Alert Override Reasons: A Failure to Communicate

Abstract

In 2007, the Leapfrog CPOE standard required that all clinical alert overrides be accompanied by an override reason. We wanted to know how many of the free text comments left by clinicians were actually override reasons, and how many were other types of communication. We reviewed 3583 free text comments left voluntarily by clinicians while responding to an alert in a CPOE system. Of the comments received, 58% were override reasons, 28% were acknowledgement of the alert, 9% were content free and over 5% were misdirected communication, written with intent to reach someone who did not receive the alert comments. This is particularly concerning because much of the misdirected communication contained clinical instructions. Those clinical instructions were stored with the alert rather than with any clinical orders, and thus were not viewed by anyone receiving the orders. Our results show that free text alert comments may cause communication failures.

Introduction

One of the advantages of CPOE, according to the Leapfrog group, is that it improves communication between physicians and pharmacists.⁽¹⁾ Unfortunately, that is not always the case. Sometimes a CPOE system instead creates the “illusion of communication”, giving the impression that information, once entered into the system, necessarily will be seen by the correct person.⁽²⁾

Several reports have commented on the complexity of health care communication^(3;4). Other reports have focused on how CPOE changes communication patterns. CPOE changes the ordering process from a team process, where orders can be verbally explained, but are often poorly documented and incomplete when written, to an individual process where orders are extensively documented, but vulnerable to misinterpretation.⁽⁵⁾ Many errors are facilitated by the constricted nature of communication through a CPOE system.⁽⁶⁾

The implied opportunity for health care communication provided by an acknowledgement comment field within an alert can be tempting for clinicians who wish to communicate with someone on the other side of the system. Previous studies that have evaluated clinician’s alert response text have largely focused on override reasons, clumping all text that was not an override reason into a single category.^(7;8) In this study, we will examine alert response text with a different categorization scheme, focusing on when the text is not an override reason.

Methods

Our institution uses a commercial CPOE system which generates alerts largely at order entry. The legacy pharmacy system with which our CPOE system communicates does not have space to display alert comments. Although this will change as soon as the involved systems are upgraded, at the time of this study, alert comments were not communicated to the pharmacy system, only stored in the CPOE database.

For this study, several queries were developed for the database underlying the CPOE system. First, the database was queried for the count of alerts triggered during a representative month (August 2007), and for the count of non-empty alert comments. Second, the database was queried for all non-empty alert comments since the system was implemented in January, 2005.

The text of each comment, together with alert specific information, such as the triggering order, the type of alert and the date of the alert were collected. At no point was the database queried for any identifying information on any patient or clinician.

Alert comments from the initial and final presentation of each alert were consolidated, and comments from the pre-implementation phase were removed from the query results. Each comment retrieved from the database was then read by the primary investigator, an internist, and placed into one of four categories. Samples of each category can be seen in Table 1.

Table 1.

Alert Comment Categories

Category	Representative Comments
Acknowledgement	“ok”, “noted”, “md notified”
Content-Free	“dd”, “hjhj”, “***”
Misdirected Communication	“peg tube okay for meds and feeds” [DC NPO alert] “WAS NOT DRAWN!!!! NEEDS TO BE DRAWN...DO NOT CANCEL!!!” [Duplicate Order alert] “please hold glipizide, rosiglitazone and metformin in am” [NPO and Hypoglycemic Drugs alert] “paged superuser multiple time, emailed tech support. no response. unable to obtain any technical assistance” [Patient Allergy alert]
Reason	“will order a diet” [DC NPO alert] “Hemolyzed” [Duplicate Order alert] “Insulin also discontinued” [NPO and Hypoglycemic Drugs alert] “Pt has previously tolerated” [Patient Allergy alert]

The requirements for placing an alert in the reason category were minimal. A response such as “done” to an alert which recommended a particular action was accepted, as were any comments indicating that the alert was invalid, that an action had been or would be taken, or that the alert was or would be discussed with another party.

A comment which consisted solely of acknowledgement of the alert, without any override reason was considered acknowledgement, while a comment which did not have any apparent meaning was considered content-free.

Any comment which appeared to be written with the expectation that a human being would read the comment and act upon it was placed in the misdirected communication category.

For validation, 50 random alert comments were selected, and a second clinician was asked to categorize each comment.

Data Analysis

The relative proportions of alert comments in each category were calculated, as well as the kappa statistic and agreement with the second clinician.

Alert comment categorization was then organized by year, to evaluate whether the type of comments left changed over time. The relative percentage of alert comments in each category were calculated for each year. The number of comments varied by year, as the first year was during implementation, and the last year only has partial data. However, since we are only interested in the relative proportions of the alert comments in their categories, the absolute number of comments per year is not important.

Finally, alert comments were organized by the type of alert to determine whether different types of alerts elicited different distributions of comments. Only the six alert types that received more than 100 comments, and which formed 89.1% of all the alert comments received, were included in this analysis. The chi square test was used to evaluate for heterogeneity in the distribution of the categorization.

Results

In August 2007, 930 clinicians received 30,590 alerts on 892,501 orders and generated 105 comments, a comment rate of 0.34%. 3,583 alert comments were recorded between 1/18/05 and 9/25/07. The 50 comments that were rated by two clinicians had an agreement of 86%, for a kappa of 0.75. The number and relative proportions of the alert comments can be seen in Table 2.

Table 2.

Categorization of Alert Comments

Category	Total	Percent
Acknowledgement	995	27.8%
Content-Free	311	8.7%
Misdirected Communication	199	5.6%
Reason	2078	58.0 %
Total	3583	100.0%

As shown in Figure 1, when comment categorization was evaluated by year, the percentage of comments that were content-free grew from under 3% during the first year of implementation to just over 12 % during the third, while the percentage of comments that were misdirected communication shrank from just over 11% during the first year to just over 4% during the second and third. The percentage of acknowledgement comments grew during the second year and shrank during the third, while the percentage of reason comments shrank during the second year and grew during the third.

Figure 1.

Alert comment categorization by year

When comment categorization was evaluated by alert type, as shown in Figure 2, marked heterogeneity was found with p< 0.0001. The following six alerts each received more than 100 comments during the study period.

Figure 2.

Alert comment categorization by alert type

The DC NPO (Discontinue Nil Per Os/ nothing by mouth) alert warns when an NPO order has been discontinued and the patient has no current diet. The Duplicate Order alert warns when two of the same item are ordered in close temporal proximity. The NPO and Hypoglycemic Drugs alert warns when an NPO order is placed and the patient is already on hypoglycemic medications. The Patient Allergy alert warns that the patient may be allergic to the drug which has just been ordered. The Renal Impairment Dosing alert warns if an ordered antibiotic dose is higher than recommended given a patient’s kidney function, or if there is no creatinine available.

Finally, the Warfarin alert provides a historical list of past doses and INRs that pop-up when Warfarin is ordered.

The highest percentage of reasons were seen for the Patient Allergy and Renal Impairment Dosing alerts while the DC NPO and Warfarin alerts had the lowest. The highest percentage of acknowledgements was seen in the DC NPO alert, while the Patient Allergy alert had the lowest. The highest percentage of content-free comments was seen in the Warfarin alert, while the NPO and Hypoglycemic Drugs and the DC NPO alerts had the lowest. The highest percentage of misdirected communication was seen in the NPO and Hypoglycemic Drugs alert, while the Patient Allergy alert had the lowest.

Discussion

This study indicates that in an environment where entering alert override reasons is entirely voluntary, fewer than 0.5% of clinicians write anything. Further, of the alert comments received, only 58% were actually override reasons, while 8.7% were meaningless. Of more concern is the high rate (5.6%) of misdirected communication regarding potentially very important clinical safety issues.

Any misdirected communication has the potential to negatively impact patient care. If a clinician believes that the appropriate party is reading their alert comments in a timely fashion, yet the comments are merely stored in the database and displayed to no one, vital patient care instructions may be overlooked. Some systems do communicate these alert comments downstream, while others do not.⁽⁹⁾ This inconsistency may exacerbate the problem of misdirected communication, since many clinicians may have been exposed to multiple different CPOE systems, or multiple different implementations of the same system.

Although the implementation decision to archive alert comments without displaying them within the CPOE, pharmacy, lab or radiology systems was specific to our site, the factors underlying this decision are common. The commercial pharmacy system at our site is in wide use and does not contain the capacity to display alert comments. Many health care organizations have legacy auxiliary systems which would similarly be unable to display alert comments.

In addition, many clinical alerts are generated from a conflict between orders, such as drug-drug interaction alerts or duplicate therapy checks.⁽¹⁰⁾ Some clinical alerts, such as those triggered by abnormal lab values, do not refer to any order. Although our CPOE system, wherein alert comments are not displayed with orders at all, may be an extreme example, it seems unlikely that any system will be able to attach alert comments to one appropriate order in all cases.

The count of misdirected communication may be even larger than these numbers show, since the second rater called 5 items misdirected communication, while the primary rater only rated 3 items as misdirected communication. Conservatively, we had one misdirected communication every 5 days, or every 3.5 working days during the study period. We do not know whether or not there were any adverse events associated with misdirected communication.

As illustrated in Figure 1, there was a large decrease in misdirected communication together with an increase in content-free reasons after the first year, possibly indicating that clinicians were learning that their alert comments were not read. Even in the second and third years since implementation, however, a 4% rate persisted of misdirected communication. This persistence may be due to the turnover of clinicians in a teaching hospital, or may be due to the desire of clinicians to communicate even with an unpromising mechanism.

Different alerts had markedly different combinations of responses. This may be due to the type of alert, as well as to whether or not the alert was marked as requiring acknowledgement. For instance, the Patient Allergy alert is well known to the medical community, which may help explain why such a large proportion of the comments left for these alerts are actual override reasons.

In this CPOE implementation, although all alerts include an acknowledgement comment field, only some alerts are marked as requiring acknowledgement, as illustrated in Figure 3. Even those alerts which require acknowledgement only require that the acknowledgement button be pressed, not that a comment be left, although clinicians may have misinterpreted the directive. The Warfarin alert, which does require acknowledgement, was the only alert in the study which generated more than 100 comments, but was more informational than critical. The absence of a specific order critique in this alert, combined with the required acknowledgement, may be related to the relatively high proportion of content-free comments.

Figure 3.

Alert Acknowledgement Comment Field

The 2007 Leapfrog standard which mandated that CPOE systems “require that physicians electronically document a reason for overriding an interception prior to doing so”⁽¹⁾ has been modified in 2008 to exclude this requirement⁽¹¹⁾. This may indicate that more sites than ours have experienced similar problems with documenting alert override reasons. Despite the voluntary natures of alert comments are out site, only 58% were actual alert override reasons. Extrapolating from our experience with our Warfarin alert, if alert comments were mandated, the proportion of content-free comments might increase significantly.

Some alerts were triggered by one order, but contained advice applicable to a different order, such as the NPO and Hypoglycemic Drugs alert, which is triggered by an NPO order, but recommends changing the hypoglycemic drug order(s). The high rate of misdirected communication for this alert supports that idea that alerts should be equipped with a mechanism to allow order modification from within the context of the alert.^(9;12). Such a mechanism might provide an outlet for much of the misdirected communication seen in these alert comments.

One limitation of this study was the relatively low rate for the collection of alert response comments. Because comment collection was not enforced, it is possible that the majority of comments were left by a small group of clinicians.

Another limitation is the generalizability of this study. Although misdirected communication may be common, the type described in this study, wherein alert comments are not displayed at all may be an extreme example. Other CPOE systems were not investigated during this study.

Conclusion

This retrospective survey of comments left by clinicians in response to alerts has illustrated a major concern, that of misdirected communication which could lead to patient care errors. Content-free comments illustrate another concern, the time and cognitive burden on clinicians when using CPOE systems.

There are several options for reducing misdirected communication. Alert override reasons could be constrained to a list of likely override reasons, rather than providing a free-text comment field. Alert response options could be created which enable clinicians to create appropriate orders and order modifications from within the context of the alert, or even to generate well-directed communication. Any of these may decrease the prevalence of misdirected communication, although we will need more studies to determine how well each of these possible interventions would work.