AMDIS Case Conference

Abstract

Clinical decision support that provides enhanced patient safety at the point of care frequently encounters significant pushback from clinicians who find the process intrusive or time-consuming. We present a hypothetical medical center’s dilemma about its allergy alerting system and discuss similar problems faced by real hospitals. We then share some lessons learned and best practices for institutions who wish to implement these tools themselves.

1. Introduction

In this feature, a fictional vignette illustrating a common applied clinical informatics dilemma is presented and discussed by two expert informaticians who have addressed similar difficulties in their own institutions. The article ends with the authors’ commentary.

A community medical center has been unable to implement drug-allergy alerts in their inpatient electronic health record (EHR) system. The hospital completed an initial implementation of their EHR system from a major commercial vendor 3 years ago. The system is capable of results reporting, clinical documentation including free text notes and problem lists, and computerized provider order entry (CPOE). After the hospital recently attempted to activate the EHR’s built-in medication safety module, numerous department heads complained to the IT Department that the alerts were too intrusive and disruptive to their workflow, and CPOE usage transiently dropped to 20% of all orders. The drug-allergy alert system was turned off last week. Yesterday, a patient with a known penicillin allergy received ampicillin and developed anaphylaxis, requiring intensive care. Today, the hospital board is asking you what went wrong.

2. Case Reports

2.1 Case Report #1

Lehigh Valley Health Network, a tertiary care center in Allentown, PA, has implemented CPOE throughout its 900 acute-care beds, including critical care and Emergency Department. After a five year implementation, CPOE was mandated throughout the hospital with approval by the Medical Executive Committee in 2006. Current utilization is 95%. This includes orders entered by attending physicians and residents.

To streamline governance and facilitate engagement by the many independent community-based physicians, only two committees were used to drive the project. The Oversight Committee consisted of C-level leaders, Medical Staff leaders, and the Physician Liaison. This group provided input to the implementation and served to support the project when resistance arose from the medical and support staff. The Physician Liaison was a practicing pediatrician who committed more than 50% of his time to the CPOE implementation and reported to the CIO.

The Physician Design Team was comprised of a multidisciplinary group of clinicians, nursing, ancillary and IT staff. The group met every other week to review screen design, workflow and other design related issues.

LVHN operates a closed loop medication administration process. In the essential first step of the closed loop, the clinician enters the order electronically into the system. Clinical decision-support alerts pop up during this process. These alerts include allergy and drug interaction alerts, dosing alerts based on creatinine clearance and other clinical data, reminders regarding missing or inappropriate orders, and other rule-based alerts. Although the majority of alerts generate “soft stops”, several require user input of a reason to override the alerts (e.g., overriding drug allergy alerts). Alerts regarding therapeutic substitutions provide alternate choices on the alert screen for the clinician to move forward.

The admitting nurse or physician enters allergy information. The system is designed such that pharmacy staff must review and confirm allergies prior to verifying any medication orders.

Clinicians raised concerns about over-alerting after the first CPOE unit was brought live. This occurred prior to the introduction of any of the more complex clinical decision-support rules. Physicians complained in several venues (department meetings, hallway conversations) that the alerts slowed down the order entry process, and that the majority of the alerts were not clinically relevant. A common example concerned “duplicate potassium” orders when ordering IV fluids and medications containing potassium. Direct communication occurred with the Physician Liaison, who made regular rounds of the hospital to receive feedback. The Physician Design Team also discussed the issue in great detail.

In response, the Physician Liaison contacted the vendor to express concerns. Various options for tailoring of the alerting system were suggested, including filtering the alerts by user role. However, it was concluded the proposed solutions were not technically feasible, and the alerts in the system either had to be “all alerts always on” or “all alerts always off.”

Given the inability to filter the alerts, the Physician Design Team and the CPOE Oversight Committee determined that potential benefits to patient safety outweighed the inconveniences of keeping the alerts in the “all alerts always on” status. The organization emphasized its strong commitment to CPOE and patient safety, and at Department and Division meetings communicated the decision throughout the organization.

CPOE utilization was measured during and subsequent to the entire implementation process. Verbal orders were not included in the total, but tracked in a separate report to ensure that no physicians were abusing the ability to use verbal orders.

In 2007 when CPOE utilization was at 95%, we undertook a review of clinical decision-support as part of a National Library of Medicine IAIMS Planning Grant (Integrated Advanced Information Management Systems), including a questionnaire distributed to all physician users. There were 80 responses to the questions regarding clinical decision support and the alerting functions in the system. In response to the question “Do you the read the pop-ups and alerts?” 18% responded “never” or “rarely”; 16% responded “occasionally”; 66% responded “sometimes” or “all of the time.” The results illustrate that despite the concerns raised by physicians, the majority of end-users clearly paid attention to the decision-support provided by the alerting system. When asked, “Do you think clinical decision support tools improve the quality of care?” 93% responded “yes.”

CPOE and embedded clinical decision support has clearly improved patient safety and quality at LVHN. Our analysis of potentially significant medication errors demonstrated a statistically significant reduction in potential events after implementation of CPOE and CDS (►Figure 1). At the ordering stage of the medication cycle there was an 80% reduction in potential significant medication errors.

Fig. 1.

Medication error rate for category B and higher event reports. Category B events are those in which an error occurred but the medication did not reach the patient (as defined by USP Medication Errors Reporting Program).

In summary, when faced with a situation similar to the one in the case vignette, LVHN utilized recognized physician leaders and CPOE governing bodies to resolve the issue. They explored the problem in detail and proposed alternative workflows and technical solutions to the vendor. When there were no easy fixes, the leadership had sufficient respect and authority to insist that the medical staff overcome inconvenience in the name of patient safety.

2.2 Case Report #2

Holy Spirit Hospital in Camp Hill, PA, is similar to the community hospital in the case study. With just 331 beds and an IT staff of 35, the hospital has confronted the challenge of implementing EHR functionality and CDS with limited resources. Most of the over 500 active members of the physician staff are community based. Seventeen employed hospitalists see 90% of medical patients.

CPOE was implemented in February 2008, six months after the system’s initial go-live. Initial activations were in the orthopedics unit soon followed by the cardiovascular ICU, both locations where mid-level providers primarily entered orders. Initial users underwent substantial training and reported no problems with false positive alerts during CPOE.

Subsequently an “exponential incrementalism” approach was taken to CPOE implementation. All beds were CPOE-eligible by August 2008, and well over 100 physicians and mid level providers were trained in CPOE by the end of 2009. By March 2010, CPOE accounted for 47% of all nonverbal orders. CPOE utilization is expected to reach 75% by the end of 2010, at which time its use will be mandated.

In 2008, within two months of training, the hospitalists began to complain about the intrusiveness of drug-allergy alerts. Allergy alerts were perceived as so disruptive that these physicians threatened to stop using CPOE entirely. An analysis determined that clinicians proceeded with medication orders approximately 90% of the time that an alert fired, suggesting they were ignoring the majority of the alerts. It was then discovered that the early users (the CMIO and mid-level providers) reviewed allergy data before entering orders, but the newer users were not following the same workflow.

The IT team analyzed the alerts, responses, and associated comments. Further investigation confirmed that allergy data from a legacy database had been appropriately preloaded into the new system. It appeared that clinicians were skipping allergy verification with the patient before entering orders. The unverified allergy list triggered an alert for every medication order entry, and was understandably intrusive. The perceived lack of utility of the alerts resulted in clinicians ignoring them.

It became obvious that this key workflow point had been underemphasized in training. Reeducation included re-emphasizing to the nursing staff the need to review and verify allergies with the patients. The expected workflow was reviewed with clinicians at special meetings with the hospitalists and during one-on-one sessions on the clinical units. ►Figure 2 shows the marked improvement in review rates for allergies after these discussions occurred in May 2009 (the 10% unreviewed allergies as of December 2009 represent patients discharged from the emergency room before allergy review).

Fig. 2.

Allergies reviewed during visit. Improvement in review of allergy review after workflow re-education in May 2009

Continued analysis revealed that alerts appeared even after allergy review when the patient’s allergy was recorded as “Other”. Staff had been using “Other” as a catchall choice despite having more appropriate options. Examples included using “Other” in place of “No Known Allergy;” unclassified food allergies; or when the user could not locate the drug in the existing list.

It was determined that the commercial drug information database generated a generic “unmapped” alert against an entry listed as “Other,” which clinicians saw as irrelevant to their decision-making. Correcting the “Other” entries in the new patient database eliminated 33% of these unnecessary alerts. ►Figure 3 displays the resultant reduction in unmapped alerts after making these corrections. ►Figure 4 shows the impact of improved nursing workflow on the rate of review of allergies before first medication administration. Clinicians expressed greatly increased satisfaction with the system as a whole after the reductions in false allergy alerts.

Fig. 3.

Frequency of unmapped allergy alerts. Reduction in unmapped allergy alerts after data clean-up (the peak in September 2009 represents a programming error during an upgrade that was quickly corrected).

Fig. 4.

Allergies before orders. Frequency of documentation of allergy review before first medication order for inpatients, including treatment in the emergency department

Several organizational strengths contribute to avoiding the outcome experienced by the hospital in the vignette. The information technology technicians and Chief Medical Informatics Officer at Holy Spirit work closely with the clinical staff and various associated committees. Communication and ongoing reevaluation are critical to all clinical decision support efforts. The ability to do a detailed analysis of the root cause of the problem and acting quickly helped avoid further downstream problems.

3. Discussion

The community medical center in the vignette has made substantial progress towards complete adoption of their electronic medical record. After 3 years of using an EHR, it is likely that most clinicians were comfortable with the hardware and software and had adopted a culture of safety. Those physicians not using the EMR would almost certainly not have been bothered by the alerts. The problem that the physician leaders encountered must therefore have been in the workflow of the alerting process for those who were using the EMR.

3.1 The Value of CDS

The evidence in support of clinical decision support (CDS) is compelling. CDS provides information at the point of care that a provider can use to improve patient care. A recent meta-analysis [1] convincingly showed that 85% of CDS studies demonstrated improved outcomes, including correct dose adjustment of medications in patients with renal impairment, and in achieving reduced length of stay. Kawamoto, et al, [2] showed that 68% of decision support trials significantly improved: compliance with protocols and guidelines; percentage of clinicians who responded to patient conditions requiring attention; and appropriateness of certain radiograph orders. Drug-allergy checking and alerting is one of the simplest yet most important CDS tools used in electronic order entry systems [3]. Why would a hospital want to turn off this function?

3.2 CDS Interface Issues

There are many kinds of alerting functions in computerized records. The most common – asynchronous pop-ups – are the most intrusive. “It is important to avoid overburdening clinicians with requests [or information] that interrupt their workflows, and . . . the system should only ask clinicians for information about which they are the definitive source.” [4] The alert must be “actionable,” meaning that one can both cancel the original order and enter a new one from within the alert message box or activity. Additional user-interface considerations include whether the alert is a “hard stop” (i.e., cannot be bypassed in any way); whether a clinician must give a reason for proceeding past the alert; and how the reason for bypassing the alert is documented (e.g., drop down menu, free-text, etc.).

A controversial area of allergy alerting concerns whether an allergy is a true reaction or an intolerance. Some clinicians feel that alerting rules based on the latter create “false positives”. Some intolerances, however, might be important, for example for a patient who experienced severe esophagitis with acetaminophen/codeine. Standardized terminology is particularly immature in this area, leading to numerous idiosyncrasies among different vendor products and interfaces.

3.3 Alert Fatigue

Physician resistance to CPOE is often due to alert fatigue. The precise threshold false positive level at which physicians will begin to ignore alerts or begin to resist use of an electronic medical record is not well documented and is probably unique to each physician. We hypothesize that this threshold depends directly on the perceived criticality of the information presented, and inversely to the total number of alerts received in the course of an ordering session.

Although current systems include the ability to act upon the information from within the alert, these alerts may still be perceived as interruptive. A certain percentage of alerts will always represent false alarms [5], but there is a paucity of empiric data to guide us regarding an acceptable level of appropriate sensitivity and specificity. Alerts for life-threatening allergic reactions will clearly need a different sensitivity level compared to intolerances, which are important but clearly not as dangerous. Without a doubt, we need more sophisticated alerting systems [6].

Clinicians have expressed concerns about personal liability for an electronically auditable record that shows that an alert was displayed. Medico-legal issues may play an important role in user acceptance of CDS systems. Informaticians must be cognizant of these concerns and work closely with physician leaders and legal advisors to ensure that clinicians are not forced into uncomfortable legal positions because of purely technical issues.

3.4 Workflow Analysis and Redesign

Electronic systems do not fix preexisting broken processes. If a workflow is faulty, exact computer mimicry will magnify that failure by amplifying unintended consequences [7] and is doomed to failure. The hospital in the second case report was forced to ask some important workflow questions as part of its investigation: were previously known allergies available in the current electronic record? Were physicians verifying them each time a patient presented for an episode of care? How complex was it for clinicians to confirm or change a patient’s allergies? These and other questions regarding workflow need attention before implementing a clinical support function.

In most hospitals, nurses as well as physicians obtain allergy histories. In some institutions, pharmacists are also responsible for this task. One could predict opposition if the physicians had never been responsible for entering allergy data, and now encountered unexpected alerts. The first step before instituting a new function is to analyze the existing workflow [8]. If the workflow is successful then the electronic process that mimics that workflow has the greatest likelihood of success. Any change in the pattern of the workflow can create disruption, and therefore requires early involvement of the users, careful planning, validation by physician (and other) champions, and then education and training in advance of the change [9].

3.5 End-User Training

Successful deployment of electronic records (and CDS for physicians in particular) depends on proper communication and training for the end users. When new features go live, physicians need to be informed prior to altering their workflow and routines. Even if physicians were involved in the creation of these alerts, the communication to other physicians of the go-live may be inadequate. These factors can make or break an EMR installation [10].

Indeed, without sufficient training and encouragement, end-users will eventually find the path of least resistance and create work-arounds to get their work accomplished. Clinicians may choose an inappropriate item from an alert override pick list, one that involves no additional justification, just to move past an alert they believe is unwarranted or invasive. As this behavior spreads among clinicians, the alerting system loses credibility, and the resultant data becomes increasingly unsuitable for quality assurance purposes.

3.6 Governance and Leadership

Clinical decision support is very complex with multiple potential points of failure. Organizations with transparent, accountable governance structures with dedicated clinical leadership will be more likely to maintain the sustained effort and resources required to succeed. The first case report hospital used the strength of its clinical leadership structures to provide sufficient resources and encouragement to help overcome their allergy alert problems.

A clear vision of the benefits and impact on patient care and quality of CPOE, and consistent/persistent reinforcement of that vision leveraged by commitment by senior administrative and medical leadership are critical success factors in implementation of CPOE and associated clinical decision support such as alerts and rules.

3.7. Troubleshooting CDS Issues

Clinical decision support is one of the most important means of improving healthcare quality and safety. Simply turning off a bothersome alert in response to concerns from the rank-and-file represents a failure to recognize and communicate the fundamental value of EHR. Failure to correct the workflow disruption while continuing to leave the alert in place will exacerbate antagonism and reduce trust among clinicians for further IT interventions.

Even if certain CDS functions need to be deactivated for troubleshooting, the onus is on the organization to introduce stopgap or downtime procedures to ensure patient safety [11]. Rather than reflexively deactivating the alerts, the hospital in the vignette should have begun by reexamining the workflow of allergy data entry and verification. They may have needed to redesign these processes before reinstituting the alerts. Organizations need to achieve agreement regarding accountability and responsibility regarding clinical alerts. Diagramming workflow often helps to define the steps. Clinicians must be directly involved in all phases of alert redesign, and the appropriate clinical governance group must agree that a new process is ready for activation.

4. Conclusions

Those impacted by CDS must be involved in the design, education and implementation of alerts. It is important to recognize that CDS is not always generally accepted on faith. Physicians object to “cookbook” medicine, and perceive CDS as “being told what to do.” Accordingly, there should be “five rights” for alerts, just as for medication administration and CDS in general [12]. An alert must be:

• 1.generally accepted as important to clinical care;
• 2.provided at the decision point without interrupting workflow;
• 3.pertinent to the individual patient based on individual data derived from the current clinical record;
• 4.require minimal clinician effort, and
• 5.actionable in a manner that suggests the correct response from the clinician [13].

The “rights,” and their applicability to the cases presented here, are shown in ►Table 1.

Table 1.

Lessons learned, according to the “five rights” of allergy alerting.

Allergy alerting best practice	Alerting perceived as intrusive	Example	Successful alerting
Pertinent to the individual patient based on individual data derived from the current clinical record	Generic alerts not specific to the current patient context	Allergy warning for a drug the patient is currently taking and tolerating	Alerting rules should take into account the full spectrum of available patient information.
Provided at the decision point without interrupting workflow	Before user has a chance to enter a relevant order	Alert that patient has an allergy upon opening chart	Only after user has entered order for that allergen
Require minimal clinician effort	Allergy review at a time and in chart location different from order entry	Alerts for unreviewed allergies at downstream workflow points	Present an actionable unreviewed allergy alert immediately before user attempts to place any orders
Generally accepted as important to clinical care	Inappropriate information	Alert for a food allergy when entering medication order	Alert should have a clear purpose defined by the hospital as critical to patient safety or other critical need
Actionable in a manner that suggests the correct response from the clinician	Non-actionable alerting	Alerts after orders have been signed	Provide the ability to add, change, or remove orders from within the alert, or acknowledge and bypass the alert

The final right deserves further comment. It is insufficient to simply suggest that an order is inappropriate. A reasonable alternative should be offered, and available directly from the alert screen. The options should be based on medical evidence, which can be provided in a non-interruptive manner [14] or available via linkage to an embedded web source [15].

5. Implications

In summary, medication safety alerts during CPOE hold great promise towards improving the safety and reliability of patient care. As the two case reports demonstrate, successful achievement of this vision requires consistent application of leadership and attention to nuances of clinical workflow, before, during and certainly after implementation.

Human Subjects

The procedures used in this work have been reviewed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects. These retrospective case reviews did not involve human subjects and thus were exempt from institutional review.

Conflicts of Interest

Dr. Graham is employed by Dearborn Advisors, a consulting firm that helps organizations implement clinical decision support. Drs. Levick and Schreiber report no conflicts of interest.