Some Unintended Consequences of Clinical Decision Support Systems

Abstract

Clinical decision support systems (CDS) coupled with computerized physician/provider order entry (CPOE) can improve the quality of patient care and the efficiency of hospital operations. However, they can also produce unintended consequences. Using qualitative methods, a multidisciplinary team gathered and analyzed data about the unintended consequences of CPOE, identifying nine types, and found that CDS-generated unintended consequences appeared among all types. Further analysis of 47 CDS examples uncovered three themes related to CDS content: elimination or shifting of human roles; difficulty in keeping content current; and inappropriate content. Three additional themes related to CDS presentation were found: rigidity of the system; alert fatigue; and potential for errors. Management of CDS must include careful selection and maintenance of content and prudent decision making about human computer interaction opportunities.

INTRODUCTION

Clinical decision support systems (CDS), broadly defined here as computer-based systems offering “passive and active referential information as well as reminders, alerts, and guidelines” [1, p. 524], are an important component of computerized physician or provider order entry (CPOE). CPOE is direct entry of orders via computer by physicians or others with the same privileges. In fact, CPOE alone may offer little benefit without CDS [2,3]. Together, CPOE and CDS can decrease medical errors [4,5] and improve hospital efficiency [6] and practitioner performance [7]. CPOE can also generate unintended adverse consequences (UACs) [8–10], which the authors have studied for the past three years. After holding a conference of experts and gathering field data at five sites, we identified nine types of UACs related to CPOE: workflow issues; new kinds of errors; changes in communication patterns; more or new work for clinicians; never ending system demands; changes in the power structure; overdependence on the technology; paper persistence; and emotional arose during analysis was that many of the unintended consequences were related to CDS. To answer the question “what are the unintended consequences of CDS?” we conducted a detailed analysis of all CDS-related examples.

METHODS

Expert Panel

We held an expert panel conference with 19 experts in April of 2004 to begin identifying kinds of unintended consequences of CPOE. Transcripts of all sessions became data sources for analysis.

Site Selection and Description

A group of experts nominated hospital study sites based on reputation for excellence in their use of CPOE, geography, and type of organization. Three had locally developed systems: Wishard Memorial Hospital in Indianapolis, IN uses the Regenstrief system, with considerable CDS; Brigham and Women’s Hospital in Boston, MA uses its homegrown system, also with extensive CDS; and Massachusetts General Hospital in Boston uses a version of the Brigham’s system, but with less CDS. The two sites with commercial systems were: The Faulkner Hospital in Jamaica Plain, MA, using MediTech, with some CDS; and Alamance Regional Medical Center in Burlington, NC, which uses an Eclipsys product, also with some CDS. The study received human subjects approval from Oregon Health & Science University, Kaiser Permanente Northwest, and each of the sites.

Informant Selection

At the sites with locally developed systems, we interviewed developers, implementers and informaticians, pharmacists, physician users, laboratory and medical records staff, and others suggested by local principal investigators. We conducted observations and informal interviews with users in a wide variety of inpatient and outpatient settings. We deliberately selected a range of users from enthusiastic to skeptical. At the hospitals with commercial systems, we were unable to interview developers, but we did interview local staff who customized aspects of the system and on-site vendor staff, along with users.

Data Gathering and Analysis

A multidisciplinary team of physicians, Ph.D. researchers, a pharmacist, and a nurse visited each site for 3–4 days. We conducted a total of 390 hours of observation of 95 clinicians and 32 formal interviews. Detailed descriptions of our methods have been published elsewhere [11]. Qualitative data analysis software (QSR N6) assisted with analysis of 1,849 pages of data. Once all of the unintended adverse consequences were identified, we reviewed all 324 of them to select 47 related to CDS and conduct axial coding [12] to gain further insight.

RESULTS

Of the 47 examples, 13 were described during the conference of experts, 20 were found during field work at sites with homegrown systems, and 14 were identified at hospitals with commercial systems.

We found two major patterns, with unintended consequences generated either by 1) the actual content of the decision support module, or 2) the presentation of the information on the computer screen. Interestingly, the conference transcripts yielded only two examples about presentation out of 13 and the sites with locally developed systems yielded only four presentation examples out of 20. On the other hand, the commercial sites yielded examples equally divided between content and presentation (seven and seven).

CDS Unintended Consequences Related to Content

The examples related to content grouped around three themes: the elimination or changing roles of clinicians and staff; the currency of the CDS content; and wrong or misleading CDS content. Representative quotes are in italics.

Elimination or shifting of human roles

Prior to CPOE, clerical staff, pharmacists, and nurses often double checked orders, but CDS content is sometimes designed to eliminate the perceived need for such verification. The ability of the system to provide assistance with scheduling orders is sometimes suboptimal, whereas prior to CPOE, clerks had helped, by monitoring x-ray orders, for example. An interviewee noted: We probably underestimated initially the gatekeeper function that the clerical staff [performed] questioning daily x ray orders after a certain amount of time. . . once we automated, chest x-ray orders went on ad infinitum.

Medication ordering is also an issue. By taking the nurse or pharmacist out of the loop and requiring the physician to manage the dosing with inadequate CDS, there can be problems. As noted in fieldnotes: Glitches with IV drips. Instead of ordering the dose per time you have to order it in drops per time, which means you have to know what the quantity of IV solution it is in and the docs then have to manually calculate the drip rate. Similarly, when ordering is done from locations away from the patient, with CDS but without full information about the patient and available assistance from other staff, problems arise. An interviewee stated: The doctors have to write the orders into the computer, and the RT isn’t always there beside them, so they are either asking the nurses what the settings are, or they have to go find RT or go check the ventilator themselves.

Currency of the CDS content

Updating CDS content becomes necessary either because outside influences like CMS and JCAHO mandate it, or because new knowledge becomes available. Even the excellent organizations we studied had difficulty keeping up with CDS changes. Coding for billing or compliance and difficulties updating order sets and rules caused problems. One expert described a coding update issue: CMS codes. change periodically. . .so if you’re using a code whose definition changes because of the government, you’re in trouble. On day one it was seven tests, on day two it was eight tests because the government redefined it. Updating to comply with JCAHO can also be a formidable task. One clinician noted: there are apparently 4000 places where “cc” is used instead of “ml” (the abbreviation now recommended by JCAHO). I can’t imagine how much work it is going to take to review all of the screens to find them, and what the incidence of new error might be during the fix. Development of order sets and building local knowledge into the system can make CDS more acceptable, but updating is problematic. An expert stated: these proliferating practices of order sets are outta [sic] control. I mean it was beneficial at first. . [but] they don’t manage [update] them personally.

Updating the algorithm-based rules behind the CDS is equally difficult: he notes that the references for the rules are often out of date and become a bone of contention, particularly when the staff doesn’t want the rule.

Wrong or misleading CDS content

This broad category includes practical issues like new CDS modules that encourage ordering even when the hospital does not have adequate supplies, alerts that are inappropriate, and information that is not trusted.

If CDS leads clinicians to order something that is not adequately stocked, there can be problems. For example, one expert stated: when reminders are introduced to remind people about doing Hemoccult tests or pneumococcal vaccine or whatever, it is that you need to make sure that the inventories of those supplies are adequate because you very quickly run out of those things. Alert content can be problematic. Inconsequential alerts are especially annoying. As one physician said: ninety-five percent of the alerts that were generated to physicians turned out to be inconsequential to the patient and that’s a problem. Sometimes clinicians do not agree with alerts. Fieldnotes noted: they would not turn on the drug allergy alerts until they were assured that there was a way to filter out specific allergy alerts that they did not agree with. Inappropriate alerts are also frustrating. As one resident said: for example, the alert to not use broad-spectrum antibiotics such as vancomycin is not appropriate in ICU. . . that’s why patients came to the ICU, to get the vancomycin.

Contradictory advice offered by alerts can be confusing, like this example where an alert warned the user against ordering something, but the system (perhaps through an order set), suggested that it be ordered: hey, you’re forcing me to place an order, a replacement order, that I may have seen an alert about that made me not want to place it. And sometimes decision support on the computer is simply inadequate. As one physician said: The online formulary kills me. It is much easier to work from a book. . . I may just want to know how fast to push a med and the online pharmacy only shows me the pharmacology of the med.

There are sometimes data quality problems which cause clinicians to mistrust information. Medication reconciliation is often an issue: our list of medications is actually a list of medications dispensed, which the patient may or may not be taking at the time you see them. There can also be mistrust about the provenance of the data. Fieldnotes said: if the source for an allergy is [hospital] X, they will not accept the allergy as being valid until they verify it independently. And one resident said about a cost reminder: I know this lab costs more [than what the clinical decision support says], so I just ignore the other costs of the labs.

CDS Unintended Consequences Related to Presentation

Many unintended consequences of CDS stem from the way alerts and other modes of decision support are presented to the user. These appear to be caused by the rigidity of systems, sources of alert fatigue, and sources of potential errors.

Rigidity of systems

The balance between the need for a system to gather and use structured data and the need for clinicians to be able to work easily and quickly is sometimes upset by CDS. Physicians have been known to use workarounds to avoid spending time entering required data. From fieldnotes: An alert that required a numerical entry and the physicians were just putting a one in. Workflow can be interrupted. One interviewee stated: one of the unintended consequences is that you can make the workflow much harder by inserting the computer. Lists arranged a certain way can be a problem: folks may think there is no dose matching what they want. The lists cannot be sorted. Finally, linear order sets may not mirror the complex reality of ordering: current order sets are organized in a linear fashion when he thought most of the problems were multidimensional.

Alert fatigue

We found more examples of alert fatigue, when clinicians feel that there are too many alerts, than any other aspect of CDS. Below are just six of many examples:

1. Drug-drug interactions-- most are ignored. 2. Alerts about weight based dosing, this is a real problem. Weights can’t be entered into the system, so this is difficult to override. IT can’t seem to disable this either. 3. We over-alert. We either need to put teeth into our warnings or don’t do them at all. 4. If you d/c a lab test that is part of an order set, a popup appears letting you know it is part of an order set. 5. Because of so many alerts and pop-ups, the docs may blithely click OK without a reason and not even read it. 6. It seems like nearly every drug has an alert associated with it.

Sources of potential errors

Many of the problems outlined above can lead to errors, but we found several additional ones that seem to be particularly error prone. Although often helpful and time saving, auto-complete features can sometimes be problematic. An interviewee noted: we have synonyms for common misspellings. The doc puts that in and an order goes into the patient’s chart with the kind of accepted name. They’ll come back later and look at that and say “I didn’t order that.” Timing is often a problem, with examples of alerts being seen when it is too late for action, or delayed action. Fieldnotes stated: X was never notified the lab test never happened, so neither did the ER, and the patient had to stay in the hospital overnight. This greatly increased the cost of the hospital stay.

Updating can lead to typing errors. Fieldnotes noted: I cannot imagine what the incidence of new error might be during the fix, such as eliminating an element in a pick list by accident or making a typo in some drug name. Several subjects pointed out that clinicians might not pay attention to important alerts because of over-alerting. Two different informants called it “crying wolf”: 1. I want systems to “quit crying wolf.” 2. From fieldnotes: He is concerned That so many [alerts] will create a cry wolf phenomenon where important alerts may be ignored. It is especially prevalent with drug-drug interactions.

DISCUSSION

The unintended consequences that arise because of CDS can of course be avoided if little CDS content is used. Without CDS, however, the major benefits of CPOE cannot be realized. The sites we studied that had locally developed CDS had more issues with content and less with presentation than did the sites with commercial systems. This is most likely either because the hospitals with commercial systems have a minimal amount of content or that users have very little control over the presentation specifics within these systems.

The hospitals with home grown systems that we studied tended to have more CDS content because of the way in which their CDS content has been developed. Individual clinicians, who were often the developers, recognized needs and built the CDS content little by little over time. Modules were added gradually, usually after considerable testing.

The hospitals we studied that had commercial systems had less CDS content. Sites with commercial systems tend to lack CDS content for several reasons. First, the vendors may not have the desired mechanisms for evaluating or displaying the clinical content in the way clinicians want it. Second, community hospitals that are not also teaching hospitals may not have the time and interest in creation of new clinical decision support content. Third, the ability to customize content is limited in commercial systems, although hospitals may want extensive customization. Finally, both the hospital and the vendors may feel that there could be legal issues if content is customized [13].

It appears that presentation of CDS is a problem for all hospitals, regardless of whether they have locally developed or commercial systems. A good part of this is undoubtedly because there is a natural tension between the need to notify the clinician before he makes a mistake and the inability of the computer to know the thought process of the clinician. In addition, the requirements of these systems to collect and use structured data and the ability for the system to allow flexible data entry methods are in direct conflict. When the system is designed so that the clinician must enter data into a certain field or must give an explanation, using a menu, for overriding decision support, the system is collecting important data. However, the clinician may resent this intrusion on his or her time and may perceive that the system is questioning his judgment.

One limitation of this study is that our sites were selected because they are successful: it is likely that the problems we discovered are more severe elsewhere. Another is that we did not set out to study CDS, although its emergence as a major underlying cause of CPOE-related unintended consequences emphasizes its importance.

We have a number of recommendations for managing the CDS issues we have outlined. Those related to content can be addressed by having a knowledge management (KM) structure in place. KM is an organized method for the selection, development or customization, organization, and maintenance of CDS modules. Some excellent suggestions for organizing KM have been published [14–16]. If issues arise because human roles are shifted, the KM plan can assure that staff besides physicians are involved in the choice and implementation of CDS so that workflow is considered and needed double checking continues. The currency of CDS can also be assured when a KM structure outlines methods for acquiring new knowledge and regularly reviewing and updating each module. Constant updating is desired and inevitable and a thorough review process before a module goes live is necessary. Content that is thought to be “bad” for any reason can be addressed through a KM structure by having a process in place for clinicians to easily provide feedback. And, finally, problems with the quality of the data entered into the system are serious, because CDS cannot accurately respond to patient data in the system if data are erroneous [17]. These problems can also be addressed through the KM plan if continuous training of users includes not only making them aware of new CDS, but also for increasing awareness about why the data are needed and why users need to be careful when entering structured data. A sound organizational structure for KM must include clinicians from different specialties and departments within the organization. In addition, it is imperative that a process for periodically measuring and reviewing key clinical decision support performance metrics is in place.

There are some possible remedies for the presentation issues we uncovered. The rigidity of the systems is often caused by the need to capture structured data. However, if it is not necessary and it is interfering with clinician workflow, it can and should be reconsidered. Alert fatigue can be mitigated by reducing the number of alerts. This can be done by carefully selecting a set of alerts that can have the most impact [18] and, if it is possible with the system, by filtering by severity levels of interactions [13]. Unfortunately, the ability to modify CDS is severely limited in many commercial systems. Clinical information systems vendors need to work closely with their customers to better understand CDS needs. Finally, there is need for research about CDS use in hospitals using commercial systems so that the lessons being learned now by those on the forefront can be passed to others.

CONCLUSION

During our study of the unintended consequences of CPOE, we found 47 examples of consequences related to CDS. After detailed analysis of them, we found that they related either to the content of the CDS or to the presentation. While these unintended consequences could be avoided completely if no CDS is implemented, without CDS, CPOE cannot offer the benefits that can lead to safety improvements. Incorporation of patient-specific clinical decision support within CPOE systems is critical for success in addressing the many challenges facing the modern healthcare industry.