The Extent and Importance of Unintended Consequences Related to Computerized Provider Order Entry

Abstract

Background

Computerized provider order entry (CPOE) systems can help hospitals improve health care quality, but they can also introduce new problems. The extent to which hospitals experience unintended consequences of CPOE, which include more than errors, has not been quantified in prior research.

Objective

To discover the extent and importance of unintended adverse consequences related to CPOE implementation in U.S. hospitals.

Design, Setting, and Participants

Building on a prior qualitative study involving fieldwork at five hospitals, we developed and then administered a telephone survey concerning the extent and importance of CPOE-related unintended adverse consequences to representatives from 176 hospitals in the U.S. that have CPOE.

Measurements

Self report by key informants of the extent and level of importance to the overall function of the hospital of eight types of unintended adverse consequences experienced by sites with inpatient CPOE.

Results

We found that hospitals experienced all eight types of unintended adverse consequences, although respondents identified several they considered more important than others. Those related to new work/more work, workflow, system demands, communication, emotions, and dependence on the technology were ranked as most severe, with at least 72% of respondents ranking them as moderately to very important. Hospital representatives are less sure about shifts in the power structure and CPOE as a new source of errors. There is no relation between kinds of unintended consequences and number of years CPOE has been used. Despite the relatively short length of time most hospitals have had CPOE (median five years), it is highly infused, or embedded, within work practice at most of these sites.

Conclusions

The unintended consequences of CPOE are widespread and important to those knowledgeable about CPOE in hospitals. They can be positive, negative, or both, depending on one’s perspective, and they continue to exist over the duration of use. Aggressive detection and management of adverse unintended consequences is vital for CPOE success.

Introduction

Implementation of computerized provider order entry (CPOE), defined as a computer-based system that allows a physician or other ordering authority to directly enter medical orders, can be an effective mechanism for improving patient safety and practitioner performance. ^1–14 There can be unintended consequences as well, however, ^15–24 and it is likely that fear of adverse unintended consequences is impeding the diffusion of CPOE throughout hospitals in the U.S. ^25–29

The unintended consequences of CPOE are especially intriguing because they can surprise implementers. Adverse unintended consequences unfortunately can become impediments to health care quality. The terms “unintended consequences” and “unanticipated consequences” are not synonymous. The “unintended” implies lack of purposeful action or causation, while the “unanticipated” means an inability to forecast what eventually occurred. Either kind of consequence can be adverse or beneficial. Unanticipated beneficial consequences are actually happy surprises. Unanticipated, unintended adverse consequences capture news headlines and are often what people imagine when they hear the term “unintended consequences.” An important goal of the current work is to identify the types of unintended consequences, so that they can be monitored and managed: once unintended consequences are predicted or detected, their management can knowingly involve tradeoffs. An example CPOE-related tradeoff is the degree to which the extra time physicians must exert to use CPOE (an undesirable consequence), is offset by the increased information physicians find available at the point of care via CPOE—making patient visits more effective. ³⁰ The ability to maintain control over consequences may give hospital decision makers more confidence when making the determination to implement CPOE.

Recent descriptions in the literature of the unintended consequences of patient care information systems include a taxonomy of kinds of errors, ³¹ a list of kinds of errors related to CPOE, ¹⁸ a hierarchical model of unintended consequences of CPOE, ³⁰ and a more general summary of the unintended consequences of patient care information systems. ¹⁹ In addition, the authors of the current study previously described in detail a typology of unintended adverse consequences of CPOE. ³² However, none of the previous studies provided a comprehensive, semi-quantitative description of the extent and importance of unintended consequences in hospitals across the nation. We therefore transformed our descriptions of types of unintended consequences into survey questions to discover how widespread and important the consequences are. We tested and, using short telephone interviews, administered the resulting instrument to representatives of 561 U.S. hospitals that have reported that they have installed CPOE systems, out of a total of 5,759 accredited hospitals. ³³

Preliminary Work

The current study protocol for this three year-project, as proposed to the granting agency and approved by appropriate human subjects committees, included an extensive qualitative study to be followed by a national survey. In the initial qualitative study, we used semi-structured interviews and observations at five sites with experience in using CPOE to find out what types of unintended consequences exist. ³² Study sites for qualitative data gathering included: Wishard Memorial Hospital, a county hospital in Indianapolis which uses a system developed by the Regenstrief Institute; Brigham and Women’s Hospital in Boston, which developed its own system; Massachusetts General Hospital in Boston, which uses a modified version of that same system; The Faulkner Hospital in Jamaica Plain, MA, which uses a commercially available system; and Alamance Regional Medical Center in Burlington, NC, which also uses a commercial product. Institutional review boards at Oregon Health & Science University, Kaiser Permanente Northwest, and each of the individual study sites approved the qualitative study.

An experienced interviewer (JSA) conducted all of the taped interviews following an interview guide of questions. Thirty-two individuals, representing clinical, administrative, and technology roles, were formally interviewed. A multidisciplinary team of Ph.D.-level informatics researchers, physicians, a nurse, and a pharmacist conducted 390 hours of observation with a purposive sample of 142 individuals representing different clinical disciplines. The team performed 120 person-hours of observation at Wishard, 144 at the three Boston sites, and 126 at Alamance. For each visit, five to six researchers spent four days at each site in the period between August 2004 and April 2005. Observations took place in medicine, surgery, emergency, and intensive care areas in addition to the pharmacy, outpatient offices, and clinics.

Qualitative data analysis followed accepted principles for assuring trustworthy results. ^34–38 All field notes and transcripts were entered into QSR N6, a qualitative analysis software program, and were individually coded by team members. The multidisciplinary team met 36 times to review transcripts and agree that what was seen or discussed was an unintended consequence and of what type.

Results of this preliminary study included identification of nine types of unintended consequences. Each of these categories can include desirable as well as undesirable consequences, depending on one’s point of view. However, our focus has been on those that are undesirable (adverse consequences), because institutions must manage them. The nine types are:

• “More/New Work Issues: Physicians find that CPOE adds to their workload by forcing them to enter required information, respond to alerts, deal with multiple passwords, and expend extra time.

• Workflow Issues: Many unintended consequences result from mismatches between the clinical information system (CIS) and workflow and include workflow process issues, workflow and policy/procedure issues, workflow and human computer interaction issues, workflow and clinical personnel issues, and workflow and situation awareness issues.

• Never Ending Demands: Because CPOE requires hardware technically advanced enough to support the clinical software, there is a continuous need for new hardware, more space in which to put this hardware, and more space on the screen to display information. In addition, maintenance of the knowledge base for decision support and training demands are ongoing.

• Paper Persistence: It has long been hoped that CIS will reduce the amount of paper used to communicate and store information, but we found that this is not necessarily the case since it is useful as a temporary display interface.

• Communication Issues: The CIS changes communication patterns among care providers and departments, creating an “illusion of communication,” meaning that people think that just because the information went into the computer the right person will see it and act on it appropriately.

• Emotions: These systems cause intense emotions in users. Unfortunately, many of these emotions are negative and often result in reduced efficacy of system use, at least in the beginning.

• New Kinds of Errors: CPOE tends to generate new kinds of errors such as juxtaposition errors, in which clinicians click on the adjacent patient name or medication from a list and inadvertently enter the wrong order.

• Changes in the Power Structure: The presence of a system that enforces specific clinical practices through mandatory data entry fields changes the power structure of organizations. Often the power or autonomy of physicians is reduced, while the power of the nursing staff, information technology specialists, and administration is increased.

• Overdependence on Technology: As hospitals become more dependent on these systems, system failures can wreak havoc when paper backup systems are not readily available.”³⁹

Methods

Survey Development

We developed an interview script, which included a short description of the purpose of the survey, five questions about the kind of CPOE system in place, and eight questions about possible unintended consequences. The five questions about the system were designed to collect descriptive information about the degree to which the system is infused into the institution. Infusion of technology is defined as the extent to which one uses an innovation in a complete and sophisticated manner, ^40–41 and it occurs as information technology applications become more deeply embedded within the organization’s work systems. ⁴²

Though we had previously discovered nine types of unintended adverse consequences, we decided not to ask interviewees about one of them, paper persistence, because as the role of paper has changed from a long-term storage medium to a temporary memory aid and disposable display device, its persistence should not now be considered adverse. The unintended consequences typology developed from the qualitative study was therefore transformed into eight survey questions, which were pilot tested (see ▶). The questions were designed to be as neutral as possible to avoid bias. We prefaced the questions by stating that the study concerned “what organizations have learned about CPOE.” We asked about surprises, “things that happened that you didn’t expect,” so the focus was on both unintended and unanticipated consequences; responses might then involve either desirable (beneficial) or undesirable (adverse) effects. The questions were asked first as yes/no queries to determine if the respondent thought the unintended consequence existed at that site (0 = No). If the answer was yes, the importance was determined on a 1 to 5-point Likert scale ranging from 1 for “not at all important” to 5 for “very important.” We received institutional review board approval from Oregon Health & Science University and Kaiser Permanente Northwest to conduct the survey.

Table 1.

Table 1 Unintended Consequences Survey Questions ^∗

Question 1. More work/New work	We think of computers as labor saving devices, but we all know that sometimes they’re not. Are there examples in your institution of new kinds of work that you didn’t do before?
Question 2. Workflow	We have noticed in our research that when CPOE systems are in use, this alters how people do their work. Have you seen this?
Question 3. System Demands	The information system typically needs a great deal of support in terms of maintenance, training, updating order sets, etc. Has this been an issue in your organization?
Question 4. Communication	Communication is really important in clinical care. Have you seen any alterations in communication patterns because of CPOE?
Question 5. Emotions	We have seen many emotional responses to the system. Have you seen users express strong feelings about CPOE?
Question 6. New Kinds of Errors	CPOE has been proposed as a solution to patient safety issues, but may have created others. Have you seen new patient safety issues with CPOE?
Question 7. Power Shifts	We have noticed the balance of power may shift when CPOE is used. Have you noticed that at your organization?
Question 8. Dependence on Technology	As we become more dependent on technology, we’ve noticed that people may have a hard time when the CPOE system is not available. If your computer went down, would this be a significant issue for your organization?

^∗ All questions asked, “Is this important at all, Yes or No. If Yes, on a scale of one to five, with one being not at all important, and five very important, how important has this been?” (“No” was rated as 0).

Survey Administration

We attempted to contact the entire population of U.S. hospitals using CPOE, instead of a sub-population from which we would make statistical inferences, because we wanted to gain insight into the nature of unintended consequences related to CPOE from organizations beyond those we visited. We selected acute care hospitals from the 2004 HIMSS Analytics™ Database that reported having CPOE in place (N= 448). Since that database did not include U.S. Veterans Affairs hospitals, which the authors believed to constitute important models of CPOE use, we also surveyed them (N= 113). We contacted hospital staff listed as appropriate contacts via electronic mail and then made phone appointments for the 10 to 20 minute survey. Follow-up included multiple phone calls until a knowledgeable informant was found. Each question asked the interviewee to respond on a 5-point Likert scale about the extent of the problem; researchers took notes on anything that was volunteered beyond that. Numerical data were analyzed with descriptive statistics and the text of comments was analyzed qualitatively, again with the aid of N6. Whenever a respondent spontaneously commented while answering a question, the interviewer wrote down the comment. At the end of the interview, we asked if the interviewee had anything else they would like to share, and these comments were also recorded.

Survey Data Analysis

Quantitative data were analyzed using SPSS software (V14.0, SPSS Inc. Chicago IL). We used descriptive statistics to understand the degree of CPOE infusion in the institutions. We explored the correlation between the length of time CPOE has been in place and the extent to which unintended consequences were present. Using bed size and ownership as established proxies for “other important organizational characteristics in studies of hospital organizational issues,” ⁴³ as well as geographic location and teaching status, we compared respondents to non-respondents using logistic regression.

Qualitative data in the form of notes taken during the interviews were analyzed using QSR N6. This analysis simply entailed grouping comments with the questions to which they related, each representing a type of unintended consequence.

Results

We were able to establish contact with 265 of the 561 hospitals and discovered that 89 of those listed in the HIMSS Analytics™ database did not actually have CPOE, so they were dropped from further consideration. We therefore conducted 176 full interviews. Thirty-four additional institutions had policies against responding to surveys, so they were added to the list of non-respondents. Using the Institute for Social and Economic Research (ISER) calculation for response rate, ⁴⁴ which takes into consideration the ineligibility of some sites (e.g., listed as having CPOE when they actually do not), our response rate was 47%. To estimate how representative our responding sites were of all hospitals that have CPOE, we categorized the sites in four ways: bed size, management type, teaching hospital status, and geographic location ⁴⁵ because these measures are considered important differentiators. ⁴³ Logistic regression (forward, stepwise, likelihood ratio) comparing respondents and non-respondents showed no difference with respect to hospital bed-size and teaching hospital status. Management type differed between respondents and non-respondents, in that the VA hospitals were most likely to respond, followed by other governmental hospitals, private institutions and non-profits (compared to VA, governmental LR = 0.63, private not for-profit LR = 0.38 and private for-profit LR = 0.35; p < 0.001). There was also a difference in geographic location with a site in the Northeast region most likely to be a responder, followed by the West, South and the Midwest, in that order (compared to South, Northeast LR = 2.11, West LR= 1.65 and Midwest LR = 0.93; p = 0.007).

Infusion Levels

The length of time that CPOE had been in place ranged from 6 months to 25 years (median = five years). The percentage of orders entered electronically ranged from 1–100% (median = 91%). Greater than 96% of the sites used CPOE to enter pharmacy, laboratory, and imaging orders; 82% were able to access all aspects of the clinical information system with a single sign-on; 86% of the respondents had at least three types of clinical decision support (order sets, drug-drug interaction warnings, and pop-up alerts); and 90% had a CPOE committee in place. The overall infusion levels were therefore at the high end along all measures. ⁴⁰

Extent of Unintended Consequences

▶ summarizes the responses to the eight questions about unintended consequences. For six of the questions, the responses indicated that the unintended consequences were important in those institutions. Workflow was rated as a 3, 4, or 5 by 88%, communication 84%, dependence on technology 83%, system demands 82%, emotions 80%, and more work or new work 72%. For two of the questions, many respondents answered that the unintended consequence had not been seen or was less important than others. Changes in the power structure was rated as 0 by 36% of respondents, and either 1 or 2 by 29%. New kinds of errors were rated 0 by 9% and either 1 or 2 by 44%. We considered that the number of years that CPOE had been in place might affect the occurrence of these consequences to either a lesser or greater extent. The response rating for these questions was evaluated for a correlation between it and duration of use in years and no correlation was found for any of the eight questions (Spearman’s rho statistic). ⁴⁶

Table 2.

Table 2 Responses to the Survey Questions Regarding Unintended Consequences ^∗

	Number (%)			n
Category	NO	YES Less Important	YES Moderately to Very Important
More work or new work	8 (5)	40 (23)	125 (72)	173
Workflow	6 (4)	15 (9)	149 (88)	170
System demands	10 (6)	21 (12)	143 (82)	174
Communication	8 (5)	20 (12)	146 (84)	174
Emotions	8 (5)	28 (10)	140 (80)	176
New kinds of errors	15 (9)	77 (44)	82 (47)	174
Power shifts	61 (36)	50 (29)	61 (36)	172
Dependence on the technology	14 (8)	15 (9)	138 (83)	167

^∗ The numbers indicate the absolute counts and the percentage of the total responses to that question. One question about each category asked if it exists with a yes or no answer possible. If a respondent answered yes, that the hospital had experienced this, he or she was then asked to rate importance on a scale of one to five, with one being not at all important, and five very important. “No” was rated as 0. In this table, the “yes” values are collapsed with a rating of 1 or 2 shown as less important, and 3–5 as moderately to very important.

Types of Unintended Consequences

Sixty-four pages of qualitative data were analyzed from the free text responses, with representative quotes from respondents shown in ▶. Comments are summarized below:

• More/New Work: Since large unanticipated changes in the amount and nature of work seem to be a regular consequence of CPOE, we asked how great a problem it is hospital-wide and discovered there are many kinds of new work generated. Over 72% said it is a moderate to very important issue. Work increases because 1) sometimes the systems are slow, 2) non-standard cases call for more steps in ordering, 3) there are training issues, 4) some tasks become more difficult, 5) with more data available more is done, 6) nurses have to be diligent about taking orders off regularly, 7) the computer “forces the user to complete all steps,” and 8) information technology staff take on many new responsibilities. Respondents often noted that there were also marked shifts in work, so that the physician must do tasks formerly done by others.

• Workflow: We asked if CPOE altered workflow and to what extent. 87% of respondents said that it is a moderate to very important issue. Many commented that CPOE increased efficiency, which is generally an expectation and therefore not an unintended consequence. Often, however, interviewees noted that while it improved workflow for some workers, it also negatively affected the workflow of others, especially physicians, who must put extra time into ordering. Some commented that it is hard to identify the effect of CPOE vs. other technological advances in communications such as cell phones, and others noted that the workflow was negatively affected in the beginning but positively affected later on.

• System Demands: We asked about the maintenance, training, and support efforts that are ongoing and the extent of the problem. 82% of informants said it is a moderate to very important problem. Demands are greatest during the first few months, but even after that “it takes an army to build and maintain” a system, even a commercial one. There is need for continuous equipment upgrades. A positive yet unintended consequence described often is “demand from the organization exceeding our supply,” meaning that expectations escalate or, as another interviewee said, “there is such an appetite for continued growth.” One informant strongly stated “do not do this unless you can support it.”

• Communication: The next question asked about alterations in communication patterns among clinical staff, with 84% of respondents noting that this was an important outcome of CPOE, but not necessarily for the better. Beliefs were strong one way or the other, with about half of the comments indicating that CPOE has cut down on the need for clarifying orders and the other half saying that people no longer communicate adequately. Many respondents commented that their organizations had to make a concerted effort to educate clinicians about using face-to-face communication when it is appropriate and not to rely solely on the system.

• Emotions: We asked if users tend to express strong feelings about CPOE and 80% said it is a moderate to very important issue. Many interviewees noted that reactions go “in both directions.” One informant said “pick your favorite terms of praise or profanity. They are all used.” Physicians get upset because it takes more of their time, but most tend to accept CPOE over time. The strength of the emotions is often unanticipated, along with the change over time from strongly negative to strongly positive.

• New Kinds of Errors: Although CPOE has been proposed as a solution to the medical errors problem, there are also new kinds of errors generated as a result of computerization. We asked if respondents had seen new patient safety issues with CPOE. 9% said it was not a problem and only 47% said it is moderately to very important. The respondents cited entering orders on the wrong patient, errors of omission, nurses not knowing an order had been generated, desensitization to alerts, loss of information during care transitions, wrong medication dosing, and overlapping medication orders. There were many comments that safeguards have been built into the systems, that there are many near misses but few actual errors and they tend to be minor, and that there is “a potential for new issues [errors] so awareness is vital.”

• Power Shifts: We asked how the balance of power shifts when CPOE is used. Although shifts in the power structure were clearly evident during our fieldwork, our interviewees did not feel they were very important. 36% said it is not a problem, 29% said it is a less important issue, and only 36% thought it was a moderate to very important issue. They noted that power flows away from physicians to pharmacists, nurses, information technology staff, and administration. Some said it is a phenomenon that occurs early during the implementation effort, that CPOE is used as a way of forcing or monitoring physician behavior. Others said that there is “not so much a change in the balance of power, but it does shine a light on how decisions are made in a very public way.”

• Dependence on the System: In fieldwork, we found that dependence on CPOE and surrounding systems is expected, but the extent of this dependence was an unintended outcome. Clinicians eventually become extremely dependent on the system, so downtime creates problems. We therefore asked about consequences when the CPOE becomes unavailable so that we could discover the extent of the problem. This is indeed an issue for over 82% of respondents and comments indicated that it gets worse over time. Most interviewees said that backup processes go into effect if there is downtime, but that “it is basically managed chaos with a very negative impact on productivity.”

Table 3.

Table 3 Comments from Respondents

Type of Unintended Consequence	Respondent Comments
More work and New work	• Needs additional educational effort and surveillance by nursing and pharmacy to prevent ADE’s • Orders are more labor intensive; repeat orders are timesaving with the system, though • Most examples of new work that come to mind really are not new, but only new to the person doing it • Prescribers are now responsible for much of the ordering and documenting that used to be done by support staff • People expect the computer to save them time, but there is a learning curve while trying to get to know the system and that creates work • More data can mean that there is more to manage and how to do that effectively becomes an issue. CPOE has made some things definitely faster • Upfront work takes more time which is the downside, but information is there and clearly available for everyone • Time saved overall, but individually some see more work than others
Workflow	• Has different effects on different jobs and people • Especially the physicians • Sequence of workflow is often required to be changed; new steps are added • Same work, different groups performing the tasks • Establishing remote access, many doctors have already checked their labs and work online before coming to work or doing rounds, which makes their floor practices easier
System demands	• It takes an army to help build and maintain X system, assuming you have not hired the vendor to do this work. There are about 50 people who support the X system directly in our IT department and it still is not enough to keep up with the demand • The need for constantly upgrading old computer equipment is a huge issue. It is a constant tug-of-war between the cost of new equipment and the cost of user time and frustration having to use equipment that is slow or in need of repair • Huge undertaking, a lot on the software side, there were cultural issues as well • Demand from the organization exceeding our supply so getting priorities sorted out once you establish the mandatory maintenance allocation is critical • Training and maintenance have been huge • Order sets are the biggest challenge and modification of them once in the working system is very intensive • Weren’t aware that so much support resources would be needed; we way under forecasted • Do not do this unless you can support it! • We’ve tackled the beast with 6 full time workers, each specializing in a particular area
Communication	• People seem to forget how to communicate, even for the most basic, routine matters. They expect the computer to tell them what to do, every step of the way • I think there is less face-to-face communication between providers • It’s always a challenge in a complex environment because the computer makes things invisible; we should educate everyone in what the system can and can’t do to uncover misconceptions and silent communication patterns • People began to assume the system had it so they did not need to tell someone • Initially, they didn’t think they had to talk any more and that isn’t the case! It is much improved over time • Improvement because patient records accessible anywhere
Emotions	• Computer phobics had a hard time; they identified each additional minute the physicians spent because of the system • There have been both strong advocates and strong opponents of the system • Doctors mostly see CPOE as a good tool and beneficial for others • In the beginning, resistance, now no • At first, everyone was upset. But now residents rely on it; they don’t know how to use the paper system • Pick your favorite terms of praise or profanity. They are all used • It’s an inanimate object, so easier to express it toward system than toward a human; people expect computers to be working all the time and when they don’t people are angry • A small but vocal minority hate it • Generational, but mostly favorable • Many reactions on both sides of the fence. Some doctors were upset because they felt like they were being asked to be typists. Others are happy because they can navigate the computer to find the information they need quickly • Some love it and other keep hoping that it will just go away • Most have been willing to adapt and have had positive responses. There was one case where a doctor told the implementation staff that they were ruining his life • Doctors don’t like feeling dumb so when they don’t know how to use the system, they get frustrated and angry
	• Someone asked me How does it feel to be the most hated person in this institution? • A doc threw a computer at me! The screaming is slowly improving after 3-4 years of meetings
New Kinds of Errors	• Wrong patient being selected. We now have an extra step to confirm the patient’s identity • Potential, but only a few minor errors so far; paper mistakes only mess up one record, but the computer has the potential to mess up multiple records • People think that computers should catch any errors in the system and therefore don’t examine things as closely • With computers you have to think in black and white and patients are in the gray • Issues have shifted. Medication problems have been reduced significantly, but there are new things like errors of omission • Orders entered on wrong patient, takes away a few cues, location, color, thickness of chart—easy to transpose numbers when typing—this is why decision support is important to safeguard • Desensitized to alerts; do things without challenging the computer • Someone gets numb looking at alert messages and skips by an important one • At points of transfer of care or areas where two systems abut and are not integrated • Lose critical thinking abilities. Don’t question order sets • Physicians have stated in the early phase that they forgot to order things as they were distracted—but over time, they feel absolutely more safe • Insufficient training and system glitches can cause safety issues with higher reliance on machines • Potential for new issues so awareness is vital
Power Shifts	• Was 1 [low] with the paper system, 5 [high] with new system, and down to 1 again after people got used to the new system • Pharmacy feels threatened … doctors are now more in charge of medications like they are meant to be • Ancillary departments have gained much more power over how patient records and patient care are handled • With the ability of the computerized system to force and monitor behaviors, primary care providers have much less control over their workflow • CPOE puts more power into the IS department • You mean the CIO becomes king? • Computer developers need to be careful and just provide the tools, not the rules • Not so much a change in the balance of power, but it does shine a light on how decisions are made in a very public way • Knowledge is power. There are now tracking patterns • Mostly I think physicians have lost power, but I also think this is happening more generally
Dependence on the Technology	• The organization implemented a downtime procedure, but some staff have never had to work with the backup paper systems before • If we lost CPOE, other programs/systems that are programmed to run in parallel would be lost as well • Despite some very well-designed and well-intentioned downtime processes, it is basically managed chaos with a very negative impact on productivity • Downtime is difficult, but even more so is when the system comes back online and they have to manage all the data from paper back into the system and that creates a lot of duplication of the work • Much harder if it’s down more than three hours

Discussion

Our preliminary qualitative study identified major types of unintended adverse consequences of CPOE. This survey study goes beyond the prior study describing these types by advancing our knowledge about how widespread the problems are. The survey results verified the existence of the eight unintended adverse consequences types we asked about at most of the 176 hospitals with CPOE, and analysis of comments has offered insight into the perceived importance of the consequences. All types of consequences are indeed widespread although two of them, power shifts and new kinds of errors, were not considered as important as the others. While the current study was based on a categorization of adverse unanticipated consequences of CPOE from the prior study, we found that there are positive as well as negative unintended consequences. Often, the same consequence can be viewed different ways by different people depending on their perspectives.

Our mixed methods approach, using extensive fieldwork to gather data about types of unintended consequences at excellent sites, followed by a phone survey to verify the existence of these same types across the country and to gain further insight, yielded rich data.

The median length of time organizations have had CPOE in place is five years, which is relatively recent since systems have been available since the early 1970’s, ⁴⁷ although only a few pioneering hospitals had it prior to ten years ago. Interestingly, there appears to be no correlation between the importance of each type of unintended consequence and the length of time that a system has been in place. Despite the recent timing of many implementations, the systems are highly infused, indicating that they are deeply embedded into the workflow, heavily used, and sophisticated.

One limitation of this study is the difficulty of asking for a simple score in response to a complex question. This difficulty was offset by allowing respondents to provide comments after each question—a distinct advantage of telephone surveys. The ability of the researcher to give examples and explain questions was helpful, but it is possible that some respondents may have misinterpreted some questions. There was no attempt to test the reliability and validity of the instrument. We asked to speak to someone in the organization who is most knowledgeable about CPOE, and we usually spoke to someone on the information technology staff with a clinical background. While these individuals seemed to offer balanced answers, they might have given answers biased in a positive way because they believe in CPOE’s benefits. This might help explain why the two categories of power shifts and new kinds of errors did not rank highly: often the power shifted to administration or information technology staff, so informants may not recognize their own gain in power. New kinds of errors may seem minor to these interviewees who seem confident that these errors are most often caught before harm is done. In addition, as representatives of these hospitals, informants may have been reluctant to discuss the sometimes-sensitive subject of errors in heath care. As previously noted, there was a bias toward eliciting adverse unintended consequences, based on the survey taxonomy being derived from a prior study of adverse consequences.

Responding sites tended to differ from non-respondents by management type, with VA hospitals the most likely to respond and for-profit centers, the least likely. Perhaps VA centers are proud of their systems and/or staff members are simply more willing to take the time to talk about them. Unfortunately, many organizations have policies against doing surveys, an increasingly common problem. ⁴⁸ The reason for geographic differences in the likelihood of response is unclear. As stated, there was no difference in the level of response with respect to bed size or whether the site was a teaching institution.

Once unintended consequences are revealed, hospitals can work to either try to avoid them or to accept them as tradeoffs. For example, the negative impact of information technology on clinical workflow noted by others ^49–52 is especially true of CPOE because it takes more of the physician’s time to enter orders than in the manual process, ^53–54 but this might be accepted as a tradeoff for the added safety benefits of CPOE.

Our chief recommendation is that developers and implementers of CPOE systems consider each of the eight types of unintended adverse consequences carefully during their planning. The workflow issues are immense, and they affect nearly every hospital staff member. Communication changes, ⁵⁵ and implementers must take steps to prevent or offset the altered communications created by electronic systems in this respect. As people become more and more dependent on such systems, ⁵⁶ preparations for downtime must be more elaborate and serious. More work and new work are inevitable, as are increasing system demands, so ample resources must be made available on an ongoing basis. Finally, emotions run high for and against CPOE, ⁵⁷ and efforts must be made to understand and manage them. Expectation management is therefore paramount. Informants indicated that power shifts and new kinds of errors were less important kinds of unintended consequences, but we must nevertheless be diligent about identifying them. ^18,58–63 Unless we make a concerted effort to avoid, manage, and/or overcome unintended consequences, the implementation of clinical information systems may lead to detrimental results. ^64–67

The eight kinds of unintended adverse consequences of CPOE we investigated are experienced by most CPOE sites, regardless of length of use of CPOE. CPOE is highly infused, or deeply embedded, at most of these sites, which is surprising considering that most implementations of CPOE are relatively recent. The telephone survey has given us insight into the extent and importance of unintended consequences. Management of possible unintended consequences is vital for maintaining control and avoiding problems. Implementation success depends on managing the unintended, as well as the intended, consequences, of CPOE.