Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2009 Jul 1.
Published in final edited form as: Int J Med Inform. 2007 Oct 10;77(7):440–447. doi: 10.1016/j.ijmedinf.2007.08.005

Assessing the Anticipated Consequences of Computer-based Provider Order Entry at Three Community Hospitals Using an Open-ended, Semi-structured Survey Instrument

Dean F Sittig 1,2, Joan S Ash 2, Ken P Guappone 2, Emily M Campbell 2, Richard H Dykstra 1,2
PMCID: PMC2668516  NIHMSID: NIHMS54353  PMID: 17931963

Abstract

Objective

To determine what “average” clinicians in organizations that were about to implement Computer-based Provider Order Entry (CPOE) were expecting to occur, we conducted an open-ended, semi-structured survey at three community hospitals.

Methods

We created an open-ended, semi-structured, interview survey template that we customized for each organization. This interview-based survey was designed to be administered orally to clinicians and take approximately five minutes to complete, although clinicians were allowed to discuss as many advantages or disadvantages of the impending system roll-out as they wanted to.

Results

Our survey findings did not reveal any overly negative, critical, problematic, or striking sets of concerns. However, from the standpoint of unintended consequences, we found that clinicians were anticipating only a few of the events, emotions, and process changes that are likely to result from CPOE.

Conclusions

The results of such an open-ended survey may prove useful in helping CPOE leaders to understand user perceptions and predictions about CPOE, because it can expose issues about which more communication, or discussion, is needed. Using the survey, implementation strategies and management techniques outlined in this paper, any chief information officer (CIO) or chief medical information officer (CMIO) should be able to adequately assess their organization's CPOE readiness, make the necessary mid-course corrections, and be prepared to deal with the currently identified unintended consequences of CPOE should they occur.

Keywords: Medical Order Entry Systems; Ethnology; Hospitals, Community; Medical Informatics

INTRODUCTION

There is currently tremendous pressure on many different fronts for hospitals to implement Computer-based Provider Order Entry (CPOE) [1]. While there have been several excellent articles describing impressive improvements in patient safety, decreases in patient costs due to decreases in redundant laboratory testing, and increased compliance with quality of care guidelines [see 2 for an excellent review of this literature], there have also been several articles describing significant negative, and often unanticipated occurrences that our research group along with others have identified [e.g., 3,4,5,6,8].

After extensive reflection on these so-called “unintended” or “unanticipated” findings and subsequent conversations with various clinical informatics experts, we began to question whether the “unintended” or “unanticipated” consequences that we had identified were perhaps already known and expected by others in the field and routinely communicated throughout their organizations prior to any CPOE implementations. In an attempt to determine what “average” clinicians were expecting to occur in organizations that were about to implement CPOE for the first time, or one that was about to experience a significant system upgrade, we conducted open-ended, semi-structured interviews with clinicians at three community hospitals. This manuscript reports on the results of these interviews.

Background

Identification of the Unintended Consequences of CPOE

Over the past three years the Provider Order Entry Team (POET) research group has been funded by the National Library of Medicine to study the “Unintended Consequences of Computer-based Provider Order Entry.” Towards that end, we convened a group of 15 experts in clinical informatics for a two-day meeting in May, 2004 at the Menucha Retreat Center outside Portland, OR to help us better understand the types of events, emotions, and process changes that often occur following CPOE implementations. The findings from this meeting included a list of 80 unintended consequences along with suggestions on how to identify additional consequences within other organizations. We took the findings from this meeting out into the field to see if we could confirm them and learn more. We visited five organizations that have been using their CPOE systems for more than 2 years and are fully operational, that is capturing over 90% of all medical orders (i.e., medications, laboratory and radiology tests, nursing actions, ancillary procedures, etc.) directly from physicians [7. While all of these sites could be classified as “successful,” they all faced significant struggles during the long and arduous implementation process. Therefore, we were able to capture ideas, concepts, and plans that worked, as well as those that failed at all of these institutions. We interviewed numerous individuals at each site. In addition we conducted many hours of observation on the clinical units. Based on our analysis of these data, we identified nine main categories of “unintended” or “unanticipated” consequences [8] (see Table 1.).

Table 1.

A list of the unintended consequences and their frequency of occurrence as identified in previous research efforts [Error! Bookmark not defined.].

Unintended Adverse Consequences (UACs) Description of UACs Frequency (%) n = 324
More/new work for clinicians CPOE systems create new work, e.g.,: (a) enter new information; (b) respond to excessive alerts that may contain non-helpful information; or (c) expend extra time in completing non-routine, complex orders. 19.8
Workflow issues If CPOE designers have not considered the entire range of workflows, the system cannot accommodate comprehensive, fully integrated clinical processes. 17.6
Never ending system demands Caused by periodic hardware and software (e.g., upgrades) purchases, implementation and training tasks, and maintenance issues. 14.8
Paper persistence Paper becomes a portable, disposable, computer output display medium for quick reference use during the workday as well as a place for temporary, handwritten data storage for later entry into the computer. 10.8
Changes in communication patterns & practices Doctors, nurses and ancillary providers report that CPOE causes unsatisfactory reductions in face-to-face communication regarding patient care. 10.1
Emotions Shifting from paper-based order generation to CPOE is bound to evoke strong emotional responses as users struggle to adapt to the new technology. 7.7
New kinds of errors Errors can result from: problematic electronic data presentations; confusing order option presentations and selection methods; inappropriate text entries; misunderstandings related to test, training, and production versions of the system; and workflow process mismatches. 7.1
Changes in the power structure CPOE system configurations control who may do what (and when) through the use of clinical, role-based authorizations. While narrowly defined authorizations may lead to much needed role standardizations that reduce unnecessary clinical variations, the constraints may also redistribute work in unexpected ways, causing frustration. 6.8
Overdependence on technology As CPOE diffuses and becomes entrenched within organizations, clinical care delivery becomes inextricably dependent upon it. System failures increasingly wreak havoc. Embedding clinical decision support within CPOE systems increases clinicians' access to educational material and may adversely affect learning and retention. 5.2
Open in a new tab

Rapid Ethnographic Assessment Methods

As a way to begin exploring the utility of this assessment method for the evaluation of clinical information systems, we developed a short interview guide (see Fig. 1) that we customized (i.e., changing the name of the organization and the names used to refer to their clinical information system) for each site. The survey, with questions based on the categories of unintended consequences we had previously identified, was a form of a rapid behavioral survey of key stakeholders. Rapid assessment using a mix of qualitative and quantitative methods has been used effectively in the public health arena to develop and evaluate intervention programs for nutrition in primary health care [9] and HIV/AIDS [10]. Researchers use this method for rapidly sizing up a situation and offering results and insights to those inside the organization so that they can take action. Also called quick ethnography [11], rapid assessment is a way of expeditiously gathering, analyzing, and interpreting high quality field data so that action can be taken as rapidly as possible. Classic ethnography can take too long (e.g., often more than one year) to be of practical, timely use for these fast-paced and rapidly changing programs. Therefore, the goals of these methods are to rapidly assess a situation and plan an intervention based on the particular context. The reason the assessment can be expedited is that it consistently utilizes structured tools across field sites at the same time participant observation and more in-depth interviews are yielding additional high quality data. The data are then triangulated in a structured manner during analysis. Tools for data collection include 1) site inventories, 2) ethnographies, or studying a small group of people in their own environment, 3) focus groups, 4) rapid behavioral surveys, 5) stakeholder interviews, and 6) evaluation forms. This manuscript describes our use of an open-ended, semi-structured, orally administered survey of clinicians in three organizations about to begin large-scale CPOE implementations.

Figure 1.

Figure 1

Open in a new tab

Customized version of the stakeholder survey used at Kaiser Permanente, Sunnyside Hospital.

The Survey Sites

The following sections describe each of the three hospitals we surveyed. Each hospital was at a slightly different stage in the CPOE implementation pathway. One organization had begun rolling out their CPOE system on one small clinical unit. A second organization was in the early, pre-CPOE phase (i.e., Admit - Discharge - Transfer (ADT), billing and patient tracking, but no CPOE) of their roll-out. The third organization was within 2 months of their planned go-live date. All of the clinicians at each of these sites had extensive exposure to various forms of clinical information systems over the past several years, in addition to an extensive pre-CPOE communication program. All sites were implementing high-quality [12], full-featured, integrated, commercially available CPOE systems (i.e., CPOE systems created by Epic Systems, Madison, WI, McKesson, San Francisco, CA and Eclipsys, Boca Raton, FL).

El Camino Hospital, Mountain View, CA

El Camino Hospital is a 395-bed community hospital serving the Mountain View, CA area; the heart of the “Silicon Valley.” The hospital first implemented the Lockheed system (then Technicon Data Systems, Alltel, and now Eclipsys, Boca Raton, FL) in the 1970's [13] and have the longest continuously operational CPOE system in the world. At the time of the survey described in this article they were less than two months away from a planned hospital-wide upgrade to the new Eclipsys Sunrise system (version 3.5) [14. This upgrade from a thirty-year-old, character-based (40 characters × 22 lines per screen), conversational-style interface in which all potential user-selectable options are clearly visible on each screen to a state-of-the-art Windows-based application with a high-resolution (1024×768 pixel display) graphical user interface that utilizes many of the currently available data entry and display widgets (e.g., drop-down lists, check boxes, nested hierarchical menus, etc.).

Kaiser Permanente, Sunnyside Hospital, Portland, OR

Kaiser Permanente's Sunnyside hospital is a 196-bed community hospital serving Kaiser Permanente, Northwest members in the greater Portland metropolitan area. While most of the clinicians have extensive experience with their highly successful ambulatory care clinical information system (Epic Systems, Madison, WI) [15], the only clinical application in use in the hospital is an older, mainframe-based clinical results review application that relies on a character-based menu interface to provide clinicians with access to a patient's clinical laboratory and radiology results. At the time of this survey, Sunnyside hospital had just “gone live” with phase I (Admission/Discharge/Transfer, new inpatient pharmacy, hospital billing, and Emergency Department tracking systems, all from Epic Systems) of their in-patient clinical information system roll-out and were approximately four months away from the planned hospital-wide roll-out of their CPOE system.

Portland Providence Medical Center, Portland, OR

Portland Providence hospital is a 483-bed, community hospital serving the Portland metropolitan area. At the time of this survey, they were approximately four months into a CPOE pilot on their rehabilitation unit, with plans to begin a phased roll-out to the rest of the in-patient clinical units over the next two years. They are implementing McKesson's Horizon Expert Orders CPOE system (McKesson, San Francisco, CA) that is based on the system developed at Vanderbilt University over the past 10 years [16].

METHODS

Following Institutional Review Board approval of the study at Oregon Health & Science University (OHSU), Kaiser Permanente Northwest, and each study site, we created an open-ended, semi-structured interview survey template that we customized for each organization. This survey was designed to be administered orally to clinicians and take approximately five minutes to complete, although we did not stop any clinician from discussing the topics in greater depth, if they desired. We administered the survey to clinicians within each organization at common gathering places - a true “convenience” sample [17]. For example, at Kaiser Permanente we approached clinicians in the cafeteria. At El Camino, we interviewed clinicians in their computer training facility. At Providence, where many of the clinicians are community-based, a member of the clinical information system staff helped us identify clinicians in and around the cafeteria, since many were in regular “street” clothes rather than the “white coats” and “scrubs” that we encountered at the other institutions. All survey responses were transcribed for data analysis.

All members of the POET research team participated in data analysis. We analyzed each set of surveys from an organization independently and created a summary of the responses. These summaries were fed back to the Principal Investigator at each site to 1) help him better understand his organization's current state of CPOE anticipation, and 2) provide member checking [18] to validate our interpretations. We combined all of the site-specific summaries to help us make more general, and hopefully transferable, statements about average clinicians' anticipation of the consequences of CPOE

RESULTS

The Survey

We interviewed a total of 83 clinicians: 31 physicians, 31 nurses, and 21 allied health professionals at the three sites. There were no large differences in the clinical training of the interviewees within professional categories, years of professional experience, age, or gender among the sites. In addition, all of the clinicians at each site were familiar with basic clinical computing features and functions such as patient lookup, clinical results review, email, and Internet access. To preserve the anonymity of the interviewees from each site and because the answers to the questions were fairly consistent across sites (i.e., there were not any significant themes that were observed at only one site), we report the answers to each question in aggregate.

All but two respondents had at least heard about the upcoming CPOE implementation at their organization. Less than one fourth (25%) of the interviewees had tested or been trained on the systems. In general, interviewees recognized that the new CPOE systems would initially slow them down since there would be a “learning curve” associated with the new system. There were very few openly hostile or negative comments about the impending system implementation.

When asked how the new system might compare to the current system, almost all respondents were able to articulate multiple perceived advantages, including increased legibility, reduced time to find charts, less paper, improved communication, and an overall improvement in patient safety. Most interviewees were also able to describe multiple disadvantages of the new system including: more difficult to use (i.e., More work/new work from our list of Unintended Adverse Consequences, see Table 1), long learning curve (Never ending demands), more frustrating (Emotions), and worries about technical issues such as downtime procedures (Overdependence on technology). When asked about the perceived effect of the new system on other clinicians within the organization, the interviewees' responses varied from cautiously optimistic to quite pessimistic. The comments ranged from anticipated improvement in communication among departments (Communication) to concerns about more work (More work/New work), to worries about physicians taking their frustrations out on nurses (Emotions/Power). One respondent even mentioned a potential “showdown” between physicians and nurses (Power).

In response to the question about the perceived effect of the new system on patients and patient care, many interviewees recognized that any patient effect would result from the effects or process changes experienced by the clinicians rather than direct effects to the patients. Overall, interviewees thought the new CPOE system would improve patient care following the difficult implementation period. Specifically, they mentioned that care would be more streamlined, faster, and less error prone.

When asked about the perceived effect of the system on the organization as a whole, many interviewees stated that it would improve the perception of the organization in the community and perhaps save the organization money. Another broadly held opinion was that the impending change was inevitable based on their experiences in other information intensive service industries (e.g., retail sales, restaurants, airline travel, banking, etc.) and that health care institutions could no longer defer the adoption of this new technology if they are to remain competitive.

Our final question regarding “what the future of CPOE at each particular organization might look like” was difficult for respondents to separate from our earlier questions. Respondents simply reiterated answers from earlier questions therefore we were not able to answer this question.

The Report to CPOE Leadership

Following the analysis of the findings from the surveys we created and presented an individualized report to each of the three sites. This report served as a form of internal validation of our findings in addition to informing the CIOs about the state of preparedness of the clinicians within their organization. The overall state of preparedness of the clinicians at each site was judged based on a consensus of the subjective opinions of the POET research team. This consensus was reached during a debriefing session immediately following the multi-day site visit during which we administered and then analyzed the survey data, interviewed many clinicians, and carried out many different participant observation sessions.

DISCUSSION

Survey Results

While our study participants were able to describe articulately both advantages and disadvantages of their impending CPOE roll-out, there were only a few instances in which these clinicians identified any of the “unintended” or “unanticipated” consequences that we have identified in our previous research (Table 1). Based on our small convenience sample of clinicians at these three community hospital sites, it appears that clinicians recognize that during the transition period, many work processes will take longer and that there will be some new work on their part as they become accustomed to the new system. None of our interviewees mentioned that the new system would continue to slow them down after the initial learning period (as we and others [8,19] have found at multiple CPOE implementation sites). In addition, many clinicians were worried about how they would take care of patients if/when the system went down. This fits into our “Overdependence on technology” unintended consequence category. No one mentioned that the system might cause “New kinds of errors” which we and others [3, 4, 5, 8] have seen and documented at many different CPOE implementation sites. Further, none of the interviewees mentioned that the new system might have a detrimental impact on communication. In fact, many thought that the new system would improve communication. While we believe that certain types of communication are improved with the implementation of CPOE (e.g., interdepartmental communication of orders), we have also seen significant negative effects on clinical communication due to 1) clinicians remotely entering orders that are not completely understood or acted upon in a timely manner, 2) changes in clinical decisions following rounds that are not clearly explained or clearly communicated by the new orders, and 3) failure on the part of nurses to recognize that “new” orders are present for a certain patient.

In addition, only one person from the three organizations mentioned that they anticipated any changes in the power structure of the organization. We have observed many instances in which “CPOE enables shifts in power related to work redistribution and safety initiatives and causes a perceived loss of control and autonomy by clinicians.” [20]

In summary, the clinicians we interviewed had a realistic view of the impact that CPOE would have on them and their peers. They understood both the upsides and the downsides of CPOE and even seemed to have a long-term perspective, acknowledging that there could be a positive payoff at the end of the long learning curve. They did not indicate that they were aware of many of the unintended consequences of CPOE implementation that we have identified, but they were surprisingly well informed about CPOE in general.

Study Limitations

When compared to a highly powered, randomly controlled clinical trial, there are too many limitations to mention. On the other hand, when an entire field is at a very early stage in our understanding of the basic scientific and organizational questions that we must begin addressing if we are to continue improving our implementations of such complex, socio-technical projects as CPOE, then these limitations can be placed in their proper perspective. For example, the selection of the 3 study sites was purely due to chance since we were looking for healthcare organizations that were about to go live with new CPOE implementations. Likewise, our decision to use a convenience sampling strategy within each organization was based solely on our need to identify quickly a relatively small group of clinicians to interview. Both of these decisions may have resulted in a biased sample of clinicians, but without spending an inordinate amount of time and effort to insure a truly representative sample of clinicians, we could not identify a better approach.

Report Feedback

The open-ended, semi-structured survey succeeded in helping the clinical information technology leadership within each organization better understand the state of preparedness of their clinicians. The CPOE leaders to whom we communicated the survey results were very positive and appreciative of our work. All of them were relieved that their pre-implementation communication strategies were for the most part very successful. By successful, they meant that clinicians throughout their organization had received and understood their messages regarding the impending CPOE implementation. They had sensed that their clinicians had a “realistic appraisal” of the new CPOE product (that is, they held neither overly positive nor negative feelings about the system), but these results validated their feelings. In addition, they were glad that clinicians recognized that there would be a difficult transition period that would hopefully improve rather quickly. Implementation leaders were most concerned that something dramatic might happen leading to a “massive outcry” that might “stop the project right away.” Because predicting the future impacts of technology on human behavior is a notoriously difficult problem [21] and one should never assume that something that has happened in the past cannot happen again [22] or again [23], the results of this survey were somewhat comforting to the leaders. On the other hand, we found very little evidence that either the clinicians we interviewed or the CIOs were preparing for, or even aware of the possibility of, the vast majority of the unintended consequences that we have identified repeatedly during our work at multiple organizations with long-standing CPOE implementations [24]. This finding led us to develop the recommendations in the final section.

In one organization we found that clinicians were fearful there would not be enough computers. In that case, we discussed what we felt was a need for far more computing devices in the clinical areas. We were reassured that more devices had been ordered and were scheduled to arrive and be available before the go-live. To this reply, we suggested that the CIO try to communicate this message to the clinical staff as a way of reducing their anxiety. At another site, they decided to delay their CPOE implementation soon after our results were in, although it did not appear that the report of our survey results played a major role in that decision.

Recommendations to increase awareness of unintended consequences of CPOE

After further discussions among POET team members, participants from the Menucha retreat [25, 26], and additional discussions with our key contacts at each of the three study sites, we developed the following recommendations. These recommendations are designed to help CPOE implementers increase awareness of, and begin putting into place the necessary strategies to help their organizations overcome, or at least ameliorate, the unintended consequences that we have previously identified.

  1. Develop and execute a sound organizational change management plan with the help of clinicians [27]. “The level of change involved with CPOE is much greater than hospitals have faced in the past, and success requires that the organization become expert at accomplishing and sustaining change.” [31]

  2. Be willing to learn about and address issues of unintended consequences that have occurred elsewhere [28]. In addition to reading as many of the reports of CPOE implementations that have appeared in the literature as possible [29], we believe that organizations interested in implementing CPOE make a few site visits to organizations similar to theirs who have successfully implemented a similar CPOE system. More often than not there are specific lessons that can be learned from these projects [30].

  3. Clinicians must be deeply involved in the selection and implementation process. CPOE must not be an information technology department led project. Rather, it must be a clinically driven project chosen to meet one or more of the organization's highest level strategic goals, for example, to improve patient safety or overall quality of care [31]. If the project is not clinically driven, it will be difficult to obtain enough clinician input to avoid many of the workflow -related unintended consequences.

  4. Communication about the upcoming CPOE implementation must come from individuals who can speak the languages of both technology and clinical workers. Hire good people with knowledge and experience in clinical, information technology, project management, and clinical informatics areas and if possible, people with experience implementing these types of complex clinical information systems in other organizations [32]. Without input and participation from experts in all of these areas who can talk to the users, many readily identifiable and preventable consequences will arise that have the potential to derail the entire project.

  5. Provide adequate opportunities for all clinicians to be trained in and to experience using the new clinical information system before its deployment. Many organizations require that all clinicians complete a minimum amount of training before they are authorized to login to the live system and enter an order. While this may seem rather draconian, usage of new systems is especially problematic for clinicians without formal training. In addition, each clinical unit should conduct multiple walk-throughs, or dress rehearsals, with all key clinical and administrative personnel of all key clinical and administrative processes that will involve the CPOE system. For example, physicians should practice writing orders on a newly admitted patient or create the orders for discharge medications while nurses should practice acknowledging the existence of each new order and creating a work list or charting their actions using the medication administration record [30].

  6. Develop and test downtime and system reactivation procedures. The act of thinking through what will be required to continue to care for patients when the computer system is unavailable will help clinicians to see the enormity of the workflow changes that will occur following CPOE implementation [33]. Assure users that processes will be in place so that system downtime can be well managed [34].

Update on the sites studied

Over the past several months, Providence Medical Center has experienced a successful CPOE rollout (i.e., they have activated additional clinical units without any major setbacks) although El Camino Hospital has acknowledged some new medication errors due to problems with their order verification and audit procedures and general unhappiness amongst the clinical staff [35]. The CPOE rollout at Kaiser Permanente, Sunnyside was delayed until late 2007 or early 2008 for financial reasons.

CONCLUSION

The open-ended, semi-structured survey instrument proved useful in helping CPOE leaders to understand user perceptions and predictions about CPOE, and it also exposed issues about which more communication is needed. Our findings did not reveal any overly negative, critical, problematic, or striking sets of circumstances at any of the three organizations, which greatly relieved all of the CPOE site leaders. But from the standpoint of our unintended adverse consequences, we only found a few clinicians who predicted more than a small proportion of unintended or unanticipated events, emotions, and process changes that are likely to result from their CPOE implementation. Using the survey, implementation strategies, and management techniques outlined in this paper, any CIO or CMIO should be able to adequately assess their organization's CPOE readiness, make the necessary mid-course corrections, and be prepared to deal with the currently identified unintended consequences of CPOE should they occur.

Figure 2.

Figure 2

Open in a new tab

Graph showing relative frequency of occurrance of unintended adverse consequences.

Acknowledgements

This study was supported by grants LM06942-02 and LM007088-14 from the U.SA.'s National Library of Medicine. We would like to thank the site PIs for this study namely, Homer Chin, Dick Gibson, and Mark Zielazinski. In addition we would like to thank all of the clinicians who gave us their time to complete the survey. Finally, we would like to thank Ashley Jones and Caroline Weathern for their help in conducting the survey.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errorsmaybe discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

REFERENCES

  • [1].Aspden P, Wolcott J, Bootman JL, Cronenwett LR, editors. Preventing Medication Errors: Quality Chasm Series. Institute of Medicine of National Academies; Washington, D.C.: 2006. p. 394. [Google Scholar]
  • [2].Kuperman GJ, Gibson RF. Computer physician order entry: benefits, costs, and issues. Ann Intern Med. 2003 Jul 1;139(1):31–9. doi: 10.7326/0003-4819-139-1-200307010-00010. [DOI] [PubMed] [Google Scholar]
  • [3].Koppel R, Metlay JP, Cohen A, Abaluck B, Localio AR, Kimmel SE, Strom BL. Role of computerized physician order entry systems in facilitating medication errors. JAMA. 2005 Mar 9;293(10):1197–203. doi: 10.1001/jama.293.10.1197. [DOI] [PubMed] [Google Scholar]
  • [4].Nebeker JR, Hoffman JM, Weir CR, et al. High rates of adverse drug events in a highly computerized hospital. Arch Intern Med. 2005 May 23;165(10):1111–6. doi: 10.1001/archinte.165.10.1111. [DOI] [PubMed] [Google Scholar]
  • [5].Han YY, Carcillo JA, Venkataraman ST, et al. Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics. 2005 Dec;116(6):1506–12. doi: 10.1542/peds.2005-1287. [DOI] [PubMed] [Google Scholar]
  • [6].Ash JS, Sittig DF, Dykstra RH, Guappone K, Carpenter JD, Seshadri V. Categorizing the unintended sociotechnical consequences of computerized provider order entry. Int J Med Inform. 2006 Jun 20; doi: 10.1016/j.ijmedinf.2006.05.017. [DOI] [PubMed] [Google Scholar]
  • [7].Sittig DF, Guappone K, Campbell EM, Dykstra RH, Ash JS. A survey of U.S.A. acute care hospitals' computer-based provider order entry system infusion levels. MedInfo Proc. 2007 [in press] [PubMed] [Google Scholar]
  • [8].Campbell EM, Sittig DF, Ash JS, Guappone KP, Dykstra RH. Types of Unintended Consequences Related to Computerized Provider Order Entry. J Amer Med Infom Assoc. 2006 Sep-Oct;13(5):547–56. doi: 10.1197/jamia.M2042. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [9].Scrimshaw SCM, Hurtado E. UCLA Latin American Center Reference Series. Vol. 11. Los Angeles: 1987. Rapid Assessment Procedures for Nutrition and Primary Health Care: Anthropological Approaches to Improving Programme Effectiveness. [Google Scholar]
  • [10].Trotter RT, Needle RH, Goosby E, et al. A Methodological Model for Rapid Assessment, Response, and Evaluation: the RARE Program in Public Health. Field Methods. 2001;13(No 2):137–159. [Google Scholar]
  • [11].Handwerker WP. Quick Ethnography. Altamira Press; Walnut Creek, CA: 2001. [Google Scholar]
  • [12]. KLAS CPOE Digest 2007 Release Date 2/23/2007.
  • [13].Barrett JP, Barnum RA, Gordon BB, et al. Final report on evaluation of the implementation of a medical information system in a general community hospital. Battelle Laboratories. 1975 Dec 19; (NTIS PB 248 340) [Google Scholar]
  • [14].Childs B. El Camino Turns the Page: A pioneering hospital recently exchanged the world's first medical information system for a next-generation solution. Healthcare Informatics. 2006 August;:56–57. [PubMed] [Google Scholar]
  • [15].Chin H, Brannon M, Dworkin L, et al. Kaiser Permanente-Northwest. In: Overhage JM, editor. Proc. 4th Annual Nicholas E. Davies CPR Recognition Award of Excellence Symposium; CPRI, Bethesda, MD. Chicago, IL: Healthcare Information Management and Systems Society; 1998. pp. 55–100. [Google Scholar]
  • [16].Miller RA, Waitman LR, Chen S, et al. The anatomy of decision support during inpatient care provider order entry (CPOE): empirical observations from a decade of CPOE experience at Vanderbilt. J Biomed Inform. 2005 Dec;38(6):469–85. doi: 10.1016/j.jbi.2005.08.009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [17].Gliner JA, Morgan GA. Lawrence Erlbaum Associates. 2000. Research Methods in Applied Settings: An Integrated Approach to Design and Analysis; p. 155. [Google Scholar]
  • [18].Patton MQ. Qualitative Research & Evaluation Methods. ed 3 Sage; Thousand Oaks, CA: 2002. [Google Scholar]
  • [19].Poissant L, Pereira J, Tamblyn R, Kawasumi Y. The Impact of Electronic Health Records on Time Efficiency of Physicians and Nurses: A Systematic Review. J. Am. Med. Inform. Assoc. 2005;12(5):505–516. doi: 10.1197/jamia.M1700. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [20].Ash JS, Sittig DF, Campbell EM, et al. An Unintended Consequence of CPOE Implementation: Shifts in Power, Control, and Autonomy. Proc. American Medical Informatics Association Fall Symposium, 2006; (in press) [PMC free article] [PubMed] [Google Scholar]
  • [21].Tenner E. Why Things Bite Back: Technology and the Revenge of Unintended Consequences. Knopf; New York, NY: 1996. [Google Scholar]
  • [22].Massaro TA. Introducing physician order entry at a major academic medical center: I. Impact on organizational culture and behavior. Acad Med. 1993 Jan;68(1):20–5. doi: 10.1097/00001888-199301000-00003. [DOI] [PubMed] [Google Scholar]
  • [23].Morrissey J. Harmonic divergence; Cedars-Sinai joins others in holding off on CPOE. Modern Healthcare. 2004 February 23; [PubMed] [Google Scholar]
  • [24].Ash JS, Sittig DF, Poon EG, Guappone K, Campbell E, Dykstra RH. The Extent and Importance of Unintended Consequences Related to Computerized Physician Order Entry. J Am Med Inform Assoc. doi: 10.1197/jamia.M2373. (in press) [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [25].Ash JS, Stavri PZ, Kuperman GJ. consensus statement on considerations for a successful CPOE implementation. J Am Med Inform Assoc. 2003 May-Jun;10(3):229–34. doi: 10.1197/jamia.M1204. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [26].Ash JS, Sittig DF, Seshadri V, Dykstra RH, Carpenter JD, Stavri PZ. Adding insight: a qualitative cross-site study of physician order entry. Int J Med Inform. 2005 Aug;74(78):623–8. doi: 10.1016/j.ijmedinf.2005.05.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • [27].Anderson JG, Ramanujam R, Hensel D, Anderson MM, Sirio CA. The need for organizational change in patient safety initiatives. Int J Med Inform. 2006 Dec;75(12):809–17. doi: 10.1016/j.ijmedinf.2006.05.043. [DOI] [PubMed] [Google Scholar]
  • [28].Stavri PZ, Ash JS. Does failure breed success: narrative analysis of stories about computerized provider order entry. Int J Med Inform. 2003 Dec;72(13):9–15. doi: 10.1016/j.ijmedinf.2003.08.001. [DOI] [PubMed] [Google Scholar]
  • [29]. For a list of CPOE-related articles see the Bibliography section on www.cpoe.org.
  • [30].Sittig DF, Ash JS, Zhang J, Osheroff JA, Shabot MM. Lessons from “unexpected increased mortality after implementation of a commercially sold computerized physician order entry system”. Pediatrics. 2006 Aug;118(2):797–801. doi: 10.1542/peds.2005-3132. [DOI] [PubMed] [Google Scholar]
  • [31].Stablein D, Welebob E, Johnson E, Metzger J, Burgess R, Classen DC. Understanding hospital readiness for computerized physician order entry. Jt Comm J Qual Saf. 2003 Jul;29(7):336–44. doi: 10.1016/s1549-3741(03)29041-9. [DOI] [PubMed] [Google Scholar]
  • [32].Ash JS, Stavri PZ, Dykstra R, Fournier L. Implementing computerized physician order entry: the importance of special people. Int J Med Inform. 2003 Mar;69(23):235–50. doi: 10.1016/s1386-5056(02)00107-7. [DOI] [PubMed] [Google Scholar]
  • [33].Kilbridge P. Computer crash--lessons from a system failure. N Engl J Med. 2003 Mar 6;348(10):881–2. doi: 10.1056/NEJMp030010. [DOI] [PubMed] [Google Scholar]
  • [34].Campbell EM, Sittig DF, Guappone KP, Dykstra RH, Ash JS. Overdependence on Technology: An Unintended Adverse Consequence of Computerized Provider Order Entry. AMIA Annu Symp Proc; 2007. (under review) [PMC free article] [PubMed] [Google Scholar]
  • [35].Tanenbaum M. Hospital vexed by new computer system. Mountain View Voice. 2006 July 14; Available at: http://www.mv-voice.com/story.php?story_id=1770.

RESOURCES