Adding insight: A qualitative cross-site study of physician order entry

Summary

The research questions, strategies, and results of a 7-year qualitative study of computerized physician order entry implementation (CPOE) at successful sites are reviewed over time. The iterative nature of qualitative inquiry stimulates a consecutive stream of research foci, which, with each iteration, add further insight into the overarching research question. A multidisciplinary team of researchers studied CPOE implementation in four organizations using a multi-method approach to address the question “what are the success factors for implementing CPOE?” Four major themes emerged after studying three sites; ten themes resulted from blending the first results with those from a fourth site; and twelve principles were generated when results of a qualitative analysis of consensus conference transcripts were combined with the field data. The study has produced detailed descriptions of factors related to CPOE success and insight into the implementation process.

1. Introduction

Computerized physician or provider order entry (CPOE) is a process, which allows direct entry of medical orders by the health care decision maker. It has been shown to be effective in improving patient care [1–6]. A team of researchers at Oregon Health and Science University, the physician order entry team (POET), began conducting studies in 1997. The first, a survey to discover what percent of US hospitals have CPOE, concluded that less than one third had CPOE and those that had it reported low usage [7]. The POET, intrigued by these results, endeavored next to discover why diffusion of CPOE was so low by identifying barriers to implementation and the key ingredients for successful implementation. We started by trying to find out why so few US hospitals were using CPOE. Since house officers are usually required to use CPOE in institutions where it is available, we decided to study teaching hospitals. We also wanted to study sites that were community hospitals, since most US hospitals are in this category. We further hoped to gain insight into outpatient as well as inpatient CPOE.

We used qualitative methods to appropriately address the new research question: What are the success factors for implementing CPOE? The study design for fieldwork included focus groups, oral history interviews, and observations that were accomplished by an interdisciplinary team. Results were blended, or triangulated, with those from an expert panel. To enhance trustworthiness (the qualitative equivalent of external validity) [8], we selected multiple sites for study. The data gathering and analysis process was iterative. Results from each phase added to and enriched those from prior phases, thus offering new insight. The following offers a summary of the research foci, strategies, and results of 7 years of study. Because details concerning methods and results have been described elsewhere, this paper simply provides a brief overview with references to prior more in-depth publications.

2. Methods

2.1. Sample selection

Criteria for site selection for the multi-center qualitative study included geography, type of hospital, and length of experience with CPOE; a panel of experts assisted with selection. The University of Virginia was selected first because its early negative experience during implementation has been well documented [9], it is on the East coast, and is a teaching hospital. The Veterans Affairs Puget Sound Health Care System campuses in Seattle and American Lake, WA were chosen because the installations were new yet they were being carefully evaluated [10], they are on the West coast, one is a teaching hospital and the other is not, and they use both inpatient and outpatient CPOE. El Camino Hospital in Mountain View, CA was selected because not only is it a community hospital without house officers, but it also is well known, having been the development site for Lockheed’s pioneering and well documented integrated hospital information system [11]. Having had CPOE since 1972, El Camino therefore has more continuous experience with CPOE than any other hospital in the world. Finally, Kaiser Permanente Northwest was selected for its extensive and successful use of outpatient CPOE [12,13].

2.2. Data collection

Fig. 1 illustrates the unfolding of our inquiry over time. Our multidisciplinary team of researchers visited each site after contact people helped with arrangements and with the often extensive and time-consuming human subjects approval paperwork. Semi-structured, oral history interviews were tape-recorded. They focused on past events and captured the dynamics of implementation issues over time. Focus groups were used either to take a snapshot picture of CPOE use, by house staff for example, or to review the past. Observation was done to verify interview data and gain the current view. Because we wanted to study multiple perspectives, we interviewed and shadowed not just physicians, but also nurses, pharmacists, information technology staff, administrators, and others. Subject selection involved identification of representatives of varying roles and perspectives, including skeptics. A one-week visit was made to Virginia in 1998, three visits were made to the Seattle sites in 1999, 2001, and 2002, and we traveled to El Camino for visits in 1999 and 2000. Kaiser Permanente Northwest was visited multiple times over a seven-month period in 2003.

Fig. 1.

POET research—–chronological view.

A consensus conference of invited experts was held in May of 2001 at the Menucha retreat center near Portland, Oregon. Thirteen people representing administrative, vendor, clinician, and technology roles discussed success factors for CPOE implementation over a two-day period. All sessions were taped and transcribed [14–15].

In 2002, 5 years after our first national survey, we repeated the same survey to find out if the percent of US hospitals with CPOE had increased and if usage was greater [16].

2.3. Analysis

A total of 2570 single spaced pages of field notes and transcripts have been analyzed to date. Analysis is conducted first by individual researchers examining the documents line-by-line. Themes are developed by consensus during face-to-face meetings. As data from each site are analyzed, new questions and foci emerge from discussions. Within the framework of the overarching research question, subsidiary research questions may arise. This is the iterative nature of qualitative research, well illustrated by Fig. 1. After the first CPOE survey, we asked “What are success factors for implementing CPOE?” The focus for the University of Virginia fieldwork and that done at the VA in 1999 (labeled VA Hospital 1 in the figure) was on house officer perceptions [17]. To broaden our understanding and gain the multiple perspectives of information technology, clinical, and administrative staff and to investigate differences between community and teaching hospitals, we did fieldwork at El Camino (labeled El Camino 1) in 1999. Analysis then included the El Camino data and data from the earlier visits to Virginia and the VA [18,19]. Numerous subsidiary research questions arose, stimulating more data gathering at El Camino in 2000 (labeled El Camino 2) to gain more depth, especially about community hospitals. Analysis resulted in the identification of ten overarching themes, several of which have been described in finer detail in published papers [20–25]. We felt the need to study the issues of integration and leadership further, so we paid several more visits to the VA in 2001 and 2002 (labeled VA 2). Analysis of all data from all sites up to that point inspired description of the “Special People” involved in CPOE success [26].

Data from the consensus conference were analyzed using methods similar to those used for field work data, although all consensus conference participants assisted in the analysis and presentation of the results. A set of ten themes emerged (see http://www.cpoe.org) [14].

Now 5 years into our study, we were curious whether the diffusion of CPOE in the US had changed since our first survey, so we duplicated the prior survey, using mail and then telephone follow-up [16].

The fieldwork represents a bottom up approach, a view from different stakeholder groups, especially users. The consensus conference represents the top down expert view. We sensed that, although we had found similar patterns in the two large data sets, there were also differences. Therefore, we conducted an analysis that blended both so that we could gain insight into the similarities and differences. While stepwise iterations kept folding new data into that blended data set, the conference data were orthogonally reconciled using a meta-matrix approach for visualization [27]. Each sub-theme of each set was compared, resulting in twelve principles of implementation. Finally, to investigate inpatient vs. outpatient issues, we studied the system at Kaiser Permanente Northwest. Although the twelve principles guided the Kaiser study by serving as a template for data gathering and analysis, we actively attempted to test the trustworthiness of our prior results while seeking new hypotheses and insights.

3. Results

After we had visited both the University of Virginia and the VA Puget Sound site in Seattle, we completed the first iteration of analysis dealing with house staff perceptions of CPOE. In summary, house officers were proud of being facile with CPOE and appreciated the decision support it offered, but were resentful of the additional time it took. Entering orders from any location, ready access to results, and having everyone reading off the same page were considered benefits. There were, however, hardware problems, service and support issues, increased paper usage, usability issues, excessive complexity of screens, and delays switching among systems. Implementation strategies that include house officers in planning, that integrate CPOE into the workflow, and that provide training and support would please house staff [17].

A second iteration of analysis included data from our first visit to El Camino Hospital and focused on discovering more about the multiple perspectives of different groups of staff and on understanding the community hospital view. Clinicians often believe that CPOE is more advantageous to the hospital administrators than it is to them. Although they resent the time it takes, they appreciate remote entry and the potential it holds for helping deliver better patient care. Administrators may view CPOE as a quality assurance tool and an aid in accreditation success. Information technology professionals are somewhat frustrated with the CPOE technology available from vendors but are committed to helping clinicians accept CPOE [19].

A third iteration of analysis included data from a second visit to El Camino, during which the focus was on how community hospitals may differ from teaching hospitals. We found that successful implementation in community hospitals depends less on mandating use, which is a common strategy in teaching hospitals that have house officers, and more on the existence and development of an organizational culture of collaboration and trust. This culture is characterized by administrative and clinical groups that trust one another and by nurse physician collaborations. Themes that emerged from the data include:

Language and misunderstandings

People using the same words are not always talking about the same thing; some words have negative connotations that offer unintentional negative attributions to individuals.

Context matters and it affects the way of doing things

The time and place, institutional context, professional and departmental context, and the individual’s role cause perceptions to vary.

Benefits and tradeoffs (upsides and downsides)

Everyone sees both benefits and tradeoffs to CPOE; what is good for some people may not be for others, though.

Contrasts, conflicts, and contradictions

There is often an “us” versus “them” mentality between administrators and clinicians; contradictions, like the computer getting either the blame or the credit for things, are common.

Collaboration and trust

Multidisciplinary collaboration and trust between administration and clinicians are necessary conditions.

Special people

Stalwart individuals in both leadership and support positions are key.

Customization and organization of information

Individual ways of doing things necessitate customization; the structured organization of information in systems gives rise to workarounds and escape valves.

Defining the boundaries of CPOE

The order creation and communication process is nonlinear; it is a myth that only doctors give orders, since other clinicians like nurses and pharmacists are part of the process as well.

The ongoing nature of implementation

Ongoing user involvement, training and support, and system modifications are needed.

Time

Speed of the process is critical; perceptions of time differ by role; a 10-year life cycle before system renewal is likely [23].

The themes that emerged from the consensus data were similar, although with more emphasis on cost issues and on project management strategies [14]. When the two data sets were carefully integrated, the outcome was twelve principles for CPOE implementation. Computer technology principles included: temporal concerns; technology and meeting information needs; multidimensional integration; and costs. Personal principles were: value to users and tradeoffs; essential people; and training and support. Organizational principles included: foundational underpinnings; collaborative project management; terms, concepts and connotations; and improvement through evaluation and learning. Finally, environmental issues included the motivation and context for implementing such systems [27].

At this point, in 2002, we repeated the 1997 survey and found that although the percent of hospitals claiming to have CPOE had decreased, its usage within those hospitals increased. Using a tenacious phone call follow-up process, we found that respondents to the first survey may have over-reported availability of CPOE [25]. When we looked closely at the 2002 responses from sites reporting full availability of CPOE, many were veterans affairs and military hospitals. When those hospitals are not counted in the “fully available” category, the percent of US hospitals with complete CPOE decreases to 6%.

Several new themes emerged from the study at Kaiser Permanente Northwest, all likely due to its being mainly an outpatient facility. First, there are many categories of unintended consequences of implementing such systems, including positive serendipitous consequences such as increased patient involvement in the order entry process. A second new theme was that of security concerns. Finally, a particularly intriguing new theme was one we are calling “The Hub” [28]. The clinical information system is so central to all of the activity at this site that it forms a conceptual Hub of all work life. It touches every aspect of people’s jobs and in essence takes on an identity as another player on the health care team.

4. Discussion

Qualitative research is iterative in nature and often offers surprises. For example, as informaticians, we understand the benefits of entering structured data, yet in the field we saw how cumbersome and time consuming this is for users. We are well aware of the evidence indicating that CPOE can reduce medical errors, yet we also saw and heard about many new kinds of errors caused by the information system. We know that good training and support are necessary, but we learned a great deal about attributes of amazing people who perform those tasks and play other roles in the implementation process. We saw not only the intended positive consequences of CPOE, but also both positive and negative unintended consequences [29].

Our second survey indicated that hospitals have not been adopting CPOE rapidly over the past 5 years. It is important to note, however, that at hospitals that have CPOE, usage is greater than reported in the first survey results. It may be that implementers are aware of and are using the success factors that the POET and other research groups have described.

In summary, publication of the results of these iterative inquiries served to promote a realization that implementation of CPOE is not easy and that the negatives must be weighed against the positives. As McDonald et al point out, CPOE implementation is a journey and not a destination and some of POET’s work “should be required reading by every health care institution’s CEO and CIO” ([30], p. 122) because it offers “some needed balance to the current excessive expectations about CPOE” ([30], p. 123). In addition, this work has motivated further research by others attempting to describe a realistic picture [31]. The most recent POET work indicates that systems are improving and that successes like that at Kaiser Permanente Northwest can provide optimism about the future.

5. Conclusion

While it is difficult to summarize the lessons from 7 years of work with multiple researchers across four organizations, which resulted in over 2000 pages of data, we were able to draw some solid conclusions about both our methods and CPOE implementation success factors. The selected methods allowed us to accomplish what we set out to do, and, although qualitative research is labor-intensive, we became more efficient using these techniques over time. Regarding success factors, CPOE implementation truly has its upsides and its downsides, along with many unintended consequences. The key is to maximize the upsides, minimize the downsides, and try to have mechanisms in place for managing the unintended consequences.