Making Sense of Clinical Practice: Order Set Design Strategies in CPOE

Abstract

A case study was conducted during the customization phase of a commercial CPOE system at a multi-hospital, academic health system. The study focused on the development of order sets. Three distinct approaches to order set development were observed: Empirical, Local Consensus and Departmental. The three approaches are first described and then examined using the framework of sensemaking. Different approaches to sensemaking in the context of order set development reflect variations in sources of knowledge related to the standardization of care.

INTRODUCTION

The development and implementation of a Care Provider Order Entry (CPOE) system is a complex matter that requires attention to the details of clinical decision-making and how those details fit into everyday clinical workflow and the overall clinical strategy of an institution [1–3]. CPOE designers have concluded that the creation of order sets that organize the clinical content of the system is a requirement for successful implementation [4–6]. However, the development of order sets can be a resource-intensive undertaking, and organizations that are currently developing clinical content for CPOE systems have minimal guidance in the academic literature on how to approach this task.

Order set development is a rationalization of practice, and proceeds from some underlying strategy, approach or philosophy. The approaches chosen by participants reflect attempts to make sense [7, 8] of clinical practice in a way that can be codified into an order set. Participants develop categories for patients, diseases, symptoms, procedures, therapies, encounters, and other domains of practice. This paper reports on three approaches to developing order sets observed in this case, referred to as Empirical, Local Consensus, and Departmental.

METHODS

Setting

This research was conducted in a multi-hospital, academic health system in the Midwestern United States. The organization, Calloway Medical Center (CMC), is a multi-hospital, academic health system in the Midwestern U.S. CMC had made a significant commitment to use technology to achieve its cost and quality objectives, including the purchase of CPOE, Nursing Documentation and Pharmacy information systems from one vendor (Emery). The research focused primarily on Information Technology (IT) staff and clinicians who were working to build and implement the systems.

Study Design

A case study was conducted to explore the strategies used for developing order sets for a new CPOE system. A case study contains lessons but not “generalizable results” in the classical sense. Methods included 64 observations of meetings, design work and the pilot clinical unit. The observations varied in length from 1 hour to 3.5 hours. Interviews were conducted with 15 key participants.

Primary data from fieldnotes and transcribed interview texts were analyzed using QSR N5 qualitative analysis software. Data were analyzed for themes related to the processes of developing order sets and classifications created for order set development. The project was approved by the Wayne State University Institutional Review Board. Pseudonyms for the institution and the participants are used.

RESULTS

Three approaches for order set development were identified: Empirical, Local Consensus, and Departmental. These approaches represent different modes of sensemaking [7]. The Empirical approach is to use clinical data to depict current practice and map the order set to that practice. The Local Consensus approach is more informed by the politics and practices of the institution implementing the order set. This approach incorporates the desires of local leadership to implement change, and the approval of existing structures of authority (e.g. medical staff committees). The Departmental approach was used in the complex area of Respiratory Therapy. In this case, the physicians were not involved in the development of the order set. The Respiratory Therapy managers from the multi-hospital system met and determined the clinical content. A common practice in the institution prior to CPOE was the physician ordering respiratory therapy and the Respiratory Therapist (RT) completing the details of the order for later physician signoff. This partnership was reflected in the order set development process.

The Empirical Approach

Dr. Hart, a surgeon, was part of the four-member Physician Leadership Team (PLT), a group that met weekly and guided clinical systems development efforts. Dr. Hart was tasked with developing order sets for CPOE. He had no formal training in informatics, but had read extensively in information theory and was particularly interested in the derivation of categories from data. His approach was to use data from the existing order entry system to depict current practice. This analysis would be used to develop the order sets so that they reflected what the physicians were currently ordering. In theory, this would place the contents of the order set beyond reproach, and thus facilitate the buy-in of the physicians. Dr. Hart was assigned two IT staff (Kristy and Lisa) to work on the order sets, and a pharmacist (Keith) also attended some of the meetings of the group.

In regular semi-weekly meetings the group discussed potential sources of data and an approach for analyzing the data. They worked with Bill Young, Kristy’s second-level manager, to obtain an extract from the extant orders data. There was about a year’s worth of “good” data in the system, based on the implementation timeframe of the existing electronic order entry system, operated primarily by nurses and clerks. Dr. Hart met with him to describe specifically the data he would like to analyze. He gave Bill a list of categories of orders he wanted, including admission, patient condition, code status, vital signs, activity, diet, nursing orders, continuous infusions, medications, lab tests, consults, gases/ vent settings, notifies, and diagnostic tests. Bill used the categories and produced a massive report that included every non-medication order for the previous year. The report reflected only the first 24 hours of each patient stay, because the group was interested in developing admission order sets. The report was organized by service (this is the clinical department, such as Psychiatry, Neurosurgery, etc.) and was sub-categorized by diagnosis.

The data came from the orders placed over the past year in the clinical information system. These are generally orders placed by nurses or clerks on behalf of physicians. The definition of service was congruent with those defined by the Medical Records department at CMC. The patients were included in a given service based on their admitting or discharging physician.

The inpatient Psychiatry unit was to be the pilot for CPOE. Dr. Hart looked at the Psychiatry portion of the report and identified 22 diagnoses, which he hoped would collapse into 7 order sets. He was pleased with the group’s progress when, in only a couple of weeks, they were able to produce one order set that combined 8 of the diagnoses. It was starting to look like there would only be one order set for Psychiatry. This was because they had found such similarity in the orders across the various Psychiatry diagnoses. The unified order set looked strikingly similar to the draft created weeks earlier by Lisa, who had done a chart review of Psychiatry patients. At that time it was known that Psychiatry would be the first unit to go live on CPOE, but the team had not gotten specific direction from Dr. Hart. Lisa met with some of the nurses from the Psychiatry Unit and obtained the medical records of several of their patients. She used the records to construct a draft order set. Lisa’s order set and the one created using the analysis of previous orders data were almost exactly the same. The next step was to include medications in the order set.

Analysis of Medications

The group was able to obtain an orders abstract of medication orders from the Psychiatry unit. In a group meeting at Kristy’s desk, Keith sat at the computer and set up spreadsheets of these orders so that he could calculate the most frequently ordered medications. He organized the orders by frequency and calculated the cumulative percentage of the total to get to the 80% number – this list contained 46 drugs. He then went through the list and eliminated drugs that he felt would not be considered Psychiatry medications, and they reached 80% at 20 drugs. Now that the medications were identified, the group needed to create order sentences for them. Order sentences specify the format of the order and contain all of the relevant categories related to that order. For example, an order sentence for a medication would include the name of the drug, the dose, the dose units, the form (e.g. inhalant), the route, the frequency, the duration, and possibly other information. An example of an order sentence is in Table 1. Some of these categories can be pre-populated with a default value, which could then be changed by the physician. Keith went though each of the specific medication orders for Psychiatry and wrote order sentences. In order to get the proper typical dosage (if he didn’t already know it), he called the pharmacy and asked a colleague.

Table 1.

Example of Order Details for Respiratory Therapy

Albuterol (albuterol 0.083% inhalation solution; First Dose 3/18/2003 10:26AM, Routine, 3 mL, Nebuliz, INHALANT, Q6H
Order Details	Detail Values
First Dose [3/18/2003 10:26AM]	[3/18/2003 10:26AM] To change a value in the order details, the user would click on the detail of interest in the left column and enter the new value in this box. All details in bold type must have a value entered.
Pharmacy Order Priority [Routine]
Dose [3]
Dose Unit [mL]
Route of Administration[Nebuliz]
Drug Form [INHALANT]
Frequency [Q6H]
PRN [No]
PRN Reason
Duration
Duration Unit
Stop Date/Time
Future Order

The Local Consensus Approach

Dr. Hart was asked to lead the order set development efforts, but there were some exceptions. Dr. Nofsinger, a PLT member and Emergency Medicine physician, was responsible for developing the order sets related to Emergency Medicine. As the go-live date for the pilot site, Hale Hospital, neared, another physician was asked to participate. Dr. Lindsey was an internist at Hale who was developing her administrative skills. Despite being a patent skeptic of technology, she was asked by the Hale Chief Medical Officer to help develop the order sets for Hale. The approaches of Drs. Nofsinger and Lindsey are both referred to as “Local Consensus” because they both use existing committees to authorize their recommendations. This approach does not attempt to replicate “current practice”, but instead uses the political authority invested in medical staff committees to create order set content that attempts to guide physician behavior in a specific way.

Consensus in the Emergency Department

Dr. Nofsinger’s role on the CPOE project was to act as an interface between the IT staff and the clinical users, especially in the department of Emergency Medicine. Before he joined the PLT there was not a specific plan to have order sets for the Emergency Department (ED). With his attention, they quickly determined that order sets would be needed for the department. The ED had long enjoyed a centralized structure, meaning that there was a CMC or system-level department to which all of the hospital-based ED physicians reported. The department had a clinical advisory group that enabled consensus-based decisions about clinical issues to be made. Dr. Nofsinger created order sets for common ED diagnoses, as well as symptom-based order sets such as chest pain. He based these on existing protocols and his own clinical experience. He then brought the order sets to the clinical decision group for approval. In this way, Dr. Nofsinger could gain approval for ED order sets in all the hospitals at once.

Local Politics at Hale

The empirical approach the Hart team had been using was not appealing to Dr. Lindsey. She did not want to design the order sets solely based on current practice. She felt that this was an opportunity to implement some controls on physician ordering. With the hypertension problem-based order set she wondered why they had pulled every order that had been written on the Psychiatry unit for the diagnosis of hypertension, instead of just including “the appropriate things” in the order set. She felt that they (the IT people, who were actually guided by Hart) were looking at it as a “frequency [of current orders] thing” and she was looking at it as a “standard of care” thing. Her opinion presented a paradox, because part of Dr. Hart’s rationale for the empiricist approach was that it would increase the chances that the results would be accepted by the physicians because it reflected their own practice. But Dr. Lindsey, an insider at Hale, felt that current practice was no basis for standardization.

Dr. Lindsey’s creation of personal goals for the implementation of CPOE implies that the organization’s goals for CPOE were flexible. This question was explored with interviewees and results showed different perspectives from different groups. The IT staff accepted CPOE as a useful tool – a combination of non-naive technological determinism and self interest. The physicians employed by IT saw it as a way to improve practice and further the reputation of the medical center as technologically advanced. The users who participated as designers (this includes Dr. Lindsey) saw CPOE as inevitable and as a result of too much political power (and money) being invested in corporate IT. The political symbolism of the system overpowered notions of its utility in their discussions.

Dr. Lindsey felt that first and foremost, the order sets needed to reflect the preprinted order sheets, where available, because these had been approved by the medical staff bureaucracy at the hospital. She was concerned about the local politics. The PLT knew that the hospital had a medical staff committee structure and were aware of its importance, but they had less local knowledge about what was important to the medical staff. Interestingly, Lindsey was not a user of the approved preprinted order sheets. She “organized [her] thinking in a different way”.

The Departmental Approach

The Respiratory Therapy (RT) order set was created by a group of RT managers with no physician input. One reason RT needed special attention vis-à-vis CPOE was the incongruence of typical physician ordering practices and what actually has to occur to provide the therapy. Respiratory services are generally provided by trained professionals who are trusted by physicians to administer the proper therapy. Therefore, frequently the physician would order the medication and dose of choice (often the drug Albuterol), and leave the rest of the details for the therapist to handle. The physician generally did not specify elements such as the use of a nebulizer and other operational details of the therapy, as these were implied in the order. With CPOE, the physician had to not only order the drug, but also specify the form, route, and other details (see Table 1). The physician also needed to order the respiratory therapy itself. As the IT staff facilitator put it, the physician had to use the order to say, “I want the Albuterol and oh yeah, I want the respiratory therapist to do it.”

The driver of this change was the pharmacy computer system. The orders for respiratory treatment preparations had to be treated like medication orders to enable the pharmacy system to perform allergy and drug interaction checking and to ensure the drugs were listed on the Medication Administration Record (MAR), the official and legal source of information about medications for a specific patient. With this change, a physician would now need to know more about the ordering and administration of respiratory therapy. Significant effort would be required to get used to the new ordering process.

The RT group was brought together to try and design an order set that would be simple to use and still get all of the necessary components ordered. The group spent hours debating the specifics of the order set, often encumbered by the vague or multiply-defined terms involved. For example, in the order sentence for the respiratory medications, the categories of “route” and “drug form” were confusing. The term “inhalant” described the route for all respiratory therapies, so why should they even have to specify it? But “inhalant” could also be a drug form, along with aerosol, nebulizer and metered dose inhaler (MDI). On the other hand, perhaps nebulizer should be considered a route.

There was also significant discussion about the proper and legal way to order ongoing respiratory treatments. In CPOE, it was not clear how to accommodate the ordering of both the initial and the subsequent treatments (the need and frequency for these was often determined by the therapist) without asking the physician to go back into the system and order the subsequent treatments. No one seemed to want to do that, and it appeared that it would be difficult to have one order in CPOE function as both the initial and subsequent treatments. The order needed to be either “initial” or “one time only”. The discussion about initial and subsequent treatments was confusing for everyone because each hospital had a unique way of handling these orders. The basic concern was the performance of treatments that hadn’t been electronically authorized by the physician, because the administration of a medication not ordered by a physician is considered fraud. One of the RT managers in the room referred repeatedly to the fact that in the current environment he had “a piece of paper” in the chart that could be referenced if there were any question about the appropriateness of the treatments being delivered to the patient. He didn’t trust the system and all its categories to effectively represent the wishes that a doctor could now write on a paper order and place in the chart.

One reason the automation of these clinical processes was particularly difficult for the managers was that the previous experience they had in classifying their work was in the area of charges. One participant brought a notebook to the meeting labeled “charges”. This book enabled him to answer questions about the procedures his staff performed and supplies they used. But it was difficult to transition from thinking about what parts of the therapy are “billable” to a more clinical, process-oriented view of the order.

DISCUSSION

Sensemaking in the standardization of care

Weick and Sutcliffe have observed that “sensemaking occurs when a flow of organizational circumstances is turned into words and salient categories” [9], and this process creates the meanings that determine action. Gasson has described two “modes of analysis” in sensemaking:

1. Reflective involvement in those local systems of social interaction, practice, and sensemaking that constitute organizational work, and

2. engagement in that detached sensemaking and analysis, by which situated knowledge is externalized, reified, and made explicit [8].

These two modes describe the experience of a physician who engages in clinical practice with her colleagues and patients, and then participates in the creation of order sets for CPOE. In the first setting she is a participant in sensemaking in an “everyday” clinical context: communicating with colleagues and patients to create meaning from clinical information and developing a plan of action. In the next she is an observer of practice, attempting to describe it in an abstract way that can be encoded into CPOE.

Weick and colleagues describe sensemaking as the “interplay of action and interpretation” and essentially asks two questions: “what’s the story here?” and “now what should I do?”[9]. Gasson [8] argues that to answer these questions the group needs to explore not only “know-what” and “know-how”, but also “know-why” and “who-knows-what”. In this way they communicate not only about practices and artifacts, but also about the rationale related to those practices, and the specific, negotiated organizational context in which practices take place.

Examining these four ways of knowing in the context of the three approaches to order set development brings out the variation in the approaches. Know-what is explicit knowledge: what do we need to build the order set? Know-how is tacit knowledge: where does the information come from? Know-why is the rationale for the approach: why do it this way? And Who-Knows-What is about the allocation of expertise: who needs to be involved to make this happen?

The Empirical Approach lends ultimate authority to the local, practicing physician. It attempts to rationalize care according to the way it is already being practiced, based on the theory that this adherence to local practice will make the order set acceptable to physicians. The Local Consensus Approach is also interested in encouraging adoption, but sees the standardization of practice as a management responsibility. The Departmental Approach proceeds on the assumption that the complexity of the RT orders demands a “process expert” to create the order set, and that this process should reflect the existing, tacit agreement between many physicians and RTs that the physician would order the therapy and the therapist would figure out the details.

CONCLUSION

Conspicuously absent from the order set development process at CMC was an explicit dedication to evidence-based medicine (EBM). Personal theories of acceptability (Dr. Hart), appropriateness (Dr. Lindsey) and role within a clinical department (Dr. Nofsinger) trumped EBM. Instead the focus was on strategies they thought would be successful to achieve the goal of the system, as they perceived it. This may, for some, reflect a certain battle-weariness among people who had been through a multi-year process of decision, negotiation, acquisition, and implementation of CPOE. Lofty goals related to EBM were replaced with more pragmatic objectives.

This case study highlights the role of local context in the development of order sets. Many other strategies exist, and it is impossible to rank them according to utility. Those engaged in the development of order sets encounter technical, political and logistical challenges. Examining sensemaking practices provides lessons for organizations engaged in the work of creating order sets. Ultimately the system must create “meaning” (or information that can be acted upon) in the same way that the people do.

Table 2.

Perspectives on appropriate knowledge from each order set design approach

Types of Knowledge	Empirical	Local Consensus	Departmental
Know-what (Explicit)	Frequency of orders	Standard of care	Standard of care
Know-how (Tacit)	Techniques for analysis	Practice of local clinical leaders and approved guidelines	Consensus of RT managers, speculation on physician adoption.
Know-why (Rationale)	To encourage adoption	To encourage adoption and improve practice	Use the experts on the ordering process – tacitagreement between physicians and RTs
Who-knows – what (Sources of expertise needed)	How to get the data	Clinical leaders, appropriate committees	RT is the source of expertise on RT orders.