A Cognitive Task Analysis of Information Management Strategies in a Computerized Provider Order Entry Environment

Abstract

Objective

Computerized Provider Order Entry (CPOE) with electronic documentation, and computerized decision support dramatically changes the information environment of the practicing clinician. Prior work patterns based on paper, verbal exchange, and manual methods are replaced with automated, computerized, and potentially less flexible systems. The objective of this study is to explore the information management strategies that clinicians use in the process of adapting to a CPOE system using cognitive task analysis techniques.

Design

Observation and semi-structured interviews were conducted with 88 primary-care clinicians at 10 Veterans Administration Medical Centers.

Measurements

Interviews were taped, transcribed, and extensively analyzed to identify key information management goals, strategies, and tasks. Tasks were aggregated into groups, common components across tasks were clarified, and underlying goals and strategies identified.

Results

Nearly half of the identified tasks were not fully supported by the available technology. Six core components of tasks were identified. Four meta-cognitive information management goals emerged: 1) Relevance Screening; 2) Ensuring Accuracy; 3) Minimizing memory load; and 4) Negotiating Responsibility. Strategies used to support these goals are presented.

Conclusion

Users develop a wide array of information management strategies that allow them to successfully adapt to new technology. Supporting the ability of users to develop adaptive strategies to support meta-cognitive goals is a key component of a successful system.

Introduction

The electronic medical record with Computerized Provider Order Entry (CPOE) has instituted a new era of electronic information processing in clinical care. ^1,2–4 Electronic documentation, electronic order entry, and decision support dramatically change the information environment of the practicing clinician. Prior work patterns based on paper, verbal exchange, and manual methods are replaced with computerized systems that are potentially less flexible, because they require prescribed methods of data entry and presentation.

Successful implementation of CPOE requires close attention to the local details of information exchange and workflow processes. Understanding workflow is difficult as many processes are often “hidden” because clinicians (and even the information technology administration) are unaware of subtle behavioral changes that occur as clinicians adapt to the environmental constraints. These adaptations consist of both the change in behavior of clinicians and the ongoing changes in the technology as the system of care evolves together. As Rasmussen notes (1994) “… an effective organization of work depends on self-organizing and adaptive ‘mechanisms’ which enable the system to change its properties in order to maintain a match with current needs … .” (p. 20). ⁵

In the process of adapting to the system, users develop multiple strategies to seek, sort, categorize, communicate, and document information. ^5–8 These strategies reflect the resolution between basic information processing needs of the user and the functional characteristics of the technological system. Often, new strategies eventually become seamlessly embedded into work processes as implementation progresses. ^5,9

The purpose of this study is to clarify how users have adapted to a well-established CPOE system. We conducted a Cognitive Task and Activity Analysis of the information management behavior of users across several primary care settings. ^10–13 Cognitive Task Analysis is a qualitative method of analysis derived from human factors and cognitive psychology. The purpose is to understand individuals’ mental representations of activities and work processes they are engaged in and how those perceptions relate to behavior.

VHA’s Computerized Patient Record System (CPRS)

The Veterans Administration (VA) has achieved widespread adoption of their Computerized Patient Record System (CPRS) since its dissemination in 1999. CPRS contains functionality for CPOE, electronic note entry, clinical decision support, electronic consult requests, results reporting, and bar code medication administration. The VA system is a full Patient Information System. Both CPOE and electronic note entry are mandated and almost all of the medical record is now electronic across the 128 medical centers in the VA. The VA is also an excellent laboratory to study the impact of CPOE. Because CPRS is adopted system-wide, the organizational and geographical factors that contribute to successful adoption of a CPOE system can be examined. In addition, because CPRS is leveraged extensively to improve the quality of care, the usefulness of technology in enhancing adoption of guidelines, clinical reminders, and quality improvement activities can be evaluated.

In early 2003 the VHA released the first new extension to CPRS called Care Management (CM). CM extended the functionality of CPRS by allowing a multiple-patient view, the ability to create generic “tasks,” and a population-based query tool. Although this functionality was much desired by providers, adoption has been very low. To clarify the issues surrounding the low adoption rate of CM, we conducted a Cognitive Task Analysis of the information management activities across multiple primary care settings. The goal was to explore current patterns of computer use as well as to identify some general information management principles that might guide future development. The results of the full evaluation are reported elsewhere (Nebeker, Weir, Hicken, and Rappaport, under review). ¹⁴

CPOE and Errors

Although information systems can significantly improve care, they may also have unintended negative results. ^15–18 When the amount of information increases and the display format changes, individuals must develop new and different strategies to manage the basic cognitive functions of memory, attention, planning, and learning. These strategies may or may not improve practice and potentially threaten the success of the implementation. Although there is no direct evidence with CPOE systems, research in cognitive psychology has demonstrated that information overload causes significant effects on individual information processing. For example, excessive information may cause individuals to underestimate the rate of events ¹⁹ and to be overconfident. ²⁰ Restrictive and limited displays of different kinds of data may result in co-variation errors (e.g., items on a list might be unconsciously perceived as equally probable), inappropriate risk assessments, or priming of inappropriate behavior scripts (e.g., focus on curative perspective when palliative care is needed). ^21–23 The impact on decision making may be subtle and may have profound consequences for the quality of care.

Other studies have found significant impact of CPOE systems on collaborative and communication processes. Many CPOE systems are designed assuming a linear process for writing orders (e.g., physician writes orders and allied staff implement them). However, often the process is not linear but rather circular, recursive, redundant, and involves extensive synchronous communication processes. For example, a nurse may write an order for a fluid bolus for a post-operative patient with low urine output in the middle of the night because she knows that the physician will cover her the next day. The pharmacist may discontinue an order and rewrite the new one if the order is incorrect or not in stock in the form ordered. The physician will be notified later. A nurse may refuse to administer what the physician ordered until it is clarified. Eliminating these processes of communication or making them more difficult may result in increased errors. 2,15,24–26

Cognitive Task Analysis

Cognitive task analysis (CTA) is defined as “the extension of traditional (behavioral) task analysis techniques to yield information about the knowledge, thought processes, and goal structures that underlie observable task performance” (p. 3). ²⁷ The aims of a cognitive task analysis are to identify the concepts, contextual cues, goals, and strategies that constitute the mental model of individuals within a domain of action. ^5,28–30 The methods used span a variety of techniques including observation of physical actions, semi-structured interviews to determine the corresponding cognitive activity, formal cognitive mapping, and observation using a “talk-aloud” technique (where individuals voice out loud what they are thinking). ³¹ Most authors promote using multiple techniques and adapting the techniques to the context. All promote some form of goal analysis. ²⁸ In this study, we used a hierarchical goal analysis approach, adapting procedures from two theoretical perspectives, Action Identification Theory ³² and general CTA models. ²⁷ Similar analyses have been conducted in relation to clinical practice guidelines ³³ and the ordering process. ³⁴

Cognitive Task Representations

Cognitive theories stipulate that cognition (thinking, decision making and subsequent behavior) arise out of activation of associative schemas of neural connections in memory that consist of behavioral sequences, semantic content, values, and emotional content. ^35,36 The more well-learned the behavior, the stronger the associative links. One particular type of representation is action representations, which have several unique characteristics. First, they are primarily goal-based and consequently, tend to stay “activated” until the resolution criteria have been reached. In contrast, “activated” memory or knowledge representations decay linearly. ^37,38 Action representations are also thought to be organized hierarchically, ranging from basic concrete procedural or mechanical representations of bodily movements to values and goals at the more abstract level in a many-to-many relationship. ^32,36,39

In this study, we adapted techniques developed by Vallacher and Wegner ³² to characterize the hierarchical structure of action representations. In this method, individuals are asked to respond to the following open-ended questions, “What are you doing?” “You are doing XX in order to do what?” “You are accomplishing XX, by doing what?” Our focus was on activities relating to information management in the primary care setting, in contrast to clinical medical tasks, although the two are highly inter-related. Information management tasks focus on the searching, handling, and manipulating of information, including diverse areas such as scheduling, ordering, or communicating with patients and include all mediums, e.g., verbal, electronic notes, and e-mail.

Methods

Site Selection

Site recruitment consisted of a two-stage process. VA institutions were selected based on institutional readiness, followed by geographic and size criterion in order to enhance representativeness. To assess readiness, Care Management implementation plans were solicited from all 135 VHA sites. The authors (CW, BH, and JN) rated each site’s readiness using a structured coding scheme that included points to be given for plan specificity, the presence of clinical champions, early marketing strategies and a training protocol. Scores could range from 1–10. Three authors (CW, BH, and JN) conducted the ratings independently and discussed differences until a consensus was reached. Sites with average scores of 6 or above were then approached to participate in the study and the final choice of enrollment balanced geographical and size representativeness. A total of 13 sites were included in the overall study and 10 sites were involved in the interview process. Two sites had been used for early piloting and another site was unable to get timely institutional review board (IRB) approval. All participating sites had IRB approval for the study.

Interviewers

Because CTA requires significant domain-specific knowledge and familiarity with the motivations and the normative situational awareness of participants, the degree of content expertise held by the interviewers can be essential to providing an accurate portrayal of tasks. The three interviewers involved in this study are very familiar with the VA system. JN is a physician in active practice at the VA. BH is a clinical psychologist who at the time of the study carried a small clinical load at the VA. CW is a nurse and social psychologist who served as Project Manager for implementing CPRS at the Salt Lake City VA.

Participants

At each site, a primary care clinic was randomly selected. Primary care teams consisted of providers, Nurse Practitioners, Physician Assistants, and/or staff physicians (we did not include residents). Also included were RNs, LPNs, and Nursing Assistants as well as social workers, dieticians, pharmacists, and clerks. A total of 88 interviews were conducted. Overall, 14 nurses, 53 ordering providers, 8 pharmacists, 2 dieticians, 3 clerks, and 8 social workers were interviewed.

Interview Content

A semi-structured interview was used that focused on how, when, what, and where information was collected in primary care, with a focus on goals and control strategies people used to manage their environment. Three investigators (CW, BH, and JN) extensively piloted the survey together in order to standardize the process. The interview included general queries regarding information management activities and had an overall chronological structure starting with questions regarding how they would begin the day, prepare for a visit, follow-up activities that were required, and clinic-related activities. Appendix A contains the interview instrument.

Interview Process

The interviews were conducted at the clinic during normal clinic hours. The interviews followed the general format, but the order of topics varied to allow for respondents’ idiosyncratic responses. To facilitate the goal-based approach, interviewees were asked probe questions, specifically “you did XX action in order to accomplish what purpose?” or “you accomplished XX goal by doing what action?” For example, interviewees would be asked, “When you come to work in the morning, what are the actions you do at that time to prepare for your day?” When a respondent replied with “I print a list of my appointments from VistA,” the interviewer would respond with “and you printed that list in order to …”. Similarly, if it were not clear, interviewer would ask “you printed that list by doing … .?” This methodology is used by Vallacher and Wegner to map cognitive action structures ³² and is consistent with the general principles of Cognitive Task Analysis. ²⁷ The interview would proceed in that manner covering the day chronologically, including actions associated with getting ready to see a patient, communicating with the staff, lunch breaks, etc. Interviewees were allowed to talk at length in an open-ended manner, with some prompting when needed.

Observations

Observations were embedded into the interview process and were quite easy to do because the researchers were visiting the clinic during patient care hours. In addition to the verbal content collected during the interview, clinicians were asked at times to demonstrate how they actually did many of the activities they described. For example, if they said “I processed my alerts,” they were asked to actually process their alerts while we observed. About a third of the interviewees allowed the researcher to sit in during a patient visit. This aspect of the interview was essential, because some clinicians had trouble reporting accurately their behavior in enough detail. Clinicians were asked to “think aloud” when doing these actions whenever possible. ^31,40

Artifacts

Finally, every time a clinician reported using some kind of artifact (e.g., paper, tools, or forms) they were asked to show it to the interviewer. For example, if they printed the daily schedule that lists patient’s appointments for that day and took ongoing notes on that list, we examined the list. Often, this would provide an opportunity for providers to describe in detail their strategies for organizing their own work (e.g., unique symbols for tracking progress, important content, or relevant “stories” about an event).

In summary, the combination of structured interviews, observations, and examination of artifacts provided a robust view of how CPOE was being used. The goal of this form of data collection was to maximize the emergence of goals, perceptions, beliefs, strategies, and concerns. All interviews were taped in order to facilitate data collection and transcribed later for analysis. In addition, field notes were typed, collated, and included in the analysis. Observation and interview time totaled approximately 110 hours.

Analysis

Qualitative data analysis involves using inductive processes to bring order, structure, and meaning to a large amount of data. Analytic procedures began with multiple readings of the interviews, development of coding schemes to facilitate abstraction, and an iterative process of validating the codes by comparing independent reviewers, discussion, and revision. Coding was expanded through processes of aggregation across levels of abstraction and by analysis across persons and events. ⁴¹

For this study, the analysis process was conducted with two of the investigators (CW and BH) plus two graduate students (Nursing Informatics and Psychology PhD candidates) meeting weekly for a total of 60 hours over 4 months of time. Initially the meetings focused on identifying themes, issues, and patterns. Later, the team worked on establishing and revising coding schemes in order to further develop the categories. To accomplish that task, each member of the team independently highlighted the transcribed text and then discussed differences and then revised the codes using NVivo, version 2.0. The process was iterative with descriptions of boundary conditions and prototypes continually being revised until agreement levels reached at least a moderate level (kappas ranged from of 0.48 to 0.64). ⁴² This level of agreement provided some assurance that the categories could be reliably described. The remaining categorization and coding of text was done by consensus. The group met for over 60 hours and the subsequent coding of all of the text took another 120 hours across investigators.

Results

The results will be reported at three levels of analysis. The first level presents the basic tasks that clinicians reported actually doing. These include the exact stated items, such as “Identify when the patient was last here” or “Determine the reason for the visit.” The second level is a common components analysis where we extracted common structures across tasks. The third analysis is at the goal level where generalized information management strategies and over-arching information management goals are identified.

Task Descriptions

Although we grouped similar tasks together to form categories, the basic wording was kept as close as possible to the original clinician’s reports as possible. A list of representative reported tasks (not the complete list) is displayed in ▶. The list is organized chronologically starting with activities that occur at the beginning of the day, followed by tasks associated with seeing the patient, those involving follow-up or tracking over time as well as general ongoing time management actions. As a result, some tasks appear twice, such as “determine what medications the patient is on” and “recent medical activity.” This action occurs both prior to a visit and during a visit.

Table 1.

Task Coverage by Medium

Black = full coverage; Grey = partial coverage; White = little or no coverage

In order to identify the relationship between tasks and available computerized functionality, we coded each task along six available mediums. CPRS refers to the patient information system display normally used by providers. CM refers to Care Management, a new multi-patient view. VistA refers to the screen that allows entrance to all of the VA computerized systems, and includes not only clinical programs, such as lab, radiology, and others, but also employee time tracking, provider education, and patient scheduling. Paper referred to either lists manually written, spreadsheets created by providers and printed, or lists printed out from VistA. The Phone column is self-explanatory. The Person category not only includes in-person conversations, but also e-mails. The cells were color coded to reflect coverage of tasks. Grey in the cell indicates the task is partially covered by the medium. Black indicates full coverage and white indicates little or no coverage.

When scanning across the table, it appears that most tasks have some technical support, either from CPRS, VistA, CM, or all three. However, very few were completely supported. Even something as straightforward as determining the patient’s actual medication profile was frustrating for clinicians because not all drugs are listed in the VA profile (especially non-VA drugs) and the actual dosage may have been changed, but a new order was not really entered. And, some tasks required manual extraction and storage on paper, such as tracking a patient’s preparation for outpatient surgery. The data could be extracted from CPRS or VistA, but the overview display of the information required that it be taken down on paper. Having to move from medium to medium was one of the most salient complaints regarding the information system heard during the interviews. Many of the tasks were done differently from site to site and those were usually the rows with all gray cells (indicating some support with no single medium providing full support).

Common Components Analysis

Common components across tasks were abstracted from the list of tasks in ▶ using an iterative process of abstraction. ▶ presents the common abstracted structure. First, all tasks require some cue to begin. Most of the time, the cue is the well-learned ordering sequence of procedures. For example, the cue to schedule an appointment often arises from the normal routine care of the patient. The patient finishes their appointment and knows to go to the check out desk and the appearance of the patient in front of the clerk asking for a next appointment cues the clerk. However, the cue to schedule an appointment could also come from reading an electronic note from another provider suggesting the need for a consult or the nurse’s putting a note on the provider’s door reporting on a phone call. In any scenario, the simple task of scheduling an appointment becomes a collaborative effort; the physician has to indicate that one is needed, the nurse may have to identify for the clerk the appropriate time, the patient has to know about the need for an appointment and the clerk has to actually enter the time into the computer. At each step, the previous action provides the cue for the next step and often, different individuals will do each component. Interestingly, the specific form that a sequence of cues would take was unique (but consistent) for each site and embedded into local workflow. However, the exact work processes used varied across sites.

Table 2.

Table 2 Common Task Components

Components	Description
Cueing	A stimulus or embedded procedure that initiates the beginning of a task, including time passed, lab result, or another provider.
Status	The current characteristic of the task, whether it is pending, resolved, etc.
Timing	Date due, latency until it begins, contingencies (e.g., on lab), or recurrent.
Communication	Results, responsibility, or status
Ownership	Person responsible for ensuring completion.
Linkage	The relationship of task to other tasks, e.g., parent-to-child.

The second common component is status. Tasks can be pending, in progress, ready to be communicated, or resolved. Status is an aspect of tasks that needs to be tracked continuously. Everyone sharing in a task requires this information. Tracking the status of tasks is a regular and repetitive component of workflow. The lack of a simple way to track the status of ongoing tasks was a common complaint and often would require substantial searching on the part of the provider. Resolved is a status that a task can take and because every task has a unique and complex manner of completion, the resolved status could be multi-faceted. Examples of types of resolutions include writing in the chart, communicating to another provider, communicating to the patient, and finishing the task itself. Commonly, the task has to be “crossed-off” from personal overview planning lists (which nearly every provider has) in addition to the other resolution actions.

Third, all tasks have a time component, including when they are due, length of time to complete the task (a test can be ordered, but scheduling requires a 2-week wait) or in the case of repetitive tasks (e.g., ordering monthly narcotics), the time interval. Often everyone in the clinic knows the time component implicitly because it is commonly held clinical knowledge (e.g., the time from a Coumadin change to the next INR test). Time information is an integral part of interpreting information and is always sought after if it is not available. When the information is not so clearly known, such as how long it has been since a patient has been called, or how many times a test has been repeated, then providers often need to resort to verbal communication.

Fourth, every task has an owner or owners. Identifying task ownership was a common information need. Unfortunately, in the current information system, identifying who owns the task is difficult, except when it is a formal order (which is only a small part of the communicated information). Not only does every task have an owner, but also as tasks evolve, sometimes the owner would change. Often, tasks involve significant coordination of activities between individuals. For example, the different tasks created as part of the process of getting an outpatient ready for surgery requires the primary care provider to provide a history and physical, nurses must schedule the surgery based on patient’s needs, clerks must determine the schedule and order protocol labs and other nurses must ensure that education is done and consents signed. Often, the person with whom the responsibility is shared is known by name, other times only by role. Determining the name of the person in a designated role is information that is currently not available on the computer and often requires substantial time to determine.

Most importantly, nearly all tasks require some form of communication to another person in the workplace. Usually this communication goes beyond simply documenting it in the notes. The material to be communicated varies, but may include one or more of the following: content, state, the responsible individual, and time. Tasks vary in the degree to which communication is an inherent component of the process, but often in order to get something done, another person has to be contacted. For many tasks, communication is central, getting an appointment means telling the patient, the clerk, and maybe the nurse when, why, and maybe even how to make the appointment. The triage nurse takes every patient interaction and often must contact 3–4 individuals before she can go on to the next patient’s call.

Some forms of communication are standardized in medical practice, such as orders and consults. Many others are less structured but are so embedded into workflow that clinicians are usually not aware of them. The variety, complexity, and idiosyncratic nature of local procedures for sharing information among providers were quite surprising. For example, the physician may not formally request that the intake nurse ask questions about medications or symptoms, but expects that those questions are asked and infers that if he or she does not get notified, there were no problems. When a patient calls in, the nurse answering the call makes a decision regarding the severity of the problem, which provider or providers should be told and when. Local procedures determine whether a decision is made by protocol, whether the physician is phoned, or whether an e-mail is sent. In some clinics, the nurse conducts an intake interview, in others a clerk asks the patients an initial set of questions and in still others, no initial interview takes place. Results from the interview may be written in a note, handwritten and taped to the door, written on paper and given to the patient, or simply communicated verbally. Sometimes several of these forms would be taking place at once.

In addition, every communication requires feedback and confirmation. Staff would go to great lengths with complicated work-arounds to ensure that their communications were received. These work-arounds ranged from requiring initials on a paper being physically passed from provider to provider to assigning a specific provider as a co-signer on an electronic note (meaning that an alert would be sent to the receiver notifying him or her of an incoming communication requiring signature).

Finally, most tasks are linked to other tasks. Specifying the links between tasks as well as role and time dependencies prior to the initiation of the task is part of planning care and an intricate component of the implementation of sequences of care, such as guidelines. Because the links must be explicit, observable, and traceable, they are often recorded on paper, such as personal lists of patients requiring follow-up, formal tracking sheets for Coumadin patients that are posted, or lists of patients who need narcotic renewals. Progress notes are rarely used because they cannot be updated and cannot be linked. Neither CPRS nor CM support detailed specification of task linkage, nor is there a common “work board” to which all relevant providers could refer.

Information Management Goals

Four general categories (▶) of information management goals emerged from the analysis. The higher-level goals are generalized patterns of strategies that resulted from extensive analysis of the transcripts. They are complex behavioral patterns that bridge the human, work processes, and the computer. Although similar goals are present across individuals, the exact strategy(ies) may differ significantly. At times these strategies are episodic and observable, but more often they are implicit, that is they are completely embedded into the work processes. Often, providers would not be aware of them and only extensive questioning would reveal their presence. In most cases, these high-level information-processing goals were only apparent to the investigators after intensive reviews of the transcripts.

Table 3.

Table 3 Information Management Goals

Goals	Description
Relevance screening	Reducing the amount of information to sort through, highlighting relevant data, making relevant cues salient.
Ensuring accuracy	Increasing the size and scope of presented information in order to not miss anything, setting up redundant processes, correlating items, or checking multiple sources.
Minimizing memory load	Tracking the status of tasks, setting up automatic procedures that serve as reminders, setting up structured division of labor or setting up lists of tasks.
Negotiating responsibility	Giving and receiving orders, consults, sharing information, and hand-offs.

The first goal, Relevance Screening included strategies to narrow the field of attention by setting up mechanisms to screen, sort, and prioritize information. These kinds of strategies were the most numerous. For example, providers would report having the clerks add the reason for the visit into the scheduling package so that they would not have to search the notes looking for a likely reason for the upcoming visit during preparation time. Every day a clinic list would be printed for all staff and the list contained the reason. Or, providers might customize their notification screen so that only the highest priority alerts would be visible on the notification window, thereby removing minimizing sorting effort. Selectively screening notes based on knowing the provider (Dr. Smith is known to be accurate and thorough) or using only one’s own notes was very common. With over 100 notes displayed in chronological order, being able to read only a few notes at a time is an important time saver. Since electronic notes can be considerably longer than written notes, taking the time to scan through all of the notes can become almost impossible. Of course, recent ER notes would be read as well as recent results from selected consults. Because of the amount and density of the information, most providers would have to resort to personally writing on paper important highlights from the patient history. They would avoid large categories of notes if they believed that the amount of material presented would not contain sufficient relevant information given the effort. For example, most providers reported avoiding notes that used templates altogether (which included many nursing notes). Other times, the strategy would exist at the whole clinic level, such as having all alerts sent to a covering or triage nurse. The nurse assigned to the task would then sift through the alerts and communicate only the most acute or important items. The overarching goal with this strategy is to minimize the information overload that accompanies the use of an electronic record.

The next category, called Ensuring Accuracy seemed almost the opposite strategy and included actions to increase the available information. The underlying goal of this group of strategies appeared to be to ensure accuracy and completeness. Many providers expressed significant distrust in the accuracy of the system and would try to build in redundancy. Some providers would have the results of all of their orders alerted back to them to avoid missing important information or have clinic nurses review daily labs and alert them of patient problems. Or, they might copy the entire results of lab panels, procedures, or x-rays into open progress notes as a reference when they are talking to the patient and then delete the ones not used for the final completion of their notes in order to ensure that they did not miss anything. Some providers would give out their personal number to make sure that the patient could get in contact with them directly or would set up clinic systems to call every patient following a visit. Calling the specialist after making a referral was often done to make sure that they understood the reason for the consult. The simplest form of this group of strategies is the ubiquitous scanning of all sources of information, ranging from the numerous electronic notes, VISTA, e-mail, and paper notes. Of course, there were significant individual differences between providers. The overarching goal associated with these sets of strategies is to maximize certainty and confidence in their retrieval of information.

The third category, Minimizing Memory Load, included strategies to set up personal reminder systems or to keep track of required tasks over time. A large variety of strategies were used for this purpose, such as a provider’s creating an unsigned addendum or progress note as a reminder to complete a task (providers receive alerts about unsigned notes). Another strategy in this category was to use the VistA e-mail to send oneself a message on a specific date. Or, a fake clinic might be created filled with appointments of patients requiring follow-up (a clinic list can then be scanned and reviewed by everyone and comes up on a pre-determined day). The patients listed in one fake clinic were those needing monthly refill of narcotics for a single provider, thereby allowing the names to come up on his appointment list for that clinic on that day. Almost all providers kept a paper “to do” list for the day, marking off tasks as they were finished and adding others as they came up. Paper calendars would be used because they could be posted and serve as highly salient visual cues. Finally, every clinic had numerous visual aids that would cue staff as to whom the patient had seen, who would be seeing the patient next, and what was left to do for that patient’s appointment. These reminders were largely in the form of check-off lists, patient folders located in different places, and names written and erased on boards visible to all.

The fourth category, Negotiating Responsibility, included a variety of processes to assign tasks or negotiate hand-offs. There are basically two kinds of hand-offs, within roles and between roles. Nurses and physicians generally divide up the workload according to standard roles, although there is a great deal of gray area between the two. Nurses would write orders and conduct procedures without orders if that was standard clinic practice. No clinic was alike in terms of the extent to which protocols were used. Other providers (such as physical therapy, social work, or dieticians) would be assigned the patient using a variety of mechanisms. Sometimes, ordering providers would simply write that they wanted the social worker to see the patient in a progress note, expecting that the social worker would read it (as a member of the team). Other times, the assignment would be done through e-mails, creation of a virtual clinic that was attached to that provider, or more formally through the CPRS consult package. The diversity in procedures between clinics was notable. The need for administration to track workload was an important consideration in making these decisions.

Hand-offs within roles occurred either at change of shift (usually nurses) or between providers when one went on vacation or residents rotated off the clinic. These issues were dealt with differently at every site, but often required significant redirecting of alerts, re-organizing the team structure in the computer, and canceling or moving appointments. This process is quite complicated because most providers work in teams and so sometimes the whole team needs to be alerted. The management of teams is an older functionality of VISTA and coordinates the alerting and notification structure of CPRS. Interestingly, it became clear in our interviews that the mechanisms of team management (and how things were set up locally) were often only known by a very small number of individuals at each site, if at all. In addition, these procedures differed substantially across sites making it quite difficult for developers of new programs (e.g., Care Management) to understand standard practice in the field. Some sites had an individual in the clinic assigned to “manage” team lists. This person would ensure that every provider was on the correct list and when clinicians rotated in and out, the lists were updated. At other sites, the clinicians were taught how to put themselves on and off lists and were expected to manage it themselves. And, in other sites, the computer office alone provided list management.

In summary, providers were observed to be engaging in a wide variety of strategies that allowed them to effectively adapt to the computerized information environment. The overall goals of minimizing cognitive load, enhancing accuracy, remembering important information, and negotiating responsibility were noted. The specific strategies used to meet these goals varied significantly between individuals within a clinic and also substantially across clinics.

Discussion

Findings from this CTA analysis extend our understanding of user’s information management strategies in a mature CPOE primary care setting. All ten sites had been fully committed to CPOE for over four years and practice patterns had evolved and developed around the system. As a result, both individuals and the system underwent a mutual transformation. This evolutionary aspect of technical systems, an issue that has been discussed by several authors associated with the sociocultural approach was clearly in evidence in this study. ^9,10,13

One of the most important findings from this work is the analysis of information management goals, which deepens our general understanding of user’s interactions with information systems. The strategies identified through this analysis highlight fundamental cognitive activities relating to the management of information in a clinical system. These strategies are congruent with the concepts of “ecological rationality” espoused by Simon, Gigerenzer, and others. ^7,43 As Girgerenzer (2000) notes, “Ecological rationality refers to the study of how cognitive strategies exploit the representation and structure of information in the environment to make reasonable judgments and decisions” (p. 57). ⁷ Any computerized information system needs to be designed to support strategies that allow for “fast and frugal” decision-making. ^23,44 These strategies are often used by decision-makers in the real world. They include multiple methods for rapidly identifying the minimally sufficient data to support a decision, ²³ attempts to efficiently balance accuracy versus speed, ⁴⁵ and creating personal information displays that support recognition-primed actions. ^8,46,47 Even sophisticated CPOE systems may not support these goals as discussed by Horsky et al. (2003). ³⁴ Designing systems to support these activities are essential, as no amount of perfection in design would eliminate the need for flexibility in a complex medical environment. The specific “epistemological goals” at play at any one time vary across individuals and even minute by minute within the same individual, suggesting that efforts to “standardize” processes might be limited.

In addition to the goal analysis a few points are especially important to note. First, an examination of the list of common tasks identified in these primary care clinics revealed that most were not fully supported by any single available technology, although most were covered by at least one. Several tasks, however, were not covered at all. Although it was possible to use a variety of communication tools, none were integrated into CPRS. The tracking, correlating, and overview tasks commonly associated with planning, forecasting, and complex decision-making were not well supported. ^48–50

Another important finding is that for nearly all tasks, some form of communication with other providers was necessary. And, although documenting in the electronic progress notes was important, other actions were usually required to draw the attention of relevant other staff, to negotiate mutual responsibility, and to confirm information exchange. These “attention-getting” activities were either embedded into the work processes (social workers knew to read every progress note by the physician) or involved shared check-off lists and even verbal interactions. The complexity and pervasiveness of communication actions become visible in a computerized environment. ^2,51–53 And when they are not well addressed, they are often a source of error. ^15,54 It was beyond the scope of this study to analyze communication patterns, but this would be an important area for future investigations.

The common components analysis revealed that it is possible to discern a common structure across tasks. The core components of each task were found to include cueing, communication, time, state, ownership, and linkage. Cueing and some aspects of time were well supported in the information system, but communication, state, and linkage were particularly not supported. The result was that users had to interact with multiple sources of data, sign-ons, and formats in order to get things done. The result was the evolution of significant work-arounds embedded into the work processes. In future redesigns, these generic components can be systematically incorporated into available functionality, depending on the specific task. Ensuring that tasks can be initiated, conducted, and finished with minimal navigation between screens, software, and tools is essential.

Another important and interesting finding was that although the tasks and goals were common, each site was unique in the specific processes used. Several factors may provide explanations. One possible factor was the variability in team coherence. Although we did not measure this factor, it was clear that those clinics where the nurses, physicians, clerks, and social workers were tightly organized had very different strategies than those where the different roles functioned independently and in parallel. Another factor creating differences between sites was the type and intensity of support from the IT office. Perhaps, the work-arounds visible in many of our interviews arose because of a failure of the technical office to adequately support the socio-clinical system. ⁵⁵ Another possible source of the variation is the degree of user expertise, both clinical and technical. Sites where there were substantial numbers of trainees, part-time workers, and frequent turn-arounds of trainees might require different strategies (more structured and automatic) than those where individuals could simply learn procedures and then would be able to stay with those procedures for long periods of time. These variables and the relationship to specific strategies used are important areas for future research.

Limitations

In a study using this approach, the goal is not to compare sites, or to make rate-based conclusions regarding the incidence of specific tasks. Nor can a study with this design provide any strong inferences about the causal relationship between the information system, task structures, strategies, and outcomes. The results from this study are valuable for providing an in-depth and structured analysis of how clinicians are handling information in a computerized environment and to generate hypotheses for future investigations.

Conclusions and Recommendations

The findings from this study support some important recommendations. First, implementation of every information system will result in users creating unique strategies that either expand functionality or work around limitations. Understanding those strategies should be a component of all design life cycles. Secondly, a CPOE system can produce significant information overload problems for users. Users can be helped to manage the problem by minimizing the different technologies, supporting all of the task core components, and by providing overview functionality. Third, communication between providers is a ubiquitous aspect of health care delivery. Information systems may not be able to replace communication systems but should not interfere with them. Communication is often embedded into artifacts and work processes that may not be clearly visible. Finally, EHR system designers should sample sites and users widely to ensure representatiaveness, because adaptation is creative and idiosyncratic.

Preparing to See a Patient

• 1 Please recall a typical day when you are working in the clinic. What do you usually do to get ready, to prepare yourself? <remember to probe with you do XX in order to and you accomplish XX, by doing ?>

  • • When do you start getting ready, usually? How do you arrange the information you need?
  • • When do you review your notifications in CPRS? When do you answer your phone messages? (How many?) When do you look at your e-mail? (How many?) Do you use e-mail?
  • • How do you keep track of things that you need to do as they arise from these sources?
  • • Do you keep a list of any kind? How do you know you have accomplished the tasks on the list?
  • • When you are reviewing notes, what note do you check for first?
  • • Are there any kinds of notes that you do not review?
  • • What are you trying to do at this point? What are your most pressing concerns?
  • • How do you keep track of items that you find that you need to focus on for each patient as you review the patient’s chart?
• 2 Do you find the documentation on the patient comprehensive, accurate, and available?

  • • What do you think of templates? What do you think of cut and paste methods?

• 3 How would you describe your team?

• 4 Do you communicate with your team on a regular basis?

• 5 Do you start a note? How do you usually set it up? What are you trying to accomplish with the set-up?

• 6 When there is something you don’t know (a disease, a drug), what do you usually do?

During the Patient Visit

• 1 As you visit with the patient, do you interact with the computer as well?

• 2 Does the physical set up of the room, the patient, and the computer work well?

• 3 What are the central areas of focus during the patient’s visit?

• 4 What are your main goals in documenting the patient’s visit?

Follow-up After the Visit

• 1 After the patient leaves, what do you usually do? Do you finish the documentation then or later?

  • • When you document, do you ever cut and paste? Do you use templates?
• 2 What kind of methods do you use to communicate with your patients?

  • • Do you mail them lab results?
  • • Do you call abnormals? Do others call for you? Do you give out your phone number?
• 3 How do you follow-up on their labs, procedures, results, etc?

  • • Do you ever know if a patient has missed a consult or procedure before the patient comes in next?
  • • Do you try to find out what happened to them?

• 4 When you are gone, who covers for you? How is that set up?

• 5 How often do you communicate with members of your team about a patient?

• 6 What are the methods you use for communicating and why?

  • • How often do you use e-mail; VISTA mail? Do you create a note with a co-signer?

Appendix A

Interview Questions

Introduction: We are conducting a research study on how the computer interfaces with clinical workflow in primary care in order to make the computer more useful. This study is funded by the computer development office through HSR&D. We would like to ask you a few questions about how you organize your workflow in a typical day. We are tape recording your answers so that we can more accurately capture what people are saying. Your name will not be on the tape recording nor will it be associated in any way with your responses. Because it is a research study, we would like you to fill out the informed consent. <INFORMED CONSENT>

We are organizing the interview chronologically, starting with what you do to prepare for your day, going through your visit with the patient and finishing with what you do to follow-up. At each stage, I am going to focus on what you are thinking, what goals you are considering, what activities you do to meet these goals, and your concerns. Remember, there are no right answers. <The items in italics are only asked if they do not come up in conversation>