When You Come to a Fork in the Road, Take It: Strategy Selection in Order Entry

Abstract

This study investigates how CPOE system users choose data input strategies for entering clinical orders. Complex systems often allow more than one way to complete a task. However, the appropriate entry strategy in the context of a specific clinical workflow situation may not be apparent to users. We have conducted a cognitive analysis of user interaction strategies for entering IV injection orders using a commercial CPOE system. We characterized the set of available information resources in the system interface and in the users’ memory, and evaluated how effectively the application supported decision-making processes. Seven internal medicine residents participated in an experiment entering IV heparin orders to manage anticoagulation therapy. The analysis showed that efficiency was contingent upon high level of procedural and conceptual system knowledge. CPOE interface design needs to conform to decision-making and workflow processes if the technology is to become an effective clinical tool.

INTRODUCTION

The considerable number of medical errors, their cost and detrimental effect on patient safety has been at the forefront of public debate for years. Errors are routine in complex human activity although most will not cause harm. A fraction of those, however, will have dramatic consequences, especially if they occur in such critical industries as mass transport, power generation or healthcare.¹

Technology solutions to error prevention in clinical medicine seem to be less effective today than initially expected. Their implementation has been often marred by numerous problems or outright failures.² Computer-based provider order entry systems (CPOE) were developed in part to eliminate errors associated with illegible handwriting and to increase speed and quality of communication among clinicians. Research evidence suggests that the use of CPOE significantly lowers the rate of serious medication errors.³ It has become apparent, however, that newly adopted technology also tends to alter familiar workflow routines, force changes to longstanding practices, and as a result may introduce new sources of error intrinsic to human interaction with complex technology.⁴

Current research findings from studies of human perception and cognition related to work with technology suggest that many errors are neither solely attributable to lapses in performance or to defects in technology, but arise as a product of their interaction.⁵

This paper describes how clinicians select and change their data input strategy when entering orders for the administration of IV heparin injections, and how their decisions are affected by both the cognitive usability characteristics of the interface and their conceptual understanding of the system. This analysis is part of an ongoing research effort by the authors to characterize the cognitive dimensions of clinician interaction with CPOE^6–8 and examines how working experience, skill, expertise and the availability of decision support features shape interactive behavior.

Our research objective was to 1) characterize the decision process leading to the selection of one completion strategy over another, and 2) examine how the availability and type of cognitive resources affects the choice of action.

MULTIPLE COMPLETION STRATEGIES

CPOE systems are multifaceted and versatile applications that allow for completing one task in several different ways. We term these goal-directed completion methods interaction strategies. Two strategies may not be equally suitable or effective, however, when the same task is carried out in different work contexts. There may be a speed or accuracy advantage in one strategy over another when working conditions change. For example, entering a routine order when all required clinical data are available may not necessitate the use of a strategy that invokes decision support features but also takes more steps and longer time to complete.

Clinical software is sometimes optimized for fast completion of simple tasks and it is expected that users will eventually refine their more advanced skills through practice.⁹ However, many users never progress beyond a minimally sufficient skill level even after years of experience.¹⁰ Bhavnani and John⁹ note that the knowledge of efficient strategies is often expressed neither in the task description nor it is suggested by design or layout of interface controls. The optimal completion method for a specific work context needs to be made explicit to users if clues in the interface cannot indicate the next step. This understanding can only be acquired from manuals and in training, or by trial and error over time. Since physician time is scarce and expensive, the cost of acquiring a strategic level of system knowledge is impractical. However, comprehension of tasks and tool usage alone is inadequate for gaining expertise.

STRATEGY SELECTION BY A HEURISTIC

Users may encounter several decision points when completing a task in which they need to select a suitable interaction strategy (see Figure 2 for an example). Fu and Gray observed in an interactive behavior study¹¹ that people chose to use suboptimal strategies even when they had knowledge of an optimal procedure. They accounted for this seemingly paradoxical finding by the considerable cognitive effort required to mentally look ahead and process task-specific information to determine which procedure is optimal. If the immediate cost of comparison outweighs the expected benefits of selecting a better strategy, users are likely to employ a decision heuristic that may select less than optimal but an acceptable strategy.

Figure 2.

Task structure of an IV heparin order

For example, subjects in this study were asked to increase the dose of a currently active IV heparin injection order. They had to discontinue the order and enter a new one with a higher dose. One strategy (DC/Reorder in Figure 1) initiated from a context menu required only two steps. First, the active order was discontinued (DC) automatically and the system created its copy with a new timestamp. The user then modified the dose and signed. The second strategy (New Order in Figure 1) necessitated four steps: manual canceling of the old order, searching menus for appropriate new order, and entering new dose. In addition, one of the idiosyncrasies of this particular CPOE was the fact that a decision support feature that calculated heparin dose based on patient weight was only triggered during the second (New Order) strategy but not when the first one was followed. However, the interface did not indicate the trade-off of speed or manual calculation this choice entailed. Users had to recall this information from memory, if they were even aware of these procedural details.

Figure 1.

Two alternative order entry strategies

Gigerenzer and Selten¹² consider the mind to function in such situations as an adaptive toolbox that reacts to the environment by choosing a heuristic contingent on task demand. Such heuristics can produce fast decisions with minimum of information by exploiting the way information is structured in particular environments.

Insights into decision processes can inform design so that interfaces may guide users to follow optimal strategy for any specific task. The combination and layout of screen controls such as buttons, context menus, pick lists, hyperlinks or visual clues may impose constraints on user behavior that can lead to the selection of strategies optimal for a given context.

COGNITIVE RESOURCES

Most cognitive tasks have internal and external components: knowledge and processes in the mind of the user and those in the environment.¹³ In computer interaction, the user brings a set of these knowledge resources to the task as skills and experience while the interface contains controls and data.

The distributed resources model proposed by Wright and colleagues¹⁴ addresses the question of “what information is required to carry out a task and where should it be located, as an interface object or as something that is mentally represented to the user.” The present authors extended this model to include descriptive constructs specific to medical domains and to analyze the cognitive complexity of a interfaces.⁸ The method has been used to analyze how the relative distribution of internal and external resources affects the choice of interaction strategy.

METHODS

We asked seven physicians to enter appropriate orders for a clinical scenario using a CPOE system. The task required adjustment of anticoagulation therapy by ordering a heparin bolus IV injection, increase of IV heparin drip dose, and ordering a PTT (partial thromboplastin time) corollary lab. Subjects were instructed to verbalize their thoughts while completing the task (a think-aloud protocol).

Clinical scenario

You are covering a patient with mechanical aortic valve that has been admitted and heparinized for reversal of oral anticoagulation and elective dental extractions. This 50 kg woman was bolused with 4000 units of heparin and placed on a drip of 900 units per hour continuous IV, and also treated with Aquamephyton 5mg IV. The PTT was drawn 6 hours after initiation of heparin was 37 (control 32). Write appropriate orders to modify this patient’s anticoagulation therapy.

Subjects

Seven internal medicine residents with 1–2 years of CPOE experience and a range of 2–5 years of clinical experience were recruited as a sample of convenience by department advertising. They had received several hours of mandatory CPOE training.

CPOE

Commercially available system, pre-loaded with simulated patient data. Each experimental entry session took about 30 minutes to complete.

Interaction analysis

Screens were videotaped so that mouse movements, actions and screen transitions could be later analyzed. User verbalizations were recorded, transcribed and coded for a cognitive task analysis. Data from Subject 2 were analyzed in detail to characterize the decision making processes during task completion. This subject was selected because of rich and detailed description of thoughts and actions that lead to changes of strategies and alternatives.

The researchers have determined the structure of the task (reference standard) and identified two possible completion strategies before the experiment was conducted. Data collected from all subjects included the number and type of strategies used and the completeness of the resulting order set. The detailed analysis of Subject 2 consisted of reconstructing the trajectory of the task, including the sequence of goals and the occurrence of decision points where changes of strategy were considered. We determined at every apparent goal and decision point what information was necessary to complete it or make a decision, and whether it had to be recalled from memory or was otherwise given by the system.

The results of a detailed analysis of one subject were compared to cumulative data collected from all participants. This approach allowed us to characterize the interaction by extrapolating performance details of an individual to group results.

RESULTS

The task structure is shown in Figure 2. Necessary actions fell into three broad categories, represented as frames in the diagram. The first stage included the gathering and interpretation of data, evaluating the case and deciding on how to change the therapy. In the second stage, users needed to enter two heparin orders (bolus and a drip) and a PTT laboratory order.

They had to select a strategy (DC/Reorder or New Order), and in case of a reorder calculate weight-based doses for heparin injections. The third part of the process consisted of reviewing completeness and finalizing by signing the new orders. While the sequence of the three categories of actions was invariant, the actions and decisions within each category could be completed in any order.

Cumulative results

Table 1 summarizes the number and choice of strategies used by all subjects and the completeness of the resulting set. All subjects except S5 ordered both the bolus and the drip. Three subjects (3, 6 and 7) did not enter the corollary PTT order. Four subjects (2, 3, 6 and 7) used a different strategy for bolus and for drip orders. S2 entered the drip order twice since it was inadvertently deleted during recovery from another error.

Table 1.

Strategy selections for bolus and drip orders; completeness of final order set for all subjects

Strategy/Subject	S1	S2	S3	S4	S5	S6	S7	Comment
DC/Reorder								No DS available but faster completion
Bolus							✓
Drip		✓	✓		✓	✓
New Order								Auto dose calculation but longer procedure
Bolus	✓	✓	✓	✓		✓
Drip	✓	✓		✓			✓
Strategies	1	2	2	1	1	2	2	Number of strategies used
PTT lab ordered	✓	✓		✓	✓			Inclusion of required corollary order

There was no clear preference for one strategy evident among the subjects although more orders (9 out of 14) were entered using the longer “New Order” strategy which triggers decision support. However, only two subjects (S2 and S3) were aware that weight-based dose calculation was performed by the CPOE and would be triggered at some point when new heparin orders were entered. Most subjects estimated or computed the dose in their heads before the system-calculated values were displayed. Three subjects expressly stated that they would have looked up dosing guideline information from various sources (e.g., a colleague, the Washington Manual) without realizing that precisely such decision support would be provided by the system.

Individual analysis

The findings from a detailed cognitive task analysis of Subject 2 are shown in Table 2. Numbered lines (1–5) in the first column mark five decision points in the process where the subject selected or switched completion strategies. Lines marked by letters (A–D) refer to the four goal states intended to be completed at that point (New bolus, Reorder drip, Order PTT lab, Sign) and also correspond with labels of the abstract task structure in Figure 2. Memory Recall column lists the data and information necessary for completing a goal that had to be recalled from memory (internal resources). System Support shows whether the interface contained data, clues or any indication of how to proceed in goal completion (external resources).

Table 2.

Analysis of goal sequence and interaction strategy for Subject 2 Legend: 1–5 Decision points where interaction strategies were selected; A–D Goal states (tasks to complete)

# Goal State/Decision	Memory Recall	System Support	Comments
Open chart	Patient name – Match	Name list
Evaluate case Dose adjustment	General medical knowledge	Electronic chart, case description	Patient data on screen and in notes
1 Drip – DC / Reorder	DC/Reorder or new order? DS available? Which faster?	none	Strategy dependent on experience, skill
B Reorder drip	Procedural knowledge	Context menu
2 Bolus – New Order	PTT values near normal, bolus order also needed	n/a	Decision driven by medical reasoning
A New bolus	Procedural knowledge	Context menu	Order A cancelled
3 Heparin IV order set 4 Heparin IV individual orders	Order set is available, may be preferable to individual orders Individual orders may be simpler, include calculation	none none	Heuristic decision, no prior experience w set Previous decision reversed, set confusing
A New bolus B Reorder drip	Procedural knowledge Procedural knowledge	Dose calculation Context menu	DS triggers reminder Aborted – needs DS
5 Drip – New Order	New orders procedure triggers auto calculation	none	Also needs to cancel current drip order
-- Error Cancellation of the previous order also deleted the just entered but unsigned bolus order --
A New bolus	Procedural knowledge	Dose calculation	DS triggers reminder
D Sign orders		Sign button	Saves current orders
B New drip	Procedural knowledge	Dose calculation	Second attempt
C Order PTT laboratory	Corollary order	Reminder	Included in bolus DS
D Sign orders		Sign button	Saves current orders

The subject used both strategies to enter orders but changed from one to another five times during the session. The initial decision was to DC/Reorder a drip (line 1). This goal was changed just after the process was started and the subject realized that a bolus would also be needed (2). Although the subject had no prior experience with a specific heparin IV order set (as stated), a heuristic decision was made based likely on the fact that order sets were in general easier to work with than single orders (3). After the set was displayed, the subject concluded that it was not appropriate in the given context, was in fact confusing, and decided to enter orders individually (4). After a bolus was ordered using the “New Order” strategy (A), a drip was started as “DC/Reorder” (B). This strategy lacked the automated dose calculation, as the subject knew, and therefore cancelled the procedure and switched to “New Order” (5). Because the just completed bolus order was not yet signed and permanently stored, cancellation of the latter order also caused inadvertent deletion of the bolus. After that, the subject repeated the entire process, this time using the same “New Order” strategy for both orders.

The subject manifested a sound conceptual grasp of some of the more intricate ordering procedures and their expected outcomes but still failed to use the system efficiently and made an error that added time and recovery effort to the task. There were three attempts made at entering each order before success, and two Sign procedures (D) when one would have been sufficient. The subject employed heuristics to decide on using an order set and to switch strategies for taking advantage of decision support. However, there were unintended consequences to both decisions of which the user was not aware that in the end made the task longer and more difficult. This subject experienced perhaps more difficulty than others but the case reflects the system’s failure to provide adequate resources to support interaction.

Although the system provided quick and useful dose calculation based on stored patient weight, users would need to be aware of its existence to use it effectively. At the points in the reasoning process where decisions were in fact made (1–5), the system lacked effective decision support resources.

Users without strong conceptual system knowledge may resort to employing a satisficing heuristic, e.g., selecting a strategy that perhaps worked previously for a similar task. Even if all data (e.g., the number of steps involved, time to compute the dose, etc.) could be recalled, users are unlikely to engage in estimates and comparisons that are cognitively too taxing and may again use a heuristic. The optimal strategy may not be learned even with extended work experience.

CONCLUSION

Entering orders into a computer-based system is an inherently complex process. It can be exacerbated or minimized by interface and support design.

This cognitive resources analysis allowed us to account for patterns of user behavior and to characterize how well the reasoning and decision making processes of users were supported by information available in the CPOE system. Although the system had integrated decision support to speed up ordering and to make heparin dosing more accurate, its implementation was not optimized to fit into the workflow and the decision process of clinical users. It did not provide necessary information at the time when the decisions were made. Successful interaction was contingent upon thorough conceptual and procedural knowledge of the system. The screens gave few clues and insufficient guidance for selecting the best possible strategy for completing orders.

If users need to rely mostly on memory and past experience to be efficient, it presumes that they need to receive extensive training and would require long time to develop proficient skills. In clinical medicine, such an approach is not feasible, as physician time is scarce. Interface design then has to incorporate such features and characteristics that relieve the user from excessive cognitive effort and accommodate support functions to fit into clinicians’ workflow and medical reasoning patterns. Methods and frameworks from cognitive engineering have been successfully applied towards that end in the past.¹⁵

A recent study of CPOE systems ¹⁶ concluded that without specific attention to usability and workflow considerations, this technology that is effective in reducing the rate of medical error may never achieve its full potential, or even make the situation worse.