The Cognitive Complexity of a Provider Order Entry Interface

Abstract

Computer-based provider order entry (POE) can reduce the frequency of preventable medical errors. However, overly complex interfaces frequently pose a challenge to users and impede clinical efficacy. We present a cognitive analysis of clinician interaction with a commercial POE system. Our investigation was informed by the distributed resources model, a novel approach designed to describe the dimensions of user interfaces that introduce unnecessary cognitive complexity. This approach characterizes the relative distribution of user’s internal representations and external representations embodied in the system or environmental artifacts. The research consisted of two component analyses: a modified cognitive walkthrough evaluation and a simulated clinical ordering task performed by seven physicians. The analysis revealed that the configuration of resources placed unnecessarily heavy cognitive demands on the user, especially those who lacked a robust conceptual model of the system. The resources model was also used to account for patterns of errors produced by clinicians.

INTRODUCTION

There is a growing recognition that many errors are neither solely attributable to lapses in human performance or to flawed technology, but develop as a product of their interaction¹. Cognitive engineering is an interdisciplinary approach to understanding the complexity of the intellectual partnership between humans and machines² and is a useful approach for the investigation of interaction errors. Errors are routine in most areas of complex human performance and a fraction will have dramatic consequences³. Newly adopted technologies tend to alter work habits and familiar practices, and as a result may introduce new sources of error².

We present an approach to the analysis of a computer-based provider order-entry system (POE), intended to characterize the cognitive demands of interaction with this complex technology. Order entry systems were developed in part to eliminate errors associated with hand-written ordering and to increase the speed and quality of communication between clinicians. There is evidence to suggest that systems currently in use have already reduced the rate of medication errors and improved patient care⁴. However, POE implementation has proven to be a significant challenge⁵, often resulting in workflow reorganization in hospitals, and requiring clinicians to learn advanced information applications. Recently, discontented physicians associated with a large California hospital network even forced a halt to the rollout of POE. Complex POE interfaces impose a steep learning curve on the novice user while the benefits of the system in speed and accuracy may not be attained for some time. This complexity can be partly attributed to the multifaceted nature of clinical medicine. However, a poorly designed POE interface not only slows down the clinician but may introduce a new source of medical errors into the ordering process, intrinsic to human interaction with information technology⁶.

These observations suggest the need for characterizing dimensions of user interfaces to identify sources of unnecessary cognitive complexity that increase cognitive load and divert attention from the clinical task at hand. Theoretical frameworks and methods from cognitive engineering can productively inform research on the evaluation of medical computer interfaces⁷. A cognitive engineering approach has been employed to diagnose the pattern of errors involved in the use of a patient controlled analgesic device⁸, and to investigate changes in diagnostic reasoning strategies of clinicians using an electronic medical record system⁷.

The research presented in this paper is informed by a theoretical framework that incorporates extensions of Norman’s theory of action⁹, in particular the cognitive walkthrough¹⁰, and recent developments in distributed cognition methods of human-computer interaction (HCI) analysis. It is useful to think of HCI as a continuous process of cyclical interaction, recognizing changes of state in the environment and acting on the environment to bring about new changes in state. The model is cyclical in the sense that action is informed by the configuration of resources represented in the interaction at a particular time - either externally in the interface or internally in the mind of the user¹¹. Cognition is then viewed as a process of coordinating distributed internal and external representations.

The design implications of this idea have long been recognized. For example, Norman⁹ argued that well-designed artifacts could reduce the need for users to remember large amounts of information, whereas poorly designed artifacts increased demands on the user’s working memory. Most cognitive tasks are described as having an internal and external component¹². The reasoning process then involves coordinating these representations to derive new information. Different but functionally equivalent displays (i.e., supporting the same set of functions) can have dissimilar representational effects. For example, certain external representations (e.g., pick-lists) can minimize the difficulty of a task by supporting recognition-based memory or perceptual judgments rather than free recall. This is analogous to the differences between GUIs and command line interfaces.

The distributed resources model proposed by Wright et al¹¹ addresses the question of “what information is required to carry out a task and where should it be located, as an interface object or as something that is mentally represented to the user.” In other words, the user brings a set of resources to the interaction in the form of his or her knowledge and experiences. Similarly, “system resources” such as dialogues boxes, buttons, and help facilities guide the interaction in specific ways. These can be categorized and quantified. The relative differences in the distribution of representations (internal and external) are central in determining the efficacy of a system designed to support a complex task. This model includes a characterization of abstract information structures (i.e., resource types) that can be used to analyze interaction. How these information structures are realized in interfaces will critically affect the quality of user interaction. This may enhance or impede performance.

The authors¹¹ propose six abstract information structures Plans - resources for action that include a sequence of actions and anticipated states. Goals - states the user wants to achieve, generated internally or emerging from system interaction. Affordances - links, buttons, or menus that suggest possible next actions at a given state of the system. History – the part of a plan already accomplished (e.g., a list of previously visited sites in a web browser). Action-effect relations - indicate the causal relationship between an action and the effected change in state. State - the current configuration of resources, as embodied in the display screen at a given point.

We needed to induce additional information structures to adequately describe the considerable interface complexity of this POE. Biomedical knowledge was differentiated into patient-specific (e.g., age, blood pressure), general medical (admission order structure) and institution-specific (formulary, locations). Each information structure was divided into internal and external representation. For instance, a patient-specific external resource could be a blood pressure reading displayed on the screen or in a clinical note, whereas the recall of the patient’s history of labile hypertension from the hospital rounds could be an internal resource. A conceptual model of the system (internal representation) corresponds to users’ understanding of how the system works.

Our research objective was to evaluate a complex provider order entry system using the distributed resources framework. Specifically, we wanted to 1) analyze how the situational distribution of cognitive resources may result in performance variation or the creation of opportunities for error, and 2) use this model to evaluate the performance of clinicians using the POE system in an experimental task.

METHODS

The analysis of this POE system consisted of two complementary approaches. First, we performed a modified version of the cognitive walkthrough informed by the distributed resources model to describe and quantify the relative distribution of cognitive resources active during clinical ordering. We then conducted an experimental task in which seven physicians were asked to enter appropriate orders for a given clinical scenario. The combination of these two methods, the cognitive analysis and empirical data collection, was intended to a) characterize the cognitive demands of the ordering task, b) to evaluate how well the demands are supported by available resources, and c) to identify possible sources of error. The focus is on an in-depth qualitative analysis of performance, thus necessitating fewer subjects.

A development version of commercially available POE system was used for both the walkthrough analysis and for data collection. A general patient admission order scenario was developed and used because it is reasonably well structured, largely invariant set of constituent orders, and the fact that it doesn’t require specialized medical expertise. The scenario is presented below.

A 65-year-old man with a medical history of untreated labile hypertension and iodine sensitivity is admitted to the hospital by his primary physician, Dr. Lesion. He has an indwelling Foley catheter in place and is admitted early in the morning for a TURP later the same day. Pre-operative testing was done as an outpatient two days prior to admission, and the patient comes with copies of the results. Dr. Lesion calls and asks you to admit the patient, get an IV going, and put the labs on the chart for the urologist that will come by later to write pre-operative orders. Write the admission orders for this patient.

The task required subjects to develop a problem representation of the clinical scenario and 1) assess the patient condition, 2) record noteworthy findings, and 3) enter orders as requested. It was important to note the patient’s iodine sensitivity and that he has an indwelling Foley catheter that necessitates a nursing order not included with the available order entry set.

System walkthrough

This analysis was designed to simulate an expert completing the patient admission order entry task. It was completed by two researchers with the assistance of a physician who was also an expert POE user. Medical orders appropriate for the given scenario were entered and the relative distribution of available resources was recorded at every system state and classified according to the notational model described in the results section (Figure 1 provides an example). Opportunities for potential errors and their possible medically adverse consequences were identified and noted.

Figure 1.

Distributed Resources Analysis of State 6

Order entry by clinicians

Seven internal medicine physicians with a year or more of daily order entry experience and a range of 2-5 years of clinical experience were given a written clinical scenario and instructed to proceed with entering appropriate medical orders while verbalizing their thoughts (a think-aloud protocol). The screen video signal was captured and recorded on a videotape so that mouse movements, actions and screen transitions could be analyzed. The subjects were also videotaped as they performed the task. Each session took about 30 minutes. Subjects’ verbalizations were transcribed and coded for a cognitive task analysis ⁷.

RESULTS

System walkthrough

The GUI provides numerous affordances (e.g., buttons and actionable objects on the display), but the configuration of resources (for example, the facilitation of successive steps) is less than optimal for achieving goals without a significant cognitive effort. This is illustrated in the context of the analysis of a system state (screen configuration of affordances and resources) described in Figure 1 where users select a subset of applicable orders from an admission order set.

In our notation, the STATE serves as a label for the current configuration of internal and external resources. Each screen transition constitutes a state change. The GOAL is formed by the user based on the current state and his or her conceptual model of the system. Here, the user needs to select a subset of orders appropriate for the clinical scenario from a default set of 36 orders. ART is an available artifact, that may be a drug dosing manual in paper or electronic version, or a list of notes. In this case it is the written scenario containing patient data. AFF signifies available system affordances and suggests possible next actions. Although these are external representations, the complexity of the screen (40 buttons with textual labels) precludes the possibility for quick perceptual judgments for “less-than-expert” users. MED and SPEC are internal (I) and external (E) instances of biomedical knowledge, as described earlier. The written scenario constituted an external reference resource of patient findings and data in this state, and decisions about the inclusion of orders in the subset were mostly supported by general and patient-specific knowledge of the physician. CSK is a conceptual system knowledge resource. There are about as many instances of conceptual system knowledge active during this state as there are instances of biomedical knowledge. The user’s attention needs to be divided between treatment planning and managing system operations (e.g., searching for the next order). HSI and HSE are internalized and externalized history resources.

Although selected orders are clearly marked, there are a total of 36 orders presented to the user, each containing some 14 textual items in 3 lines of text. The user needs to scroll through three screens to browse all available orders, without the possibility of a single view of the selected subset. PLI is an internalized action plan that refers to the sequence of actions that the user will need to execute to accomplish the goal and advance to the next state. The next step of activating the selected orders needs to be recalled from memory.

This frame-based template was used to describe each state, with additional abstract information structures used as necessary. For example, AEI or internal action-effect relations are predicated on the user’s conceptual model of the system and specific knowledge of action consequences (i.e, activating orders by clicking a button). AEE, their external representations such as explicit labels or entries in manuals afford the user an additional and explicit semantic mapping of action to consequence.

To complete the task with optimal efficiency and accuracy, a user needs to navigate through 12 system states. Many of these states make considerable demands on users’ internal resources, in particular on conceptual models of the system. Summary results of the analysis are presented in Table 1. Internal and external resources are subcategorized as patient and system centered, depending on which aspect of the ordering task they support. There is more than twice the number of internal resources (44 to 17) required for system operation then there are for patient-centered clinical reasoning. This unfavorable ratio indicates that users must direct attention away from the clinical task. A similarly adverse 2 to 1 ratio (61 to 27) characterizes the internal/external distribution of all available resources. A well-designed system minimizes the cognitive overhead of users by providing more resources as reflected in the external representation in the interface. The relative distribution of resources in this system places heavy cognitive demand on users and renders the order entry task as difficult, especially in the absence of a robust conceptual model. From this analysis we can infer that the system will require an especially steep learning curve and may increase the likelihood of user errors.

Table 1.

Number of Resources Active During an Ordering Task

		Internal							External
		Patient			System			Patient	System
GOAL	States	Spec	Med	Inst	CSK	HSI	AEI	PLI	Spec	HSE	AEE
Open chart	1				1		1	2	2		1
Select set	4	2	1	1	5		3	4		1	1
Select subset	1	4	2		4	1		2	1	7
Change defaults	4	2	2	1	4	2	1	2	2	7
Add order	1		1		6		1	2	1	1
Review and sign	1	1				1		2	1	1	1
Total	12	9	6	2	20	4	6	14	7	17	3
Total Patient / System		17	44	7	20
Total Internal / External		61	27

Resources: Patient– Patient-centered reasoning support, System- System-centered reasoning support

Knowledge: Spec– Patient-specific, Med– General medical, Inst– Institution-specific

Order entry by clinicians

No subject produced a flawless set of orders as compared to a reference model. The entries were coded as correct, partially correct, incorrect and omitted, as shown in Figure 2. Errors of omissions were made by five subjects, ranging from one to three items missed out of the possible nine that required entries. Five subjects entered several incorrect entries. These errors may have resulted in delays or extra requests for clarification by the order recipient. Two subjects recorded erroneous allergy information with potentially serious medical consequences (“NKDA” instead of the documented iodine sensitivity). This error seemed to be the result of an oversight and probably not attributable to interface complexity. The number of both types of errors (omission and commission) per subject ranged from one to five.

Figure 2.

Accuracy and completeness of orders

The system walkthrough identified particular states in which a given configuration of resources were likely to pose problems for the users. This was evidenced by users’ actions and error patterns. For example, a subject mistakenly selected a urology post-operative transfer order set. He subsequently needed to recognize and eliminate inapplicable orders and to reconstruct the admission set by entering individual orders. This was a time-consuming and laborious process. This error was precipitated by a lack of clarity in the presentation of order sets in the pick list. The clinician needed to rely on specific conceptual system knowledge to successfully navigate the hierarchical menu of order sets. In addition, the system does not afford easy backtracking or error recovery. The walkthrough analysis of this particular state enabled us to explain why inappropriate selections may eventuate and that error recovery would be difficult given the limited external navigation resources.

CONCLUSION

Provider order entry is an inherently complex process, but the configuration of system resources can either exacerbate or minimize its complexity. This research was predicated on a two-pronged approach to the study of human computer interaction. The first component involved a distributed resources task analysis carried out by the team of investigators. The second involved usability testing of clinicians entering clinical orders into the system. The distributed resource analysis enabled us to account for patterns of user behavior. In turn, usability testing allowed us to refine our intuitions about the ways in which configurations of resources can facilitate order-entry tasks.

This research was guided by the belief that cognition is best construed as a distributed process that stretches across humans and artifacts. Well-designed technologies reduce the need for users to remember large amounts of information and appropriate external representations can minimize the difficulty of a task by supporting recognition-based memory. In our view, the resource model is a valuable tool for the study of complex medical information technologies. A distributed resource analysis could inform design decisions by making task demands more transparent and providing guidance for externalizing resources that alleviate the working memory burden. Towards that end, a designer may examine the ratio between external and internal resources and also determine how to reallocate user and system resources. Although the application of this model to explain user performance is still at an early stage, it was useful in accounting for certain patterns of errors and interactive strategies. The redistribution and reconfiguration of resources may suggest guiding principles and design solutions in the development of complex interactive systems.