Use of Simulation Approaches in the Study of Clinician Workflow

Abstract

This paper describes a methodological framework for conducting evaluations of clinical workflow and systems impact based on simulated user interactions. The approach involves the collection of a rich set of data consisting of audio and video recordings of interactions between healthcare workers and health care information systems and their associated devices. Methodological considerations and issues in conducting such studies are discussed. The steps involved in carrying out such studies are also described along with a description of our current work in the application of the approach to the analysis of clinician workflow.

Introduction

Methods based on simulations have been used in biomedical informatics to study aspects of human computer interaction in a number of health research domains including: human factors, usability, doctor patient interactions involving technology, health professional information needs, health professional decision-making, new device testing and medical error [1–5]. In this paper we describe a new approach towards evaluating the effects of health information systems upon clinician workflow in the performance of both routine and complex clinical tasks in a laboratory environment. The approach involves video and audio taping clinicians thinking aloud while observing their interactions with health information systems and their associated devices while undertaking representative routine and atypical tasks.

Background

Health information systems can have a significant impact upon clinician work. The impact of health information systems upon clinician work has been both positive (e.g. reducing medical error, length of stay) and negative (e.g. altering traditional workflow patterns that promote clinician communication and coordination of work). This has led a number of biomedical informaticians to theorize about the socio-technical implications (i.e. task-technology fit) between health information systems and real world clinical environments [6]. More specifically, some researchers have suggested many of difficulties experienced by biomedical informaticians when implementing health information systems arise from a poor fit between clinical tasks and technology, leading to unintended clinical practice [7] and workflow effects [6]. Over the past decade considerable researcher effort and energy has been directed towards documenting the unintended effects of technology upon clinician workflow using observational approaches in real life settings well after a health information system has been deployed. More recently, biomedical informatics researchers have been advocating the use of simulations as a methodology for assessing the potential impact of health information systems upon clinician work before a health information system is deployed in order to prevent undue aggravation and disruption to clinical workflow [5–7]. Simulations allow clinicians and informaticians to determine the impact of a system and improve workflow changes before the system is deployed in a real world context.

Simulations have been used in a number of research domains to study varying aspects of human computer interaction with significant success. There are a number of differing types of simulations, including computer based simulations that attempt to model human behaviour [2], and simulations that are developed to test specific system components [5]. In this paper we demonstrate the potential value of using simulations to document and evaluate the impact of health information systems upon clinician workflow before real world implementation is undertaken. Specifically, we explore the value of using simulations that involve real users interacting with information systems and devices in simulated clinical environments as they perform routine and atypical tasks to determine the systems impact on clinical workflow. Historically, such simulations have been used to effectively develop and pilot test systems across the System Development Life Cycle (SDLC) from user needs assessment through to system design. Our work extends this research, to examining the impact of health information systems upon workflow as it allows software and hardware developers to document, develop, refine and improve clinician workflow and thereby ensure a more seamless fit between task and technology during systems implementation.

Using Simulations to Document, Develop and Refine Health Information System Workflow

The development of representative, ecologically valid simulations that allow for the study of workflow to enhance task-technology fit requires careful attention to factors that may influence internal and external validity and thereby the generalizability, applicability and value of findings. In order to ensure internal validity there is a need to control extraneous variables and their impact on the quality of the data that can be obtained from the simulation. External validity must also be considered, including attention to subject, setting, case, task, and scenario representativeness. External validity ensures the “research has generality, is representative of the so-called real world, and does not distort the question under investigation”[9]. External validity requires that real behaviours are studied in realistic situations.

Methodological Approach

In this section of the paper we describe a step-by-step approach to developing workflow simulations. Although there may be some variation in the method, the development of simulations to evaluate health information system impact on clinical workflow involves consideration of each of these steps to ensure the generalizability, applicability and value of the findings in informing health information systems deployment and implementation (see Figure 1 for an overview of the steps).

Figure 1.

Overview of Simulation Approach to Health Information System Workflow Study

Step 1: Subject Selection

This step involves identifying and selecting representative subjects for the simulation. Subjects should be representative of health information system end users in terms of their level of disciplinary, domain and technology expertise [9,10].

Step 2: Task Selection

This stage involves the selection of representative tasks (i.e. routine and atypical) that end users are expected to undertake when using a specific type health information system. A range of representative tasks can be selected [8]. For example, in a study of a medication administration system, this may include routine tasks such as administering a once a day oral medication to a patient to more atypical tasks such as mixing an IV medication and then administering it via IV pump.

Step 3: Scenario Design

Scenarios can involve simple written descriptions of a task that are given to subjects to read and perform when using a health information system or they can be as complex as scenarios involving actors playing patients or computer controlled mannequins (e.g. programmed with a range of physiologic responses intended to emulate human behaviour) following a script [13].

Step 4: Designing the Attributes of a Scenario

Attention should be paid to the attributes or qualitative dimensions of each scenario. Researchers should consider varying levels of scenario complexity, task urgency and the role of time constraints in scenario design as each of these factors may influence clinician workflow [13]. Scenarios should also be representative of the range of clinical tasks (i.e. routine and atypical) that would be performed by clinicians to ensure all possible workflows and their variations are adequately represented and studied. For example, when studying the impact of an online documentation system upon clinician information needs at the bedside, a researcher might use scenarios that stimulate a clinicians’ desire for additional information (e.g. a clinician has encountered a symptom of a disease that is unfamiliar to them in their day to day work) at the bedside. The impact of the clinical environment upon workflow associated with health information systems should also be considered when developing scenarios. For example, if the researcher wishes to understand the impact of the clinical environment upon clinician workflow there may be a need to vary the demands of the environment – clinicians should be observed attempting to acquire information from a health information system given environmental constraints in terms of available time to perform a task and the sense of urgency associated with managing life threatening situations.

Step 5: Equipment and Recording Methods

The complexity of equipment required to conduct simulations varies from low fidelity to high fidelity. A low fidelity simulation roughly approximates the real world. For example, a simple study may involve presenting physicians with a short written case description of a patient and asking the physician to enter information about the patient into the patient record system while recording the interaction with simple video or audio devices. A high fidelity simulation would more closely reproduce the real world. For example, a simulation may involve actors playing the role of patient and staff in a clinic in a study of how doctors use patient record systems in clinics. Such a study may involve multiple recording devices to precisely document all subject interactions (e.g. audio and video recordings of all verbalizations, computer activities and the examining room to document actions).

Step 6: Data Collection

As described in step 5, varying types of data can be collected. We recommend that audio data be collected in all cases. Audio data provides information about what is being focused on and considered by the subjects during simulations [15]. Other forms of data collection include video and computer screen recordings of human computer interaction. Video data and computer screen recordings can provide additional insights when triangulated with audio data. Increasingly, the role of computer screen recordings and video data has been found to both inform and contextualize collected audio data – helping the researcher to better understand the underlying cognitive processes and the effects of computerization upon them. For example, in one of our recent studies examining the relationship between medical error and system usability, video and computer screen data informed audio transcripts of subjects interacting with a system resulting in the emergence of new findings. In this study audio data indicated subjects believed they had entered the correct prescription when using an electronic prescribing program, while corresponding video data and computer screen recordings revealed usability issues led subjects to unknowingly enter incorrect prescriptions [5].

Step 7: Qualitative Analysis: Coding Schemes

Fundamentally, all coding schemes should be theoretically motivated [11,14]. Theory drives qualitative coding of audio, video and computer screen recording data. Data is coded using an inductive, deductive or mixed method approach [14,15]. In deductive coding, existing workflow theory/models are used to code data. In inductive coding, research findings lead to the development of workflow theories/models [15]. Mixed method approaches involve the use of both inductive and deductive approaches. There are a number of benefits associated with using mixed method approaches: (1) empirically validated theories/models from previous workflow research are used and (2) in cases where existing theory/models cannot explain a phenomena theory/models can be extended or falsified [15].

Step 8: Quantitative Data Analysis

Qualitative data (i.e. coded verbal transcripts, coded observations from video data or recorded computer screen information) can be converted into quantitative data or quanticized [16,17]. Coded verbal data can be reduced to concepts or representations “intended to mean one thing” ([16], p. 253). For example, frequencies can be tabulated for each aspect of workflow that has been coded, and as a consequence inferential statistics can be undertaken to make comparisons between health information system workflows [16].

Step 9: From Laboratory to Real World Health Information Systems Design

In step nine we use our findings to redesign or modify workflows to improve task-technology fit to meet the demands of end users and the clinical environment. This is done before institutional deployment to ensure the “right” workflow is present.

Use of Simulation Early in the Development Cycle: Assessing Workflow

As described above the methodological approach that we have refined and can be used to study the effect of health information systems upon workflow (e.g. physician use of electronic prescription writing systems while undertaking patient care rounds). Simulations can be used to study a wide range of health information systems [18]. In addition, we have been recently working on extending the simulation approach to the assessment of user workflow as the basis for the development of new systems. This can involve observing clinicians (e.g. physicians or nurses) interacting with the paper record under simulated conditions to assess what parts of the record are relevant to assessment, planning and decision-making tasks in real world environments. Such an approach allows one to study workflow without influencing patient care prior to system implementation. Simulations can be conducted early in the SDLC during system development to inform the design to ensure task-technology fit. In such studies, subjects are asked to carry out routine and atypical clinical tasks. An analysis of the type and frequency of specific workflows can be used to guide selection, design and implementation of health information systems.

Discussion

In this paper we have described our work in the development, refinement and application of an approach to evaluating the impact of health information systems upon clinician workflow based on use of simulated user interactions involving routine and atypical clinical tasks. The approach builds on previous work in the area of simulation [3] and leads to collection of a rich set of qualitative and quantitative data. In addition, as described, the approach can be used throughout the System Development Life Cycle, from requirements gathering (as the basis for system design) to assessment of the impact of information systems upon workflow. In particular, simulations can form the basis for subsequent evaluation of system use in clinical settings. Results from such simulations can guide and provide focus for health information systems development and implementation in real-world settings. It is argued that such an approach has considerable potential for areas where it may be difficult to obtain useful data under uncontrolled conditions (e.g. workflows involving information systems during life threatening events for patients).