Applying Direct Observation to Model Workflow and Assess Adoption

Abstract

Lack of understanding about workflow can impair health IT system adoption. Observational techniques can provide valuable information about clinical workflow. A pilot study using direct observation was conducted in an outpatient chronic disease clinic. The goals of the study were to assess workflow and information flow and to develop a general model of workflow and information behavior. Over 55 hours of direct observation showed that the pilot site utilized many of the features of the informatics systems available to them, but also employed multiple non-electronic artifacts and workarounds. Gaps existed between clinic workflow and informatics tool workflow, as well as between institutional expectations of informatics tool use and actual use. Concurrent use of both paper-based and electronic systems resulted in duplication of effort and inefficiencies. A relatively short period of direct observation revealed important information about workflow and informatics tool adoption.

INTRODUCTION

Informatics tools have tremendous potential for improving patient care, but can face problems in acceptance and adoption^1,2. Poor knowledge of workflow can undermine adoption of efficacious systems^3,4. Adoption is defined as the extent to which people utilize informatics tools. Workflow describes the processes people follow to complete tasks including the order in which tasks are completed. Workflow also incorporates how people interact with other people, processes, and technology to accomplish tasks. Informatics tools that are consonant with and support user workflow may be more readily adopted. Unfortunately, individuals may have difficulty providing a complete description of their workflow because they are immersed in it. Ethnographic techniques such as direct observation can elicit details of workflow that other methods would miss⁵. Previous studies have shown that direct observation can provide formative data for understanding workflow and user needs, as well as improving user satisfaction^6–11.

A pilot study was conducted using direct observation in preparation for developing informatics tools to assist in chronic disease management. The goals of the study were to rigorously delineate workflow in one chronic disease clinic, understand the adoption of existing informatics tools, and develop generalizable models of workflow and information behavior for chronic disease care.

METHODS

This pilot study was initiated in a single subspecialty disease management clinic at Vanderbilt University Medical Center (VUMC). Fifteen chronic disease clinics were evaluated for inclusion. The VUMC Multiple Sclerosis (MS) Clinic was chosen as the pilot site due to its modest staff size and patient load, espoused use of existing informatics tools, interest in participating, and interest in developing tools for chronic disease management. The MS clinic, located in the adjacent rehabilitation hospital (physically separated from all other ambulatory clinics) provides chronic disease management care to approximately 2500 patients. The clinic is staffed by two physicians, one full-time registered nurse, one clinical receptionist, and several part-time nurses and administrative personnel.

Patients come to the clinic for routine care at ~6 month intervals and more frequently when disease exacerbations occur. Patients are treated with multiple medications both to suppress disease progression and to mollify an array of symptoms. Tests are routinely performed to track MS progression (e.g. MRI scans) and to ensure that chronic medications are not causing organ damage. Because the clinic attracts patients from a wide geographic area, many of these tests are performed at outside facilities. The clinic represented itself at the start of the study as “fully electronic”. The clinic has been using Vanderbilt’s outpatient electronic medical record (EMR) system, StarPanel, for approximately 3 years. StarPanel enables users to capture vital signs, medication lists, problem lists, and clinic notes at the point of care. StarPanel also has a message basket function that allows staff members to communicate electronically. Message baskets can be shared between users based on roles, such as a message basket for all nurses in a single clinic. In addition, the clinic utilizes software for scheduling and billing and PACS software for viewing MRI scans.

A direct observation technique^12–13 was utilized to understand workflow in the clinic and assess how thoroughly StarPanel had been adopted. After Vanderbilt Institutional Review Board approval, a graduate student in informatics (KMU) observed in staff work areas and exam rooms. As well as recent coursework and directed study on observation techniques, she had previously conducted contextual inquiries in the medical device industry. The student also read extensively about MS disease management prior to starting the observations.

Before each observation, the project was explained to staff members as well as to individual patients and verbal assent was obtained. Observations focused on the interactions between people, processes, and technology and on information flow in the clinic. Throughout each observation, the observer recorded detailed notes using a Logitech io2 digital writing system. The system uses an ink pen with an optical sensor to capture information written on patterned paper for later upload to a computer.

Data were organized in an electronic notebook according to date of observation and regularly reviewed to extract conceptual constructs and general themes. A set of flowcharts was developed demonstrating a systems engineering perspective¹⁴ to task and information flow in the clinic. Iterative construction of the flowcharts facilitated identification of gaps in knowledge about the clinic’s processes and guided subsequent observation.

Early observations focused on gaining a detailed understanding of care processes and the domain. Later observations were used to build and refine the model as well as to evaluate evolving themes. Data from early observations were reviewed at later stages of the project to ensure accuracy and internal validity.

RESULTS

Research themes

Based on over 55 hours of observation, five research themes were developed. The five themes were:

• Information from disparate sources needs to be collected, organized, and understood;

• Effective communication between physicians, nurses, patients, and administrative staff is critical to chronic disease care;

• Difficulty in managing information causes problems throughout the care delivery process;

• Patients are key partners in the collection and use of information in chronic disease clinics; and

• Technology should be a partner in chronic disease care, but currently it is not.

Task and information flowcharts

The task and information flow in the clinic was organized by segmenting the overall process into subprocesses. Subprocesses included: pre-appointment activities, check-in, RN intake, MD workup, testing, treatment, patient counseling, making a follow-up appointment, and post-appointment activities. Each subprocess was divided into specific tasks. The handoffs between tasks and between people were shown in detail. The flowcharts graphically present the clinic’s workflow. The overview flowchart is shown in Figure 1.

Figure 1.

Overview of workflow in an MS clinic

Gathering and using information

Information originated from both inside and outside VUMC. The types of information included: laboratory results, MRI scans, medical records, and communications. Lab results arrived in the clinic electronically through StarPanel, by fax machine, by email, and also through hard copy either mailed by the testing location or brought in by the patient. Medical records included both StarPanel and paper “shadow” charts maintained by and exclusive to the clinic. Patients who were referred to the clinic for diagnosis typically brought a hard copy of their medical records from other healthcare providers.

This information was used for a variety of tasks. For new patients, the most common use of information was to make the diagnosis of MS. Information was also used to predict overall prognosis, which was determined by the type of MS, history of exacerbations and disease progression, and literature evidence. For returning patients, information was used primarily to manage their disease. Disease management included impeding disease progression, addressing specific symptoms, and helping with lifestyle issues such as depression and coping strategies. Therapy changes, if needed, were based on disease-specific needs as well as literature and previous therapies.

Communication

Communication modalities were diverse and included conversations in exam rooms and work areas, phone calls, faxes, emails, postal mail, and StarPanel message basket communications. Message baskets were used to communicate within the clinic and between the clinic and other areas within VUMC, such as with the main Neurology Clinic.

Nursing staff also spent appreciable amounts of time using the telephone to discuss medication refill requests, disease progression, symptom management, and requests for assistance with insurance-related issues. Nurses frequently had to call insurers to determine what paperwork was needed and to assure prior authorization. During the study period, changes in Tennessee’s Medicaid program, TennCare, and the implementation of Medicare Part D prescription drug coverage increased the communication required among nurses, patients, and insurers.

Communication gaps occurred occasionally within the clinic. During the handoff of the patient from the nurse to the physician, little communication took place. Some information that patients shared with the nurse was not communicated to the physician because the nurses recorded data in the EMR and the physicians rarely consulted the EMR. The shadow chart sometimes, but not always, mitigated this gap.

Use of informatics tools

The MS clinic made use of many of the functions of the informatics tools available to them. The nursing staff entered the patient vitals of every patient into StarPanel and also updated each patient’s problem list and medication list. The physicians used the telephone to dictate their notes and these were transcribed into the EMR. The results of laboratory and imaging tests performed at VUMC were available in StarPanel and MRI scans could be viewed with PACS. Electronic systems were used for scheduling patient appointments. StarPanel’s message basket feature was used extensively by clinic staff. The clinic’s nursing staff shared a role-based message basket to ensure continuity of care. Many hard copy documents were scanned as PDF documents into StarPanel by an external service provider.

Use of non-electronic artifacts

Although the clinic utilized many electronic data management tools, it also used multiple non-electronic artifacts. These artifacts included: paper charts, paper forms, records and test results provided by patients, and tools for tracking patient flow through the clinic.

The clinic maintained a paper chart for every patient. Each day before clinic started, an administrative assistant gathered the paper records needed for that day. Physicians consulted the paper charts, rather than the EMR, to review previous patient exams and to assess disease progression. After the clinician dictated his or her notes summarizing the patient visit, the paper chart was filed.

Multiple paper forms were used within the clinic. New patients used paper forms to provide information on past medical history, social history, and family medical history. Both new and existing patients used paper forms to provide information on pain, risks to safety, current medical history, current medications, changes in medical or social status, and a review of systems. The nurse copied data provided by patients from the paper forms into the EMR. The volume of data and the free text structure of the EMR made this a tedious and time-consuming process.

Physicians used a paper form to document their neurological examination. The form included vital signs, mental status, cranial nerve function, reflexes and coordination, motor exam and sensory exam. A homunculus and several graphical diagrams allowed the physician to quickly record highly specific clinical data. Other paper forms were used in the clinic for a variety of purposes. Few of the forms scanned into StarPanel were ever reviewed again.

New patients usually brought paper copies of their medical records and test results. The type and scope of these records varied widely depending on the patient. Returning patients provided copies of results from tests performed outside Vanderbilt.

Patient flow through the clinic was managed with the aid of non-electronic artifacts. A list of patients for each day’s clinic was printed out and posted on the wall of the staff work area. In addition, a whiteboard mounted in the hallway listed each exam room, the initials of the patient in that room, and the visit stage. The whiteboard was updated sporadically and did not include workflow information such as the next patient who should be seen. Physicians frequently needed to ask if a patient was ready and on occasion patients were seen out of order.

User expectations versus system performance

Gaps existed between the performance of the informatics tools and the expectations of the users. At one point, a form within StarPanel used to collect data about patients’ pain level and risk of domestic violence changed without advance warning. The paper form used in the clinic to actually collect these data was not updated until later, resulting in staff confusion and workflow disruption.

There were problems with a message basket shared by the clinic’s nursing staff. Two nurses worked on the same message in the shared basket without realizing the duplicated effort. Nurses noted that some messages were being removed from the basket by administrative staff in another clinic and sent to individual nurses, defeating the purpose of the shared basket. Nurses commented frequently about the high volume of messages in this basket and how difficult it was to actually respond to them all each day.

Physicians had difficulty reviewing MRI scans done outside VUMC. If patients brought hard copy films of the MRI, the process worked well. If patients brought a CD containing the images, the clinic was frequently unable to view the images. Proprietary viewers for myriad different MRI image formats could not be loaded on the clinical workstations in the examination rooms. The clinic only had one administrative workstation that could handle these applications but clinic staff was not aware that this software existed or could be installed on it.

DISCUSSION

This study identifies aspects of workflow that are complicated by, rather than complemented by, technology. This clinic adapted to meet an institutional mandate for a paperless office even though the provided technology did not fully support users’ workflow. The results suggest that if clinical informatics tools are not designed with a detailed understanding of workflow, then adoption may be impaired. The study also demonstrates the value of direct observation to fully appreciate clinical workflow and the role of health information technology to either enhance or impair it. Finally, the study identified numerous opportunities to enhance processes for chronic disease management.

Workflow and adoption

Gaps existed between the workflow of the clinic and the workflow demanded by the informatics tool. In some cases, the informatics tools actually made it more difficult to complete tasks. Attributes of IT design can force changes in workflow that decrease efficiency and increase the potential for use errors.

StarPanel does not currently have a provision to allow patients to input data such as their current medical status or medications. The lack of a patient-accessible electronic intake form necessitates patients having to fill out lengthy and difficult-to-read paper forms. Many MS patients have cognitive impairment as well as degraded handwriting reducing both the reliability and the legibility of these critical data. Nursing staff must then copy relevant information from completed forms into the EMR. This inefficient process may increase the risk of missing key or subtle patient-provided data. While the neurological exam form could be computerized, its use during the physician-patient interaction makes both its design and physician acceptance a challenge.

Disconnect between design and use

The observed effects on clinic staff of the sudden unexpected change in one of the EMR forms exemplified the apparent disconnect between users’ expectations of system performance, actual system use and performance, and the designers’ expectations of system use. Another example of this disconnect was seen in the IT support for the clinic. Solutions to some of the issues encountered in the clinic, such as programs to read different MRI formats, have been implemented elsewhere at VUMC. However, clinic staff was unaware that solutions were available and the clinic’s IT staff was unaware that the problem existed. Although the message basket feature was highly valued by staff, it was also a source of frustration and lead to subtle or unexpected sources of miscommunication.

The informatics tool did not support the clinicians’ need to integrate disparate clinical information to formulate a complete picture of patient status in the past, present, and future. Using the paper chart, physicians could quickly flip through previous neurological examination forms and see how the patient’s status had changed over time. The presentation of the same data in the EMR appeared to be suboptimal for supporting this essential clinical need. A variety of informatics solutions including better data entry modalities, improved methods of tagging and organizing scanned documents, and better data visualization paradigms could significantly reduce the need for a paper chart.

Observation reveals gaps

Gaps were identified between how the institution expected StarPanel would be used and how end users in the MS clinic were actually using it. StarPanel was designed to be a care management tool and it was anticipated that paper charts would no longer be necessary once it was deployed¹⁵. Yet in this clinic, StarPanel is being used primarily as a data repository and shadow charts are still an integral part of the workflow. The presence of the informatics tool has increased workload, as effort is duplicated to maintain both types of records. Interestingly, in preliminary observations in the VUMC Cystic Fibrosis Clinic, it appeared that StarPanel was better integrated into clinical workflow and there was less reliance on paper artifacts.

In initial conversations, the MS clinic staff asserted that they were fully electronic. Direct observation showed that “fully electronic” may mean different things to different people and does not necessarily correlate with “fully functional”. To support the institutional mandate, the clinic scanned paper documents into the EMR while maintaining their shadow paper charts. The adoption of StarPanel into the workflow of the MS clinic is very shallow.

Implications for design

This study raises several implications for future informatics design. Failing to understand the workflow and information behavior of users before developing a tool can have negative consequences. First, users may not use the tool as developers expect. Second, the tool may cause duplication of effort, inefficiency, workarounds, and errors thus impairing adoption and user satisfaction. A tool’s full potential will not be attained if user needs are not met.

This study shows that even if users say they are using an informatics tool, this does not tell the whole story. Users may believe that they are using the tools as intended, when in fact they are not. Direct observation can provide data that would not be revealed through other methods such as data audits or use surveys. Thus, this study confirms the value of direct field observation in informatics design and demonstrates that even a relatively short amount of time spent observing in a use environment can yield invaluable data on information behavior and workflow.

Future directions

Based on the success of the pilot study, the investigators intend to extend the study into other chronic disease clinics. Complementary methods including subject interviews and targeted surveys will also be used to formulate a generalized model of information behavior and workflow in chronic disease care. Using this generalized model, methods for developing informatics tools can then be improved by accounting for existing information behavior and workflow.