Abstract
We discuss a novel methodological approach to the analysis of a medication order entry system prior to system release. The approach involved use of realistic scenarios (where physicians and nurses interacted with a system and dummy patient) where the sessions were video recorded in their entirety. The data were analyzed using a qualitative coding scheme for identifying usability problems and changes in workflow.
Methodology
The approach initially involved creation of a set of realistic scenarios of use of a medication order entry system in a large hospital setting. The system under study was a new medication order entry system that was integrated with patient and medication bar coding technology. The materials used in the study included a patient mannequin and an assortment of a realistic hospital room artifacts, including intravenous bags and other realistic props. Sixteen subjects (5 nurses and 10 doctors) participated in the study. Each subject worked through the scenarios of giving patients (using the mannequin) IV injections while interacting with the new medication order entry system. All subject interactions were video recorded in their entirety, including full recordings of all computer screens and activities taking place within the room and all interactions with the dummy patient (e.g. the subject scanning the patient id from the mannequin’s wrist band) and IV bags. A low cost approach to use of simulations was taken (using low cost portable recording equipment consisting of the screen recording software Hypercam and inexpensive portable digital cameras for recording of behavioral activities from multiple views). In addition, all subjects participated in a brief interview at the end of each session to collect further data on their impressions of the system. The analysis of the data involved use of software coding program to inter-relate and correlate video sequences from all views with audio recordings of interactions. The corresponding audio portion of the sessions were coded by two researchers in order to: (a) identify potential usability problems, and (b) to characterize workflow using the new system.
Evaluation Results
The approach was found to be useful for identifying changes in workflow that would result from implementation of the system and in predicting potential user problems and errors prior to release of the system. For example, from observation and interview with subjects, prior to use of the new system, physicians and nurses were able to conduct certain activities in parallel (e.g. hanging of IV bags). Analysis of the video data revealed that the process involving use of the computer lead to a specific serial ordering of activities, where for example, all activities associated with hanging one bag must be completed prior to moving to the next task. Implications for improvement of workflow and error prevention in health informatics are discussed. For example, it was found that the approach could identify situations where physicians and nurses encountered inefficiencies while using the system, leading to potential refinements prior to release of the system, ranging from software issues to issues related to the ergonomic arrangement of equipment within the hospital room.
Conclusions
We have found that a low-cost and portable approach to the use of simulations and detailed video analysis of realistic healthcare situations (involving use of new technology) can be successfully and economically applied to identify potential changes in workflow and usability problems prior to system release. Furthermore, results from such simulation can guide and provide focus for collection of data in real-world settings (we currently undertaking). It is argued that such an approach has considerable potential for areas (such as the study of user needs and the study of technology-induced error) where it may be difficult to obtain useful data under uncontrolled conditions. Furthermore, the real benefits of use of such approaches may come from detailed analysis of error and changes in workflow observed during simulations conducted prior to widespread dissemination of new healthcare information systems.