Abstract
Vital sign documentation is an essential part of perioperative workflow. Health information technology can introduce complexity into all facets of documentation and burden clinicians with high cognitive load3-4. The Mayo Clinic enterprise is in the process of documenting current EHR-mediated workflow prior to a system-wide EHR conversion. We compared and evaluated three different vital sign documentation interfaces in pre-operative nursing assessments at three different Mayo Clinic sites. The interfaces differed in their modes of interaction, organization of patient information and cognitive support. Analyses revealed that accessing displays and the organization of interface elements are often unintuitive and inefficient, creating unnecessary complexities when interacting with the system. These differences surface through interface workflow models and interactive behavior measures for accessing, logging and reviewing patient information. Different designs differentially mediate task performance, which can ultimately mitigate errors for complex cognitive tasks, risking patient safety. Identifying barriers to interface usability and bottlenecks in EHR-mediated workflow can lead to system redesigns that minimize cognitive load while improving patient safety and efficiency.
Introduction
More than 200 million surgeries are performed worldwide every year1. Surgery is a high acuity setting with an immensely complex workflow that need to be better understood and modeled to identify potential barriers to the quality of patient care. Surgical adverse events are second only to the incidence of adverse drug events2. Electronic health records (EHRs) are essential for informing, documenting and coordinating the surgical care process, but have not realized their potential as a tool that facilitates coordination of care and patient safety3,4. EHRs are known to place a high cognitive burden on clinicians, impede clinical workflow and hinder task performance3. Following Meeks et al.5, we conceptualize EHRs as adaptive systems where safety risks often surface from user interactions leading to variation in the workflow. Variations in clinical workflow can be seen in the pre-operative (PreOp) nursing assessment, the processes occurring before a patient is transferred to the operating room, for particular tasks across individuals, interfaces and clinical sites. This variation can often lead to suboptimal, overly complex or inefficient procedures that affect workflow, which in turn influences task completion and error mitigation. This paper focuses on EHR-mediated workflow in the context of the vital signs documentation task in PreOp nursing assessment.
Research surrounding vital signs documentation workflow has received comparatively little attention making it difficult to understand the mediating effects of technologies6. In the PreOp nursing assessment, one of the first tasks performed is the collecting and charting of patient vital signs. Vital signs can be used to gauge a patient’s hemodynamic stability and provide a point of reference for comparison during surgery. Stevenson et al. 7 found that the lack of adequate facilities and functions for documenting vital signs in an EHR made it difficult for physicians and nurses to enter, locate and retrieve the data. This was found to have an adverse impact on patient safety. Yeung and colleagues6 found that vital signs documentation process was highly variable within groups and between settings despite the fact that clinical assessment of vital signs was consistent.
Because of their importance, it is critical that point-of-care vital signs documentation is understood at fine level of granularity. Towards this end, we draw on a cognitive engineering (CE) methodological framework9. It is an interdisciplinary approach to the development of principles, methods, and tools to assess and guide the design of systems to support human performance. CE highlights the discrepancy between user’s goals and the physical controls embodied in a system. We draw on that framework in employing video-analytic capture methods, think-aloud protocols, and quantification of interactive behavior to capture EHR-mediated workflow. CE has been widely used to assess the task-related performance of EHR users10,11. Design and development of surgery specific information systems often raise a variety of challenges for designers7. When designing interfaces, there are many aspects of user interactions to consider optimizing to relieve cognitive burden on clinicians while maximizing efficiency, especially in high-risk, high-cost settings such as surgery.
EHRs are often perceived by providers as difficult to use. Usability analysts have cited issues with difficult-to-read interfaces, confusing displays and iconography that lacks consistency and intuitive meaning12. These usability issues can often lead to heavy cognitive load. Even small differences in an interface can have significant impact on efficiency and effectiveness in accessing and documenting patient information. Horsky et al.13 found that clinicians made three times as many errors in EHRs with single column medication lists, as compared to using side-by-side medication lists. Plaisant et al.14 similarly discovered that having access to needed information on a single screen proved to be a considerable advantage in EHR workflow. EHR interfaces differ considerably in terms of interaction design, resulting in marked differences in the process of task completion15.
The Mayo Clinic ROOT (Registry of Operations and Tasks) project aims to identify current workflows at all of their clinical sites16 in advance of a large-scale EHR conversion with a main goal being to understand and consolidate currently siloed applications into a standardized workflow and a uniform system-wide EHR. This project partially aims to understand and characterize variation across both individual clinicians and clinical sites. This is accomplished through a broad range of methods including interviews, log file analysis and video ethnography to investigate EHR-mediated workflow in different clinical settings17. In a related study, Grando et al.,18 used process mining of log files in combination with observational methods to study variations in clinicians’ EHR-mediated workflow. The study compared workflows between two PreOp settings at different Mayo Clinic hospitals and found significant differences in physical organization, patient workflow, nursing roles and time on task. The approach employed emphasized workflow processes stretched across settings, clinicians and time. Duncan. et al.18 studied medication reconciliation by characterizing the variation in EHR-mediated workflow. In that study, they analyzed design differences in two different medication reconciliation interfaces through identifying violation in usability heuristics in the interfaces in the context of PreOp nursing assessment18. These interfaces differed in their modes of interaction along with the ways in which they supported user interaction15, 18. The analysis revealed unnecessarily complexities that impacted the efficiency of the reconciliation process.
The process of analyzing and understanding clinical workflow is crucial for adoption and assessment of technology in healthcare. However, the current work in vital signs and the EHR-mediated workflow in settings associated with vital sign documentation has received little attention and is not thoroughly understood6. The main goal of our research program is to leverage methodology proposed in [15-18] to document and understand EHR-mediated workflows in the context of clinical work and the different facets of interface design and how they impact clinician performance and cognition on a range of variables. The goals of this paper are to: (1) closely scrutinize aspects of vital signs charting interfaces and determine how these aspects differentially mediate task performance and (2) investigate variations in vital signs documentation across clinical sites.
Methods
We employ the methodological approach described in Duncan et al.,18 for identifying variation across clinical sites and individuals and analyzing the impact of interface elements. This includes: (1) analytic methods, as exemplified in the description of interface elements, keystroke level model (KLM) models, and workflow representations (e.g., pop-up screens, buttons, drop-down menus), (2) quantitative analysis of interactive behavior based on video observation and (3) narrative representations of complex cases that surface some of the reasons for variation.
Settings
Observations of clinicians took place at the following Mayo Clinic Hospitals: Mayo Clinic Hospital, Methodist Campus (Rochester, MN), Saint Marys Campus (Rochester, MN), Arizona Campus (Phoenix, AZ), Florida Campus (Jacksonville, FL) and Eau Claire Campus (Eau Claire, WI). In this report, results for Arizona, Florida and Eau Claire campuses are presented. Video analysis was performed for 15 different patient cases which involved 11 nurses across sites.
Data Capture
We focused on nursing assessments in preoperative (PreOp) care. We employed Morae™ 3.3 video analytic software to capture workflow. The software records the clinicians screen, provides a set of analytics (e.g., mouse clicks, web-page changes), and audio and video recording of the clinicians’ hands (e.g., use of paper documents, checklists) via a webcam. An observer was also present during the data collection to record notes and points of interest during observations. Individual recordings of clinicians performing the nursing assessment on individual patients were collected in various clinical settings and geographic locations.
Data Analysis
Videos were reviewed for integrity and gaps in time (e.g., where no activity was observed), then segmented into individual tasks. Once segmented, various quantitative and qualitative analyses were performed. The quantitative analyses focused on the analytics of interactive behavior (e.g., mouse clicks, screen transitions). Each individual task was reviewed and, for each task, mouse clicks and screen changes were recorded. Analyses involved keystroke-level analysis, process models for the vital signs documentation task, interactive behavior measures and narrative summaries. These summaries are constituted by descriptions of stretches of interaction over some period of time. The vital signs documentation task was then selected for detailed analysis. Process models were created to capture clinicians’ cognitive process of completing the vital signs documentation task for the various clinical sites.
The KLM analytic method was performed on the set of operations required to complete the tasks of accessing the vital signs documentation interface and entering the series of vital signs. In the KLM model, there are six operators: key or mouse button press (K), mouse pointing (P), moving hands to home position on the keyboard or to the mouse (H), button press (B), mentally preparing to perform an action (M), and typing a string of characters (T(n), n*K seconds)19. Each action has estimated execution: K is 0.20 seconds, P is 1.10 seconds, H is 0.40 seconds, B is 0.10 seconds, M is 1.20 seconds, and T(n) is 0.20 *n seconds18. Findings from the KLM analysis were then compared with observed user data. It has been shown to be a reliable indicator of effort in various studies19 and we have found it to approximate users’ performance as reflected in observed time measures well.
Results
We compared vital signs documentation for three different EHRs (Cerner SurgiNet, PowerChart and Surgical Information Systems (SIS)). Figures 1 and 2 each show a schematic representation an interface used for vital signs charting. Figure 1 represents Cerner SurgiNet (a dedicated surgery charting application within PowerChart). Figure 2 represents SIS (the main tool used for charting in surgical settings in the Eau Claire Campus). During the PreOp nursing assessment, the patient’s vital signs are collected prior to the surgical procedure to serve as a baseline measure and point of reference throughout the perioperative phases of care. This information is typically collected by a patient care assistant or certified nursing assistant and entered into the EHR by the nurse assigned to that patient. Additional vitals may be recorded during the patient encounter. There are a series of standard vital signs entered across all sites with minimal variation. They include height, weight, heart rate, non-invasive blood pressure, respiration rate, core temperature and oxygen saturation, and may also include manual observations that are needed. Variation across nurse and sites include vitals entered for oxygen devices as well as the documenting of patient position at the time the vitals were collected (i.e., sitting or lying).
Figure 1:
A schematic representation of the “Vital Signs Charting” interface through Cerner SurgiNet
Figure 2:
A schematic representation of the “Vital Signs Charting” interface through Surgical information Systems (SlS)
Description and schematic representation of interfaces
Although the specific vital signs collected were generally uniform across sites, the interactions with the systems differed markedly. This was reflected in both the steps required for accessing the vital documentation page and the process of entering patient information. In the Florida and Arizona hospitals, the main tool used for charting was Cerner SurgiNet with Cerner PowerChart being used as a secondary system in some cases. In Eau Claire, the primary tool used for charting was SIS. However, on one occasion, patient information was entered into PowerChart in Eau Claire for a specific patient case. In SurgiNet, information was displayed in columnar format where all information is visible on the screen. This provides more flexibility to chart in any order the nurse chooses (Figure 1). However, it does not provide any form of data-entry validity checking, for example for completeness of charting or erroneous data entry. In SIS, a user entered patient vital sign measures through a series of pop-up screens, each screen relating to a different vital sign (Figure 2). Once a measurement was entered and saved, the screen transitioned to the next vital sign pop-up screen. This approach allows for less flexibility in navigation but ensures that the necessary vital signs are charted because clinicians have to actively skip entering patient information to move to the next screen.
In SurgiNet, nurses navigated to the “IView” function of the interface that allows for charting vital signs. Once selected (indicated in red in Figure 1), a series of “quick view” selections were available that allowed for easy navigation between sections. Subsequently, when the “Vital Signs” section is selected (indicated in yellow in Figure 1), the interface transitioned to the vital signs section. The clinician must then insert the correct date and time by right-clicking on the screen and clicking on the “Add Date/Time” button, which inserted a new column with the current date and time. Once completed, the clinician can begin to chart. Each vital sign box is a free text box where data can be entered. Of note, there are several boxes that, once selected, prompt the user to make a selection from a drop-down menu. These vital signs are “Patient Location”, “Weight Scale Type”, “Temperature Mode”, and “O2 Device Type”. Clinicians tend to follow the order of entering vital signs as they are listed on the screen but can enter data in any order they choose. Once the appropriate vital signs are entered and their accuracy is confirmed, the clinician saves selections and transitions to the next section of the nursing assessment.
In SIS, there is a button labeled “CNA” and, once selected, turns into a volumetric status bar that incrementally fills (bright blue on white background) as vital signs are completed. This can be seen in red in Figure 2. When the “CNA” button is selected, a pop-up screen appears where vital signs are entered and cycled through. Each vital sign is entered into the “Comments” section of the pop-up screen and, once entered, transitions to the next vital signs screen. The section where vitals are entered can be seen in yellow in Figure 2. The method employed in SIS provides an invariant order in which vitals are entered. It confers an advantage of uniformity in data entry and affords visual verification to ensure that no necessary vitals are overlooked. However, the rigid sequence provides little flexibility to accommodate variation in workflow, especially in circumstances when information is not yet available.
Keystroke Level Model
Table 1 presents the KLM model estimates for the processes of (1) accessing the page for vital signs charting and (2) entering new vital signs, as well as the task completion time. In Arizona and Florida, where SurgiNet is used, there was an average of 7.22 seconds to access the interface for documenting vital signs. The KLM Modes estimated this time as 8.00 seconds. This value is high in comparison to Eau Claire, using SIS, with actual values of 1.56 seconds and a KLM estimate of 2.40 seconds. This difference is largely attributed to the interface layouts. Specifically, in SurgiNet, users are required to create a new column for the series of vital signs being entered while in SIS, a button is clicked, and the date/time are automatically saved for that series of vitals. This design aspect in SIS more easily facilitates access, but does not have the flexibility to adjust the times for retrospective vital sign documentation unlike SurgiNet. Time to complete the series of vital signs are also shown in Table 1, with SurgiNet users requiring 58.49 seconds for actual completion of interactions that KLM predicted to take 68.40 seconds. In Eau Claire, SIS users needed significantly less time for vital sign documentation, requiring 37.06 seconds for actual completion time of interactions that KLM predicted to take 38.40 seconds. It should be noted that there was slight variation in the number of vital signs entered at each site. In Arizona and Florida (both use SurgiNet), there were 13 vital signs documented with a mix of open-text fields and multiple-choice selections. In Eau Claire, there were nine vital signs documented with only one multiple choice selection while all others were free-text.
Table 1:
KLM Model of all sites showing the KLM predicted task duration value versus the actual task duration
| Site | System | Accessing Vitals - Actual | KLM | Charting Vital Sign - Actual | KLM | Total Time | KLM |
|---|---|---|---|---|---|---|---|
| Arizona/Florida | SurgiNet | 7.22 | 8.00 | 51.27 | 60.40 | 58.49 | 68.40 |
| Eau Claire | SIS | 1.56 | 2.40 | 35.50 | 36.00 | 37.06 | 38.40 |
Figure 3 represents the sequential process of completing the vital signs charting task in Eau Claire using SIS. However, we did observe, on one occasion, that PowerChart was used as a secondary charting tool for specific cases where a patient is being transferred to another hospital ward following surgery. The reason for this is that all other wards of the Eau Claire hospital utilize PowerChart as their main documentation tool, so perioperative patient vitals need to be accessible in PowerChart for reference as well. In SIS, clinicians select a button labeled “CNA”, which generates a pop-up window. In PowerChart, nurses select the “AdHoc” charting button and under the “Assessments” section, select “Vitals and Measurements”. From this point on, SIS and PowerChart employ nearly identical steps. Nurses record patient’s weight, heart rate, non-invasive blood pressure, respiration rate, temperature, oxygen saturation and oxygen rate through a pop-up screen that cycles through the vital signs. This interface leaves little room for variation in the order in which vitals are entered. Finally, nurses reach the decision point of whether there was a manual observation required for patients. Once all vitals are entered on the pop-up screen, the task is completed. On the other hand, PowerChart requires nurses to click a save button on the top of the screen.
Figure 3:
A process model of the “Vitals signs Charing” task in SlS and powerChart (Eau Claire)
Figure 4 represents the process model in Arizona where SurgiNet is the main tool used for charting. The first step is entering the “IView” function of SurgiNet. Then, clinicians must decide whether vital signs have been entered or not. If there are vital signs already entered, the nurse confirms whether the vital signs entered are the most recent and accurate. If the vitals are accurate and up to date, the process is complete. If vital signs have not yet been entered, a date and time column are added. This column automatically fills with the current date and time when the column was created, but time often needs to be modified to the time the vitals were collected. Patient location is recorded for all patients, and then nurses enter the “Vital Signs Activity” (e.g. the point of care which vitals were recorded). When entering vital signs, SurgiNet displays patient information in a column form with all fields shown on one screen. This means that the nurse can navigate between various fields for entering vital signs. Though there is freedom to switch between fields for data entry, all vital signs listed in the workflow must be entered before moving on. Once all necessary vitals are entered, nurses can move to the next decision as to whether the patient has case-specific vital signs that needs to be documented and then the task is complete.
Figure 4:
A process model of the “Vital Signs Charing” task in SurgiNet (Arizana, Florida)
Interactive behavior measure for individual clinicians
Table 2 shows measures of interactive behavior collected from video ethnography for the task of “Vital Signs Charting” to include the mean, standard deviation and range, the time to task completion and the amount of mouse clicks and screen changes for performing the task. Results show that the mean time for completion is highest in Arizona subjects with 2 minutes 24 seconds while in Florida, the mean time is 55 seconds and in Eau Claire, a mean time of 48 seconds. In terms of mouse clicks, Arizona also had the highest amount with 30 clicks, with Florida having the second highest amount at 29.3 clicks and Eau Claire with the lowest amount of mouse clicks at 9.5 clicks. Though Arizona had the highest amount of mouse clicks, it had the lowest amount of screen changes with a mean of 0.83 screen changes. Florida and Eau Claire had very similar number of screen changes with means of 5.5 and 6 screen changes. The fewer screen changes in Arizona can be attributed to the columnar format interface design contrasted with Eau Claire using a pop-up screen that transitions through vital signs.
Table 2:
Summary of interactive behavior measures for the task “Vital Signs Charting” in each site
 |
Though Arizona and Florida use the same system, SurgiNet, Florida has more screen transitions due to the process in which vital signs are entered. In general, Arizona and Florida nurses employ similar patterns of screen changes with the exception of one outlier at each site. In Arizona, one patient needed a comparatively greater number of mouse clicks and screen changes, 84 clicks and 5 changes, respectively. This can mainly be attributed to the missing measurement of height in the patient’s record. The height measurement for the patient, a transfer from the hospital floor into surgery, could not be readily located in their record. To locate this information, the nurse had to serially negotiate each instance of previously recorded vital signs. This required a significant amount of searching and manual navigation through various screens.
There is an outlier in the Eau Claire observations (denoted with * in Table 2), with a significantly higher amount of total time, mouse clicks and screen changes than the other observations. This had nearly 10 times the amount of interactive behavior measures compared to the other two observations (i.e., 190 mouse clicks and 46 screen changes, versus 10 and 9 mouse clicks and 6 screen changes each). The main reason for the high amounts of interactive behavior is due to patient case specifics—procedural sedation called for vital sign collection every 5 minutes. In this instance, quantitative measures are elevated compared to the other two patient cases. Here each had 1 instance of vital signs charting, between 42 and 54 seconds, 10 and 9 mouse clicks and 6 screen changes each. The interface design aspects of SIS caused the high frequency of screen changes and mouse clicks, since each instance of vital signs charting required nurses to go through this standard set of vital sign screens.
Narrative summary for vital signs documentation in SIS (Eau Claire) for a complex patient
Table 3 provides a sequential breakdown of the various tasks performed in the nursing assessment for patient 3 in Eau Claire. This was a case of particular complexity because there were a greater number of vital signs charting instances which resulted in more interactions. The patient, a 75-year-old male receiving a right total knee arthroplasty adductor canal block, received a particular type of anesthesia prior to surgery where protocols required the collection of vital signs every five minutes until the patient was taken into the operating room. This case shows the standard series of vital signs collected during the PreOp nursing assessment. Due to the design of the vital signs charting interface in SIS, a high frequency of screen changes was required to enter each series.
Table 3:
Patient Case Narrative of Eau Claire patient 3
| Time | Activity | Dialogue/Notes |
|---|---|---|
| 0:10:33 | Nurse clicks “CNA” button to start vital signs | |
| 0:10:33 – 0:10:37 | Nurse begins charting vital signs in SIS | |
| 0:10:37 – 0:10:51 | Nurse searches for previous vital signs | |
| 0:10:51 – 0:11:05 | Nurse accesses vital signs page in PowerChart | |
| 0:11:06 – 0:11:26 | Nurse records vitals on sheet of paper | |
| 0:11:27 – 0:12:20 | Nurse enters vital signs into PowerChart | |
| 0:12:22 – 0:12:55 | Nurse switches to SIS | |
| 0:12:22 – 0:12:55 | Nurse charts vital signs in SIS | |
| 1:23:04 – 1:27:46 | SIS vital signs charting | RN: So, we monitor the vital signs every 5 minutes because of his procedure. Deep breath again, deep breath. I think some of this is just where his leads are placed. |
| 1:30:01 – 1:30:36 | SIS vital signs charting | |
| 1:36:05 – 1:36:35 | SIS vital signs charting | |
| 1:41:05 – 1:42:36 | SIS vital signs charting | RN: We just started using the CO2 monitors, but his CO2 is at 100%. The CO2 monitor catches his oxygen level dropping when he’s breathing normally.” |
| 2:00:55 – 2:01:22 | SIS vital signs charting |
Discussion
Variation and inconsistencies in EHR documentation can result in varying levels of inefficiencies, lack of uniformity in the charting process and may lead to challenges in the coordination of care. We contrasted the EHR workflow process for PreOp vital signs completion across two EHRs (three Mayo Clinic hospitals). We observed both large (i.e., interaction process) and small (e.g., arrangement of elements on the screen) differences in the interface design. Both sets of differences are consequential in the charting process. SurgiNet utilizes an interface that allows for more flexibility to navigate documentation sections yet had a higher time to task completion and required more mouse clicks.
SurgiNet had a higher KLM estimate to access the vital sign documentation interface but allowed for more accurate capture of date and time of vital signs documented. Because SurgiNet employs one screen and prompts scrolling to access information, there are additional steps required to document date and time for current vital signs that can increase cognitive load. Nurses must create a new column, which will automatically contain the date and time that the column is created, and then modify the time to match when the vitals were collected. In the context of the nurses’ work, they assess patients’ vital signs at the appropriate frequency; however, they often must retrospectively document patients’ vital signs in the EHR. SurgiNet accommodates this, but the process creates another level of burden for clinicians as they must have additional interactions with the system. The KLM analysis identified an increase in mental and physical operations in SurgiNet compared to SIS, which could lead to increases in task completion times, cognitive load and errors.
The impact of displaying information on one screen versus using transitioning screens requires fewer interactions, yet captures less patient information overall. In SIS, as shown in the Eau Claire case study, task completion time was shorter yet the number of vital signs documented and the overall effort to complete the task was higher. The methodology employed allows for the identification of the impact that interface design elements have in clinician cognition and EHR-mediated workflow. The methods used can identify, analyze and manage bottlenecks in EHR-mediated workflow, as well as be used to increase functionality of interfaces and decrease cognitive load on clinicians. In order to understand and identify barriers to workflow, the impact that interface elements have on task completion need to be understood. This work employs an analytic framework that utilizes employs both qualitative and quantitative measures to gain a deeper understanding of what interface design aspects promote workflow or create bottlenecks. To optimize the vital signs documentation process, workflow must be understood thoroughly and interface design aspects should be well integrated and optimally functional in EHRs. Notably, the Mayo Clinic is endeavoring to harmonize clinical workflows across all sites. This work serves to highlight the nature of the differences between systems at a fine-grained level and underscores the challenges ahead.
A limitation to our data collection is that there is a small number of observations and number of patients. However, these findings underline the importance of assessments of usability of software in order to create the best system for use clinically. Although we have small sample sizes, there are vast opportunities through the ROOT project to employ a combination of analytic and empirical method through the use of log file analysis in future work. This allows us to discern patterns of interactions from thousands of events across large number of patient cases to surface patterns of interface interactions. Because log files enable analysis of large data sets (e.g., interaction events), hundreds of siloed applications can be observed, allowing for a more comprehensive assessment of currently heterogeneous states of EHR-mediated workflows. These converging methods, along with triangulating qualitative and quantitative analyses could serve to strengthen the results of this work, as well as provide a more thorough understanding of clinicians’ EHR-mediated workflow and interactive behaviors.
Conclusions
Seamlessly integrating EHRs into clinical workflow has become a vital and necessary process to optimize patient safety. However, there are still numerous barriers to reaching this ideal state. Current EHR system designs often hinder task completion due to unintuitive and inefficient interactions within the interface while placing high cognitive burden on clinicians. In clinical settings, such burden can possibly affect patient safety and quality of care. This study aimed to identify barriers in interface design related to EHR-mediated workflow and illustrate how these barriers created challenges to task completion. A combination of analytic and empirical methods was used to better understand clinicians’ usability challenges and provide suggestions for interface redesign. The analyses completed here identified elements of design that facilitate tasks such as vital signs documentation, as well as those that impede them. Enhanced user experience can lead to safer and higher quality care in perioperative workflow and allied disciplines.
Acknowledgements
This research was funded by the Center for Translational Informatics and Knowledge Management at the Mayo Clinic. We thank the volunteer clinicians who participated in the study. We would also like to thank Sarah Hirn, Robert Sunday, Sheryl Castaneda and Joanne Bouchereau of Mayo Clinic.
References
- 1.Kim FJ, da Silva RD, Gustafson D, Nogueira L, Harlin T, Paul DL. Current issues in patient safety in surgery: a review. Patient safety in surgery. 2015 Dec;9(1):26. doi: 10.1186/s13037-015-0067-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.DeVon HA, Patmon FL, Rosenfeld AG, Fennessy MM, Francis D. Implementing clinical research in the high acuity setting of the emergency department. Journal of Emergency Nursing. 2013 Jan 1;39(1):6–12. doi: 10.1016/j.jen.2012.08.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Middleton B, Bloomrosen M, Dente MA, Hashmat B, Koppel R, Overhage JM, Payne TH, Rosenbloom ST, Weaver C, Zhang J. Enhancing patient safety and quality of care by improving the usability of electronic health record systems: recommendations from AMIA. Journal of the American Medical Informatics Association. 2013 Jun 1;20(e1):e2–8. doi: 10.1136/amiajnl-2012-001458. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Schumacher RM, Lowry SZ. National Institute of Standards and Technology. 2010. Nov, NIST guide to the processes approach for improving the usability of electronic health records. [Google Scholar]
- 5.Meeks DW, Takian A, Sittig DF, Singh H, Barber N. Exploring the sociotechnical intersection of patient safety and electronic health record implementation. Journal of the American Medical Informatics Association. 2013 Sep 19;21(e1):e28–34. doi: 10.1136/amiajnl-2013-001762. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Yeung MS, Lapinsky SE, Granton JT, Doran DM, Cafazzo JA. Examining nursing vital signs documentation workflow: barriers and opportunities in general internal medicine units. Journal of clinical nursing. 2012 Apr 1;21(7-8):975–82. doi: 10.1111/j.1365-2702.2011.03937.x. [DOI] [PubMed] [Google Scholar]
- 7.Stevenson JE, Israelsson J, Petersson G, Bath PA. Journal of clinical nursing. 2017. Nov 17, Factors influencing the quality of vital signs data in electronic health records: a qualitative study. [DOI] [PubMed] [Google Scholar]
- 8.Jalote-Parmar A, Badke-Schaub P. Workflow integration matrix: a framework to support the development of surgical information systems. Design Studies. 2008 Jul 1;29(4):338–68. [Google Scholar]
- 9.Norman DA. In: Cognitive engineering. Norman DA, Draper SW, editors. User centered system design. Hillsdale, NJ: Erlbaum Associates; 1986. pp. 31–61. [Google Scholar]
- 10.Kushniruk AW, Patel VL. Cognitive and usability engineering methods for the evaluation of clinical information systems. Journal of biomedical informatics. 2004 Feb 1;37(1):56–76. doi: 10.1016/j.jbi.2004.01.003. [DOI] [PubMed] [Google Scholar]
- 11.Johnson CM, Johnson TR, Zhang J. A user-centered framework for redesigning health care interfaces. Journal of biomedical informatics. 2005 Feb 1;38(1):75–87. doi: 10.1016/j.jbi.2004.11.005. [DOI] [PubMed] [Google Scholar]
- 12.Kanry J, Kushniruk A, Koppel R. Meaningful usability: health care information technology for the rest of us. In: Ong K, editor. Medical Informatics: An Executive Primer. Chicago: HIMSS; 2011. pp. 1–20. [Google Scholar]
- 13.Horsky J, Ramelson H. Z. Cognitive Errors in Reconciling Complex Medication Lists. AMIA Annual Symposium Proceedings. 2016;2016:638–646. [PMC free article] [PubMed] [Google Scholar]
- 14.Plaisant C, Wu J, Hettinger A, Powsner S, Shneiderman B. Novel user interface design for medication reconciliation: an evaluation of Twinlist. Journal of the American Medical Informatics Association. 2015;22:340–349. doi: 10.1093/jamia/ocu021. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Duncan B, Solomon A, Doebbeling B, Furniss SK, Grando A, Poterack KA, Miksch TA, Helmers RA, Kaufman DR. Human Factors in Ergonomics Healthcare Symposium Proceedings. 2018. Preoperative Medication Management: Reconciling Differences Across Clinical Sites. [Google Scholar]
- 16.Doebbeling B. N, Burton M, Kaufman D, R, Poterack K, McCullough M, Grando A, Miksch T, Helmers R. 2016. “Integrated Workflow Capture in an EHR Conversion: Standardizing on Best Practice Methods.”.AMIA. [Google Scholar]
- 17.Grando M. A, Groat D, Furniss S. K, Nowak J, Gaines R, Kaufman D. R, Poterack K. A, Miksch T, Helmers R. A. AMIA Annu Symp 2017 on November 4-8. 2017. “Using Process Mining Techniques to Study Workflows in a Pre-operative Setting”. [PMC free article] [PubMed] [Google Scholar]
- 18.Duncan B, Solomon A, Doebbeling B, Furniss SK, Grando A, Poterack KA, Miksch TA, Helmers RA, Kaufman DR. AMIA Annual Symposium Proceedings, WISH; 2017. A Comparative In-Situ Study of Two Medication Reconciliation Interfaces in Pre-Operative Nurse Assessments. [Google Scholar]
- 19.Kieras D. Ann Arbor: Department of Psychology, University of Michigan; 2001. Using the keystroke-level model to estimate execution times. Unpublished report. [Google Scholar]
- 20.Markowitz E, Bernstam EV, Herskovic J. Medication Reconciliation: Work Domain Ontology, Prototype Development, and a Predictive Model. AMIA Annual Symposium Proceedings. 2011;2011:878–887. [PMC free article] [PubMed] [Google Scholar]