Abstract
Objective
To systematically examine clinical workflows before and after a major electronic health record (EHR) implementation, we performed this study. EHR implementation and/or conversion are associated with many challenges, which are barriers to optimal care. Clinical workflows may be significantly affected by EHR implementations and conversions, resulting in provider frustration and reduced efficiency.
Patients and Methods
Our institution completed a large EHR conversion and workflow standardization converting from 3 EHRs (GE Centricity and 2 versions of Cerner) to a system-wide Epic platform. To study this quantitatively and qualitatively, we collected and curated clinical workflows through rapid ethnography, workflow observation, video ethnography, and log-file analyses of hundreds of providers, patients, and more than 100,000 log files. The study included 5 geographic sites in 4 states (Arizona, Minnesota, Florida, and Wisconsin). This project began in April 2016, and will be completed by December 2019. Our study began on May 1, 2016, and is ongoing.
Results
Salient themes include the importance of prioritizing clinical areas with the most intensive EHR use, the value of tools to identify bottlenecks in workflow that cause delays, and desire for additional training to optimize navigation. Video microanalyses identified marked differences in patterns of workflow and EHR navigation patterns across sites. Log-file analyses and social network analyses identified differences in personnel roles, which led to differences in patient–clinician interaction, time spent using the EHR, and paper-based artifacts.
Conclusion
Assessing and curating workflow data before and after EHR conversion may provide opportunities for unexpected efficiencies in workflow optimization and information-system redesign. This project may be a model for capturing significant new knowledge in using EHRs to improve patient care, workflow efficiency, and outcomes.
Abbreviations and Acronyms: EHR, electronic health record; HIT, health information technology; ROOT, Registry of Operational Tasks
US investment in electronic health record (EHR) systems has markedly increased owing to the implementation of the meaningful use policies of the American Recovery and Reinvestment Act (2009). EHR systems support care delivery, reducing medical error rates, supporting decision-making activities, ordering tests and treatments, improving the cost–benefit ratio, and improving the quality of health services.1
EHR implementation or conversion is associated with many challenges. The integration of fully functional EHR systems and these challenges are barriers to optimal care at both provider and organizational levels.2, 3 In hospitals, nurses are usually the frontline clinical staff that most incorporate EHR systems into their clinical workflow.4 Perceptions of clinical workflow impact are strongly associated with successful implementation and acceptance of EHR systems.1
Workflow is defined as the sequence of physical and mental tasks performed by individuals within and among work environments.5 Clinical workflow is a directed series of steps comprising clinical processes performed by people, equipment, and computers, which consume, transform, and/or produce information.6 Workflow is a dynamic and sometimes unpredictable system depending on user-centered design and collaboration and communication among independent caregivers. Clinical workflow must be taken into consideration to integrate the EHR into routine clinical practice.1 EHR design significantly affects workflow in both pre- and postoperative care by increasing access to information and documentation time at the expense of direct patient care.7, 8 The fit of HIT into clinical workflow is critical for safe, effective, and efficient HIT use.9
Mayo Clinic has invested significant resources in developing rich electronic ecosystems for clinical information management. Much effort by health information technology (HIT) professionals and providers was invested in eliciting and understanding the needs of clinicians to result in functionality and capabilities in numerous niche applications continuously evolved to enable high-quality care. The Plummer Project has transitioned all Mayo Clinic practice sites in 5 states to a single EHR. This presents a unique opportunity to compare clinical workflows pre- and postconversion directly.
To date, we have found no published mixed-method (qualitative and quantitative) self-study of any large-scale EHR conversion. Collecting a rich mixed-methods dataset on current EHR practices was envisaged to enable responses to whatever unintended consequences might occur postconversion.
Workflow challenges are particularly notable in both preoperative and postoperative care. Surgery has 3-fold more barriers to information flow than facilitators; often, these barriers result in undesirable outcomes for the patient, family, and providers.10 Surgical providers often spend time tracking down data that was delayed, missing, incomplete, or inaccurate, sometimes delaying surgery.10
Workflow can be assessed and described at many levels of aggregation and dimensions that are often in conflict.11, 12, 13 There is a significant body of research evaluating complex clinical workflows.13, 14, 15 These fall into clusters of research that address, at best, a few dimensions of clinical work and frameworks of limited scope.16 This restricts the ability to describe work that involves highly complex, distributed, collaborative, continuously evolving, interruptive, cognitive/informational, and physical elements.1, 2
Multiple factors mediate successful implementations of HIT and frameworks that endeavor to better reflect this complexity. Sittig and Singh proposed an 8-dimensional model of interrelated concepts that can be used to explain performance in adaptive systems in the health care arena. The model has been applied in a range of settings model to understand and improve HIT applications at various stages of development and implementation.17 Others have proposed multifaceted sociotechnical approaches to the study of HIT-mediated workflow in clinical settings. 18 Collectively, they make a persuasive case for a comprehensive mixed-methods approach. In addition, an emerging class of techniques leverage automated methods such as process mining for characterizing the distribution of patterns of workflow and quantifying dimensions of interactive behavior.19 Although still at an early stage, the process mining of log files (eg, as recorded in audit logs) is a promising method for understanding, quantifying, and modeling EHR use.20 We developed a project to learn from the Plummer Project, the goal of which is to collect and curate current clinical EHR workflow data to describe the current clinical workflow accurately and obtain and make available the highest-value baseline data and information necessary to maximize postimplementation optimization efforts.21 The ROOT project (Registry of Operations and Tasks) uses a structured, measured approach to capturing the data as completely and consistently as possible; this process and documentation will be considered a “reference implementation” for the capture and management of workflow for the Mayo Clinic. Clinical leaders recommended that the initial focus of the ROOT project be the Surgical and Emergency Department practices in 4 regional sites (Minnesota, Arizona, Florida, and Wisconsin), as operating room and ED throughput are key metrics for Mayo Clinic practice.
The convergence of a single-enterprise EHR has necessitated the standardization of many operational processes across Mayo sites. This is having a profound impact on the workflow of providers and staff, given the number and degree of niche, siloed applications in existence to meet the specific job duties of very specialized groups. Documentation will not only serve an archival purpose but also provide invaluable “prior state” data that can be used to characterize existing preimplementation processes in sufficient depth to reveal what has been lost in the conversion. This project began in April 2016, to capture and archive existing HIT-mediated workflow with the current 3 EHRs (GE Centricity and 2 versions of Cerner) before and after the conversion to Epic in all Mayo Clinic sites. It is a systematic effort to capture and curate a portion of the current electronic ecosystem.
Methods/Examples
Through the ROOT project, we are conducting a mixed-method approach to reveal an in-depth understanding of work components (eg, individuals, groups, contextual factors), as well as the interactive behavior and dependencies between the components that influence HIT-mediated performance (Figure). Toward this end, we have leveraged methods common to sociotechnical and human factors approaches. We employed a comprehensive, multidisciplinary, and mixed-method clinical workflow capture and analysis methodological framework that draws on numerous expert and evidence-based best practices. In addition, we can review, reanalyze, and repurpose the data at later dates. Emergent problems and solution strategies can be derived from both interviews and observations. They enable us to focus on pressing issues and contribute to future solutions strategies.
Figure.
Schematic of methods used in the project. The left-hand column illustrates the dimensions of interest and specific resources (eg, internal policy documents) that inform data collection. The second column characterizes the 5 central methods of data collection, including rapid ethnography (eg, key informant interviews, shadowing, walkthrough), EHR video-capture, log files, artifacts, and network analysis.20 The third column lists the type of information captured by each method. The analysis yields 4 categories of output.
The findings from various disparate yet complementary approaches can be normalized through formal task structures that explicitly define “who” does “what” (activities), “when,” “where,” and “how” (subactivities) by employing “which resources or tools,” consuming, altering, or producing “what information,” and in “what relation” to other tasks or entities formally expressed in a work domain ontology and visually depicted as activity network diagrams.22 The informal task structures are further elicited to express the actions of individual entities interacting dynamically in a complex activity system through mechanisms of coordination, awareness of coworkings, and qualitative descriptors that preserve the richness of findings from distinct methods.
A series of interrelated, interdisciplinary projects (Figure) are currently underway for (1) capturing and archiving a multisite, multiprovider observational, interview and video dataset of EHR, clinical workflow, and contextual information; (2) discovering workflows from event logs generated by various information and communication systems; and (3) creating a repository of work data that serve as digital signatures or patterns in an organization to understand common practices, deviations, and need for intervention and monitoring. Our study began on May 1, 2016, and is ongoing.
Objectives
The objective of this research is to employ best practices from clinical and research informatics to advance the study of clinical workflow, elucidate barriers to productive workflow, and identify opportunities for optimization and to contribute to operations and outcomes. The data can be used to understand better the unanticipated consequences of any large-scale EHR conversion. The ultimate objective is to develop a standardized, scalable process to analyze clinical workflow activities related to EHR use and leverage this process to drive the optimization of health-information systems.
State of Research
In the first phase of the project, we developed a comprehensive approach to data collection, analysis, and workflow modeling. The project also surfaced a range of findings on workflow barriers and optimization opportunities. The first round of research yielded findings that vary in terms of their immediate clinical implications with some work best described as exploratory and others that can have a more immediate impact. We have recently begun the second phase to extend the approach to make it scalable, standardized, and more robust. We envisage a third phase in which we engage predictive modeling of workflow, leveraging simulations and data mining methods, and work to contribute to learning health-systems solutions that can enhance the HER-user experience, better support workflow and decision making, and contribute to clinical practice improvement and system redesign goals. We have also begun to collect postimplementation data. Although the data collection is still in progress, and the analysis in the relatively early stages, some preliminary findings are reported here. We can collect postimplementation data once a period of 6 months has elapsed. That is the generally accepted minimum time frame for workflow and clinical activities to stabilize postimplementation.
The ROOT project amassed a large set of data that captures high definition snapshots of workflow in different settings. The project has resulted in a coherent methodological framework to allow us to elucidate patterns of workflow. In this section, we illustrate findings derived from different methods. The Table presents an overview of data collected prior to the implementation and postimplementation (thus far).
Table.
An Overview of Data Collected Prior to EHR Conversion and Postconversion
| Location | Interviews (Employees) |
Observations (Hours) |
Event logs |
|||
|---|---|---|---|---|---|---|
| Pre | Post | Pre | Post | Pre | Post | |
| Phoenix | 32 | Pending | 111 | Pending | 76,000 logs, 15 providers, 142 patients | Pending |
| Jacksonville | 28 | Pending | 113 | Pending | 116,706 logs, 31 providers, 275 patients | Pending |
| Eau Claire | 17 | 23 | 74 | 26 | 577,466 logs, 73 providers, 83 patients | 1,436,350 logs, 3 providers |
| Saint Marys | 33 | 9 | 111 | 28 (in progress) | 105,712 logs, 112 providers, 1997 patients | Pending |
| Methodist | 11 | 9 | 42 | 36 (in progress) | Pending | |
Interviews
Before implementation, 121 semistructured leadership and clinician interviews related to the use of the EHR and planned conversion have been conducted across the 5 Mayo Clinic systems. Analysis of the interviews identified a range of themes that clustered around key points including IT tools, quality and performance, teamwork, coordination, management and leadership, and optimization opportunities. Salient themes include the importance of prioritizing clinical areas with the most intensive EHR use, the value of tools to identify bottlenecks in workflow that cause delays, and desire for additional training to optimize navigation of the systems. One interviewee said, “Part of our challenge is there is so much work to be done that it is a bit siloed. None of us has seen the big picture flow of what happens from the time a procedural patient enters the system to when they leave.” Leaders have also identified a broad range of ideas and practices for further optimization of the EHR, other IT tools, and workflow.
EHR-Mediated Workflow
We employed Morae TM 3.3, video-analytic software to capture EHR workflow. Clinicians were recorded performing EHR tasks in the context of their routine workflow in surgical and ED settings. The software captured the participants’ screens, a set of analytics (eg, mouse clicks and Web-page changes), and, through the use of a Webcam, audio and video recording of the participants’ hands (to capture additional documentation via paper and checklists to coordinate clinical care). We were able to quantify tasks and processes and develop measures of efficiency: for example, as reflected in time on task, interruptions, keystrokes and mouse clicks. We also developed in-depth qualitative analyses of longitudinal events such as preoperative nursing assessment workflow. We were able to discern particular patterns of care coordination (eg, handoff) and problem-solving (eg, workaround) concerning EHR use.
We completed detailed analysis of Morae video captures of nurses performing preoperative assessment at 5 Mayo Clinic sites: Arizona and 2 Hospitals in Minnesota, Wisconsin, and Florida. The study documented significant variation across many measures. Observed variation in workflow was driven by (1) idiosyncratic interaction patterns for individual nurses; (2) patient condition and complexity; (3) elective vs emergent surgery; (4) role differences at each site; (5) situational differences in tempo, velocity, and scheduling on a given day, and perhaps most importantly: (6) differences in the application interface that create varying levels of work and cognitive burden. In addition, we investigated high-priority tasks such as medication reconciliation, vital signs documentation, patient-order management, and medication administration record interfaces at multiple sites. The interfaces differed in their modes of interaction and how they supported user interaction.7, 19 We found unnecessary complexities that affected the efficiency of the reconciliation process. Different EHR and HIT designs differently mediate task performance, which can have critical impact on the user’s experience and may thus affect patient safety. For example, one system used a more minimalist approach and required fewer interactive behaviors tha other systems, which are more complex but offer additional functionality. The tradeoff between ease of use and enhanced functionality can be quantified to determine the relative value of providing additional tools vs a more streamlined process. For example, functions that are used infrequently can be removed from the navigation path yet remain accessible to the user should they be needed.
We also contrasted the EHR workflow process for preoperative vital-signs completion across 2 EHRs (3 Mayo Clinic hospitals). We observed both large (interaction process) and small (arrangement of elements on the screen) differences in the interface designs. Both sets of differences are consequential in the charting process. Contrary to the MedRec task, the SurgiNet interface for vital signs allowed greater flexibility to navigate documentation sections, yet had a higher time to task completion and required more mouse clicks. Variation and inconsistencies in EHR documentation can result in varying levels of inefficiencies, lack of uniformity in the charting process, and may lead to challenges in coordination of care.
We have begun to perform similar comparisons and analyses postimplementation.23 Specifically, we scrutinized interface differences and quantified interactive behavior with regard to the MedRec task. The new system employed a very different interface and approach to documentation. Compared with other systems used before implementation, we observed a shorter mean time to task completion and fewer mouse clicks and screen changes per medication during the MedRec task with the new system. We can hypothesize that the fewer screen changes also reduced the burden on working memory and cognitive load. Thus, we can infer that the different mode of interaction supported by the system may result in a more efficient documentation process.
In a related study, we investigated variation in preoperative workflow findings in 2 regional referral hospitals. We applied a new novel methodological framework using workflow microanalysis to characterize the EHR-mediated workflow across the preoperative care process. Analyses focused on the distribution of work for EHR tasks and off-screen tasks. Suboptimal patterns were identified and reasons for variation explored. Although both settings used the same EHR system, we observed marked differences in patterns of workflow. Arizona nurses devoted more time to using the EHR for nursing assessment and completing the case. Florida nurses spent a higher proportion of their time delivering patient care and performing the medication-administration task. The EHR tasks and off-screen tasks time distribution and patterns were notably different across sites. We extended the same approach to studying workflow postimplementation. Following the conversion, there was a significant reduction in total time devoted to preoperative EHR tasks. Of the total preoperative time, there was also a sizable shift in the distribution of effort toward increased patient care and less time on EHR tasks. The workflow is in part driven by different by different protocols, patterns of information gathering, and in the coordination of care.
The ROOT project has successfully mined rich veins of qualitative data to develop insights about variation in clinical workflow.24 Process mining provides a quantitative analysis of workflows based on computer event logs. Event logs are automated recordings of system events maintained by institutions for different reasons including matters of quality and safety. This analysis was used in concert with the methods noted above to develop a richly detailed picture of workflow in different settings.10, 24 In one substudy, the team contrasted workflow at 2 different Mayo Clinic sites in preoperative surgery. The dataset included 116,706 event logs generated over 7 days in the preoperative setting including data from 31 providers and supporting staff and 375 patients. Although these 2 particular facilities share the same EHR applications, significant differences in the physical organization of surgical areas and personnel roles were observed. Differences in personnel roles led to differences in patient–clinician interaction as well as time spent using the EHR and paper-based artifacts. The findings were also supported by social network analysis and personnel’s interactions with patient cases, captured through observational techniques and process mining. We are in the early phases of process mining the data to understand the dimensions of change.
Conclusion
This major clinical transformation project was studied thoroughly, qualitatively, and quantitatively for clinical optimization as well as to gain significant knowledge about the impact of EHR implementation on clinical workflow. This knowledge may help to discern ways to minimize the impact on clinical care and to optimize clinical care during large EHR implementations. Mayo Clinic’s ROOT project provides opportunities for further optimization, tool development, and research and development. It will serve as a basis to assess the extent of workflow change empirically through process and outcome measures for both providers and patients. Additional benefits include informing data semantics for historical records and research initiatives that leverage longitudinal clinical data. We believe we have only scratched the surface of this rich prospective, multisite, multimethod dataset.
Although EHRs offer considerable promise for enhancing patient care, implementations have been fraught with numerous challenges, leading to suboptimal and unsatisfying results. Despite significant research on EHRs and workflow, comparatively few large-scale studies of EHR conversion have been conducted, and none of this scope. There is a pressing need to develop a cumulative knowledge base so that medical centers can make data-driven decisions and that insights can be gained into high-performing practices.
The ROOT project will examine clinical workflows in each of the different legacy EHR domains and compare these with clinical workflows in the new single shared EHR. The project is at an early stage of learning from this rich prospective, multisite, mixed-method dataset. The Mayo Clinic ROOT project may be a model for capturing significant new knowledge in using EHRs to improve patient care.
Acknowledgments
We thank the volunteer clinicians who graciously allowed us to observe, interview, or record them in clinical practice contexts. We are very grateful to the Translational Informatics and Knowledge Management Program at the Mayo Clinic Kern Center for funding the project. We would also like to thank Benjamin Duncan, Lu Zheng, A.J. Solomon, Pooja Paode, and Shweta Yadav of ASU, and Robert Sunday, Sarah Hirn, Joanne Bouchereau, and Sheryl Castaneda of Mayo Clinic for assistance with data collection and analysis.
Footnotes
Potential Competing Interests: The authors have no financial relationships with any organizations that might have an interest in the submitted work in the previous 3 years and no other relationships or activities that could appear to have influenced the submitted work.
References
- 1.Castillo V.H., Martinez-Garcia A.I., Pulido J.R.G. A knowledge-based taxonomy of critical factors for adopting electronic health record systems by physicians: a systematic literature review. BMC Med Inform Decis Mak. 2010;10(1):60. doi: 10.1186/1472-6947-10-60. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.McAlearney A., Hefner J.L., Sieck C.J., Huerta T.R. The journey through grief: insights from a qualitative study of electronic health record implementation. Health Serv Res. 2015;50(2):462–488. doi: 10.1111/1475-6773.12227. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Boonstra A., Broekhuis M. Barriers to the acceptance of electronic medical records by physicians from systematic review to taxonomy and interventions. BMC Med Inform Decis Mak. 2010;10(1):23. doi: 10.1186/1472-6963-10-231. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Xiao Y., Montgomery D.C., Philpot L.M. Development of a tool to measure user experience following electronic health record implementation. J Nurs Admin. 2014;44(7/8):423–428. doi: 10.1097/NNA.0000000000000093. [DOI] [PubMed] [Google Scholar]
- 5.Agency for Healthcare Research and Quality What is workflow? https://healthit.ahrq.gov/health-it-tools-and-resources/evaluation-resources/workflow-assesment-health-it-toolkit/workflow [DOI] [PubMed]
- 6.Clinical Workflow Center Workflow and Process Definitions. https://www.clinicalworkflowcenter.com/resources/workflow-process-definitions
- 7.Saleem J.J., Militello L.M., Arbuckle N., et al. Vol. 2009. AMIA Symposium; San Francisco, CA: 2009. pp. 558–562. (Provider Perceptions of Clinical Decision Support at Three Benchmark Institutions). [PMC free article] [PubMed] [Google Scholar]
- 8.Murphy J. Leading from the future: leadership makes a difference during electronic health record implementation. Front Health Serv Manag. 2011;28(1):25–30. [PubMed] [Google Scholar]
- 9.Carayon P., Karsch B., Cartmill R., et al. Agency for Healthcare Research and Quality; Rockville MD: 2010. Incorporating Health Information Technology Into Workflow Redesign: Summary Report. [Google Scholar]
- 10.Schultz K., Carayon P., Hundt A.S., et al. Care transitions in the outpatient surgery preoperative process; facilitators and obstacles to information flow and their consequences. Cogn Technol Work. 2007;9(4):219–231. [Google Scholar]
- 11.Sockolow P.S., Weiner J.P., Bowles K.H., Lehmann H.P. A new instrument for measuring clinician satisfaction with electronic health records. Comput Inform Nurs. 2011;29(10):574–585. doi: 10.1097/NCN.0b013e31821a1568. [DOI] [PubMed] [Google Scholar]
- 12.Duncan B.J., Solomon A.H., Doebbeling B.N., et al. AMIA WISH; Washington, DC: November 4-8, 2017. A Comparative In-Situ Study of Two Medication Reconciliation Interfaces in Pre-operative Nurse Assessments. [Google Scholar]
- 13.Grando M.A., Groat D., Furniss S.K., et al. November 4-8, 2017. Using process-mining techniques to study workflows in a pre-operative setting. AMIA Annual Symposium. [PMC free article] [PubMed] [Google Scholar]
- 14.Duncan B., Zheng l., Doebbeling B., et al. Reconciling differences in preoperative medication management across clinical sites. Proceedings of the International Symposium on Human Factors and Ergonomics in Health Care. 2018;1(4):44–51. [Google Scholar]
- 15.Paode P., Doebbeling B.N. In: Cognitive Informatics: Reengineering Clinical Workflow for More Efficient and Safer Care. Zheng K., Westbrook J., Kannampallil T.G., Patel V.L., editors. Springer; Hoboken, NJ: 2019. Workflow at the edges of care. [Google Scholar]
- 16.Vellore V.R., Grando M.A., Duncan B., et al. Process mining and ethnography study of medication reconciliation tasks. AMIA Annu Symp. 2019 In Press. [PMC free article] [PubMed] [Google Scholar]
- 17.Sittig D.F., Singh H. A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. Qual Saf Health Care. 2010;19(suppl 3):68–74. doi: 10.1136/qshc.2010.042085. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Holden R.J., Carayon P., Gurses A.P., et al. SEIPS 2.0: a human factors framework for studying and improving the work of healthcare professionals and patients. Ergonomics. 2013;56(11):1669–1686. doi: 10.1080/00140139.2013.838643. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Zheng K., Haftel H.M., Hirschl R.B., O'Reilly M., Hanauer D.A. Quantifying the impact of health IT implementations on clinical workflow: a new methodological perspective. J Am Med Inform Assoc. 2010;17(4):454–461. doi: 10.1136/jamia.2010.004440. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Hribar M.R., Read-Brown S., Goldstein I.H., et al. Secondary use of electronic health record data for clinical workflow analysis. J Am Med Inform Assoc. 2017;25(1):40–46. doi: 10.1093/jamia/ocx098. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Doebbeling B.N., Burton M.M., Kaufman D., et al. AMIA Annual Symposium; Washington, DC: November 12-16, 2016. Integrated Workflow Capture in an EHR Conversion: Standardizing on Best Practice Methods. [Google Scholar]
- 22.Zhang J., Walji M.F. TURF: toward a unified framework of EHR usability. J Biomed Inform. 2011;44(6):1056–1067. doi: 10.1016/j.jbi.2011.08.005. [DOI] [PubMed] [Google Scholar]
- 23.Grando A., Groat D., Furniss S.K., et al. AMIA Annual Symposium; Washington DC: 2017. Using Process-Mining Techniques to Study Workflows in a Pre-operative Setting. [PMC free article] [PubMed] [Google Scholar]
- 24.Gillum R.F. From papyrus to the electronic tablet: a brief history of the clinical medical record with lessons for the digital age. Am J Med. 2013;126(10):853–857. doi: 10.1016/j.amjmed.2013.03.024. [DOI] [PubMed] [Google Scholar]