Workflow Integration Analysis of a Human Factors-based Clinical Decision Support in the Emergency Department

Abstract

Numerous challenges with the implementation, acceptance, and use of health IT are related to poor usability and a lack of integration of the technologies into clinical workflow, and have, therefore, limited the potential of these technologies to improve patient safety. We propose a definition and conceptual model of health IT workflow integration. Using interviews of 12 emergency department (ED) physicians, we identify 134 excerpts of barriers and facilitators to workflow integration of a human factors (HF)-based clinical decision support (CDS) implemented in the ED. Using data on these 134 barriers and facilitators, we distinguish 25 components of workflow integration of the CDS, which are described according to four dimensions of workflow integration: time, flow, scope of patient journey, and level. The proposed definition and conceptual model of workflow integration can be used to inform health IT design; this is the purpose of the proposed checklist that can help to ensure consideration of workflow integration during the development of health IT.

1. Introduction

Health IT has the potential to improve the safety of patient care (Lin, Jha, & Adler-Milstein, 2018; Yuan, Dudley, Boscardin, & Lin, 2019); however, the usability and uptake of these technologies continues to be a challenge (HealthIT.gov, 2019; Ratwani, Reider, & Singh, 2019; Ratwani et al., 2018). Poor usability of health IT contributes to patient safety risks (Institute of Medicine, 2012) as well as negative consequences to clinicians such as increased workload, low job satisfaction and increased burnout, a major problem affecting approximately 50% of clinicians (National Academies of Sciences, 2020; Shanafelt et al., 2016). Workflow integration has emerged as a key concept related to the usability, acceptance, and use of health IT (Bowens, Frye, & Jones, 2010), and numerous experts have emphasized the importance of seamlessly integrating health IT in clinical workflows (Bates et al., 2003; Shortliffe & Sepúlveda, 2018; Sittig et al., 2008; Tcheng et al., 2017). Challenges with the integration of health IT in clinical workflow persist, largely relating to inadequate consideration of the work and work system of users (Blijleven, Koelemeijer, Wetzels, & Jaspers, 2017; Greenes et al., 2018; Karlin-Zysman, Zeitoun, Belletti, McCullagh, & McGinn, 2012; Militello et al., 2004). Human factors (HF) methods and approaches have been recommended in order to improve the usability and workflow integration of health IT (The Office of the National Coordinator for Health Information Technology, 2020).

While integration of technologies into clinical workflows is critical to health IT acceptance and use, the concept of workflow integration is inconsistently defined in the literature. In this study, we propose a definition and conceptual model of workflow integration of health IT based on HF concepts of workflow (Carayon et al., 2012), IT usability (Marcilly, Ammenwerth, Vasseur, Roehrer, & Beuscart-Zéphir, 2015; Scapin & Bastien, 1997) and the work system model (Carayon, 2009; Smith & Carayon-Sainfort, 1989). We apply our definition and conceptual model of workflow integration in a study of barriers and facilitators to workflow integration of a clinical decision support (CDS) in the emergency department (ED) of a large academic health system. Better understanding the concept of workflow integration can inform the design of health IT, which may ultimately increase clinicians’ acceptance and use of health IT, reduce its negative impact on clinicians (i.e. burnout), and improve patient safety.

1.1. Workflow

Before defining workflow integration, we need to first describe workflow. Traditional perspectives of workflow are often simplistic, with workflow defined as a sequence of work activities that unfold over time. In the Workflow Elements Model, Unertl et al. (2010) propose an enhanced perspective that acknowledges the multiple factors and dimensions of workflow, such as organizational policies and teamwork. Carayon et al. (2012) defined workflow as “the flow of people, equipment (including machines and tools), information, and tasks, in different places, at different levels, at different timescales continuously and discontinuously, that are used or required to support the goals of the work domain” (pg. 509). They described 4 aspects of workflow: time, flow, space, and level. In essence, “workflow” is the “process” in the Systems Engineering Initiative for Patient Safety (SEIPS) model (Carayon et al., 2006; Carayon et al., 2014), where the process is the combined product of the work system elements interacting and unfolding over time. In line with Unertl et al. (2010), this approach recognizes the complexities of workflow. In our research, we use the definition of workflow by Carayon et al. (2012), as it is grounded in a systems approach, i.e. the SEIPS model (Carayon et al., 2006; Carayon et al., 2014), and allows us to develop a systems perspective on workflow integration of health IT.

1.2. Perspectives on workflow integration

We found 3 perspectives on workflow integration in the literature (see table 1), which include different dimensions of workflow as defined by Carayon et al. (2012). The perspectives propose different relationships between workflow integration and usability (see table 1). One perspective conceptualizes workflow integration (or compatibility) as a dimension of usability. A second perspective views workflow integration as having several components, such as usability and workload. However, ‘workload’ is related to efficiency, one of the dimensions of usability according to the definition by ISO (International Organization for Standardization, 2018). The components of workflow integration (e.g. usability, workload) are not independent as suggested by this second perspective. The third perspective proposes that workflow integration is separate from usability. In line with the work of Scapin and Bastien (1997) and Marcilly et al. (2015) (i.e. first perspective in table 1), we posit that workflow integration is one aspect of the broader usability of a technology. This perspective (unlike perspective 3 in table 1) recognizes the inherent relationship between workflow integration and usability outcomes as defined by ISO (International Organization for Standardization, 2018), i.e. efficiency, effectiveness and satisfaction. We propose that when a technology is well integrated in clinician workflow, a technology’s efficiency and effectiveness, and user satisfaction (i.e. the 3 outcomes of usability as defined by ISO) may be enhanced (Carayon et al., 2019). For example, a CDS designed to support diagnosis of pulmonary embolism can be integrated in the workflow by proposing next steps such as ordering a diagnostic test (e.g. CT scan) and leading users to the appropriate ordering screen. By supporting workflow in this way, the CDS can increase efficiency by reducing the time to place orders and may improve the order accuracy (i.e. effectiveness) as well as user satisfaction.

Table 1:

Perspectives of workflow integration

Authors	Approach	Definition of workflow integration	Relationship between usability and workflow integration	Dimensions of workflow considered in this approach*
Scapin and Bastien (1997); Marcilly et al. (2015)	Usability heuristics for health IT	“how the characteristics of the system under design/evaluation fit: the characteristics of the tasks to be performed with/supported by the system, the characteristics of the typical end-user(s) (mental model, knowledge organization, cognitive tasks), and the characteristics of the typical end-user(s) workflow”
Flanagan et al. (2011); Doebelling et al. (2011)	Survey to assess workflow integration of CDS	4 aspects: • Navigation (e.g. patient information is easy to find) • Functionality (e.g. the CDS helps you perform the tasks you need to during face-to-face patient encounters) • Usability (e.g. the CDS is challenging to use) • Workload (e.g. using the CDS during face-to-face patient encounters adds effort).
Press et al. (2015)	Design of CDS in the emergency department	“consideration of what timing in the patient interaction the CDS is triggered”

^*The dimensions of workflow are those identified by Carayon et al. (2012)

1.3. Proposed definition of workflow integration

Expanding on the work of Scapin and Bastien (1997) and Marcilly et al. (2015) (see table 1), the work system model (Carayon, 2009; Smith & Carayon-Sainfort, 1989), and the definition of workflow by Carayon et al. (2012), we propose that workflow integration of a technology means that: “The technology is seamlessly incorporated within the work system elements (i.e. people, tasks, other technologies/tools, physical environment, organization) and their interactions over time (i.e. process), specifically considering the temporal order in which work is accomplished and the point in time in which the technology will be used. This includes how the technology fits within the sequence and flow of tasks, people, information, and tools/technologies; at the individual, team, and organizational level; across different scopes of the patient journey.”

When a new technology is implemented, it alters the entire work system, i.e. its elements and their interactions, resulting in the emergence of a new workflow within the system (Wilson, 2014). As the interactions between work system elements change with the new technology, new properties emerge within the system leading to changes in the temporal nature of work, i.e. workflow. Therefore, workflow integration is about system interactions and how well the new technology fits (or does not fit) within the temporal flow of work (i.e. process). A technology is integrated in the workflow if the system interactions resulting from the introduction of the new technology fit in the flow of work. A technology is not integrated in the workflow if the system interactions do not fit in the flow of work. Workflow integration (or lack of) leads to work system outcomes. Positive outcomes include satisfaction with the technology, acceptance and use of the technology, and reduced workload and time to complete tasks. Negative outcomes include frustration, stress, workarounds, increased workload and time to complete tasks, and dissatisfaction with the technology.

Below (see table 2) we outline four proposed dimensions of workflow integration based on the definition of workflow by Carayon et al. (2012). We clarified the dimension of “Space” to “Scope of patient journey”.

Table 2:

Dimensions of workflow integration

Dimensions of workflow integration	Sub-dimensions	Description
Time The technology is integrated with the temporal nautre of work. The technology fits in the workflow, which unfolds over time.	Sequential	The technology fits with the sequential flow of tasks/people/information/tools one after the other (occurring in a sequence)
	In parallel	The technology fits with the flow of multiple tasks/people/information/tools at the same time; the technology supports the flow of tasks/people/information/tools concurrently.
	Discontinuous	The technology fits with the flow of tasks/people/information/tools that is interrupted throughout time. The flow of tasks/people/information/tools is discontinuous; for example, a task may start, then be interrupted and stop, and then resume at a later point in time. Does the technology support this, i.e. going back to the original task/tools-technologies/information after interruption?
	Timing	The technology fits in the flow of tasks/people/information/tools at a specific point in time (e.g. when is it used? when does the technology show up?). The technology is available at the point in time in which it is needed by the physician.
	Extended time	The technology is integrated in the workflow as it occurs over an extended period of time, such as days, weeks, or months. For example, residents in the ED rotate to other services such as cardiology. When the residents rotate to these other services, they adapt to a new workflow during that time (e.g. over months of time) that may differ than the workflow used in the ED. Does the technology fit the workflow over this extended period of time?
Flow The technology is integrated in the flow of tasks, people, information, and other tools/technologies	Tasks	The technology is integrated in the flow of tasks, which occurs over space and time. In order for a technology to integrate in the workflow according to the flow of tasks, the technology should be available/accessible in a location near the preceding and succeeding tasks (e.g. reviewing vitals, completing documentation) and support next steps (e.g. placing an order).
	People	The technology is integrated in the flow of people including the physical environment in which people flow through. The flow of people occurs over space and time.
	Information	The technology is integrated in the flow of information. For instance, if the technology relies on lab results or information from another clinic, this information is available and integrates with the technology. Triggers represent one type of the flow of information. There are two specific considerations for triggers: • Trigger versus no trigger: Whether or not a trigger is used as a reminder to use the technology • Point in time for trigger: The point in time in the workflow for the trigger to occur
	Other tools/technologies	The technology is integrated in the flow of other tools and technologies; this includes the technology’s integration with other CDS that are used by the physician for other diagnoses. It also includes integration of the CDS with other parts of the EHR and other tools such as MDCalc.
Scope of patient journey The scope of the patient journey (i.e. part of the process) in which the technology is integrated within	Within patient interaction	The technology is integrated in the workflow during a patient interaction (e.g. with a clinician, staff); the technology is integrated with a small portion of the patient journey.
	Within patient visit	The technology is integrated in the workflow within the patient (e.g. ED) visit; this is a larger part of the patient journey.
	Before, within, and after patient visit	The technology is integrated in the workflow before (e.g. urgent care visit), within, and after (e.g. hospital stay) the ED visit; the technology is integrated in a large part of the patient journey. For example, the technology is integrated in the workflow as a patient moves from the ED to the hospital by facilitating the transfer of information between the ED attending and the hospitalist.
Level The technology is integrated into the different levels of workflow including the individual workflow, the workflow of a team, and the organizational workflow. The dimension, LEVEL, relates to how many people are involved in the workflow in which the technology is integrated	Individual	The technology is integrated in the workflow of an individual.
	Team	The technology is integrated in the workfolw of a team working together.
	Organization	The technology is integrated in the workflow of the organization.

1. Time

   1. Sequential
   2. In parallel
   3. Discontinuous
2. Flow of

   1. Tasks
   2. People
   3. Information
   4. Tools
3. Scope of patient journey

   1. Intra-visit
   2. Intra-organizational
   3. Inter-organizational
4. Level

   1. Individual
   2. Team
   3. Organization

Our conceptualization of workflow integration is based on the following premise:

1. Workflow integration (or lack of) is related to the entire work system, its elements and their interactions. When a new technology is introduced, it interacts with one or several work system element(s), and these new interactions influence workflow integration.

2. Workflow integration is tightly coupled with the temporal nature of work, i.e. process.

3. Workflow integration (or lack of) produces work system outcomes, which can be positive or negative.

In this study, we applied our proposed definition of workflow integration to understand the barriers and facilitators to workflow integration of an HF-based CDS used in the ED of a large academic health system.

2. Methods

2.1. Context of the study

This study was part of a larger study funded by the Agency for Healthcare Research and Quality (AHRQ) on health IT-supported processes for venous thromboembolism (VTE) diagnosis and prevention (https://cqpi.wisc.edu/research/health-care-and-patient-safety-seips/vte-and-health-it/). Pulmonary embolism (PE), a blood clot causing a blockage in the lung, is a common source of diagnostic problems and safety risks (e.g. over-use of harmful CT scans) in the ED (Hall, Schenkel, Hirshon, Xiao, & Noskin, 2010). Clinical decision support (CDS), which combines individual patient data and evidence-based guidelines to inform diagnostic decisions at the point of care (Garg, Adhikari, McDonald, & et al., 2005; Patterson et al., 2019), has the potential to improve diagnostic safety for health problems such as PE (Carayon et al., 2019; Raja et al., 2015).

Using HF principles and approaches, we designed a CDS to support the PE diagnostic process (Carayon et al., 2019; Hoonakker et al., 2019; Salwei et al., 2019). The PE Diagnosis (Dx) CDS, so called PE Dx, is embedded within the electronic health record (EHR) and combines two risk scoring algorithms, the Wells’ criteria (Wells et al., 2001) and the PERC rule (Kline, Peterson, & Steuerwald, 2010), which are recommended to support diagnosis of PE. Prior to implementation of the PE Dx CDS, physicians typically used an external online calculator, MDCalc (www.mdcalc.com) to calculate the Wells’ and the PERC and assess a patient’s risk of PE (Carayon et al., 2019). Leveraging the Wells’ and PERC, the PE Dx auto-populates data from the EHR (e.g. heart rate, age), calculates the patient’s risk score, provides a recommendation for the appropriate next step, supports ordering the recommended diagnostic test (e.g. D-dimer, CT scan), and automatically documents the decision and order in the physician’s note. One academic health system implemented the PE Dx in their ED in December 2018.

2.2. Sample and setting

We used a qualitative approach to deepen our understanding of workflow integration. Our goal was to define the concept of workflow integration and identify all possible dimensions relating to the concept. To do this, we conducted semi-structured interviews with a total of 12 emergency medicine physicians: 6 medical residents (post-graduate years 1, 2, and 3) and 6 attending physicians. At the start of each interview, we asked participants to classify themselves as one of the following PE Dx user groups:

1. Previous users: ED physicians who have used the PE Dx, but no longer use it.

2. Current users: ED physicians who use the PE Dx.

During the data collection process, a survey was conducted in the ED evaluating the implementation and use of the PE Dx. To ensure variety in the sample according to physician role and PE Dx user type, we purposively sampled interviewees using the survey data. In order to ensure maximum coverage of the domain of workflow integration, we aimed to get a heterogeneous sample of both residents and attendings as well as physicians who were current and previous users of PE Dx; this sampling strategy helped to elicit a wide variety of barriers and facilitators to workflow integration of the CDS. The final distribution of interviewees included 7 previous users of PE Dx (5 residents and 2 attendings) and 5 current users of PE Dx (1 resident and 4 attendings). We tracked data saturation throughout the interview process (i.e. number of unique barriers and facilitators mentioned by each new interviewee), and stopped interviewing participants once we determined saturation had been reached. Our final sample size of 12 is appropriate given the study aim, sample specificity, quality of dialogue, and the analysis strategy (Malterud, Siersma, & Guassora, 2016) and is in line with similar HF qualitative studies on acceptance and use of health IT (Asan, 2017; Melnick et al., 2017; Whittaker, Aufdenkamp, & Tinley, 2009). We only included physicians who worked in the ED at the participating health system and who were available during the data collection period (September-December 2019). Participation in the study was voluntary. This study was approved by the associated Institutional Review Board.

2.3. Data collection

Each interview was conducted face-to-face by one or two HF researchers. In each interview, we used the EHR “playground” environment, which mirrors the actual EHR used in the ED, to display the PE Dx to interviewees. The use of the EHR playground environment served as a visual cue to participants to remember the various features of the PE Dx and to support discussion of specific barriers and facilitators of PE Dx in their workflow. During the interview, participants viewed the PE Dx as they answered the interview questions, and they were able to navigate through the EHR playground to demonstrate what they liked or did not like about PE Dx. Each interview was audio-recorded and then transcribed by a professional transcription service. The interviews lasted an average of 27 minutes (standard deviation: 6 minutes; Range: 17–41 minutes).

We developed the interview guide (https://cqpi.wisc.edu/wp-content/uploads/sites/599/2020/05/PEDxCDS-Implementation-Interview-Guide.pdf) based on interview guides used in previous studies investigating CDS and workflow integration (Khan et al., 2016; Saleem et al., 2009), which asked questions about workflow integration such as “At what point do you interact with clinical reminders for inpatients (before, during, or after seeing the patients on each shift)?”. Based on our definition of workflow integration and the SEIPS model (Carayon et al., 2006; Carayon et al., 2014), we used probes on the work system elements to elicit information about how the PE Dx interacts with other work system elements.

2.4. Data analysis

To analyze the interview data, we used deductive content analysis (Elo & Kyngas, 2008) guided by the work system model (Carayon, 2009; Smith & Carayon-Sainfort, 1989). We created a codebook, which outlined the objective of the coding, the coding framework, and definitions of the codes. Two researchers independently coded a randomly selected transcript for barriers and facilitators, the work system elements, and for workflow integration according to our proposed definition; the researchers then met to discuss the coding, challenges faced, and modifications needed in the codebook. We refined our coding process and tested the new process by re-coding the previously coded interview transcript as well as an additional interview transcript, which was randomly selected from the alternative PE Dx user group. We compared the coding of the two interviews, which helped to further refine the codebook. Next, one researcher coded a randomly selected transcript in Dedoose©, a qualitative data analysis software, and a second researcher reviewed the coding and identified any excerpts in which they disagreed with the coding. The two researchers met to discuss and resolve the coding differences. We repeated this process for an additional transcript. One researcher then coded all remaining transcripts in Dedoose©, interacting with the second researcher when questions arose. All excerpts were exported from Dedoose© into Microsoft Excel for further analysis.

In Excel, we reviewed all of the excerpts coded for workflow integration to ensure that they fit our definition of workflow integration. Using the constant comparative method (Boeije, 2002; Glaser, 1965), we grouped the workflow integration excerpts in an inductive manner (Elo & Kyngas, 2008). First, we wrote short descriptions for each of the excerpts coded as workflow integration. We created a piece of paper for each workflow integration excerpt, which included the excerpt number, the description of the excerpt, and the actual excerpt text. One researcher then reviewed each paper document one by one. The researcher compared each document to another and began creating preliminary groups of the excerpts based on similarities that emerged in the data. The researcher continued grouping the paper documents, comparing the current groups to remaining data until all the data were grouped. The preliminary excerpt groups were then moved to an online collaboration software, MURAL. In MURAL, two researchers iteratively reviewed and refined the groups until consensus was reached on the final groups (henceforth referred to as “components” of workflow integration).

Once we finalized the workflow integration components, we specified the dimensions of workflow integration (i.e. time, flow, scope, level) for each component. Throughout the process of reviewing and refining the components of workflow integration, we continuously reviewed the data in light of the conceptual model of workflow integration; this process of going back and forth between the data and the conceptual model helped to clarify, specify, and refine the conceptual model.

3. Results

Out of the 12 interview transcripts, we identified a total of 270 excerpts related to barriers and facilitators of PE Dx. We coded 134 out of the 270 excerpts for workflow integration, meaning that the excerpt specifically described (1) the temporal nature of work and (2) any of the other dimensions of workflow integration (i.e. flow, scope, and level). Table 2 provides a description of each dimension of workflow integration.

3.1. Components of workflow integration

The 134 excerpts were grouped into 25 components of workflow integration. We described 20 of the components according to the 4 dimensions of workflow integration (time, flow, scope, and level); see table 3 for the components of workflow integration, a description of each component, and the corresponding dimensions of workflow integration. The remaining five components are about outcomes of workflow integration; see table 4 for a description of these components and their corresponding positive and negative outcomes. In the remaining results, we use UPPER CASE to denote a dimension of workflow integration (TIME, FLOW, SCOPE, LEVEL) and italics to denote a sub-dimension.

Table 3:

Components of workflow integration of PE Dx CDS

#	Components of workflow integration (# of interviewees)	Description (− indicates a barrier to workflow integration; + indicates a facilitator to workflow integration)	Dimensions of workflow integration
			TIME	FLOW	SCOPE	LEVEL
1	Combination of Wells’ and PERC risk scoring algorithms (5 physicians – 3 attendings, 2 residents)	(−) PE Dx is not integrated in the workflow because the CDS is designed to be used in a sequence with Wells’ first and then PERC second. In reality, physicians may want to use both Wells’ and PERC together at the same time or in some instances they want to skip Wells’ and go straight to PERC.	• Sequential • In parallel	• Tasks	• Within ED visit	• Individual
2	PE Dx is more useful in CareStart, early on in patient visit (1 physician − attending)	(+) PE Dx is integrated in the workflow early on in the patient visit when a physician is working in CareStart. The physician places more CTs for PE when seeing patients early on in the visit in CareStart.	• Timing	• Tasks	• Within ED visit	• Individual
3	Fit with workflow of attending and resident physicians working together (3 physicians - attendings)	(+) PE Dx is integrated in the workflow as it supports the work of attending and resident physicians as they are deciding what tests to order for the patient. The attending and resident discuss the patient before the resident uses the CDS or the attending and resident use the CDS together after the resident talks to the patient.	• Sequential • In parallel	• Tasks	• Within ED visit	• Team (attending + resident)
4	Fit with the resident workflow learned when rotating to other services out of the ED (4 physicians – residents)	(−) PE Dx is not integrated in the workflow of residents that they adapt to when working in other services outside of the ED. ED residents rotate into other services such as cardiology where they spend a lot of time writing notes and do not have an ED navigator. When they return to the ED, they keep their learned workflow, which does not fit with the workflow of going to the ED navigator to use PE Dx. When off-service, residents also learn to use MDCalc and use it on the phone when they are in other services intern year.	• Extended time	• Tasks • People	Not applicable	• Individual
5	Fit with interruptions in the ED (1 physician – attending)	(+) PE Dx is integrated in the workflow because it allows the physician to be interrupted and saves the work so the physician can go back to finish the PE Dx later.	• Discontinuous	• Tasks	• Within ED visit	• Individual
6	Fit with physical environment when seeing patients (1 physician – attending)	(−) PE Dx is not integrated in the workflow because it is not mobile and therefore the physician needs to be in front of the computer in order to use it. ○ The physician is not able to use the CDS while walking between patient rooms ○ The physician cannot pull up the CDS on their phone in the patient room; the computers in patient rooms are slow so it is preferable to use the phone to calculate the patient risk while in the room. PE Dx does not fit this workflow, but MDCalc does.	• Timing	• Tasks • People	• Within patient interaction	• Individual • Team (patient)
7	Integration of multiple CDS (5 physicians – 2 attendings, 3 residents)	(−) PE Dx is not integrated in the workflow because there are no other CDS like PE Dx in the EHR. The PE Dx is designed well but because it is the only CDS of its kind in the EHR, it does not fit with the other tools that the physician will use while making diagnostic decisions for the patient. The physician prefers to use MDCalc because the physician can look up multiple risk scores at the same time such as for PE, pneumonia, etc. As more CDS like PE Dx are built in the EHR, PE Dx will better fit the broader workflow of the physician and will become increasingly integrated into physician workflow.	• In parallel	• Tools	• Within ED visit	• Individual
8	Placing orders via PE Dx versus placing all the other orders (2 physicians – attendings)	(+) PE Dx is integrated in the workflow because it is easy for a physician to place the D-dimer order separately from the other orders in the PE Dx.	• Sequential	• Tasks	• Within ED visit	• Individual
		(−) PE Dx is not integrated in the workflow because physicians prefer to place all of their orders for the patient at one time, not only the orders for diagnosing PE. PE Dx supports the D-dimer/CT order but does not fit the workflow of placing all of the orders together. If the physicians places the D-dimer order with PE Dx separately from the other orders, it may interfere with nursing workflow of drawing blood.	• In parallel	• Tasks	• Within ED visit	• Individual • Team (nurse)
9	Supporting next steps (2 physicians – residents)	(+) PE Dx is integrated in the workflow because it supports the appropriate clinical pathway and therefore, does not require a PERC to be calculated if the Wells’ score is moderate or high. In these cases, the PE Dx goes straight to the next step by supporting the physician to place the order.	• Sequential	• Tasks: supports next step	• Within ED visit	• Individual
		(+) PE Dx could be expanded to integrate in the workflow by supporting other steps in the PE diagnostic process including activating the PE Response Team based on the results of the CT scan.	• Sequential	• Tasks: supports next step	• Within ED visit	• Team (PE response team)
10	Integration in the EHR (5 physicians – 1 attending, 4 residents)	(+) PE Dx is integrated in the workflow because it is embedded within the EHR; therefore, the physician can pull the documentation from the CDS directly into the note and they do not need to go to an internet browser for the risk calculator.	• Sequential	• Tasks • Tools	• Within ED visit	• Individual
		(−) PE Dx is not integrated in the workflow within the EHR because in order to use the PE Dx, the physician needs to exit out of their task in the EHR and go to another spot in the chart to use PE Dx. The physician, therefore, cannot access the CDS at the same time as they are placing orders.	• In parallel	• Tasks • Tools	• Within ED visit	• Individual
11	Alternative locations for PE Dx in the EHR (2 physicians – 1 attending, 1 resident)	(−) PE Dx is not integrated in the workflow because it is not in a location within the EHR that is convenient for the workflow. The CDS would better integrate in the workflow if it was placed at the top of Epic or in the workups page close to orders.	• Timing	• Tasks: in a location near preceding/succeeding task	• Within ED visit	• Individual
12	Physician is never in ED navigator so they did not know about tool or do not use (4 physicians – 1 attending, 3 residents)	(−) PE Dx is not integrated in the workflow because physicians never use the ED navigator where the PE Dx is located. Because physicians are never in the ED navigator, they either do not know PE Dx is available or it is never used because it is inconvenient to get to.	• Timing	• Tasks: in a location near preceding/succeeding task	• Within ED visit	• Individual
13	Location of PE Dx in ED navigator (8 physicians – 5 attendings, 3 residents)	(+) PE Dx is integrated in the workflow because the ED navigator is easy to get to within the EHR and can be accessed in one click. The ED navigator is close to where physicians are starting their notes and it is a common area the physician will be working in.	• Timing	• Tasks	• Within ED visit	• Individual
		(−) PE Dx is not integrated in the workflow because physicians do not use the ED navigator except when they are working in CareStart. When in the regular ED, physicians do not use the ED navigator so the location in the EHR does not support the workflow.	• Timing	• Tasks	• Within ED visit	• Individual
		(−) PE Dx is not integrated in the workflow because of its location in the EHR in the ED navigator. The PE Dx is not near other tasks that physicians are working on when they need the decision support. Before Epic was upgraded, physicians used to work their way down the ED navigator to look at notes, vitals, orders, use the PE Dx and write notes. After the Epic upgrade, PE Dx is not embedded in this workflow.	• Sequential	• Tasks: in a location near preceding/succeeding task	• Within ED visit	• Individual
		(−) PE Dx is not integrated in the workflow because physicians would not go to the ED navigator until the end of the ED visit when they are ready to write their note.	• Timing	• Tasks	• Within ED visit	• Individual
14	Integration with notes (4 physicians – 1 attending, 3 residents)	(−) PE Dx is not integrated in the workflow because it is not accessible within the notes. To use PE Dx, the physician needs to exit out of their notes and go to the ED navigator. PE Dx would better integrate in the workflow if it could be used directly within the notes. This would enable the physician to see their notes as they are filling out the CDS.	• In parallel	• Tasks	• Within ED visit	• Individual
15	Integration with orders (7 physicians – 5 attendings, 2 residents)	(+) PE Dx is integrated in the workflow because a physician uses it to support their decision immediately before placing an order. If the physician has already placed the order, they can use PE Dx to confirm the decision and dismiss the option in the CDS to place the order.	• Sequential	• Tasks	• Within ED visit	• Individual
		(−) PE Dx is not integrated in the workflow because it is difficult to use in the patient room while the physician is placing orders and talking with the patient. The computers in the patient rooms are slow, therefore physicians do not go to other places in the EHR (e.g. ED navigator) while placing the orders. The CDS does not integrate in this workflow.	• In parallel	• Tasks	• Within patient interaction	• Team (patient)
		(−) PE Dx is not integrated in the workflow because it is not accessible within the ordering screen. ○ The physician wants the CDS when they are in the ordering screen, deciding what orders to place. The CDS would better integrate in the workflow if it was easily accessible/available at this point in the workflow. If the physician wants to use the PE Dx, they need to close out of the orders and go to the ‘ED navigator’. If PE Dx was available in the orders, they could continue their work without interruption. ○ Physicians do not go to the ED navigator until the end of the ED visit to write notes. If the PE Dx was integrated in the orders, it would better fit the workflow of physicians placing the orders early on in the patients ED visit.	• In parallel	• Tasks: in a location near preceding/succeeding task	• Within ED visit	• Individual
16	Fit with temporal flow of patient interaction (7 physicians – 4 attendings, 3 residents)	(+) PE Dx is integrated in the workflow because it helps the physician remember the questions to ask a patient. The physician will start a note, talk with the patient, and then use the PE Dx after talking with the patient.	• Sequential	• Tasks • People	• Within patient interaction • Within ED visit	• Individual • Team (patient)
		(+) PE Dx is integrated in the workflow because after talking with the patient, the physician always returns to the computers and goes into the EHR. The PE Dx fits in this workflow. If the physician is unsure on what to order, they will go to PE Dx.	• Sequential	• Tasks • People	• Within patient interaction • Within ED visit	• Individual • Team (patient)
		(−) PE Dx is not integrated in the workflow because after the physician talks with the patient, they return to the computer, and go directly to placing orders. Since the PE Dx is not within the ordering screen/near the ordering screen, it does not fit in the physician workflow.	• Sequential	• Tasks • People	• Within patient interaction • Within ED visit	• Individual • Team (patient)
		(−) PE Dx is not integrated in the workflow because the physician is usually listening to the patient presentation, deciding what orders to put in, looking at what the CareStart physician has done. PE Dx does not support this workflow.	• In parallel	• Tasks	• Within patient interaction • Within ED visit	• Individual • Team (patient)
		(−) PE Dx is not integrated in the workflow in cases where the physician forgets to ask a patient a question (e.g. do they have hemoptysis?) before using the tool. In this case, the physician would need to exit the PE Dx, leave the physician computer station and go back into the patient room to ask the question before completing the tool. The physician typically completes their documentation in the physician computer work station not in the patient room.	• Timing	• Tasks • People	• Within patient interaction • Within ED visit	• Individual • Team (patient)
17	Timing of documentation from PE Dx versus other documentation (5 physicians – 2 attendings, 3 residents)	(+) PE Dx is integrated in the workflow after the physician reviews the patient orders, vitals, and history. The physician uses PE Dx before going into the note. PE Dx is integrated in the workflow because the documentation from the PE Dx goes directly into the physician note, therefore the physician does not need to go back later and re-document what was calculated in MDCalc.	• Sequential	• Tasks: supports next task	• Within ED visit	• Individual
		(−) PE Dx is not integrated in the workflow because the physician goes back to edit the patient note throughout the ED visit. ○ The PE Dx documentation should be able to be edited throughout the patient’s ED visit. • The PE Dx documentation support does not always work if the physician has already started a note before using the tool. This can cause the PE Dx to overwrite the old documentation with the PE Dx documentation. In order to get the documentation text in, the physician needs to refresh their note, which can cause a problem if the physician forgets to refresh their note and the PE Dx documentation never populates in.	• Discontinuous	• Tasks	• Within ED visit	• Individual
18	Trigger vs no trigger (9 physicians – 5 attendings, 4 residents)	(+) PE Dx is integrated in the workflow as there are not any hard stops or repetitive typing to use it. PE Dx does not pop-up and interrupt the workflow.	• Timing	• Information: no trigger	• Within ED visit	• Individual
		(−) PE Dx is not integrated in the workflow because it is not a pop-up or alert so sometimes the physician forgets to use the tool. It may be more annoying, consequently physicians would use the tool more if it was an alert.	• Timing	• Information: no trigger	• Within ED visit	• Individual
		(−) PE Dx would be more integrated in the workflow if it was optimized to support the order decision. Physicians want the PE Dx in the background checking patient vitals, chief complaint, and risk factors and then triggering an alert that suggests the appropriate order based on those factors. PE Dx could also work if the CDS came up when the physician placed an order for a CT scan. The triggers for prompting decision support would be challenging so they do not overtrigger.	• Timing	• Information: trigger • Tasks: support next task	• Within ED visit	• Individual
19	Timing of trigger in the workflow (4 physicians – 3 attendings, 1 resident)	(+) PE Dx is integrated in the workflow because it is not a BPA and therefore it does not come up at the wrong time and interrupt the work	• Timing	• Tasks	• Within ED visit	• Individual
		(−) If the PE Dx triggered in the workflow, the timing for the trigger is difficult. Before placing the D-dimer, it may be too late in the workflow because the physician has already decided on the order to place. Before the CT scan will be annoying for really high-risk patients that do not require decision support. Forcing physicians to use decision support before placing the orders will be frustrating especially for experienced physicians. CDS could also be a pop-up that comes up in the physician note.	• Timing	Information: point in time for trigger • Tasks	• Within ED visit	• Individual
20	Flag for relevant information (1 physician – attending)	(−) PE Dx would be better integrated in the workflow if there was a prompt/cue for specific information found in the patient chart (e.g. prior DVT). This would take out the initial screening step that physicians need to do and prompt them to look further into potential risk factors.	• Timing	• Information	• Within ED visit	• Individual

Table 4:

Components of workflow integration related to clinician outcomes

#	Components of workflow integration (# of interviewees)	Description (− indicates a barrier to workflow integration; + indicates a facilitator to workflow integration)	Clinician outcomes
			Positive	Negative
21	Hard to find PE Dx and remember to use (4 physicians – residents)	(−) PE Dx is not integrated in the workflow so the physician forgets to use the tool and/or has a hard time remembering to use the tool. It is challenging to find the CDS in the EHR the first time using it which is a hurdle to remembering to use it. PE Dx would be more integrated in the workflow if it was not in the ED navigator because that part of the EHR is rarely visited, and if it is used, it is not until the end of the ED visit when a physician is ready to write the notes.		• Lack of adoption/use
22	Physician was not initially aware of PE Dx (3 physicians – residents)	(−) Some physicians were not initially aware that PE Dx existed because it is in the ED navigator, which they do not often use. PE Dx is not integrated in the workflow because the physician does not think to use the PE Dx in their workflow. The physician goes to MDCalc or straight into the notes and documents their medical decision making without thinking to use PE Dx. When attending physicians are at UW, they do not think to use PE Dx because the residents are the ones placing most orders and writing the notes.		Lack of use
23	Impact on efficiency, time to complete tasks and workload (7 physicians - 3 attendings, 4 residents)	(+) PE Dx is integrated in the workflow because it is easy to access with one click within Epic. PE Dx is easier than MDCalc because you do not need to open a web browser and type in looking for the risk calculators. If the tool was not as easy to use and that had additional prompts, physicians would not use it.	• Reduced time to complete tasks • Reduced workload
		(−) In order for the physician to change their workflow and switch to using PE Dx, it needs to offer more value and efficiency to integrate in the workflow. ○ PE Dx is not integrated in the workflow as it takes extra clicks to go from the orders to the ED navigator to use PE Dx. ○ Time pressure makes it less likely that the physician will access PE Dx. ○ MDCalc is easier/faster than PE Dx since physicians usually have a web browser open during the shift anyway. ○ It takes the physician extra clicks/time/steps to exit out of the orders/documentation, find the PE Dx within the ED navigator and the return to the orders/documentation to complete their work. ○ Although the PE Dx supports documentation, using PE Dx is a source of extra clicks that the physician does not need in order to determine the order.		• Increased time to complete tasks • Increased workload • Lack of adoption/use
24	Impact on workflow - Adaptation/lack of adaptation to new workflow (6 physicians - 4 attendings, 2 residents)	(+) PE Dx is integrated in the workflow over time after the physician adapts to using it. As more CDS like PE Dx are available, PE Dx will become a part of the habit to use.	• Adaptation/adoption
		(−) When there is time pressure, residents revert to the workflow that is quickest and that they are most comfortable with. Physicians are less likely to use PE Dx and revert to their old workflow.		• Lack of adoption/use
		(−) Physician used PE Dx 2–3 times and then gradually forgot and reverted to old workflow		• Lack of adoption/use
		(−) Physician lack of awareness/habit of using PE Dx is a barrier		• Lack of use
		(−) Physician sometimes does not think to use PE Dx until after placing the order. PE Dx is not integrated in the natural work process		• Lack of adoption/use
		(−) PE Dx is not integrated in the workflow because the physician prefers to dictate or type the note rather than using the documentation support from the CDS.		• Lack of adoption/use
25	Workaround for placing orders via PE Dx versus placing all the other orders (1 physician – attending)	(−) PE Dx is not integrated in the workflow because physicians prefer to place all of their orders for the patient at one time, not just the orders for diagnosing PE. PE Dx supports the D-dimer/CT order but does not fit the workflow of placing all orders together. The physician uses a workaround by using PE Dx and then placing all orders or placing all orders and then using PE Dx to confirm their decision on the order.		• Workaround

3.1.1. TIME

3.1.1.1. Sequential

We identified 9 components of sequential time. For instance, the PE Dx is designed to be used in a sequence: first, one risk scoring algorithm is used (i.e. Wells’), followed by a second risk scoring algorithm (i.e. PERC), depending on the results of the first algorithm (component #1 in table 3). This sequence of risk scoring algorithms is in line with hospital policy and recommended guidelines (Raja et al., 2015) for determining the appropriate diagnostic pathway for patients with suspected PE. In our interviews, we identified a barrier relating to the physicians’ sequence of tasks: a physician may want to skip Wells’ and go directly to PERC or do the PERC first and then use Wells’ afterward rather than always following the sequence of Wells’ followed by PERC. An ED resident explained “sometimes...I’m looking at the patient, and I’m already thinking low risk, without having used the Wells’ criteria explicitly. And it’s actually fairly common... to say, okay, this patient is low risk for PE. Can we PERC them out without already calculating a Wells’ less than two?”. We found the technology does not always integrate in the workflow of physicians because of the sequential flow of tasks.

3.1.1.2. In parallel

We identified 8 components relating to the time sub-dimension in parallel. For instance, in our data, we identified a barrier relating to how physicians place orders in parallel (component #8 in table 3). The workflow of ED physicians is to place all of their orders for a patient at the same time, from a large ordering screen where multiple orders can be selected and submitted together. An attending physician explained: “I generally do order all my tests at one time, and that’s why I’ll either do the PE diagnosis, you can flip back and forth in the order screens. You can order some things, and then do it [the CDS]. Or I’ll do the PE diagnosis, then order all my tests.” While our CDS supported the ordering of a D-dimer or CT scan based on the patient’s risk score, this does not integrate well with the physician workflow of placing all of the patient’s orders, such as urine sample, CT scan, and bloodwork, at the same time. This is an example of the in parallel sub-dimension of TIME.

3.1.1.3. Discontinuous

We identified 2 components relating to the discontinuous flow of tasks. For example, we identified a facilitator to workflow integration due to our CDS allowing a physician to be interrupted in the middle of completing the CDS (component #5 in table 3). The ED is a chaotic environment with frequent interruptions and distractions; our CDS is integrated in this discontinuous flow of tasks because it supports interruptions by saving the physician’s work if they get interrupted while filling out the CDS. Another example of the discontinuous flow of tasks is documentation in the ED. A physician starts and stops their documentation for a patient throughout the ED visit (component #17 in table 3). For instance, an ED resident stated, “I’d want to be able to edit it in the actual note because if I have to go back later and change it, then I’d have to go back into the decision support tool instead of just my note...lots of patients in the emergency department, we are like in and out of the note multiple times ”. We found that our CDS was not integrated in this discontinuous workflow. If the physician had already started their note before using our CDS, the CDS would overwrite the previous documentation of the physician rather than add it to the note.

3.1.1.4. Timing

We identified 9 components for the TIME sub-dimension, timing. Several of the components were related to the timing of a trigger in the workflow (components #18 and #19 in table 3). For instance, one physician described the timing of a trigger in the workflow: “I don’t know that you want that [PE Dx] to pop-up every time somebody orders a D-dimer, but potentially, if you click D-dimer, and then the PE diagnosis tool popped up, but...you’ve already decided you ‘re ordering D-dimer, so you already decided your Wells’ score was medium risk. So that’s probably not the best time either”. We also identified several other components not related to triggers, such as the component “PE Dx is more useful in CareStart, early on in patient visit” (component #2 in table 3). When a patient arrives in the ED before they are seen in the main ED, they are first sent to a triage area called “CareStart”. In CareStart, the patient is seen by a physician and advanced practice providers (APP) and orders can be placed if needed. We found that our CDS was integrated in the workflow of physicians working in CareStart because the physician is seeing the patient early on in the ED visit before any orders have been placed. The CDS being used at this point in time in the ED visit fits with the physician workflow. Timing is an important part of workflow integration.

3.1.1.5. Extended time

We identified one component relating to the sub-dimension extended time (component #4 in table 3). Extended time represents how the technology fits (or does not fit) in the clinical workflow over a long period of time, such as over days, weeks, or months; this is different than the dimension of scope, which is specifically focused on the portion of the patient journey in which the technology is integrated. During residency, the ED residents rotate to other services including the Intensive Care Unit (ICU) and cardiology, which helps them to understand the services where their patient may transition following an ED visit. When the residents are on these other services, they learn a new workflow in use in that service. For instance, in the ICU, residents spend a majority of their time in the “notes” section of the EHR rather than in the “ED Navigator”, the section of the EHR in which the CDS is located. When residents return to the ED after rotating in other services, they continue to use the “notes” section of the EHR rather than using the “ED Navigator”, resulting in the CDS no longer integrating in their ED workflow. For example, an ED resident said: “I think the reason I use the notes tab versus the ED Navigator is when you’re off service, there isn’t an ED Navigator...And really all you ever do on those off-service rotations is write notes…So I think it’s just been kind of my workflow that….my best guess as to why I use the note tab versus the ED Navigator.”. This demonstrates how the sub-dimension, extended time, can be related to the workflow integration of a technology. While the CDS fit resident workflow initially when they are in the ED, the CDS did not fit their workflow over an extended period of TIME (e.g. after they rotated to other services).

3.1.2. FLOW

3.1.2.1. Tasks

Almost all of the components of workflow integration related to the flow of tasks. We specified that in order to integrate in the flow of tasks, a technology should be accessible in a location near the preceding and succeeding tasks; this enables the physician to easily flow from one task to the next. In our data, we identified a barrier to workflow integration of our CDS due to the location of the CDS within the EHR. The CDS is located in a section of the chart called the “ED navigator”, which was previously used by physicians throughout the ED visit to place orders, write notes, and view the patient’s medications and medical history (components #12 and #13 in table 3). For example, a physician explained: “it worked well when we lived in ED Navigator, and our orders were driven there... there used to be an orders thing right around here...I would start here and be like, okay what are their allergies? what’s their history?... I would just be working down the navigator, and so it was nice, because it was right there ”.

Following the implementation of the CDS, the hospital upgraded the EHR, which changed the location of many tools (e.g. placing orders) and moved them out of the ED navigator. Because of this, ED physicians no longer used the ED navigator. In order to use our CDS, the physician would need to leave the section of the EHR they are in, click through the EHR to find the ED navigator, and then use the CDS. An ED resident explained: “I never actually use this ED Navigator for anything. I would be going into the ED Navigator specifically to do this [PE Dx].” The location of the CDS in the ED navigator was no longer near the other tasks the physician was doing or near the tools they were using, which hindered the integration of the CDS in the workflow. An ED attending stated “cognitively, it’s in a different place than where I’m thinking about making this decision...So, to leave this place [the orders tab] where I’m getting things done and then go over to the ED Navigator...now you’ve interrupted what I’m doing. So, I don’t like that it’s in the wrong place cognitively”. To facilitate workflow integration of technology, the technology should be accessible near the preceding and succeeding tasks. To integrate in the workflow, a technology should also support the next step in the process. We identified a facilitator because our CDS supported the physician placing an order and automatically documented the patient’s risk score based on the recommendation from the CDS (component #9 in table 3).

3.1.2.3. People

We identified 5 components related to the flow of people, which includes the movement of people (e.g. patients, clinicians) through space and time. For instance, we identified a barrier to workflow integration relating to the flow of people, specifically, the flow of the physician throughout the ED. When a physician is moving between patient rooms in the ED, they like to pull up the CDS on their phone in order to calculate a patient’s risk for PE as they walk (component #6 in table 3). Our CDS did not fit with this flow of people because it is not accessible on the physician’s phone, and therefore needs to be used while at the computer. An ED attending explained this workflow: “the advantage of my phone is I can do that while I’m walking back from the patient’s room, when I’m walking to the patient’s room, when I’m, you know, doing whatever. This [PE Dx] requires me to be sitting in front of the computer...and as you’ve seen by being down there, we’re pretty much moving around continuously.” This also demonstrates the TIME dimension, timing, in which the CDS was not available at the point in time it was needed, i.e. while walking between patient rooms.

3.1.2.4. Information & tools/technologies

We identified two components related to the flow of information (i.e. use of triggers), and two components related to the flow of tools/technologies. These components described how the PE Dx cannot be used at the same time as physicians are using other CDS (e.g. for pneumonia) or while placing orders (components #7 and #10 in table 3). For example, an ED resident stated: “it’s probably hampered by the fact that it’s, it is a good tool, and it would be good to have this tool for everything, but we don’t have it for everything. We just have it for the one thing.”.

3.1.3. SCOPE

We describe 3 SCOPES of the patient journey that a technology can be integrated within: (1) within a patient interaction, (2) within an ED visit, and (3) before, within, and after ED visit. We identified 3 components within a patient interaction and 18 components within the ED visit. The components on within a patient interaction described the workflow of physicians talking with patients to gather information for the CDS. We did not identify any components with the SCOPE, before, within, and after ED visit.

3.1.4. LEVEL

We identified 19 components of the individual workflow LEVEL, and 6 components of workflow integration relating to team workflow. This included the team workflow of physicians and patients, physicians and nurses, ED physicians and clinicians in the PE Response Team (PERT), and attending and resident physicians. An attending physician described this team workflow, “by the time I saw the patient, the resident had already seen the patient. They had a bunch of bloodwork and stuff done. I had talked to the resident. The resident was, you know, debating on what to do about, in terms of PE workup, and in my mind, I was like, you know what, I think this person is going to meet the D-dimer and then CT, but I wasn’t 100% sure. And so I was like, all right, let’s just go work, go through the flow ” (component #3 in table 3). In this example, the resident and the attending worked through the CDS together to make a final, collaborative decision on what test to order for the patient.

3.2. Conceptual model

We developed an initial conceptual model of workflow integration based on our proposed definition and dimensions of workflow integration (see section 1.3). As we analyzed the data on barriers and facilitators to workflow integration of PE Dx, we iteratively went back and forth between the initial conceptual model and the dimensions of workflow integration (see table 2) and our data. This iterative process helped us to identify areas to specify and expand in the conceptual model in order to fully capture the concept of workflow integration of health IT. Figure 1 depicts the final conceptual model of workflow integration.

Figure 1:

Conceptual model of workflow integration

In the conceptual model on the left (figure 1), we depict the new technology (i.e. health IT) embedded within the existing work system and interacting with all system elements. Workflow integration emerges from the interactions of the new technology in the existing work system. The center of the conceptual model details 4 components of workflow integration: TIME, FLOW, SCOPE, and LEVEL. TIME includes 5 sub-dimensions of different aspects of time that influence how a technology integrates in a workflow: sequential, in parallel, discontinuous, timing, and extended time. FLOW represents the flow of tasks, people, information, and tools/technologies. Based on the concept of “space” as described by Carayon et al. (2012), the third dimension of workflow integration is SCOPE of the patient journey. The patient journey is defined as “the temporal series of work systems interacting over time” (Carayon, Wooldridge, Hoonakker, Hundt, & Kelly, 2020). SCOPE represents what portion of the patient journey a technology is integrated within, for instance, within a patient interaction, within an ED (or another type of) visit, and before, within, and after an ED visit. LEVEL represents whose workflow the technology is integrated within - the individual, team, or organization. Workflow integration results in outcomes. The right side of the conceptual model outlines positive and negative outcomes as a result of workflow integration (or lack of workflow integration).

4. Discussion

In this research, we propose a definition and conceptual model of workflow integration that builds on previous research on health IT usability (Marcilly et al., 2015; Scapin & Bastien, 1997), workflow (Carayon et al., 2012), and the work system model (Carayon, 2009; Smith & Carayon-Sainfort, 1989). We propose that workflow integration involves the entire work system and interactions between the person, tools and technologies (including health IT), tasks, physical environment, and organization. This expands on the work of Marcilly et al. (2015) who described workflow integration as relating to the interaction between the person-technology-task work system elements. Using data gathered through interviews with 12 ED physicians, we identified 134 coded data elements of barriers and facilitators to workflow integration of the PE Dx, which were combined into 25 components of workflow integration of the CDS (including 5 components related to the outcomes of the workflow integration). We described 20 of the components according to the 4 dimensions of workflow integration proposed in our conceptual model (see table 3); the other five components related to outcomes of workflow integration (see table 4).

Previous studies of workflow integration of health IT have focused on a couple of pieces of workflow integration such as the TIME sub-dimensions, sequential and timing, the FLOW of tasks and information (e.g. triggers), and the workflow of an individual (i.e. LEVEL). Our definition and conceptual model depict a much more complex view of workflow integration. The concept of workflow integration of health IT includes multiple dimensions and layers; our data support this multi-dimensional view of workflow integration of health IT (see figure 1). In line with the work of Scapin and Bastien (1997) and Marcilly et al. (2015), we propose that workflow integration is one part of the broader usability of health IT (see table 1). We expanded on the usability heuristic ‘compatibility’ of Scapin and Bastien (1997) by specifying the multiple dimensions and layers of workflow integration. In order to improve patient safety and reduce the negative impact on clinicians (i.e. burnout) (National Academies of Sciences, 2020), it is important to consider workflow integration of the technology as a part of the broader goal of developing usable health IT.

4.1. Temporal nature of tasks and work

We expand on previous research by specifying and defining 5 different sub-dimensions of time that influence workflow integration of health IT: sequential, in parallel, discontinuous, timing, and extended time. Most studies on workflow integration of health IT have focused on the time sub-dimensions sequential and timing (Mann et al., 2011; Marcilly et al., 2015; Press et al., 2015; Saleem et al., 2011). For instance, Khan et al. (2016) developed workflow diagrams that depicted the sequential flow of tasks and the fit of a CDS within that sequential flow over time. Press et al. (2015) conducted usability testing to determine the optimal timing for a trigger in the workflow. Our conceptual model expands on and clarifies the different dimensions of time that can influence workflow integration of health IT, and our data about PE Dx supported these proposed sub-dimensions of time. In the design of health IT, it is important to not only consider the sequential flow and timing of the work activities, but also consider workflows that occur in parallel, discontinuously, and over extended time.

While previous studies on workflow integration of health IT have described the flow of tasks (Mann et al., 2011; Marcilly et al., 2015; Press et al., 2015; Saleem et al., 2011), we expand on this by providing 2 specific considerations for the flow of tasks. First, the technology should be in a location near the preceding and succeeding tasks. This concept relates to the dimension “navigation” outlined in the Workflow Integration Survey of Flanagan et al. (2011). Second, we specify that to integrate in the flow of tasks, a technology should support the next step in the process (e.g. placing an order based on the CDS recommendation, which PE Dx does). In their Workflow Integration Survey, Flanagan et al. (2011) describe a similar concept, “functionality”, which specifies that the technology should help the user perform the tasks needed during patient encounters. We expand on this by specifying that to integrate in the workflow, the technology should support the clinician’s next steps, which may (or may not) take place outside of a patient encounter.

4.2. Triggers of the technology in the workflow

We expand previous research by clarifying the concept of triggers. Triggers are commonly used in the form of an alert as a reminder to use a technology such as a CDS. Press et al. (2015) described workflow integration as “consideration of what timing in the patient interaction the CDS is triggered” (pg. 2). Previous CDS implementations have struggled to find the appropriate time in the workflow (i.e. timing) for a trigger (Mann et al., 2011; Press et al., 2015; Tan et al., 2020), resulting in low acceptance of technologies and alert fatigue (Van der Sijs, Aarts, Vulto, & Berg, 2006). Triggers represent one aspect of the flow of information in our concept of workflow integration of health IT. In designing health IT such as CDS, it is important to decide whether a trigger (i.e. an alert) will be used as a reminder to the user. If there is rationale for the use of a trigger, the point in time in the workflow for the trigger is an important (and challenging) consideration. Designers of health IT should carefully consider whether, how and when to implement a trigger in clinician workflow.

In addition to triggers, we describe several components of timing that were related to other factors. For instance, we uncovered a barrier to using the PE Dx due to the timing of using the CDS within the physical environment: a physician preferred to use the CDS while walking between patient rooms; this does not fit with the fact that our CDS is not available on a phone and cannot be used at the point in time when the physician is in between rooms. This demonstrates another work system element of timing, related to the physical environment, that influences workflow integration of health IT. Timing is an important part of workflow integration, and it includes consideration of triggers as well as other factors not previously described in the literature.

4.3. Scope and level of workflow integration

Designers of health IT should consider the SCOPE of the patient journey in which the technology will be integrated within as well as whose workflow will be involved (LEVEL). As described in the SEIPS model (Carayon et al., 2006; Carayon et al., 2014), the process represents the interaction between each of the work system elements over time. Workflow integration of health IT relies on a technology’s integration in various parts of the process (i.e. SCOPE). In their study of a CDS for colorectal cancer screening, Saleem et al. (2011) describe the scope, before, within, and after visit, by stating that the CDS should integrate results from outside the primary care clinic (i.e. before the visit). Although we did not identify any instances of the SCOPE before, within, and after ED visit in our data, we included this in our conceptual model as it will likely be relevant to workflow integration of a technology as a patient moves from one work system to another; for instance, throughout care transitions in the patient journey (Carayon et al., 2020). It is important to note that although the conceptual model specifies SCOPE as relating to a patient’s ED visit, the SCOPE of patient journey can be modified to apply to any type of patient interaction with the health system, for example, within primary care visit rather than the SCOPE within ED visit.

We show that workflow integration not only relies on the technology’s fit with the workflow of an individual clinician, but also the fit of the technology in team and organizational workflows; this relates to the LEVEL dimension. Previous studies describing workflow integration have focused on the integration of a technology in the workflow of a single clinician (Mann et al., 2011; Marcilly et al., 2015; Press et al., 2015; Saleem et al., 2011). As health care is increasingly relying on the work of teams to provide timely and safe care to patients, our findings demonstrate the importance of considering how a technology integrates into a workflow that includes multiple team members rather than solely focusing on the workflow of an individual (Carayon & Hoonakker, 2019; Li, 2016; Walker & Carayon, 2009). The technology should support the workflow of an individual who interacts with members of a team.

4.4. Implications for the design of health IT

This research has important implications for the human-centered design of health IT. Based on our conceptual model of workflow integration and the identified barriers and facilitators to workflow integration of PE Dx, we developed an initial checklist on workflow integration of health IT (see figure 2). The checklist can be used by technology designers, clinicians, and researchers to support the explicit consideration of workflow integration in the design process. In a human-centered design process, the series of questions included in the checklist can provide a framework or structure for health IT design teams to focus systematically on workflow integration. A participatory process with multiple perspectives could benefit from this checklist in the initial technology design phase as well as subsequent continuous design iterations. Future studies can continue the work of refining and evaluating this checklist as it is based on our one study.

Figure 2:

Checklist for workflow integration of health IT

4.5. Limitations and future research

One limitation of this research is that it took place in one ED of an academic health system. The results may be limited in their generalizability to other settings, including EDs in non-academic health systems. The data are focused on the design and implementation of one type of health IT, an EHR-integrated diagnostic CDS, which was not widely used at the time of data collection; the definition and conceptual model of workflow integration may need to be refined to apply to other types of health IT. Conceptual models are simplified depictions of reality; therefore, the conceptual model of workflow integration may not fully capture the variation and complexities of integrating technologies in clinical workflow. The goal of this analysis was to identify all of the possible dimensions of workflow integration. However, it is worth noting that the proposed dimensions of workflow integration were reported by a range of interviewees, i.e. between 1 and 9 (see table 3). For example, the dimension “Fit with interruptions in the ED” was discussed by 1 physician whereas “Location of PE Dx in ED navigator” was discussed by 8 physicians. Future research could further investigate the frequency and impact of the proposed dimensions of workflow integration. Despite reaching data saturation, our overall sample size of 12 remains small. Finally, future research could explore and mine EHR data to assess the use of the CDS in the ED and its impact on relevant care process and outcome measures.

This study provides several opportunities for future research. First, the conceptual model and checklist of workflow integration can be used to inform the design of health IT. Researchers should apply the checklist in the design of health IT and evaluate if the checklist improves the usability outcomes (i.e. efficiency, effectiveness, satisfaction) of health IT and consequently work system outcomes, in particular patient safety. Researchers can then identify modifications to the checklist to support consideration of workflow integration in the human-centered design process. Second, future research should apply the definition and conceptual model of workflow integration to other contexts and settings, for instance, to study patient-facing health IT or health IT used by health care teams; this may identify differences in the concept of workflow integration in settings outside of the ED and for other types of health IT. Third, the conceptual model of workflow integration could be used to inform the development of additional heuristics to consider in a heuristic evaluation; this would build off of the “compatibility” heuristic of Scapin and Bastien (1997) and Marcilly et al. (2015). Finally, in this research, we identified 5 components of workflow integration specifically related to outcomes. These components demonstrate the potential positive and negative outcomes of workflow integration of health IT. Future research could further examine the impact of workflow integration on usability outcomes (i.e., efficiency, effectiveness, satisfaction) and on clinical outcomes.

5. Conclusion

Workflow integration is essential to health IT acceptance, use, and impact on patient safety. The proposed definition, conceptual model, and checklist of workflow integration of health IT clarify the multiple dimensions of workflow integration. This conceptual model and checklist can support consideration of workflow integration during the design of health IT in order to improve usability of the technology when implemented.

Highlights:

• We propose a definition and conceptual model of workflow integration of health IT

• We identify barriers and facilitators to workflow integration of a CDS in the ED

• We describe 25 components of workflow integration of a CDS

• We propose a checklist to help consider workflow integration in health IT design