Examining Workflow in a Pediatric Emergency Department to Develop a Clinical Decision Support for an Antimicrobial Stewardship Program

Abstract

Background  Clinical decision support (CDS) embedded into the electronic health record (EHR), is a potentially powerful tool for institution of antimicrobial stewardship programs (ASPs) in emergency departments (EDs). However, design and implementation of CDS systems should be informed by the existing workflow to ensure its congruence with ED practice, which is characterized by erratic workflow, intermittent computer interactions, and variable timing of antibiotic prescription.

Objective  This article aims to characterize ED workflow for four provider types, to guide future design and implementation of an ED-based ASP using the EHR.

Methods  Workflow was systematically examined in a single, tertiary-care academic children's hospital ED. Clinicians with four roles (attending, nurse practitioner, physician assistant, resident) were observed over a 3-month period using a tablet computer-based data collection tool. Structural observations were recorded by investigators, and classified using a predetermined set of activities. Clinicians were queried regarding timing of diagnosis and disposition decision points.

Results  A total of 23 providers were observed for 90 hours. Sixty-four different activities were captured for a total of 6,060 times. Among these activities, nine were conducted at different frequency or time allocation across four roles. Moreover, we identified differences in sequential patterns across roles. Decision points, whereby clinicians then proceeded with treatment, were identified 127 times. The most common decision points identified were: (1) after/during examining or talking to patient or relative; (2) after talking to a specialist; and (3) after diagnostic test/image was resulted and discussed with patient/family.

Conclusion  The design and implementation of CDS for ASP should support clinicians in various provider roles, despite having different workflow patterns. The clinicians make their decisions about treatment at different points of overall care delivery practice; likewise, the CDS should also support decisions at different points of care.

Background and Significance

Clinical Decision Support in Emergency Departments and Clinical Workflow

Electronic health record (EHR)-based clinical decision support (CDS) systems have the potential to change clinical behavior in the emergency department (ED) setting. Decision support interventions tailored to context and end-user needs have been shown to be more likely to result in desired behavior change. 1 Although CDS has promise to impact prescribing practices, the ED setting presents unique challenges to CDS implementation and effectiveness. 2 3 Clinicians in the ED are frequently forced to make rapid decisions in a chaotic setting. Decision making is often complicated by patient turnover, multiple interruptions, the need for empiric therapy, and limitations such as a lack of perceived follow-up and medicolegal fears. 4 5 6 7 To be successful, current literature and CDS experts strongly recommend that developers assess and account for local ED processes and clinician perceptions during the creation and refinement of CDS, but prior to implementation. 8 9 10 11

Antimicrobial stewardship programs (ASPs) aim to optimize appropriate antibiotic prescribing in the clinical setting. 12 CDS can deliver antibiotic prescribing recommendations at the point-of-care, and overcome barriers unique to the ED setting. 13 14 15 Therefore, CDS is a potentially effective method to implement ED-based ASPs. However, an understanding of ED workflow prior to development of CDS for antibiotic prescribing is necessary to develop an effective and sustainable ASP intervention. Proper design and implementation of CDS can be integrated into the usual workflow without disrupting ongoing practice. 16 17 The purpose of this study was to characterize workflow for four clinician roles in a pediatric ED, to inform the future development of EHR-based CDS, as the centerpiece for an ED-based ASP.

Potential Role for CDS in Antimicrobial Stewardship Programs

Antimicrobial resistance is among the greatest threats to health care quality and pediatric public health. 18 In outpatient settings such as the ED, it is estimated that 25 to 63% of patients are prescribed inappropriate or unnecessary antibiotics, contributing to the development of antimicrobial resistance. 19 20 21 22 ASPs can improve guideline adherence for antibiotic prescribing and minimize the impact of antibiotic-resistant organisms. 23 However, ASPs are not sufficiently utilized in ED settings, owing to the erratic workflow and variety of providers unique to this setting. 24 Innovative methods are required to implement ASPs into the ED setting.

Use of health information technology is one possible method to potentially optimize ASPs in the ED. 13 14 Incorporation of health information technology and use of EHR is viewed as an essential method for future CDS interventions to mitigate antibiotic prescribing. 14 15 25 Implementation of CDS using the EHR as a vehicle to deliver recommendations has positively impacted provider antibiotic prescribing in various clinical settings. 15 25 26 27 28 29 30 Trials of CDS have demonstrated success in improving antibiotic prescribing in inpatient and office settings, though ED-based studies are limited. 25 27 28 31 32 33 34 35 36 However, the success of these CDS interventions largely depends on proper integration into the clinical workflow. 26 37 38

Methods

Setting

Workflow was systematically examined in a single, tertiary-care academic children's hospital ED that serves as regional referral center, receiving ∼73,000 visits annually. The ED consists of 40 acute care beds and is staffed by a variety of providers, including attending physicians (pediatric emergency medicine [PEM] specialists and general pediatricians), nurse practitioners (NPs), physician assistants (PAs), and resident physician trainees. At any given time, the ED is usually staffed by 2 to 3 PEM specialists, 5 advanced practice providers (NPs or PAs), and between 2 and 4 residents. The ED is grouped into three teams of providers: two teams are a combination of PEM specialists, residents, and advanced practice providers. A third team is composed of two advanced practice providers who evaluate patients of low acuity. In the system, all advanced practice providers act independently, without direct supervision. Residents present to PEM specialists only, and are fully supervised.

Sample

ED clinicians of four provider types were observed in the ED over a 5-month period (March–July 2016). Data were collected from a total of 23 participants (11 attendings, 3 residents, 5 NPs, and 4 PAs). Table 1 shows the participating providers and observation characteristics. Participants volunteered from a pool of 40 attending physicians, 30 residents (on ED rotation), and 70 advanced practice providers. We focused recruitment on the day shift, to ensure homogeneity of the data.

Table 1. Study sample.

Type of providers	Clinicians ( n )	Sessions ( n )	Time spent in the ED (h)	Time captured with observation tool (h)	Decision point questions ( n )
Attending	11	13	41	38	65
Resident	3	4	11	10	12
Nurse practitioner	5	8	24	22	32
Physician assistant	4	6	14	13	18
Total	23	31	90	83	127

Abbreviation: ED, emergency department.

Procedure

Data were collected by direct observations supported by probing questions. Workflow activities and information regarding decision points were recorded on a tablet computer, using a user-friendly software tool for collection of time and motion data, where the tablet automatically records the time each recording of an activity was entered ( Fig. 1 ). The software tool 39 was initially developed for and validated in an ambulatory care setting. We modified the tool by adding the ED setting and observer names while keeping the task list and other settings. Coding of workflow activities is outlined in Appendix A . A total of 90 hours of observations were conducted.

Fig. 1.

Data collection tool.

Appendix A.

CATG_ID	CATG_DESC
A1	Computer–Communicating
B1	Dictating
C1	Computer–Entering
C2	Computer–Login
C3	Computer–Logout
C4	Computer–Processing
C5	Computer–Printing
C6	Computer–Reading
D1	Paper–Copying/Faxing
D2	Paper–Reading reference
D3	Paper–Reading/Writing
D4	Paper–Retrieving/Accepting
D5	Paper–Sorting/Filing/Delivering
D6	Paper–Writing
E1	Phone–Answering
E2	Phone–Calling
E3	Phone–Transferring
F1	Talking–Coworker
F2	Talking–Patient
F3	Talking–With others
G1	Walking–Alone
G2	Walking–With coworker
G3	Walking–With patient
H1	Meeting
I1	Performing–Exam room preparation/Cleaning
I2	Performing–Hand sanitization
I3	Performing–Handling
I4	Performing–Measuring
I5	Performing–Medical procedure
I6	Performing–Physical exam
I7	Performing–Other
J1	Personal–Cell phone
J2	Personal–Computer
J3	Personal–Eating/drinking + Socializing/chatting
J4	Personal–Restroom
J5	Personal–Other
K1	Cell phone/iPad–Reading/typing

To better understand the critical decision-making moment, clinicians were queried in real time about the timing of diagnosis, and disposition decision points, for each patient that had a prescription or discharge instruction entered in the EHR. Decision points were based on the following question: “For [this specific patient] at what point during her/his visit did you decide how to proceed with the treatment?” A total of 127 decision points were identified for providers ( Tables 2 and 3 ). We primarily targeted diagnosis point rather than disposition with this question. However, we acknowledge that diagnosis and disposition can be very close temporally for some health care encounters. We acknowledge that consideration of both decision points is very important. Regardless, from the ASP CDS design perspective the ambiguity does not matter because in both cases (diagnosis or disposition) the function of the CDS would be the same.

Table 2. Summary of decision points.

Decisions (total frequency of responses for each decision)	Provider type	Frequency
1. After/during examining or talking to patient or relative (55)	Attending	27
	NP	17
	PA	6
	Resident	5
2. After talking to a specialist (13)	Attending	8
	NP	3
	PA	1
	Resident	1
3. After test/image is resulted and discussion with patient/family (13)	Attending	8
	NP	3
	PA	2
4. During/after discussion with a resident (5)	Attending	5
5. After administration of medication and reexamining (4)	Attending	1
	NP	2
	PA	1
6. During/after discussing with an attending (2)	Resident	2
7. After reviewing patient history and conducting exam (2)	NP	2
8. Reading through the patient chart (2)	Attending	2
9. After seeing the test results and conducting a second physical exam (2)	PA	1
	Attending	1
10. After making calculations on a Web site (1)	Attending	1
11. Talking to a respiratory therapist (1)	Attending	1
12. After going through head injury protocol; and observation protocol (1)	NP	1
13. After asthma assessments (1)	Resident	1
14. After gathering her history, conducting a physical exam, and talking to an attending (1)	PA	1
15. After obtaining a second opinion (1)	NP	1
16. After obtaining medical history and final exam (1)	NP	1
17. After referring to TBI algorithm (1)	PA	1
18. After medical procedure (1)	Attending	1
19. When hearing other attending taking the transfer phone call (1)	Attending	1
20. Prior to the patient's arrival at the ED, as well as after speaking with the patient's doctor who referred the patient (1)	Attending	1
21. After reading triage notes (1)	Resident	1
Total		110

Abbreviations: ED, emergency department; NP, nurse practitioner; PA, physician assistant; TBI, traumatic brain injury.

Table 3. Summary of cases with no specific decision.

Reasons for no decision	Provider type	Frequency
1. Complex patient so no decision during the observation period	Attending	5
	NP	2
	PA	1
	Resident	2
2. Social worker consultation	Attending	1
3. Psychiatry patient	Attending	2
	PA	3
4. Observer could not ask	Attending	1
Total		17

Abbreviations: NP, nurse practitioner; PA, physician assistant.

The observer recorded reflection statements after each of the 31 observation sessions. These statements were used to clarify both qualitative and quantitative data. Reflection statements are important to explain why there was a decision or if there is any additional important information (e.g., location) that might be important for analysis.

Decision Analysis

We used a qualitative approach to analyze decision points and a quantitative approach to analyze workflow data. Qualitative analysis of the 127 decision point responses was accomplished by summarizing the decision points within the overall care delivery process and highlighting the patterns by roles of the four types of providers. Responses by the participants were categorized using constant comparison. Key events associated with decision making were identified.

Sequential Pattern Analysis

The quantitative analysis of workflow data consisted of two parts. In the first part, the analysis was accomplished by sequential pattern mining, 40 41 a subfield in data mining that focuses on identifying frequent patterns in sequence data. Prior to applying the sequential pattern mining methods, we manipulated the workflow activity data into an appropriate format for the algorithm input. Specifically, tasks observed in the care delivery process were categorized ( Appendix A ) and transformed into a string sequence for each observation. For example, if a clinician walks alone (G1), talks to a patient (F2), and then performs a physical exam during the observation (I6), a sequence of “G1-F2-I6” for this observation is generated. To make the patterns more interpretable for clinical experts, we constrained the sequential pattern mining method to identify only adjacent events with a focus on start and end points. We ignored frequent task patterns that did not occur in a consecutive manner. To accommodate this constraint, we added a pseudo start and end point, “S1” and “S9” correspondingly, to each observation sequence.

The manipulated workflow activity data were then fed into a computerized algorithm developed by the second author (D.T.W.) to identify frequent sequence patterns. This identification was guided by the a priori algorithm, which has two parameters. The algorithm constrained a minimum support of 20%, i.e., a pattern is considered frequent when one-fifth of the observations contained this pattern, and the minimum confidence of 80%, i.e., a task is added to the frequent pattern list if this task is observed 4 out of 5 times using conditional probability. Once the frequent task sequences were identified, we visualized these patterns in a directed network graph ( Fig. 2 ) for each clinical role, with the nodes being the frequent categories and the edges being the connection in the patterns. Based on these parameters, the network graphs were laid out in a circle to facilitate interpretation. It is worth noting that the research team iteratively reviewed and interpreted the frequent patterns in the network graphs to adjust the parameters to include a reasonable amount of meaningful patterns, resulting in the threshold of minimum 20% support and the 80% confidence.

Fig. 2.

Workflow patterns for the four providers.

The second part of the quantitative analysis focused on determining whether the frequency of, and the time allocated in the tasks, were significantly different among the providers based on their clinical role. In particular, the frequency (count) and the time allocation (percentage) were aggregated based on the task categories per observation using the same mapping definition in Appendix A . Multitasking events were split into several single events and their elapsed time were evenly distributed to the single events. For example, if event A lasts 10 seconds, event B lasts 15 seconds, A and B are overlapped by 5 seconds, then the multitasking part is split equally between A and B so that A is 7.5 seconds and B is 12.5 seconds in a single tasking manner. The mean of the frequency of, and the time allocated to the categories by clinical roles, was compared using the Kruskal–Wallis test, assuming two independent samples with a nonnormal distribution. The significance level was set at 0.05.

The data were manipulated in Python 2.7 and stored in a SQLite database ( https://www.sqlite.org/ ), a relational file-based database format. The sequential pattern mining procedure and the statistical test were also conducted using Python 2.7. Specifically, the string matching procedure was implemented using the generic Python regular expression library; the Kruskal–Wallis test was implemented using the Python Numpy and Scipy libraries. The network graphs were produced using the igraph library in R-package in the R-Studio environment, an open-source interactive code editor for R.

During the study, we collected qualitative and quantitative data simultaneously. Our analysis of the qualitative data showed saturation before data collection was complete. We argue that our sample is sufficiently representative because: (1) we did not pick the participants purposefully, it was based on volunteering; (2) observers did not note a significant event during the observations that may affect the usual workflow; and (3) the sample size is consistent with similar studies.

Results

The 90-hour observations resulted in 6,060 records with 64 distinct activities. Among these activities, Five were conducted at different frequency or time allocation across the four clinical roles, including computer entering (C1) and reading (C6), talking with coworkers (F1) and walking alone (G1), and personal activities (J3) as listed in the second part of Fig. 2 . We also identified the frequency of decision points and compared the sequential patterns of the activities across the four roles.

Decision Points

During the observation period, of 127 decision point questions, in 17 questions (9 attending, 2 NPs, 4 PAs, and 2 residents), no specific decisions were made toward the treatment (see Table 3 ). A common reason for the lack of a decision point was among those are complex patients; still need further tests and consultations, and those coming to ED for mental health reasons; these patients were being screened in the ED for medical clearance. The 110 remaining decision points within the care delivery process are presented in Table 2 .

The most frequent decision point for each of the four providers was “after/during examining or talking to patient or relative” in 55 (50%) responses. Other frequent decision points were: “after talking to a specialist” and “after test/image result is resulted and discussion with patient/family” with 13 (12%) total responses for each. The second and third most frequent decision points for attending physicians and NPs were “after talking to a specialist” and “after test/image result is resulted and discussion with patient/family.” “After test/image result is up (and talking to patient/family)” was the second frequent decision point for PAs, while “during/after discussing with an attending” was the second most frequent decision point for residents. Out of 110 visits, antibiotics were prescribed in 20 cases. In 12 of those cases, the decision was made “after/during examining or talking to patient or relative.”

The reflection notes highlight how various activities were conducted in various places within the ED. For example, conversations among ED staff members took place at physician stations, hallways, or in front of electronic board. Similarly (face-to-face or phone), conversations between ED clinicians and specialists took place in various places within the ED. Although the majority of physical exams took place in patient rooms, at least in one case it took place in a triage room. Test results were accessed by clinicians through desktop computers a majority of the time; however, attendings could be aware of test results by residents in hallways.

Sequential Patterns

Fig. 2 illustrates the most frequent sequential pattern in a network graph for each clinical role. Attending physicians demonstrated the most clear-cut pattern (fewer frequent patterns in total) compared with other providers. The attendings frequently started with talking to coworkers (F1), walking alone (G1), and personal activities such as socializing (J3). They were frequently involved in computer entering (C1) and reading (C6) toward the end of the observations. Other providers, on the other hand, involved more tasks that the attendings did not frequently do, such as answering phone calls (E1), hand sanitization (I2), and performing physical exams (I6). This pattern reasonably reflects the roles these clinicians play on a care team. It is worth noting that residents and PAs were more likely to talk to patients (F2) in the beginning of the observations. They also tended to talk to their coworkers at the end of the observations (F1–>S9), which likely involved reporting to attending physicians and seeking guidance and approval. Residents exhibited the most complicated and frequent task patterns compared with other providers, meaning that residents had higher numbers of frequent patterns than others, and the sequence of these patterns had more steps and more switches between task categories.

Frequency and Time Allocation

Significant differences ( p  < 0.05) were seen in the frequency and task categories among providers ( Table 4 ). Attending physicians demonstrated a very different pattern compared with the other clinical roles. For example, they more frequently talked to coworkers (F1) and read electronic documents (C6) than other providers. Attending physicians talked to coworkers about 20 minutes longer than other providers. NPs, on the other hand, tended to allocate more time than attendings to entering tasks on the computer (22.29 vs. 8.27 seconds), but performed measures (1.0 vs. 2.0%) less frequently than attendings. Residents walked alone longer then NPs (4.68% vs. 1.86%).

Table 4. Activities that were conducted in different amounts by four roles.

Measure	Category	Roles compared (median) a		Number of observations	Median difference	p -Value
Time allocation (s)	F1: Talking–Coworker	Attending (2926.87)	NP (1140.65)	13, 8	–1786.22	0.000
		Attending (2926.87)	PA (893.22)	13, 6	–2033.65	0.002
		Attending (2926.87)	Resident (1520.75)	13, 4	–1406.12	0.000
Time allocation (%)	C1: Computer–Entering	Attending (8.27)	NP (22.29)	13, 8	14.02	0.001
	F1: Talking–Coworker	Attending (30.58)	NP (12.54)	13, 8	–18.04	0.000
		Attending (30.58)	Resident (19.02)	13, 4	–11.56	0.001
	G1: Walking–Alone	NP (1.86)	Resident (4.68)	8, 4	2.82	0.006
Frequency	C6: Computer–Reading	Attending (57.38)	NP (22.13)	13, 8	–35.25	0.000
		Attending (57.38)	PA (26.67)	13, 6	–30.71	0.000
		Attending (57.38)	Resident (30.25)	13, 4	–27.13	0.002
	E1: Phone–Answering	Attending (9.92)	NP (1.8)	13, 5	–8.12	0.001
		Attending (9.92)	PA (2.0)	13, 3	–7.92	0.001
		Attending (9.92)	Resident (3.0)	13, 4	–6.92	0.006
	E2: Phone–Calling	Attending (6.45)	PA (1.8)	11, 5	–4.65	0.009
	F1: Talking–Coworker	Attending (50.92)	NP (26.0)	13, 8	–24.92	0.000
		Attending (50.92)	PA (15.17)	13, 6	–35.75	0.001
		Attending (50.92)	Resident (29.75)	13, 4	–21.17	0.003
	I4: Performing–Measuring	Attending (2.0)	NP (1.0)	3, 2	–1	0.000
	I5: Performing–Medical procedure	Attending (1.14)	NP (2.0)	7, 2	0.86	0.001
Time allocation (s)	C1: Computer–Entering	Attending (707.50)	NP (2186.75)	13, 8	1479.25	0.001
		Attending (707.50)	PA (1914.00)	13, 6	1206.50	0.011
	C6: Computer–Reading	Attending (1546.50)	NP (923.58)	13, 8	–622.92	0.010
	E1: Phone–Answering	Attending (419.17)	NP (80.00)	13, 5	–339.17	0.006
	F1: Talking–Coworker	Attending (2880.50)	NP (992.33)	13, 8	–1888.17	0.000
		Attending (2880.50)	PA (677.42)	13, 6	–2203.08	0.002
		Attending (2880.50)	Resident (1520.75)	13, 4	–1446.50	0.005
	G1: Walking–Alone	NP (182.5)	Resident (303.0)	8, 4	120.50	0.009
Time allocation (%)	C1: Computer–Entering	Attending (7.72)	NP (22.89)	13, 8	15.17	0.001
		Attending (7.72)	PA (23.64)	13, 6	15.92	0.005
		Attending (7.72)	Resident (16.92)	13, 4	9.20	0.005
	E1: Phone–Answering	Attending (4.43)	NP (0.86)	13, 5	–3.57	0.009
	F1: Talking–Coworker	Attending (31.52)	NP (10.95)	13, 8	–20.57	0.000
		Attending (31.52)	PA (7.43)	13, 6	–24.09	0.011
	G1: Walking–Alone	Attending (2.80)	Resident (4.62)	13, 4	1.82	0.012
		NP (1.86)	Resident (4.62)	8, 4	2.64	0.003
	I6: Performing–Physical exam	Attending (3.84)	Resident (6.39)	13, 4	2.55	0.012
Frequency	C6: Computer–Reading	Attending (58)	NP (22)	13, 8	–36.00	0.000
		Attending (58)	PA (24)	13, 6	–33.50	0.001
		Attending (58)	Resident (29)	13, 4	–29.00	0.006
	E1: Phone–Answering	Attending (9)	NP (1)	13, 5	–8.00	0.003
		Attending (9)	PA (2)	13, 3	–7.00	0.011
	E2: Phone–Calling	Attending (5)	PA (1)	11, 5	–4.00	0.008
	F1: Talking–Coworker	Attending (48)	NP (22)	13, 8	–26.00	0.000
		Attending (48)	PA (10)	13, 6	–38.00	0.001
		Attending (48)	Resident (31)	13, 4	–17.00	0.003

Abbreviations: NP, nurse practitioner; PA, physician assistant.

^aNonnormal distribution, using Kruskal–Wallis test to compare the pre- and postmedian.

Discussion

We examined general workflow to provide context around a specific task, i.e., antibiotic prescription to inform the development of ASPs, in EDs. Although the parent project of this study was ASP implementation and we conducted the study within the ASP context, the main focus of this study is to highlight workflow variability across clinical roles. Because of higher relevancy, we identified decision points within the workflow. The most frequent decision point for each of the four providers was after/during examining or talking to patient or relative in 55 (50%) responses. Where an antibiotic is prescribed, in 60% of the cases, the decision is made at this same point. However, as we demonstrated in this study, decisions are made at different times and at different locations. The sequential pattern analysis also demonstrated that clinicians with different roles may have different and more frequent patterns in their activities. Our analysis showed that residents have the most complicated frequent patterns involving more diverse (e.g., talking and computer-related) and interconnected activities. On the other hand, while attendings have more frequent and longer conversations with coworkers overall, they tend not to end their processes with talking to coworkers but computer reading and entering. This is consistent with their role in demonstrating participation in decision making and “signing off” on the appropriateness of the plan of care. Therefore, the design of CDS should consider clinical roles, the accessibility at different points of care, and physical locations. CDS should provide best decision using up-to-date evidence-based clinical guidelines and most available data about the patient. These findings highlight the following three main principles regarding the design and implementation of the CDS for antibiotic prescribing that would support development of an ED-based ASP.

1. CDS should support a variety of workflows: CDS should activate at the time when clinical decisions are made. 42 Our results indicate that providers with different backgrounds often take different steps and make decisions at different points within care delivery. This difference reflects their role and expertise in health care. CDS has the potential to understand a user's practice and provide decision support individualized to clinicians' roles. For ASP, the CDS could not only identify the choice of antibiotic and potential variability in prescribing practices, but consistency of practice with guidelines. CDS can benefit best when providing some standardization without harming flexibility that is due to the nature of the work.

2. CDS should support clinicians within different physical locations in the ED: Clinicians are often at different locations when they make decisions regarding treatment plans. For example, talking to a patient or their families mostly occurs at the patient's bedside while talking to a specialist often occurs in the hallways or on the phone in front of the computer screen. CDS should ideally be activated the moment when the decision regarding antibiotic prescription is made at these diverse locations.

3. CDS should support decisions at different points of care: Qualitative analysis ( Table 2 ) shows that depending on the patient's symptoms and diagnosis, prescription decisions are made at different temporal points within the care. Quantitative analysis showed that overall workflow varies across roles. Decision support systems should be contextually sensitive in terms of understanding the clinician's overall workflow for a specific patient, including identifying the timing of care to ensure that decision support is provided at the right time. If CDS is not activated in time, the recommendation is more likely to be ignored.

If the CDS is embedded in the EHR, convenient access to EHR at all locations and situations should be provided to clinicians. Having computer access (e.g., strategically placed screens on walls informed by workflow) and providing easy logging in/out features can be solutions to support these principles. Another solution would be providing mobile access to clinicians. CDS that follows these principles will be more context-sensitive and will potentially provide better cognitive support to clinicians who then can make better prescription decisions. Although our focus was CDS for ASP, these principles apply to CDS for other purposes as well. CDS vendors could improve CDS performance by developing diverse interaction points that allow clinicians to receive needed decision supports at different physical locations as well as stage in process. An important bottleneck of CDSs is availability of needed data (e.g., laboratory and imaging results) in real time. For sooner access to needed data, vendors can also improve performance by focusing on better interoperability across systems that provide the needed data.

While writing an outpatient antibiotic prescription may be similar to other medication orders, the decision to prescribe is based on more complex data than other medications. With antibiotic prescriptions, data from patient exam, results of diagnostic testing, prior antibiotic use, etc., combined with local factors such as bacterial resistance for the area, all must be considered. This decision making is usually less straightforward than selecting a bronchodilator or analgesic. However, from a workflow perspective, antibiotic prescription can be considered a prescription task. This specific study is the first step in designing an ASP initiative. We intended to understand workflow so that the new ASP initiative is congruent with the current practice.

Two recent studies 42 43 highlighted the necessity of examining workflow for the success of ASP implementation through EHR in EDs. As opposed to previous studies 44 45 that focused on time distribution of tasks across various roles, this study also examines the differences in the sequence of activities by various roles in ED. Sequence is an important building block of workflow in EDs. 3 46 Development of next-generation CDS for ASP which follows the three principles listed above may involve mobile applications and systems, which may significantly change existing workflow. Therefore, the new workflow should be consistent with the variability of care delivery work and should not hurt needed flexibility of clinicians' workflow to accomplish high-quality patient care and avoid unintended consequences.

In terms of methodological contribution, we recorded the clinical workflow activities using a tablet-based application and analyzed the data using a set of computational methods developed by the authors. 39 The successful completion of the data collection and analysis of this study demonstrates the potential generalizability of the application and the analysis methods, which were designed based on two previous publications about standardizing time motion data collection 47 and advancing time motion data analysis to uncover hidden patterns. 48 Another methodological contribution of this study is the combination of statistical and sequential pattern analysis on the workflow activity data. The former provides an overview of the frequency and time allocation of tasks in each clinical roles; the latter dives into the processes to provide a more granular view of activity sequences. This combined method enables both macro and micro view of workflow data, facilitating a more detailed understanding of the nuances of behavior patterns in clinical workflow.

There are limitations to our findings. This study was conducted in a single setting and the majority of observations were conducted by a single observer. However, pilot observations (observations before we started collecting data that was not included in the analysis) were conducted by two observers before data collection, and demonstrated excellent interobserver reliability. Another limitation is that the workflow activity data collected in this study, using a time and motion design, may not be comprehensive due to the medium sample size. However, we believe our samples are representative and we focused only on important tasks in the data collection process. This specific study is the first step in designing an ASP initiative. We intended to understand workflow so that the new ASP initiative is congruent with the current practice. We focus on day shift to ensure homogeneity of the data. However, we added this as a limitation.

Future studies will include leveraging EHR data to complement observation data with patient-specific and operational information such as diagnosis, demographics, and workload demands of the ED. Future studies will also include applying the data collection tool in other scenarios (such as hand-offs) to improve its ability to be customized for the setting and the generalizability of tool. This study examined general workflow and diagnostic decision in general. Our future efforts will also focus on cases in which antibiotics were prescribed and more detailed steps specific to antibiotic prescription and its unique characteristics.

Conclusion

The design and implementation of the CDS to implement an ED-based ASP should support all four ED provider roles, who often have different workflows. Clinicians make their decisions about treatment at different points of care delivery; a CDS for antibiotic prescribing also should support decisions at these different points of care.

Clinical Relevance Statement

CDS is the preferred method for implementation of ASP; 24 however, timing of CDS activation and presentation to clinicians is challenging in the ED setting. The ED is characterized by an erratic interrupted workflow, and a variety of providers and provider types that make decisions regarding antibiotic prescribing at different time points during the ED visit. 6 Integration into the clinical workflow is among the most important preferences of ED providers with respect to ED-based ASP implementation. 42 The results of our study demonstrate the complexity of the ED workflow, and the differences that exist among various ED providers. These workflow patterns will be essential to inform and develop our CDS for antibiotic prescribing, which will serve as the centerpiece for an ED-based ASP.

Multiple Choice Question

Which of the following statements is incorrect when designing and implementing clinical decision support systems in EDs?

1. CDS should support workflow for various type of clinical roles

2. Clinical decision support should support clinicians within different physical locations in the ED

3. CDS should support decisions at different points of care

4. The functionality of CDS should depend time of the day

Correct Answer: The correct answer is option d. As data analysis revealed in this study, CDS should support workflow for various clinical role. CDS also support clinicians within different physical locations and CDS should support decisions at different points of care. Although functionality of CDS can depend on various contextual factors such as workload in ED or individual characteristics of the clinicians, there is no evidence that the functionality of CDS should depend on time of the day.

Funding Statement

Funding This study was funded by grants from Emergency Medicine Foundation, University of Colorado School of Medicine and Children's Hospital Colorado.