Table 3:

Components of workflow integration of PE Dx CDS

#	Components of workflow integration (# of interviewees)	Description (− indicates a barrier to workflow integration; + indicates a facilitator to workflow integration)	Dimensions of workflow integration
			TIME	FLOW	SCOPE	LEVEL
1	Combination of Wells’ and PERC risk scoring algorithms (5 physicians – 3 attendings, 2 residents)	(−) PE Dx is not integrated in the workflow because the CDS is designed to be used in a sequence with Wells’ first and then PERC second. In reality, physicians may want to use both Wells’ and PERC together at the same time or in some instances they want to skip Wells’ and go straight to PERC.	• Sequential • In parallel	• Tasks	• Within ED visit	• Individual
2	PE Dx is more useful in CareStart, early on in patient visit (1 physician − attending)	(+) PE Dx is integrated in the workflow early on in the patient visit when a physician is working in CareStart. The physician places more CTs for PE when seeing patients early on in the visit in CareStart.	• Timing	• Tasks	• Within ED visit	• Individual
3	Fit with workflow of attending and resident physicians working together (3 physicians - attendings)	(+) PE Dx is integrated in the workflow as it supports the work of attending and resident physicians as they are deciding what tests to order for the patient. The attending and resident discuss the patient before the resident uses the CDS or the attending and resident use the CDS together after the resident talks to the patient.	• Sequential • In parallel	• Tasks	• Within ED visit	• Team (attending + resident)
4	Fit with the resident workflow learned when rotating to other services out of the ED (4 physicians – residents)	(−) PE Dx is not integrated in the workflow of residents that they adapt to when working in other services outside of the ED. ED residents rotate into other services such as cardiology where they spend a lot of time writing notes and do not have an ED navigator. When they return to the ED, they keep their learned workflow, which does not fit with the workflow of going to the ED navigator to use PE Dx. When off-service, residents also learn to use MDCalc and use it on the phone when they are in other services intern year.	• Extended time	• Tasks • People	Not applicable	• Individual
5	Fit with interruptions in the ED (1 physician – attending)	(+) PE Dx is integrated in the workflow because it allows the physician to be interrupted and saves the work so the physician can go back to finish the PE Dx later.	• Discontinuous	• Tasks	• Within ED visit	• Individual
6	Fit with physical environment when seeing patients (1 physician – attending)	(−) PE Dx is not integrated in the workflow because it is not mobile and therefore the physician needs to be in front of the computer in order to use it. ○ The physician is not able to use the CDS while walking between patient rooms ○ The physician cannot pull up the CDS on their phone in the patient room; the computers in patient rooms are slow so it is preferable to use the phone to calculate the patient risk while in the room. PE Dx does not fit this workflow, but MDCalc does.	• Timing	• Tasks • People	• Within patient interaction	• Individual • Team (patient)
7	Integration of multiple CDS (5 physicians – 2 attendings, 3 residents)	(−) PE Dx is not integrated in the workflow because there are no other CDS like PE Dx in the EHR. The PE Dx is designed well but because it is the only CDS of its kind in the EHR, it does not fit with the other tools that the physician will use while making diagnostic decisions for the patient. The physician prefers to use MDCalc because the physician can look up multiple risk scores at the same time such as for PE, pneumonia, etc. As more CDS like PE Dx are built in the EHR, PE Dx will better fit the broader workflow of the physician and will become increasingly integrated into physician workflow.	• In parallel	• Tools	• Within ED visit	• Individual
8	Placing orders via PE Dx versus placing all the other orders (2 physicians – attendings)	(+) PE Dx is integrated in the workflow because it is easy for a physician to place the D-dimer order separately from the other orders in the PE Dx.	• Sequential	• Tasks	• Within ED visit	• Individual
		(−) PE Dx is not integrated in the workflow because physicians prefer to place all of their orders for the patient at one time, not only the orders for diagnosing PE. PE Dx supports the D-dimer/CT order but does not fit the workflow of placing all of the orders together. If the physicians places the D-dimer order with PE Dx separately from the other orders, it may interfere with nursing workflow of drawing blood.	• In parallel	• Tasks	• Within ED visit	• Individual • Team (nurse)
9	Supporting next steps (2 physicians – residents)	(+) PE Dx is integrated in the workflow because it supports the appropriate clinical pathway and therefore, does not require a PERC to be calculated if the Wells’ score is moderate or high. In these cases, the PE Dx goes straight to the next step by supporting the physician to place the order.	• Sequential	• Tasks: supports next step	• Within ED visit	• Individual
		(+) PE Dx could be expanded to integrate in the workflow by supporting other steps in the PE diagnostic process including activating the PE Response Team based on the results of the CT scan.	• Sequential	• Tasks: supports next step	• Within ED visit	• Team (PE response team)
10	Integration in the EHR (5 physicians – 1 attending, 4 residents)	(+) PE Dx is integrated in the workflow because it is embedded within the EHR; therefore, the physician can pull the documentation from the CDS directly into the note and they do not need to go to an internet browser for the risk calculator.	• Sequential	• Tasks • Tools	• Within ED visit	• Individual
		(−) PE Dx is not integrated in the workflow within the EHR because in order to use the PE Dx, the physician needs to exit out of their task in the EHR and go to another spot in the chart to use PE Dx. The physician, therefore, cannot access the CDS at the same time as they are placing orders.	• In parallel	• Tasks • Tools	• Within ED visit	• Individual
11	Alternative locations for PE Dx in the EHR (2 physicians – 1 attending, 1 resident)	(−) PE Dx is not integrated in the workflow because it is not in a location within the EHR that is convenient for the workflow. The CDS would better integrate in the workflow if it was placed at the top of Epic or in the workups page close to orders.	• Timing	• Tasks: in a location near preceding/succeeding task	• Within ED visit	• Individual
12	Physician is never in ED navigator so they did not know about tool or do not use (4 physicians – 1 attending, 3 residents)	(−) PE Dx is not integrated in the workflow because physicians never use the ED navigator where the PE Dx is located. Because physicians are never in the ED navigator, they either do not know PE Dx is available or it is never used because it is inconvenient to get to.	• Timing	• Tasks: in a location near preceding/succeeding task	• Within ED visit	• Individual
13	Location of PE Dx in ED navigator (8 physicians – 5 attendings, 3 residents)	(+) PE Dx is integrated in the workflow because the ED navigator is easy to get to within the EHR and can be accessed in one click. The ED navigator is close to where physicians are starting their notes and it is a common area the physician will be working in.	• Timing	• Tasks	• Within ED visit	• Individual
		(−) PE Dx is not integrated in the workflow because physicians do not use the ED navigator except when they are working in CareStart. When in the regular ED, physicians do not use the ED navigator so the location in the EHR does not support the workflow.	• Timing	• Tasks	• Within ED visit	• Individual
		(−) PE Dx is not integrated in the workflow because of its location in the EHR in the ED navigator. The PE Dx is not near other tasks that physicians are working on when they need the decision support. Before Epic was upgraded, physicians used to work their way down the ED navigator to look at notes, vitals, orders, use the PE Dx and write notes. After the Epic upgrade, PE Dx is not embedded in this workflow.	• Sequential	• Tasks: in a location near preceding/succeeding task	• Within ED visit	• Individual
		(−) PE Dx is not integrated in the workflow because physicians would not go to the ED navigator until the end of the ED visit when they are ready to write their note.	• Timing	• Tasks	• Within ED visit	• Individual
14	Integration with notes (4 physicians – 1 attending, 3 residents)	(−) PE Dx is not integrated in the workflow because it is not accessible within the notes. To use PE Dx, the physician needs to exit out of their notes and go to the ED navigator. PE Dx would better integrate in the workflow if it could be used directly within the notes. This would enable the physician to see their notes as they are filling out the CDS.	• In parallel	• Tasks	• Within ED visit	• Individual
15	Integration with orders (7 physicians – 5 attendings, 2 residents)	(+) PE Dx is integrated in the workflow because a physician uses it to support their decision immediately before placing an order. If the physician has already placed the order, they can use PE Dx to confirm the decision and dismiss the option in the CDS to place the order.	• Sequential	• Tasks	• Within ED visit	• Individual
		(−) PE Dx is not integrated in the workflow because it is difficult to use in the patient room while the physician is placing orders and talking with the patient. The computers in the patient rooms are slow, therefore physicians do not go to other places in the EHR (e.g. ED navigator) while placing the orders. The CDS does not integrate in this workflow.	• In parallel	• Tasks	• Within patient interaction	• Team (patient)
		(−) PE Dx is not integrated in the workflow because it is not accessible within the ordering screen. ○ The physician wants the CDS when they are in the ordering screen, deciding what orders to place. The CDS would better integrate in the workflow if it was easily accessible/available at this point in the workflow. If the physician wants to use the PE Dx, they need to close out of the orders and go to the ‘ED navigator’. If PE Dx was available in the orders, they could continue their work without interruption. ○ Physicians do not go to the ED navigator until the end of the ED visit to write notes. If the PE Dx was integrated in the orders, it would better fit the workflow of physicians placing the orders early on in the patients ED visit.	• In parallel	• Tasks: in a location near preceding/succeeding task	• Within ED visit	• Individual
16	Fit with temporal flow of patient interaction (7 physicians – 4 attendings, 3 residents)	(+) PE Dx is integrated in the workflow because it helps the physician remember the questions to ask a patient. The physician will start a note, talk with the patient, and then use the PE Dx after talking with the patient.	• Sequential	• Tasks • People	• Within patient interaction • Within ED visit	• Individual • Team (patient)
		(+) PE Dx is integrated in the workflow because after talking with the patient, the physician always returns to the computers and goes into the EHR. The PE Dx fits in this workflow. If the physician is unsure on what to order, they will go to PE Dx.	• Sequential	• Tasks • People	• Within patient interaction • Within ED visit	• Individual • Team (patient)
		(−) PE Dx is not integrated in the workflow because after the physician talks with the patient, they return to the computer, and go directly to placing orders. Since the PE Dx is not within the ordering screen/near the ordering screen, it does not fit in the physician workflow.	• Sequential	• Tasks • People	• Within patient interaction • Within ED visit	• Individual • Team (patient)
		(−) PE Dx is not integrated in the workflow because the physician is usually listening to the patient presentation, deciding what orders to put in, looking at what the CareStart physician has done. PE Dx does not support this workflow.	• In parallel	• Tasks	• Within patient interaction • Within ED visit	• Individual • Team (patient)
		(−) PE Dx is not integrated in the workflow in cases where the physician forgets to ask a patient a question (e.g. do they have hemoptysis?) before using the tool. In this case, the physician would need to exit the PE Dx, leave the physician computer station and go back into the patient room to ask the question before completing the tool. The physician typically completes their documentation in the physician computer work station not in the patient room.	• Timing	• Tasks • People	• Within patient interaction • Within ED visit	• Individual • Team (patient)
17	Timing of documentation from PE Dx versus other documentation (5 physicians – 2 attendings, 3 residents)	(+) PE Dx is integrated in the workflow after the physician reviews the patient orders, vitals, and history. The physician uses PE Dx before going into the note. PE Dx is integrated in the workflow because the documentation from the PE Dx goes directly into the physician note, therefore the physician does not need to go back later and re-document what was calculated in MDCalc.	• Sequential	• Tasks: supports next task	• Within ED visit	• Individual
		(−) PE Dx is not integrated in the workflow because the physician goes back to edit the patient note throughout the ED visit. ○ The PE Dx documentation should be able to be edited throughout the patient’s ED visit. • The PE Dx documentation support does not always work if the physician has already started a note before using the tool. This can cause the PE Dx to overwrite the old documentation with the PE Dx documentation. In order to get the documentation text in, the physician needs to refresh their note, which can cause a problem if the physician forgets to refresh their note and the PE Dx documentation never populates in.	• Discontinuous	• Tasks	• Within ED visit	• Individual
18	Trigger vs no trigger (9 physicians – 5 attendings, 4 residents)	(+) PE Dx is integrated in the workflow as there are not any hard stops or repetitive typing to use it. PE Dx does not pop-up and interrupt the workflow.	• Timing	• Information: no trigger	• Within ED visit	• Individual
		(−) PE Dx is not integrated in the workflow because it is not a pop-up or alert so sometimes the physician forgets to use the tool. It may be more annoying, consequently physicians would use the tool more if it was an alert.	• Timing	• Information: no trigger	• Within ED visit	• Individual
		(−) PE Dx would be more integrated in the workflow if it was optimized to support the order decision. Physicians want the PE Dx in the background checking patient vitals, chief complaint, and risk factors and then triggering an alert that suggests the appropriate order based on those factors. PE Dx could also work if the CDS came up when the physician placed an order for a CT scan. The triggers for prompting decision support would be challenging so they do not overtrigger.	• Timing	• Information: trigger • Tasks: support next task	• Within ED visit	• Individual
19	Timing of trigger in the workflow (4 physicians – 3 attendings, 1 resident)	(+) PE Dx is integrated in the workflow because it is not a BPA and therefore it does not come up at the wrong time and interrupt the work	• Timing	• Tasks	• Within ED visit	• Individual
		(−) If the PE Dx triggered in the workflow, the timing for the trigger is difficult. Before placing the D-dimer, it may be too late in the workflow because the physician has already decided on the order to place. Before the CT scan will be annoying for really high-risk patients that do not require decision support. Forcing physicians to use decision support before placing the orders will be frustrating especially for experienced physicians. CDS could also be a pop-up that comes up in the physician note.	• Timing	Information: point in time for trigger • Tasks	• Within ED visit	• Individual
20	Flag for relevant information (1 physician – attending)	(−) PE Dx would be better integrated in the workflow if there was a prompt/cue for specific information found in the patient chart (e.g. prior DVT). This would take out the initial screening step that physicians need to do and prompt them to look further into potential risk factors.	• Timing	• Information	• Within ED visit	• Individual