Post-implementation usability evaluation of a human factors-based clinical decision support for pulmonary embolism (PE) diagnosis (Dx): PE Dx Study Part 1

Abstract

While there is promise for health IT, such as Clinical Decision Support (CDS), to improve patient safety and clinician efficiency, poor usability has hindered widespread use of these tools. Human Factors (HF) principles and methods remain the gold standard for health IT design; however, there is limited information on how HF methods and principles influence CDS usability “in the wild”. In this study, we explore the usability of an HF-based CDS used in the clinical environment; the CDS was designed according to a human-centered design process, which is described in Carayon et al. (2020). In this study, we interviewed 12 emergency medicine physicians, identifying 294 excerpts of barriers and facilitators of the CDS. Sixty-eight percent of excerpts related to the HF principles applied in the human-centered design of the CDS. The remaining 32% of excerpts related to 18 inductively-created categories, which highlight gaps in the CDS design process. Several barriers were related to the physical environment and organization work system elements as well as physicians’ broader workflow in the emergency department (e.g., teamwork). This study expands our understanding of the usability outcomes of HF-based CDS “in the wild”. We demonstrate the value of HF principles in the usability of CDS and identify areas for improvement to future human-centered design of CDS. The relationship between these usability outcomes and the HCD process is explored in an accompanying Part 2 manuscript.

1. Introduction

With the growing use of health IT and associated technologies such as computerized Clinical Decision Support (CDS), emerging opportunities exist to leverage these technologies to improve patient safety. For instance, the Emergency Department (ED) is a complex environment with frequent interruptions, high time pressure, severe patient acuity, and numerous decisions that need to be made, usually with limited knowledge on the patient (Wears & Leape, 1999; Wears et al., 2010). CDS can provide benefit in these challenging environments by supporting the complex decision-making process of clinicians (El-Kareh et al., 2013; Patterson et al., 2019). However, poor usability has limited the widespread acceptance and use of health IT, including CDS (Ratwani et al., 2019). The US Office of the National Coordinator recommends Human Factors (HF) methods and principles to improve the usability of health IT (The Office of the National Coordinator for Health Information Technology, 2020).

1.1. Usability of HF-based technologies

Several studies have demonstrated the value of HF in improving the usability of health IT in experimental settings (Beuscart-Zéphir et al., 2010; Carayon et al., 2020; Clark et al., 2017; Georgsson & Staggers, 2016; Russ et al., 2014). For instance, Russ et al. (2014) utilized HF principles and methods to design a CDS for identifying drug-drug, drug-allergy, and drug-disease interactions. In a scenario-based simulation, they demonstrated superior usability of the HF-based CDS with a 50% reduction in prescribing errors and a 34% reduction in task time compared to the currently used system. Carayon et al. (2020) found that a CDS developed in a Human-Centered Design (HCD) process, which integrated HF principles (e.g. consistency), improved diagnostic decision-making and reduced physician workload compared to the currently used CDS. Despite promise in experimental settings, few studies have explored the usability of HF-based technologies once they are implemented in the real clinical environment (Catchpole et al., 2022; Salwei, Hoonakker, et al., 2022). In this study, we investigate the usability outcomes (i.e., barriers and facilitators) of an HF-based CDS implemented in the ED for diagnosing pulmonary embolism (PE).

1.2. Pulmonary embolism diagnosis

PE is a blood clot in the lung, which can be fatal if not treated quickly. Approximately 100,000 people die from PE in the US each year (Khan et al., 2021). PE is particularly challenging to diagnose as patients often present with non-specific symptoms (e.g., generalized chest pain). Computed Tomography (CT) scans represent the most reliable method to diagnose patients with PE. However, there has been over-use of CT scans, which is costly and exposes patients to harmful radiation and renal injury (Kline et al., 2014). Therefore, physicians must balance the risks of under-testing (i.e., missing PE) and over-testing (i.e., over-use of CT scans). The D-dimer test (conducted with a blood draw) is another diagnostic test that can be useful in the PE diagnostic process. The D-dimer test has a high negative predictive value in low and moderate risk patients (Segal et al., 2007); however, it has a poor positive predictive value (Yan et al., 2017).

Due to the complicated nature of PE diagnosis, multiple risk scoring algorithms have been developed. Two of these algorithms, the Wells’ criteria (Wells et al., 2001) and the Pulmonary Embolism Rule-out Criteria (PERC) rule (Kline et al., 2010), are recommended for risk stratifying patients suspected of PE (Raja et al., 2015). First, the Wells’ criteria can be used to determine if a patient is low, moderate, or high risk for PE, based on seven questions about the patient’s history (e.g., surgery in the previous 4 weeks) and current symptoms (e.g., heart rate greater than 100). If a patient is high risk according to the Wells’, a CT scan should be ordered to further rule out PE. If a patient is moderate risk, a D-dimer test should be ordered. If a patient is low risk according to the Wells’, the PERC rule can then be applied. Similar to the Wells’, the PERC rule includes a series of 8 questions about the patient (e.g., prior PE or Deep Vein Thrombosis (DVT)?); a patient will be PERC positive if a clinician responds affirmative to any of the 8 questions. PERC positive patients likely merit D-dimer testing whereas PERC negative patients require no further work-up for PE diagnosis.

1.3. Human-centered design of a CDS for PE diagnosis

To support the challenging PE diagnostic process, an interdisciplinary team of HF researchers, emergency physicians, and an IT specialist designed a CDS to support PE diagnosis in the ED, i.e., PE Dx (Carayon et al., 2020; Hoonakker et al., 2019). Throughout 7 steps in the PE Dx design process (see left side of Figure 1), we systematically considered 9 HF design principles (Table 1). PE Dx combined the Wells’ and PERC in a tier-approach, where first the Wells’ was presented and then, if indicated (e.g., patient is low risk according to Wells’), the PERC rule appeared. PE Dx automatically populated some patient data from the EHR into the Wells’ and PERC criteria, such as the patient’s recorded heart rate and age. Using the Wells’ and PERC, the CDS calculated a patient’s risk for PE and provided a recommendation on the appropriate next step (e.g., D-dimer test). The CDS then supported ordering the recommended diagnostic test and documenting the decision in the note. In a scenario-based simulation (Carayon et al., 2020; Salwei, Carayon, et al., 2022), we demonstrated high usability of PE Dx, which improved diagnostic accuracy and reduced physician workload compared to the currently used CDS (i.e., MDCalc). PE Dx was implemented in an ED in December 2018; this study focuses on the evaluation of PE Dx “in the wild”.

Figure 1.

Conceptual framework of the study

Table 1.

HF principles used in the design of PE Dx (Carayon et al., 2020)

HF principles	Example of implementation in PE Dx
1. Automation of information acquisition (Parasuraman et al., 2000)	Automatic population of data from the patient’s chart into the risk scores
2. Automation of information analysis (Parasuraman et al., 2000)	Automatic calculation of Wells’ risk score
3. Support of decision selection (Parasuraman et al., 2000)	Provide recommendation for next step based on score, e.g., order CT scan
4. Explicit control / flexibility (Parasuraman et al., 2000; Scapin & Bastien, 1997)	Able to change risk criteria values, e.g., patient heart rate
5. Minimizing workload (Scapin & Bastien, 1997)	Minimize data entry, e.g., only need 1 positive criteria in the PERC to get a result
6. Consistency (Scapin & Bastien, 1997)	CDS format follows the conventions of the EHR, e.g., dark blue color for selected button
7. Chunking/grouping (Sanders & McCormick, 1993; Scapin & Bastien, 1997)	Visually separating Wells’ and PERC criteria
8. Visibility (Zhang et al., 2003)	Depict point values for each Wells’ criteria to convey how the score is calculated
9. Error prevention (Scapin & Bastien, 1997; Zhang et al., 2003)	All Wells’ criteria must be addressed to receive a Wells’ score

2. Problem statement

We aim to expand our understanding of the usability of HF-based CDS when used in the real clinical environment and gather insights for future design of health IT.

3. Methods

This study was part of a larger project investigating health IT-supported processes for Venous Thromboembolism (VTE) diagnosis and management (https://cqpi.wisc.edu/research/health-care-and-patient-safety-seips/vte-and-health-it/). The study took place in one ED of an academic health system in the Midwest of the Unites States and was approved by the associated Institutional Review Board. Figure 1 depicts the conceptual framework guiding this study.

3.1. Data collection

Nine months after the implementation of PE Dx, we conducted semi-structured interviews with 12 emergency physicians with varying levels of experience (i.e., year 1-3 residents and attending physicians). One or two HF researchers conducted each interview, which lasted an average of 27 minutes (standard deviation: 6 minutes). The interviews focused on physicians’ use of PE Dx and the barriers and facilitators (Carayon et al., 2006; Smith & Carayon-Sainfort, 1989) to using PE Dx within their workflow (full interview guide here). Due to the power differential in the ED team (e.g., between an attending physician, a resident physician, and a nurse) as well as the logistical challenge of scheduling multiple clinicians at one time, we decided to conduct individual interviews for this project. We included specific probes in the interview guide asking about the role of other team members in the use of the CDS; for example, “do you usually interact with anyone (e.g., the patient, other clinicians) while using the tool?”. We also probed about how the CDS interacted with other work system elements, such as tasks, the physical environment, and tools and technologies used in the ED (Carayon, 2009; Smith & Carayon-Sainfort, 1989). During each interview, we used the EHR “playground” environment, which allowed physicians to interact with the CDS in a simulated environment that mirrors the actual EHR. The use of the EHR playground facilitated the interview by reminding physicians about the PE Dx and how they have used the technology in the ED.

Each interview was audio-recorded and transcribed by a professional transcription service. We uploaded the interview transcripts into a qualitative data analysis software, Dedoose^©.

3.2. Data analysis

To analyze the interview data, we used a combination of deductive and inductive content analysis (Elo & Kyngäs, 2008). The deductive analysis was guided by the HF principles applied during the design of PE Dx (Table 1); see Carayon et al. (2020) for more details on the design of PE Dx and see Figure 1 and Table 1 for the list of HF design principles. We coded each interview for barriers and facilitators and the HF principle(s) related to each barrier or facilitator. If no HF principles applied to an excerpt, we inductively coded the excerpt. Two researchers started by independently coding two transcripts and meeting to discuss and resolve any discrepancies in the coding. We calculated the total number of excerpts coded and the total number of excerpts that were coded the same by both researchers, and then we calculated the Inter-Rater Reliability (IRR). After coding the first two transcripts, IRR was 63%. We updated the codebook to clarify the concepts and definitions and then coded two more transcripts, reaching an IRR of 98%. One researcher then coded the remaining interview transcripts, consulting the second researcher as questions came up in the coding.

All coded excerpts were exported from Dedoose© into Microsoft Excel. We grouped the excerpts according to the HF design principles coded. Next, we reviewed the inductively coded excerpts; one HF researcher printed out a slip of paper for each excerpt. After reviewing each excerpt, we grouped excerpts representing similar ideas into preliminary categories. The excepts and preliminary categories were then reviewed by another research team member. The two researchers then discussed and finalized the categories of inductively coded excerpts.

4. Results

We identified a total of 294 excerpts with barriers and facilitators of PE Dx. Out of the 294 excerpts, 199 (68%) were coded as HF design principles; a description of the barriers and facilitators for each HF principle can be seen in Table 2, along with the number of excerpts coded as barriers and facilitators for each principle. The remaining 95 excerpts (32%) were coded inductively, representing barriers and facilitators not related to the HF design principles. Through an inductive process, we grouped the 95 excerpts into 18 categories, which are described in Table 3. Figure 2 details the percentage of barriers and facilitators for excerpts coded as HF principles and as inductive categories.

Table 2.

Description of barriers and facilitators related to the HF design principles

HF principles	Barriers	Facilitators
Automation of information acquisition (3 – 9)	- PE Dx may auto-populate incorrect data (e.g., incorrect vitals in the chart) - PE Dx does not pull in enough data	- Automatically populates data (e.g., vital signs, age)
Automation of information analysis (2 – 7)	- Physician does not need the exact Wells’ score	- PE Dx helps determine if the present risk factors (e.g., prior DVT) put the patient in the D-dimer or CT scan threshold - Exact Wells’ score and recommendation helps physician remember the cut-offs for risk
Support of decision selection (4 – 24)	- There is a pop-up alert when the PE Dx recommends a D-dimer but the physician does not have high enough suspicion to place the order	- PE Dx gives a clear recommendation based on patient risk
Explicit control/flexibility (5 – 1)	- PE Dx may force the physician down a PE pathway; it is hard to not order a diagnostic test after a high score is recorded in the chart - The physician cannot edit PE Dx documentation throughout patients’ ED visit	- PE Dx is a resource but is not mandatory
Minimization of workload (29 – 53)	- PE Dx is slower than other risk calculators (e.g., MDCalc) - PE Dx documentation support does not work if the physician has already started their note - The physician has to click “refresh” to get PE Dx documentation in the note - It is easier for physicians to use dictation than to use documentation support in PE Dx - PE Dx is hard to access in the EHR due to its location - PE Dx is slower than using gestalt; PE Dx is extra clicks for things the physician is already thinking about	- PE Dx supports ordering the recommended test, which saves clicks - PE Dx does not require a lot of scrolling or clicking - PE Dx reduces the need to go searching in the chart for information - PE Dx is embedded in the EHR, so physicians do not need to go to another website to calculate the risk scores - PE Dx does not require the physician to use the PERC if the Wells’ is high enough to go straight to ordering a test
Consistency (1 – 6)	- Do not like the color scheme of PE Dx, which is the color scheme of Epic	- PE Dx is similar to other online risk scores for PE (e.g., MDCalc); the criteria are listed in the same order and the yes/no boxes are the same
Chunking/grouping (27 – 10)	- Physician considers Wells’ and PERC at one time, not as separate criteria - Physician often starts with PERC rather than Wells’ - Location of PE Dx in the EHR (ED navigator) does not fit in physician workflow	- PE Dx is easily accessible; the ED navigator is a common place physicians are working - PE Dx sequence of using Wells’ before PERC reminds the physician of the intention for the risk scores
Visibility (2 – 6)	- Physicians did not understand all of the functionalities of the PE Dx (e.g., physician did not know how to edit the documentation or vital signs)	- Point values are listed on the yes/no buttons - PE Dx provides interpretation of patient risk scores - Exact patient scores get documented in the chart
Error prevention (1 – 9)	- Physician does not always use Wells’ before PERC; PE Dx forces this workflow to prevent errors	- PE Dx reminds physicians of the risk information and questions to ask patients (e.g., hemoptysis) - Cannot access PERC without low Wells’ score

^*The numbers in parentheses represent the total number of excerpts coded as barriers and facilitators

Table 3.

Inductively identified categories and corresponding barriers and facilitator

Categories *	Barriers	Facilitators
Mobile workflow (2 – 0)	- PE Dx cannot be used on a phone and is not mobile therefore, it cannot be used while walking between patient rooms or on the phone in a patient room while discussing the patient’s risk
Computer access (2 – 2)	- The computers in the patient rooms are slow making it harder to use PE Dx while talking with patients	- PE Dx is easy to use because the physician has easy access to computers in the ED - PE Dx can be used on computers in the patient room in The American Center (TAC)
Resident workflow in other services (3 – 0)	- Resident physicians rotate out of the ED into other services where they learn new workflows. PE Dx does not fit in this new workflow when the resident returns to the ED (e.g., residents spend a lot of time in the notes and use MDCalc when in other services)
Preference for another CDS (5 – 0)	- MDCalc is easier to use than the PE Dx because physicians usually have an internet browser open on their computer during the shift
Integration of multiple CDS (7 – 1)	- There are no other CDS in the EHR like PE Dx, which makes it less appealing to use - MDCalc has multiple different risk scores making it a “one-stop shop” for calculating the patient’s risk for different possible conditions	- As more CDS like PE Dx are in the EHR, PE Dx will better fit in physician workflow; the physician can then go to one spot in the EHR to calculate all applicable risk scores
Time pressure (3 – 0)	- Time pressure in the ED makes it less likely that a physician will use PE Dx - Physicians may revert to gestalt or use MDCalc on their phone instead of PE Dx
Integration within EHR (4 – 2)	- Using PE Dx interrupts the workflow because the physician needs to leave their current task in the EHR to access and use PE Dx - The ED navigator where PE Dx is located in the EHR is cluttered making it hard to find the CDS	- PE Dx is a good adjunct for the EHR and works well because it is embedded in the EHR
Physician-patient workflows (3 – 3)	- PE Dx is difficult to use if the physician forgets to ask patients one of the criteria questions; in this case, the physician needs to return to the patient room before completing PE Dx	- PE Dx can be used while placing orders in the patient room - PE Dx best fits the workflow after the physician sees the patient and before placing orders - As the physician continues to use PE Dx, it will become part of the workflow to ask patients the criteria questions
Availability of residents (1 – 2)	- The attending is less likely to use PE Dx at UW because the residents are there	- PE Dx is more useful when the attending is at TAC without residents
Interruptions (0 – 1)		- PE Dx is short, so interruptions do not affect use of the tool
Preference for another PE workflow (3 – 0)	- The physician does not use PE Dx because they already have a PE workflow - The physician tried PE Dx and then reverted to their old workflow - Physician prefers to use the Geneva score versus the Wells’
Attending-resident tasks (2 – 0)	- Attendings are not the main users of PE Dx as residents place the orders for patients - PE Dx does not fit with the other tasks attendings are doing when deciding what to do for PE, such as listening to residents about the patient presentation and reviewing the chart and current orders
Gestalt and memorization of criteria (9 – 0)	- Physician does not need PE Dx because they have the risk criteria memorized - Physician gestalt is good enough to determine appropriate diagnostic pathway (e.g., if the physician gestalt is that Wells’ is low, the physician may only want to use PERC)
Teaching tool (0 – 10)		- PE Dx is useful as a teaching tool for medical students, residents, and advanced practice providers (APPs)
Adaptation for patient risk (6 – 2)	- Use of PE Dx varies depending on the patient scenario such as patient age (e.g., over 50 years old) and risk level (e.g., young, cancer patient, pregnant). For example, the PERC criteria are not relevant if the patient is over 50 - If the patient is high risk, the physician goes straight to ordering a CT scan	- PE Dx is helpful for moderate and high-risk patients as it helps justify the diagnostic pathway and order chosen - PE Dx is helpful when a physician thinks a patient is low risk, but is uncertain on the true risk
Ordering workflow (4 – 3)	- The physician prefers to place all orders at the same time instead of only placing the order for PE diagnosis. The CDS does not fit this workflow of placing all orders at once - Physicians use a workaround of placing all orders and then using PE Dx to confirm their decision or using PE Dx and then placing all of the orders together afterwards - If the physician places the D-dimer order separately from the other orders, it may interfere with the nursing workflow of drawing blood - The physician is less likely to use PE Dx if they have already placed other orders while in the patient room	- PE Dx can be used while placing orders in the patient room - It is easy to place the PE Dx order separate from the other orders - PE Dx fits in the ordering workflow, in which a physician returns to the computer station after talking with a patient
Access to CDS evidence (2 – 0)	- MDCalc is better than PE Dx because it provides literature that supports the recommendation and rationale for the criteria
Alerts (10 – 3)	- Because there is no alert, the physician sometimes forgets to use PE Dx - The physician did not know PE Dx existed - It would be hard to find the best time in the workflow for an alert - PE Dx would better integrate in the workflow if there was an alert or a reminder to use it	- PE Dx is better than other CDS because there is no alert or pop-up to use it, which would interrupt the workflow

^*The numbers in parentheses represent the total number of excerpts coded as barriers and facilitators

Figure 2.

Percentage of barriers and facilitators for excerpts coded as HF principles and inductive categories

4.1. Facilitators

4.1.1. HF principles

The majority (63%) of excerpts coded as HF principles were facilitators. The most frequently discussed facilitator related to the HF design principle minimization of workload. Physicians liked that PE Dx automatically populated data from the EHR (e.g., heart rate, age) into the CDS criteria, which eliminated the need to search in the chart for information. Physicians also liked that PE Dx was built within the EHR, eliminating the need to exit the chart to calculate a patient’s PE risk score. Related to support of decision selection, physicians liked that PE Dx provided a clear recommendation on next steps based on a patient’s risk score. For example, if a patient’s Wells’ score was 3, PE Dx indicated that this was a moderate risk and that the appropriate next step was to order a D-dimer blood test. Physicians also liked that the PE Dx helped to prevent errors, for instance, by reminding them of the necessary questions to ask patients (e.g., are you coughing up blood?).

4.1.2. Inductive categories

We identified several facilitators not related to any of the HF design principles. One code teaching tool was a common facilitator mentioned by physicians. Attending physicians liked that PE Dx provided a guideline-based structure for residents, medical students, and Advanced Practice Providers (APPs) to learn about risk stratifying patients suspected of PE. The PE Dx also supported physician workflow when they were working at the community hospital without the support of residents; in these instances, PE Dx supported their tasks by helping them place orders and document care efficiently. Physicians also liked that PE Dx was not an alert and therefore, it did not interrupt their workflow.

4.2. Barriers

4.2.1. HF principles

We identified 74 excerpts of barriers related to HF design principles. For instance, we identified several barriers for the HF principle chunking/grouping. One of these barriers was related to the location of PE Dx in the EHR called the “ED navigator”, which is the left sidebar of a patient’s chart. PE Dx could be accessed by clicking a button in the “ED navigator”, which is used for several other tasks such as reading notes and placing orders. Following implementation of PE Dx, physicians stated that they did not use the ED navigator and instead used other sections of the chart; it did not fit physician workflow to access PE Dx in the “ED navigator”. We also found that the grouping of Wells’ separately from PERC was described as a barrier. Physicians stated that they sometimes wanted to use PERC on its own rather than first completing Wells’ followed by PERC; some physicians reported that they had the Wells’ criteria memorized and therefore wanted to go straight to the PERC criteria without completing the Wells’. Other physicians described that they used the PERC first if they thought the patient was very low risk in order to quickly rule out PE.

Related to the HF principle automation of information acquisition, some physicians were concerned that the PE Dx would automatically populate incorrect EHR data (e.g., heart rate), which was a barrier to use. Finally, physicians described several barriers related to the principle minimization of workload. Physicians stated that other risk calculators (e.g., MDCalc) were faster to use than PE Dx. Physicians also experienced some issues with the PE Dx documentation support, stating that the documentation required them to refresh their note if it was already started before using the tool.

4.2.2. Inductive categories

Most of the inductively coded categories were barriers (69%). One barrier was related to physician movement throughout the physical environment in the ED (code: mobile workflow). Physicians described that they often use their phone to calculate a patient’s risk score of PE while walking between patient rooms; because PE Dx could only be used on the computer, it did not support this workflow. Another barrier related to the availability of other CDS in the EHR, corresponding to the code integration of multiple CDS. Physicians described how they used MDCalc to calculate multiple risk scores for a patient at one time, such as for PE, pneumonia, and heart failure. Because PE Dx was the only CDS integrated in the EHR, it did not support their workflow.

A unique aspect of resident workflow hindered use of PE Dx (code: resident workflow in other services); we found that ED residents frequently rotated to other services outside of the ED. When in these services, residents would adopt new workflows in the EHR (e.g., using the “notes” tab frequently). We found that this switch in workflow during rotation to other services created barriers to use of the CDS since the newly adopted workflows did not support use of PE Dx. Finally, several physicians mentioned that they did not need the CDS because they had the risk criteria memorized (code: gestalt and memorization of criteria), and physicians also forgot to use the CDS because there was not an alert reminding physicians to use it (code: alerts).

5. Discussion

CDS has the potential to improve patient safety and care quality (Garg et al., 2005; Hunt et al., 1998), yet poor usability has hindered widespread use of these tools (Patterson et al., 2019). In this study, we interviewed 12 emergency physicians following the implementation of an HF-based CDS in the ED. We identified 294 excerpts describing usability barriers and facilitators of the CDS use in clinical workflow, i.e. “in the wild”. Many of the excerpts related to HF principles explicitly considered in the design of the CDS (Table 2). In addition, we inductively identified 18 categories of barriers and facilitators not related to the HF principles (Table 3).

We expand our understanding of the usability of HF-based CDS in the real clinical environment. Given the persistent usability challenges of CDS (Gong & Kang, 2016), and more broadly health IT (Ratwani et al., 2019), numerous agencies have recommended the use of HF principles and methods in the design of CDS (The Office of the National Coordinator for Health Information Technology, 2020). Several studies have previously demonstrated that HF methods and principles produced highly usable technologies in simulated environments (Carayon et al., 2020; Russ et al., 2014). Yet, few studies have evaluated the usability of HF-based technologies implemented in the real clinical environment (Catchpole et al., 2022; Salwei, Hoonakker, et al., 2022). This study expands our understanding of the usability outcomes of HF-based CDS when implemented.

5.1. Impact of HF principles on CDS usability

We demonstrate the value of systematically applying HF principles in the design of health IT. When asked about usability barriers and facilitators of PE Dx, physicians frequently described aspects related to the HF design principles that were considered in the design process and incorporated in the design of PE Dx, such as automation of information acquisition and consistency. Further, physicians mostly described these as facilitators to use of PE Dx, demonstrating value of HF-based design. Overall, the most frequently discussed topic by physicians was minimization of workload; this principle was systematically considered in the PE Dx design and resulted in numerous facilitators when the PE Dx was implemented. HF principles can improve the usability of CDS when implemented “in the wild”.

Notwithstanding the observed benefits of HF design, we still identified some HF usability barriers. Physicians most frequently discussed barriers relating to the principles minimization of workload and chunking/grouping. For instance, physicians did not like the location of PE Dx in the “ED navigator” section of the EHR. This perhaps highlights gaps or limitations in the design process that could be improved. Some physicians also did not like how the CDS forced them to use the Wells’ criteria before the PERC rule. While skipping the Wells’ may appear to be an efficient short-cut to rule out PE, it presents a safety concern as physicians may get an inaccurate risk assessment for a patient, resulting in over- or under-treatment. Therefore, grouping the Wells’ and PERC separately represents a tension between efficiency and safety. In the human-centered design process used to develop PE Dx, this tension was carefully considered; the design team decided to force the use of Wells’ before PERC in order to support safety. Such design tradeoffs can be challenging to manage.

5.2. Emergent categories related to CDS usability

This study highlights important factors that should be considered in future HF design. We identified 18 categories of usability barriers and facilitators that were not explicitly incorporated in the design of PE Dx. Perhaps unsurprisingly, most of these categories were described by physicians as barriers. These categories represent opportunities to improve future HF design processes for CDS. For instance, several categories related to teamwork (e.g., attending-resident tasks, teaching tool, physician-patient workflows). To date, most health IT design has focused on use of technologies by individuals rather than by teams (Carayon & Hoonakker, 2019), which has contributed to low adoption and use (Walker & Carayon, 2009). Focusing on these factors during CDS design may improve CDS usability when implemented. Additionally, we found that some participants viewed features of PE Dx as facilitators while others perceived the same feature as a barrier, for example, the integration of the CDS in the EHR and the lack of an alert to use the CDS. Some physicians liked that the CDS was integrated in the EHR as it was easy to access, whereas other physicians reported that they preferred a CDS outside of the EHR since then they did not need to exit their task (e.g., writing notes) to use the CDS. Similarly, some physicians wanted an alert to remind them to use the CDS whereas others did not like alerts and were glad the CDS did not pop-up and interrupt their workflow. These divergent perspectives present a challenge in design. CDS developers should consider what level of flexibility and customization is feasible and also which design best supports performance (e.g., efficiency, safety).

5.3. Integration in the work system

We found that many of the usability barriers related to the “physical environment” and “organization” work system elements (Smith & Carayon-Sainfort, 1989) as well as integration of PE Dx within physicians’ broader workflow (Carayon et al., 2012; Salwei et al., 2021). For instance, the codes mobile workflow and computer access described aspects of physician workflow and the physical environment that hindered use of PE Dx. It can be challenging to fully understand the impact of the physical environment and organizational factors on CDS use prior to implementation. Therefore, it is important to continue monitoring use of the CDS after go-live to support a continuous design process (Carayon, 2006, 2019; Carayon et al., 2017; Carayon et al., 2008). For instance, could PE Dx be re-designed to be accessible via an app on physicians’ phones in addition to on the computer? Could the computers in patient rooms be updated to ease the use of PE Dx in patient rooms? The design process does not end when the technology is implemented.

Several other codes related to organizational factors that influenced CDS use, such as the role of residents in ED care. One frequently mentioned barrier, integration of multiple CDS, related to the usability of PE Dx in the context of physicians’ broader workflow; while PE Dx supported tasks for PE diagnosis, it did not support physicians’ broader workflow of considering other potential diagnoses for a patient, such as pneumonia. This demonstrates that CDS usability is dependent, not only the technology’s interface, but also on its fit within the broader work system. CDS designers should expand the HF methods used to promote explicit consideration of the physical environment, organizational context, and broader workflow of clinicians in CDS design. Methods such as organizational scenarios (Clegg et al., 1996; Hughes et al., 2017), role network analyses (Salwei, Carayon, Hundt, Hoonakker, et al., 2019; Salwei, Carayon, Hundt, Kleinschmidt, et al., 2019), and SEIPS-based process maps (Carayon et al., 2006; Wooldridge et al., 2017) could be used to support broader consideration of clinical workflow during health IT design.

5.4. Limitations and future research

One limitation of this study is that we had a small sample size (12 physicians); however, we determined that data saturation had been reached with this number. Another limitation is that this study took place in one ED of an academic medical center; the results may not apply to other types of CDS and those in other, non-ED settings. Future work should explore the usability outcomes of diverse types of CDS implemented in differing clinical settings. Another limitation is that we did not measure actual use of the tool, only its usability; future work should leverage EHR data to explore use of PE Dx. Another limitation is that the interviews were conducted with individuals rather than with the whole team that cares for a patient in the ED; future studies should further explore how teams use CDS together to manage patient care. This study demonstrates the impact of HF principles on the usability of PE Dx used “in the wild”. However, we still do not know how the specific methods used in the design process (e.g., needs assessment, heuristic evaluation, scenario-based evaluation) impacted the CDS design (Salwei et al., Under revision). Future research should further explore the design process to better understand how the HF principles and methods led to our identified usability outcomes.

6. Conclusion

While HF principles and methods have been recommended to improve CDS usability, there remains limited information on the usability of HF-based CDS “in the wild”. We identified the usability barriers and facilitators of an HF-based CDS implemented in the ED and demonstrated the value of HF principles in the design of CDS. This study highlights other important factors, such as the physical environment and organizational context, that should be considered in future HF-based design; this should further enhance CDS usability, adoption, and impact on patient outcomes.

7. Implications and applications

This work demonstrates the value of HF principles in the usability of CDS and identifies areas for improvement to future human-centered design of CDS. Explicit consideration of HF principles in the design of CDS improves usability; HF principles (e.g., consistency, explicit control) should be identified and systematically applied throughout the design process. Several factors emerged that were not considered by the design team that influenced the usability of the CDS in the ED. Many of these factors related to the physical environment and organization work system elements. We recommend that additional methods, such as organizational scenarios (Hughes et al., 2017) and SEIPS-based process maps (Wooldridge et al., 2017), should be used in the HCD process to adequately incorporate all of the work system elements in the design of health IT. Health IT designers and developers can leverage our description of the 18 categories of barriers and facilitators that emerged when the CDS was implemented to inform future CDS design. For instance, the following should be considered during design: How will clinician workflow within the physical environment impact use of the CDS? Where and when do clinicians have access to computers? Does this align with the intended CDS workflow? Do physicians in training use differing workflows when training in other services? Will this impact their use of the CDS? How does teamwork influence use of the CDS? Does the CDS support training of other clinical staff? These questions can guide design to support clinician workflows. Finally, we recommend that designers carefully monitor CDS use following implementation to identify and address issues that emerge.

8. Impact statement

Health IT, such as CDS, has the potential to improve patient safety and reduce clinician workload, however, usability challenges have limited the effectiveness of these tools. While HF principles and methods remain the gold standard for health IT design, there is limited information on how HF methods and principles influence CDS usability “in the wild”. This study expands our understanding of the usability outcomes of HF-based CDS in the real clinical environment. These findings can be used to improve the design of future health IT, such as CDS, which can improve the usability and impact of these tools on patient care.