Application of human factors to improve usability of clinical decision support for diagnostic decision-making: a scenario-based simulation study

Abstract

Objective

In this study, we used human factors methods and principles to design a clinical decision support (CDS) that provides cognitive support to the pulmonary embolism (PE) diagnostic decision-making process in the emergency department. We hypothesized that the application of human factors methods and principles will produce a more usable CDS that improves PE diagnostic decision-making, in particular decision about appropriate clinical pathway.

Materials and methods

We conducted a scenario-based simulation study to compare a human factors-based CDS (so-called PE-Dx CDS) with a web-based CDS (MDCalc); 32 emergency physicians performed various tasks using both CDS. PE-Dx integrated human factors design principles such as automating information acquisition and analysis, and minimizing workload. We assessed all three dimensions of usability using both objective and subjective measures: effectiveness (e.g. appropriate decision regarding the PE diagnostic pathway), efficiency (e.g. time spent, perceived workload) and satisfaction (perceived usability of CDS).

Results

Emergency physicians made more appropriate diagnostic decisions (94% with PE-Dx; 84% with web-based CDS; p<0.01) and performed experimental tasks faster with the PE-Dx CDS (on average 96 seconds per scenario with PE-Dx; 117 seconds with web-based CDS; p<0.001). They also reported lower workload (p<0.001) and higher satisfaction (p<0.001) with PE-Dx.

Conclusions

This simulation study shows that human factors methods and principles can improve usability of CDS and diagnostic decision-making. Aspects of the human factors-based CDS that provided cognitive support to emergency physicians and improved diagnostic performance included automation of information acquisition (e.g. auto-populating risk scoring algorithms), minimization of workload, and support of decision selection (e.g. recommending a clinical pathway). These human factors design principles can be applied to the design of other CDS technologies to improve diagnostic safety.

BACKGROUND AND SIGNIFICANCE

Diagnostic errors have emerged as a critical patient safety issue.[1-4] Technological solutions such as CDS have the potential to improve diagnosis and reduce diagnostic errors.[1 5] However, in general, CDS adoption, use and impact have been limited,[6] partly because of poor usability and workflow integration.[7-10] Human factors (HF) methods and principles can enhance CDS usability, improve workflow integration and positively impact patient safety.[8 12] This is especially important in settings such as the emergency department (ED) where diagnostic decisions, e.g. making a pulmonary embolism (PE) diagnosis, are often made under time pressure and in chaotic environments,[13] therefore leading to diagnostic errors.[14] In this study, we applied a HF-based approach to design a CDS to support PE diagnostic decision-making in the ED. The study objective was to assess whether a HF-based approach improves CDS usability, including cognitive support to PE diagnostic decision-making, efficiency, workload and satisfaction.

CDS Technologies for Improving Diagnosis

Health information technologies (IT), including CDS, can facilitate access to patient-related information and support diagnostic decision-making.[1 5 15 16] Several systematic reviews [17-19] indicate the potential for CDS technologies to improve ED diagnostic processes. However, the evidence on the benefits of CDS for ED patient outcomes is rather weak,[17 19 20] often because the technologies are neither accepted nor used.[21] For instance, studies have demonstrated the positive (albeit small) impact of CDS on appropriate ordering of Computed Tomography Angiography (CTA) for PE diagnosis;[19 22 23] but many ED physicians (nearly 27% in one study [22]) may not use the CDS because of the time and data entry requirements. Achieving improvements in diagnostic safety with CDS implementation partly depends on the usability of CDS technologies and their integration into clinical workflow.[1 5 24] For PE diagnosis, CDS should support the diagnostic pathway and associated cognitive activities to avoid ordering wrong or uncessary tests.[23] Indeed, suspected PE is a challenging diagnostic scenario in the ED, with the morbidity and mortality of delayed or missed diagnoses [25 26] needing to be weighed against potential harms of overtesting.[27] In a study of 328 patients receiving CTA for PE diagnosis,[28] emergency physicians indicated reasons for ordering a CTA: to confirm or rule out PE (93%), elevated D-dimers (66%) and fear of missing PE (55%). This latter reason (fear of missing PE) may represent a defensive behavior, which can contribute to unnecessary radiation exposure.[29] Systematic risk assessment of patients using validated rules such as the Wells’ score and the PERC rule [30] is recommended so that PE is appropriately diagnosed in a timely manner [31] while avoiding unnecessary CTA scanning and radiation.[29] CDS can support the diagnostic process of emergency physicians when suspecting PE by suggesting appropriate clinical pathways, which are in concordance with these validated rules (e.g. ordering a D-dimer test for patient with moderate PE risk as opposed to proceeding directly to CTA).

Human Factors-based Design of Health IT

Health IT, such as CDS, need to incorporate HF methods and principles to improve usability.[11 32-34] Clinical decision-making such as a diagnostic process involves cognitive activities of acquiring and analyzing information and using this information to select a decision.[1] HF experts [35] recommend different levels of automation for cognitive activities at different information processing stages. High automation is useful for efficient information acquisition and analysis; the stage of decision selection should be supported by the CDS but not fully automated to preserve clinician judgment. Those HF design principles and usability heuristics, e.g. consistency and visibility,[36 37] should be used in designing CDS. See Table 1 for the HF design principles applied to the CDS tested in this study.

Table 1 –

Human Factors Design of PE-Dx CDS

Human Factors Design Process	Design Activities and Outcomes
1. Human factors analysis of PE diagnosis in the ED	Identification of multiple work system barriers and facilitators to cognitive work involved in PE diagnosis; in particular technology-related barriers for information gathering activities in PE diagnostic process.
2. Analysis of suggestions made by ED physicians regarding health IT solutions	Main suggestions for CDS to support PE diagnosis in ED included: Integration with EHR Auto-population Automated scoring Support for decision regarding order.
3. Nine interdisciplinary design sessions over one-year for a total of 13.5 hours (each session was between 1 and 2 hours)	Decision by design team to focus the CDS on how to support diagnostic pathway for PE diagnosis. Multiple disciplines involved in design sessions: human factors, emergency medicine and software engineering/programming. Use of iterative design process with multiple mock-ups of PE-Dx CDS (created in PowerPoint).
4. Two one-hour focus groups with four ED physicians (interspersed between design sessions) for review of PE-Dx CDS mock-up	Confirmation of preference for two-step CDS: presentation of Wells’ and then PERC; match with hospital policy. Use of highest heart rate/pulse value and lowest O2 saturation value; but need to allow physicians to update these values.
5. Creation of PE-Dx CDS in EHR Playground	Programming of PE-Dx CDS in EHR Playground based on PowerPoint mock-up.
6. Two-hour participatory heuristic evaluation of PE-Dx CDS with 7 participants (human factors engineers, physicians, systems analyst)	Identification of 19 usability problems; 13 problems resolved through design changes to PE-Dx CDS; 6 problems not addressed because of constraints of flowsheet functionality of local EHR. Production of final PE-Dx CDS tested in usability evaluation.
Human Factors Design Principles	Implementation in PE-Dx CDS
Automation of information acquisition [35]	Auto-population of some Wells’ and PERC criteria using EHR data.
Automation of information analysis [35]	Computation of Wells’ score by CDS.
Support of decision selection [35]	Provision of recommendation for next step in diagnostic pathway, e.g. doing nothing, ordering D-dimer, ordering CTA scan.
Explicit control/flexibility [35 36]	Ability to change values for Wells’ criteria – e.g. possible to change heart rate/pulse – to support physician clinical judgment and unique patient situations.
Minimization of workload [36]	Minimization of data entry; e.g. data for Wells’ automatically populated in PERC. PERC appears only if Wells’ score is low. No need to enter data for all PERC criteria once any PERC criterion is positive.
Human Factors Design Process	Design Activities and Outcomes
	Automatic generation of text for documentation of medical decision-making.
Consistency [36 37]	Consistency with how information is presented in other parts of the EHR; e.g. use of Yes/No toggle. Consistency with how Wells’ and PERC criteria are listed on MDCalc website that is routinely used by physicians.
Chunking/grouping [36 53]	Wells’ criteria and PERC rule presented separately. Placement of CDS in ED navigator of EHR.
Visibility [37]	Indication of points/weights assigned to each Wells’ criterion to make it clear/transparent how Wells’ score is computed.
Error prevention [36 37]	In order to avoid documenting wrong Wells’ score, all Wells’ criteria must be addressed.

Few studies have demonstrated the value of HF-based design in actually improving the usability and safety of CDS technologies. For instance, a simulation study [38] showed how the redesign of medication alerts based on HF design principles (e.g. proximity compatibility) improved usability, e.g. enhanced efficiency and reduced prescription errors. Studies that incorporate HF methods and principles have focused on medication safety [38 39], preventive care (e.g. colorectal cancer screening [40]), and clinical reminders.[41] The value of HF-based design of CDS to support the diagnostic decision-making process has been suggested,[1 42-44] but has not been demonstrated yet.

Study Objectives

Using HF methods and principles, we created a CDS (known as PE-Dx CDS) to support the PE diagnostic pathway for ED physicians. The study objective was to assess whether the HF-based approach improved CDS usability and diagnostic decision-making as compared to a widely used web-based CDS. We assessed the new CDS (PE-Dx) and the web-based CDS on the three usability dimensions defined by the International Organization for Standardization [45] (effectiveness, efficiency, satisfaction) and tested the following hypotheses:

• Hypothesis 1 (effectiveness). Because PE-Dx provided cognitive support for the diagnostic clinical pathway, it would be more likely to lead to appropriate decision regarding diagnostic pathway, and greater confidence in decision as compared to the web-based CDS.

• Hypothesis 2 (efficiency). Because the PE-Dx was designed according to HF principles (e.g. automation and workload minimization), it would be more efficient and would lead to faster performance, fewer clicks and scrolls, and lower perception of workload as compared to the web-based CDS.

• Hypothesis 3 (satisfaction). Because the design of the PE-Dx CDS incorporated HF methods and principles, it would be more satisfying and produce a more positive score of perceived usability as compared to the web-based CDS.

METHODS

Organizational Setting

We conducted an experimental study in collaboration with the ED of a large US academic teaching hospital, which has been using an EHR (Epic Systems) since 2008. The study was approved by the hospital’s institutional review board. We used the STARE-HI guidelines to write this article.[46 47]

Description of the Clinical Decision Support for PE Diagnosis (PE-Dx CDS)

While there are several strategies for evaluating patients presenting with symptoms of PE, validated clinical guidelines both nationally and specifically in use in the study ED suggest a tiered approach using two risk scores: the Wells’ score [48] and PERC (pulmonary embolism rule-out criteria).[49 50] These scores are available on websites and apps, e.g. MDCalc, a widely used source for medical scores and algorithms (www.mdcalc.com). Patients are initially screened with the Wells’ score. If the Wells’ score is higher than 6, PE risk is high and cross-sectional imaging such as CTA should be ordered. If the Wells’ score is between 2 and 6, PE risk is moderate and guidelines recommend ordering a D-dimer (a blood test which, when normal, effectively excludes PE in intermediate to low risk scenarios, and if positive prompts cross-sectional imaging). If the Wells’ score is lower than 2, PE risk is low. In this case it is appropriate to apply the PERC. If any PERC criteria are positive, a D-dimer is recommended; if all PERC criteria are negative, a PE diagnosis can be excluded without any further work-up.[30 49 51] An objective of this staged diagnostic process recommended by the American College of Physicians [49] is to reduce patients’ exposure to radiation.

Using a HF-based approach, we created PE-Dx, a computerized CDS integrated in the EHR to support the PE diagnostic pathway. The human-centered design process relied on multiple, iterative steps that combined work system analysis, design sessions, focus groups and a heuristic evaluation,[52] and incorporated HF design principles (Table 1).

Using the flowsheet functionality in Epic, a clinical systems analyst of the participating hospital programmed the PE-Dx CDS based on a PowerPoint mock-up developed in the HF design process (Table 1). PE-Dx incorporated Wells’ criteria for PE and the PERC rule in accord with hospital policy and established guidelines.[49] The clinician first considered the results of the Wells’ criteria and, when a patient was “Wells’ low,” then s/he completed PERC. Data available in the EHR were used to autopopulate the PE-Dx CDS. Some autopopulated fields could be overridden by physicians (i.e., heart rate and O2 saturation). A recommendation regarding the diagnostic pathway appeared when the final criterion was completed. If the Wells’ score was “low” (i.e., < 2), the autopopulated PERC criteria appeared. An affirmative response to any of the PERC criteria concluded the process and a recommendation appeared to order a D-dimer. See Figure 1 for screenshots of the CDS, and Table 2 for a comparison of the usual diagnostic process (MDCalc) and the PE-Dx CDS process.

Figure 1 –

PE-Dx CDS: Wells’ Criteria, Wells’ Score and Recommendation (A); PERC Criteria (B)

Table 2 –

Comparison of Diagnostic Decision Pathway With MDCalc and PE-Dx CDS

	MDCalc (web-based CDS)	PE-Dx CDS
Integration with EHR
	Not integrated in EHR; separate website.	Integrated in EHR.
Wells’ score (7 criteria)
Initiation	User needs to open browser, type MDCalc PE in search engine in browser and click on URL for MDCalc website.	User clicks on PE-Dx button in ED Navigator.
Criteria responses	User does not have to answer all (7) criteria before obtaining a score.	User has to address all (7) criteria before obtaining the Wells’ score.
Auto-population	No criteria are auto-populated; therefore, the user may need to navigate between the EHR and MDCalc to search for relevant information.	One criterion (heart rate > 100) is auto-populated; user may need to navigate within the EHR to search for other relevant information.
PERC rule (8 criteria)
Initiation	User must access a different website within MDCalc to review PERC criteria.	PERC is integrated in CDS and becomes active when indicated per hospital policy.
Criteria responses	User must respond yes to only one criterion to receive PERC positive result.	User must respond yes to only one criterion to receive PERC positive result.
Auto-population	Does not auto-populate, nor copy results from Wells’ criteria. Therefore, user may need to navigate between the EHR and MDCalc to search for relevant information.	Auto-populates 3 questions (age, heart rate, O2 saturation) and copies forward responses from comparable Wells’ criteria (prior PE or deep vein thrombosis, hemoptysis). Therefore, user must address only 3 criteria (recent trauma or surgery, exogenous estrogen, unilateral leg swelling).
Risk assessment and decision-making
Result/score	Provides the user with a score (e.g. 3.0 points for Wells’ score) and compares that score with the population (e.g. moderate risk group: 16.2% chance of PE in ED population).	Provides the user with a score.
Recommendations/next steps	Provides the user with suggestion for next steps; suggestion uses two different models, which can be confusing.	Provides the user with a single recommendation in accord with hospital policy and established guidelines.[49]
Order *	Does not automatically provide option to enter an order.	Provides the option to go to CPOE to order D-dimer or CTA scan when applicable.
Documentation *	Allows user to copy results (summary and scores on individual questions) into an EHR note, but user must register to use the feature.	Allows the user to populate the results and recommendation directly into his/her note.

^*The automated creation of an order and documentation of the decision were not tested in this study, but are available in the implemented version of PE-Dx CDS.

Study Design

We conducted a scenario-based simulation study with a repeated measures design. Participants were first exposed to the web-based CDS used in clinical practice (MDCalc; www.mdcalc.com) and then to the PE-Dx CDS in two separate sessions, each lasting approximately 30 minutes and occurring about one week apart. Five randomly ordered patient scenarios were presented during each session (Appendix for scenario details). The scenarios represented the range of decisions (no further action, order D-dimer, order CTA scan) that physicians make when suspecting PE, and all possible combinations of the Wells’ score and the PERC rule.[49] A physician on the study team drafted the scenarios, which were pilot tested with two other physicians. The scenarios were designed such that objectively presented data placed patients in one of five possible pathways, each with an appropriate decision:

1. Wells’ low (<2) / PERC negative; appropriate decision=no further action required

2. Wells’ low (<2) / PERC positive; appropriate decision=order D-dimer

3. Wells’ low (<2) / PERC positive due solely to age >49; appropriate decision=order D-dimer

4. Wells moderate (≥2 and ≤6), PERC inappropriate; appropriate decision=order D-dimer

5. Wells’ high (>6), PERC inappropriate; appropriate decision=order CTA scan.

Differences existed between the scenarios of session 1 and those of session 2 to minimize recall; differences were minor and not relevant for PE diagnosis. The creation of the scenarios integrated best practices outlined by Russ and Saleem.[54]

Participants

Twenty-four emergency medicine (EM) residents (eight from each of three residency classes) and eight EM faculty participated in the study. The study took place over a ten-week period from April through June 2018. Results of a power analysis showed that, for a power of 0.80 with 5 different scenarios, we needed a sample size of at least 21 participants to obtain an effect size of 0.70 with an alpha of 0.05; this was equivalent to recruiting 6 physicians in each of the four targeted physician groups (3 residency years and attendings). To ensure sufficient power and given the uncertainty around completion of both sessions by participants, we recruited 8 participants in each group, for a total of 32 physicians. Physicians were recruited via email, and all presented for both sessions. Total compensation of $100 was provided: $20 after session 1 and $80 after session 2.

Study Procedures

For study purposes, the PE-Dx CDS ran in the Epic “Playground,” an electronic environment that mimics the actual EHR with fictitious patients and is used for testing and training. Following a precise script (available upon request), trained researchers explained to the study participants the goal of the experiment, i.e. “to evaluate a new clinical decision support tool – a CDS – to facilitate making decisions about PE.” They then directed the study participants to complete the following tasks for each patient scenario: (1) review prepopulated ED notes containing relevant clinical information on the patient, (2) complete MDCalc (Wells’ and PERC when necessary) in session 1 and the PE-Dx CDS in session 2, (3) indicate on the paper survey what clinical pathway they would follow (order nothing, D-dimer or CTA) and their level of confidence with the decision, and (4) complete an electronic survey. After completing the final scenario, the electronic survey included additional questions. Upon completion of session 2, we conducted a short debrief interview to ask participants’ preference for either MDCalc or PE-Dx and perceived barriers and facilitators for implementing PE-Dx. Comments by participants during debriefing are used as quotes when discussing study results.

Data Collection and Measurement

We used multiple data collection methods to assess usability: screen capture software to video record task performance (Camtasia^®), stopwatch, paper survey and electronic survey.

Screen capture software

The screen capture software (Camtasia^®) started at the beginning of the session and recorded continuously every activity performed by participants until the end of the session. Researchers reviewed the video recordings and extracted data to assess:

• Number of clicks for each scenario. Counting of clicks used to navigate or choose/select an item began once the participant typed the patient’s name and stopped when the participant documented his/her decision.

• Number of scrolls for each scenario. Counting of uninterrupted scrolls on a page or side bar started and ended similar to the counting of clicks.

• Number of navigation elements used for each scenario. Navigation elements included windows in the EHR (e.g. ED provider note, medication list, medical history) and browser windows (e.g. MDCalc for Wells’) that physicians interacted with during navigation.

Stopwatch

Time to complete each scenario was measured with a stopwatch on a smart phone. Start time was defined as the moment that the participant started typing the patient’s name in ED dashboard. End time was defined as the moment that the participant made his/her choice for the diagnostic pathway and recorded it on the paper survey.

Paper survey after each scenario

After completing each scenario, participants filled out a two-question paper survey. The first question asked for their decision regarding the diagnostic pathway: order nothing, a D-dimer test or a CTA. Their decision was compared to the guideline-supported decision for that particular scenario; this produced the measure of decision appropriateness. In the second question, participants indicated their confidence in their decision about the diagnostic pathway, measured on a 100-millimeter visual analogue scale, ranging from 0 (no confidence at all) to 100 (very high confidence).

Electronic survey after each scenario

After completing the paper survey, participants filled out a short questionnaire on an iPad, programmed in the Qualtrics web-based survey software. The NASA Task Load (TLX) questionnaire [55] measured six dimensions of workload on a scale from 1 (low) to 10 (high): mental workload, physical workload, temporal workload, effort, frustration and performance. We adapted the question on physical workload to address physical activities of computer work, replacing the original text about pushing, pulling, turning, controlling, activating, etc. with text on navigating, scrolling, clicking, etc.

Electronic survey at the end of each session

At the end of each session, participants answered 17 questions from the Computer System Usability Questionnaire (CSUQ).[56] The original CSUQ has 19 questions, but two questions about error messages and online help were removed because MDCalc and PE-Dx did not have those features. The Cronbach’s alpha for the 17-item CSUQ was 0.95 in our study, which is similar to the reliability score in the original CSUQ study.[56] The electronic survey used at the end of each scenario and at the end of each session can be found at: XXX.

Data Analysis

All outcome measures were transformed for normalcy in the models used for data analysis, and for the marginal graphics were back transformed to the original metrics for easier interpretation. Data analysis was conducted using a three-level (physician-session-scenario) empirical Bayesian regression model, allowing us to model the pre-post and scenario dependencies. We used a Bayesian estimation framework to analyze our models. Bayesian methods are a useful alternative to iterative generalized least squares (IGLS) estimation in multilevel regression models, particularly with smaller samples.[57] To obtain the benefits of Bayesian methods, a prior distribution must be informative.[58] We estimated our effects with Empirical Bayesian (EB) Markov Chain Monte Carlo (MCMC) estimates,[59 60] using data-derived estimates from our dataset as our EB informative priors.[57 59 61] The Bayesian estimates and the analysis of our multilevel models were constructed using MLwiN Version 3.02. We used the three-level empirical Bayesian model to obtain the estimates of marginal means and report the comparison of outcome measures for session 1 (MDCalc) and session 2 (PE-Dx).

RESULTS

Participant Characteristics

Eight of the 32 (25%) study participants were women and most (88%) were between 25 and 34 years-old. Study participants used MDCalc primarily on the computer (75%) or their phone (25%). Most study participants used Wells’ score (81%) or PERC rule (91%) on MDCalc. All participants completed both sessions of the study.

Effectiveness

The percentage of appropriate decisions increased from 84% using MDCalc to 94% with the PE-Dx CDS (p<0.01) (Table 3). Physician confidence in their decision about the diagnostic pathway did not change with PE-Dx; confidence was high for both CDS (mean around 80 on a 0-100 scale).

Table 3 –

Impact of PE-Dx CDS on Usability (mean score [standard deviation], effect size [95% confidence interval] and p-value) [N=32]

Dependent Variables	MDCalc	PE-Dx CDS	Effect Size (CI)	p-value
Effectiveness
% appropriate decision	83.75% [0.37]	94.38% [0.23]	0.35a [0.03, 0.17]	p<0.01
Confidence level*	80.21 [18.94]	82.71 [18.21]	0.13b [−0.05, 0.31]	p=0.14
Efficiency
Time per scenario (in seconds)	117.37 [38.91]	95.84 [95.84]	−0.55b [−0.71, −0.38]	p<0.001
Number of clicks per scenario	16.49 [4.82]	17.90 [3.91]	0.29b [0.09, 0.48]	p<0.01
Number of scrolls per scenario	7.47 [2.92]	6.31 [2.50]	−0.39b [−0.57, −0.21]	p<0.001
Number of navigation elements per scenario	15.53 [5.01]	10.23 [2.74]	−1.05b [−1.26, −0.83]	p<.001
Perceived workload**	4.45 [1.53]	3.64 [1.78]	−0.52b [−0.68, −0.37]	p<0.001
Satisfaction ***
CUSQ – overall satisfaction	5.62 [0.94]	6.18 [0.66]	0.58b [0.18, 0.98]	p<0.001
CUSQ – system usefulness	5.65 [0.95]	6.26 [0.67]	0.62b [0.19, 1.05]	p<0.01
CUSQ – information quality	5.73 [0.98]	6.17 [0.73]	0.43b [0.03, 0.83]	p<0.05
CUSQ – interface quality	5.48 [1.15]	6.02 [0.86]	0.45b [0.07, 0.84]	p<0.05

^*One question: 0 (no confidence) to 100 (very high confidence)

^**Scale: 1 (low) to 10 (high) [NASA TLX Workload] [55]

^***Response categories: 1 (strongly disagree) to 7 (strongly agree) [CUSQ=Computer System Usability Questionnaire] [56]

^a= Cohen (h) effect size based on the inverse sine of the square root of the proportional values.

^b= Becker [62] adjusted effect size for repeated measures.

Efficiency

Efficiency was measured with five variables; four of them were favorably impacted by the HF-based CDS and one variable was worse with PE-Dx (Table 3). Physicians spent significantly less time per scenario with PE-Dx (117 seconds with MDCalc versus 96 seconds with PE-Dx; p<0.001). The difference of means was 20 seconds, which represented about 20% of time saved per scenario. Physicians used fewer scrolls per scenario (7 with MDCalc versus 6 with PE-Dx; p<0.001) and interacted with fewer interface elements with the HF-based CDS (16 elements with MDCalc versus 10 elements with PE-Dx; p<0.001). There were significant differences in the NASA TLX global and individual scores (except for performance) between the two CDS (Figure 2). PE-Dx was associated with lower overall perceived workload, as well as lower scores on mental workload, physical workload, temporal workload, effort and frustration. The number of clicks per scenario increased slightly with PE-Dx (16 with MDCalc versus 18 with PE-Dx; p<0.01).

Figure 2 –

Impact of PE-Dx CDS on Perceived Workload. Results are shown as mean+/−SEM. Higher scores indicate higher perceived workload. The differences on the NASA-TLX subscales were statistically significant, except for performance

Satisfaction

The CUSQ produced one overall satisfaction score, and three separate scores of (1) system usefulness, (2) information quality and (3) interface quality. The HF-based CDS performed better on all four measures (Table 3). Physicians were more satisfied with the usability of PE-Dx compared to MDCalc; including overall perceived usability, usefulness (e.g. “I can effectively complete my work using this system”), information quality (e.g. “It is easy to find the information I needed”) and interface quality (e.g. “The interface of this system is pleasant”).

Twenty-eight of the 32 participating physicians (88%) indicated their preference for PE-Dx. One physician would continue using MDCalc as s/he may not agree with the recommendation of PE-Dx. Three other physicians were unsure which CDS they preferred.

DISCUSSION

In this study, we used HF methods and principles to design a CDS that supports the PE diagnostic process and compared this CDS (PE-Dx) to an existing web-based CDS (MDCalc). Results demonstrated that the HF-based CDS produced better usability and improved the PE diagnostic process. The majority of our hypotheses were supported: 9 of the 11 variables of usability improved with PE-Dx (Table 3).

Improvements in both objective (e.g. time) and subjective (e.g. satisfaction) measures of usability were achieved with PE-Dx. Physicians reported greater satisfaction with PE-Dx, including usefulness, information quality and interface quality (Table 3). After being exposed to it, a majority of physicians (88%) indicated their preference for PE-Dx.

Cognitive Support of PE-Dx CDS to PE Diagnostic Decision-Making

PE-Dx was more effective than MDCalc as it provided cognitive support for the PE diagnostic pathway: 94% of physicians chose the appropriate clinical pathway when using PE-Dx (versus 84% with MDCalc). With MDCalc, physicians could decide not to follow the sequence of computing the Wells’ score first and then looking at PERC if appropriate. In a follow-up analysis,[63] we found that study participants followed the recommended workflow almost always with PE-Dx (98%), and only half of the time with MDCalc (51%). PE-Dx was built to guide physicians in a clinical workflow that started with the Wells’ score and continued with the PERC rule; this led to a clear recommendation to either do nothing, order a D-dimer or order a CTA.

PE-Dx helped physicians make a decision about which clinical pathway to take (HF design principle of support for decision selection).[35 43] However, in nine cases (out of 160 cases or 6%), physicians did not make the appropriate decision. In one case, the study participant made a data entry error, which led to a recommendation different from the one intended for this specific scenario. In seven cases, study participants chose no further testing whereas PE-Dx recommended a D-dimer as the Wells’ score was below 2 with PERC positive; in one case, the physician ordered a CTA whereas PE-Dx suggested a D-dimer. It is possible that in those 8 cases (out of 160 cases) physicians disagree with the recommendation from the CDS or did not trust the CDS.[64] But we are unsure why study participants made those decisions as we did not ask them to think aloud in order to obtain accurate estimates of time needed to complete tasks.

Using clinical decision and prediction rules has been identified as a health IT functionality to support making a diagnosis, ordering tests and determining a diagnostic plan;[5] this is what PE-Dx does. PE-Dx does not fully automate the decision selection, but provides a recommendation to the physician regarding next steps, i.e. doing nothing, ordering a D-dimer or ordering a CTA. This partial automation of decision selection is a HF design principle.[35] High automation at the stages of information acquisition and analysis is another HF design principle applied to PE-Dx: some criteria were auto-populated (information acquisition) and scores were automatically calculated (information analysis) (Table 1). Physicians liked this PE-Dx CDS functionality as indicated in a debrief interview: “I like that It pulls the vital signs in particular and also is smart enough to look at the entire vital signs from the entire encounter, another area in which I do feel more confident; that a computer would be able to do that more reliably than me with less error long-term.”

Saving Time and Reducing Workload with PE-Dx CDS

Physicians completed the scenarios about 20% faster with PE-Dx, and used fewer scrolls and navigation elements (e.g. browser windows, EHR elements). During a debrief interview, a resident confirmed the importance of saving time with PE-Dx: “It took the criteria from the Wells’ score that matched the PERC score and just auto-populated that as well, which is extremely, you know, efficient. That would help the workflow a lot. Instead of having to go back and forth in MDCalc and trying to type in the numbers between MDCalc and Epic, it would be much quicker.” However, we found that PE-Dx increased the number of clicks per scenario. On one hand, when using PE-Dx a physician needs fewer clicks because s/he does not have to leave the EHR, open a browser, and go back and forth between the EHR and MDCalc website (Table 2). On the other hand, PE-Dx requires more clicks as the physician needs to provide information for all seven Wells’ criteria, whereas MDCalc does not require the user to answer all questions before calculating the Wells’ score (Table 2). We made the decision to ask physicians to address all Wells’ criteria to avoid wrong decisions (principle of ‘error prevention’ in Table 1). Overall, the PE-Dx CDS had a slightly higher number of clicks per scenario than MDCalc (Table 3). Even though PE-Dx slightly increased clicks, physicians reported lower workload, except for the performance dimension (Figure 2).

Several HF design principles contributed to the impact of PE-Dx on efficiency, in particular auto-population and minimizing workload. For the Wells’ score, only one criterion was auto-populated from the EHR, and for PERC, 6 criteria were auto-populated. Future improvement in the PE-Dx CDS should include additional auto-population, e.g. using natural language processing. A benefit of PE-Dx was to use data entered in the Wells’ score to populate the PERC rule; this reduced data entry. As indicated above, automation of information acquisition is a HF design principle,[35] which was applied to PE-Dx. This was confirmed as a good approach by a participating physician: “It Is great that It Is all within [the EHR]… and easily accessible with one click. That it auto-populates is a huge function. To be able to take out that step of having to manually look up information and input information,… I felt [it] sped things up entirely and also made me more confident in the sense that it took out my own doubts about my ability to consistently transcribe information.”

The performance subscale of the NASA TLX produced relatively high scores (means around 7 on a 1-10 scale) for both CDS. Physicians perceived high satisfaction with their performance on the experimental tasks; this matched the data on confidence in decision that was also high and similar for both CDS (Table 3).

Evidence for Value of Human Factors-based Design

Our study provides empirical evidence for the value of HF-based design in increasing usability of health IT, such as CDS, and improving diagnostic decision-making. We showed that a human-centered design process integrating multiple HF methods and design principles (Table 1) produced a more usable CDS that provided cognitive support for the diagnostic process. The HF design principles implemented in PE-Dx (Table 1), in combination with other HF criteria such as those in heuristic usability evaluations,[36 37] could be applied to other CDS, in particular CDS targeting diagnostic processes that involve cognitive activities of information gathering, information integration and interpretation, and working diagnosis.[1]

The HF design principles can address the information fragmentation and overload of current EHR technologies, which contribute to physicians’ frustration, stress and burnout.[65] Our HF-based CDS improved efficiency (e.g. faster performance, lower perceived workload), although the number of clicks per scenario was slightly higher with PE-Dx (Table 3). These positive results were likely related to autopopulation, minimization of data entry and chunking/grouping, i.e. HF design principles aligned with suggestions made by ED physicians early in the design process (Table 1). Therefore, a HF-based design that involves analyzing the actual work of clinicians, participation from clinicians in the design process and implementation of HF design principles can enhance the usability of health IT such as CDS. As data continue to emerge about poor usability and workflow integration of health IT and adverse patient safety consequences,[66] it is essential to integrate HF methods and principles in technology design processes.[12 67-70] This is particularly important with the emergence of artificial intelligence (AI), which can be used to design CDS technologies that complement and enhance clinical decision-making. [71-73]

Strengths and Weaknesses of the Study

Findings of this scenario-based simulation study may not extend to the actual implementation of PE-Dx. When a technology is implemented in the real work environment, challenges may occur that affect technology acceptance and use. Oftentimes challenges are related to problems with technology usability and workflow integration, and misfit of the technology with the rest of the work system.[74] As our CDS technology was designed according to HF methods and principles, its usability was rated positively in our experimental study. But implementation issues may still arise. Therefore, future research should evaluate the actual implementation of PE-Dx, its use in the clinical environment, and its impact on key process and outcome measures, e.g. appropriate use of lab or imaging tests. This research should use qualitative and quantitative research methods,[75] as well as objective and subjective measures of usability.[16] This follow-up research can assess two functionalities of PE-Dx that were not tested in the simulation study: automated creation of lab test order (D-dimer) or imaging study (CTA) in CPOE, and automated documentation of decision in ED provider note. Because we focused on cognitive support provided in the diagnostic pathway, the experimental set-up and data collection stopped right after the physician made a diagnostic decision regarding next steps (order nothing, D-dimer or CTA).

Because the study took place in a single ED of a US academic hospital, it is difficult to generalize the results to other settings and emergency physicians. A strength of the study was its experimental approach; therefore, differences between the sessions are likely attributable to differences in the two CDS technologies (MDCalc and PE-Dx). Whereas scenarios were presented in a random order during each session, the order of CDS presentation was not counterbalanced: participants were first exposed to their current CDS (MDCalc) in session 1 and to PE-Dx in session 2 in order to assess the incremental impact of using the new HF-based CDS compared to usual practice. The lack of counterbalancing of conditions could affect our results if the participants’ performance improved over time with their involvement in the study, which is unlikely. Further bias may have occurred because neither participants nor researchers were blinded; however, this is unlikely as we used a repeated measures design and implemented study procedures with detailed scripting of instructions given to participants by trained researchers. Another strength is that improvements were shown on all usability dimensions: with the newly designed CDS, the diagnostic pathway of PE was more effective and efficient, and physicians perceived lower workload and were more satisfied with the technology.

CONCLUSION

Ensuring that human factors principles are incorporated into CDS design is critical to improve CDS usability, which includes supporting appropriate diagnostic pathway, saving time, reducing perceived workload, and improving physician satisfaction with the technology. This research suggests that human factors principles such as automating information acquisition (e.g. auto-population) and information analysis (e.g. computing risk scores), minimizing workload (e.g. minimizing data entry) and supporting decision selection are key to improving CDS usability and consequently diagnostic safety. Clinicians report greater satisfaction with a CDS designed according to human factors principles. Future research should examine the effectiveness of these human factors principles for other health IT and assess the implementation of PE-Dx and its clinical impact.

Appendix – Patient Scenarios

Patient Scenario 1: Wells’ low, PERC negative (session 1)

A 24 year old female presents to the ED with a chief complaint of cough and chest pain. For the past four days she has been having upper respiratory symptoms. She reports having a subjective fever, cough, runny nose, and a sore throat. She has chest pain with coughing that worsens with deep breaths. The pain is sharp and located on her right side. She rates the pain at 4 on a scale from 1 to 10 (4) while coughing or breathing deeply. She has no abdominal pain or vomiting.

Past Medical History

No pertinent medical history.

Past Surgical History

Wisdom teeth extracted age 17, no complications

Medications

None. She does not take oral contraceptive pills (OCPs).

Family History

No family history of blood clots or heart disease at an early age.

Social History

Never Smoker

VITALS

Temperature (Temp): 98.9

Pulse: 77

Respiratory Rate (RR): 14

Oxygen (O2) Saturation: 99% on room air

Blood Pressure (BP): 126/74

PHYSICAL EXAM

General appearance (Gen): well appearing

HEENT (head, ears, eyes, nose and throat): Pupils ERRL (equal, round, reactive to light), no pharyngeal exudate, neck supple

Chest: no chest wall ttp (tenderness to palpation), lungs CTAB (Clear to Auscultation Bilaterally), no increased work of breathing

CV: RRR (regular rate and rhytm), s1 +s2 nl (heart sounds normal), no m/r/g (no murmurs, rubs, or gallops)

Abdomen: soft, non-tender, non-distended

Neuro: no deficits

Ext: no edema, no calf tenderness

Patient Scenario 1: Wells’ low, PERC negative (session 2)

A 27 year old female presents to the ED with a chief complaint of cough and chest pain. For the past five days she has been having upper respiratory symptoms. She reports having a sore throat, cough, and subjective fever and runny nose. She has chest pain with coughing that worsens with deep breaths. The pain is sharp and located on her right side. She rates the pain at 4/10 while coughing or breathing deeply. She has no abdominal pain or vomiting.

Past Medical History

No pertinent medical history.

Past Surgical History

Wisdom teeth extracted age 17, no complications

Medications:

None. She does not take OCPs.

Family History

No family history of blood clots or heart disease at an early age.

Social History

Never Smoker

Family History

No family history of blood clots or heart disease at an early age.

VITALS

Temp: 97.7

Pulse: 68 RR: 16

O2 Sat: 98% on room air

BP: 134/85

PHYSICAL EXAM

Gen: well appearing

HEENT: Pupils ERRL, no pharyngeal exudate, neck supple

Chest: no chest wall tenderness to palpation, lungs CTAB, no increased work of breathing

Heart: RRR s1+s2 nl, no m/r/g

Abdomen: soft, non-tender, non-distended

Neuro: no deficits

Ext: no edema, no calf tenderness

Patient Scenario 2: Wells’ low, PERC positive (session 1)

A 38 year old male presents to the ED with chest pain. He lifted weights at the gym 4 days ago and reports that he woke up the next morning with chest pain located on his right side. The pain is sharp and constant. It is a 4/10 on the pain scale that worsens when he takes a deep breath. He reports no shortness of breath with his pain. He thinks the pain worsens when he is exerting himself, but is not sure. He has no fever or abdominal symptoms. He reports that he has never had pain like this before.

Past Medical History

Asthma, well controlled

Deep vein thrombosis (DVT), provoked 5 years ago, on anticoagulants for 6 months but off since then

Past Surgical History

Ankle ORIF (Open Reduction with Internal Fixation) 5 years ago c/b (complicated by) DVT

Medications

Albuterol

Family History

No family history of blood clots or heart disease at an early age.

Social History

½ pack per day smoker

VITALS

Temp 98.6

Pulse: 82 RR: 14

O2 Sat: 94% on room air

BP: 150/90

PHYSICAL EXAM

Gen: well appearing

HEENT: Pupils ERRL, no pharyngeal exudate, neck supple

Chest: Some right sided ttp about intercostals, lungs scattered wheezes b/l, no increased work of breathing

Heart: RRR s1+s2 nl, no m/r/g

Abdomen: soft, non-tender, non-distended

Neuro: no deficits

Ext: no edema, no calf tenderness

Patient Scenario 2: Wells’ low, PERC positive (session 2)

A 36 year old male presents to the ED complaining of chest pain for the past 3 days. He lifted weights at the gym 3 days ago and reports that he woke up the next morning with chest pain located on his right side. The pain is sharp and constant. It is a 4/10 on the pain scale that worsens when he takes a deep breath. He reports no shortness of breath with his pain. He thinks the pain worsens when he is exerting himself, but is not sure. He has no fever or abdominal symptoms. He reports that he has never had pain like this before.

Past Medical History

Asthma, well controlled

DVT, provoked 5 years ago, on anticoagulants for 6 months but off since then

Past Surgical History

Ankle ORIF (Open Reduction with Internal Fixation) 5 years ago c/b (complicated by) DVT

Medications

Albuterol

Family History

No family history of blood clots or heart disease at an early age.

Social History

½ pack per day smoker

VITALS

Temp: 98.6

Pulse: 82 RR: 14

O2 Sat: 94% on room air

BP: 150/90

PHYSICAL EXAM

Gen: well appearing

HEENT: Pupils ERRL, no pharyngeal exudate, neck supple

Chest: Some right sided ttp (tenderness to palpation) about intercostals, lungs scattered wheezes b/l, no increased work of breathing

Heart: RRR s1+s2 nl, no m/r/g

Abdomen: soft, non-tender, non-distended

Neuro: no deficits

Ext: no edema, no calf tenderness

Patient Scenario 3: Wells’ low, PERC positive due to age alone (session 1)

A 65 y/o female presents to the ED with a chief complaint of cough and chest pain. For the past four days she has had a dry cough with no other infectious symptoms, and since yesterday she has noticed a chest pain with coughing that also worsens with deep breaths. The pain is sharp and located on her right side. She rates the pain at 4/10 while coughing or breathing deeply. She otherwise feels well with no shortness of breath, inability to lie flat, or decreased exercise tolerance. She denies fevers, and has had no abdominal pain or vomiting.

Past Medical History

HTN (hypertension)

HL (hyperlipidemia)

Past Surgical History

None

Medications

HCTZ (hydrohlorothiazide)

Aspirin

Family History

No family history of blood clots or heart disease at an early age.

Social History

Never smoked

VITALS

Temp: 98.9

Pulse: 77 RR: 14

O2 Sat: 99% on room air

BP: 126/74

PHYSICAL EXAM

Gen: well appearing

HEENT: Pupils ERRL, no pharyngeal exudate, neck supple

Chest: no chest wall tenderness to palpation, lungs CTAB, no increased work of breathing

CV: RRR s1+s2 nl, no m/r/g

Abdomen: soft, non-tender, non-distended

Neuro: no deficits

Ext: no edema, no calf tenderness

Patient Scenario 3: Wells’ low, PERC positive due to age alone (session 2)

A 67 y/o male presents to the ED with a chief complaint of cough and chest pain. For the past four days he has had a dry cough with no other infectious symptoms, and since yesterday he has noticed a chest pain with coughing that also worsens with deep breaths. The pain is sharp and located on his right side. He rates the pain at 4/10 while coughing or breathing deeply. He otherwise feels well with no shortness of breath, inability to lie flat, or decreased exercise tolerance. He denies fevers, and has had no abdominal pain or vomiting.

Past Medical History

HTN

HL

Past Surgical History

None

Medications

HCTZ

Aspirin

Family History

No family history of blood clots or heart disease at an early age.

VITALS

Temp: 97.7

Pulse: 68 RR: 16

O2 Sat: 98% on room air

BP: 134/85

PHYSICAL EXAM

Gen: well appearing

HEENT: Pupils ERRL, no pharyngeal exudate, neck supple

Chest: no chest wall tenderness to palpation, lungs CTAB, no increased work of breathing

Heart: RRR s1+s2 nl, no m/r/g

Abdomen: soft, non-tender, non-distended

Neuro: no deficits

Ext: no edema, no calf tenderness

Patient Scenario 4: Wells’ moderate, PERC inappropriate (session 1)

A 46 y/o male presents to the ED with chest pain. He reports having chest pain and upper respiratory symptoms for 1 week. He said he was feeling well until about a week ago when he noticed some right side chest pain and cough. Initially, he thought it was a cold but it hasn’t gone away. The pain is worse with breathing. His pain persists and he expels clear mucus when coughing. Today he noticed a few streaks of bright red blood in the mucus. He thought he would feel better and could shake the cough but the blood worried him, so he came to the ED.

Past Medical History

DVT, provoked 10 years ago, on anticoagulants for 6 months but off since then

Past Surgical History

Femur Fx (Fracture) after MVC (motor vehicle collision) 10 years ago complicated by DVT

Medications

None

Family History

No family history of blood clots or heart disease at an early age.

Social History

Never smoked

VITALS

Temp: 98.7

Pulse: 70 RR: 12

O2 Sat: 99% on room air

BP: 140/82

PHYSICAL EXAM

Gen: well appearing

HEENT: Pupils ERRL, no pharyngeal exudate, neck supple

Chest: no cw ttp, lungs with scattered rhonci b/l, no increased wob

Heart: RR s1+s2 nl, no m/r/g

Abdomen: soft, non-tender, non-distended

Neuro: no deficits

Ext: no edema, no calf tenderness

Patient Scenario 4: Wells’ moderate, PERC inappropriate (session 2)

A 46 y/o female presents to the ED with chest pain. She reports having chest pain and upper respiratory symptoms for 1 week. She said she was feeling well until about a week ago when she noticed some right side chest pain and cough. Initially, she thought it was a cold but it hasn’t gone away. The pain is worse with breathing. Her pain persists and she expels clear mucus when coughing. Today she noticed a few streaks of bright red blood in the mucus. She thought she would feel better and could shake the cough but the blood worried her, so she came to the ED.

Past Medical History

DVT, provoked 10 years ago, on anticoagulants for 6 months but off since then

Past Surgical History

Femur Fx (Fracture) after MVC (motor vehicle collision) 10 years ago complicated by DVT

Medications

None

Family History

No family history of blood clots or heart disease at an early age.

Social History

Never smoked

VITALS

Temp: 98.7

Pulse: 70 RR: 12

O2 Sat: 99% on room air

BP: 140/82

PHYSICAL EXAM

Gen: well appearing

HEENT: Pupils ERRL, no pharyngeal exudate, neck supple

Chest: no cw ttp, lungs with scattered rhonci b/l, no increased wob

Heart: RR s1+s2 nl, no m/r/g

Abdomen: soft, non-tender, non-distended

Neuro: no deficits

Ext: no edema, no calf tenderness

Patient Scenario 5: Wells’ high, PERC inappropriate (session 1)

A 62 y/o female presents to the ED with increased shortness of breath. She is three weeks post-surgery for ovarian cancer. She reports shortness of breath for the past 3 days. She is currently taking neoadjuvant chemotherapy but is unsure exactly what meds she is on. She was feeling better after her surgery but three days ago awoke with shortness of breath and found it difficult to perform activities of daily living without being breathless. She reports having tightness in her chest but no pain She denies fever or other symptoms.

Past Medical History

Ovarian CA (cancer), on neoadjuvant Chemotherapy

No hx of DVT/PE

Past Surgical History

TAH/BSO (total abdominal hysterectomy with bilateral salpingo-oopherectomy) for ovarian CA (cancer) 3 weeks prior

Medications

None other than chemotherapy

Family History

No family history of blood clots or heart disease at an early age.

Social History

Former smoker, quit when diagnosed with CA

VITALS

Temp: 98.6

Pulse: 115 RR: 20

O2 Sat: 91% on room air

BP: 95/60

PHYSICAL EXAM

Gen: uncomfortable appearing

HEENT: Pupils ERRL, no pharyngeal exudate, neck supple

Chest: no cw ttp, lungs scattered wheezes b/l, no increased wob

Heart: tachycardic s1+s2 nl, no m/r/g

Abdomen: soft, non-tender, non-distended

Neuro: no deficits

Ext: 1+ edema to R leg, 2+ to L, some calf ttp on the L side

Patient Scenario 5: Wells’ high, PERC inappropriate (session 2)

A 55 y/o male presents to the ED with shortness of breath. He was recently diagnosed with colon cancer and is 3 weeks post colon resection. He reports that 3 days ago he began having shortness of breath. He is currently taking neo-adjuvant chemotherapy but is unsure exactly what meds he is on. He reports that he was feeling better post-surgery but awoke 3 days ago with shortness of breath. He finds it difficult to perform activities of daily living without feeling breathless. He reports having tightness in his chest but no frank pain, fever or other symptoms.

Past Medical History

Colon CA, s/p (status post) recent resection with no known metastases

No hx of DVT/PE

Past Surgical History

Colon resection 3 weeks ago

Medications

FOLFOX chemo

Allergies: None Family History

No family history of blood clots or heart disease at an early age.

Social History

Former smoker, quit when diagnosed with CA

VITALS

Temp: 98.7

Pulse: 112 RR: 20

O2 Sat: 92% on room air

BP: 97/58

PHYSICAL EXAM

Gen: uncomfortable appearing

HEENT: Pupils ERRL, no pharyngeal exudate, neck supple

Chest: no cw ttp, lungs scattered wheezes b/l, no increased wob

Heart: tachycardic s1+s2 nl, no m/r/g

Abdomen: soft, non-tender, non-distended

Neuro: no deficits

Ext: 1+ edema to R leg, 2+ to L, some calf ttp on the L side