The Design of PE Dx, a CDS to Support Pulmonary Embolism Diagnosis in the ED

Abstract

Designing and implementing clinical decision support (CDS) in health care has been challenging. Attempts have been made to design and implement CDS to support clinical procedures, but many of these CDSs have met user resistance. One possible explanation for the lack of acceptability can be the poor design of the CDS. In this study, we describe the design of PE Dx, a CDS built to support the diagnosis of pulmonary embolism (PE) in the emergency department (ED) using human factors methods.

1. Introduction

Estimates show that between 650,000 and 900,000 individuals are diagnosed with a pulmonary embolism (PE) each year in the US, 200,000 of which will be fatal.[13] A PE is the sudden blockage of an artery in the lung, usually by a blood clot. If the clot is large and stops blood flow to a substantial portion of the lung, it can result in sudden death.[6] It is difficult to correctly diagnose a PE. Individual signs and symptoms are often not accurate enough to rule in or rule out a PE.[6] No single test is enough to diagnose a PE. The literature shows that less than 50% of patients who die from a PE were correctly diagnosed.[8] Several tests are needed to either rule out a PE or make the PE diagnosis. To support the diagnostic pathway for PE, risk-scoring algorithms have been developed.

1.1. PE diagnosis and PE Risk scoring algorithms

The most reliable method to diagnose a PE is the computerized tomography pulmonary angiography (CTA) scan. However, CTA scans are expensive and expose the patient to potentially hazardous radiation and renal injury because of the administration of IV contrast.[5; 11] Therefore, CTA scans should be avoided when possible.[11] The D-dimer test is a diagnostic test that has significantly less risk and a high negative predictive value in low to moderate risk patients.[12] However, D-dimer is limited by a relatively poor positive predictive value.[14] As CTA scans have excellent positive and negative predictive value, many physicians order them when they suspect a PE, resulting in over-testing. The number of CTA scans for PEs has dramatically increased in the past decade without a decrease in mortality.[10] There are several clinical decision aids that help physicians choose the appropriate clinical pathway to rule out PE or confirm it with tests. The most often used aids are the Wells’ criteria and the Pulmonary Embolism Rule-out Criteria (PERC) rule. Often these two decision aids are used together to rule out a PE diagnosis. The Wells’ criteria consist of seven questions. On two of the questions, you can get 3 points: “clinical signs and symptoms of PE” and “PE is #1 diagnosis or equally likely”. Three questions can result in 1.5 points (heart rate >100, immobilizations during last 3 days or surgery in previous 4 weeks, and previously, objectively diagnosed with PE or deep vein thrombosis (DVT)). Finally, two questions result in 1 point each: having been diagnosed with hemoptysis or malignancy with treatment in last 6 months. The answers to these seven questions result in a score that reflects the risk of a PE (see Figure 1). Depending on the risk, the clinician needs to either apply the PERC (low risk), proceed with D-dimer testing (moderate risk) or go directly to CTA (high risk). If a clinician responds affirmative on one of the 8 PERC questions, PERC is positive.

Figure 1.

Best practices for the PE rule in/rule out process, American College of Physicians [10]

1.2. CDS for PE in the emergency department

The emergency department (ED) environment uses many clinical decision rules and is therefore an ideal environment to design and implement CDS. As a result, EDs have seen explosive growth in CDS implementation. However, although most studies (84%) in a recent review by Patterson et al. [7] showed an impact of CDS on processes and process measures, few studies have shown an impact on clinical outcomes, such as length of stay (LOS) or readmissions. Several studies that have examined CDS to provide support for PE in the ED showed an impact of the CDS on the PE diagnostic process.[2; 3; 9–11] For example, studies by Raja [10; 11], Prevedello [9] and Jiminez [3] resulted in a significant reduction of CTA use. However, few ED clinicians use the CDS that were designed to improve the PE diagnostic process.[2; 4] In other words, results of studies on CDS for PE do show effectiveness but lack acceptance. Several authors suggest that poor CDS usability is the cause of low acceptance: “The computerized decision support system, however, was poorly accepted by emergency physicians (partly because of increased computer time), leading to possibly selective use, reducing the effect on overall yield, and leading to removal of the computerized decision support system from the computer order entry.”[2] Our study uses a sociotechnical systems approach to the design of CDS to support diagnosis of PE. This way, we can address the range of sociotechnical system issues that have limited the acceptance of prior CDS designed for PE. Based on a user-centered design process (an iterative process in which designers focus on the end users and their needs in each phase of the design process) with the participation of both human factors engineers and clinicians, CDS design requirements were developed. Those design requirements were embedded in CDS mock-ups; and the CDS mock-ups have been evaluated through scenario-based usability evaluation and group debriefings.

2. Methods (CDS tool development)

To better understand the PE diagnostic processes in the ED, in the early stages of this research project, we interviewed several ED clinicians. We also conducted detailed workflow studies to better understand the Wells’ and PE workflows. For the actual design of the CDS (so called PE Dx), we used design sessions, focus groups and usability evaluation. The study and associated data collection procedures were approved by the Institutional Review Board (IRB) of our university.

2.1. Design sessions

We conducted 9 design sessions in which the design team consisting of 6 human factors experts and 2 clinicians discussed advantages and disadvantages of several design solutions for PE Dx. Each design session lasted between 1 and 2 hours. In total, the 9 design sessions lasted 13.5 hours. During most of the design sessions, we used mock-ups of PE Dx created in MS PowerPoint. Based on the feedback received during a design session, we would make changes to the mock-up and discuss the redesign in the following design session.

2.2. Focus groups and interviews

We organized two focus groups and two interviews with clinicians to gather feedback on early designs of PE Dx and asked specific questions about particular parts of the design, such as which vital sign data to pull from the EHR, hover over/information buttons, error messages, and calculate buttons. Each focus group lasted about one hour. The interviews lasted about 30 minutes.

2.3. Survey

We conducted a survey for the design team to help make design decisions. The survey consisted of 20 questions. The first set of 4 questions was about possible triggers for the tool. The second set of 5 questions asked about the Wells’ criteria and the third set of 7 questions asked about the PERC rule. The next question was about placing orders and the final set of 3 questions asked about documentation.

2.4. From designing to building the CDS

This iterative process, using different methods, resulted in a design of PE Dx that was agreed upon by the whole design team. We then asked an electronic health record (EHR) programmer to program PE Dx in an existing EHR. After the CDS had been programmed, we conducted a heuristic usability session with the programmer to make PE Dx more user-friendly.

3. Results

User-centered design is an iterative process, and the design of PE Dx went through different stages. During those stages, decisions needed to be made to be able to move forward. In this paper, we focus on three major design decisions because we believe they not only had a major impact on the design of PE Dx, but will affect the design of other CDS as well: (1) considering the whole process: designing from start to finish, (2) auto-populating vs. supporting clinical autonomy, and (3) (auto) calculating.

3.1. Designing for the whole process, not only for a specific task

If an ED physician suspects a PE based on observed physical symptoms, they have several tools to rule in or rule out a PE. The tools provide support for the next steps of the process, such as ordering a D-dimer or a CTA scan, and documenting these steps. We believe that a CDS should also support these next steps, and for example, depending on the test results, provide the physician with an easily accessible option to order a D-dimer or CTA scan. Further, test results should be “automatically” documented in the doctor’s notes. We strongly believe that, if you add a task for a clinician, you should design your health information technology (IT) in such a way that other task(s) in the same process are eliminated.

3.2. Auto-populating vs. supporting clinical autonomy

To make the CDS as efficient as possible and prevent physicians entering data that are already available in the EHR, we tried to auto-populate as much as possible the CDS. The CDS is auto-populated with items that have numeric values, such as heart rate, blood pressure and oxygenation (SpO₂). However, although physicians like auto-population, they also like their autonomy. Especially in the ED, there is sometimes confusion about the reliability of some these values. For example, heart rate (HR) is measured by the triage nurse at intake, and then periodically rechecked during the ED stay. The number that is auto-populated in PE Dx from the EHR is the highest recorded HR, but this does not always best reflect the observed medical condition of the patient. Therefore, physicians would like to be able to edit that (auto-populated) value, sometimes to see how that would impact the final risk score. Evidently, there is an interesting trade-off between automation and autonomy. In our CDS, both options are available: the PE Dx is auto-populated, but clinicians can edit the data and the edited value is used in (auto) calculating the score.

3.3. (Auto-) Calculating

ED physicians work with many clinical diagnostic decision rules to evaluate the probability of a medical condition (e.g. Ottawa rule for ankle fracture). Most rules require some combination of symptoms, physical exam findings, and laboratory results. The values assigned to each element produce a score that helps guide the next step in the diagnostic process. Often a diagnosis can be ruled out if none of the elements is present. For example, results of the Wells’ criteria can vary from 0–1 (PERC Rule needed to rule out PE), 2–6 (Order D-dimer) and >6 (Order CT scan). One of the questions presented during design of PE Dx was whether to “simplify” execution of the different rules. For example, as soon as a certain score has been achieved (for example >6 points in the Wells’ criteria), it is often not necessary for the diagnostic process to continue: there is no difference between 7, 8, 9 or 10 points, all are above the cut-off score. One way to speed up the work process is to provide the physician immediately with a score as s/he goes through the different items and as soon as she has reached a certain cut-off point, to provide her with decision support (e.g. recommendation to order CT scan). However, from a clinical perspective, there are advantages to obtain the responses to all questions, and to have all responses documented. Local hospital policy supports physicians responding to all questions before calculating the score. Further, data collected in this way can be used to further improve clinical decision rules.

4. Discussion

In this study, we designed a computer decision support tool (PE Dx) to support PE diagnosis in the ED, following user-centered design methods and principles. Preliminary results of a simulation study show that the CDS is well accepted by clinicians, more efficient and effective and results in higher end-user satisfaction compared to an online CDS [1]. PE Dx has been implemented and will be evaluated.

4.1. Lessons learned

The procedure to rule in/out a PE is challenging from a human factors design perspective. A two-step process is needed to rule out a PE. There is considerable overlap between the two tests (PERC, Wells’): 3 of the 12 questions are the same. Further, administration of PERC is dependent on the results of the other test (Wells’). In the early stages of PE Dx design, we focused on integrating the two tests as much as possible to avoid duplicate data entry, which can be frustrating and does not encourage clinicians to use the tool. However, apart from trying to deal with the similarities between the two tests, we should also have focused on the differences between them. The Wells’ criteria need the responses to several questions to calculate a score, while the PERC only needs one single item to be positive. That makes integration of the two tests challenging; it took us a while to realize that and to let go of our initial design approach. Further, only in certain situations (a Wells’ score lower than 2) is the PERC needed. Therefore, we decided to gray out the PERC, and only make it active when the Wells’ criteria indicated use of PERC (see Figure 1). The lesson is not to be narrowly focused on certain design features before the whole process is completely understood.

During the design of PE Dx, we were confronted with difficult decisions. A design decision from a human factors point of view would have been different from that from a clinical perspective. Evidently, a CDS is created for and used by clinicians and therefore, they make the ultimate decision. However, in some instances, the human factors experts could explain why certain (design) solutions would be better. The combined input of clinicians and human factors experts contributed to the design process. The human factors experts developed basic knowledge of the PE diagnostic process, and input from clinicians helped to understand why physicians made specific choices, and also to explain specific tasks and processes, such as the actual D-dimer test and its advantages and disadvantages.

Defining design requirements for a CDS for PE was the original goal of the study. However, we were able to actually design and program PE Dx, and to conduct a usability evaluation in a simulation experiment. In this experimental study, we evaluated the usability (effectiveness, efficiency and end-user satisfaction) by comparing PE Dx with an existing web-based CDS. Results of the experiment are promising, which underlines the central message of this paper, i.e. that it is possible to use human factors principles to design health IT that will be accepted by end-users. However, the ultimate test is whether clinicians actually use the tool.

5.2. Study limitations

In this study, we describe the design of a specific CDS (PE Dx) for a specific environment (ED). We realize that designing CDS is context dependent; designing a CDS for the ED is not the same as designing a CDS for the inpatient wards or for ambulatory care clinics, because the work system characteristics and workflow are very different in those clinical settings. Nevertheless, we think that the issues we described here can also inform the design of CDS in these other settings.