User-Centered Redesign of a Clinical Decision Support System for Pneumonia in the Emergency Department

Abstract

Background

Deviation from evidence-based guidelines is common and associated with worse patient outcomes, especially in hectic emergency departments (EDs). Clinical decision support (CDS) systems can improve outcomes by promoting guideline adherence while allowing patient-specific adaptation. Implementation of a CDS system for pneumonia (“ePneumonia”) in the ED has been associated with improved guideline adherence and reduced 30-day mortality. However, adoption of ePneumonia has been hindered by a suboptimal user interface (UI).

Objective

This study aimed to redesign the ePneumonia UI to improve usability and adoption.

Methods

We conducted a user-centered design study involving ED clinicians at Vanderbilt University Medical Center. Across two rounds of one-on-one usability interviews with ED clinicians held via videoconference we (1) identified user requirements, and (2) iteratively refined a UI prototype. During each usability interview we presented realistic pneumonia cases, observed the clinician interact with a prototype, and elicited feedback with a semi-structured interview guide. We applied rapid thematic analysis and iteratively updated ePneumonia UI prototypes between interviews.

Results

Among 21 invited ED clinicians, 19 (90%) participated, including 15 attendings, 3 residents, and 1 advanced practice provider; 6 (32%) participants were women. Initial findings revealed that the original step-by-step UI did not align with the dynamic ED workflow. Clinicians expressed a need for both flexibility (e.g., skipping sections, overriding recommendations) and constraints (e.g., alerts for guideline deviations). We identified UI features to meet these needs that resulted in greater subjective usability including: a three-step navigation scheme, tiered information display, and multimodal indicators (text, icon, color) of agreement between clinician choices and CDS recommendations.

Conclusion

A user-centered design approach identified UI features that were associated with greater perceived usability of a CDS for pneumonia in the ED. Future work will evaluate real-world usability and adoption in a clinical trial.

Background and Significance

Medical decision-making and care delivery are highly variable and often deviate from evidence-based guidelines leading to worse patient outcomes, higher mortality, increased costs, and greater disparities.^1–5 This is especially true in hectic emergency departments (EDs), where time pressure, high acuity, and diagnostic uncertainty make adhering to guidelines particularly challenging.⁶ At the same time, deviation from guidelines is sometimes necessary, as individual patients are complex and do not always neatly align with the evidence underlying guidelines.⁷ Clinical decision support (CDS) systems embedded in the electronic health record (EHR), if made well, can standardize care toward evidence-based guidelines while allowing patient-specific adaptation with clinical judgment. CDS systems have been associated with improved outcomes in many clinical domains including ED management of sepsis, acute heart failure, and pneumonia.^8–12 Unfortunately, CDS systems frequently have a suboptimal user interface (UI) and are either ignored >50% of the time or force clinician interaction which worsens alert fatigue and automation complacency.^13,14

In this study, we focus on CDS for the ED management of adults with community-acquired pneumonia to investigate what UI design features enhance usability and adoption. Community-acquired pneumonia afflicts millions worldwide annually and is the leading infectious cause of death in the United States with a 1-year mortality rate of 30%.^15–17 Treatment of pneumonia in the ED, including selection of empiric antibiotic therapy and the site of care following the ED (i.e., disposition), varies more than two-fold between individual clinicians and EDs, resulting in substantial deviation from guidelines (>25% in some cases).^18–21 To address these issues, over the past 20 years Intermountain Health has refined an EHR-embedded CDS system called “ePneumonia” to support the diagnosis and treatment of ED patients with pneumonia.^22–25 In a recent stepped-wedge cluster-controlled clinical trial at 16 Intermountain sites, deployment of ePneumonia was associated with reduced 30-day mortality (8.6% before vs. 4.8% after, adjusted odds ratio 0.62 [95% CI: 0.49–0.79], p < 0.001), safely increased discharge disposition (29.2 vs. 46.9%), and increased guideline-concordant antibiotic prescribing (79.5 vs. 87.9%).²⁶ However, adoption of ePneumonia has been variable, reaching 80% in some sites but falling below 30% in others, particularly rural and community-based EDs, in large part due to suboptimal UI design.²⁷ Furthermore, to date ePneumonia has not been disseminated beyond the Intermountain Health system.

Scaling promising CDS systems, like ePneumonia, across multiple institutions has been impeded by issues with both usability and interoperability. First, many CDS systems have suboptimal UI design, including inappropriate use of interruptive alerts, non-intuitive interactions, or confusing displays.^28–31 Although the “five rights” of CDS are well established, designing UIs that effectively fulfill these standards remains a challenge.^9,32 User-centered design (UCD) is a human factors engineering approach that can help overcome many usability pitfalls.³³ UCD employs iterative cycles of design and evaluation in usability interviews, where users interact with prototypes in scenario-based simulation, to align the technology design with user needs.^34,35 Second, many CDS systems are custom-built within a single institution’s EHR. Even when the underlying logic is well described, the CDS software itself is often not readily translatable to other hospitals with the same EHR platform let alone those with a different platform.³⁶ More recent SMART (Substitutable Medical Applications and Reusable Technologies) on FHIR (Fast Healthcare Interoperability Resources) standards that enable a single CDS application to be integrated with multiple EHRs may overcome the interoperability barrier.³⁷ Applying UCD to develop SMART on FHIR-based CDS applications may overcome these two interacting barriers to scaling effective CDS systems.

Objective

We applied UCD methods to identify UI features that would enhance the usability of ePneumonia for development of an interoperable SMART on FHIR application and future implementation at Vanderbilt University Medical Center (VUMC).

Methods

Study Design and Setting

We conducted a formative qualitative UCD study, following international standards and best practices,^38,39 involving ED clinicians at VUMC to redesign the UI for ePneumonia. We focused on the UI for the main display of ePneumonia once the system is opened and assumed the full range of UI functionality available in SMART on FHIR applications. The underlying logic of ePneumonia has been described in detail previously and was maintained without changes.^22–25 In brief, ePneumonia screens all ED patients for possible pneumonia using real-time EHR data and prompts ED clinicians to open ePneumonia with a non-interruptive icon on the ED trackboard. Upon clicking the icon, ePneumonia opens in a window within the patient’s chart and guides the clinician to complete risk assessments for pneumonia severity and antibiotic-resistant bacteria which are prefilled with available EHR data. After completing the risk assessments, ePneumonia recommends a guideline-concordant order set including ED disposition, laboratory testing, and empiric antibiotics (for more details, see the Supplementary Material, available in the online version only).

VUMC is an academic health system in Tennessee with a flagship quaternary urban hospital and five affiliated community-based hospitals. A single division of Emergency Medicine comprised of attending physicians, residents, and advanced practice providers, hereafter referred to collectively as ED clinicians, staff the EDs at all VUMC hospitals. Clinicians work at one or more ED sites within the system. VUMC had been using the Epic EHR (Verona, Wisconsin, United States) for 7 years at the time of the study. The multidisciplinary research team (all authors) included experts in pneumonia, ED operations, CDS systems, UCD, and development of SMART on FHIR applications.^40–47

Study Procedures and Data Collection

We used purposive sampling to recruit ED clinicians at VUMC across a range of targeted user roles (attendings, residents, and advanced practice providers), years of experience, and clinical site (quaternary ED, community-affiliated EDs, or both). Clinicians were invited via email to participate in a one-on-one hour-long usability interview during their non-clinical time. Participation was voluntary and there was no compensation.

We conducted two separate rounds of formative usability interviews, led by authors M.A.C., R.B., and A.C., with increasing prototype fidelity that were guided by distinct protocols (Supplementary Material, available in the online version only). Initial low-fidelity prototypes (used on rounds one and two) were made with Microsoft PowerPoint (Redmond, Washington, United States), and later high-fidelity interactive prototypes (used in round two) were made using Figma (San Francisco, California, United States), an online UI prototyping program.⁴⁸ We used Microsoft Teams (Redmond, WA) videoconferencing to conduct, record, transcribe, and securely store all usability interviews. We obtained participant demographic data including age, years in practice, role, and distribution of clinical work among ED sites via a REDCap survey administered at the beginning of each interview.^49,50

In the first-round usability interviews, we conducted interviews and assessed clinicians’ responses to the baseline ePneumonia UI design to develop user requirements for redesign. During each interview, we provided a realistic case of a patient who may have pneumonia, developed by clinicians on the research team, and asked the clinician about his or her workflow and any modifying factors. We then observed the clinician interact with the baseline ePneumonia design in a clickable PowerPoint prototype and asked questions directed at usability, including perceptions, desired and expected actions, and points of confusion. After completing first-round interviews, we analyzed the data as described below, formulated user requirements, and constructed new prototypes.

In the second-round usability interviews, we conducted iterative design and formative usability testing with progressively increasing prototype fidelity. During each interview we presented between one and three standardized realistic patient cases as time allowed, observed clinicians interact with different prototypes, and again asked questions directed at usability as described above. The standardized pneumonia scenarios were developed by two clinicians (M.A.C. and M.J.W.) based upon their real-world clinical experience. The three cases were designed to include edge scenarios, where a clinician may reasonably differ between ED disposition (discharge, admit to ward, admit to ICU) and broad versus standard empiric antibiotic therapy (Supplementary Material, available in the online version only). Throughout the second round we iteratively analyzed data, making comparisons across participants, after each usability interview and made updates to the prototypes for ongoing formative assessment in subsequent interviews. For example, during early interviews in round two we exposed clinicians to different tree diagram UIs but found that clinicians had difficulty navigating all of these UIs. In response, we created a new prototype with a three-step linear navigation scheme which clinicians were able to navigate more easily. In subsequent interviews we exposed participants to different UIs (button behavior and coloring) for indicating agreement and disagreement between clinician selections and ePneumonia recommendations and again evaluated qualitative usability.

Data Analysis

We followed standard UCD practices in formative usability studies for pragmatic rapid thematic analysis,^34,39 informed by a human–computer interaction framework (Nielsen’s usability heuristics) and a sociotechnical systems framework (systems engineering initiative for patient safety [SEIPS]).^51–53 We chose to use both of these frameworks during analysis so that we could capture how direct interactions with the computer (e.g., clicking buttons) and broader contextual factors (e.g., local practice norms and workflow) influenced the usability of prototype UI features. We used field notes, transcripts, and subsequent targeted review of screen recordings to clarify interactions as the primary data for analysis. One author (M.A.C.) reviewed the primary data manually looking for patterns across participants during each round using rapid thematic analysis with mixed deductive and inductive analysis informed by the sociotechnical frameworks (Nielsen’s and SEIPS). All members of the research team met every 2 weeks to discuss emerging themes and findings. For example, clinicians’ descriptions of their work when caring for patients with pneumonia in the ED were deductively matched to the SEIPS’ model of a cognitive process carried out by a work system and combined with an inductive analytic process to generate the finding that clinicians wanted to be able to skip around sections of ePneumonia. We set a minimum sample size of five ED clinicians per round, which is considered standard practice in formative UCD studies and reveals up to 90% of design problems.³⁹ After five interviews in the first round no new themes were uncovered and we proceeded to the second round. After completion of 14 interviews in the second round the UI design stabilized, and no further usability issues or themes were found and we considered the design finalized for this formative stage of UCD.

Demographic data of participants were summarized using means, ranges, counts, and percentages as appropriate.

Results

Among 21 invited ED clinicians, 19 (90%) participated, including 15 attendings, 3 residents, and 1 advanced practice provider (Table 1). Five clinicians participated in round one and 14 participated in round two.

Table 1.

Characteristics of participating ED clinicians in the ePneumonia UCD study

Characteristic	(N = 19)
Age, mean (range), y	35 (28–44)
Sex, N (%)
Female	6 (32)
Male	13 (68)
Role, N (%)
Attending	15 (79)
Advanced practice provider	1 (5)
Resident	3 (16)
Years in practice, mean (range)	6 (1–14)
ED shifts per month, mean (range)	12 (7–20)
Average proportion of work in ED sites, mean (range)
Quaternary	86 (33–100)
Community based	14 (0–67)

Abbreviation: UCD, user-centered design.

Contextual Factors and User Requirements (Round One)

Contextual sociotechnical factors that influenced the potential use of CDS for pneumonia included non-linear workflows, the presence of a team-based decision-making with shared task ownership, and time pressure. Clinicians’ descriptions of the general process of an ED encounter followed the common “ABCDE” structure: Arrival and nurse triage, Bedding (i.e., placement in a treatment room),Clinician assessment and treatment, Disposition selection, and Exit.⁵⁴ However, clinicians described highly complex and disrupted workflows within this structure, where decisions around pneumonia care occurred across people and time (Fig. 1). In the quaternary ED setting but not in community EDs, the presence of a continuous flow unit (CFU),⁵⁵ a separate space and team in the ED designed to quickly care for patients with lower acuity problems, further complicated the workflow. Patients first sent to the CFU were sometimes transferred to the main section of the ED if they became sicker, where they received a new team of clinicians. These separate clinician teams in the CFU and main ED unit could include a single attending, multiple attendings between different shifts, or a combination of residents or advanced practice providers with variable degrees of supervision from an attending. Thus, the number of clinicians, roles involved, and division of tasks between roles could be variable depending upon the initial and subsequent unit (CFU or main ED or both). Together, these factors led to a nonlinear complex workflow, with variability in who, when, and in which order treatment decisions were made for patients with pneumonia.

Fig. 1.

Presumed versus observed ED workflow for pneumonia. Time is the horizontal axis, divided into the “ABCDE” structure. D in circles represent when a decision is made on ED disposition. A in diamonds represent when a decision is made on empiric antibiotic therapy.

The original stepwise design of the ePneumonia UI (Fig. 2, baseline navigation design) did not align with this complex workflow. Both the time commitment necessary to complete the tool (number of clicks and pages), and the dictated order of pages was met with exasperation. For instance:

“So I get that we’re starting with severe hypoxemia, [and if they have that] they need to go to the ICU, but that’s clearly not where this patient’s at… Get out of my way ICU screen!”

(participant 1 [p1], attending at quaternary ED, Bold denotes participant’s own verbal emphasis)

Fig. 2.

Navigation design concepts for ePneumonia user interface (UI). Top: The baseline design present before the user-centered design (UCD) study and tested in round-one UCD interviews. Middle: One example of an intermediate navigation design, with a tree diagram, tested during round-two UCD interviews. Bottom: The finalized navigation design that resulted after completion of all round-two UCD interviews.

Clinicians expressed a desire to skip components of ePneumonia, typically the disposition assessment and selection, because they felt that they would already have made that decision by the time they opened ePneumonia. Conversely, clinicians recognized the value of the ePneumonia to highlight information, which they may not have noticed otherwise, that would substantially change their decision.

“Once I use this tool a few times, I’m going to probably skip ahead because I know what the next box is going to be and it’s not of utility to me to click through them. I’ve already decided the patient does not meet ICU level care before I open this tool. Granted, there may be times in which I overlooked something that is quite pertinent. For example, the P/F ratio, which I don’t routinely calculate on my patients, if that’s calculated for me and it alarms me that I’ve overlooked that and this patient may be better suited for ICU care then that would be helpful.”

(p12, attending at quaternary and community EDs)

A list of user requirements developed by the research team, according to round-one data, are presented in Table 2. Both the frequency and valence of comments around the length of time needed to complete the tool made it clear that the highest priority was a method to enable clinicians to skip sections that they deemed unnecessary. To enable this flexibility, it became clear that there were two other major requirements to maintain constraints and guidance: (1) a method to indicate the agreement/disagreement status between a clinician’s selection and the ePneumonia recommendation, and (2) a method to indicate whether data had been reviewed and confirmed by the clinician.

Table 2.

User requirements for ePneumonia UI developed from round-one interviews

Category	Description
Navigation	A method to display all sections and enable the user to “skip” forward and backward to a section of interest.
Navigation	A method to indicate whether a recommendation was preliminary (because of missing or unreviewed data) or finalized.
Navigation	A method to indicate when the user-selected choice differs from that recommended by ePneumonia, but still allows deviation from ePneumonia.
Process	A method to enable users to leave/exit ePneumonia and pick up where they left off, upon returning.
Process	A method to enable multiple different clinicians to edit and view ePneumonia during an encounter.
Data interaction	A method to allow clinicians to edit individual data elements imported from the EHR.

Abbreviations: EHR; electronic health record; UI, user interface.

Design Features to Meet User Requirements (Round Two)

Representative examples of UI concepts in baseline, intermediate, and final prototypes developed during round-two interviews are shown in Fig. 2. Although the baseline design did allow skipping by clicking at tabs at the top of the screen, none of the participants recognized that this action was possible. In intermediate prototypes, participants viewed decision tree diagrams positively, in part because of their similarity to other protocols used in their practice (e.g., UpToDate).

“When I see a tool and it’s got eight tabs at the top, I start to get worried that it’s gonna take forever to get through. Whereas [with this tree diagram] I can see immediately [where] it’s trying to guide me.”

(p11, attending at quaternary and community EDs)

However, in subsequent interviews, tree diagrams had their own usability problems. Clinicians had difficulty splitting attention between the navigation diagram in the top half and the data review form in the bottom half of the screen (Fig. 3). Most clinicians were unsure of where to start and simply directed their attention to the bottom-half of the screen and worked through each data review page in a step-by-step fashion without realizing that they could skip sections by clicking on the tree diagram.

Fig. 3.

Intermediate ePneumonia user interface (UI) design. Decision tree navigation is shown on the top-half while a data review form is simultaneously shown in the bottom-half. At right, the suggested order set is highlighted in yellow, and the selected non-concordant order set is highlighted in red.

“Answering questions and checking boxes. Yeah, I can do that pretty easily. But like having to navigate a flow diagram like that in the middle of a shift, it’s not something I can do.”

(p16, attending at quaternary ED)

Clinicians also had difficulty interpreting the use of color indicators on the tree diagram or had negative reactions to them. One clinician, upon viewing the red color when a selection differed from the CDS recommendation (Fig. 3), stated:

“It’s red. It’s telling me not to do it, so I’d be like, forget you score and X out of it and just write whatever.”

(p14, advanced practice provider at quaternary ED)

The final design included three features which were associated with greater subjective usability (Fig. 4). First, we used a three-step navigation schema with a box summarizing the data and instructions for each decision: select disposition, select antibiotics, confirm order set. Second, we used tiered information displays by placing the data review components into subpages that became visible (popped open) upon clicking “edit assessment.” Third, we used a combination of text, icons, and color to denote whether data were preliminary or confirmed and agreement status between a clinician’s selections and CDS recommendations. The yellow fill color for disagreement combined with the “acknowledge disagreement” options in the order checkout component avoided the negative reaction observed with the red fill color.

Fig. 4.

Finalized ePneumonia user interface (UI) design. Key design features shown include: (1) three-step navigation schema with data summary, (2) tiered information display with data review form shown in pop-up subpage that opens when clicking the summary box, (3) multimodal indication of preliminary versus confirmed status of clinical decision support (CDS) recommendation and of agreement status between clinician selection and CDS suggestion.

Discussion

Principal Findings

In this UCD study of a CDS system for ED management of pneumonia, we identified UI features that achieved greater subjective usability. The complex ED workflows described by clinicians differed substantially from a prior simplistic model.⁴² The baseline step-by-step design did not align with this complex workflow and led to frustration, which is a common finding for many CDS systems.^27,56 Clinicians in our study had competing needs for flexibility (e.g., to skip past sections) and constraint (e.g., to be still notified and given a chance to adjust when their initial decisions deviated from recommendations). UI features that fulfilled these needs included a simplified three-step navigation scheme, tiered information display, and multimodal (text, graphic, color) indicators of agreement status between clinician selections and CDS recommendations. Although these findings arose in the specific context of ED management of pneumonia, these UI features may improve usability of CDS systems for other diseases in acute care settings.

Implications of the Findings

The UI features we identified differ from previously described CDS UIs in several ways. Because we planned to develop a SMART on FHIR application, we had a much larger suite of possible UI features to select from compared to CDS configured natively within commercial EHRs. The baseline step-by-step ePneumonia UI design was a result of the limitations of the commercial EHR in which it was developed. Clinicians’ exasperation with the step-by-step baseline UI is consistent with other studies of complex CDS that require “too many clicks.”^57,58

Our redesigned UI offers a different strategy to make complex multi-step clinical guidelines easier to navigate. Common navigation approaches include decision trees (i.e., pathways) and dashboards. Decision trees, often seen in professional guidelines and online references like UpToDate (Waltham, MA), efficiently display the overall logic graphically. In order for clinicians to navigate decision trees the branch points must be limited to the value of just one, perhaps two, categorical variables. However, many clinical decisions are improved with objective multivariable risk prediction models that are not readily manually computable. We tested combining a decision tree with prefilled clinical risk scores and found that clinicians had difficulty splitting their attention between the two tasks of navigating the tree and reviewing the score (Fig. 3). Dashboards, which display both a summary result and the data used in a model, collocate the navigation and review tasks in the same area of the display. However, dashboards can produce information overload for users rendering them less usable.^31,59

We found that a simplified three-step display made it easy for clinicians to quickly recognize the three guided tasks: select disposition, select antibiotics, and confirm order set placement. The tiered information delivery, in which the details of the risk scores were placed in sub-pages that were open upon selection, was effective to avoid information overload that occurs with decision trees and dashboards. These design features may increase the usability of CDS for other clinical scenarios that include a handful of decisions that are guided by objective risk scores calculated from EHR data. Conversely, for situations with a single decision (e.g., a single risk score for ED disposition in heart failure, or avoidance of an allergy) these navigation features may not be necessary. For situations with a greater number of nested decisions (e.g., management of diabetic ketoacidosis), the logic visibility afforded by decision trees may be preferrable.

The competing needs for flexibility and constraint that was expressed by clinicians in this study has been described in human–computer interactions in other industries (e.g., airplane cockpits).⁶⁰ Except at the uncommon extremes of where there is clearly a correct action (constraint is needed) or high ambiguity (flexibility is needed), balancing these needs is both important and challenging.³³ Constraints in areas of high ambiguity, such as mandatory pop-up warnings, drive clinicians to find work-arounds or bypass CDS.⁶¹ Conversely, making it easier for clinicians to complete CDS by removing warnings or using preselected defaults risks automation complacency.⁶² Balancing flexibility and constraint requires clear indication of system status from the computer to the human operator.⁶³ During the iterative formative UCD design process, we found that the color red successfully indicated deviation from a recommendation but the salience was perhaps too strong, as it made clinicians want to close the CDS. We suspect that overuse of the color red, and other indicators with salience that mismatches the scenario, may partially explain why many CDS systems are closed within a few seconds and bypassed without consideration.^13,64 Medium salience indicators of deviation, neutral text and yellow-colored selections, had greater acceptance in this study. The finalized UI, which combines navigation and indicators of deviation, provides one mechanism to balance the demands of flexibility and constraint. Taken together, the UI features we identified generally follow established design prescriptions, including Nielsen’s heuristics used during analysis (e.g., the importance of visibility of system status and of options) and Shneiderman’s visualization mantra (overview first, zoom and filter, and details on demand).⁶⁵ Although the importance of these concepts for usability is not new, determining what UI features achieve these concepts in the complex ED environment is novel. The specific UI features of our final design may inform development of other CDS in acute care environments (ED, hospital, intensive care unit), especially those that take the SMART on FHIR application approach.

Limitations

In this study we applied rigorous UCD methods and used high-fidelity interactive prototypes in PowerPoint and Figma, which offer substantially increased realism over low-fidelity static options. However, this study has several important limitations. First, although we used realistic simulation in UCD interviews, these methods were formative not summative, and do not capture all the complexities of the clinical environment nor quantitative measures of usability, guideline adherence, or influence on clinical outcomes. While qualitative data included participants’ stated and observed interactions, and themes were derived across participants in each round, quantitative measures of usability would further support internal validity. We have begun a prospective multisite clinical trial with quantitative measures to assess the real-world usability, adoption, acceptability, and clinical effectiveness of the updated ePneumonia UI. Second, our study was limited in scope by setting (single academic center), site (ED), disease (pneumonia), and the underlying functionality of a single CDS system. Although our findings may be informative for CDS design generally, they could be overfit to a particular setting and they are less likely to generalize to more disparate settings, like chronic outpatient care or daily inpatient management. Third, we used rapid thematic analysis, which utilized manual review of multiple data types (field notes, transcripts, videos) with mixed inductive and deductive approach, which may yield slightly different results than traditional inductive grounded theory methodology with sentence-level coding. Fourth, our UCD methods pragmatically focused on ED clinicians and their use of ePneumonia. We did not include nurses, pharmacists, or clinicians in other roles (e.g., accepting hospitalist or intensivist, or primary care providers) with potential differing needs or perspectives that could influence CDS usability and adoption. Fifth, by designing for a SMART on FHIR application, we had a broader range of UI capabilities of the main screen available to explore, but fewer options for integration within the EHR workflow (e.g., alerts, launching, and order placement operations). As such, we did not focus on alert and order placement UI designs, which are also important for overall usability and will require further evaluation in future studies.

Conclusion

We applied a UCD approach to redesign the UI for an interoperable pneumonia CDS system. We found that the original baseline design, which assumed clinical decisions were finalized at one point in time, did not match the complex reality of ED clinicians’ decision-making processes. By applying the UCD methods of iterative scenario-based simulations where ED clinicians interacted with low- and high-fidelity prototypes, we identified design features that fulfilled ED clinician’s dual needs for flexibility and constraint. A navigation scheme with a simplified three-step process and tiered information display enabled clinicians to optionally “skip” past components, such as disposition selection, when they had already made a decision. Simultaneously, multimodal indication of disagreement with a CDS recommendation fulfilled clinicians’ need to be made aware of information that may sway their decision. Although these UI features had increased subjective usability in simulation, evaluation of the real-world usability and effect of these UI features on adoption in a clinical trial is needed and ongoing.

Supplementary Material

Supplementary Material is available at https://doi.org/10.1055/a-2808-9190.

Clinical Relevance Statement.

Clinical decision support (CDS) systems can improve patient outcomes by promoting adherence to evidence-based guidelines, but adoption is impaired by poor usability. We sought to identify user interface features to enhance CDS usability by redesigning an existing CDS system for the management of pneumonia in the emergency department. We applied user-centered design methods, consisting of iterative cycles of design and scenario-based simulation with clinicians. We found that a simplified three-step navigation scheme, tiered information display, and multimodal indication of clinician–recommendation agreement were associated with greater subjective CDS usability. Evaluation of the effect of these user interface features on real-world usability and CDS adoption in a clinical implementation trial is ongoing.

Multiple-Choice Questions.

1. What is one user requirement for usable CDS in the ED indicated by clinicians in this study?

   1. The ability to set alerts to “always ignore”
   2. A setting to auto-select CDS recommendations without oversight
   3. A method to enable the user to “skip” non-relevant sections of the CDS
   4. A setting that ensures that clinicians cannot ignore or disagree with CDS data
      Correct Answer: The correct answer is option c. In this study, the CDS system for Pneumonia included multiple evaluations to guide clinicians on risk-stratification, disposition selection, and antibiotic selection. Clinicians’ often wanted assistance with only one of those components of ED pneumonia care. I method to “skip” nonrelevant CDS sections was a major user requirement.
2. In this study, what user interface navigation design had the greatest subjective usability?

   1. A step-by-step approach where clinicians are guided through each relevant screen
   2. A graphic decision tree diagram demonstrating the overall logic
   3. A single pop-up alert that asks to confirm an order set selected by the CDS system
   4. A simplified three-step approach showing major decisions, suggested selections, and summarized data with additional information available via pop-open subpages
      Correct Answer: The correct answer is option d. In this study, an updated CDS user-interface that used a three-step approach with additional detail available on-demand had the greatest subjective usability (Fig. 4)