Designing Real-time Decision Support for Trauma Resuscitations

Abstract

Background

Use of electronic clinical decision support (eCDS) has been recommended to improve implementation of clinical decision rules. Many eCDS tools, however, are designed and implemented without taking into account the context in which clinical work is performed. Implementation of the pediatric traumatic brain injury (TBI) clinical decision rule at one Level I pediatric emergency department includes an electronic questionnaire triggered when ordering a head computed tomography using computerized physician order entry (CPOE). Providers use this CPOE tool in less than 20% of trauma resuscitation cases. A human factors engineering approach could identify the implementation barriers that are limiting the use of this tool.

Objectives

The objective was to design a pediatric TBI eCDS tool for trauma resuscitation using a human factors approach. The hypothesis was that clinical experts will rate a usability-enhanced eCDS tool better than the existing CPOE tool for user interface design and suitability for clinical use.

Methods

This mixed-methods study followed usability evaluation principles. Pediatric emergency physicians were surveyed to identify barriers to using the existing eCDS tool. Using standard trauma resuscitation protocols, a hierarchical task analysis of pediatric TBI evaluation was developed. Five clinical experts, all board-certified pediatric emergency medicine faculty members, then iteratively modified the hierarchical task analysis until reaching consensus. The software team developed a prototype eCDS display using the hierarchical task analysis. Three human factors engineers provided feedback on the prototype through a heuristic evaluation, and the software team refined the eCDS tool using a rapid prototyping process. The eCDS tool then underwent iterative usability evaluations by the five clinical experts using video review of 50 trauma resuscitation cases. A final eCDS tool was created based on their feedback, with content analysis of the evaluations performed to ensure all concerns were identified and addressed.

Results

Among 26 EPs (76% response rate), the main barriers to using the existing tool were that the information displayed is redundant and does not fit clinical workflow. After the prototype eCDS tool was developed based on the trauma resuscitation hierarchical task analysis, the human factors engineers rated it to be better than the CPOE tool for nine of 10 standard user interface design heuristics on a three-point scale. The eCDS tool was also rated better for clinical use on the same scale, in 84% of 50 expert–video pairs, and was rated equivalent in the remainder. Clinical experts also rated barriers to use of the eCDS tool as being low.

Conclusions

An eCDS tool for diagnostic imaging designed using human factors engineering methods has improved perceived usability among pediatric emergency physicians.

Traumatic brain injury (TBI) is a leading cause of childhood morbidity and mortality. In the United States, pediatric head trauma is responsible for 7,400 deaths, 60,000 hospitalizations, and more than 600,000 emergency department (ED) visits annually.¹ More than half of children with head injuries undergo computed tomography (CT), with most having minor head trauma. Fewer than 10% of those scanned have TBI, and fewer than 1% have clinically important TBI.² Given the number of children evaluated each year for head injury, the established risks of radiation exposure^3,4 and the added cost from overuse of head CT represent avoidable health care burdens.

Similar to other clinical conditions for which testing or treatment is either overused or inconsistently applied,^5–8 a Pediatric Emergency Care Applied Research Network (PECARN) clinical decision rule (CDR) has been derived and validated to guide the use of head CT in injured children.² The current PECARN criteria for TBI is stratified as age < 2 years old and age ≥ 2 years old.² Each list contains seven conditions, signs, or symptoms that the TBI patient must be assessed for to determine the risk of clinically significant TBI. According to the PECARN rule, high-risk features are altered mental status, low Glasgow Coma Scale (GCS) score, and skull fracture. Medium risk is defined as the absence of high-risk criteria, and presence of at least one of the remaining four age-specific criteria. Low risk is defined as the absence of all seven age-specific criteria.

Unfortunately, physicians often fail to adopt CDRs and other forms of evidence-based practice through traditional means of knowledge acquisition such as journal articles and scientific presentations.^9–12 This implementation gap has led to interest in electronic clinical decision support (eCDS) tools as a means to improve physician adoption rates.^13,14 An eCDS tool is deployed within the electronic health record system, often connected to computerized physician order entry (CPOE) functionality, to help the provider in his or her decision-making process. Many existing eCDS tools have been designed and implemented without taking into account the context in which clinical work is performed. As a result, implementation research confirms little demonstrable improvement in physician adoption or quality of care using these systems.^15,16 At the very least, poor implementation of eCDS systems leads to wasted effort and money by failing to accomplish its goal.^17,18 At worst, it leads to unnecessary patient morbidity or mortality by serving as a time-consuming distraction to physicians.^19,20

In the ED at Children’s National Medical Center (CNMC), the current PECARN CDR implementation for head CT imaging after TBI is a CDR questionnaire delivered when ordering a head CT using Cerner FirstNet (Cerner Corporation, North Kansas City, MO). Although more than half of clinicians use it in their daily practice, it is used in less than 20% of trauma resuscitation cases.²¹ Rather than citing physician resistance to the existing CPOE tool, a human factors engineering approach seeks to address this noncompliance by identifying the usability challenges and workflow gaps that serve as barriers to use. Trauma resuscitations at CNMC involve a coordinated team of emergency medicine and trauma surgery providers, which includes a trauma recorder (typically a nurse or physician) who collects the elements of assessment and management as they are uttered by other team members and a nurse outside the trauma bay who enters orders into the CPOE. In the trauma bay, there is a wall-mounted 40-inch monitor that displays pertinent patient information. The patient information presently displayed is only vital signs data, but the trauma flowsheet is being converted from paper form to a digital tablet entry to enable the monitor to display the ongoing collective assessment in real time. A human factors evaluation may discover that the CPOE tool, which is being presented to the nurse outside the trauma bay who is entering the orders, is being delivered to the wrong team member and well after the decision has already been made to perform the test. Our objective is to use human factors engineering methods to design a pediatric TBI eCDS tool that is more appropriate than the existing CPOE tool for the fast-paced environment of trauma resuscitations.

METHODS

Study Design

For this study we used a mixed-methods approach that is an adaptation of the Hix and Hartson²² star life cycle usability evaluation (Figure 1). This study was approved by our institutional review board (IRB).

Figure 1.

Research approach as a modified Star Life Cycle Evaluation.

Study Setting and Population

The CNMC is an urban tertiary care teaching hospital with Level I trauma and burn center designation. The ED sees almost 87,000 ED visits annually, with over 500 trauma resuscitations. In addition to the study physicians, we recruited into our team three usability experts; human factors engineers with domain knowledge of medicine; and five clinical experts, board-certified pediatric emergency medicine academic faculty members.

Study Protocol

We used a video surveillance system already in place at our hospital that allows for performance improvement review of trauma resuscitations. Each trauma bay is equipped with two ceiling-mounted cameras that record video and audio. All of the audiovisual streams, including the vital signs monitor, are stored together on a server and can be reviewed simultaneously through a secure portal.

Usability Evaluation Phase 0: Survey of Providers

We created an anonymous survey for CNMC pediatric EM staff physicians to identify barriers to use of the existing CPOE tool for pediatric TBI. We developed the questions from identified barriers to use of electronic decision support from a prior systematic review and pilot-tested the questions prior to the actual survey.²³ Electronic survey data were collected via e-mail solicitation and managed using Research Electronic Data Capture (REDCap) electronic data capture tools hosted in the Biostatistics Division of Children’s National Health System.²⁴

Usability Evaluation Phase 1a: Hierarchical Task Analysis

We performed a hierarchical task analysis to reorganize the existing PECARN TBI decision rule for pediatric TBI.²⁵ Using existing trauma protocols (Advanced Trauma Life Support, Pediatric Advanced Life Support, and our own department protocols), the study physicians performed a hierarchical task analysis of pediatric TBI evaluation. The clinical experts then iteratively modified the hierarchical task analysis until no new modifications were made (saturation) and the clinical experts were in consensus. The PECARN TBI assessment criteria were then identified within the hierarchical task analysis to understand how they fit the workflow pattern of how trauma patients are assessed once they present to the ED.

Usability Evaluation Phase 1b: Design and Design Representation

Guided by the hierarchical task analysis, the software team (KY, SC) developed a computer display of a prototype eCDS. We created a Microsoft PowerPoint slideshow with embedded links to navigate between the different states of the tool. The goal of the prototype was to display the information in a way that correlated with workflow, making it natural and intuitive to the providers who were assessing the trauma patient.

Usability Evaluation Phase 2a: Heuristic Evaluation

For the heuristic analysis, a team of three human factors engineers with medical domain knowledge were consulted. These experts evaluated the design of the existing CPOE tool and our prototype eCDS tool based on Jacob Neilsen’s “10 Usability Heuristics for User Interface Design,” which is a commonly used and well-respected reference for user interface design.²⁶ This consensus evaluation provided a score for each tool for the 10 usability heuristics on a three-point scale (heuristic met, partially met, not met).

Usability Evaluation Phase 2b: Rapid Prototyping

The prototype eCDS was programmed by the software team using HTML and Javascript because it allowed for portability between systems without added software development. Using this interface, the tool can be run on any computer or portable device with an Internet Web browser. Iterative refinements were made and tested based on the heuristics evaluation and continuous researcher review of the prototype performance with routine video review of trauma resuscitation cases.

Usability Evaluation Phase 3: Near-live Usability Testing

The eCDS tool was subject to iterative usability evaluation by the clinical experts using video review of trauma resuscitation cases. Based on prior usability evaluation sample size literature,²⁷ five clinical experts, all of whom are emergency physicians, completed 10 simulations each over a 1-month period. To vary case content, the 10 most recent trauma resuscitation videos were accessed from the server at the time of each expert session. For usability evaluation, we conducted “near-live” video-guided simulated resuscitations using a “talk-aloud” protocol.²⁸

The “near-live” simulation was set up so that the clinical expert was viewing two display screens. The first display showed recorded video footage of the resuscitation in addition to the audio stream and vital sign monitor from the case. The second display projected the eCDS prototype that was displaying real-time information as the trauma resuscitation was being viewed. During the simulations, a member of the research team was the trauma recorder acting as a “Wizard of Oz,”²⁹ picking up cues from the video and triggering the eCDS tool. Similar to their behavior as members of the trauma resuscitation team, the clinical experts were encouraged to verbalize their thoughts, perceptions, and actions as they worked through the simulations with the eCDS tool.

Qualitative Analysis

The participant’s utterances and feedback during the simulation were recorded by direct observation and with a digital audio recorder to understand the decision-making in real time. In addition, physician experiences were explored at the end of each case-based simulation using semistructured interviews and a survey that rated perceptions of the eCDS utility on a three-point Likert scale. Audio and written transcripts were reviewed independently by two study investigators (KY, JJ) using NVivo 10 (QSR international, Melbourne, Australia). Content analysis was performed inductively to generate themes, informed by previously identified barriers to use of eCDS.²³ Data were double-coded and examined for discrepancies to establish consensus.

RESULTS

The Phase 0 implementation survey was completed by 40 of the 53 CNMC pediatric emergency medicine providers (75.5% response rate). The primary barriers to use of the existing CPOE tool are that the information provided is redundant and does not fit clinical workflow (Table 1).

Table 1.

Survey of Clinician Barriers to Use of Existing CPOE Tool

Barrier	CPOE Tool, %
Information redundancy	69.2
Fit to clinical workflow	53.8
Too much time to complete	30.8
Medicolegal concerns	15.4
Family expectations	15.4

N = 26.

CPOE = computerized physician order entry.

We developed and iteratively modified a trauma resuscitation hierarchical task analysis until we reached clinical expert consensus. The Phase 1a hierarchical task analysis was then used to reorganize the original PECARN rule for the Phase 1b prototype eCDS tool (Table 2). To make the tool concise and not distracting to the user, we presented the criteria in the form of short yes or no questions abstracted from the PECARN rule. In addition, we included color markers and a status indicator that would not only draw the users’ attention to the important changes in the assessment, but also provide a familiar indication of the severity of the assessed criteria and what kind of risk level the patient was at. Red indicated presence of a high-risk criteria, yellow indicated medium-risk criteria, and green indicated low-risk criteria (Figure 2).

Table 2.

Reorganization of PECARN TBI rule by age

HTA-modified TBI Rule
Age ≥ 2 yr	Age < 2 yr
Severe mechanism	Severe mechanism
Loss of consciousness	Loss of consciousness
Abnormal mental status	Abnormal mental status
GCS < 15	GCS < 15
Severe headache	Scalp hematoma
Basilar skull fracture	Palpable skull fracture
Vomiting	Not normal per parent

GCS = Glasgow Coma Scale; HTA = Health Technology Assessment; PECARN = Pediatric Emergency Care Applied Research Network; TBI = traumatic brain injury.

Figure 2.

Organization of (A) high-, (B) intermediate-, and (C) low-risk elements. CT = computed tomography; GCS = Glasgow Coma Scale; TBI = traumatic brain injury.

During Phase 2a, the human factors engineers scored the prototype eCDS tool better for user interface design on all but one of the 10 usability heuristic measures when compared to the CPOE tool (Data Supplement S1, available as supporting information in the online version of this paper). The eCDS tool was considered deficient in the “help and documentation” category for not having links to the pediatric GCS and criteria for severe mechanism of injury for reference. To address this deficiency, we planned to add wall-mounted severe mechanism of injury posters in every trauma resuscitation room to accompany the existing GCS reference posters.

As a result of the heuristic evaluation, design improvements were incorporated during Phase 2b rapid prototyping before usability testing. Major changes included phrasing each of the seven questions so that all “yes” answers were positive findings and that “yes” and “no” options were clearly visible. When an answer is selected, the query area is marked and the question above it changes to a color that indicates the contribution of that finding to risk: red indicating contribution to high risk, yellow indicating contribution to medium risk, and green indicating no contribution to risk. The decision bar on the bottom of the tool is made to stand out from the rest as it displays the recommendation. This bar prompts the user to continue assessing the trauma until enough criteria are met for a recommendation to be made. If a user error is made, or a criterion is reassessed and changes during the resuscitation, the yes/no status is visible and can be corrected. The human factors engineers advised that changes in the state of the responses be automatically reflected in the recommendation.

In anticipation of integration into the electronic display of the trauma flowsheet, we condensed the tool into a single column occupying about a third of the screen, so that other pertinent patient information could be displayed on the monitors in the trauma bay. We used a continuous loop that reassessed when a new criterion was checked off so that the system state would update instantaneously (sample code available from author on request). The recommendation made by the tool then would reflect real-time updates in the trauma assessment (Figure 3). In addition, any input was made to be reversible so that a wrong input could be changed immediately. Likewise, if criteria changed after patient reassessment during the resuscitation, the system would change immediately to reflect the new information and update the recommendation.

Figure 3.

Examples of (A) updating eCDS tool and (B) change in recommendation. CT = computed tomography; GCS = Glasgow Coma Scale; TBI = traumatic brain injury.

Recommendations from the Phase 3 usability testing with clinical experts led to further substantive improvements in the eCDS tool. One major consideration that needed to be accounted for was that the PECARN rule is stratified into two criteria sets based on age and that the age of a patient presenting to a trauma resuscitation was not always known immediately. The appropriate PECARN TBI criteria could not be displayed immediately, and nothing could be displayed on the eCDS tool screen until age was determined. Because reorganization of the two criteria sets had shown significant overlap early in the assessment (Table 2), an initial starting display was created so that the common criteria could be displayed for assessment without age being known, and information could carry over to the age-specific list once the age was determined (Figure 4).

Figure 4.

The eCDS tool allows collection of risk score elements prior to knowing age. CT = computed tomography; eCDS = electronic clinical decision support; GCS = Glasgow Coma Scale; TBI = traumatic brain injury.

The other consideration was for the tool to provide intermediate feedback to the provider about how the assessment is going even if a final recommendation cannot yet be made. Specifically, it was possible to indicate that the patient was no longer considered high risk but should be continued to be assessed to determine low versus intermediate risk. This prompt could only be triggered if all three of the age-specific high risk PECARN criteria are absent (Figure 5).

Figure 5.

Intermediate decision support guidance.

After five clinical experts gave feedback on 10 video cases each (with a three-point scale of better, equivalent, or worse), the iteratively refined eCDS tool was rated better than the existing CDR in 84% of respondent–case pairs and rated equivalent in the remainder. Content analysis did not identify any new barriers to eCDS tool use. Barriers to acceptability were not matching clinical workflow (6%), medicolegal concerns (4%), and the timing/redundancy (8%) of data display (Table 3). Although not directly comparable, these rates were substantially lower than the previously solicited CPOE tool barriers (Table 1).

Table 3.

Frequency of Identification of Barriers to Decision Support

Barrier	New eCDS Tool, % (Expert–Video Pairs), n = 50
Information redundancy	8
Fit to clinical workflow	6
Medicolegal concerns	4

eCDS = electronic clinical decision support.

DISCUSSION

Our study demonstrates the importance of considering workflow and usability in design of CDS tools. Use of the PECARN TBI decision rule occurs in less than 20% of trauma resuscitations at our institution and physicians cite workflow and usability as major obstacles. Using a human factors approach and iterative design, we were able to substantially improve the match to workflow and perceived usability of the tool to support decision making in the trauma resuscitation setting.

The Agency for Healthcare Research and Quality has promoted the use of human factors analyses of usability and workflow to improve health information technology (IT) implementation.³⁰ Usability evaluation methods have improved work in ambulatory care settings by engaging end users and simulating the clinical realities of use.³¹ Previous work applying paper checklist interventions has shown that human factors implementation is possible in operating rooms and intensive care units.³² Little human factors research has been conducted in other critical care medical settings,^33–35 including trauma resuscitation rooms.³⁶ This study is a novel translation of human factors usability and workflow methods to inform and improve diagnostic imaging eCDS in the pediatric trauma resuscitation environment.

The survey-identified barriers are consistent with prior research in that the two primary errors with health IT are problems with data entry and data display and poor provider communication and coordination with clinical workflow.^23,37 The value of human factors methods for health IT implementation has been highlighted previously^38,39 and is considered core content in the specialty of clinical informatics.⁴⁰ The benefit of a human factors approach to improve usability and workflow of CDS was apparent in our study, as those two primary barriers were markedly reduced with the new eCDS tool (Table 3).

Dissemination and implementation remains a challenge for the translation of research findings into clinical practice, including emergency care.⁴¹ In contrast to clinical practice guidelines that have had well-publicized difficulties with adoption,^42–44 CDRs have been a focus of clinical researchers because they are designed with bedside application in mind. Despite this intent, challenges remain for their implementation, as highlighted by previous work and our experience with the CPOE tool in the trauma resuscitation room.^45–48 Part of this implementation barrier is inherent from CDR research methodology. Elements of CDRs are selected for reliability and then the statistical modeling organizes the criteria for optimal discrimination of the outcome of interest. When disseminated as peer-reviewed publications, CDRs are often presented in a way that matches the statistical model output or are perhaps organized in a way perceived to be easier to remember. Our work highlights how engaging usability experts and end-user practitioners to consider the clinical work environment can help reorganize and adapt CDRs as eCDS tools to better fit workflow and promote adoptability. When considered within a conceptual framework for implementation science,⁴⁹ human factors design is a valuable strategy for improving intervention characteristics. Our study complements other ongoing human factors research using recommended implementation planning strategies⁵⁰ to achieve better use of the PECARN TBI CDR.⁵¹

LIMITATIONS

The results of this study should be considered with some limitations in mind. First and foremost, we are describing a usability study to design a better eCDS tool and therefore are focused on implementation outcomes, not clinical outcomes. In fact, usability testing should not be undertaken in the clinical environment without extensive protections against patient harms. This is one of the major issues identified in prior studies and case reports where electronic health record features are put into production without scrutiny by human factors methods.^17–20 Instead, we opted to use sound human factors methods to do simulated usability testing. Having now demonstrated that the new eCDS tool is usable and acceptable using valid human factors techniques, we have IRB approval to pilot test the eCDS tool in the trauma bay for feasibility and effect on actual clinical care, with all members of the trauma team participating.

Because of the workflow-specific nature of this usability evaluation, the eCDS tool end product is specific to the institution and clinical area where it was studied. It cannot and should not be transplanted into a different location, but rather the whole human factors engineering approach outlined should be undertaken to design a custom-fit eCDS tool. Couched within some of the popular conceptual frameworks of implementation science, our usability approach considers the inner setting (culture) and individual characteristics when planning an implementation, and adheres to the recommendation that implementation approaches should be “multifaceted, multilevel implementation plans tailored to local contexts.”^49,50

The usability evaluation approach operates under a risk of bias. In particular, the heuristic evaluation and usability evaluation steps involve person-to-person communication with the study investigators. This interaction may result in a bias to perform evaluations that favor the eCDS tool. A prior systematic review did show that those eCDS tools implemented where they were created are often found to be more successful.⁵² Rather than consider those studies to be biased, it could be argued that those studies used better implementation methods. This highlights the tension between mitigating bias in the execution of an empirical scientific approach and pursuing an evidence-based mixed-methods implementation approach. Creating blinding to mitigate bias would limit the depth and richness of these key improvement steps, where exchanges of ideas allowed for substantive iterative changes to the eCDS tool.

Our study design did not allow direct comparison of the relative merits of the existing CPOE tool and the new eCDS tool. The CPOE tool barriers in Phase 0 were elicited as a general question to providers anonymously, and the eCDS tool barriers in Phase 3 were evaluated on an expert–video pair case-by-case basis. It is therefore not possible to compare the proportions of reported barriers. Our formative assessment was focused on creating a refined eCDS tool, and the future summative evaluation with testing in the live clinical environment would provide better comparison measures. A planned follow-up anonymous survey of eCDS tool would allow a more direct comparison with the CPOE tool, in addition to other outcome metrics to measure effective implementation.⁵³

CONCLUSIONS

This study shows the utility of human factors engineering in the design of an electronic clinical decision support tool. The iteratively refined electronic clinical decision support tool is now ready for live clinical feasibility and acceptability testing. The goal of this work is to incorporate the electronic clinical decision support tool into the process of care to display real-time guidance to medical providers for head CT imaging in children with head injuries during trauma resuscitations. Effective implementation of an electronic clinical decision support tool may reduce the number of unnecessary CT requests for children at very low risk for traumatic brain injury. This methodologic approach could be applied to improve the implementation of electronic clinical decision support tools for other medical conditions evaluated in the ED.