Design Challenges in Converting a Paper Checklist to Digital Format for Dynamic Medical Settings

Abstract

We describe a mobile digital checklist that we designed and developed for trauma resuscitation—a dynamic, fast-paced medical process of treating severely injured patients. The checklist design was informed by our analysis of user interactions with a paper checklist that was introduced to improve team performance during resuscitations. The design process followed an iterative approach and involved several medical experts. We discuss design challenges in converting a paper checklist to its digital counterpart, as well as our approaches for addressing those challenges. While we show that using a digital checklist during a fast-paced medical event is feasible, we also recognize several design constraints, including limited display size, difficulties in entering notes about the medical process and patient, and difficulties in replicating user experience with paper checklists.

1. INTRODUCTION

Checklists have become an integral component of healthcare safety procedures and are now routinely used in many medical settings, including emergency departments [9], operating rooms [1] [10], and intensive care units [3]. While successful in preventing and reducing errors in some domains, checklists have been criticized for poor design and a failure to achieve their full potential in healthcare [4]. Most checklists are paper-based and static, even when displayed on a screen, limiting adaptability to dynamic and often unpredictable medical scenarios. Although paper checklists still dominate, attempts have been made to digitize checklists to improve compliance and allow easier access to information from them. Digitization of checklists may also lead to: improved clinical decision making and care coordination by allowing faster access to data for process improvement and research; integration with other hospital systems; shared access to information through common displays; and context-based adaptation. The process of converting paper checklists to a digital format, however, is not straightforward and has several challenges, including the need to present the correct tasks at the right time; to adapt to contextual information, like patient status, available providers or equipment; to prevent mistakes commonly found with paper checklists, such as skipping items or choosing the wrong list; and, to be easy to access and use [7][12]. Checklists for dynamic, time-pressured crisis situations, such as medical emergencies, face an additional set of design challenges associated with rapidly emerging data.

We describe a mobile digital checklist system that we developed for trauma resuscitation—a dynamic, fast-paced medical process of treating severely injured patients early after injury. Clinicians responding to medical emergencies such as trauma resuscitation must rapidly perform a defined set of tasks that are critical to successful team performance. Despite the widespread use of standard evaluation and management protocols, compliance with performing critical tasks during patient evaluation is low [11]. To assist with memory recall of procedural items in settings with high cognitive load, clinicians have used mnemonics when designing protocols (e.g., “ABC” for “Airway, Breathing, Circulation” in resuscitation), but even these strategies are prone to error [5]. Drawing from experiences with checklists in other safety-critical domains such as aviation, researchers have designed a trauma resuscitation checklist as an alternative approach to improving team performance [14]. This paper-based checklist contains a total of 65 items grouped into four sections: pre-arrival plan, primary survey, secondary survey and prepare for travel (Figure 1). It was deployed in actual resuscitations, with a recent study showing its positive impact on team performance and protocol compliance [9]. Here, we focus on the process of converting this paper checklist into an electronic format. We first review prior research that informed the initial system prototype, and then follow with the checklist design, the rationale for our design decisions, and the challenges we encountered. We then present findings from a feasibility study in which the system was tested in real resuscitations to further explore the application area. Our key contributions to pervasive health are:

• Design of a digital checklist for dynamic medical domains

• Identified challenges in converting paper forms into computerized systems for dynamic medical domains

• Proposed approaches for addressing these challenges

Figure 1.

Paper-based checklist with handwritten notes from an actual trauma resuscitation

2. BACKGROUND & RELATED WORK

Trauma resuscitation is one of the most challenging processes in healthcare. It requires the focused effort of an interdisciplinary team of up to 15 specialists over a short time (on average, 20–30 minutes). Trauma patients are evaluated using a two-part protocol based on Advanced Trauma Life Support (ATLS). Team members first conduct the “primary survey,” a rapid evaluation of major physiological systems such as Airway, Breathing, Circulation, and Disability (ABCD). This initial evaluation is followed by a more detailed assessment or “secondary survey,” to identify other injuries. Over the course of the patient evaluation, trauma team members must make rapid decisions while managing a large amount of information obtained from inside and outside the room in which the patient is being evaluated.

Trauma resuscitation differs from other clinical domains in which checklists have been implemented. During resuscitation, most team members are busy performing time-critical tasks, which does not always allow a “time-out” for performing a checklist. Modifications to some tasks may also be needed to adjust to the anatomical and physiological aspects of the patient and the patient’s injuries. In addition, the checklist administrators must actively use the checklist to ensure task completion, which may interfere with their performance of direct hands-on patient care. These differences require a domain-specific approach to checklist development and implementation.

2.1 Digital Checklists in Medical Settings

Existing digital checklists are either plain translation or screen projection of their paper-based counterparts. For example, Hulfish et al. [8] compared trauma team performance across three scenarios—no checklist, a handheld paper checklist, and a checklist displayed on a monitor visible to all team members. They found that the displayed checklist significantly reduced completion time for both the primary and secondary surveys. Thongprayoon et al. [18] developed an electronic checklist with 24 interactive items grouped by organ systems and a decision support tool that provides relevant information for each checklist item. After comparing the use of paper vs. electronic checklists in an ICU, they showed that the electronic checklist significantly reduced provider workload and errors, with no measurable increase of time for checklist completion. The SURgical PAtient Safety System (SURPASS) checklist is another example of a checklist converted to an electronic version [6]. The SURPASS Digital, a web-based checklist support system for any platform, was developed over the course of six weeks and was integrated into the hospital’s information systems [17]. Despite its mobility and ease of access, the checklist contains 18 webpages of items, directly translated from the paper-based SURPASS and presented in a sequential order. It also includes several advanced features, such as reviewing the log of checks and preventing task skipping through the application’s validation logic. This previous research showed that digital checklists are feasible in critical care settings, but did not describe checklist design or the challenges associated with converting paper forms to digital formats.

Although medical checklists are largely paper-based and static, there have been proposals to make them dynamic through computerization. For example, Nan et al. [13] proposed Tracebook, a dynamic, context-aware checklist system. The proposed system would support both pre-defined and ad-hoc processes, filter tasks and display information based on medical contexts, automatically trigger tasks based on clinical events, and support note taking and information exchange among care providers using a social networking platform. Similarly, Avrunin et al. [2] proposed a smart checklist system that would monitor and guide clinicians through a medical process based on a precisely defined process model. Using context information, the system would dynamically adapt and filter tasks, while addressing specific scenarios. Most of this work is still at the conceptual level, offering a vision for smart, interconnected systems that dynamically adapt content based on the patient context. Despite their potential, it is unclear how these proposed systems would function in the real world and whether they would improve patient outcomes. There is also a lack of detail about the design process and the rationale for including features such as social networking.

Wu et al. [19] developed dpAid, a web application that displays procedure aids and resources, synchronized across multiple displays for medical crisis situations. The system was built using fieldwork and participatory design with anesthesiologists. Although their design implemented established protocols that are currently presented on paper aids, the design process did not focus on converting any particular paper forms. The resulting system has several components in addition to the checklist. Its primary purpose, however, is decision support, guiding users through the emergency medical protocols and suggesting tasks and treatments. We build on this work by contributing new knowledge in the area of medical checklist design and, more specifically, by identifying challenges and approaches to addressing them when converting paper checklists into their digital counterparts.

For medical checklists to achieve their goals, new design approaches are needed. A recent publication called for this shift in design paradigms, stating: “(t)he continued success of checklists in healthcare requires solving an interaction design problem” [7]. Checklists should remain simple, but also become more dynamic and adaptive to different contexts and users. Our main design goal was to maintain simplicity of paper checklists while providing the important additional functionality of adaptability.

3. FROM PAPER TO DIGITAL FORMAT

Our digital checklist design process followed an iterative, user-centered approach. The initial design was based on preliminary research, in which we analyzed user interactions with paper checklists. As the design progressed, we involved medical experts on our research team, asking them to trial the checklist and evaluate the design from a user perspective.

3.1 Preliminary Research to Inform the Initial Checklist Prototype Design

We began the process of converting the paper-based trauma resuscitation checklist to a digital format by analyzing user interactions with the paper checklist. Based on an initial review of paper checklists completed during actual resuscitations, we found that physician leaders frequently scribbled notes on their checklist sheets. To better understand these use practices and the nature of note taking, we obtained paper checklists from 163 resuscitations that occurred in a pediatric level 1 trauma center during a four-month period in 2012. We first used an open coding technique to identify types of information written down by physician leaders. We then performed statistical analyses to quantify notes under each category. We measured the frequency of checked and unchecked items, as well as the number of recorded notes and their type. To gain additional insight, we examined whether the amount and type of checklist notes depended on the physician-leader’s expertise level.

The results showed that physician leaders recorded 27 information types grouped into nine high-level categories: (1) patient values (e.g., vital signs); (2) physical assessment findings and body location; (3) pre-hospital information (e.g., mechanism of injury, prehospital treatments, medical history); (4) care plan (e.g., labs, consults); (5) injury type and location; (6) task completion status (e.g., deferred, not done); (7) treatments and procedures (e.g., fluids, medications); (8) laboratory results; and, (9) other information. We observed that more experienced physician leaders (e.g., attending physicians) took notes more frequently than less experienced leaders (e.g., fellows and residents). In addition, we found that more experienced leaders recorded more patient values (e.g., vital signs, temperature) and findings from physical assessments (i.e., secondary survey findings), two types of information necessary for decision making. They also recorded more notes about the patient’s subsequent care plan. Conversely, less experienced leaders recorded more notes about their activities, such as treatments and procedures, as well as about their task completion status.

Our preliminary research suggested that a checklist designed for a high-risk, fast-paced medical event has evolved into a dual-function tool, serving both as a compliance and decision-making aid. Based on these findings, we formulated early checklist designs for a tablet screen and focused on supporting this dual function. We used Android as our mobile operating system to allow for easier integration with existing hospital systems, as well as integration with the decision-support system components that we are currently developing.

3.2 User Participation in the Design Process

In addition to using the insights from analysis of paper checklists, we conducted two evaluation sessions with medical experts on our research team to ensure that the checklist design meets the needs of experts. The first evaluation session involved an hour-long focus group via teleconference with four medical experts to obtain user feedback on an early prototype. We first demonstrated our digital checklist, walking the users through the checklist features. We then asked for general feedback on the overall design, as well as for specific feedback about the features and functionalities. The second evaluation session took place at our research site—a level 1 trauma center in an urban, teaching pediatric hospital. We conducted usability tests with two medical experts, each lasting an hour. We replicated a trauma resuscitation scenario by asking each user to complete the digital checklist as they watched a video recording of an actual resuscitation. We started with a short introduction to the checklist, showing the main features, and then observed users while they were completing the checklist. Each test concluded with a debriefing and questionnaire. In the next section, we highlight the main features and functionalities of the digital checklist. For each feature, we discuss our design decisions, and how our preliminary research and user participation informed those decisions.

3.3 Checklist Features & Design Rationale

We initially focused on converting all features and functionalities from the paper checklist for uninterrupted use and smooth transition between the two form factors. We then adapted the design to fit the requirements for mobile interaction and to make access to information easier.

3.3.1 Checklist Navigation

A major design question was how best to model the sections of the paper checklist for easy navigation, including the Pre-arrival Plan, Primary Survey, Secondary Survey, and Preparation for Travel sections. A tablet-size display limited our ability to show the entire paper checklist on one screen, requiring an approach for organizing the checklist sections to allow seamless navigation between them. We opted for a tab-based system, with each tab representing a section of the paper checklist (Figure 2(a)). This layout allows users to rapidly select any section of the checklist during resuscitations, as well as to swipe between tabs for a smoother transition.

Figure 2.

Primary Survey and Pre-arrival Plan screens from the digital checklist: (a) tab-based design highlighting the current tab and showing the number of remaining items on each tab; (b) drawing area for taking “margin” notes; (c) expanded “If intubating” checklist section; (d) white rectangle fields for showing recorded notes; (e) drawing area for taking item-associated notes.

3.3.2 At-a-Glance Overview of the Process & Tasks

To provide an “at-a-glance” overview of the resuscitation process across tabs (i.e., completed vs. remaining tasks), we included the number of remaining checklist items on each tab as a subheading underneath the tab name (Figure 2(a)). Whenever a user checks an item on a specific tab, the remaining number of checkboxes on that tab decreases, reducing the total number in the subheading. We further enhanced this mechanism by color-coding the subheading. The subheading remains red until all items are checked, after which it turns green. In addition, the text changes from “Items remaining: n” to “Complete”. Because checklist sections have different number of items, some tabs required scrolling after their content was converted for a tablet (e.g., Primary Survey tab). A medical expert during the first usability session observed that scrolling might be problematic because the rapid pace of the resuscitation prevents complex interactions with the system. To reduce the number of checklist items on the Primary Survey tab, we moved the items related to vital signs to their own tab, “Vitals” (Figure 3). This change reduced scrolling on the Primary Survey tab while also organizing all of the vitals information into a single tab.

Figure 3.

Vital signs tab.

We used a different mechanism to provide a quick overview of unchecked items within tabs. We initially decided to color-code unchecked items as red and checked items as black. Once the item was checked, its text color transitioned from red to black. We assumed that the difference in contrast between each state would provide a simple visual cue throughout the use of the checklist. This coloring scheme turned out to be counterintuitive, as observed by a medical expert during the second usability session. Red-colored checklist items may indicate a degree of importance or priority, rather than simply an unchecked item. This expert user suggested using a black color for unchecked items and fading them into gray after they are checked. Our current system uses a black text color for unchecked items and 30% opacity for items that are checked. It was important to deemphasize items that are checked because prior research has found that people often check items before task completion, in anticipation that the task will be performed [14]. They then end up skipping the task or performing it, but forgetting to check it off. Our design addresses this challenge by reducing the clutter and increasing user awareness of tasks that are completed and tasks that remain.

3.3.3 Adaptive Checklist Content

Another important design decision was how to implement optional checklist items, indicated by a “Not Applicable” or “N/A” checkbox. Prior research has shown that checklists containing irrelevant or not-applicable information routinely lead to task skipping and deviations from well-established procedures [7]. Our initial approach was to implement an optional item in a manner that mimicked user interactions with the “N/A” checkboxes on the paper form. If a user checked an “N/A” item on the paper, he or she would skip all of the subordinate checkboxes associated with that “N/A” item (e.g., “For Attending level activations” checklist item in the Pre-Arrival Plan section, Figure 1). Once an “N/A” item was checked in our initial implementation, all checkboxes associated with that item became disabled, reducing the count of the unchecked items in the tab by the number of items under that “N/A” item. This design, however, contradicted our findings from paper checklists showing that tasks under optional items were rarely performed, thus almost never checked. For example, “Prepare for Travel” was checked off in only 8% of the checklists that we analyzed, and “If intubating” in 10% of the checklists. A medical expert during the second usability session added that team leaders often check the “N/A” checkboxes at the very end of the resuscitation, while finalizing the checklist. Our current design incorporates these insights by having all of the N/A checkboxes checked off at the initiation of the checklist by default with the exception of the “Prepare for Travel” section. In case any of those items become applicable during the resuscitation, the user can uncheck them and complete the subordinate items.

The “Prepare for Travel” optional section (Figure 1) became a separate tab, with just one N/A checkbox for the section, mirroring the paper checklist. Because this tab contains only these four items, checking them all by default at the checklist initiation would make this tab complete, potentially leading to skipping this task. We also used a different approach for the “If intubating” optional item in the Primary Survey section and “For Attending level activations” item in the Pre-Arrival Plan section (Figure 1). We decided on a simple expand-and-collapse button that hides the subordinate items until the user clicks the expand button, which then reveals and enables those items (Figure 2(c)). This design iteration significantly reduced the needed screen space, while also preserving ease of use (i.e., less scrolling). In general, the way we implemented and changed the optional items reduced the overall number of clicks. It also made the checklist more focused. By explicitly showing only items that are relevant to specific scenarios, we are increasing the likelihood that the checklist will be used.

3.3.4 Note Taking

Our design process also addressed the note-taking functionality, given its critical role on the paper form. We wanted to allow users to scribble notes on the digital checklist as easily as they do on paper. Although handwriting recognition appears to be a straightforward solution, it has had limited success in emergency medical domains because automatic character recognition is unreliable for rapid writing [16]. We used an alternative approach, in which users still write or draw notes using a stylus or finger, and the system captures an image of the note and displays it next to the related checklist item.

Our initial decision was to make note taking possible anywhere on a tab. This choice, however, posed several challenges. First, scribbling notes next to the small text-sized checklist items was difficult. Writing small letters on a mobile device can be a challenge, even when using a stylus. Second, unconstrained location of writing would prevent correlating a note with a specific checklist item for either research or archival purposes. For example, a user could jot down a note on the top of the screen that was related to a checklist item on the bottom of the screen. Capturing this relation, unless specifically stated in the note, would be impossible when analyzing completed checklists. To address these challenges, we modified our design by constraining note taking next to the associated checklist item. The user can take a note related to a checklist item by tapping a white rectangle to the right of that item (Figure 2(d)). When tapped, this white rectangle expands a hidden, larger drawing area at the bottom of the screen (Figure 2(e)). The user can then write notes in this section and tap a checkmark button when finished. Once the checkmark button is tapped, the large drawing area retracts and the white rectangle next to the checklist item is replaced with a miniaturized image of the note. This modified design allows users to make clearer and more accurate notes. It also allows correlating each note with a specific checklist item. The notes can be edited at any time by re-tapping the white rectangle and adding to or erasing parts of the existing note. We implemented the editing feature because we found that physicians using the paper checklists frequently added new information to their existing notes (e.g., trends in vital signs) or made corrections based on subsequent observations (e.g., struck-through notes).

An important functionality missing from this implementation was allowing users to take notes that were not related to any specific checklist item. These “margin” notes frequently appeared on paper forms and included information such as patient demographics, details about the injury, prior hospitalizations, or lab results. To enable margin notes on the digital checklist, we added a note-taking area at the top of the screen that is accessible from any tab (Figure 2(b)). For example, a note taken in the margin area for the Pre-Arrival Plan tab is also visible from the Secondary Survey tab. This area is always active and does not require button clicks for note taking.

3.3.5 Recording Patient Values

Our content analysis of paper checklists showed that physician leaders write down a great deal of information on their checklists. Patient values, such as weight, vital signs, distal pulses, neurological score called Glasgow Coma Score (GCS), and temperature, comprised 35% of all notes written on the checklists we analyzed. We observed notes about vital signs appearing next to their corresponding checklist items and GCS values scribbled next to the “State GCS” item (Figure 1). These findings led to a design feature that better supported inputting patient values for vital signs and GCS, the most frequently written values on paper forms. For these items, we replaced the white note field with a simple text-input field that, when tapped, expands a numeric keyboard from the bottom of the screen, allowing the user to type in the values (Figure 3).

This design change raised an issue about whether recording patient values on the checklist should be mandatory. On the one hand, recording values for vitals or GCS is not mandatory on the paper checklist. As we have observed, recording values is a matter of preference, which suggests that it should be kept optional on the digital checklist. Yet a clearly delineated space for typing a value may lead to conclusion that entering values is mandatory. To make entering values truly mandatory, we would need to treat the corresponding checklist item as incomplete until the value is entered, i.e., it would not be counted towards completed items in the subheading of each tab. Possible confusion about whether or not entering values is mandatory may potentially be managed by training. On the other hand, by associating the recording of values with item checking, we would create an effective means for improving checklist compliance, as described next.

3.3.6 Automatic Check Off for Items with Values

Prior studies of checklist compliance have shown that people often check off items without performing the tasks, or they perform tasks, but then forget to check them off on the checklist [12][14]. While paper forms can do little to improve compliance, digital checklist systems can do so by using any user input to infer context and apply rules. For example, entering a blood pressure value and a GCS score provides enough indication that the user had taken the patient’s blood pressure and performed neurological exam, i.e., completed those checklist items. Having received this input, the system can save the user some clicks by automatically checking off those items. At the same time, if the user checked off an item that asks for a patient value but did not enter any values, the system could interpret this checklist item as incomplete and issue a warning that the value is missing. Our users also felt that it was redundant to type in a value and then check the box, because entering a value is a valid approach for inferring task completion.

We have implemented the automatic check-off for items that allow entering patient values as follows: (a) when the user enters a patient value (i.e., weight, GCS score and vital signs), the system automatically checks that item off, counting it toward completed tasks by decreasing the number of remaining checkboxes on the tab subheading; (b) if the user checks off the item but does not record its value, the system leaves the item unchecked and counts it as incomplete; and (c) when neither checkmark nor value are entered, the system counts this item as incomplete. Once the user is ready to finalize the checklist, the system alerts the user with a warning box, listing all unchecked items and those with missing values, prompting the user to complete them.

Based on a user suggestion, we initially implemented the automatic check-off for all items that have a white rectangular note field next to them. This implementation was problematic because handwritten notes are a poor indication of task completion; notes are not required and may be entered before or after task completion. As we learned over the course of the study, leaders sometimes take a note well before the task is completed (i.e., during multi-step tasks) and then go back to check off the box after the task is completed. Although this automatic check-off for all notes reduced the overall number of clicks, it proved counterintuitive and we removed it.

3.3.7 Checklist Reports & Logs

An important factor in developing the digital checklist was preserving the ability to analyze user interactions with the checklist, just as we did with paper forms. An added value of the digital format is a linear log of checked items and notes with timestamps, allowing us to broaden our analysis of interactions and use the findings for further design improvements. We have implemented a report system that integrates with a cloud computing service. Once the user has finalized the checklist and selects the checklist icon to complete the checklist, a “Report” screen is shown with the log of checked items with timestamps, recorded margin notes, and notes related to specific checklist items. The user can then click the “Submit Report” button, which displays a success or failure message, depending on whether the file was successfully uploaded. If the upload is successful, a text file and an image file are generated and sent to a cloud storage service. If the upload is not successful, a dialog box displays a message with the error report.

4. FEASBILITY STUDY

We conducted a limited user study to assess the feasibility of using the digital checklist during trauma resuscitations.

4.1 Research Setting & Methods

Two experts in trauma resuscitation on our research team trialed the digital checklist while observing three resuscitations at our research site, a pediatric level 1 trauma center. As part of the study, we asked the experts to provide feedback after each trial about usability issues with the checklist, general observations, or areas for improvement. We received this feedback via email after each checklist trial while the impressions were still fresh with the user. We also used our weekly research meetings to further discuss checklist design issues and clarify any ambiguities.

4.2 Findings from Digital Checklist Trials

4.2.1 Checklist Completion Rates

Our analysis of checklist logs from all three events showed that users left some items unchecked. Of all sections on the checklist, the Pre-Arrival Plan had the largest ratio of unchecked items to total items in that section, at 28% unchecked. This is consistent with our findings from the paper checklist analysis, where the pre-arrival plan had the highest number of unchecked items of all sections. The Primary Survey and Vitals tabs had 100% completion rate in all three events. The Secondary Survey tab had an average of 12 out of 14 items completed. The “Pause” section with two items located at the bottom of the Primary Survey tab was completed once and left unchecked the other two times. We considered this a usability issue because the user must scroll down to view the “Pause” section. The average completion rate of checked items to total items across all three events was 86%.

4.2.2 Note Taking

Users took a total of eight notes across the Pre-Arrival Plan, Primary Survey and Secondary Survey tabs. Two notes corresponded to the “State GCS” item and two to the “Pupil size and response” item. The remaining notes corresponded to the “Brief team on incoming patient,” “Check distal pulses,” “Temperature,” and “Lower extremities” items. Of these eight notes, five were text-based and three numeric (e.g., “4 3 14 6” next to “State GCS”). Two of the five text-based notes were one word only (e.g., “reactive” next to “Pupil size and response”), one note had two words (e.g., “equal bilateral” next to “Pupil size and response”), one note had three words (e.g., “R hip broken” next to “Upper extremities”), and one note had more than four words (e.g., “intubated en route, some pain” next to “Brief team on incoming patient”).

When asked to comment about the note-taking feature, one user explained that she had difficulties writing notes legibly because she forgot to bring the stylus. In another event, this user mentioned that her note-taking behavior has changed with the digital checklist compared to the paper version:

“I am taking less notes because it is not as easy or intuitive for me. Notes are very useful for me, when talking to surgeons and updating them on patients, I have a record of what happened, so it is nice to have notes. So it’s more about the utility of the notes section on the digital form and I am not utilizing it the same way as the paper notes.”

4.2.3 Navigating the Checklist

We also examined the order in which the items were checked. One user frequently switched between the checklist tabs. At first, the user checked off the second item in the Primary Survey, went to complete the Vitals tab, and then returned to the Primary Survey to check off five more items. The user then moved to the Secondary Survey to check off three items, then back to Primary to note the patient’s temperature, and then back and forth a few more times between the Primary and Secondary survey tabs. When asked about the navigation issues, the user did not mention any problems with switching between tabs, noting that she was checking off items as the team was completing their tasks. The other user was switching less between tabs. An interesting observation was that in one event, this user checked temperature and the vitals signs at the very end of the resuscitation event.

Currently, it appears that switching between tabs did not have an impact on the time it took to complete the checklist, as completion times corresponded to the duration of the resuscitation events. The time from start to finish for the checklist with the most switching between tabs was 23 minutes 57 seconds. The checklist with the fewest switches between tabs had the longest completion duration, at 25 minutes and 1 second. The checklist that was completed in the shortest amount of time (18 minutes and 5 seconds) included switching between tabs only at the end of the checklist to fill out unchecked items on the Primary Survey and Vitals tabs.

5. DISCUSSION: DESIGN CHALLENGES

Our process of converting a paper checklist for a time-critical medical setting to digital format uncovered several key challenges. Most challenges arose from conflicting demands of the limited screen size for mobile devices and the need to display all logically related checklist items at once, while also allowing smooth transition between paper and digital formats. A design that considerably deviates from the paper version of the checklist may make the transition difficult. But paper, too, has limitations, so our question was what features to replicate and what features to design specifically for the digital medium. We next discuss these design challenges and the approaches we used to address them.

Grouping checklist items into tabs

The need for grouping all logically related checklist items into one tab may require scrolling the page to reach all items. But page scrolling, especially on a mobile device, makes the use of the checklist challenging. As we observed in our feasibility study, users may not scroll below the fold to check the items that are not visible by default. Although we provide subheadings under tab names to inform users about the number of unchecked items, they may ignore or not see this information in time. To address this challenge, we included a popup window listing unchecked items, if any, and asking for user confirmation that the checklist has been finalized. While inclusion of this feature may help improve the checklist compliance, the prompt may be appearing late in the process as some users may forget the values or tasks they performed by this time.

Related to the first challenge is placement of periodic-monitoring checklist items that have to be performed on an ongoing basis (unlike items that may or may not be repeated, such as assessment of breath sounds). It is well known that checklists are best suited for linear processes, where tasks are performed in sequence and the checklist ensures compliance. In many work processes, including trauma resuscitation, monitoring tasks need to be performed periodically throughout the process. The design challenge then is in arranging the corresponding checklist items on the display. As an example, checking the patient’s vital signs is one of several monitoring tasks during resuscitations. On paper, the issue of arranging the items was solved by placing the Vitals box under the Primary Survey. In reality, however, vital signs are checked more than once during the resuscitation. Because existing checklist designs do not support repeated performance of a same task, checking vitals was placed in this section to ensure that vitals are taken at least once and early in the process.

We placed the Vitals section on a separate tab for two reasons: (1) checking vitals is independent of the process phase, and (2) switching to another tab was considered more optimal than scrolling down the page. Users in our feasibility study initially looked for the vitals in the Primary Survey tab because of their prior experiences with the paper checklist. They commented, however, that they quickly familiarized with the new placement of vitals. Even so, further investigation is needed to identify a better solution for supporting periodic monitoring activities and adaptation to the actual work practices (“work as is”) as opposed to imposing “logical” solutions that disregard those practices (“work as imagined”) [15].

Checklist navigation and effects of the tabbed design

We observed different frequencies of switching between tabs for our users. Because resuscitation activities are performed in parallel, they may not necessarily occur in the order listed on the checklist. This parallel process progression requires navigation between different tabs to complete the checklist. A design challenge lies in making this navigation efficient. Here we notice again that it will be critical for the checklist to adapt to work as is. Observed switching between tabs showed that the process execution is nonlinear and needs to be supported as such. We implemented two modes of switching between tabs (tapping on the specific tab or swiping between adjacent tabs) to support different navigation modes, thereby minimizing a chance of slowing down the user.

Note taking

Prior work on using character recognition in real-time documentation during trauma resuscitation exposed several challenges in accurate recognition of rapid writing [16]. Our current solution is to capture images of handwritten notes, but this strategy prevents processing the written information and using it contemporaneously. It is also challenging to transfer such data into the patient record. An option is to provide typing and auto-completions though this solution may also face challenges due to the fast-paced nature of trauma resuscitation.

Related to note taking is the challenge of entering free-form vs. structured notes and validating item check-off. Our initial design used white rectangular fields for all notes, i.e., all notes were images of handwriting. After observing that users frequently recorded notes on vital signs that were mostly numeric values or structured text, we replaced note fields with text typing on a line for vital signs and GCS score. This approach, however, changed the manner in which the checklist is used for checking vital signs. Before this change, users were required to verbally report and validate vital-sign values, with note taking being optional. With the new approach, the line indicated that entering vital-sign values is mandatory, which is an unintended consequence of our design. Furthermore, when the user checks any vitals item but does not input the value, the system does not count that item as completed. Our future research will examine the implications of requiring the recording of patient values for interactions with the checklist.

Structured notes allow simple validation of user input and contribute to checklist compliance. For example, the system could automatically evaluate whether the entered value is logical and within normal limits for age, which is done manually for paper checklists (Figure 1). The system could also warn the user if the values were invalid or not entered. We observed, however, that even simple notes on vital signs contained more than just numeric values. For instance, blood pressure was recorded as “105 over palp[ation],” and oxygen saturation was recorded as “100% on nonrebreather” or “per ER was upper 80s on RA, 97% low NRB.” Further analysis is needed to determine whether these notes can be categorized into a set of templates to support structured input that can be automatically analyzed.

5.1.1 Study Limitations

Our study has several limitations. First, our validation of the current checklist design is incomplete. Data from the feasibility study were insufficient to determine the effects of the digital checklist on team performance, but they provided insights about the workflow, as well as about design and usability issues. Similarly, our second limitation was the sample size of users that have participated in this research. To date, only medical experts from our research group took part in the usability and feasibility studies, limiting the type and scope of feedback for the checklist design. Even so, we believe that researchers and practitioners will benefit from our results because they show how we identified and addressed the design challenges as we were converting a paper form to a digital format. It is important to note that design solutions described here represent our initial steps toward reaching our long-term research goal of developing a suite of decision-support tools to reduce errors and improve team performance during trauma resuscitation. Because this is a complex problem that may take years to solve, it was critical to assess if digital checklists are feasible in such fast-paced medical domains before we invest additional resources.

6. CONCLUSION & FUTURE WORK

We described a digital checklist system that we have designed and developed for trauma resuscitation, a dynamic medical setting with a high cognitive workload. Our initial design was informed by analysis of interactions with a paper checklist and was improved through several iterations based on expert-user participation and feasibility study. The main challenges arose from the limited display size and difficulties in automatic recognition of user handwriting. Additional concerns were about the extent to which the digital checklist should mimic the paper checklist to facilitate smooth transition. Finally, some choices of note-taking modes (handwriting vs. typing) had unintended consequences of changing the checklist completion requirements.

Our future work will move in several directions. First, we are planning extensive evaluation studies and clinical trials for the digital checklist and different checklist administration methods (e.g., calling out each item and confirming task completion before moving onto the next item, or calling for a pause at various times and then confirming task completion) to assess their effects on team performance and patient outcomes. Second, we will keep improving the design based on the observed use of the checklist during resuscitations. More users will be trialing the system, which will in turn provide rich feedback. Third, we are using current checklist design to initiate our smart checklist project and develop a system that will automatically adapt to different patient contexts. This is part of our larger effort through which we are developing a smart resuscitation room, and the checklist will be one of several tools critical to team performance. For example, an important functionality that our current digital checklist does not have is recording multiple instances of the same task when necessary. Some resuscitation tasks may need to be repeated during the process, but current checklists allow recording only the first instance. Finally, we also plan to integrate the digital checklist with the electronic health record for easier access to patient data.