Understanding Digital Checklist Use Through Team Communication

Abstract

Introducing technology support in a complex, team-based work setting requires a study of teamwork effects on technology use. In this paper, we present our initial analysis of team communications in a trauma resuscitation setting, where we deployed a digital checklist to support trauma team leaders in guiding patient care. By analyzing speech transcripts, checklist interaction logs, and videos of 15 resuscitations, we identified several tensions that arose from the use of a checklist in a team-based process with multi-step tasks. The tensions included incorrect markings of in-progress tasks as completed, failure to mark completed tasks due to missed communications, failure to record planned tasks, and difficulties in recording dynamic values. From these findings, we discuss design implications for checklist design for dynamic, team-based activities.

Introduction

Trauma resuscitations are time-critical medical events, in which a multidisciplinary team of care providers follows the Advanced Trauma Life Support (ATLS) protocol to evaluate and manage critically injured patients. Most resuscitations take place in the emergency department and are led by a team leader, usually a fellow or senior surgical resident. The rest of the team includes junior residents, nurses, respiratory therapists, anesthesiologists, technicians, and social workers. Although critical, the process is prone to errors. A recent study has shown that fifty percent of trauma resuscitations had errors due to missed tasks, process delays, or poor judgement [4]. When combined, these errors can have adverse effects on patient safety and can contribute to patient morbidity and mortality. Prior research has introduced technology in this setting to reduce errors, and improve teamwork and protocol compliance [6–8, 13, 20]. While most prior studies have focused on understanding the effects of technology on teamwork [1, 13, 15, 20], few have studied the effects of teamwork on technology use.

In this paper, we present an analysis of team communication in conjunction with the team leader’s use of a digital checklist for trauma resuscitation. The design of the digital checklist was based on a previously used paper checklist that was introduced to support protocol adherence. The checklist contains five sections–Pre-arrival Plan, Primary Survey, Secondary Survey, Vitals, and Prepare for Travel–each listing the tasks associated with a phase in the ATLS protocol (Figure 1) [13]. Users can also enter notes in designated fields next to each item and in the top, “margin” area of the checklist. To determine if tasks have been completed (and can be checked off), the leader relies on verbal reports about task status or findings from team members who are performing the tasks. If verbal reports are missing, the leader clarifies with the team before they check them off. Our study of team communication showed several mismatches between the information recorded on the checklist and the state of the tasks performed during the resuscitation, highlighting tensions that arise from the use of a static checklist for dynamic, team-based processes. This work represents our initial step towards understanding how complex teamwork affects the use of a newly introduced technology, offering design recommendations that could alleviate the identified tensions.

Figure 1:

Digital checklist screenshot

Background & Related Work

Checklists have been introduced in medical settings to reduce errors and improve patient outcomes as memory-and decision-making aids [12]. Studies of electronic checklists in critical care settings have shown mixed results, with some studies finding reduced errors and workload [17] and others showing irregular use and low compliance rates [18]. Physicians in crisis situations frequently just “sample” various parts of the checklist at different times [3], while acute care leaders administer the checklist depending on the details of a particular case [9]. Checklist use is especially challenging in team settings where multiple activities are occurring simultaneously and not all team members share the benefit of viewing the checklist. If designed with teams in mind, checklists can be used in these settings as a “teamwork and communication tool” to support work coordination [5]. However, physicians often struggle to balance the work of interacting with a cognitive aid while also interpreting information and coordinating the team [2]. Checklists can also provide a false sense of security if they incorrectly indicate that certain tasks have been completed [19]. We therefore need a better understanding of how checklists are used in complex teamwork settings and how the team communication shapes their use.

Our previous work studied the use of both paper and electronic checklists in trauma resuscitation. We found that paper checklists were used as both compliance and memory tools, with patient values and task status jotted down on the checklists [11, 16]. We observed the increased frequency of notes and completion rates with the introduction of the digital checklist [13]. While studying the nature of team interactions during resuscitations, we found that the leaders mostly relied on communication with other team members, instead of direct observation, to gather information [10, 14]. In this study, we used data from the timestamped digital checklist logs and examined communication patterns through video review and audio transcripts to understand how team communication influenced digital checklist use.

Methods

Data were collected from 15 resuscitations occurring between January and May 2017 at a pediatric trauma center in the northeast region of the United States. The study was approved by the hospital’s Institutional Review Board (IRB). A digital checklist for trauma resuscitation, installed on a Samsung Galaxy tablet, was used by the team leader in all 15 resuscitations (Figure 2).

Figure 2:

A mock-up illustration of a physician using the digital checklist.

Six unique team leaders led the resuscitations, four of whom were senior residents and two were fellows. Two resuscitations were led by two or more leaders. The patients ranged from infants to 14 year old children, and all sustained blunt injuries. Two resuscitations were activated at the “attending” level, which indicates the highest acuity patients. Seven resuscitations involved patients who were transferred from another hospital. The remaining six resuscitations were activated at the standard, “stat” level. Eleven resuscitations occurred during the day and four occurred at night.

Each resuscitation was video- and audio-recorded. Audio recordings were manually transcribed after receiving patient/care-giver consent to use these data for research purposes. The transcripts include a line for each utterance, along with timestamps and the speaker’s role. Medical experts reviewed each video to record the start and finish times for all performed resuscitation tasks, generating the ground truth activity logs. The digital checklist produced a log with the timestamps of activity check-offs and any associated values or notes entered for each resuscitation.

Data analysis involved a three-step process. First, we reviewed each audio transcript in conjunction with the corresponding video. While reviewing, we assigned every line of communication from the team leader one of four communication categories: a team leader instructing a team member to perform a task, a team leader asking a question about a task, a team leader sharing knowledge about prior events, and a team leader confirming they heard another team member. Next, we compared every line of communication from the leader with their use of the checklist at that time according to the checklist logs. Finally, we compared the activity in the checklist log with the ground truth activity log to see if the checklist record matched the activities that were actually performed during the resuscitation. The output of this analysis was a file containing (a) each utterance from the team leader from all 15 resuscitations, (b) the communication category of those utterances, (c) any action that was performed on the checklist at the time of the utterance, and (d) a description indicating if the information recorded on the checklist matched the log of performed activities.

Findings

Our study identified several instances where team communication showed a discrepancy between the state of the resuscitation and the information recorded on the checklist. We classified these instances into three categories: tasks in progress, missed tasks, and planned tasks.

Tasks in Progress:

In three of the 15 cases, we observed one instance of team leaders prematurely marking tasks as completed on the checklist after discussing task performance with the team. During these discussions, team leaders asked a team member to perform a task and then checked it off before it was actually completed. For example, the leader in one case instructed the team to establish the intravenous (IV) access and immediately checked off the corresponding item on the checklist (Table 1(a)). Shortly after, two nurses attempted to place the IV six times over the course of the next nine minutes but were unsuccessful. The team then discussed the next steps and decided they would try again after moving the patient to another room post resuscitation. The team leader participated in this discussion and acknowledged that IV access was not established after the failed attempts. However, they did not record this nuance in task execution in the checklist, which incorrectly showed that IV access had been established during the resuscitation. In each of these three cases, we did not observe any notes or other indications on the checklist reflecting that the task was not eventually completed or was completed after the item was marked on the checklist. The checklist therefore inaccurately demonstrated that the task was completed at the time the leader checked it off.

Table 1:

Examples of mismatches between the digital checklist record & state of the resuscitation

(a) Tasks in Progress: 21:00:38 “Let’s work on getting IV access when there’s someone available to do that” [Leader] 21:00:40 Marked Complete on Checklist Activity Completed: No

(b) Missed Tasks: 12:14:53 Activity Completed 12:18:43 Marked Complete on Checklist 12:18:48 “Did we look in his mouth, I didn’t hear” [Leader]

(c) Planned Tasks: “Alright for right now, let’s just do [step] c, lateral c-spine films and then we’ll see how those look before we go any further. I’m gonna hold off on any labs.” [Leader] Recorded on Checklist: No

“Given the mechanism of injury, I think we should x-ray his c-spine as well as his chest and a CT of his head” [Leader] Recorded on Checklist: No

Missed Tasks:

In five of the 15 cases team leaders missed recording one completed task on the checklist and in one case, the team leader missed recording two tasks. In six of those seven cases, a team member had reported the results of the completed task. However, the team member was also sharing the status of other tasks in the same report or another team member was concurrently discussing a different task. We observed team leaders later inquiring about these missed tasks and then marking them as complete on the checklist upon hearing the confirmations. In one case, for example, the junior resident performing the exam asked the patient to open their mouth and then reported “Dental tissue intact.” Later in the resuscitation, the team leader asked the resident if they had checked the mouth, saying that they didn’t hear if they did, and then checked off the task (Table 1(b)).

Planned Tasks:

In six of the 15 cases, we observed team leaders planning the subsequent tasks and tests with team members. These conversations usually occurred towards the end of the resuscitation, as the leader summarized the case and outlined the next steps. The checklist contains a checkbox for “Summarize findings and brief team on plan of care” at the end of the secondary survey. However, this box does not provide any room for notes on the plan of care. Additionally, the Prepare For Travel section does not have any free space for notes at the bottom of the section. Most often, team leaders discussed with the team the laboratory or imaging tests (e.g., x-rays) that should be performed, but they rarely noted this information in any section of the checklist (Table 1(c)).

Additionally, we observed that two cases involved more than one senior leader. In these cases, one person would interact with the digital checklist while the other(s) interacted with team members and communicated about the tasks. The leader using the digital checklist rarely interacted with the other leaders or team members, and mostly focused on capturing data on the checklist. In the first case with a highly acute patient, an attending surgeon and fellow directed the team, while the resident administered the checklist. The second case was less severe, but still had one physician interacting with team and the other administering the checklist.

Finally, we found that team leaders checked off the patient vital signs at the end of the resuscitation (rather than at the end of the primary survey) in three of the 15 cases. In addition, vitals were checked off as completed without any actual values recorded in four cases. Upon closely examining the team communication around the blood pressure vital sign in particular, we found that a nurse verbally reported the blood pressure in 12 of the 15 cases. The analysis of checklist logs then showed that the leader entered a blood pressure value in nine of those 12 cases, usually later in the resuscitation, and the entered value did not match the verbalized value in seven of the nine cases.

Discussion

By nature of their design, checklists are static, allowing users to record only whether a task is completed or not. In most settings, checklists are used by a single user to record the tasks they themselves performed. In complex team settings like trauma resuscitation, however, the checklist is administered by a person who is overseeing the team and who may not perform any tasks (Figure 3). Our study highlighted the tensions that can arise when static tools like a checklist are used to aid dynamic, team-based processes.

Figure 3:

Diagram of a trauma resuscitation event featuring the team leader using the digital checklist and team members performing the evaluation on the patient.

The “Tasks in Progress” category illustrates the challenges that leaders face in coordinating multi-step tasks. The current design of the checklist does not allow users to indicate that a task is in progress or to record who is performing which task. Once a team leader instructs a member to perform a task, they might consider this task complete and check it off on the checklist. However, this premature check-off can result in a false checklist record and a mismatch between the record and the actual state of the process if a team member cannot complete the task during the resuscitation. This mismatch can be especially pronounced in multi-step tasks, such as establishing an IV. If the purpose of a checklist is to aid protocol adherence by differentiating between completed and non-completed tasks, a false indication that a task has been completed is problematic.

The “Planned Tasks” category illustrates the tool’s lack of support for planning and coordinating future work. Prior work has shown that leaders often recorded care plan information in the margins of the paper checklist [16]. In our previous study that compared the paper and digital checklists, we observed that the digital checklists contained fewer notes about the care plan than paper checklists, highlighting the design need to include a space for care plan notes as well [13]. This current study reinforces that need, as we found that team leaders discussed the care plan with the teams in ten of the 15 cases, but did not record this information on their checklists.

The “Missed Tasks” category highlights the challenges that arise when one person is responsible for maintaining a record of the concurrent work performed by the rest of the team. In this category, team members have reported out loud (or verbalized) task status or exam findings but the team leader missed the verbalization and did not capture the information on the checklist. Because the checklist is only viewed by the team leader, others on the team have no insight into the information recorded on the checklist, so they cannot inform the leader of any missed or inaccurate information. The missed information is only corrected if the leader inquires about it at a later time. This finding also provides insight into delays between the activity completion and the time of task check-off on the checklist. We have observed how team leaders were trying to be compliant with the checklist and ensure its accuracy, but had difficulties in tracking and recording the concurrent, rapidly evolving work by other team members.

When a case had multiple leaders, one leader appeared to take on the role of a scribe, focusing on recording information on the checklist rather than interacting with the team. This finding implies that administering the checklist felt like a measurable amount of work, forcing the leaders to divide their work and have one team leader focus on the checklist while the other leader(s) directed the resuscitation.

Finally, we observed team leaders filling in the vital sign values at the end of the resuscitation or only checking them off without entering any values. Vital sign values vary throughout the resuscitation and need to be continuously reevaluated, unlike other tasks, which are considered done after being performed once (e.g., examining the mouth). Static values for vitals on the checklist do not meet the needs of the team, so the leaders recorded them when it was most convenient (at the end) or did not enter any values at all.

Design Implications

We next discuss design recommendations for improving checklists that guide dynamic team activities.

Support In-Progress Tasks:

Our findings showed that team leaders had difficulties capturing the status of in-progress tasks on the checklist. Currently, the checklist only indicates if a task is performed or not (through a check-off mark or the lack thereof). An indicator for tasks that are in progress could be added to alleviate this challenge, along with recording which team member is working on the task. A decision support feature could also be incorporated to alert the leader if a task has been in progress for longer than expected and to provide guidance on how to proceed.

Automatically Capture Information:

It is challenging for team leaders to capture information on all concurrent work being performed by the team, especially dynamic information that keeps changing. An advantage of a digital checklist, when compared to a paper form is that it can interact with other technologies in the environment. Values from the vital sign monitors could be displayed on the checklist and updated in real time. Video and audio recognition systems could indicate tasks that are missing or have been done. Streamlining the automatic capture of information for the checklist would lessen the leader’s burden, making them rely less on other team members for capturing the information and manually recording it. This approach would then allow the leader to spend more time interacting with the team.

These design implications, however, raise questions about the purpose and role of digital checklists in dynamic and safety-critical settings like trauma resuscitation. Adding more functionality and features may lead to new types of checklists that are moving away from a traditional notion of a checklist–a simple list of tasks that user check off when completed. Our findings may signal that a different type of a system is required to support team leaders in guiding concurrent performance of multi-step tasks and coordinating a team. Our future work will address these questions by implementing our design recommendations, either in simulations or in the wild, to evaluate if they reduce the number of missed tasks and provide better support for in-progress tasks without increasing the leader’s cognitive load. If the tensions between the digital checklist record and the state of the resuscitation persist, we may need to explore different systems which can better support this type of work.

Conclusion

By studying the team communication in a trauma resuscitation setting along with the leader’s interactions with the digital checklist, we were able to identify the tensions that can arise when static artifacts like checklists are used for team-based, dynamic processes. Shortly after assigning tasks, team leaders marked the tasks as complete on the checklist, even when team members had not yet completed them. The leaders also missed verbal communications from team members about completed tasks, and only checked those tasks off when they inquired about them at later points. Finally, the leaders discussed the future tasks that should be performed after the resuscitation, such as lab or imaging studies, but did not record this information on the checklist. We concluded by discussing design recommendation in light of these findings and their implications for designing the checklists of the future.

CCS Concepts.

• Human-centered computing → Computer supported cooperative work;