Checklist as a Memory Externalization Tool during a Critical Care Process

Abstract

We analyzed user interactions with a paper-based checklist in a regional trauma center to inform the design of digital cognitive aids for safety-critical medical teamwork. An initial review of paper checklists from actual trauma resuscitations revealed that trauma team leaders frequently wrote notes on the checklist. To understand this notetaking practice, we performed content analysis of 163 checklists collected over the period of four months. We found nine major categories of information that leaders recorded during resuscitations, including patient values, physical assessment findings, and pre-hospital information. An analysis of types and amount of notes written by leaders of different experience levels showed that more experienced leaders recorded more patient values and physical findings, while less experienced leaders recorded more notes about their activities and task completion status. These findings suggested that a checklist designed for a high-risk, fast-paced medical event has evolved into a dual function tool, serving both as a compliance and memory aid. Based on these findings, we derived requirements for designing digital cognitive aids to support safety-critical medical teamwork.

Introduction

A growing body of work has shown the value of checklists in general hospital care, suggesting that their use can be extended to other clinical domains prone to human error¹. Checklists are now commonly used to support a range of complex medical activities, including verifying the steps of safe surgery, performing infection control procedures, and defining daily care tasks in a hospital setting^2–4. Trauma resuscitation is a high-risk, time-critical and team-driven process of treating severely injured patients, posing a four-fold higher risk of death from errors than general care⁵. Due to the fast-paced and complex nature of the process, adoption of decision-support systems and cognitive aids in this environment, including the checklist, has been slow. To attain and retain resuscitation skills, providers rely on individual and team training, but training alone has been found insufficient for ensuring patient safety and long-term compliance with protocols⁶. Recently, however, several U.S. trauma centers have started implementing checklists in actual resuscitations, showing their positive impact on team performance and protocol compliance^7–9.

Although initial reports showed the benefits of checklists in medicine, some follow-up studies have questioned their effectiveness¹⁰. Medical work is often characterized as messy, ad hoc, and unpredictable¹¹, yet most medical checklists are static and linear. Checklists typically cover a mix of important and frequent activities, but do not take into account repeated tasks, changing patient and environmental conditions, and less frequent but critical tasks. These checklist limitations have led to both low adoption and poor compliance rates^12,13. To achieve optimal performance, particularly in time-pressured and safety-critical settings such as trauma resuscitation, checklists may need to be more flexible and dynamic to allow for adaptability to different patient and user scenarios¹⁴.

To inform the design of a dynamic checklist for fast-paced medical settings, we studied the use of a paper-based trauma resuscitation checklist in an urban, pediatric teaching hospital, with a level 1 trauma center. A checklist is commonly seen as a tool for ensuring process compliance by performing and checking off a set of tasks. We found that trauma team leaders, in addition to checking the items, frequently wrote notes next to those items. The notes contained valuable information for task performance and decision making during trauma resuscitations. This finding suggested that the resuscitation checklist has evolved into a dual-purpose tool to support both the process compliance and leaders’ cognitive functions. In this paper, we performed a mixed-method study to analyze the nature of note taking and other observed checklist use practices, and to derive design requirements for augmenting the checklists to better support complex and highly dynamic teamwork.

Trauma Resuscitation Checklist

The resuscitation checklist was designed to help trauma teams prepare for patient arrival and to guide their performance throughout the patient evaluation. Its goal was to ensure protocol compliance and reduce delays in treatments. The checklist contains 53 items grouped into four sections: pre-arrival plan, primary and secondary surveys, and departure plan (Figure 1). The pre-arrival section lists preparatory steps for patient arrival. The primary and secondary survey sections contain the protocol steps, including the ABCDEs, head-to-toe evaluation, and vital signs checking. The departure plan items facilitate discussion about diagnostic tests and patient disposition. Checklist administration was assigned to the physician leader, a role least likely to be hands-on during patient evaluation.

Figure 1.

Trauma resuscitation checklist with physician’s notes.

Methods

Research Site & Data Collection

Our research site was an urban, pediatric teaching hospital and a regional level 1 trauma center. Each year, the center treats about 600 high-acuity and high-risk injured children in the emergency department. Patients are treated in two adjoining rooms dedicated to the initial resuscitation. Each room is equipped with vital signs monitors, large wall displays for augmenting vital signs, and a wall clock and stopclock to help teams manage their time. Additional low-tech decision-support artifacts include a dry erase board for displaying the patient’s weight, trauma flowsheet for documenting the process, wall-mounted charts with patient parameters, and recently—the resuscitation checklist.

Data collection took place over a four-month period in 2012 (05/01/2012-08/31/2012), three months after the checklist was implemented. We obtained checklists from 163 resuscitations that occurred during this time. Checklists were administered by physician leaders, a role responsible for leading an interdisciplinary team of up to 15 specialists, assigning tasks, making decisions, and overseeing the process. This role is usually assigned to an attending surgeon, a surgical fellow, a senior surgical resident or an emergency medicine physician. The leaders rarely participated in hands-on evaluation, and mostly stood at the foot of the bed to maintain overview of the patient, team and resuscitation room. The study was approved by the hospital’s Institutional Review Board (IRB).

Data Analysis

All checklists were first transcribed into a spreadsheet. For each checklist, we transcribed its study ID, the physician leader’s experience level (surgical resident [PGY-4], surgical fellow, surgical attending, emergency medicine physician), weight information recorded in the pre-arrival plan section, list of notes written down by the leader, location on the checklist for each note that was written down, and items on the checklist that were left unchecked. This transcript was then reviewed to resolve any ambiguous and unreadable notes. Two experts in trauma resuscitation also reviewed the transcribed notes for clarity and accuracy.

We first performed quantitative analyses on the checklists to determine the frequency of checked and unchecked items, the number of checklists with and without notes, the amount of notes for each checklist section and item, and the amount of structured and free-form text notes. We then focused on the 125 checklists (77%) that contained handwritten notes. We used an open coding technique to identify and categorize types of information recorded by physician leaders. During the first pass, we identified 50 codes characterizing the notes. These codes were then discussed in a group session to determine which codes to keep, remove, or merge. This process resulted in nine high-level notes categories and 27 subcategories (Table 1). The transcript was updated with these final codes in the second pass to reflect these changes. We also determined the number of notes for each high-level category.

Table 1.

Notes categories and subcategories, percent of all notes, and examples.

Notes Categories and Subcategories			Examples
Patient Values (35%)	Vitals	Heart Rate (HR)	87 HR; 118; 90s; 19?
		Respiratory Rate (RR)	20s; 21; 30; 41
		Oxygen Saturation (SO2)	100% sat; 97% RA; 98
		Blood Pressure (BP)	117/74; 90/palpable; 100 over 65
	Pulse		2+ DP; rad 2+; 2+ bilat.
	Glasgow Coma Score (GCS)		4, 3, 6 13; GCS: 3; 15; 4+4+6=14
	Pupils		5mm sluggish but reactive; 3→2; 4 brisk
	Temperature		36.6 capillary; 37; 37.2 rectal
Physical Findings & Symptoms (22%)			WNL; soft; stable; TMs clear; Ø deformities; TTP; pain; cleared
	Location		Tender midline; RUQ tenderness; pain L posterior
Pre-Hospital Information (12%)	Transport	Est. Arrival Time (ETA)	10 min
	Demographics	Age	2 yo; 7 mo/o; 3 wks; Age 12
		Name	(names spelled out)
		Gender	Male; female
		Weight	11kg; 120 lbs
	Medical History		PMH 3x concussions; All: cats; H/O Asthma on steroids
	Previous Treatments (e.g., en route, outside hospital)		100mg fent; CPR; 2 x saline; CT head + neck; PE tubes; intubated
	Mechanism of Injury		Fall 5ft; MVC vs Ped; motorbike vs. SUV
	Physical Findings		Ø LOC; + LOC; lethargic; alert & awake
	Injuries/Complaints		Neck pain; R skull fx; leg bruise
Care Plan (11%)	Labs		Basic trauma lab; tox screen
	Consults		Ortho; N Surg consult
	Imaging Tests		Chest, c-spine; Xray; CXR; CT Head
	Disposition		PICU; Neuro
	Medications/Fluids		Bolus fluids; LR 28
	Other		Consider NAT; obs pl ambulate
Injury Type & Location (9%)			L LQ abrasion; multiple lacs; facial bruising
Task Completion Status (4%)			Deferred; Not done; In progress; N/A
Treatments & Procedures (3%)	IV/IO	Size/Location	R A/C – 20; 22g L hand; 16 ga
	Meds & Fluids	Type/Rate/Dosage/Time	Start LR @ 65; 600 ml LR; NS @ 50; etom in; atropine → 15.30
	Intubation	Tube Size & Location	4 cuffed, 13 @ lips
	Chest Decompression		L needle decompression
Other (3%)	Notes to trauma office, unreadable		Bair hugger not available
Laboratory Results (1%)			Dstick = 111; HH values: 50 over 30, 15 over 40, 16 over 48; AST/ALT – 117/91

To gain additional insight into note taking, we compared the number and types of notes based on the leader’s experience level by grouping the checklists into those administered by surgical residents and fellows (less experienced) and those administered by attending surgeons and emergency medicine physicians (more experienced).

Findings

We report our findings in three parts. First, we present the results from our general observations of checklist use. We then describe the results of content analysis by categorizing and describing information types that physician leaders recorded on the checklists. We conclude by describing note-taking differences based on leaders’ experience levels.

General Observations about Checklist Use

Our analysis of the frequency of checked and unchecked boxes revealed two groups of checklist items (Figure 2(a)): (1) almost always checked, and (2) rarely checked. The first group consisted of primary and secondary survey items such as “Confirm O₂ placement” and “State GCS [Glasgow Coma Score],” which were checked in 80% to 95% of the checklists. Optional items such as “For attending activations” in the pre-arrival plan section and “Prepare patient for travel” in the departure plan section were rarely checked (7%), as was the primary survey item “Give fluid or blood” (14%). These findings suggest that a multi-tiered organization and adaptation of the checklist to specific contexts may be useful. Although prior studies of checklist compliance showed that users sometimes check a box without performing the corresponding task or perform a task without checking the corresponding box¹⁵, there is no evidence that physician leaders in our study chose to exclude any specific checklist items. It is therefore appropriate to assume that failures in completing checklist items are uniformly distributed across the checklist, and that the items that were rarely or never checked were also rarely or never performed and vice versa (Figure 2(a)).

Figure 2.

(a) The frequency an item was checked off. (b) The number of times a note was taken for frequently checked items. Items in (a) and (b) are ordered by frequency, but the same item may be in different positions in (a) and (b). (c) Scatterplot of the relationship between the frequencies of item checking and their corresponding notes.

We observed several styles of checklist use. Notes were sometimes written next to an item, but the item was left unchecked. In other cases, notes associated with an item appeared in the same checklist section as the item, but not next to it. Physician leaders appeared to prefer free margin space, filling both top and bottom margins with notes on most checklists. We also found a few checklists with all 53 items checked off but without a single note. Items with the largest number of notes were “State GCS” under step D(isability) and vital signs evaluation tasks under Vitals in the primary survey section. Items with the least number of notes included the “If intubating” box in the primary survey section, equipment preparation steps in the pre-arrival section, and departure plan items. A closer look at the relationship between the frequencies of item checking and note taking revealed that for items that were rarely checked, notes were almost never taken (left corner of Figure 2(c)). For items that were frequently checked, the number of notes varied uniformly between 0 and 83 (Figure 2(b) as well as the right side of Figure 2(c)).

We further observed that leaders’ handwritten notes belonged to three groups based on the note structure: (1) freeform text, (2) numeric, and (3) numeric with attributes. Free-form text notes were most common, comprising 63% of all notes taken. These notes appeared in the margins and in the secondary survey section, especially for head and cervical spine exams. The length of free-form notes varied between 1 to 46 characters per note. Most notes (78%), however, were short with fewer than 16 characters. Numeric notes comprised 30% of all notes, mostly appearing next to the vital signs, Glasgow Coma Score (GCS), and temperature assessment tasks. Numeric notes with attributes (e.g., “90/palpable” for blood pressure, “2 mm reactive” for pupil size) comprised 7% of all notes, and were also found next to vital signs, GCS score and pupil size. Here, we observed that the attributes were limited to a small set of choices. For example, leaders wrote one of the following for pupil size: “sluggish,” “brisk,” “reactive,” “blown,” “unreactive,” and “equal”. GCS score notes almost always showed its value broken down into three components: eyes, verbal and motor response. Attributes for the GCS score were therefore implied by the initial of the score component name (e.g., “E3V3M5”) or by the order in which they were recorded (e.g., “4, 3, 6, 13”).

Checklist Notes: Information Types and High-Level Categories

Our content analysis of physician leaders’ notes written on the checklists showed that leaders recorded a great deal of information during resuscitations. These checklists (including the leaders’ notes), however, were not part of the official medical record and were usually discarded after the resuscitation, unless kept for research purposes. We identified 27 information types and grouped them into nine high-level categories (Table 1). Below we describe each high-level category in greater detail.

Patient Values: Patient values were the most commonly found notes, comprising 35% of all notes written down on the checklists. These values included Glasgow Coma Score (GCS), temperature, pulses, pupil size, and vital signs such as heart rate, respiratory rate, oxygen saturation and blood pressure, and were usually recorded next to their corresponding checklist items (Figure 3). For example, GCS and pupil size values were found in section D of the primary survey, temperature was written in section E, and vitals were written down in the vitals section.

Figure 3.

Instances of recorded vital signs trends: Oxygen saturation (top); Blood pressure (bottom).

The amount of detail for each value also varied, especially for vital signs. Although most leaders wrote both systolic and diastolic numbers for blood pressure, we also found several instances with the systolic number only (e.g., 100), either alone or with an attribute (e.g., 145-sbp, 90/palpable). Oxygen saturation was sometimes accompanied by “RA” (e.g., 97% RA), meaning that the patient was breathing room air rather than 100% oxygen. Another patient value with varying levels of detail recorded was Glasgow Coma Score (GCS). Most often, physician leaders broke down the score into parameters and then wrote the final score summing up the individual scores, e.g., “4, 4, 6 = 14.” Other, less frequently observed variants of this value included noting only the final score (e.g., “15”) or describing each parameter (e.g., “eye injury, talks, obeys commands”).

In some instances, physician leaders wrote more than one note for the same value. For example, they recorded changes in oxygen saturation levels, as well as more than one blood pressure value (Figure 3). Although no time information was written next to these values, these notes suggested that some leaders used their checklists to record trends in patient data, especially in vital signs. Current vital signs monitors can be setup to show trends but teams do not have time to change the monitor settings and mostly use the default, continuous monitoring of vital signs. Similar findings were observed for the GCS values, with initial GCS assessments crossed out and new values recorded next to them. Although trends in the data were found in only 3% of the checklists, these findings suggest that trends are important during time-critical patient management.

Physical Findings and Symptoms: Physical findings and symptoms were the second most frequently observed category, comprising 22% of all notes. These notes were mostly related to findings from the secondary survey and were usually found in the secondary survey section of the checklist (Figure 4). Status for each secondary survey item was written next to its corresponding checkbox. The notes included both normal (e.g., stable, within normal limits [WNL], non distended [ND]) and abnormal (e.g., tenderness to palpation [TTP] diffusely, blood in nares, pain) findings. We also noted a few instances of patient complaints written as part of the findings (e.g., “Pt complains of chest tenderness”). Sometimes, however, leaders would jot down only general status notes using this space, without referring to any of the particular survey components (e.g., “multiple lac to scalp 10.5cm” accompanied by a sketch of the head with lacerations marked). Finally, leaders were observed specifying location of the findings and symptoms. For example, if an abnormality were found on the patient’s chest, the notes would also specify parts of the chest, as in “RUQ [right upper quadrant] tenderness.”

Figure 4.

Notes about physical findings.

Pre-Hospital Information: Information about incoming patients, such as mechanism of injury, received treatments and patient medical history, serves a critical role in helping medical teams prepare for patient care¹⁶. It was not surprising then to see 12% of total notes dedicated to pre-hospital care, mechanism of injury and patient medical history. These notes were mostly written on the top or left margins, or between the patient label and primary survey section of the checklist (Figure 1). For injured patients coming directly to the trauma center, pre-hospital notes included estimated time of arrival, patient demographics (e.g., age, gender), mechanism of injury, physical findings, and injuries and complaints. For patients that were transferred from another hospital, leaders also added notes about their medical history (e.g., allergies, any chronic illnesses and current medications), as well as the results from previous tests, such as imaging and laboratory tests (Figure 5).

Figure 5.

Pre-hospital information notes for a transfer patient.

Care Plan: Care plan notes included notes on any decisions that leaders made about the subsequent steps in patient care. These notes accounted for 11% of total notes and were typically found in the margins, e.g., the lower right corner of the checklist, below the departure plan section (Figure 6, left). Most of the time, these notes specified laboratory tests (i.e., basic vs. comprehensive labs), imaging tests (e.g., head CT, shoulder x-ray), needed consults (e.g., neurosurgery, orthopedics), and patient disposition. The care plan notes were often presented in the form of an additional list of checklist items, with checkboxes drawn next them (Figure 6, right). This observation suggests that physician leaders who presented their care plan in this form felt that the corresponding items were missing from the checklist and should therefore be considered for a future iteration of the checklist.

Figure 6.

Care plan notes examples.

Injury Types and Location: In addition to physical findings and symptoms, we observed notes about injuries identified during secondary survey. Similar to physical findings, physician leaders noted the location of injuries, e.g., left forehead bruise, R[ight] eye ecchymosis, or abrasion over right eyebrow. Although not as frequent as physical findings, injury notes accounted for about 9% of all notes written on the checklists. As part of these notes, physician leaders recorded injury types such as abrasions, lacerations, or fractures. Most of the time, notes about injury type and location were recorded along with physical findings in the secondary survey section.

Task Completion Status: We observed that leaders’ notes sometimes provided a rationale for either checking an item or leaving it unchecked. For example, for “Confirm O₂ placement” in the primary survey section, leaders often wrote “N/A sats 100%” or “100% defer,” implying that oxygen mask placement was unnecessary because oxygen saturation was already at 100%. Or, for preparatory steps in the pre-arrival plan section, one leader crossed the entire section noting that they did not have time to complete the pre-arrival steps because the patient already arrived. Task completion status notes were found throughout the checklist, next to different items, and comprised 4% of all notes.

Treatments and Procedures: Although rare (only 3% of all notes), notes about treatments and procedures represent a significant finding because they suggest that some leaders needed to remember the type and amount of administered medications or fluids (Figure 7, left). Most commonly seen notes in this category were about location and size of intraosseous (IO) or intravenous (IV) placement. These notes typically appeared in the bottom margin of the checklist. Notes about procedures were even rarer because only few patients in our sample required procedures such as intubation or chest tube placement. When found, these notes contained the tube size and location.

Figure 7.

Treatment notes (left) and laboratory results notes (right).

Laboratory Results: The least frequently found notes were about laboratory results (3% of all notes). These notes mostly appeared on the checklists for transfer patients because test results came through documents accompanying the patient and were reported upon arrival (Figure 7, right). Some results from basic tests like blood gases and glucose levels were also written for patients that arrived from the injury scene, as these tests could be performed in the trauma bay.

Note Taking Differences between Physician Leaders

To gain more insight into leaders’ note taking, we also examined the differences in types and number of notes based on leaders’ experience levels.

First, we examined the percentage of leaders within each experience level that took notes during resuscitations (Figure 8). More than 70% of team leaders of any experience level took notes on their checklist sheets during resuscitations. We found that 75% of all attending surgeons, 77% of all surgical fellows, 71% of all surgical senior residents, and 83% of all emergency medicine physicians took notes on their checklists. Second, we looked at the average number of notes per checklist by leaders of all experience levels. Attending surgeons were the most prolific note takers, with an average of 15 notes per checklist, followed by emergency medicine physicians (on average, 12 notes per checklist), followed by senior surgical residents (on average, 10 notes per checklist), followed by surgical fellows (on average, 9 notes per checklist). An interesting observation is that more experienced team leaders (attending surgeons and emergency medicine physicians) took more notes than less experienced team leaders (surgical fellows and residents). Finally, we wanted to know if there were any differences in the types of information recorded by physician leaders of different experience levels (Figure 9). We found that more experienced leaders, on average, recorded more patient values (e.g., vital signs, temperature and Glasgow Coma Scores) and findings from physical assessments (i.e., secondary survey findings), two types of information necessary for decision making. They also recorded more notes about the patient’s subsequent care plan. On the other hand, less experienced team leaders recorded more notes about their activities, such as treatments and procedures, and task completion status. These differences in behaviors may be related to the patient injury severity level. Attending surgeons act as leaders only during highly acute cases, while fellows and residents lead in less critical cases.

Figure 8.

Experience-based fractions of leaders taking notes on the checklist.

Figure 9.

Experience-based differences in the types of notes taken on the checklist.

Discussion

Our findings showed that trauma team leaders of all experience levels write down a great deal of information on their checklist sheets. Initially, this finding was not surprising because prior work has shown that paper records afford many qualities that are critical in medical work, including mobility, flexibility, and visibility of work^17,18. Notes on paper are also used as transitional records for more efficient communication and coordination¹⁹. Upon closer examination, however, we found that leaders’ notes contained valuable information about patient and task performance during trauma resuscitations. Although the checklist was primarily designed to assist with protocol compliance, its use was appropriated for additional purposes during resuscitations, such as memory externalization and decision making. Prior research on distributed cognition systems has shown that cognitive aids (e.g., maps, charts, diagrams, paper scraps) have an important role during complex, knowledge-based processes because workers need to lessen their cognitive load by externalizing information²⁰. These findings suggest that complex medical processes such as trauma resuscitation require externalizing information from memory, much like navigating a ship or piloting an airplane^20,21. Although these other distributed cognition systems have computerized support for externalizing critical process information, this kind of support is still paper-based in the resuscitation and other critical care settings. Paper, however, is static (i.e., checklist items remain the same despite changing patient conditions) and limited in size, which constrains the number of checklist items; otherwise, it would be inconvenient to manage a large paper sheet. Also, paper checklists cannot be connected to electronic sources of information, requiring manual capture of information. Our study offers several implications for computerizing checklists and other cognitive aids to better support complex medical processes.

Provide Information about Patient Data Values over Time

The most common information type scribbled on the checklist was patient values. Physician leaders needed patient vital signs, GCS values and temperature externalized to analyze the data and detect any abnormalities. It appeared that leaders used their checklists for memory externalization, suggesting the need for additional cognitive aids to support these tasks. Although vital signs monitors can provide information about patient values over time, clinicians rarely set them to those modes because they preferred the continuous monitoring view. Even so, supplemental displays in the room could facilitate access to patient data over time by capturing, analyzing and visualizing information such as trends (upwards, downwards or flat), sudden changes of levels, or spikes in the data.

Make the Checklist Adaptive to Different Patient Contexts and Leaders’ Experience Level

Our findings about the frequency of checked and unchecked items suggested the need for checklist adaption to different patient contexts. Current resuscitation checklist contains only one complex procedure (endotracheal intubation), and ignores other major interventions, such as chest tube insertion, chest decompression, or pelvic stabilization. Because of the limited amount of free space on the checklist and the need to keep the checklist comprehensive but succinct for easy use, the number of checklist items will always be constrained, regardless of the medium. The implication here is that only the items that apply to the current patient scenario should be displayed, which can be achieved through real-time checklist adaptation. As we have observed, some physician leaders performed “manual adaptation” of the checklist by listing action items under the plan of care as newly added checkboxes. We also observed a few instances of this approach for items in the treatments and procedures category, like medications and fluids (Table 1). These observations imply that leaders needed additional checklist items to indicate the kinds of laboratory or imaging tests they wanted to perform, or to maintain a record of medications and fluids they administered. An electronic checklist could afford such an extension by adaptive selection of checklist items or by using multiple tabs. Additionally, items that were rarely checked could be called on demand using the collapse and expand feature, or shown automatically based on patient-context recognition. Finally, we found that leaders of different experience levels showed different preferences for types of information, suggesting that some adaption based on user contexts may also be needed.

Allow Selection-Based rather than Description-Based Note Taking

We found that physician leaders marked not only task completions, but also task outcomes. These observations suggest that interactions with checklists could be augmented by functionalities that allow easy input of descriptive information. Writing descriptive notes requires more thought and effort than selecting an item. Documenting during resuscitations, however, requires efficiency due to the rapid information flow, which makes selection-based data entry more appropriate²². This type of interaction could be supported by creating and structuring selection lists based on the notes we identified through the analyses. Given that most free-form notes were relatively short (≤15 characters), we expect that the selections lists will have a limited number of items (the shorter the note, the more limited the number of choices of what it can say).

We also observed that leaders sometimes explained the rationale for skipping some checklist tasks. Having a selection-based menu that was structured using their notes could make this practice faster and easier as well. With advancements in sensors and automatic data capture, some of the conditions for skipping a checklist task may be possible to automatically detect and alert the leader that a task is not applicable. For example, as described above, leaders often justified their skipping of the oxygen mask placement by noting that the patient was already at 100% O₂ saturation. Integrating the output of the O₂ saturation sensor into the system could automatically infer that the oxygen mask is not necessary.

Handwritten checklist notes depend on personal styles and preferences. For example, pupil size and response were variously described as “Pupils Equal and Equally Reactive to Light,” “Pupils Equal, Round, Reactive to Light,” and “Pupils Equal, Round & Reactive to Light & Accommodation,” all of which conveyed the same status. By structuring leaders’ observations and providing a selection list, we can ensure standardization across different leaders and checklist use cases.

Allow Free-Form Text Input for Note Taking

The mode of note taking should in general match the note structure. Numeric keypads should be considered for entering numerical notes, such as vitals, GCS, and temperature. Some types of notes cannot be easily reduced to a list of options. As we found, notes about patient medical history and mechanism of injury, as well as questions for the trauma services staff served an important role, even though they were less frequent than notes about task completion and outcome. Future checklists should allow the leaders to write these notes by providing a designated space and free-form text input. Rapid input of free-form text could be further facilitated by selection lists and autocompletion of unique text, which is again possible based on our analysis of note content. Using previously seen notes as completions is feasible because free-form notes are relatively short and it is very likely that the user will find a previously seen match for his or her intended note.

Conclusion

In this paper, we analyzed user interactions with a paper-based checklist in a regional trauma center based on content analysis of physician leaders’ notes scribbled on 163 checklists collected over a four-month period. Our findings showed that leaders recorded nine major categories of information during resuscitations, including patient values, physical assessment findings, pre-hospital information and care plan steps. We also found that more experienced leaders, on average, recorded more patient values and physical findings, while less experienced leaders recorded more notes about their activities and task completion status. These findings suggested that a checklist designed for a high-risk, fast-paced medical event evolved into a dual function tool, serving as both the compliance and memory aid. Based on these findings, we discussed several implications for designing future dynamic cognitive aids for high-risk medical events to better support fast response medical teams.

This paper represents our first look into the checklist note-taking phenomenon. Our future work will continue collecting and analyzing paper checklists to further examine the differences in physician leaders’ use practices based on experience level. We also plan to examine the extent to which factors such as patient scenarios and injury acuity affect types and amounts of notes. Physician leaders will be surveyed to provide their perceptions of the checklist effectiveness, as well as their reasons for note taking. Finally, we will continue refining design recommendations for dynamic checklists and other cognitive aids for complex medical processes.