Emergency Manuals Improved Novice Physician Performance During Simulated ICU Emergencies

Abstract

Background

Emergency manuals, which are safety essentials in non-medical high-reliability organizations (e.g., aviation), have recently gained acceptance in critical medical environments. Of the existing emergency manuals in anesthesiology, most are geared towards intraoperative settings. Additionally, most evidence supporting their efficacy focuses on the study of physicians with at least some meaningful experience as a physician. Our aim was to evaluate whether an emergency manual would improve the performance of novice physicians (post-graduate year [PGY] 1 or first year resident) in managing a critical event in the intensive care unit (ICU).

Methods

PGY1 interns (n=41) were assessed on the management of a simulated critical event (unstable bradycardia) in the ICU. Participants underwent a group allocation process to either a control group (n=18) or an intervention group (emergency manual provided, n=23). The number of successfully executed treatment and diagnostic interventions completed was evaluated over a ten minute (600 seconds) simulation for each participant.

Results

The participants using the emergency manual averaged 9.9/12 (83%) interventions, compared to an average of 7.1/12 (59%) interventions (p < 0.01) in the control group.

Conclusions

The use of an emergency manual was associated with a significant improvement in critical event management by individual novice physicians in a simulated ICU patient (23% average increase).

Background

Every year a changeover of trainees occurs in teaching hospitals where novice physicians (PGY1), or interns, take on new responsibilities while simultaneously facing the stressors encountered at the beginning of residency. Consequently, evidence suggests that mortality increases and efficiency decreases in hospitals in the first month after year-end changeovers¹. Reasons may include the loss of experienced staff, relative inexperience of each new group moving up in rank (e.g. new interns, new senior residents, and new supervisor physicians), and a general lack of institutional and tacit knowledge (e.g. new teams, figuring out “how we do things here”). In the context of these safety vulnerabilities, novice physicians may find themselves with the responsibility of managing acute life-threatening events in various clinical settings^2,3. Caring for critically ill patients may occur in stressful and high stakes situations where backup (e.g. supervising physicians) is not immediately available. Despite an intern's limited clinical experience in caring for the critically ill, patients and their families still expect physicians of all levels, and the hospital system at large, to provide safe, high quality care, while minimizing errors.

Not surprisingly, it has been established that adherence to Advanced Cardiovascular Life Support (ACLS) guidelines during an in-hospital cardiac arrest is associated with improved outcomes^4–6. Yet, studies have shown that even experienced providers have significant decay of ACLS knowledge after their one-day recertification courses^7–9. To compound matters, experts and beginners alike, in both medical and nonmedical domains, suffer from cognitive performance degradation when stress and workload are high¹⁰. Therefore, additional measures are needed to support novice physician decision-making, especially under conditions of high-stakes outcomes.

Checklists have been used to enhance safety in some medical^11–16 and high-reliability fields^17–20 (e.g., aviation and nuclear power plants) for many years. Recently, emergency manuals have become increasingly advocated for use in medical environments where critical events occur, especially intra-operative settings^21–23. Individual checklists have evolved into comprehensive emergency manuals, such as the “Emergency Manual: Cognitive Aids for Perioperative Critical Events” by the Stanford Anesthesia Cognitive Aid Group²⁴ and the “Operating Rooms Crisis Checklist” by Ariadne Labs²⁵ (A Joint Center at Brigham and Women's Hospital & Harvard School of Public Health). In addition to each group's individual websites, both of the emergency manuals, along with implementation tools and resources, can be accessed from the Emergency Manuals Implementation Collaborative website²⁶. These emergency manuals help clinicians ensure comprehensive and accurate diagnostic and treatment interventions needed during critical events such as a malignant hyperthermia or local anesthetic toxicity. The presence of emergency manuals in simulated operating room environments has been associated with significant improvement in the management of operating room crises^27–29. However, researchers are trying to determine the effectiveness of emergency manuals in other clinical settings. Limited data is available regarding the use of emergency manuals in the intensive care environment. We are aware of one other study that explicitly evaluates the impact of ICU-centric emergency manuals³⁰. These researchers focused on team performance with experienced clinicians, with mandatory use of an emergency manual. Additionally, much of the research that has been done regarding emergency manuals has focused on physicians with at least some meaningful experience in managing critically ill patients (e.g. PGY-2 or greater clinical anesthesia residents and attending physicians^31–33). However, data is lacking to show whether emergency manuals impact care when the clinician in charge is relatively inexperienced, such as an intern (PGY-1). We wondered whether providing an emergency manual, without explicit instructions to use it, would help novice physicians' individual performance during a critical event in an ICU when the help of experienced providers was unavailable.

The goal of the study was to investigate the impact of an emergency manual on novice physician performance in managing a critical event in the ICU. We hypothesized that the use of an emergency manual would significantly improve management of unstable bradycardia in a simulated ICU patient.

Methods

After receiving institutional review board (IRB) exemptions at the University of North Carolina and the University of Kentucky, a multi-year (2013–2016), multi-institutional study commenced with volunteer PGY1 anesthesiology interns (n=41) during their fifth or sixth month in residency training. All of the study participants graduated from an accredited US-based medical school with similar baseline clinical experiences during their internship and American Heart Association ALCS certification within the previous 12-months.

All of the events occurred in a simulation center either at the University Of North Carolina School of Medicine or the University of Kentucky School of Medicine. At both centers, a high fidelity patient simulator (SimMan 3G, Laerdal Medical, Wappingers Falls, NY) was used in the setting of a standardized simulated ICU environment.

Group allocation was performed each year by assigning one group of participants to have access to the emergency manual (intervention group) and the other group of participants to not have access to the manual (control group) during the simulated case. The participants were blinded to the group they were assigned. Participants assigned themselves to groups only taking into account their individual schedule availability. Because the allocation process did not depend on the participants' abilities, age, sex, previous clinical rotation experience, or standardized testing scores, the allocation process was likely to create groups with randomly distributed resident abilities. Each year, the testing was completed over a one day period with a confidentiality contract in place for each participant that prohibited discussion regarding the scenario with other participants in the study. In total, 41 interns participated in the study, 18 in the control group and 23 in the intervention group.

Since the “Emergency Manual: Cognitive Aid for Perioperative Critical Events”, by the Stanford Anesthesia Cognitive Aid Group²⁴, was implemented into the operating rooms at the study institutions in the preceding academic year, it was the emergency manual chosen for the study. The implementation of the emergency manuals in the operating room environments included discussions at departmental grand rounds lectures and during anesthesiology resident simulation debriefing sessions. It should also be noted that no emergency manuals had been implemented in any other clinical environment outside of the operating room at the two institutions. Additionally, since the study subjects were in their PGY-1 of residency training, they did not have any exposure to the manuals prior to the study. The emergency manual was placed in the simulated ICU environment on the patient's bedside table so the participants in the intervention group could immediately identify its location for usage while caring for the patient. None of the participants were presented with information about the simulation scenario prior to arriving at the testing site. However, approximately three days prior to the simulation testing sessions, all of participants received a thirty-minute lecture from a content expert on application of emergency manuals. No baseline testing of the interns was completed prior to the study.

During the simulated event, all participants were individually assessed on their ability to treat unstable bradycardia (HR=25, BP=60/40, SaO₂ = 100%). The checklist used exclusively to assess the participants in the study was created from elements in the Stanford Emergency Manual²⁴ after exercising a modified Delphi method technique. As part of the technique, four anesthesia providers participated in three separate rounds of eliminating or adapting items from the checklist in order to create a consensus list that was comprehensive of the actions applicable to a non-operating room ICU crisis. For each round of the process, all of the anesthesia providers reviewed the interventions on the Stanford Emergency Manual regarding unstable bradycardia to decide whether to maintain, eliminate, or alter the interventions. The goal of the iterative process was to maintain authenticity to the emergency manual items and applicability to the simulated environment. Ultimately, three interventions from the Stanford Emergency Manual were modified on the assessment checklist to fit the ICU environment in the simulated scenario. However, the actual emergency manual used in the simulation encounter was not altered from the original format and the participants had to adapt the recommended interventions to the ICU environment. Additionally, one of the interventions from the manual was excluded from the assessment checklist (“inform OR team”) since the participants were told they were the primary service team (Table 1).

Table 1:

Unstable bradycardia checklist. This analytical checklist was adapted from on the recommended treatment strategies from the Stanford Emergency Manual²⁸ for use in the ICU.

Prior to the start of the scenario, each participant received an orientation to the physical environment (e.g. simulated ICU) and a verbal report about the simulated patient's status (Table 2). For scoring purposes, participants were instructed to clearly verbalize the diagnostic and therapeutic interventions they desired to implement for the simulated patient. Participants were also told that they could use “any items in the room” to help care for the patient. The intention was to let the participants know that using the emergency manual was acceptable during the simulation without explicitly instructing them to do so. Additionally, the participants could have used any item that was on their person, such as smartphones, handbooks, or ACLS cards. After the orientation, each individual participant was instructed to enter the simulated ICU to evaluate and care for the patient. Upon arrival in the simulated ICU, each participant found themselves alone, with an unresponsive patient mannequin and no other care providers physically present. However, there was a faculty member present in the simulation control area to assist the participant with any questions regarding fidelity of the scenario.

Table 2:

Scenario Case Stem: Unstable bradycardia in the ICU. The clinical scenario verbally presented to each of the study participants before the simulation began.

The participants were scored in real-time by the two anesthesia providers as observers, stationed in the simulation control area, on the 12-item “unstable bradycardia” checklist of interventions (Table 1). The observers completed a single evaluation checklist for each participant's performance. Clear demonstration of the intervention, with unanimous agreement between the two observers, must have occurred for the participant to receive credit for the checklist interventions. If there was disagreement between the observers, which was infrequent, or it was ambiguous whether an intervention was completed, the participant did not receive credit for the intervention. The observers were not blinded to participants' group allocation as the emergency manual was easily visualized by the observers in the simulation space. For interventions “feel for pulse” and “confirm oxygenation and ventilation” participants were given credit if they were observed performing the intervention. For every other item on the checklist, the participant was required to clearly verbalize the actions to receive credit. In order to create an opportunity for all items on the checklist to be attempted, the scenario was designed so that no intervention would resolve or exacerbate the unstable bradycardia. The scenario lasted up to 10 minutes, or 600 seconds, which was sufficient to perform all of the interventions on the checklist and clinically representative of true hazard to the patient if the condition remained unresolved. If a participant performed all of the interventions on the checklist prior to the end of the designated time limit, the simulation ended.

The study was designed to detect a difference in the intervention group compared to the control group at a power level of 80%. The standard deviation was conservatively estimated to be 2 interventions difference between groups based on prior studies investigating the impact of emergency manuals in simulated case scenarios³³. The required minimum sample size based on these assumptions was 16 subjects in each study group. All power calculations were performed a priori.

Data was analyzed via descriptive and inferential statistics. Differences between means were tested using two-tailed Student's t-tests. All analyses were performed on an intention to treat basis, and participants who had access to the emergency manual but did not access it were included in their originally assigned group. All data analysis was performed using Stata Data Analysis and Statistical Software (Release 12, StataCorp, College Station, TX).

Results

Individually, the control group (no access to the emergency manual) averaged 7.1 out of 12 recommended interventions per participant, while the intervention group (with access to the emergency manual) averaged 9.9 out of 12 recommended interventions (p<0.01) per participant. The control group collectively performed 128/216 (59%) of the recommended interventions in the allotted 600 second timeframe. The intervention group collectively performed 228/276 (83%) of the recommended interventions in the allotted timeframe. Figure 1 demonstrates the range of interventions performed by each participant. No participant in the control group completed all 12 interventions and everyone used the entire allotted time frame. Two participants in the control group accessed another form of cognitive aid during the simulated scenario which included a smartphone ACLS application (n=1) and an anesthesiology textbook via smartphone (n=1). Those participants who accessed the manual in the intervention group averaged 205 seconds to begin using it to guide treatment of the patient. Six of the subjects in the intervention group performed all of the interventions on the checklist, leading to a mean time for completion of the scenario in the intervention group of 582 seconds (p=0.04). Two participants in the intervention group, one from each participating institution, did not access the manual. These two participants performed 5/12 and 8/12 of the potential interventions respectively.

Figure 1.

Number of interventions performed by subjects with and without access to emergency manual. The group without access to the emergency manual performed between 4 to11 interventions per participant, whereas the group with access to the manual performed between 5 to 12 interventions per participant. Dashed lines represent the average number of interventions for each respective group. Asterisks denote participants in the intervention group who did not access the manual.

Discussion

Critical events in hospital-based settings can be complex, stressful situations characterized by time pressure (e.g. decisions that must be made quickly) and high stakes outcomes (e.g. life or death). Even experienced clinicians may experience cognitive loading during critical events that exceeds their working memory capacity, leading to errors of incorrect or incomplete management. Such stress and cognitive load can be magnified for a novice clinician with less clinical expertise and emergency management experience. For decades, junior physicians have carried around some version of a pocket-sized “cheat sheet”, handbooks, or manuals to help them manage such clinical scenarios. This study provides novel evidence that use of a relevant, familiar, accessible emergency manual was associated with a significant improvement in novice physician's clinical management of a simulated life-threatening ICU event. By having an emergency manual at the bedside of a critically ill patient, the novice physician will have a familiar “handbook” that contains the clinical scenarios they are likely to encounter in an ICU environment. Importantly, although many study participants had access to their own pocket aids, such as ACLS cards, and were told they were free to use any items in the room, only two of the control group participants used such aids. There are many possibilities as to why this occurred including a lack of familiarity with the pocket aids, a deficiency in the training related to the proper use of the pocket aids, stress related to managing a critical event, or simply forgetting to access the pocket aids since they were not their immediate field of vision. Additionally, this study showed a significant decrease in time to performance of recommended interventions when an emergency manual is used. No intervention participant delayed initial pulse check or life saving interventions because of inappropriate or distracting emergency manual use. These findings have important downstream implications for improving the quality of care provided to critically ill patients via increasing thoroughness and timeliness of interventions.

To date, a large percentage of emergency manuals that have been developed for medical contexts have been intended for use in intraoperative settings. This study, in conjunction with the one other study³⁰ demonstrated that emergency manuals could be useful if adapted to ICUs and other non-operative settings. To our knowledge, there is no existing rigorously validated emergency manual or other checklist specifically intended for use in the ICU environment. Adaptations for the non-operative environment are necessary because logistics of support personnel (e.g.. respiratory therapists and bedside nurses) and diagnostic or therapeutic resources (e.g. radiology, point-of-care lab testing, medication order and delivery) differ considerably from one clinical environment to the next. Therefore, the creation of a validated emergency manual written specifically for ICU would be very useful for both inexperienced and experienced clinicians.

Since emergency manuals intended for use outside of the operating rooms are not yet widely available, we still suggest that hospitals place operating room emergency manuals, such as the Stanford Emergency Manual used in this study, in ICUs where they could potentially help interns and other providers manage critical events. Paper versions of cognitive aids, which appear to be more favorable to providers than electronic versions³⁴, would be appropriate for ICUs as they could be stored in a standardized location in each patient room. More importantly, clinicians in these units should receive implementation training regarding usage of the manuals that are deployed, since familiarity with format and content is related to checklist failure, or nonuse, and enhances overall usability^28,35. Additionally, timely use of emergency manuals can lead to more timely and appropriate patient care. This was demonstrated in the study when analyzing the three lowest performing participants in the intervention group. On average, these three participants accessed the manual 297 seconds into the scenario as compared to an average of 205 seconds for the entire intervention group and an average of 190 seconds for the intervention group minus these three lowest performing participants. This delay in accessing the manual subsequently lead to a delay in providing the appropriate patient care interventions.”

We perceive that our findings may be generalizable beyond the study group (e.g. anesthesia interns) and suspect that novice physicians in many other specialties would benefit from the use of emergency manuals in intensive care units and other acute care settings. Hospitals should consider customizing emergency manuals to create institution specific manuals tailored to their institutional resources and norms. Potential obstacles to widespread implementation of emergency manuals include the costs to obtain the manuals, manpower needed for customization for the clinical care areas of interest, education of providers on the proper use of the emergency manuals, and the cultural barriers that may need to be overcome for successful deployment. However, the potential for improved care of patients and decreased stress for providers during critical events likely outweighs the effort required to overcome the above-mentioned obstacles.

It is notable that although all the participants in the study were told that they could use any resource to help them treat the patient, only two used a cognitive aid other than the emergency manual. In the age of smartphones, no provider is ever far from an unlimited source of information to help them treat complex medical scenarios. Despite two participants in the control group accessing cognitive aids on their smartphones, they still did not treat the unstable bradycardia as promptly or accurately as those with the emergency manual at the bedside. This is consistent with a prior study³⁴ that showed providers favor paper cognitive aids over electronic aids. It also shows that placing the familiar manual at the bedside, with accurate and relevant information, may provide a “nudge” to the provider to use the aid in a timely manner, which can directly improve patient care. It is interesting to note that the rate of emergency manual use in this study was much higher than rates seen in other similar studies with trainees^31–33. This may be a result of the familiarity and training with the Stanford Emergency Manual, since study subjects had been given an educational session on the manual three days prior to the intervention. They would not, however, have had any prior clinical experience with the manual since the operating room was the only place it was used at the study institutions. Additionally, the increased rate of use may be some initial evidence that inexperienced providers are more likely to use cognitive aids due to their lack of familiarity with clinical situations or lack of confidence caring for patients without additional information.

As with any study, this study had some limitations. First, the assignment of the study subjects occurred via a blinded, self assigned, group allocation process and not a randomization process. Further, the observers were not blinded and thus may have favored crediting the intervention group. Second, the study design differed from many other immersive human patient simulator studies as the participants were instructed to verbalize their diagnostic and therapeutic interventions. As a result, this study deign may not have accurately reflected the cognitive load of task switching or the impact that interruptions on emergency manual reading and clinical tasks have on one another. Third, our analysis did not specifically account for incorrect actions performed. Certainly, performing inappropriate interventions could lead to deterioration in a patient's clinical status. Our secondary outcome measure – time to correct performance – may offset this limitation though, since inappropriate interventions would lead to delay in successful treatment. Fourth, clinical teams were somewhat artificial (e.g. a lone resident) for this study; with realistic clinical teams that include more providers, such as a team leader or a reader of a cognitive aid, actions could have been completed in a more coordinated, parallel manner. In our study, the lone resident provider may have been distracted while trying to read the emergency manual and simultaneously provide clinical care. Fifth, we only evaluated interventions adapted from the Stanford Emergency manual (Table 1) and not other interventions that may have positively impacted patient care in the scenario. Additionally, we did not assign weights to the interventions that may have provided a greater impact to the patient over another intervention. Lastly, like many other simulation studies investigating trainee performance in emergency situations^31,36,37, our study consisted of a single clinical scenario and a single-specialty group of interns. While we did study subjects from two distinct academic institutions, all participants are future anesthesiologists, and therefore may display a level of aptitude for crisis events that may not be true of physicians across all specialties. Future work evaluating the impact of emergency manuals on novice physician care should include more scenarios, other clinical environments, and novice physicians from other intended specialty groups.

Conclusion

This study shows that use of emergency manuals significantly improved novice physician performance (completeness and timeliness) in managing a simulated clinical crisis in the ICU setting. Patient care in real clinical environments outside of the operating room may be improved when an emergency manual is used, particularly when the physician managing the event is a relative novice.