Latent safety threat identification during in situ simulation debriefing: a qualitative analysis

Abstract

Background

Latent safety threats (LSTs) in healthcare are hazards or conditions that risk patient safety but are not readily apparent without system stress. In situ simulation (ISS), followed by post-scenario debriefing is a common method to identify LSTs within the clinical environment. The function of post-ISS debriefing for LST identification is not well understood.

Objectives

This study aims to qualitatively characterise the types of LSTs identified during ISS debriefing.

Methods

We conducted 12 ISS trauma scenarios followed by debriefing at a Canadian, Level 1 trauma centre. We designed the scenarios and debriefing for 15 and 20 min, respectively. Debriefings focused on LST identification, and each session was audio recorded and transcribed. We used an inductive approach with qualitative content analysis to code text data into an initial coding tree. We generated refined topics from the coded text data.

Results

We identified five major topics: (1) communication and teamwork challenges, (2) system-level issues, (3) resource constraints, (4) positive team performance and (5) potential improvements to the current systems and processes.

Conclusions

During simulation debriefing sessions for LST identification, participants discussed threats related to communication and interpersonal issues. Safety issues relating to equipment, processes and the physical space received less emphasis. These findings may guide health system leaders and simulation experts better understanding of the strengths and limitations of simulation debriefing for LST identification. Further studies are required to compare ISS-based LST identification techniques.

BACKGROUND

Latent safety threats (LSTs) in healthcare are conditions, not immediately apparent, that under certain circumstances manifest and threaten patient harm. ¹ These hazards range from equipment failures and physical hazards to inefficient protocols and communication breakdowns. Detection of LSTs is inherently difficult and frequently occurs retrospectively, only after the harm occurs.

In situ simulation (ISS), a simulation technique that occurs within the real clinical workspace, can be used to proactively identify LSTs within the clinical environment. ^1–5 The application of ISS for LST detection is analogous to crash testing an automobile, whereby the system is rigorously tested before the patient is affected. Typically, LST identification occurs by participant feedback, often facilitated using structured debriefs or post-ISS surveys. ^6–9 While these reporting techniques are pragmatic, they may be subject to participant recall biases who may not be aware of critical LSTs if their situational awareness is impaired during a high-stress resuscitation. ¹⁰ Participants accustomed to a specific workflow may not perceive the potential for that process to manifest as a safety threat. There is uncertainty regarding which LSTs may not be detected during simulation debriefings. Alternative LST detection techniques exist, including structured video review and live teamwork self-assessment tools; however, studies comparing efficacy are lacking. ^{4 11 12}

A structured analysis of participant feedback is required to better understand how best to apply this technique for LST identification. Although debriefing methods have been critically analysed, ¹³ there are limited data that evaluates the effectiveness of participant feedback specifically for LST identification. Without better understanding the types of LSTs that are identified by participants and how this compares to other methods, we may be systematically selecting for certain types of LSTs at the expense of other less apparent threats. To address this, we conducted an exploratory study to qualitatively characterise the types of LSTs identified during debriefing sessions following ISS trauma scenarios.

METHODS

This study is nested within the Trauma Resuscitation Using In situ Simulation Team Training (TRUST) study, which aims to identify and prioritise LSTs using a video-based framework analysis. ¹¹ We recorded and transcribed each debriefing session that comprises the data for this study. The complete TRUST study setting, design, participants and protocol are published previously; however, a summary of the methods is described later. ¹¹ The Data analysis of the debriefing transcripts section is novel and specific to this paper.

Setting

We conducted this study at a Canadian, Level 1 Trauma centre with approximately 800 annual trauma team activations managed by the on-call trauma team. The unpredictable nature of trauma care, inherent diagnostic uncertainty and high patient acuity provide an ideal setting for LST identification. The research ethics board of St. Michael’s Hospital approved this study.

Study design

We conducted a 2-year review of trauma cases at our institution containing an unexpected death or adverse events. We identified themes to develop four risk-informed trauma simulation scenarios that covered the following topics:

1. Surgical airway

2. Blunt trauma and massive haemorrhage

3. Medical event leading to significant injury

4. Penetrating injury in an agitated patient

We pilot tested each scenario among an interprofessional trauma team (non-study participants) for clarity, flow and logistical considerations. Planned scenario duration was 15 min to minimise disruptions within the clinical environment followed by a 20-min debriefing session.

We conducted 12 monthly, unannounced ISSs. We selected scenarios on a rotating basis, each conducted thrice with the exception of topic #2 (blunt trauma) repeated in place of #4 (penetrating injury) for logistical reasons during the final rotation. Following the usual the trauma team activation process, the team arrived in the trauma room and cared for the patient in the usual manner, using the same equipment, personnel, resources and protocols that are available during actual trauma resuscitations. We placed four video cameras (GoPro Hero4, Sony Handycam Exmor R) at different locations within the trauma bay to capture all areas of clinical care and the debriefing. An overhead microphone (Aputure V Mic D1 Directional Condenser Shotgun) and handheld voice recorders (Sony ICD-UX533 digital voice recorder) captured audio recordings during the scenario and the debriefing.

Participants

Participants consisted of the on-call trauma team on the day of the simulation, including a trauma team leader (TTL, either a staff emergency physician or general surgeon), two to three emergency department nurses, two general surgery residents, one to two orthopaedic residents, an anaesthesia resident, a respiratory therapist, a social worker, an X-ray technician and a clinical assistant. While all participants knew of this study in advance, which occurred during the consent process, only the TTL was notified in advance of the precise timing of the ISS.

Protocols

Before each ISS session, the study investigators sought and received approval from the staff emergency physician and charge nurse to conduct the simulation prior to each ISS session, based on departmental staffing levels and current patient acuity levels. Two members of the simulation team acted as confederates during the scenario including (1) a respiratory therapist who functioned as a trauma team member and ensured any simulation equipment issues were quickly remedied and (2) a paramedic who provided a scripted handover to the team.

Debriefing design

Upon completion of the ISS scenario, two simulation and debriefing experts (AP and KW) led an interprofessional 20-min debriefing session following established system integration debriefing best practices. ^{14 15} Participation in the debriefing was optional but encouraged. At any point during the simulation or debriefing, participants could attend to urgent clinical duties if they deemed it an essential patient care situation. We began the debriefing by promoting psychological safety among the participants, notably highlighting that individual performance would not be evaluated and all participant feedback would be anonymised prior to data analysis. ¹⁵ This serves to allow participants to speak freely and provide honest feedback without fear of repercussions. All participants in attendance during the debriefing were invited but not required to speak.

The debriefing objectives during this study differed from typical training-based simulation sessions. Instead of focusing on individual learners’ knowledge and communication skills, we sought to solicit participant-identified LSTs. ^16–21 This aligns with a recently described, systems-focused debriefing framework. ¹⁴ To anchor and familiarise participants with the concept of LSTs, we aligned the discussion with elements from the Systems Engineering Initiative for Patient Safety (SEIPS) framework. ²² The SEIPS framework is a human factors approach to healthcare quality and patient safety based on five elements that can affect the work system including the patient/person, tools/technology, tasks, physical environment and organisational characteristics. We initiated the discussion by asking participants to voluntarily share the ‘most significant safety threat or barrier to safety’ that they observed or experienced during the simulation. Member checking, restating and summarising participant statements, was performed periodically throughout the debriefing process.

The audio recordings were used to generate transcriptions, and one researcher (AP) reviewed each transcript for accuracy.

Data analysis of debriefing transcripts

We applied a qualitative content analysis following the conventional approach. ²³ The observations are direct responses to questions by participants that require low inference, which makes a descriptive qualitative approach ideal for this analysis. First, one researcher (SS) read the transcribed debriefings to understand the data. Next, the debriefings were imported into NVivo (Version 11, QSR International), a qualitative data analysis software. We applied a qualitative content analysis approach because of its flexibility and to provide a systematic and objective means to make inferences of contextual meaning within text data. ^24–26 We devised an initial coding tree with an inductive approach rather than a priori codes due to minimal existing qualitative research on LST identification. One researcher (SS) developed an initial coding tree with the first four transcripts based on manifest content meanings. A second researcher (AP) also independently coded the first four debriefings into an initial coding tree developed by the first researcher. The two researchers then reviewed the coding scheme together and came to a consensus on a revised coding tree. The first researcher then re-coded the first four transcripts and then coded the subsequent eight transcripts, while continuing to test and refine the coding scheme. Through constant comparison, we achieved saturation in our coding scheme by the seventh transcript, though we analysed all 12 transcripts.

Two researchers (SS and AP) generated candidate topics following a review of all transcripts. The set of candidate topics were reorganised and refined until a final categorical framework was generated through consensus of both researchers. Triangulation allowed us to mitigate potential biases and improve the validity of results. ²⁷ We were able to achieve this in two ways: First, we had repetitions of each type of trauma simulation, allowing us for multiple observations of a standardised scenario. Second, the two researchers independently reviewed the transcripts before arriving to a consensus, ensuring a comprehensive understanding of the transcripts. Researchers actively attempted to moderate their own biases towards ISS and LST identification and further minimised them by asking open-ended questions and practising reflexivity throughout the debriefing and coding process. ²⁸

RESULTS

During 12 ISS sessions, a total of 124 individuals participated in the simulation and 110 attended the debriefings (table 1). The difference (124 vs 110) occurred when participants left to attend clinical duties and were absent for the debriefing.

Table 1.

Participant information

Participant discipline	n (%)
Nurse	21 (17)
Physician
General surgery resident	19 (15)
Trauma team leader (TTL)*	16 (13)
Anesthesia resident	11 (9)
Orthopedics resident	8 (6)
Emergency physician	5
Respiratory therapist	14
Clinical assistant	9
X-ray technologist	8
Chaplain/Social worker	5
Porter	3
Security	2
Pharmacist	2
Clerical	1

*Four scenarios had both resident and staff physician TTL.

We identified five major topics from the data. Representative quotes from participants under each of these topics are presented in table 2. The five topics are listed as follows:

Table 2.

Representative examples of identified topics

Theme	Representative quotes
1. Communication and teamwork challenges	‘I think sometimes that close-loop communication with the documenting is really hard for us. So, like what’s been done—I can’t see the patient because there’s so many people. So sort of that declaration I think just takes practice. Like yelling out your findings as they go.’ ‘So, me being the charting nurse, I did feel at the beginning, there wasn’t—like it was sort of a closed loop.., I couldn’t actually hear what was happening, so that’s why I had to ask a couple times because it wasn’t communicated to me what, like if you had medical histories, if you had allergies, so I had to go ahead and ask him.’ ‘every time EMS is here, it’s—they always have to repeat the story numerous times, no one really hears it and it’s always that kind of challenge, so I think even just by making our best efforts to try to quiet the room there, there’s obviously still a bit of a lack of getting that story across.’
2. System-level issues	‘We had to have the blood come back (during MTP), but there are all these little things. And then the last thing is that we have this check-list. So we have three things that were up on the board, that no one used any of them. It’s interesting because we’re using them as primers for—so TXA, history, and a trauma care check-list—and none of those got used…. Nobody probably knows about those, so there needs to be some emphasis on—you come into the trauma, this is your trauma, you need to know about these three things’ “Yeah. We always have the battle of …, I wanted blood and no phenylephrine. We tend to be a ‘phenyl less’ environment in here on purpose. But there is that sort of discussion, first of all, just around pre-mixed drugs in general, I think, is something to think about. Ketamine pre-mixed in a syringe, certainly they do that in the U.S. when they’re intubating, code of hospital, where you just grab your thing and you know exactly how much you’re pushing. But the debate around phenylephrine in trauma, people have different feelings around that.
3. Resource constraints	‘we’re short-staffed’ ‘Right. And the biggest difficulty, with the outside of this, stopping and knowing what’s happening, having a bit of a finger on the pulse, and we can’t spare one nurse for that. Already, our two nurses are very busy, we’re running trauma, and you guys are always running around’ ‘because there are things I was asked as an RT—like to check the pupils and then she was putting an IV while there was no one else to check pupils—I don’t know how to check pupils…. And I had to wait for the doctor to come, and she was multitasking in multiple different areas. I think she was just spread thin more than she should have been. That’s my assessment.’
4. Positive team performance	‘I think in fairness, and in summary, that declaration was made quite nicely by you. You were pretty clear about that’ ‘He’s a clinical assistant—to go get the blood for the MTP. He returned six minutes later, 9:41 with the blood, and three minutes later the nurses had it hanging. So that time’s pretty great’ ‘Well, from the patient’s point of view, what I liked is—especially for you, I had you here. From my eyesight, I could actually see someone because there’s a lot going around, so just having a contact with someone was good. That was nice to always be there because sometimes a lot of people would disappear and I felt abandoned. But then after, as long as someone was there saying take deep breaths, it’s going to be okay and answer my questions, I felt reassured. It made me calm down a lot. So, I liked always having someone there telling me, like just breathe. It’s going to be okay.’
5. Potential improvements to current systems and processes	‘And the idea of a protocol—that’s I would call it, is sort of a routine restatement of the major issues by the trauma team leader at a step point in the resuscitation also makes sense. Because you’re right. In certain circumstances like this, you’re compelled to act quickly, but that doesn’t negate the importance of saying … very early in the resuscitation when the whole team is present, the importance of restating the major issues as they appear at the time. And that could be another system-based solution. I don’t know if it’s either by routine delay or there’s some sort of team-based summary that happens fairly shortly after the whole team arrives.’ ‘Can I just add that I’ve worked in other facilities where sometimes there is a clerical that’s able to be here during a trauma. It’s really helpful—half the time, there’s phone calls going on and missing have the story because I’m busy answering the phone. It’s OR calling, it’s locating, it’s blood—so to have them there, even for the first fifteen minutes of a trauma might be an idea just to facilitate those calls and to sort of….’ ‘As soon as a trauma is activated, maybe we have blood coming immediately, before we know anything about the patient. Then, you just run a risk—I don’t know how hematology would feel about that, because then you run a risk of wasting a lot of blood products.’

1. Communication and teamwork challenges

Communication and teamwork LSTs were repeatedly discussed and became a primary focus during every debriefing, despite the moderators attempt to identify a variety of types of LSTs. These LSTs related to interpersonal interactions include communication errors, conflicting mental models, lack of role clarity, and uncoordinated teamwork. Interpersonal issues were the predominant focus of all the debriefs, and all debriefings had at least one identified communication or team cohesiveness issue.

2. System-level issues

Participants also identified system-related LSTs. These included concerns regarding clinical protocols and processes, including the massive transfusion protocol (MTP). These were issues that existed throughout a protocol, process or workflow rather than discrete safety threat events (eg, tripping over a cable).

3. Resource constraints

The most frequently identified resource constraint related to issues about the lack of human capital. This translated into feelings of task overload. At times, trauma team members felt overwhelmed with the number or intensity of tasks they were responsible for and expressed desire for help.

4. Positive team performance

Participants took note of instances when things were done well. Although the facilitators did not directly ask about what went well, participants repeatedly highlighted the positive aspects of care.

5. Potential improvements to current systems and processes

Participants also suggested opportunities for improvement after LSTs were identified, including predrawn medication syringes, a simplified MTP activation process and addition of clerical personnel to coordinate phone calls and trauma bay logistics. Feedback was not limited to a specific type of LST, but it tended to be predominantly system-based improvements.

DISCUSSION

During this exploratory study of simulation debriefing for LST identification, we observed that participants focused on topics related to communication or teamwork-related interactions. Specifically, participants addressed elements related to positive performance and opportunities for team-based improvements. Interestingly, our debriefing scripts did not explicitly solicit feedback related to team performance, and we noted it challenging for participants to shift their focus towards other LSTs including those pertaining to equipment, processes or systems. While team performance is valuable information, there is an opportunity cost to allocating limited debriefing time to such topics when the goal is to explicitly identify LSTs.

In this study, participants described LSTs related to communication lapses, lack of leadership, poor role clarity and a lack of human capital. These topics provide valuable insights into team dynamics; however, we learn very little about system and equipment hazards. Our findings support the notion that people have a predilection to discuss socially relevant topics over other LSTs. ²⁹ It appears that participants are more sensitive towards issues relating to interpersonal behaviour, than issues with the equipment, systems and the physical workspace.

Highlighting these observations, our TRUST study team published a video workflow analysis from this same simulation data set evaluating clinician movement during surgical airway performance. ³⁰ We identified several workflow inefficiencies during the simulations including lack of task delegation, knowledge gaps on equipment location and lack of equipment bundling. These LSTs were associated with increased time to completion for a time-sensitive procedure. During the debriefing for these simulations, participants rarely discussed these LSTs. It is unclear why, but it may be related participant prioritisation of their experience or lack of awareness to these LSTs. Our results suggest that LSTs related to interpersonal issues may be better identified during debriefings than equipment or physical space issues.

These results are of interest for organisations as they seek to elucidate interpersonal LSTs and develop strategies to minimise them. Furthermore, benefits and limitations of debriefing feedback is important for simulation organisers to understand a priori when planning and implementing simulation for LST identification. Various debriefing strategies may be needed to maximise a focus on all types of LSTs. This can include acknowledging at the start of the debriefing the tendency for participants to focus on interpersonal interactions despite the focus on equipment, system and physical space LSTs. Alternatively, facilitators can identify a specific time during the debriefing to explicitly address interpersonal and communication challenges before shifting towards other LSTs. This could include a more structured debriefing script designed to elicit feedback pertaining to LSTs linked to systems, processes and the physical space. Inevitably there is a trade-off with this approach as increasing structure may prevent participants from discussing their actual experiences. A combination of unstructured discussion and LST focused prompts may represent a pragmatic balance, but further investigation is warranted to better understand this approach.

In addition, simulation debriefings may be useful to identify LSTs related to long-standing, multifactorial themes not evident at a single moment. In our study, participants reported system-level deficiencies such as issues with the MTP protocol. These discussions may have been influenced by a combination of simulation and real-world experience.

Based on this study, to capture the wide range of LSTs that may exist within a system, simulation programmes may need to consider multiple approaches to ISS analysis. Specifically, exclusive reliance on participant debriefing feedback may be insufficient to identify all critical LSTs and it may be prudent to combine debriefing with other strategies. ^{11 31} For example, post-simulation debriefing could be combined with the video workflow analysis mentioned earlier to capture interpersonal LSTs as well as LSTs related to the equipment/environment. The video recording of the simulation could also be replayed to participants during the debrief to stimulate discussion of other types of LSTs. However, further work is needed to compare and contrast LST identification and analysis techniques.

This study has several limitations. This is a simulation-based study, and participants may not perform in the same manner as when they participate in real trauma care. This could result in underemphasis or overemphasis of certain LSTs during debriefing. However, we conducted this study ‘in situ’ within the workplace, to minimise this effect. Participants are aware of video recording during the simulation session, and despite clear statements that this was not a performance evaluation, the Hawthorne effect may still result in behaviour modifications. ³² We did not conduct an analysis of the feedback types between disciplines and professions given our study’s sample size. Further studies should seek to more precisely understand the feedback from specific healthcare professions. Finally, there is also the possibility that participants identified certain LSTs but did not mention them during the debriefing. However, facilitators encouraged participant feedback and exploration of all types of LSTs during the debriefing.

This study contributes to an improved understanding of how participants respond during ISS debriefings for LST identification. The next steps in translational simulation, or simulation connected to patient outcomes, are evaluating interventions that address such issues. ³³ Participant feedback should remain an essential part of this process, and future work should address in more detail how to maximise the participant perspectives.

CONCLUSION

Post-simulation debriefing is commonly used to identify LSTs. In our study, we found that this technique readily identifies communication breakdowns and teamwork interactions yet it may fail to adequately capture other relevant LSTs (eg, related to equipment, systems and physical space). An improved understanding of the strengths and limitations of LST identification methods will allow health system administrators, educators and simulation experts to tailor an approach that addresses the spectrum of potential LSTs. Further research is required to compare different simulation-based LST identification techniques.

What is already known on this subject.

• Debriefing in situ simulation is a common method used to identify latent safety threats.

• Debriefing may inherently be subject to participant biases (eg, recall bias) and this may impact if and how latent safety threats are identified.

• No studies have evaluated the type of LSTs identified during in situ simulation debriefing.

What this study adds.

• This study characterises the types of LSTs identified during debriefing of in situ trauma simulations.

• During the debriefing, participants frequently discussed interpersonal interactions and positive team performance while rarely identifying LSTs related to the physical space, equipment or task-specific items.

• Further studies are needed to better understand the efficacy of simulation-based LST identification techniques.