Crash testing the dummy: a review of in situ trauma simulation at a Canadian tertiary centre

Abstract

Background

In situ trauma simulations allow for the trauma team and emergency department to practise team dynamics, resuscitation and logistics in a safe environment. The goal of this investigation was to show the feasibility of an in situ trauma simulation program at a Canadian level 1 trauma centre.

Methods

We performed a retrospective review of in situ simulations (maximum 20 min, followed by a 10-min debriefing session) at a level 1 trauma centre from 2015 to 2017. Errors were categorized according to the National Patient Safety Agency risk assessment matrix by 3 independent raters and assigned consequence scores (assessing potential harm) and likelihood scores (assessing the likelihood of potential harm). A risk score was calculated as the product of the mean consequence and likelihood scores. Errors per simulation and the number of simulations required for error resolution were recorded.

Results

We reviewed 8 in situ simulations and identified 54 errors, of which 7 were related to medications, 20 to equipment, 21 to environment/staffing and 6 to training. The mean consequence score was 2.85/5 (standard deviation [SD] 0.75, intraclass correlation coefficient [ICC] 28%), indicating minor to moderate harm. The mean likelihood score was 2.82/5 (SD 0.55, ICC 41%), indicating unlikely to possible. The mean risk score was 8.42/25 (SD 3.19, ICC 43%). One error (2%) was low risk, 23 (43%) were moderate risk, 26 (48%) were high risk, and 4 (7%) were extreme risk.

Conclusion

In situ trauma simulations are feasible in a Canadian centre and provide a safe environment to identify and rectify errors.

Trauma resuscitation is complex, unpredictable and prone to medical error.^1–5 The ability of trauma teams to safely manage rare events often depends on the frequency of exposure, which, in many Canadian centres, may be inadequate to maintain expertise.⁶ Simulation is an established tool in medical education and has been used to improve trauma team performance and patient safety.^7,8

Within a simulation laboratory that is removed from the actual environment in which care is performed, important system, equipment and environmental influences are challenging to fully evaluate. Furthermore, individual performance may be influenced by the artificial environment in which tasks are performed, and “suspension of disbelief” may be more difficult in the simulation laboratory.⁹

In situ simulation, in which team-based training is conducted in the actual patient care area, and the resources and equipment available to the usual treating team are used, has been described as “crash testing the dummy.”^10–14 In situ simulation has been used to proactively identify latent system errors in emergency medicine, pediatric anesthesia and trauma.^15–18 Latent system errors are system-based conditions that do not appear to be creating active harm and may lie dormant for some time but, in certain circumstances, may materialize and result in serious morbidity.^19,20 Multidisciplinary in situ simulation may inspire reflection from multiple clinical viewpoints and provide a better evaluation of latent safety errors. It may also provide an opportunity for team learning that may not be achievable within a traditional simulation laboratory.^21–23

The goal of this investigation was to show the feasibility of an in situ trauma simulation program at a Canadian level 1 trauma centre. We hypothesized that the introduction of in situ simulations would identify actual and latent safety errors that could be corrected by subsequent evaluations and provide unique learning opportunities for team members in a safe and supportive environment.

Methods

Setting

We evaluated in situ simulation involving the trauma team at the QEII Health Sciences Centre, a level 1 trauma centre in Halifax. It is a tertiary/quaternary referral centre for Atlantic Canada, serving a population of about 1 million. It performs over 400 major trauma activations each year.²⁴ Trauma team activations follow established predefined mechanistic, physiologic and anatomic criteria and are standardized across the province of Nova Scotia.

Participants

The trauma team includes an attending trauma team leader, resident trauma team leader, general surgery resident, orthopedic surgery resident, anesthesia resident, paramedic, respiratory therapist, radiology resident, radiology technologist and 3 emergency nurses. Three evaluators (the QEII Health Sciences Centre medical director of trauma [general surgery/critical care and trauma team leader attending], a simulation specialist [paramedic] and the Trauma Nova Scotia Program education nurse) were physically present for each of the in situ simulations. The evaluators developed test scenarios, provided feedback and led a focused team debriefing session.

Design

Initiation of in situ simulations in the emergency department were vetted through the QEII Health Sciences Centre trauma services committee, surgery executive committee and department head of Emergency Medicine, and the Nova Scotia provincial trauma committee. Concern regarding any potential impact on patient care was discussed, and all committees were in agreement that the benefits of the exercise exceeded the risks.

Starting in November 2015, in situ simulations were planned at monthly intervals and ran from November 2015 to May 2017. To evaluate the trauma system with various staff and variables that may be temporally related, the simulations were performed at a variety of times during weekdays, including evenings. One hour before activation, the intent to activate the trauma team was discussed in confidence with the emergency charge physician and charge nurse to determine whether there were any barriers to running the simulation. A “no-go” criterion was established whereby the simulation would be cancelled if the charge nurse or physician felt that the simulation would result in definite patient harm. Examples of no-go situations included a trauma activation already in progress or all monitored emergency department beds’ being occupied with critically ill patients in unstable condition who could not be transferred to lower-acuity beds. The trauma team was activated in the usual manner via the standard provincial trauma communication system and was not informed that the activation was a simulation.

A SimMan 3G simulator (Laerdal Medical) was brought into the trauma bay from the ambulance bay, and handover to the trauma team was conducted by paramedics as per standard protocol. The trauma team used equipment, medications and supplies from the trauma room in a manner similar to that with a real patient with trauma. This included using regular monitors, drugs and procedural equipment as well as transport to the computed tomography scanner and operating room when appropriate. To limit time away from other clinical activities, scenarios were limited to 20 minutes in length, followed by a 10-minute debriefing session.

Test scenarios were developed to investigate team behaviours, communication issues or equipment readiness. Most scenarios were developed with the use of real cases from the trauma database appropriate to meet predetermined educational goals. If an actual or latent error was identified during the simulation, it was added as an educational goal to the next simulation to test whether the efforts to correct it had been successful. During the debriefing session, observations regarding actual and latent safety errors as well as possible solutions were solicited from members of the trauma team. After each simulation, the medical director of trauma documented the identified safety issues and proposed solutions.

Data analysis

The primary data source for our study was the debriefing reports created by the medical director of trauma between November 2015 and May 2017. Each error was noted and presented to 3 independent reviewers (S.M., R.G. and S.J.) for framework analysis. An error was defined as a deviation in best care, and a latent error was defined as a potential set-up for patient harm. Best care was defined as care consistent with evidence-based practice guidelines, such as use of tranexamic acid, or with institutional trauma protocols that were not necessarily based on evidence-based guidelines. Errors were identified by the 3 evaluators during the simulation or by the participants during the debriefing session.

Framework analysis uses a matrix format to systematically reduce qualitative data to be uniformly analyzed.²⁵ The framework used was the National Patient Safety Agency risk assessment matrix²⁶ (Fig. 1). Errors were categorized into medication-related, equipment-related, environment-related or training-related. Each error was attributed a consequence score, a likelihood score and a risk score. Consequence scores ranged from 1 (negligible) to 5 (catastrophic) and were based on the severity of the outcome or potential outcome that would have resulted from the error. Likelihood scores ranged from 1 (rare) to 5 (almost certain) and were based on the probability of the outcome’s occurring. A risk score was then computed as the product of the consequence and likelihood scores, with a range from 1–3 (low risk) to 15–25 (extreme risk), reflecting the impact of the identified error. We summarized consequence, likelihood and risk scores of each reviewer as means and standard deviations (SDs) and assessed agreement among reviewers using intraclass correlation (ICC), through a 2-way mixed-effects methodology. The final presented scores are the mean scores across all 3 raters, thus adjusting for rater bias. For example, failure to give tranexamic acid in the setting of massive bleeding was allocated an average consequence score of 5 (death from exsanguination) and an average likelihood score of 4 (based on strong evidence for its use in this setting), for a risk score of 20 (extreme risk). In contrast, failing to use the mobile recording desk (institutional standard operating procedure because the recording nurse can hear better) had an average consequence score of 1 and an average likelihood score of 3 (whenever not used, our personal experience was that the nurses frequently complained they could not hear), for a risk score of 3 (low risk).

Fig. 1.

National Patient Safety Agency risk assessment matrix. Adapted from reference ²⁶ with permission of NHS Improvement.

Results

Over the study period, 8 in situ simulations were reviewed. The overall rate of cancellation of simulations was 27%, with a higher rate observed after simulations were introduced, relative to later in the study period. The most common reason for cancellation was Emergency Medicine staff overburden.

A total of 54 errors were identified, of which 7 were medication-related, 20 were equipment-related, 21 were environment-related, and 6 were training-related. Examples of identified errors included failure to administer tranexamic acid (medication), inability to locate a Thomas splint (equipment), inability of recording nurse to hear trauma team leader (environment) and no person able to perform focused assessment with sonography in trauma (FAST) (Table 1). A mean of 6 errors (SD 3.35) were identified per simulation.

Table 1.

Identified errors by category

Category	Error
Medication	Tranexamic acid incorrectly or not administered
	Tetanus not administered
	Mannitol not available or not administered
Equipment	Communication system malfunction
	Raney clips not available
	Nursing mobile desk not used
	Nursing recording sheet used improperly
	HoverMatt not used
	Thomas splint not available
	Level 1 infuser used incorrectly
	Glucometer not used
Environment	Recording nurse could not hear
	Imaging not available
	Neurosurgery unavailable
	Orthopedic surgery unavailable
	Radiology unavailable
	Paramedic overwhelmed with too many tasks
	Room not warmed
	Sign-in board not used
	Identification stickers not used
Training	Improper technique for spinal precautions
	Universal precautions not used
	No one trained in focused assessment with sonography in trauma (FAST) available

The overall mean consequence score was 2.85/5 (SD 0.75, ICC 28%), the mean likelihood score was 2.82/5 (SD 0.55, ICC 41%), and the mean risk score was 8.42/25 (SD 3.19, ICC 43%). Mean consequence, likelihood and risk scores by category are summarized in Table 2. Most errors identified were moderate or high risk (Fig. 2). A median of 2 simulations (interquartile range 1–5) were required to satisfactorily resolve the error.

Table 2.

Mean consequence, likelihood and risk scores^* by error category

Error category	Mean consequence score ± SD	Mean likelihood score ± SD	Mean risk score ± SD
Medication (n = 7)	3.00 ± 0.27	2.57 ± 0.32	8.05 ± 1.37
Equipment (n = 20)	2.77 ± 0.83	2.85 ± 0.54	8.18 ± 3.12
Environment (n = 21)	2.83 ± 0.78	2.90 ± 0.63	8.80 ± 3.82
Training (n = 6)	3.06 ± 0.80	2.66 ± 0.41	8.27 ± 2.84

SD = standard deviation.

^*Consequence errors were rated on a scale of 1 (negligible) to 5 (catastrophic).

Likelihood errors were rated on a scale of 1 (rare) to 5 (almost certain). Risk score was computed as the product of the mean consequence and likelihood scores, with a range from 1–3 (low risk) to 15–25 (extreme risk), reflecting the impact of the identified error.

Fig. 2.

Distribution of risk scores.

Discussion

Our study shows the feasibility of introducing training based on in situ simulation into a Canadian trauma centre with a goal of identifying latent safety issues. Each simulation identified a broad array of errors, with the most common categories being equipment and environmental. Once an error was identified, potential solutions were devised by the trauma team and implemented over the following month. Whether a solution was effective was specifically evaluated with the use of scenarios designed to test the solution until it was clear that the issue had been resolved. It took a median of 2 further simulations to solve the identified problem.

Our overall simulation cancellation rate was 27%, which is similar to that in other in situ programs.¹⁰ However, this rate was much higher when the simulations were first introduced. We subsequently met with the QEII Health Sciences Centre trauma services committee and chief of the emergency department to discuss the high rate of cancellation. It was felt that, although in situ simulation had some potential to negatively affect or delay patient care, the potential positive outcomes of running the simulations outweighed those risks. We subsequently agreed that the only criterion for cancelling the simulation was causing definite patient harm. This varies from other no-go criteria described in the literature²⁷ but was successful in increasing the number of simulations performed and may be helpful to understand potential errors that are the result of high workload strain. Monitoring the frequency of cancelled simulations was also informative as a marker of the potential impact that the trauma program might have been having on the rest of the emergency patient population.

Other investigators have detailed the successful implementation of in situ simulation programs in the emergency department. Patterson and colleagues¹⁰ introduced 90 in situ simulations over 1 year at a large centre with about 3000 resuscitation activations per year and were able to identify a latent safety error for every 1.2 simulations performed (rate of 3.5/simulation). Geis and colleagues²⁸ used in situ simulations to identify 37 latent safety errors and 46 errors in clinical proficiency over 24 in situ simulations in a new satellite emergency department. Similar to our study, most of the identified errors involved equipment and resource issues, and 86% were corrected by the time the facility opened. The ability of in situ simulation to identify a high rate of latent safety errors was validated by video analysis of actual trauma resuscitations, which also showed a high error detection rate, 2.4 errors/case. However, for many centres, video analysis may be impractical owing to privacy issues and the logistics of having a delay between obtaining the video footage and being able to view it.²⁹ In a comparison of in situ simulation versus traditional simulation, in situ simulation was able to identify more latent safety errors per simulation than traditional simulation.³⁰

We had a high rate of safety errors identified (mean 6 per simulation), with half of the errors identified as high risk. Our high rate of error detection may have been secondary to having 3 trained evaluators for each simulation or because we invited feedback from the entire trauma team. The high rate may also reflect a system that was in need of substantial improvement.

We found in situ simulation to be an excellent method to identify irregularities in care and to provide practical solutions. The errors, which likely would not have been identified through traditional means of error reporting, were resolved without any patient harm.³¹ During an actual trauma there are practical barriers to providing feedback, as some of the team members depart with the patient while others return to their regular clinical duties.²⁰ The use of in situ simulation accomplishes the dual goals of identifying and remedying latent safety errors as well as providing continuous opportunities to deliberately practise technical and nontechnical skills. This provides immediate benefit to the individual health care provider, the health care team and the next patient with trauma. On a strategic level, it also contributes to the evolution of the culture of safety.³² The realism and actual clinical environment of in situ simulation engages participants to improve the system and thereby patient safety as few other training methods can.^33–35

Although we feel that performing in situ simulations has improved patient safety at our centre, future studies could investigate whether identification and resolution of latent safety errors through this training activity actually translates into important patient outcomes, which is the ultimate goal of any quality-improvement program.

Limitations

The single-centre design of our study is a limitation, and the observations and utility of the in situ simulation program we have described may not be generalizable to other centres. However, in light of the success of the other in situ simulation interventions described,^10–28 we feel that this study adds to the understanding of the opportunities with and barriers to in situ simulation. Furthermore, our study is limited by a relatively small number of simulations reviewed (8 in total). There was also considerable subjectivity in our analysis. The classification of events as errors, which were not predetermined, depended on the evaluators’ and participants’ input for identification. In addition, although the National Patient Safety Agency risk assessment matrix is validated, it remains subjective, as evidenced by our relatively high ICC. To combat potential rater bias, we used the mean scores across 3 raters; however, should the raters have the same biases, bias would persist.

Conclusion

This project shows that an in situ simulation program is feasible in a Canadian trauma centre and provides a strategy that simultaneously allows for the identification of latent safety errors, deliberate practice of teamwork and communication skills, and multiple opportunities to improve patient safety.