Is in situ simulation in emergency medicine safe? A scoping review

Abstract

Objectives

To provide an overview of the available evidence regarding the safety of in situ simulation (ISS) in the emergency department (ED).

Design

Scoping review.

Methods

Original articles published before March 2021 were included if they investigated the use of ISS in the field of emergency medicine.

Information sources

MEDLINE, EMBASE, Cochrane and Web of Science.

Results

A total of 4077 records were identified by our search strategy and 2476 abstracts were screened. One hundred and thirty full articles were reviewed and 81 full articles were included. Only 33 studies (40%) assessed safety-related issues, among which 11 chose a safety-related primary outcome. Latent safety threats (LSTs) assessment was conducted in 24 studies (30%) and the cancellation rate was described in 9 studies (11%). The possible negative impact of ISS on real ED patients was assessed in two studies (2.5%), through a questionnaire and not through patient outcomes.

Conclusion

Most studies use ISS for systems-based or education-based applications. Patient safety during ISS is often evaluated in the context of identifying or mitigating LSTs and rarely on the potential impact and risks to patients simultaneously receiving care in the ED. Our scoping review identified knowledge gaps related to the safe conduct of ISS in the ED, which may warrant further investigation.

Strengths and limitations of this study.

• First complete overview of safety of conducting in situ simulation (ISS) in emergency medicine.

• The methodology includes a detailed search strategy, clear inclusion and exclusion criteria, duplicate screening and data extraction by independent reviewers.

• The main limitation is the widespread design variations of the included studies and of the terminology and concepts related to ISS.

• Isolating the impact of ISS from various potential confounding factors remains a major challenge.

• Our study identified major knowledge gaps related to the safe conduct of ISS in emergency medicine, which require further investigation.

Introduction

Emergency medicine (EM) is a complex and challenging specialty requiring the mastery of numerous technical and non-technical skills. Most emergency departments (ED) are overcrowded, chaotic environments with reported rates of errors up to 10%.¹ Regular simulation is needed to establish and maintain skills competency for ED professionals.^{2 3} However, recent literature suggests a bias towards simulations for rare and unexpected cases (cardiac arrest, difficult airway, disaster medicine and/or rare cases) rather than for more common, routine cases.^4–6 The retention rate of acquired knowledge and skills can also be quite low.^{7 8}

In situ simulation (ISS) offers an optimal training solution for these practical EM-related issues.⁹ ISS is defined as simulation taking place in the participant’s everyday work environment.^{10 11} The ED environment is very different from a simulation centre and this can greatly impact the learning process.¹² The value of ISS is predicated on principles from the ‘situated learning theory’ which states that learning is closely linked to the context of the experience.¹² ISS is a ‘point of care’ type of simulation training used to respond to specific local teaching and training needs and, as a result, may enhance patient safety by exposing latent safety threats (LSTs) and trialling mitigation strategies.^{10 13}

As with any intervention (or any clinical research project),^{14 15} there are potential harms linked to ISS including inadvertent use of simulation equipment/medications for real patient care.^16–18 There is a potential for tension between the present and future state of ED care related to the delivery of ISS. During an ISS session, patients in the ED may be at risk of negative impacts because of the misdirection of resources towards the ISS. The potential risks associated with ISS and the published no go criteria seem to have been reached empirically through expert consensus.^16–18 The main objective of this scoping review is to provide an overview of the available evidence regarding the safety of ISS in the ED. Our secondary objective is to explore the benefits of ISS on all levels of Kirkpatrick’s pyramid in EM.^{19 20}

Methods

Design

We followed published guidance for conducting a scoping review.²¹

The results of this scoping review are reported as per the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines.²²

Patient and public involvement

Patients or the public were not involved in the design or the conduct of this study.

Eligibility criteria

We used broad inclusion criteria to present a comprehensive overview of ISS in EM, without any language or date limitations. Original studies were included if they investigated the use of ISS in the ED. Specifically, this comprised studies pertaining to clinician education, system/process evaluation, patient outcomes and patient safety.

We included single-group pretest, non-randomised and randomised studies, parallel-group and cross-over designs, and studies of ‘adjuvant’ instruction in which simulation was added to other instructions common to all learners.

Protocols, commentaries, conference abstracts, posters and correspondences were excluded. This review excludes systematic reviews or meta-analyses and editorial-style reviews. However, those reviews (invited, narrative or systematic), editorials and letters to the editor were scrutinised and citation tracking was conducted to retrieve potentially relevant studies.

Search strategy and information sources

An experienced librarian designed a search strategy (online supplemental eAppendix 1) to search MEDLINE, EMBASE, Cochrane and Web of Science, using keywords specific to each database from inception to January 2020. The search was updated in March 2021.

Since there is no MESH keyword for ‘ISS’, a manual search was performed in our four targeted databases.

Selection process

All identified studies were exported to Endnote V.X9 (Clarivate Analytics, Philadelphia, Pennsylvania, USA) and duplicates were then removed. Four authors (JT, WL, GM and EJ) independently assessed the eligibility of each study. Title, abstract and full text were first screened and then cross-referenced between reviewers. Disagreements were resolved by consensus. Any unresolved disagreement was discussed with a third researcher with experience in the field of simulation studies (ER-D).

Given the nature of a scoping review, risk of bias and quality appraisals were not performed.²¹

Data collection

Four independent reviewers completed data extraction, ensuring a double reviewing for each article.

Outcomes

The main outcome, safety, is defined in this study as the absence of incidents, accidents or a state with the minimal acceptable level of risk. This definition of safety has been used in other simulation studies.^23–25 We selected specific criteria such as wait time, patients leaving without being seen or accident reports, which are based on literature on safety metrics in EM.^{26 27} We explored two safety concepts: ‘ongoing’ safety of patients being managed while ISS was ongoing in the ED (measured by ISS cancellation rate, accident reports, ED median wait time, number of patients who left without being seen) and ‘future patient’ safety with more long-term benefits from ISS (measured by LST evaluation). We sought to explore the impact of ISS on patients’ safety throughout time. Indeed, future patients may benefit from ISS if its output translates into improved skills, clinical care and patient safety.

Secondary outcomes were assessed using the validated Kirkpatrick pyramid, which is a tool used to rank educational interventions on a scale of 1–5 (figure 1).¹⁹ Because of the specificity of ISS, we also distinguished studies that used ISS to teach and train from those that aimed to explore procedures and systems.

Figure 1.

The validated Kirkpatrick pyramid used to rank educational interventions.

Data synthesis

Descriptive statistics and percentages were used to describe our results. No statistical analysis, quantitative meta-analysis or pooling was conducted due to the heterogeneous study designs and methodologies.

Results

Study selection

A total of 4077 records were identified by our initial search strategy. After removing 1601 duplicates, 2476 titles and abstracts were screened. Of those, 2366 did not meet our eligibility criteria. A second search was conducted in March 2021 after which seven original articles were added to our review. A total of 81 full articles were analysed (online supplemental eAppendix 2).

General characteristics of included studies

All reviewed articles were published in English between 2006 and 2021 in specialised journals (81%, n=66).

Most included studies used a prospective design (94%; n=76), 47 of which were observational (58%). Only 3 studies were randomised controlled trials and 28 were pre/post intervention studies (35%). For the most part, the included studies were single centre (80%; n=65) and quantitative (67%; n=54). A high majority included interprofessional participants, defined as >2 professions (84%; n=68).

The clinical topics explored in the studies included ED care (77%; n=62) and extrahospital care settings (such as HEMS, prehospital care, etc) (22%; n=18). Most studies focused on ISS training for advanced life support/basic life support (n=44, 54%). The remainder focused on a variety of clinical topics including trauma (n=35, 43%) and airway management (n=21, 26%). Three studies particularly focused on COVID-19 (4%). Very few studies used consistent scenarios and as a result more detailed thematic analysis was not performed. Ethical approval was obtained for 51 studies (63%). The full characteristics of the included studies are presented in table 1.

Table 1.

General characteristic of studies (n=81)

Study characteristics	n (%)
Study design
Pre/post design	28 (35)
Comparative
≥2 groups	16 (20)
1 group	55 (68)
Prospective	76 (94)
Retrospective	4 (5)
Randomised controlled trial	3 (4)
Single centre	65 (80)
Quantitative	54 (67)
Qualitative	17 (21)
Mixed	9 (11)
Participants
Mean no of participants per study ±SD (min; max)	88.9±66 (4; 398)
Medical students	0 (0)
Residents	4 (4.9)
Emergency department teams	62 (76.5)
Nurses and nursing students	2 (2.5)
Physicians in practice	3 (3.7)
Paramedics	2 (2.5)
Interprofessional	68 (84)
Clinical topics
ALS, BLS	44 (54)
Trauma	35 (43)
Intubation	21 (26)
Peadiatrics	22 (27)
COVID-19	3 (4)
Goal of ISS
to train skills	34 (49)
to assess skills or a procedure	24 (35)
to train and assess	16 (18.5)
NA	7 (9)

ALS, advanced life support; BLS, basic life support; ISS, in situ simulation; NA, not available.

Is ISS safe for patients in EM?

We found that 33 studies (40%) evaluated safety issues, among which 12 (15%) chose a safety-related primary outcome (table 2).

Table 2.

Safety

Satefy-related parameters	n (%)	References
Safety assessement	33 (40)	Amiel et al.28 Abulebda et al.29 Chan et al.31 Fan et al.45 Hamman et al.59 Hamman et al.60 Hargestam et al.61 Hunt et al.44 Kerner et al.62 Kobayashi et al.52 Kobayashi et al.63 Mondrup et al.33 O'Leary et al.64 Patterson et al.10 Patterson et al.65 Petrosoniak et al.30 Shrestha et al.34 Sørensen et al.66 Theilen et al.41 Ullman et al.67 Walsh et al.68 Whitfill et al.69 Zimmermann et al.70 Bradley et al.71 Paltved et al.32 Bredmose et al.35 Couto et al.36 Geis et al.72 Lakissian et al.73 Shah et al.37 Wong et al.74 Petrosoniak et al.75 Aljahany et al.76
Quantitative data such as number of departures without being seen, median waiting time	0
Accident reports	3 (4)	Chan et al.31 Patterson et al.10 Paltved et al.32
Cancellation rate	9 (11)	Chan et al.31 Mondrup et al.33 Patterson et al.10 Shrestha et al.34 Paltved et al.32 Bredmose et al.35 Couto et al.36 Shah et al.37 Auerbach et al.38
LST evaluation	24 (25)	Chan et al.31 Hamman et al.59 Hamman et al.60 Hunt et al.44 Kerner et al.62 Kobayashi et al.52 Kobayashi et al.63 O'Leary et al.64 Patterson et al.10 Shrestha et al.34 Theilen et al.41 Ullman et al.67 Walsh et al.68 Whitfill et al.69 Zimmermann et al.70 Bradley et al.71 Couto et al.36 Geis et al.72 Lakissian et al.73 Shah et al.37 Wong et al.74 Petrosoniak et al.75 Aljahany et al.76 Zern et al.77 Shrestha et al.78 Gray et al.42
Others (survey, qualitative analysis)	5 (6.3)	Amiel et al28 Abulebda et al29 Petrosoniak et al30 Wong et al74 Sørensen et al79

LST, latent safety threat.

Ongoing safety

As for ongoing safety of patients during ISS, none of the included studies assessed the following: (1) The number of patients in ED who left without being seen during the period of the ISS training, or (2) The impact of ISS on patient wait times and (3) Safety-related quality parameters in the ED.

No studies included official institutional accident reports, but three studies included surveys on accident reports.^28–30 No accident was reported during ISS.^{10 31 32} The cancellation rate of ISS was described in nine studies (11%).^{10 31–38} One study included analyses on feasibility (based on department status, such as patient waiting to be seen) in their secondary outcomes. The authors showed that 3 ISS were delayed, and none were cancelled but no other patient-related outcome was presented.³⁹

The impact of ISS on real ED patients was assessed in two studies. In one multicentre clinical trial, ISS was used to prepare and facilitate the identification and mitigation of threats to study participation and patient safety.²⁶ In this case, ISS was not really used to teach and directly improve skills but rather to prepare teams for the implementation of a clinical trial. As an ongoing research project can also jeopardise the safety of other patients by redirecting attention on the study and not on real patients, safety was analysed with parameters such as latent threats identification, mitigation and protocol errors and deviation. Patterson et al’s work on LST identification also addressed the impact of ISS training on patient care,¹⁰ which was assessed using a questionnaire. Four participants out of 118 reported that the simulation was disruptive or affected the participants in a negative way. These are the only studies exploring the impact of ISS on patients being managed in the ED.

Future safety of patients

The LST assessment was conducted in 24 studies (30%).

Kirkpatrick evaluation level

Most studies employed Kirkpatrick-related endpoints (n=75; 92%). Fourteen studies (17%) reported comparison with a preintervention assessment or a control group using knowledge as the outcome (KP1), mostly through a pre/post design. The assessment of KP2 level was provided for 10 studies (12%), and 27 (33%) using ISS as an intervention in EM to assess a level 3 outcome.

Twenty-two studies assessed a level 4 outcome. We considered LST identification as a level 4, because of the direct impact of this methodology on patient care. This classification was decided on consensus among the authors. For instance, in the study by Gray et al (38), the authors showed that the identification of various LSTs regarding massive blood transfusion led to improved patient outcomes (measured by a decrease in the delays between massive haemorrhage protocol activation and blood component administration). We identified three studies with a direct impact on patient outcomes (morbidity, mortality, length of stay). Using a pre–post study design, Steinemann et al demonstrated a significant improvement in mean teamwork scores of real patient care in addition to a 16% reduction in the mean resuscitation time in the ED.⁴⁰ The other study that included a patient outcome analysis examined the effect of introducing a paediatric medical emergency team combined with regular ISS training on patient and system-level outcomes.⁴¹ Even though the decrease in paediatric intensive care unit mortality was nonsignificant, the authors found an association between ISS and a reduction of healthcare-related costs (level 5). Using a prospective observational design, Theilen et al showed that regular ISS training of ED healthcare professionals was associated with a reduction in costs and overall patient mortality. Another more recent study showed direct patient outcome improvement.⁴² The authors showed that a novel ISS-based quality improvement (QI) intervention for blood component administration in bleeding trauma patients led to a 21% mean reduction in time between massive haemorrhage protocol activation and blood component administration. The impact of changing the institution’s massive haemorrhage protocol after the identification of LSTs during ISS was explored in this study using a pre/post design.

No studies showed a negative impact when exploring the improvement of knowledge or skills. The efficacy of ISS was always superior when compared with any another training method or to a control group. We were unable to categorise 10 of our included studies (12.3%) within the 5 levels of KP, either because the studies assessed guideline adherence^{38 43} or because ISS was employed to assess a process or a system (table 3).^{44 45}

Table 3.

Outcomes

Kirkpatrick level	n (%)	References
KP 1: satisfaction, feelings and perceptions	14	Bischof et al.80 Burke et al.81 Couto et al.13 Davison et al.82 Gangadharan et al.83 Gundrosen et al.84 Hunt et al.44 Mannenbach et al.85 Meurling et al.86 O'Leary et al.87 Shrestha et al.34 Sørensen et al.88 Ullman et al.67 Katznelson et al.89
KP 2: knowledge and skills	10	Walsh et al.68 Bradley et al.71 Thomas et al.90 Lemke et al.91 Kalidindi et al.92 Ben-Ari et al.93 Auerbach et al.94 Abualenain et al.95 Bischof et al.80 Sørensen et al.88
KP 3: behaviours	27	Abu-Sultaneh et al.96 Amiel et al.28 Armstrong et al.97 Auerbach et al.38 Barker et al.98 Bayouth et al.99 Bredmose et al.35 Campbell et al.100 Coggins et al.101 Farah et al.102 Generoso et al.103 Jörgens et al.104 Jung et al.105 Katznelson et al.106 Lakissian et al.73 Miller et al.107 O'Leary et al.87 Patterson et al.65 Petrosoniak et al.30 Pirie et al.108 Qian 109et al Saqe-Rockoff et al.110 Truta et al.111 Wong et al.74 Zern et al.77 Zimmermann et al.70 Walsh et al.68
KP4: patient outcome	22	Abulebda et al.29 Aljahany et al.76 Barni et al.112 Chan et al.31 Couto et al.36 Geis et al.72 Gray et al.42 Hamman et al.59 Hamman et al.60 Hargestam et al.61 Kerner RL et al.62 Kobayashi et al.63 Lavelle et al.113 Paltved et al.32 Patterson et al.10 Shah et al.37 Shrestha et al.34 Steinemann et al.40 Wang et al.114 Whitfill et al.69 Petrosoniak et al.75 Theilen et al.41 Walsh et al.68
KP5: costs	2	Theilen et al.41 Petrosoniak et al.75
Other, NA	10	Hunt et al.44 Kobayashi et al.52 Mondrup et al.33 Kessler et al.115 Siegel et al.43 Stevens et al.116 Wieck et al.117 Nadkarni et al.118 Auerbach et al.119 O'Leary et al.87

NA, not available.

Indeed, for 24 studies (29%), ISS was used to explore and assess a procedure, an organisation, or guideline adherence and not to teach or train. Seven studies (8.6%) assessed the feasibility of ISS itself without exploring learning or safety outcomes.

Discussion

This scoping review assessed the safety of ISS in EM.

ISS and patient safety

Conducting ISS within the chaotic and busy ED environment may raise concerns for the ongoing care of real patients. However, we found that few studies evaluated the impact of ISS on the safety of actual ED patients. Our scoping review focused on the available evidence regarding two safety concepts: ongoing ED patient safety during ISS, and the safety of future patients (long-term benefits of ISS). It is important for educators and ED professionals to be aware of the associated risks of ISS. Specific, rigorous guidelines are needed to help create a framework aiming to enhance ISS ‘ongoing’ safety. Surprisingly, we did not find any studies evaluating the impact of ISS on ongoing patient care, clinical adverse events, wait times or patients leaving without being seen. Some editorials warn against these risks,^16–18 but we found no studies exploring nor demonstrating the potential consequences or risks of ISS in a busy ED for the ongoing care of patients. Most studies seemingly overvalue the ‘future’ safety of patients while undervaluing the ‘ongoing’ safety of ED patients during ISS. This seems crucial because ISS is increasingly used to teach and train EM professionals. As for any method aiming at improving the quality of care, the risks for patients should be controlled. In that sense, any type of clinical research may jeopardise patient safety. For example, the time it takes to consent and enrol a patient in a therapeutic trial could impact ED safety measures. The time-sensitive nature of EM research has been reported in research primers.¹⁴ Our scoping review shows the absence of scientific evidence from studies with rigorous methodology to confirm the absence of risks for patients. The Foundation for Healthcare Simulation Safety published a 10-item ‘pledge’ of ‘best practices’ for simulation programmes to reduce simulation-related hazards.^{17 46} However, even if those principles seem logical, they are not evidence based.

Educators and simulation teams should consider guidelines from existing literature on QI to ensure the sustainability of these interventions. In the case of ISS, the ongoing safety of patients could be referred to as a ‘balancing measure’. In the QI literature, balancing measures represent checkpoints that ensure there are no potential unintended consequences (risk to ongoing patient safety) resulting from an intervention (ISS).^{37 47–51}

Patient-reported measures, which appear to be underutilised based on our study results, represent a useful lens to understand the value of ISS more comprehensively. For example, at one investigator’s institution (AP), a Code Orange simulation during which dedicated volunteers checked in with patients waiting and patients had positive perceptions regarding this ongoing training (unpublished data).

Outcomes

One would think that being closer to the patient and to the patient care facility would make it easier and more intuitive to assess ISS-related patient outcomes, but only two studies assessed direct patient outcomes.^{41 42} This raises the question of the relevance of the Kirkpatrick pyramid. Indeed, we deemed appropriate to classify the identification of LST as a level 4 on the KP scale, because it is directly linked to improving the quality of care. This decision was reached on consensus among the authors and is debatable because direct patient outcomes are not measured. However, we found data exploring the improvement of care following the identification of LSTs to validate our choice of classification.^{41 42} We believe even though this was not directly measured in every study, the identification of LSTs leads to improved patient outcomes by reducing medical errors.⁵² We were unable to classify a great number of studies using this tool. As other authors have previously suggested, it may be wise to adopt a new lexicon and new endpoints to assess the impact of ISS.^{53 54} Some studies did, however, explore level KP4 and even KP5.⁴¹ Their inspiring results should encourage researchers to pursue their efforts and lead projects aiming at exploring higher levels of the KP pyramid.

Study methodology limitations

As highlighted in previous studies, studies exploring the impact of simulation-based interventions are still needed.^{2 55–57} We observed that even though the included studies were quite recent, as ISS seems to have emerged in the last 20 years, the vast majority used observational prospective designs. With ISS, the proximity between the intervention and the possible patient-centred outcomes is inspiring and educators should seek collaboration with methodologists to elaborate research protocols leaning to prove the translational dimension of ISS. Simulation should be about improving patient outcomes, and the causative effect of isolating ISS should be a priority.

Strengths and limitations

Because there is no mesh word for ‘ISS’ we conducted a search on simulation and EM followed by more detailed reviews for ‘in situ’ specific articles, which made the screening process more complex. Furthermore, the terms used to describe ISS are highly heterogeneous. It is possible that this resulted in missed articles that otherwise should have been included. However, the corollary is this process required a close assessment of each article resulting in a more thorough review.

Also, several systematic reviews were excluded from our analysis. However, we reviewed and cross-referenced their reference list against our literature search. We found no discrepancies, supporting the accuracy of our approach.

Conclusion

Our scoping review identified major knowledge gaps related to the safe delivery of ISS in the ED, which require further investigation. Patient safety during ISS is often evaluated in the context of identifying or mitigating LSTs with little focus on the potential impact and risks to actual ED patients. For all these reasons, our research team sought to address this issue in a mix-method study, which will aim at demonstrating the safety and impact of ISS.⁵⁸

Data availability statement

No data are available.

Ethics statements

Patient consent for publication

Not applicable.

Ethics approval

Not applicable.