In situ simulation and its effects on patient outcomes: a systematic review

Abstract

Background

The use of in situ simulation has previously been shown to increase confidence, teamwork and practical skills of trained professionals. However, a direct benefit to patient outcomes has not been sufficiently explored. This review focuses on the effect of in situ simulation training in a hospital setting on morbidity or mortality.

Methods

A combined search was conducted in PUBMED, OVID, WEB OF SCIENCE, CINAHL, SCOPUS and EMBASE. 478 studies were screened with nine articles published between 2011 and 2017 meeting the inclusion criteria for analysis.

Results

This review selected eight prospective studies and one prospective-retrospective study. Three studies isolated in situ simulation as an experimental variable while the remaining studies implemented in situ programmes as a component of larger quality improvement initiatives. Seven studies demonstrated a significant improvement in morbidity and/or mortality outcomes following integrated in situ simulation training.

Conclusion

Existing literature, albeit limited, demonstrates that in situ training improves patient outcomes either in isolation or within a larger quality improvement programme. However, existing evidence contains difficulties such as isolating the impact of in situ training from various potential confounding factors and potential for publication bias.

Introduction

Simulation-based education has been an increasingly applied teaching method with a growing body of supporting evidence.¹ There are a variety of techniques, tools and methodologies that can be used within simulation-based education.² One such modality, in situ simulation-based training, is defined as simulated encounters in the exact setting where they are expected to occur, as opposed to dedicated simulation labs or centres. In situ simulations create an encounter within the real working environment. It is particularly relevant to train teams and/or individuals within an institution. More so, employing in situ simulation may mitigate challenges in constructing a simulation centre by negating the need for dedicated physical space other than storage of the equipment. In the context of healthcare, this has been widely incorporated into education programming ranging from early degree programmes to continuing medical education for various providers including physicians, dentists, nurses and physician assistants.³ In situ simulations are often used when training for medical emergencies or ‘codes’, and correspondingly may be referred to as a mock code.⁴ Another name for in situ simulations is point-of-care simulations, meaning its scope is not limited to medical emergencies and can be used within the context of other healthcare responses.⁵

In situ simulation-based training may be an important training modality to improve care delivery in high-pressure situations that demand the coordination of many healthcare providers, actions and resources.⁶ While it has been demonstrated that knowledge and technical skills are vital to providing competent emergency care, non-technical skills referred to as crisis resource management have also been shown to affect overall outcome, and are improved through in situ simulation training.^7–10

Studies have described in situ training improving process measures of care delivery. Process measures refer to properties of the care response itself such as the number of errors, time to intervention and healthcare provider comfort, rather than outcomes of the response.¹¹ Conversely, literature reporting the direct impact of in situ simulation on patient outcomes is scarce.¹² The definition of ‘patient outcomes’ in literature is broad but the concepts of morbidity and mortality can be used. They are simplistic in their scope yet are a fundamental starting point when assessing the efficacy of a clinical or educational intervention.¹³ For example, in the context of cardiopulmonary resuscitation, while certain metrics such as time to compressions and time to defibrillation are established surrogate endpoints to code blue response efficacy, it cannot be assumed that the combination of these process measures leads to improved patient outcomes. In one study investigating the outcomes of cardiopulmonary arrest for 290 patients, 95 were successfully resuscitated; however, only 35 were alive at discharge and may have had significant adverse outcomes that were unreported.¹⁴ This trend holds true in similar studies reporting that only 15%–35% of in-hospital arrests survive to discharge.^{15 16} Process measures for emergency responses are more closely associated with individual resuscitation success, but less so with subacute and long-term patient outcomes such as survival-to-discharge and morbidity. Overall, process measures of in situ simulation are not a substitution for assessing patient outcomes directly.

Previous reviews have shown that simulation-based education has positive effects on patient outcomes but have not demonstrated these findings with in situ simulation-based training in particular.^{17 18} Other reviews acknowledged the need to further investigate in situ simulation as an educational modality to improve patient morbidity and mortality.^{3 12 19} However, these studies focused on all levels of Kirkpatrick’s model, looking at both integration of knowledge and measures that result in changes in patient outcomes.²⁰ We felt a more robust review of the up-to-date literature is warranted to identify the impact of in situ simulation training on direct patient outcomes.

Methods

Aim

This systematic review seeks to determine if there is evidence in the literature of in situ simulation training having a correlation with patient outcomes in the domains of morbidity and mortality.

Study identification

Six independent online databases were used in this review: PUBMED, OVID, WEB OF SCIENCE, CINAHL, SCOPUS and EMBASE. The major keywords in all searches were: in situ, simulation and patient outcomes. The ‘in situ’ keyword was combined with its relevant synonyms for the complete search query: ((in situ OR mock code OR point of care) AND (simulation) AND (patient outcome)). With PUBMED, OVID and CINAHL, relevant MeSH terms were included within the search query with an OR operation between each major keyword. The separation of keywords ‘in situ’ and ‘simulation’ provided an extra catchment net due to the possible different uses of the expression within relevant studies. The time frame ranged from inception until April 2018. Two independent librarians were recruited to help construct the search query. They were acknowledged but did not meet the criteria for authorship.

Abstracts identified within each search were imported into Covidence,²¹ a systematic review data management software, and duplicates were removed. Articles were screened independently by two reviewers (DG and CK) with conflicts being resolved through discussion, consensus and input from a third reviewer (SP).

Study eligibility

The abstract screening process followed the predetermined inclusion criteria: (1) conducted simulations were explicitly in situ; (2) subjects of the studies must have included healthcare providers or trainees; and (3) studies used direct patient outcomes such as morbidity or mortality as a metric of analysis (result in Kirkpatrick level 4²⁰ or improvement in 7I framework²²).

Studies that were not chosen matched the following exclusion criteria: (1) simulation was not used as an intervention; (2) simulation used was not in situ; (3) dependent variable of the study was not a direct patient outcome (see above) but rather some sort of indirect measure such as latent safety threats or teamwork management; (4) studies published in languages other than English; (5) secondary research studies such as review articles or systematic reviews, although the authors reviewed references for any missed primary studies; (6) studies from non-peer-reviewed articles, magazine and newsletter publications or online publications, abstracts and conference proceedings, and so-called ‘grey literature’. Abstracts and conference proceedings were excluded due to limited extractable information. The population of interest were licensed healthcare professionals and trainees, including nurses, physicians and students. All patient populations were considered. The primary intervention of interest was the implementation of in situ simulation training alone or as a component of a robust training programme. All primary study designs were included. All years were considered.

Article review process

Forty-five articles then underwent full-text screening resulting in 37 articles being removed due to ambiguous simulation protocols (not clarified whether in situ or at simulation lab or centre), non-English full text, unrelated outcomes and exclusive poster presentations. One primary study which was found while reviewing references from an excluded secondary review article was also included. This resulted in a total of nine studies as seen in figure 1.

Figure 1.

Search and selection of included studies.

Quality assessment

During the full-text screening, risk of bias was assessed independently by two reviewers (DG and CK) based on the existing Risk of Bias in Non-randomised Studies of Interventions tool for assessing the risk of bias of interventional, non-randomised studies.²³ Any conflicts were discussed and resolved. Risk of bias was determined to be low, moderate or serious depending on factors such as confounding variables, selection criteria, intervention classification and completeness in data reporting.

Data extraction

The extraction of the nine articles was conducted independently by the two reviewers (DG and CK) and verified by a third reviewer (SD) using the Covidence software. The extraction process focused on team composition, choice of outcome, intervention length and data analysis methodology.

Data synthesis

Due to the potential variable study designs and methodologies (heterogeneity), a quantitative meta-analysis was not performed. Instead, a narrative synthesis analysis that evaluated effectiveness was conducted.^{24 25} Our theoretical model was based on the hypothesis that in situ simulation does affect patient outcomes through providing unique opportunities for training in the real clinical environment. Primarily we wanted to see if there is literature evidence for a change in direct patient outcomes through this intervention. A preliminary synthesis then followed through an initial description of included studies. Afterwards, the studies were compared and contrasted based on the characteristics, findings and relationships between the variables. Finally, the robustness and methodological quality of the studies and synthesis is assessing with respect to their limitations and biases.

Results

Study characteristics

Of the nine selected studies,^26–34 seven were conducted in the USA,26–30 32 34 one in the UK³³ and one in India.³¹ All studies used some variation of a prospective cohort approach, with one study incorporating a retrospective analysis as well.³¹ The publishing dates range from 2011 to 2017. The spectrum of clinical scenarios for which in situ simulation was performed included code emergencies (n=6),26 27 29 31 33 34 detection of in-hospital deterioration (n=1)³⁰ and obstetrical emergencies (n=2).^{28 32} The patient populations consisted of paediatric (n=4),26 29 33 34 obstetric (n=2)^{28 32} and adult inpatients (n=3).^{27 30 31}

Study methodologies

The selected studies used different methodological approaches to assess the effectiveness of their variable patient safety interventions (table 1). Only three studies isolated in situ simulation as the primary intervention at onset,^{26 27 34} while an additional study temporally separated a multitude of interventions, including in situ simulation, to assess for individual effects.³² The majority of studies (n=5) incorporated in situ simulation as part of a broad educational intervention.^{28–31 33} Seven studies were single arm,26 27 29 31–34 and seven studies used a clearly defined preintervention baseline period for statistical comparisons.^27–33 Two studies used other hospital inpatient units as the control group.^{28 30} Four of the selected studies had intervention periods of 1 year or less,27 29 30 34 with the remaining five studies implementing intervention periods lasting between 2 and 6 years.26 28 31–33

Table 1.

Summaries of included studies

Study	Summary
Andreatta et al 26	Investigated the role of mock codes in paediatric cardiopulmonary arrest (CPA) survival rates. Mock codes were held monthly at random times during Monday through Friday day shifts, increasing in number as the study progressed. All regular code team members were involved including medical residents, paediatric intensive care unit nurses, medical students, hospitalists and pharmacists. High-fidelity manikins were used in either the clinical simulation centre or a functional paediatric patient room. Scenarios ranged from sepsis, respiratory distress, increased intracranial pressure/herniation, anaphylactic shock and cardiogenic shock. Each code encompassed one or more scenarios with initial emphasis on pulseless rhythms in year 1, followed by rhythms with a pulse in year 2 and a composite in years 3 and 4. Debriefing events led by trained clinical faculty followed the mock codes where video recordings were used. The chosen outcome of interest was survival rates, defined as a patient survival-to-discharge percentage. CPA survival rates increased during the first year and slightly increased again for the last 2 years of the study.
Steinemann et al 27	Examined the effect of a novel 4-hour team training curriculum, consisting of online teaching, multiple in situ simulation exercises and simulation debriefs. Participants consisted of all healthcare professionals involved in an acute resuscitation, including staff physicians, residents, nurses, respiratory therapists and emergency department technicians. Outcomes of interest were primarily skills based; however, this study also evaluated the effect of the simulation-based training curriculum on patient survivability after intervention, reporting a non-significant impact.
Riley et al 28	Investigated the effects of didactic training alone or in combination with in situ simulation on adverse prenatal outcomes. Three hospitals were randomly assigned to either have no intervention (control), TeamSTEPPS didactic training, or in situ simulation integrated with TeamSTEPPS training (full intervention). Participants included all the labour and delivery staff. Eleven in situ simulations were conducted across half a year after didactic training in the third hospital. Outcomes included the Weighted Adverse Outcome Score which decreased in the full-intervention hospital, stayed the same in the didactic-only hospital and increased in the control hospital.
Knight et al 29	Examined the clinical effects of a composite resuscitation team training programme. The training included in situ simulation along with other interventions. The preintervention baseline period was 4 years with an intervention period of 1 year. The study demonstrated an improvement in the primary outcome which was survival-to-discharge following a CPA. Exclusion criteria included events which used extracorporeal membrane oxygenation (ECMO) at code initiation.
Braddock et al 30	Investigated the effect of a multifaceted patient safety programme on clinical outcomes. Baseline measurements were conducted over 1 year followed by a 1-year intervention period and a 6-month sustainability period. The intervention comprised in situ simulation training adjunctive to other interventions. In situ simulations were conducted four times per month during the intervention period and monthly during the sustainability period on both day and night shifts. Scenarios were designed to mimic clinical states preceding acute deterioration where both technical and non-technical skills were emphasised. Measured outcomes included hospital-acquired severe sepsis/septic shock, acute respiratory failure, rate of unplanned transfers to higher level of care (HLOC) and weighted risk adjusted observed to expected mortality ratio. All outcomes except for the rate of unplanned transfers to HLOC improved significantly when compared with both the baseline period and to control hospital units which did not receive the training programme.
Sodhi et al 31	Investigated cardiopulmonary resuscitation outcomes with interventions including mock codes along with other optimisation protocols. The mock codes were conducted at least twice a year in different departments around the hospital. The study demonstrated improvement in the rate of return of spontaneous circulation (ROSC) as well as survival-to-discharge.
Riley et al 32	Analysed the effect of a quality improvement collaborative on prenatal outcomes from 14 hospitals. This initiative included a 2-year baseline followed by a 5-year intervention period which encompassed three primary interventions: a standardised care process, teamwork training through in situ simulation, and education and performance feedback. Phase 1 introduced education and performance feedback as well as a standardised care process over 3 years. It was followed by phase 2 which occurred during the last 2 years of the study and included in situ training. This study demonstrated a decrease in the adverse outcome index after intervention.
Theilen et al 33	Examined the effect of introducing a paediatric medical emergency team (PMET) coupled with weekly in situ simulation training on hospital outcomes. Non-PMET staff, registrars and senior nurses from all hospital wards were also included in this training programme as a hospital-wide initiative. The study demonstrated an insignificant decrease in paediatric intensive care unit (PICU) mortality yet showed a significant decrease in hospital-wide deaths, which was not a predetermined outcome.
Gibbs et al 34	Implemented and investigated the effect of an in situ simulation programme to combat a methicillin-resistant Staphylococcus aureus (MRSA) outbreak in a level 4 neonatal intensive care unit. Physicians, nurses, respiratory therapists and environmental service workers completed the training programme which incorporated 30 min in situ simulations along with debriefing. The main educational principles of interest were proper techniques of personal protective equipment, hand hygiene, handling potentially contaminated materials and entering/existing infected rooms. This study demonstrated a significant decrease in the number of infections.

The frequencies of in situ simulation varied, ranging from only once (n=2),^{27 34} to weekly (n=2),^{30 33} biweekly (n=1),²⁸ monthly (n=2)^{26 29} and biannually (n=1).³¹ One study did not specify the number of in situ simulations performed.³² Debriefing following an in situ simulation exercise was performed in eight studies, using either video (n=4)26 27 29 32 or non-video (n=4) modalities.28 30 33 34

Team characteristics

All the selected studies included both nurses and physicians in the in situ simulation intervention. Eight studies26–29 31–34 included the entire emergency code team with only one study focusing exclusively on Postgraduate Year 1 residents and nurses.³⁰ For all studies, turnover rates for participating teams were not clearly reported. Only one study specified the team size and individual composition.³³ As well, there were varying reporting methodologies for participation rates as it was generally unclear if participants were involved in one or several in situ simulation exercises.

Outcome metrics

Five of the selected studies used mortality metrics as the primary measured outcome.26 27 29 31 33 The remaining four studies chose specific morbidity metrics including sepsis, septic shock and acute respiratory failure (n=1),³⁰ methicillin-resistant Staphylococcus aureus infection (n=1)³⁴ and perinatal adverse events (n=2).^{28 32} In seven of the selected studies, there was a significant improvement in the outcomes of interest.26 28–32 34 One study found no statistically significant improvement in mortality.²⁷ Another study did appreciate a significant improvement in hospital-wide mortality, but the primary scope of the study (paediatric intensive care unit) showed insignificant results.³³ Table 2 outlines the intervention and outcome characteristics of selected studies, as well as the results as reported.

Table 2.

Included studies with reported interventions and outcomes

Study	Design	Intervention	Investigated outcome	Baseline measurement	Postintervention measurement	P value	OR	95% CI
Andreatta et al 26	A 4-year prospective research study	In situ simulations	Paediatric CPA survival rate	33%	56%	0.000	–	–
Steinemann et al 27	6-month prospective study	In situ simulations	Absolute number of mortalities	8	4	Reported as NS, p>0.05	0.67	–
			Mean hospital LOS days (survivors)	5.1	3.4	Reported as NS, p>0.05	–	–
			Mean ICU days (survivors)	1.9	0.3	Reported as NS, p>0.05	–	–
Riley et al 28	A 4-year prospective study	TeamSTEPPS didactic training alone or in combination with in situ simulation	Weighted Adverse Outcome Index	1.15 (full intervention)	0.72	P<0.05	–	–
				1.46 (didactic only)	1.45	P>0.05	–	–
				1.05 (control)	1.5	P>0.05	–	–
Knight et al 29	A 5-year prospective observational study	Define and teach institution-specific code roles and responsibilities. Familiarisation with and training on emergency equipment for resuscitation team. Monthly in situ high-fidelity cardiac arrest simulation with video debriefing. BLS for pharmacists and security. ACLS for paediatric residents (PALS), hospitalists, PICU fellows and attendings in-house pharmacists. Lab-based code blue simulation for acute care and ICU nurses	Paediatric CPA survival rate	40.3%	60.9%	–	2.30	1.15 to 4.60
Braddock et al 30	A 1-year prospective interventional study was conducted, followed by a 6-month sustainability phase	In situ simulation training. Debriefing of medical emergencies. Monthly patient safety team meetings. Patient safety champion role. Interdisciplinary patient safety conferences. Recognition programme for exemplary teamwork.	Hospital-acquired severe sepsis/septic shock complications per 1000 discharges	1.78	1.21 and 0.64 (intervention and sustain period)	0.04	0.53	0.29 to 0.96
			Acute respiratory failure complications per 1000 discharges	2.44	2.10 and 0.43	0.03	0.58	0.35 to 0.96
			Number of unplanned transfers to higher level of care (HLOC)	715	763 and 764	0.08	1.27	0.97 to 1.66
			Weighted risk-adjusted mortality observed-to-expected ratio	0.5	0.44 and 0.40	<0.001	0.95	0.94 to 0.97
Sodhi et al 31	A 6-year prospective and retrospective observational study	ACLS and BLS refresher courses. In situ simulation drills. Optimisation of the environment (eg, crash carts placed in each ward, etc). Code blue team restructuring. ACLS and BLS algorithms placed on crash carts and critical areas. Modification of code blue sheets for better documentation and postanalysis. More efficient public address system. Formulation of postresuscitation care protocol.	Return of spontaneous circulation (ROSC)	26.7%	40.8%	<0.05	–	–
			Survival-to-discharge rate	23.4%	66.6%	<0.05	–	–
Riley et al 32	A 7-year prospective study	Standardisation of evidence-based care. Interdisciplinary teamwork training through in situ mock codes. Routine education with performance feedback.	Adverse Outcome Index	0.055	0.047	0.32	–	–
			Weighted Adverse Outcome Index	1.192	1.081	0.1	–	–
			Severity Index	21.88	22.62	0.46	–	–
			Patient Safety Index 17	0.0019	0.0016	0.163	–	–
Theilen et al 33	Prospective cohort over three 1-year periods	Introduction of PMET team. Weekly team training with mock codes (for registrars and senior nurses as well)	Absolute number of PICU mortalities	7	2 and 2 (intervention and sustain period)	0.3	–	–
			Number of PICU bed days	527	336 and 196	<0.001	–	–
			Hospital deaths per 1000 admissions	2.9	1.4	<0.001	–	–
Gibbs et al 34	6-month prospective study	In situ simulations	Absolute number of MRSA-infected infants	18	0	–	–	–

ACLS, Advanced Cardiac Life Support; BLS, Basic Life Support; CPA, cardiopulmonary arrest; ICU, intensive care unit; LOS, length of stay; MRSA, methicillin-resistant Staphylococcus aureus; NS, not significant; PALS, Paediatric Advanced Life Support; PICU, paediatric intensive care unit; PMET, paediatric medical emergency team.

Quality and bias

Table 3 outlines the assessment of each study related to quality and bias. The most prevalent methodological limitation was in situ simulation not being isolated (n=5).^{28–31 33}

Table 3.

Included studies with limitations and risk of bias

Study	Risk of bias	Limitation/methodological issues
Andreatta et al 26	Moderate	Absence of baseline period for comparison. Participant composition not fully elucidated. Presence of confounding variables.
Steinemann et al 27	Moderate	Though number of involved participants recorded, no mention of participant retention during study duration (1 year).
Riley et al 28	Moderate	Presence of confounding variables (in situ simulation not isolated). Arms of study differed at baseline.
Knight et al 29	Moderate	Participant composition not fully elucidated. Presence of confounding variables (in situ simulation not isolated).
Braddock et al 30	Moderate	Non-intervention control group differed from intervention group. Presence of confounding variables (in situ simulation not isolated).
Sodhi et al 31	Moderate	Presence of confounding variables (in situ simulation not isolated). Participant composition not fully elucidated.
Riley et al 32	Serious	Presence of confounding variables (in situ simulation not isolated). Potential outcome selection bias due to reporting on four of the eight initial outcomes. Potential bias due to influence of private third-party entity. Participant composition not fully elucidated.
Theilen et al 33	Moderate	Presence of confounding variables (in situ simulation not isolated).
Gibbs et al 34	Serious	Potential outcome selection bias due to reporting on non-initial outcome. Results affected by confounding factors (related to disease outbreak dynamics). Participant composition not fully elucidated. Small sample size. Team characteristics not fully elucidated.

Discussion

Summary

The research question posited in this systematic review is whether in situ simulation affects patient morbidity and mortality. The choice of selecting direct outcomes was made to limit the review to the most impactful studies. Only nine studies met the criteria to be included in this review. This relatively small number of studies demonstrates a tendency in literature to focus on the acquisition and improvement of skills and process measures as surrogate markers for improved clinical competency rather than patient outcomes. While conducting this review, articles reporting on these surrogate endpoints were abundant. This finding is congruent with other systematic reviews published in the last several years.9 12 17 18 Recently, however, an increasing number of studies have been published that directly demonstrate the positive impact of in situ simulation training on patient morbidity and mortality. Furthermore, the in situ element of simulation training is a much less explored factor when assessing traditional high-fidelity simulation approaches, contributing to a narrow scope of captured studies.

Limitations of selected studies

Causative effect of isolating in situ simulation

We included all studies that had in situ simulation as an intervention either independently or part of a complex training programme. As simulation-based interventions become more common and training programmes more sophisticated, it is becoming increasingly challenging to isolate the impact of the in situ simulation training as an independent intervention. Moreover, in one study with in situ simulation temporally isolated, while a visual decrease in mortality was seen in the provided figure, no statistical analysis was offered with respect to the isolation of the in situ simulation after introduction of other interventions.³² Thus, while in situ simulation was isolated temporally, its statistical significance is unknown.

In practice, in situ simulation will most likely not be implemented independently and will be a component of a complex training programme. What is critical is that training interventions that include an in situ component have been shown to lead to improved patient morbidity and mortality.

Study methodology limitations

All but two of the selected studies used a single-arm prospective approach that chose a specific baseline period for comparison with the intervention results. While perhaps simpler to develop and implement a single-arm study design, the absence of a control group and subsequent population randomisation limits the validity of the findings. Braddock et al investigated in situ simulation as part of a training intervention in a single hospital department, using other hospital wards as the control.³⁰ However, not all hospital wards are equal, thus this may introduce further confounders. Riley et al randomly assigned three hospitals as a control, didactic only, or didactic along with in situ simulation.²⁸ Similarly, not all hospitals are equal which introduced cofounders. Despite these limitations, this is a prudent first step to increasing the validity of the results. However, further efforts should include randomisation of the study population and larger sample sizes to reduce confounding variables such as secular trends in outcomes.³⁵

The dilemma of study methodology can be traced back to the chosen population of interest. Studies that focused on the code team are hindered by issues such as limited sample size, rotating teams and inability to create control groups. Often the number of in situ simulations is reported, but it is unclear if the same people are participating in each one. The ultimate effect of improved patient outcomes is difficult to attribute to any factor, instead correlations are drawn to an abstract general change in safety culture or safety awareness. A greater focus on documentation of individual participation and team change-over may increase transparency of the effectiveness of the in situ simulation intervention.

Interdisciplinary practices

All selected studies used an interprofessional approach to training which highlights the importance of investing in whole-team training. This is beneficial as in situ training mirrors real-life emergency codes which are attended by a rotating interdisciplinary team. Teams must rely on each member’s technical and non-technical skills to succeed; therefore, training is imperative for all members. Moreover, one study demonstrated that by inviting non-core team members to in situ training, a decrease in general hospital mortality was demonstrated.³³ A possible explanation for this was that general ward nurses and medical trainees who participated in the in situ training brought their educational experiences back to other hospital units. Though only demonstrated by a single study, by expanding training interventions department-wide or even hospital-wide beyond the code team, patient outcomes can be further influenced.

Limitations of current review

This systematic review is limited by exclusion of non-English articles and being unable to contact researchers for additional information. Our search criteria were narrow by design to capture only the studies about in situ simulation deemed most impactful with relation to patient outcomes, which may have limited the scope of our search. A larger limitation of this review is possible publication bias since no studies captured in the search reported negative results. This could suggest the search criteria are too narrow, by focusing entirely on simulations with an in situ component, that few studies are investigating morbidity and mortality as an outcome, or that studies with negative or neutral results are not published. As well, the majority of the selected studies are single-arm pre-post intervention studies, thus the risk of bias is relatively high. Long-term trends are another limitation involved in the before-and-after studies we analysed. For example, it has been shown that a hospital without a paediatric medical emergency team has decreased mortality over time, suggesting that multiple interventions or secular trends in outcomes of cardiac arrest can account for this decrease.³⁵

Recommendations for future studies

Current literature tends to investigate in situ simulation independent of patient outcomes, or as a part of a larger educational initiative. However, the relationship between the two is rarely investigated in isolation. To better explore the potential impact of in situ simulation, future studies could implement protocols that effectively isolate its impact on patient outcomes. A prospective study in an isolated unit such as an emergency department can be used to compare patient outcomes before and after the implementation of an in situ training programme. A resource intensive method could be implemented where all potential members of the response team receive in situ training, or instead implement an intent-to-treat approach where specific members (eg, full-time staff and code leaders only) receive training. The emergency department is a strong candidate for this small-scale trial since it often acts as its own code response team (ie, less responders that require in situ training). This smaller scale pilot project could provide data for cost-benefit analysis of an institution-wide initiative for all code responders. To reduce confounding factors, sites can be chosen that have demonstrated relatively consistent code outcomes over a retrospective time period that is equal to or longer than the planned prospective study period. For example, a 3-year prospective in situ study should take place at an institution where code outcomes have remained consistent for the prior 3 years. Additionally, patient outcomes from other institutions without in situ training can be monitored for relative changes. This would reflect and account for confounding factors such as technological and research advancements. Another study could include participants randomised into either the in situ or traditional arms will elucidate the specific impact of in situ training. Overall, a well thought-out prospective study design specifically focused on evaluating the effect of in situ training on patient outcomes is required to truly reveal its impact; however, paucity in current literature persists as the relationship between the two is rarely the sole outcome being investigated.

Conclusion

The present review discovered evidence in the literature that incorporation of in situ simulation training is statistically correlated with improved patient morbidity and mortality. However, supporting evidence remains limited by the number of studies and an array of confounding factors to grasp the true validity of the findings. To determine the true impact of in situ simulation either independently or as part of a larger training programme, future research should make use of more isolating protocols with fewer confounding factors.