Simulation-based multiprofessional obstetric anaesthesia training conducted in situ versus off-site leads to similar individual and team outcomes: a randomised educational trial

Abstract

Objective

To investigate the effect of in situ simulation (ISS) versus off-site simulation (OSS) on knowledge, patient safety attitude, stress, motivation, perceptions of simulation, team performance and organisational impact.

Design

Investigator-initiated single-centre randomised superiority educational trial.

Setting

Obstetrics and anaesthesiology departments, Rigshospitalet, University of Copenhagen, Denmark.

Participants

100 participants in teams of 10, comprising midwives, specialised midwives, auxiliary nurses, nurse anaesthetists, operating theatre nurses, and consultant doctors and trainees in obstetrics and anaesthesiology.

Interventions

Two multiprofessional simulations (clinical management of an emergency caesarean section and a postpartum haemorrhage scenario) were conducted in teams of 10 in the ISS versus the OSS setting.

Primary outcome

Knowledge assessed by a multiple choice question test.

Exploratory outcomes

Individual outcomes: scores on the Safety Attitudes Questionnaire, stress measurements (State-Trait Anxiety Inventory, cognitive appraisal and salivary cortisol), Intrinsic Motivation Inventory and perceptions of simulations. Team outcome: video assessment of team performance. Organisational impact: suggestions for organisational changes.

Results

The trial was conducted from April to June 2013. No differences between the two groups were found for the multiple choice question test, patient safety attitude, stress measurements, motivation or the evaluation of the simulations. The participants in the ISS group scored the authenticity of the simulation significantly higher than did the participants in the OSS group. Expert video assessment of team performance showed no differences between the ISS versus the OSS group. The ISS group provided more ideas and suggestions for changes at the organisational level.

Conclusions

In this randomised trial, no significant differences were found regarding knowledge, patient safety attitude, motivation or stress measurements when comparing ISS versus OSS. Although participant perception of the authenticity of ISS versus OSS differed significantly, there were no differences in other outcomes between the groups except that the ISS group generated more suggestions for organisational changes.

Trial registration number

NCT01792674.

Strengths and limitations of this study.

• To the best of our knowledge, this is the first randomised trial conducted to assess the effects of two different simulation settings, in situ simulation versus off-site simulation, on a broad variety of outcomes.

• Previous non-randomised studies have recommended in situ simulation. However, in this randomised trial, no significant differences were found regarding knowledge, patient safety attitude, stress measurements, motivation or team performance when comparing in situ simulation versus off-site simulation. The participants in the in situ group scored the authenticity of the simulation significantly higher than did the participants in the off-site simulation group. However, this perception did not influence the individual and team outcomes. On the outcome on the organisational level, the in situ group generated more suggestions for organisational changes.

• A strength of this trial is the involvement of authentic teams that mirrored teams in real life that resembles the real clinical setting in every possible way. This seem to be important for the so-called sociological fidelity.

• A limitation of the trial is the fact that the outcomes were based only on immediate measurements of knowledge level and of team performance. Only perceptions of simulation were measured after 1 week (evaluation and motivation) and safety attitudes after 1 month. No clinical outcome was measured.

Introduction

Frequently recommended as a learning modality,^1–5 simulation-based medical education is described as “devices, trained persons, lifelike virtual environments and contrived social situations that mimic problems, events, or conditions that arise in professional encounters.” ⁵ However, its key elements remain to be studied in depth in order to improve simulation-based medical education. One potential aspect that may influence the effect of this kind of education is the level of fidelity, or authenticity in more layman's terms. Fidelity is traditionally described to be assessed on two levels: (1) engineering or physical fidelity, that is, does the simulation look realistic? (2) psychological fidelity, that is, does the simulator contain the critical elements to accurately simulate the behaviours required to complete a task? ^{6 7}

Simulation-based medical education has traditionally been conducted as an off-site simulation (OSS), either at a simulation centre or in facilities in the hospital set up for the purpose of simulation. Recently, in situ simulation (ISS) has been introduced and described as “a team based simulation strategy that occurs on the actual patient care units involving actual healthcare team members within their own working environment.” ^8–12 An unanswered question is whether ISS is superior to OSS. It has been argued that ISS has more fidelity and can lead to better teaching and greater organisational impact compared with OSS. ^8–14

We hypothesised that the physical setting could influence fidelity, and hence ISS could be more effective for educational purposes. To the best of our knowledge, no randomised educational trials have been conducted comparing the ISS versus the OSS setting. Two articles that do use randomisation focused on frequency of training and not setting, and did not include a relevant control group.^{15 16} Previous studies have been criticised for having small sample sizes, weak study designs and a lack of meaningful evaluations of the effectiveness of the programmes.⁸ A recent retrospective video-based study showed that the performance was similar in all the tested simulation settings, but the participants favoured ISS and the authors argued that prospective studies are needed.¹⁷

Human factors such as stress and motivation impact learning,^18–26 which is why we set out to investigate how stress and motivation were affected by ISS versus OSS. We anticipated that the participants would experience ISS as more demanding and as creating higher levels of stress and motivation, which might enhance their learning. Furthermore, we hypothesised that ISS might provide investigators with more information on changes needed in the organisation to improve quality of care.

In this trial, we wanted to apply simulation-based medical education in the field of obstetrics, as delivery wards are challenging workplaces, where patient safety is high on the agenda and unexpected emergencies occur.^27–34 Simulation-based medical education is thus argued to be an essential learning strategy for labour wards.^{4 35} The objective of this randomised educational trial was to investigate the effect of ISS versus OSS on knowledge, patient safety attitude, stress, motivation, perception of the simulation, team performance and organisational impact among multiprofessional obstetric anaesthesia teams.

Methods

Design

An investigator-initiated, single-centre randomised superiority educational trial was previously described in a design article.³⁶

Setting and participants

The setting was the Department of Obstetrics and the Department of Anaesthesiology, Juliane Marie Centre for Children, Women and Reproduction, Rigshospitalet, University of Copenhagen, which has approximately 6300 deliveries per year. Participants were healthcare professionals who worked in shifts on the labour ward: consultant and trainee doctors in obstetrics and anaesthesiology, midwives, specialised midwives, auxiliary nurses, nurse anaesthetists and operating theatre nurses. Participants gave written informed consent. Exclusion criteria were lack of informed consent, employees with managerial and staff responsibilities, staff members involved in the design of the trial and employees who did not work in shifts.³⁶

Recruitment of participants

Eligible participants were provided with information via meetings, a website and personal letters, but additional verbal and written information could also be obtained from the principal investigator (JLS). Informed written consent was obtained if people decided to participate in the trial.³⁶

Interventions

The experimental intervention was a preannounced ISS,^{8 9} that is, simulation-based medical education in the delivery room and operating theatre. The control intervention was an OSS, which took place in hospital rooms set up for the occasion but away from the patient care unit.

An appointed working committee consisting of representatives from all the healthcare professionals participating in the trial developed its aims and objectives, and they designed simulated scenarios for ISS and OSS.³⁶ The two simulation scenarios were: (1) management of an emergency caesarean section after a cord prolapse; and (2) a postpartum haemorrhage including surgical procedures to evacuate the uterus. Focusing mainly on interprofessional skills and communication, the scenarios gave each healthcare profession a significant role to play.³⁷

All participants recruited for a training day were told to arrive at a specific time dressed in work clothes, but had not been told what kind of simulation they were randomised to. The OSS room that was to function as the delivery room was in the doctors’ on-call room, which was small compared to the usual delivery room. A roller table prepared with the usual labour ward equipment had been placed in the room. The OSS room that was to function as the operating theatre was set up in the corner of a lecture hall. An anaesthetic trolley with the usual equipment was placed in the room and equipment for the operating theatre nurses was placed on a roller table. An introductory presentation was given to all participants on how the simulation was organised and then the participants recruited for OSS were shown the fictitious delivery room and fictitious operating theatre.

In the first part of the simulation in the delivery room, someone who has been instructed in role playing acted as the patient in the ISS and OSS settings. In the real and the fictitious operating theatre, a full-body birthing simulator, a SimMom, was used for parts of the simulation scenario.³⁸ Recruited from the working committee, the instructors conducting the simulations were trained in facilitating simulations and doing debriefings. The working committee was trained in local organised courses and attended a British National train the trainers course: PROMPT (PRactical Obstetric Multi-Professional Training).³⁹ They worked in groups of two comprising either a consultant obstetrician with a nurse anaesthetist or a consultant anaesthetist with a midwife. The debriefings lasted 50–60 min and comprised three phases: description, analysis and application.⁴⁰ In addition to the simulation-based medical education, the training day also included video-based, case-based⁴¹ and lecture-based teaching sessions.

Primary outcome

The primary outcome was the results from a knowledge test based on a 40-item multiple choice question (MCQ) test developed specifically for this trial.⁴² The choice of a knowledge test as the primary outcome was mainly a pragmatic choice. MCQ testing is feasible for testing many participants in a relatively short time and at a low cost.⁴³ Furthermore, previously used knowledge tests could be used for inspiration and for sample size calculation.^{44 45} The participants completed the MCQ test at the beginning and at the end of the training day. They were asked not to discuss the MCQ test with other participants or instructors during the training day.

Exploratory outcomes

The Safety Attitudes Questionnaire (SAQ) is validated in a Danish context.⁴⁶ It included 33 items covering five dimensions: (1) team work climate; (2) safety climate; (3) job satisfaction; (4) stress recognition; and (5) work conditions.^{47 48} The participants did the SAQ 1 month prior to and 1 month after participating in the training day.

Stress: Salivary cortisol levels were used as an objective measure of physiological stress.³⁶ The salivary cortisol samples were obtained as a baseline before the first and the second simulation and at three additional times after the two simulations (figure 1). The subjective stress level was measured using the Stress-Trait Anxiety Inventory (STAI) and cognitive appraisal (CA) (figure 1).^{21 23 49 50}

Figure 1.

Timing of the simulations and measurement of stress: Objective stress was measured by salivary cortisol and subjective stress was measured by State-Trait Anxiety Inventory and cognitive appraisal.

Intrinsic Motivation Inventory (IMI) included 22 items with four dimensions: (1) interest/enjoyment; (2) perceived competence; (3) perceived choice; and (4) pressure or tension (reversed scale).⁵¹

Evaluation questionnaire: Together with the IMI, each participant received an evaluation questionnaire at the end of the training day and they were asked to return it within a week.³⁶

Team performance was video recorded and assessed by experts using a Team Emergency Assessment Measure (TEAM).^{36 52 53} The TEAM scale was used in the original version in English and supplemented with a translated Danish version. The scoring of team performance was done by two consultant anaesthetists and two consultant obstetricians from outside the trial hospital. All four video assessors jointly attended two times 3 h training sessions on video rating, but assessment of the trial videos was conducted individually. Each video-assessor received an external hard disc with 20 simulated scenarios in random order of teams and scenarios of management of an emergency caesarean section and a postpartum haemorrhage, respectively.

Organisational outcomes were registered using: (1) two open-ended questions included in the evaluation questionnaire on suggestions for organisational changes; and (2) debriefing and evaluation at the end of the training day, where participants reported ideas for organisational changes. The principal investigator (JLS) took notes during these sessions, which were then discussed in the previously mentioned working committee, which included authors MJ and KE.

Sample size calculation

We chose data from knowledge tests from previous studies to conduct our sample size estimation.^{44 45} We assumed the distribution of the primary outcome (the percentage of correct MCQ answers) to be normally distributed with an SD of 24%. If a difference in the percentage of correct MCQ answers between the two groups (ISS and OSS) was 17%, then 64 participants had to be included to be able to reject the null hypothesis with a power of 80%. Since the interventions were delivered in teams (clusters), observations from the same team were likely to be correlated.^{54 55} The reduction in effective sample size depends on the cluster correlation coefficient, which is why the crude sample size had to be multiplied by a design effect. With a design effect of 0.05, the minimum sample size was increased to 92.8 participants.⁵⁵ We therefore decided to include a total of 100 participants.

Randomisation and blinding

Randomisation was performed by the Copenhagen Trial Unit using a computer-generated allocation sequence concealed to the investigators. The randomisation was conducted in two steps. First, the participants were individually randomised 1:1 to the ISS versus the OSS group. The allocation sequence consisted of nine strata, one for each healthcare professional group. Each stratum was composed of one or two permuted blocks with the size of 10. Second, the participants in each group were then randomised into one of five teams for the ISS and OSS settings using simple randomisation that took into account the days they were available for training.

Questionnaire data were transferred from the paper versions and coded by independent data managers. The intervention was not blinded for the participants, instructors providing the educational intervention, the video assessors or the investigators drawing the conclusions. The data managers and statisticians were blinded to the allocated intervention groups.

Data analysis and statistical methods

Owing to the low number of missing values, no missing data techniques were applied. Single missing items in the MCQ test or more than one answer to an MCQ item were treated as incorrect answers. Single missing items in the inventories SAQ, IMI and STAI were excluded from the overall calculation of the summary scores.

Calculation of 95% CI obtained after the simulation intervention (post-MCQ, post-SAQ, stress measurements, IMI) was based on generalised estimating equations (GEE)⁵⁶ since observations from individuals on the same team were potentially correlated.

The evaluation data measured on a Likert scale were analysed as comparisons of location of the ordinal responses from items in the evaluation questionnaire performed by the Kruskal-Wallis rank sum test, and the p values were adjusted for multiple testing using the Benjamini-Hochberg method.⁵⁷

The mean outcomes obtained after the simulation intervention (postmeasurements) in the two intervention groups were compared by a linear model including intervention and baseline (premeasurements) as explanatory variables (analysis of covariance (ANCOVA)), and inferences were based on GEE to account for the potential correlation within teams. To assess whether there was a difference in mean between pre and postmeasurements in each of the intervention groups, overall tests of whether the intercept equals 0 and the slope equals 1 from a linear model of the postmeasurements on the premeasurements were performed.

The team data, that is, the ratings from the four assessors, were analysed using linear mixed models to take into account the repeated measurements on the teams by the same assessors. Random effects for each team nested in the randomisation group and in each assessor were included. A model including the interaction between the randomisation group and simulation was used to estimate means, whereas an additive model was used to determine the overall difference in mean between the ISS versus the OSS intervention and the first (emergency caesarean section) and the second (postpartum haemorrhage) simulation (no interaction between randomisation and simulations was found).

Ideas for organisational changes were registered by participants and the reported suggestions were categorised as qualitative data and analysed using part of the framework from the Systems Engineering Initiative for Patient Safety model.⁵⁸

SAS V.9.2, R V.3.0.2 and IBM SPSS Statistics V.20 were used for statistical analysis. Two-sided p values <0.05 were considered significant.

Results

Recruitment, basic characteristics and follow-up of participants

Informed written consent for participation in the trial was provided by 116 healthcare professionals. The two randomised intervention groups were comparable (table 1).

Table 1.

Baseline characteristics of participants in the ISS and OSS groups (n=100)

	ISS group	OSS group
Number of participants	48*	49†
Number of females/males	42/6	43/6
Median age (range)	44.5 (26–63)	42 (27–65)
Median years of obstetric work experiences (range)	7 (0.6–38)	7 (0.6–39)
Previous simulation experiences‡
No experience	8	10
Simple simulation	25	24
Full-scale simulation	15	15
Pregnant participants	2	2
Participants on any kind of medication	19	20
Participants on medication with no expected influence on cortisol measurement§	12	9
Participants on medication with potential influence on cortisol measurement	7	11
Intranasal and inhaled corticosteroids (mometasone furoate, budesonide/formoterol, budesonide, fluticasone/salmeterol)	2	3
Levothyroxine	1	2
Metformin	1	1
Norethisterone/estradiol acetate	0	1
Oral contraceptives	1	3
Beta blockers (metoprolol)	0	1
Antidepressants (nortriptyline, fluoxetine)	2	0

*Not included due to illness: A consultant obstetrician and an operating room nurse (n=2).

†Not included due to illness: An auxiliary nurse (n=1).

‡A simple simulation experience is, for example, skills training using a low-tech delivery mannequin and no video recording of the simulation scenario. Full-scale simulation is, for example, done in teams with fully interactive mannequins and video recorded scenarios.

§Intrauterine contraceptive devices, angiotensin II receptor antagonists, angiotensin-converting-enzyme inhibitors, simvastatin, alendronate, pantoprazole, antihistamine and tinzaparine.

ISS, in situ simulation; OSS, off-site simulation.

The flow of participants is described in figure 2 and in table 2.

Figure 2.

Flow diagram for participants in a trial determining the effects of ISS versus OSS on (1) primary outcome: knowledge (MCQ test); and (2) exploratory outcomes: patient safety attitudes (SAQ), stress (salivary cortisol, STAI, CA), motivation (IMI), perceptions of simulation (evaluation questionnaire), video-assessed team performance (TEAM) and organisational impact.

Table 2.

Reasons for loss to follow-up (n/100 randomised participants (%))

	ISS group	OSS group
Pre-MCQ test	2 (2%)*	1 (1%)*
Post-MCQ test	2 (2%)*	1 (1%)*
Salivary cortisol level at emergency caesarean section simulation	2 (2%)*	3 (3%)*†
Salivary cortisol level at postpartum haemorrhage simulation	2 (2%)*	2 (2%)*‡
STAI at emergency caesarean section simulation	2 (2%)*	1 (1%)*
STAI at postpartum haemorrhage simulation	2 (2%)*	2 (2%)*‡
CA at caesarean section simulation	2 (2%)*	1 (1%)*
CA at postpartum haemorrhage simulation	2 (2%)*	2 (2%)*‡
Evaluation questionnaire	3 (3%)*§	1 (1%)*
IMI	4 (4%)*¶	1 (1%)*
Pre-SAQ	1 (1%)**	4 (4%)*††
Post-SAQ	5 (5%)*‡‡	4 (4%)*‡‡

*Participants ill and did not participate (n=3).

†Two measurements were clear outliers. A re-evaluation of the data collection indicated that the two samples had most likely been swapped between two participants, which is why these measurements were excluded from all analyses (n=2).

‡Since one participant was temporarily called away for clinical work, the cortisol measurement after the simulation in postpartum haemorrhage is lacking and he was unable to answer parts of the questionnaires (n=1).

§Questionnaires not returned (n=1).

¶Questionnaires not returned (n=2).

**Of the individuals who did not participate due to illness (n=3), one filled out the pre SAQ anyhow.

††For three participants, pre SAQ data were excluded because these participants were employed in other departments prior to participating in the training days; hence, their responses did not refer to the department in question (n=3).

‡‡Questionnaires not returned (n=6).

CA, cognitive appraisal; IMI, Intrinsic Motivation Inventory; ISS, in situ simulation; MCQ, multiple choice question; OSS, off-site simulation; SAQ, Safety Attitudes Questionnaire; STAI, Stress-Trait Anxiety Inventory.

Intervention delivery

The trial was conducted from April to June 2013. Out of 100 participants included, 97 participated (tables 1 and 2 and figure 2). The 10 simulations were conducted as planned, although one ISS had to be postponed for 15 min due to an ongoing, real emergency caesarean section. The mean number of minutes spent on the caesarean section simulation in ISS and OSS was 18 and 15 min, respectively (p=0.70), while the mean for the postpartum haemorrhage simulation was 26 and 24 min, respectively, (p=0.40).

Primary outcome

MCQ test: There was no difference in mean post-MCQ scores between the ISS versus the OSS group adjusted for the pre-MCQ scores (table 3). Additional analyses based on the MCQ test, including 33 or 29 of the 40 items, gave similar results (data not shown). These additional analyses were performed because validation of the MCQ test revealed that 7–11 of the 40 MCQ items were disputable.⁴²

Table 3.

Means (95% Cl) of percentages of correct answers in the MCQ test before (pre-MCQ) and after (post-MCQ) in the ISS and OSS groups

		Descriptive statistics
MCQ test Per cent correct	Simulation intervention	Pre-MCQ mean* (start of training day)	Post-MCQ mean* (end of training day)	Mean difference*†
	ISS	69.4 (65.4 to 73.4)	82.6 (79.3 to 85.8)	−0.02 (−2.13 to 2.09) p=0.98
	OSS	70.6 (66.0 to 75.2)	83.3 (80.4 to 86.1)

*Based on generalised estimating equations to account for potential correlation within teams.

†Adjusted for pre MCQ (ANCOVA).

ANCOVA, analysis of covariance; ISS, in situ simulation; MCQ, multiple choice question (range: 0–100%); OSS, off-site simulation.

Post hoc analysis: The average increase in percentage of correct answers in the MCQ test following training was 13.1% (95% CI 11.0% to 15.3%) in the ISS group and 12.7% (95% CI 10.3% to 15.2%) in the OSS group (overall tests of no difference between pre and post MCQ: both p<0.0001).

Exploratory outcomes

SAQ: No differences were found in the ISS versus the OSS group for any of the post-SAQ dimensions (table 4).

Table 4.

Means (95% CI) of SAQ (converted to percentages) for five dimensions 1 month before (pre-SAQ) and 1 month after (post-SAQ) the simulation training day with ISS and OSS

		Descriptive statistics
	Simulation intervention	Pre-SAQ mean (1 month before)	Post-SAQ mean* (1 month after)	Mean difference*†
SAQ teamwork climate	ISS	80.5 (76.7 to 84.3)	81.1 (76.7 to 85.5)	−1.4 (−5.8 to 3.1) p=0.54
	OSS	78.4 (74.1 to 82.2)	81.2 (77.5 to 85.0)
SAQ safety climate	ISS	66.7 (61.8 to 71.6)	70.6 (65.9 to 75.2)	1.6 (−2.0 to 5.1) p=0.39
	OSS	69.2 (65.4 to 73.0)	70.8 (66.8 to 74.8)
SAQ job satisfaction	ISS	86.4 (82.9 to 89.8)	87.5 (83.3 to 91.7)	0.6 (−2.9 to 4.1) p=0.74
	OSS	85.6 (81.6 to 89.6)	85.7 (81.9 to 89.5)
SAQ stress recognition	ISS	69.7 (63.5 to 76.0)	68.8 (62.4 to 75.1)	−2.6 (−9.2 to 4.0) p=0.44
	OSS	67.3 (61.2 to 73.3)	69.2 (64.0 to 74.4)
SAQ work condition	ISS	66.4 (60.8 to 72.1)	64.9 (59.0 to 70.8)	−0.3 (−5.7 to 5.1) p=0.91
	OSS	65.9 (59.9 to 71.8)	64.0 (58.1 to 69.8)

*Based on generalised estimating equations to account for potential correlation within teams.

†Adjusted for pre SAQ (ANCOVA).

ANCOVA, analysis of covariance;ISS, in situ simulation; OSS, off-site simulation; SAQ, Safety Attitudes Questionnaire (range: 0–100%).

Salivary cortisol, STAI and CA: The mean change in baseline to peak was similar for ISS versus OSS for both the first (caesarean section) and the second (postpartum haemorrhage) simulation (table 5).

Table 5.

Mean (95% CI) of salivary cortisol (nmol/L), STAI and CA during simulation in management of an emergency caesarean section and postpartum haemorrhage conducted as ISS and OSS

		Baseline	Postsimulation 0 min Mean*	Postsimulation 5 min Mean*	Postsimulation 10 min Mean*	Peak-level mean*†	Mean difference*‡ of baseline to peak of ΔOSS versus ΔISS
First simulation: emergency caesarean section
Cortisol	ISS	7.0 (6.3 to 7.8)	8.9 (7.2 to 10.6)	8.1 (6.6 to 9.6)	8.1 (6.6 to 9.5)	9.3 (7.6 to 11.0)	−0.5 (−1.6 to 2.5) p=0.64
	OSS	7.3 (5.3 to 9.2)	8.2 (6.3 to 10.2)	7.8 (6.1 to 9.6)	8.0 (6.2 to 9.8)	9.0 (6.9 to 11.1)
STAI	ISS	32.2 (30.4 to 34.0)	34.8 (32.7 to 37.0)		31.3 (29.5 to 33.1)	36.5 (34.3 to 38.7)	−0.2 (−2.1 to 2.5) p=0.85
	OSS	33.1 (31.1 to 35.0)	34.8 (32.2 to 37.3)		30.7 (29.0 to 32.4)	37.0 (34.7 to 39.3)
CA	ISS	1.0 (0.9 to 1.1)	0.8 (0.7 to 1.0)		0.8 (0.7 to 0.9)	0.8 (0.7 to 1.0.)	0.0 (−0.2 to 0.2) p=0.93
	OSS	1.0 (1.0 to 1.1)	0.8 (0.7 to 0.9)		0.8 (0.6 to 0.9)	0.9 (0.7 to 0.9)
Second simulation: postpartum haemorrhage
Cortisol	ISS	7.4 (6.5 to 8.3)	9.2 (7.7 to 10.7)	7.7 (6.6 to 8.8)	7.4 (6.3 to 8.5)	9.4 (7.9 to 10.9)	−1.2 (−0.1 to 0.25) p=0.07
	OSS	6.9 (5.9 to 7.9)	7.5 (6.6 to 8.4)	6.7 (5.8 to 7.7)	6.8 (6.0 to 7.6)	7.7 (6.7 to 8.7)
STAI	ISS	31.8 (30.0 to 33.6)	31.8 (30.1 to 33.6)		28.5 (27.3 to 29.7)	32.2 (30.5 to 33.9)	−0.5 (−2.2 to 1.3) p=0.61
	OSS	32.1 (29.9 to 34.2)	32.4 (30.5 to 34.3)		30.1 (28.5 to 31.8)	32.8 (31.0 to 34.7)
CA	ISS	1.0 (0.9 to 1.1)	0.8 (0.7 to 0.9)		0.8 (0.7 to 0.9)	0.8 (0.7 to 0.9)	0.1 (−0.2 to 0.1) p=0.56
	OSS	1.1 (1.0 to 1.2)	0.9 (0.7 to 1.0)		0.8 (0.7 to 0.9)	0.9 (0.7 to 1.0)

*Based on generalised estimating equations to account for potential correlation within teams.

†Peak level is the maximum of the measurements obtained at 0, 5 and 10 min after the end of the simulation.

‡Adjusted for pre-cortisol, pre-STAI and pre-CA (ANCOVA).

ANCOVA, analysis of covariance;CA, cognitive appraisal (range 0.1–10); ISS, in situ simulation; OSS, off-site simulation; STAI, Stress-Trait Anxiety Inventory (range 20–80).

Post hoc analysis: The salivary cortisol and STAI levels increased significantly from baseline to peak in the ISS and OSS groups following the first (caesarean section) and the second (postpartum haemorrhage) simulation (overall tests for no difference between pre and post: all p<0.0001). CA decreased significantly from baseline to peak in the ISS and OSS settings in both the caesarean section and in the postpartum haemorrhage simulations (p<0.0001).

IMI: No differences were found in the ISS versus the OSS group for the IMI score (table 6).

Table 6.

Mean (95% CI) motivation after participation in either ISS or OSS. Analysis comprised a comparison of the mean IMI and the mean of the ISS and OSS groups

Simulation intervention	IMI mean (1 week after)*
Interest/enjoyment
ISS	5.2 (4.9 to 5.5)
OSS	5.3 (5.1 to 5.5)
	p=0.72
Perceived competence
ISS	5.1 (4.8 to 5.4)
OSS	4.9 (4.7 to 5.1)
	p=0.24
Perceived choice
ISS	5.8 (5.6 to 6.1)
OSS	5.5 (5.2 to 5.9)
	p=0.15
Pressure tension (reversed)
ISS	2.8 (2.5 to 3.1)
OSS	2.9 (2.6 to 3.3)
	p=0.65

*Based on generalised estimating equations to account for potential correlation within teams.

IMI, Intrinsic Motivation Inventory (range: 1–7); ISS, in situ simulation; OSS, off-site simulation.

Participant evaluations and perception: For almost all 20 questions in the evaluation questionnaire, the ISS and OSS groups did not differ significantly. However, the two questions addressing the authenticity or fidelity of the simulations were scored significantly higher by the ISS participants compared with the OSS participants (table 7).

Table 7.

Participant evaluations after participation in either ISS or OSS in medians with 25% and 75% quartiles. Analysis comprised a comparison of the evaluation medians of the ISS versus the OSS group

	ISS	OSS
	Median (1st Q–3rd Q)	Median (1st Q–3rd Q)	p Value*
Evaluation questions (shortened version, original version in Danish)
1. Overall, the training day was (1=very bad to 5=very good)	5 (4–5)	5 (4–5)	0.70
2. Multi-professional approach with all healthcare groups involved was (1=very bad to 5=very good)	5 (4–5)	5 (4–5)	0.70
3. I thought the level of education of the training was (1=very much over my level to 5=very much below my level)	3 (3–3)	3 (3–3)	0.70
4. Will recommend others to participate (1=never to 5=always)	5 (5–5)	5 (4–5)	0.70
5. Did simulations inspire you to change procedures or practical issues in the labour room or operating theatre (1=no ideas to 5=many ideas) (included open-ended questions)	3 (2–3)	3 (2–4)	0.70
6. Did simulations inspire you to change guidelines (1=no ideas to 5=many ideas) (included open-ended questions)	2 (1–2)	2 (1–2)	0.70
Simulation of an emergency CS
7. Overall, my learning was (1=very bad to 5=very good)	4 (3–4)	4 (3–4)	0.90
8. The authenticity of the CS simulation was (1=not at all authentic to 5=very authentic)	4 (3–4)	3 (3–4)	0.02
9. The authenticity of the CS simulation influenced my learning (1=not at all important to 5=very important)	4 (4–4.5)	4 (4–4)	0.65
10. Collaboration in the CS team was (1=very bad to 5=very good)	4 (4–4.5)	4 (3.8–4)	0.27
11. Communication in the CS team was (1=very bad to 5=very good)	4 (3–4)	4 (3–4)	0.23
12. The CS team leader was (1=very bad to 5=very good)	4 (3–4)	4 (3–4)	0.26
13. My learning at the debriefing after the CS was (1=very bad to 5=very good)	4 (4–5)	4 (4–4)	0.88
Simulation in PPH
14. My learning overall was (1=very bad to 5=very good)	4 (4–4)	4 (4–4)	0.70
15. The authenticity of the PPH simulation was (1=not at all authentic to 5=very authentic)	4 (3–4)	3 (3–4)	0.01
16. The authenticity of the simulation in PPH influenced my learning (1=not at all important to 5=very important)	4 (4–4.5)	4 (4–4)	0.23
17. Collaboration in the PPH team was (1=very bad to 5=very good)	4 (4–4.5)	4 (4–4)	0.64
18. Communication in the PPH team was (1=very bad to 5=very good)	4 (3.5–4)	4 (3–4)	0.64
19. The PPH team leader was (1=very bad to 5=very good)	4 (4–4)	4 (3–4)	0.23
20. My learning at the debriefing after the PPH was (1=very bad to 5=very good)	4 (4–4)	4 (4–4)	0.57

*Kruskal-Wallis rank sum test. p Values adjusted for multiple testing.

CS, caesarean section; ISS, in situ simulation; OSS, off-site simulation; 1st Q–3rd Q, 25% and 75% quartiles; PPH, postpartum haemorrhage.

TEAM: No significant differences were found in the team scoring of performance between the ISS versus the OSS group (table 8).

Table 8.

Mean (95% CI) of video assessment performance scores with the TEAM scale

	ISS	OSS
	Mean	Mean	p Value*
Video assessment scoring of performance
TEAM (means of item rating) Simulation in emergency CS*	2.6 (2.3 to 3.0)	2.4 (2.1 to 2.8)
TEAM (means of item rating) Simulation in PPH*	2.9 (2.5 to 3.2)	2.8 (2.5 to 3.2)
Estimated overall difference in mean between ISS and OSS†	0.1 (−0.2 to 0.5)		0.36
TEAM (global rating) Simulation in emergency CS*	6.1 (4.8 to 7.3)	5.3 (4.0 to 6.5)
TEAM (global rating) Simulation in PPH*	6.8 (5.5 to 8.1)	6.3 (5.0 to 7.6)
Estimated overall difference in mean between ISS and OSS†	0.7 (−0.4 to 1.7)		0.18
Differences in video assessment scores of performance between emergency CS (1st) and PPH (2nd) simulation scenarios
Differences in mean of TEAM (means of item rating) of the simulation in emergency CS versus PPH†	0.3 (0.1 to 0.5)		0.0003
Differences in mean of TEAM (global rating) of the simulation in emergency CS versus PPH†	0.9 (0.3 to1.5)		0.005

Four consultants recruited outside the research hospital did the video assessment scoring. Analysis comprised a comparison of the mean TEAM score of the ISS versus the OSS group.

*Means found from a linear mixed model including an interaction between the simulation group (ISS and OSS) and simulation scenario (emergency CS and PPH).

†Overall difference in means found from an additive linear mixed model based on the simulation group and simulation scenario.

CS, caesarean section; ISS, in situ simulation; OSS, off-site simulation; PPH, postpartum haemorrhage; Q, quartile; TEAM, Team Emergency Assessment Measure (range for item rating: 0–4; range for global rating:1–10).

TEAM post hoc analysis: A significant increase was found in the team scoring of performance from the first simulation (emergency caesarean section) to the second (postpartum haemorrhage) (table 8).

Organisational changes: A qualitative analysis showed that more ideas for organisational changes were suggested by ISS participants than OSS participants. For details, see online supplementary table S1. The quantitative analysis, however, showed that participants in the ISS and OSS groups scored equally concerning whether the simulations inspired making changes in procedures or guidelines (table 7, questions 5 and 6).

Discussion

In this randomised trial, we did not find that simulation-based medical education conducted as ISS compared with OSS led to different outcomes assessed on knowledge, patient safety attitude, stress, motivation, perceptions of the simulations and team performance. Participant perception of the authenticity of the ISS and OSS differed significantly, but this had no influence on other individual or team outcomes. We observed that ISS participants provided more ideas for organisational changes than did OSS participants. This is in accordance with several non-randomised studies describing a positive impact of ISS on the organisation.^{8 10 11 13 59–61}

In the evaluation questionnaire (table 7), participants were asked about their perceptions of the authenticity of the simulations, which can be interpreted as their perception of the simulation's fidelity. The participants scored the authenticity to be significantly higher in ISS compared with OSS; however, there were no differences in any of the other outcomes between the ISS and OSS groups. The results from this randomised trial are not consistent with traditional situated learning theory, which states that increased fidelity leads to improved learning.^{62 63} The conclusions from this trial, however, are in alignment with more recent empirical research and discussions on fidelity and learning.^{6 64–66} Our study indicates that the change in simulation fidelity, that is change in setting for simulation, does not necessarily translate into learning. Another randomised trial, which compared OSS as in-house training at the hospital in rooms specifically allocated for training with OSS in a simulation centre, also showed that the simulation setting was of minor importance and that there was no additional benefit from training OSS in a simulation centre versus OSS in-house.^{44 67}

The present trial involved simulation-based training with six different healthcare professions. A relevant perspective is the discussion on expanding the traditional concept of fidelity to include the recently introduced term sociological fidelity, which encompasses the relationship between the various healthcare professionals.^{37 68} After completing the trial, we decided to explore more closely the experiences between the healthcare professionals in a qualitative study.⁶⁹

Post hoc analyses showed similar educational effects in the ISS and OSS groups with a knowledge gain of approximately 13% in both groups. It can be argued that this knowledge gain was due to the test effect.^{70 71} We believe, however, that the test effect was minimised as feedback was not given after the initial testing, which is viewed as crucial to learning from a test, and furthermore only one MCQ test was used.⁷¹

No differences were found in the mean SAQ score after simulation-based medical education in the ISS versus the OSS group. Earlier studies have described that high SAQ values mean that SAQ cannot be influenced by an intervention.^{72 73} The values for SAQ were generally high in this trial compared with various other studies from non-Scandinavian countries.^72–75

There were no differences in the stress level when measured as salivary cortisol levels, STAI and CA in the ISS versus the OSS group. The post hoc analysis showed that simulation-based medical education triggered objective stress, measured by salivary cortisol, to the same extent in the ISS and OSS groups. CA seemed to be without discriminatory effect and a decrease was observed where an increase would have been expected, and the levels of CA were low compared with other studies. Previously used among students and medical trainees,^{22 76 77} CA appeared to have a less discriminatory effect in these more senior groups of healthcare professionals.

IMI^{24 51} revealed no differences between ISS versus OSS. Motivation has not previously been tested in educational simulation studies, and it is argued that a gap appears to exist in the simulation literature on motivational factors and further research has been encouraged.²⁵ Some argue that simulation in the clinical setting, as with ISS, should increase motivation,¹⁴ but this was not confirmed by findings in this trial.

The evaluation data showed no differences between ISS and OSS. Both the ISS and OSS participants gave very high scores on the evaluation. This is in accordance with what is generally seen in interprofessional training.⁷⁸

The team performance showed no differences between ISS versus OSS. The post hoc analysis showed that teams performed statistically significantly better in the second compared to the first simulation, which indicates that the simulations were effective. Validated in previous studies, the TEAM scale has been found to be reasonably intuitive to use,^{52 53} which was also our impression in this study.

According to the participants’ own perceptions, they found that ISS and OSS were equally inspirational with regard to suggesting organisational changes in the delivery room and operating theatre and for clinical guidelines. The qualitative analysis, however, revealed that ISS participants provided more ideas for suggested changes, especially concerning technology and tools in the delivery ward and the operating theatre.⁵⁸ Previous non-randomised studies have suggested that ISS has an impact on organisations, but this has, to the best of our knowledge, never been confirmed in a randomised trial.^{8 11 13 17 59}

Strength and limitations

This trial has several strengths. It was conducted with an adequate generation of allocation sequence; adequate allocation concealment; adequate reporting of all relevant outcomes; had very few dropouts; and was conducted on a not-for-profit basis.^79–81 The trial was also blinded for data managers and statisticians. Generally, ISS programmes have been criticised for their lack of meaningful evaluations of the effectiveness of the programmes.⁸ A strength of this trial was its use of a broad variety of outcome measures using previously validated scales to assess the effect on the individual, the team and the organisational level.

A limitation of the study is the fact that the outcome was based only on immediate measurements of knowledge level and of team performance. Only perceptions of simulation were measured after 1 week (evaluation and motivation) and safety attitudes after 1 month. No clinical outcomes or patient safety data were measured.

A strength of this trial is the involvement of authentic teams that mirrored teams in real life, which seem to be of importance for the so-called sociological fidelity.^{37 68} The teams in this trial were authentic in their design and hence resemble the real clinical setting in every possible way.^{65 82} These kinds of teams are called ‘ad hoc’ on-call teams and are very difficult to follow and observe in the real clinical setting, and assessment of the clinical performance of ad hoc teams for a long period is almost impossible. The authentic teams may also be a limitation because two-thirds of the participants had some simulation experiences. The findings in this trial therefore need to be confirmed among other kinds of healthcare professionals with less experience in simulation-based education.

Previous research on assessment suggests that knowledge-based written assessments can predict the results of performance-based tests, and hence knowledge-based assessment could be used as a proxy for performance.^83–85 However, a better approach to the assessment could have been performance-based tests of clinical work, but this was considered unfeasible.

In this trial, we did not measure long-term retention. The literature on retention of skills suggests that deterioration of the non-used skills appears to occur about 3–18 months after training. More research within the field of retention and on the effect of short booster courses is necessary.^{45 86–88}

There is a risk of type II error and the trial is most likely underpowered, as many randomised trials are. On the other hand, it should be discussed whether performing a larger trial to detect a statistically significant effect of ISS is relevant or feasible and appears to have a clinically or educationally relevant effect.⁸⁹

The improvements on knowledge and team performance may also be due to the Hawthorne effect, that is, due to individuals changing behaviour as a result of their awareness of being observed.⁹⁰ From an educational perspective, a major problem with the Hawthorne effect is an intervention group versus a control group, where the control group is given no intervention.⁹⁰ This issue was avoided in this trial as exactly the same intervention was used for both groups, the only difference being the physical setting, thus most likely minimising the Hawthorne effect in our trial.⁹⁰

Conclusions

This randomised trial compared ISS versus OSS, where OSS was provided as in-house training at the hospital in rooms specifically allocated for training. From this trial, we concluded that changes in settings from OSS to ISS do not seem to provide key elements for improving simulation-based medical education. Although participant perception of the authenticity or fidelity of ISS versus OSS differed significantly, there were no differences in knowledge, patient safety attitude, stress measurements, motivation and team performance between the groups, except that the ISS group generated more suggestions for organisational changes. This trial indicated that the physical fidelity of the setting seemed to be of less importance for learning; however, more research is necessary to better understand which aspects of simulation to be most important for learning.