Simulation Training to Improve the Ability of First-Year Doctors to Assess and Manage Deteriorating Patients: a Systematic Review and Meta-analysis

Abstract

Many simulation courses now exist which aim to prepare first-year doctors for the task of assessing and managing potentially deteriorating patients. Despite the substantial resources required, the degree to which participants benefit from such courses, and which aspects of the simulation training are optimal for learning, remains unclear. A systematic literature search was undertaken across seven electronic databases. Inclusion criteria were that the intervention must be a simulation of a deteriorating patient scenario that would likely be experienced by first-year doctors, and that participants being first-year doctors or in their final year of medical school. Studies reporting quantitative benefits of simulation on participants’ knowledge and simulator performance underwent meta-analyses. The search returned 1444 articles, of which 48 met inclusion criteria. All studies showed a benefit of simulation training, but outcomes were largely limited to self-rated or objective tests of knowledge, or simulator performance. The meta-analysis demonstrated that simulation improved participant performance by 16% as assessed by structured observation of a simulated scenario, and participant knowledge by 7% as assessed by written assessments. A mixed-methods analysis found conflicting evidence about which aspects of simulation were optimal for learning. The results of the review indicate that simulation is an important tool to improve first-year doctors’ confidence, knowledge and simulator performance with regard to assessment and management of a potentially deteriorating patient. Future research should now seek to clarify the extent to which these improvements translate into clinical practice, and which aspects of simulation are best suited to achieve this.

Introduction

For the last decade, there have been growing concerns that doctors feel inadequately prepared when they enter clinical practice [1, 2]. Surveys sent to first-year doctors in the UK since 2001 consistently find that only half of graduates felt that their medical experience prepared them for the tasks they faced in clinical practice [3–6], and these views are often shared by senior colleagues [6–8]. A specific and high-stakes aspect of first-year doctors’ jobs that is often cited by both themselves and their senior colleagues as a cause for concern is the assessment and management of potentially deteriorating patients [6, 7].

Underpreparedness in this specific area is not only the cause of stress for the first-year doctors involved [1, 9, 10] but also a serious patient safety issue that has received substantial attention in recent years [11, 12]. A systematic review of studies that assessed patient outcomes in the period after the changeover of first-year doctors found a statistically significant increase in patient mortality of around 5% [12]. Perhaps equally concerning is another review which suggests that first-year doctors’ competence in acute care is declining over time [13].

Numerous studies have explored first-year doctors’ management of potentially deteriorating patients and identified several deficiencies in this domain: lack of fundamental knowledge relating to acute illness [6, 14]; lack of experience managing deteriorating patients [6, 15], with one study noting a median of one to two experiences with deteriorating patients amongst first-year doctors prior to graduation [15]; and limited opportunities to practice independent decision-making in undergraduate curricula [13, 16]. These factors suggest that gaps in both knowledge and experience applying this knowledge when assessing patients contribute significantly to first-year doctors’ feelings of ill-preparedness.

In response to this, many institutions have developed simulation courses where senior medical students or first-year doctors can practice in an ethically and educationally safe environment. Simulation offers a reproducible learning environment, meaning that the same educational intervention or assessments can be delivered to multiple learners [17]. This is an advantage over workplace-based learning, where competing learner interests, random variation of cases and ethical concerns of potential harm to patients often limit the number of appropriate cases that first-year doctors can assume responsibility for and learn from [18, 19]. Furthermore, in contrast to simulation, significant time pressures and the lack of educational training amongst clinicians mean that explanation and feedback about a case, an important phase in adult learning theory [20, 21], is frequently omitted in the workplace [19, 22, 23].

One of the main disadvantages of using simulation rather than other educational tools is the large costs associated with start-up and utilisation [17]. Consequently, a major issue facing simulation courses is how to prove that knowledge or skills learnt in the simulator translate into the clinical environment, thus justifying the cost of the exercise [17]. As with other educational interventions, there are multiple ways of demonstrating evidence of translation into the workplace. A popular tool used to classify this evidence is the Kirkpatrick hierarchy of training evaluation which identifies four different levels shown in Table 1 [24, 25].

Table 1.

Kirkpatrick hierarchy of outcomes of educational interventions [25]

Kirkpatrick level	Definition	Example in medical education
1	Participants’ reactions to the educational intervention	• Participant feedback on the value of the intervention or confidence to perform certain tasks
2	Modification of participants’ knowledge or skills following the intervention	• Performance in a simulated scenario (scored by an observer using a formal rating tool) • Score in a written test
3	Changes in participant behaviours or performance in the workplace	• Hand hygiene compliance • Score in a workplace-based assessment, such as a mini-CEX
4	Changes to the results of the organisation as a result of the intervention	• Patient outcomes

In general, whilst it may provide more compelling evidence in terms of healthcare quality improvement, demonstrating evidence at higher Kirkpatrick levels becomes more difficult due to confounding by other participant and workplace factors [26–28]. For example, two recent reviews of simulation courses to prepare junior doctors for surgical roles found only articles reporting outcomes at the first two Kirkpatrick levels [29, 30] and largely focussed on procedural skills for which there is already reasonable evidence that simulation can improve performance [26, 31].

Another shortcoming of using Kirkpatrick’s evidence hierarchy is that it fails to take study design and quality into account, meaning that less robust articles can significantly bias the results of a review. This is demonstrated by one of the reviews of surgical preparation courses for junior doctors which found that 87% of its included studies were single-group designs, which led to several potential biases [30]. Unfortunately, this type of study design is common in modern medical education literature [32], despite being identified as having a limited scientific value more than 50 years ago [33].

To conclude, first-year doctors’ feelings of ill-preparedness with regard to assessing and managing potentially deteriorating patients are a significant health and educational concern and are largely due to shortcomings in undergraduate curricula, with limited opportunities to apply learned knowledge to clinical situations. There are a growing number of studies which have used simulation to address this issue. Previous reviews of these studies have concentrated largely on surgical specialties and have been subject to contamination from procedural skill outcomes or bias due to study design limitations [29, 30]. Our review therefore seeks to quantitatively evaluate the evidence of simulation improving first-year doctors’ assessment and management of potentially deteriorating patients and to identify which aspects of such simulation courses are most effective for learning. We plan to frame this evidence within the Kirkpatrick model, using a hierarchy of methodological strength and a quality appraisal tool to address issues of poor study design and bias, respectively [25].

Aim Statement

The study aims to determine whether simulation training for first-year doctors and final-year medical students improves their assessment and management of potentially deteriorating adult patients.

Methods

Search Strategy

A literature search was conducted in March 2017 using the following electronic databases: Medline, ERIC, Scopus, Google Scholar, Web of Science, PsycINFO and Embase. The Database of Abstracts of Reviews of Effects, the Cochrane Library and the Evidence for Policy and Practice Information and Co-ordinating Centre were also searched for relevant review articles. The search strategy consisted of using a term related to simulation in a Boolean-AND combination with a term to indicate learner level. The first five databases listed also required a Boolean-AND term related to education or medical education to limit the search to appropriate numbers and study themes (Table 2).

Table 2.

Search strategy

Database	Search terms
PubMed 561 hits	“Education, Medical” [MeSH] AND (“Patient Simulation” [MeSH] OR “High Fidelity Simulation Training” [MeSH] OR “Simulation Training” [MeSH]) AND “Internship and Residency” [MeSH]
Scopus 175 hits	Simulation AND (“junior doctor” OR “intern”) AND “medical education”
Web of Science 247 hits	Topic: simulation AND topic: “junior doctor” OR “intern” AND topic: “medical education” OR “training”
Google Scholar (only first 50 hits included)	Simulation AND [“junior doctor” OR “intern”] AND “education”
ERIC 511 hits	Explode term “Simulation” AND (explode term “Medical Students” OR explode term “Graduate Medical Education”)
Embase 152 hits	Simulation AND (“junior doctor” OR “intern”)
PsycINFO 48 hits	Explode term “Simulation” AND (explode term “medical internship” OR explode term “Medical residency”)

All original research articles and conference proceedings were included. The reference lists of the included articles were also searched until no additional papers matching the inclusion criteria were found.

Study Selection

Rationale for Selection Criteria

For the purposes of this review, we have chosen to limit the scope of scenarios of the deteriorating patient to adult general medical and surgical patients, for the following reasons. Firstly, at least in New Zealand, these are the only specialties in which first-year doctors are placed. Not coincidentally, adult general medical and surgical wards also happen to be the environments which have received the most attention in the quality and safety field in New Zealand [34]. Focussing on this area also limits the scope of common clinical scenarios that might be simulated, such as sepsis or breathlessness, which allows for more appropriate comparison across studies. Finally, deteriorating patient scenarios encountered in other sub-specialties such as obstetrics and otorhinolaryngology often incorporate procedural components [35, 36], for which, as noted above, there is already reasonable evidence for the benefit of simulation training [26, 31].

We have also chosen to focus our review at the level of training of the final-year medical student and first-year doctor. This is because the educational aims for these two groups are very similar, at least in New Zealand. That is, the primary focus is to be able to assess patients at a particular level of complexity, synthesise the information of that assessment, initiate management and know when to escalate care [37]. In comparison, trainees at more junior levels of training are still learning the basic skills of how to extract relevant information by way of history and examination, whilst more senior levels of training tend to focus on definitive management of more complex patients.

Inclusion criteria are as follows:

1. The educational intervention is a simulation of a deteriorating or potentially deteriorating patient scenario that would occur on an adult medical or surgical ward

2. Simulations must involve participants both

   1. Eliciting medical information from or about a simulated hospital patient
   2. Making decisions regarding their medical care

3. The type of simulator must be described (e.g. mock phone call, high-fidelity mannequin)

4. Educational outcomes of participants following the simulation must be described

5. Participants are final-year medical students or doctors in their first year of practice only. If participants from other levels of training are included, the study must report on isolated outcomes from eligible participants in order to be included

6. For studies that combine different types of simulations such as procedural or life support skills, the simulations meeting the inclusion criteria must comprise at least 50% of the total number of simulations. Alternatively, outcomes from simulations meeting inclusion criteria must be distinguishable from those that do not

7. English language studies only

8. Publications from any year are accepted

Exclusion criterion is as follows:

1. Technical or validity studies of simulation courses

Data Extraction

The following data were collected from each article included in the review and tabulated in a Microsoft Excel spreadsheet (Redmond, USA): study design; type of outcome measured (such as confidence, knowledge, simulated or real-world performance); measurement tools, including reliability and validity data; study outcomes; duration and number of simulations; and duration of feedback. In addition, the features of the simulation intervention shown in Table 3 were also collected and entered as discrete variables. Finally, each paper was reviewed for quantitative or qualitative information that assessed the participants’ response to a particular simulation feature.

Table 3.

Simulation features and types of each feature

Simulation feature	Possible types
Type of simulator	Human actors (HAs) High-fidelity mannequins (HFMs) Telephone consult with actor portraying a ward nurse enquiring about a patient (‘mock page’)
Participant numbers in the simulator	Individual Group
Format of simulation course	Single or half-day (‘one-off’) Continuous course over more than 1 day (‘boot camp’) Single sessions distributed over a longer period of time (‘distributed’)
Participant level	Final-year medical student - For the purposes of classification, participants who undertook the simulation immediately before commencing work as first-year doctors were considered as medical students First-year doctors
Participant numbers in the feedback session	Individual group
Presence of additional feedback tools	Video-assisted feedback Human actor scoring or feedback Resident-led feedback

Data Analysis

Where quantified numerically, performance in a simulated scenario between simulation and non-simulation groups underwent meta-analysis using Cochrane Review Manager 5.3 software (Cochrane Community, London) which allowed the creation of a forest plot [38]. In order to analyse the simulation features that were collected as discrete variables (Table 3), a scoring system was used which allocated points to each paper based on the methodological strength of its study design. Similar approaches have been used in other systematic reviews to rank the quality of heterogeneous findings in healthcare reports [39, 40]. Studies that used two different types of the same feature (for example, medical students and doctors) were not counted in the analysis of that feature, as it would be unclear which variable led to the outcome.

From a pilot search and the two existing reviews on simulations for junior doctors, four common study designs were identified and ranked within an established hierarchy of medical educational evidence [33, 41–43]. In addition, other quantitative and qualitative data relevant to each simulation feature were compared alongside its point-based score for a more complete view of the evidence.

Finally, due to the anticipated heterogeneity of the included studies, the Mixed Methods Appraisal Tool (MMAT) was chosen as a quality assessment tool [44]. The categories in the MMAT differ slightly to the main study designs listed above, mainly in that cohort and before-and-after studies are combined, whilst single-group post-test studies are stratified into quantitative, qualitative and mixed-methods categories; however, this has a limited effect on the scoring scheme in this review (Table 4).

Table 4.

Points allocated to each design, from the most rigorous design (randomised controlled trials) to the least (single-group post-test only), with corresponding MMAT category

MMAT category	Study designs identified by pilot search and other reviews of simulation training	Points allocated in this review
Quantitative randomised controlled trials	Randomised controlled trial	4
	Randomised cross-over trial	4
Quantitative non-randomised controlled trials	Controlled trial without randomisation (‘cohort’)	3
	Single-group pre-test and post-test (‘before and after’)	2
Quantitative descriptive studies	Single group, post-test only	1
Qualitative descriptive studies	Single group, post-test only	1
Mixed-methods studies	Single group, post-test only	1

Results

Literature Search

The literature search yielded 1744 publications, of which 1444 remained after exclusion of duplicates (Fig. 1). From this search, 229 abstracts were screened, of which 96 full-text articles were assessed for eligibility. From the references of these 96 full-text articles, 43 abstracts were screened, resulting in a further 20 full-text articles being assessed. In total, 116 full-text articles were assessed for eligibility, of which 48 met all inclusion and exclusion criteria.

Fig. 1.

PRISMA flowchart of included and excluded studies [45]

Features of Included Studies

The included studies showed a preponderance towards less rigorous study designs, as shown in Table 5. Studies in the lower Kirkpatrick levels tended to be single-group designs, meaning that no control group was used. This occurred in 96% of Kirkpatrick level one studies, as opposed to 50% of Kirkpatrick level two studies and no Kirkpatrick level three studies. Where two-group designs were used (14 studies), only 6 studies (43%) used an alternative educational intervention, such as problem- or workplace-based learning, in the comparison group [46–51].

Table 5.

Results of the included studies’ designs, stratified by Kirkpatrick level

Study design	Number of studies	Kirkpatrick level			Mean MMAT
		1	2	3
Randomised control or cross-over trial	6	0	5	1	3.3
Controlled trial without randomisation	8	1	6	1	2.9
Single-group pre-test and post-test	18	8	10	0	1.9
Single-group post-test only	16	15	1	0	2.7
Total number of studies	48	24	22	2	2.5

The MMAT proved to be a useful measure of quality across different study designs, reflected by the higher scores achieved by more rigorous designs. The only discrepancy in this trend was caused by qualitative studies within the ‘single group post-test’ category, which looked at what factors made the simulation training effective. As single-group post-test study design is generally appropriate to answer this question, these studies tended to score relatively high.

Kirkpatrick Level One Outcomes

Studies reporting Kirkpatrick level one outcomes comprised half (n = 24) of the included articles, with 20 using quantitative methods such as Likert scales to grade participants’ reactions, and four reporting only qualitative data, which were either written or in the form of verbal feedback from participants about the simulation.

Most of the quantitative data was unsuitable for meta-analysis, as phrasing of the questions asked by participants and the method of outcome reporting were different between studies, and standard deviation data were not reported. However, for their chosen measure, every article reported a positive educational outcome. For example, six studies asked participants to rate the global benefit of the course on a Likert scale, and all reported that at least 90% of their participants gave a positive response [49, 52–56]. Twelve studies demonstrated a self-reported improvement in participants’ knowledge, confidence or skills relating to working as a junior doctor [49, 53–57] or managing deteriorating patients specifically [9, 58–62].

Kirkpatrick Level Two Outcomes

Studies reporting on objective measures of participants’ knowledge or performance following simulation (Kirkpatrick level two outcomes) comprised another 22 articles. Of these, 13 used performance in a simulation as assessed by an observer, eight used written tests and one used both.

For the 14 studies that used simulation performance as assessed by an observer, 12 used measures where the difference between the simulation and control groups could be calculated as a percentage and thus were amenable to meta-analysis. The other two utilised checklists for which a numerical difference between the simulation and control groups could not be calculated [63, 64].

Seven of the 12 studies reporting results suitable for meta-analysis did not publish all the required data. Attempts were made to contact these authors, and three responded with the required data [48, 65, 66] and four did not provide the requested information. This resulted in eight studies being amenable to meta-analysis.

Of these eight studies, four reported outcomes from individual simulated scenarios without an overall mean. However, for three of these studies, only one scenario from each study was relevant to the review question, so only these outcomes were included for the meta-analysis [47, 67, 68]. The fourth study reported outcomes for three relevant scenarios but as the outcomes were very similar across all three, the median score was chosen for the meta-analysis [50].

Therefore, eight studies were included in the meta-analysis and demonstrated that a total of 286 participants who had undertaken simulation training performed on average 16% (95% confidence interval 14–17%, p < 0.00001) better in simulated scenarios than the 245 participants who had not [47, 48, 50, 65–68] (Fig. 2). Of the studies included in the meta-analysis, one was a before-and-after study [65], four were cohort designs [50, 67–69] and three were RCTs [47, 48, 66]. The exclusion of either the before-and-after study or the four cohort studies did not result in a change to the pooled effect estimate of the meta-analysis, and in fact, this increased the heterogeneity (I²) by 1% and 2%, respectively. All but one of the included studies scored at least three out of four on the MMAT quality appraisal [65].

Fig. 2.

Meta-analysis of improvements in simulator performance following simulation training

Of the nine studies that used written tests to measure participants’ improvement in knowledge following simulation, six used multiple-choice questionnaires, two used short answer questions and one, Rogers et al. [70], used both. Given this tendency, the results from multiple-choice questionnaire testing from Rogers et al. [70] were selected as the outcome for inclusion in the meta-analysis.

Three studies lacked the necessary data to be included in the meta-analysis [66, 71, 72]. This left six studies that could be included in the meta-analysis, which showed that a total of 225 participants who had undertaken simulation training performed on average 7% (95% CI 6–8%, p < 0.00001) better in written tests than 273 participants who had not [46, 51, 70, 73–75] (Fig. 3).

Fig. 3.

Meta-analysis of improvements in written tests following simulation

Again, the exclusion of before-and-after studies [73–75] or cohort studies [51] did not change the pooled effect estimate and had a minimal effect on heterogeneity.

Kirkpatrick Level Three and Four Outcomes

A further two studies analysed learner performance in the workplace following simulation (Kirkpatrick level three outcomes). Schroedl et al. [76] utilised a validated checklist to grade performance of an assessment of a real patient between first-year doctors who had been randomised to receive 4 h of additional simulation-based training on top of their usual departmental teaching. Krajewski et al. [77] compared nursing staff and teaching faculty’s opinions of a cohort who had undertaken simulation training to the previous year’s cohort who had not undertaken training. Both reported outcomes favoured the simulation cohort, but the number of studies in this category is too small to perform a meta-analysis. No studies evaluated Kirkpatrick level four outcomes.

Quantitative Analysis of Simulation Features

The quantitative analysis allocated points to studies based on the strength of their methodologies and revealed several favoured types of simulation features. High-fidelity mannequins, group simulations and medical student participants each had approximately two thirds of the points available for the features of ‘simulator type’, ‘number of participants in each simulator’ and ‘participant level’, respectively. For ‘course format’ and ‘number of participants in each feedback session’, the evidence was much more equivocal, with points shared more evenly amongst the types of features.

Only nine studies made use of additional feedback features, such as human actor- or video-assisted or resident-led feedback. Where the studies explored participant opinions of these feedback features, only the resident-led approach appeared to be viewed favourably.

There were 11 studies that used multiple types of simulators and were thus not counted in the quantitative ‘type of simulator’ evidence category [55, 58, 59, 71, 73, 75, 78–82]. No other studies needed to be excluded for using multiple types of the same simulation feature. In addition, relevant qualitative and quantitative data of participants’ responses to certain simulation features in the included studies was directly extracted and detailed in Table 6. The absence of any consistent method of reporting this information and other priorities of this review meant that a formal analysis of this data was not undertaken.

Table 6.

Typology of simulation features

Simulation feature	Type of feature	Support in principle (percentage of total points that favoured that type of feature)	Additional quantitative evidence	Additional qualitative or quantitative information gathered from participants
Type of simulator	Human actor	16 points (24%)	89% of learners agreed or strongly agreed that the simulation was realistic [49]	Human actors preferred to mannequins due to difficulties taking the history [78]
	High-fidelity mannequin	45 points (66%)	26% of learners felt that the simulation was ‘realistic’ or ‘very realistic’ [60]	Needed time to get used to the mannequin [61]
	Mock page	7 points (10%)	Nil	Nil
Number of participants in each sim	Individual	29 points (34%)	Mean duration per simulation (where specified) = 16 min	Preferred individual simulation and feedback compared to previous group simulation experiences [54]
	Group	57 points (66%)	Mean duration per simulation (where specified) = 21 min	Nil
Format	One-off	45 points (45%)	Nil	Nil
	Boot camp	30 points (30%)	Nil	Nil
	Distributed	24 points (24%)	Nil	Nil
Participant level of training	First-year doctors	28 points (30%)	Nil	90% (both first-year doctors and medical students) felt that the simulation course should be undertaken prior to starting work [83]
	Medical students	64 points (70%)	Nil
Feedback	Individual	25 points (42%)	Nil	Individualised feedback cited as a strength [54]
	Group	34 points (58%)	Nil	Nil
Additional feedback features	Duration of feedback	N/A	Mean duration (where specified) = 21 min	Benefit in both condensed (10 min) feedback with longer simulation and longer feedback (45 min) with shorter simulation [84]
	Human actor feedback	7 points	Nil	Feedback from tutors more helpful than feedback from actors [81]
	Video-assisted	10 points	Nil	Some participants disliked video feedback [61, 85]. Beneficial for some learning points, but these are very specific and often predictable [61]
	Resident-led	2 points	Nil	Both studies reported positive feedback from using residents to lead debrief [53, 56]

Discussion

Although evidence was sparse at Kirkpatrick levels three and four, this review shows that simulation is at minimum, an important tool to improve first-year doctors’ confidence, knowledge and simulated performance in the ‘deteriorating patient’ domain. Kirkpatrick level one studies showed a very positive self-reported participant response to simulation courses, whilst the meta-analyses of Kirkpatrick level two studies demonstrated significant improvements in performance of a simulated patient encounter and written test scores. Whilst it seems likely that these educational improvements could translate to improvements in first-year doctors’ performance in the clinical environment (level three outcomes) or patient outcomes (level four outcomes), healthcare organisations may yet require more direct evidence of this connection.

Given the paucity of Kirkpatrick level three evidence in the literature, a potentially valuable area of future research may lie in validity studies of simulations for junior doctors. An example of this in a similar field is a study that used a structured framework to code communication patterns to show that the way anaesthesiologists interacted in a simulated operating room was similar to their behaviour in real-life operating room situations [86]. Future studies which prove that first-year doctors behave similarly in simulated and real deteriorating patient scenarios could therefore be useful to demonstrate the relevance of the existing body of Kirkpatrick level one and two evidence to healthcare organisations.

An important limitation of validity study evidence is that, typically, it is not able to quantify how much exposure to simulation training is required to change first-year doctors’ workplace performance. Answering this question is essential for an optimal design of future simulation-based curricula. To highlight this issue, one survey of first-year doctors reported that those who had experience of simulation felt more globally prepared for practice [15]. However, this effect was only significant when comparing those with five or more simulation experiences to those with none [15]. It is then perhaps surprising that more studies do not take advantage of the widespread use of workplace-based assessments, such as the mini-CEX, to measure Kirkpatrick level three outcomes following various levels of exposure to simulation teaching [87]. This potentially could be a feasible opportunity to not only directly demonstrate the effects of simulation on workplace outcomes but also quantify how much simulation training is needed to achieve this change.

In contrast, given the large confounding effect of modern healthcare environments, demonstrating improvement in patient outcomes following junior doctor simulation training is likely to remain difficult [41]. To date, only one study exists that has shown an improvement in patient outcomes following simulation training for ward staff to better recognise deteriorating patients; however, this required analysis of 2000 patient outcomes, and the intervention also included revision of vital sign charts [88]. As such, validity studies of junior doctor simulator performance and Kirkpatrick level three studies using existing workplace-based assessments currently appear to be the most feasible ways of demonstrating the value of simulation to healthcare organisations.

Strengths and Limitations

The main strength of this review is the synthesis of a large number of studies that assessed the value of placing a first-year doctor or final-year medical student in a simulation of a potentially deteriorating patient. These studies were taken from a systematic search and underwent quality appraisal. We were also able to perform a meta-analysis on screened quantitative data at Kirkpatrick level two evidence, which, to our knowledge, has not been performed before. The main limitations of this review stem from study heterogeneity, bias towards certain types of simulation features, and methodological flaws of the included studies.

Study Heterogeneity

When conducting meta-analyses of randomised controlled trials, it is commonly considered inappropriate to combine studies which demonstrate a heterogeneity statistic (I²) over a certain threshold (typically 75%) [89]. Techniques are then often used to reduce the heterogeneity by excluding the most deviant study estimate, calculating the heterogeneity again and repeating this process until the heterogeneity is reduced below the ‘acceptable’ threshold [89, 90].

Given that we were studying educational interventions, substantial heterogeneity should be expected and so an arbitrary rule-of-thumb for acceptable heterogeneity is not useful [91]. In fact, Higgins, one of the developers of the I² statistic and co-author of the Cochrane Handbook for Systematic Reviews of Interventions, has stated that any amount of heterogeneity is acceptable as long as study eligibility criteria are clear. He has also claimed that the ‘exclude one’ and try again approach is tantamount to manipulation of the predetermined eligibility criteria and should not be used on philosophical and technical grounds [90, 92]. In fact, excluding the non-RCT study designs in our meta-analysis increased heterogeneity rather than reducing it.

Giving us confidence in our results is the fact that all individual study estimates fall on the side of the null-effect line that favours simulation training over control. In addition, we conducted a fixed-effects meta-analysis, which is known to yield a more conservative pooled-effect estimate when small-n studies are included (as in our review). Hence, we believe our finding that simulation training is effective in improving performance and knowledge is secure, since this is a clear binary determination based on our results. Readers may wish to treat our estimate of the size of the benefits of simulation training with more caution.

Bias Towards Certain Types of Simulation Features

A limitation of using the prevalence of different types of simulator features in the quantitative analysis table is that some types of features may have been used by more studies due to their feasibility rather than reflecting optimal learning conditions for simulation. This was evident in the features of ‘type of simulator’ and ‘participant numbers in the simulator’, where high-fidelity mannequins and group simulations may have been more likely to be used due to their ease-of-use and ability to cater to larger participant numbers.

This discrepancy is demonstrated by more detailed evidence from several studies which suggest that both mannequins and group simulations are less preferred than their alternatives. For example, the only study comparing mannequins with actors as simulator types noted a qualitative participant preference for actors [78], whilst two other studies showed that participants felt that actors were more realistic [49, 60]. In reality, each option is probably more suited to a particular scenario. Human actors are likely superior for simulating a ward call of a stable patient where a history is more important than physical signs and immediate intervention, such as chest pain, whilst a high-fidelity mannequin may be preferred for deteriorating patient scenarios as it can allow interventions and provide physiological feedback. Similarly, the only feedback found in our review about numbers of participants in the simulator also favoured simulation participation as an individual rather than in a group [54], which happens to be more consistent with the structure of a real-life on-call shift.

In summary, the most prevalent types of features used in the simulation literature may not necessarily indicate what is optimal for learning. Only five studies in this review commented on or compared different types of simulation features, and all used Kirkpatrick level one outcomes [49, 54, 78, 83, 84]. These findings are detailed in Table 6. Therefore, more studies that directly compare different types of a simulation feature are needed in order to answer which is most optimal for learning for a given scenario. Such studies could be achieved using either one- or two-group designs, at either of the first two Kirkpatrick levels. For example, distributed sessions could be compared to boot camps using a two-group design, or individual and group simulator sessions could be compared by getting feedback from a single group that undertakes both types of simulations.

Methodological Flaws of Included Studies

This review highlights the continued inappropriate use of single-group design studies in the educational literature. For example, the uniformly positive participant responses demonstrated by the Kirkpatrick level one studies appear to suggest that simulation can help alleviate participants’ feelings of unpreparedness for their first year of practice. Unfortunately, 23 out of 24 of these studies used a single-group design. This resulted in participants being asked to rate their confidence or ability in a skill which they had just practised, which would invariably lead to more positive results [41]. Ultimately, these studies contribute less to the educational literature but are likely to continue being reported as such course designs can be easily incorporated into existing simulation curricula [41]. If this is to be the case, as noted above, future Kirkpatrick level one studies might instead ask participants which aspect of the course contributed to their learning, rather than simply if they felt the course was beneficial. The latter question has been answered by this review in the affirmative.

The Kirkpatrick level two studies presented a different methodological concern. Namely, only one of 13 studies which used simulation as an assessment as well as an intervention allowed the comparison group access to simulation training [66], meaning the other 12 inherently favoured the intervention group as they had more experience in the simulator [47, 48, 50, 63–65, 67–69, 85, 93, 94]. The importance of this factor was highlighted by several studies which reported qualitative learner feedback that simulations could be stressful [49, 78, 95] or took some time to learn how to interact with the simulated patient [61]. Additional evidence of this effect could be seen in this review’s meta-analysis, where the improvement in simulation performance following simulation-based teaching was more than double the improvement in written tests. To compound this issue further, only three of the 14 studies blinded the observers scoring the simulator performance to which intervention the participants had received [47, 48, 50], and only seven of the studies reported some form of reliability or validity process for their assessment tool [50, 63, 66–69, 96].

Conclusion

This systematic review critically summarises the existing literature on simulation courses for first-year doctors to assess and manage potentially deteriorating patients. It demonstrates that such courses improve participants’ simulator and written test performance, as well as improve participant confidence and self-rated knowledge. Future research should now focus on two main areas.

First is using the Kirkpatrick level three evidence to show translation of learning into workplace performance following simulation training. The priority should be using workplace-based assessments to measure the benefit of simulation training, as this is surely one of the most direct ways of showing that the participant has improved in their ability to manage deteriorating patients. Second, institutions with less resources should consider studying what aspects of simulation training are optimal for learning, rather than simply reporting an educational improvement.

Notes on Contributors

Dr. Nicholas Buist is an emergency medicine registrar at the Northland District Health Board. He has research interests in how medical education can improve patient safety. This review was undertaken as a part of his post-graduate diploma in clinical education at the University of Auckland.

Dr. Craig Webster is an associate professor at the University of Auckland. He has been involved in research in human factors and safety in healthcare for around 20 years.

Compliance with Ethical Standards

Conflict of Interest

The authors declare that they have no competing interests.