Effect of just-in-time simulation training on provider performance and patient outcomes for clinical procedures: a systematic review

Abstract

Background

Providing simulation training directly before an actual clinical procedure—or ‘just-in-time’ (JiT)—is resource intensive, but could improve both provider performance and patient outcomes.

Objectives

To assess the effects of JiT simulation training versus no JiT training on provider performance and patient complications following clinical procedures on patients.

Study selection

We searched MEDLINE, Cochrane Library, CINAHL, PsycINFO, ERIC, ClinicalTrials.gov, simulation journals indexes and references of included studies during October 2014 for randomised trials, non-randomised trials and before-after studies comparing JiT simulation training versus no JiT training among providers performing clinical procedures. Findings were synthesised qualitatively.

Findings

Of 1805 records screened, 8 studies comprising 3540 procedures and 1969 providers were eligible. 5 involved surgical procedures; the other 3 included paediatric endotracheal intubations, central venous catheter dressing changes, or infant lumbar puncture. Methodological quality was high. Of the 8 studies evaluating provider performance, 5 favoured JiT simulation training with 18–48% relative improvement on validated clinical performance scales, 16–20% relative reduction in surgical time and 12% absolute reduction in corrective prompts during central venous catheter dressing changes; 3 studies were equivocal with no improvement in intubation success, lumbar puncture success or urological surgery clinical performance scores. 3 studies evaluated patient complications; 1 favoured JiT simulation training with 45% relative reduction in central line-associated blood stream infections; 2 studies found no differences following intubation or laparoscopic nephrectomy.

Conclusions

JiT simulation training improves provider performance, but currently available literature does not demonstrate a reduction in patient complications.

Background

Healthcare providers from various backgrounds are often expected to master multiple complex procedures. Historically, learners gained these technical skills as described by the ‘see one, do one, teach one’ mantra. While clinical procedures have become increasingly complex in recent decades, the training has remained unstandardised and even haphazard, resulting in growing concern about deficits in learners’ technical competence and the potential relationship to adverse patient outcomes.¹

Systematic reviews of simulation training for procedures suggest that it improves procedural performance on actual patients, increases provider confidence and has the potential to reduce patient complications.^2–6 Despite these findings, the most effective and efficient methods of providing simulation training remain unknown. Of particular interest is the issue of ideal timing of both initial and refresher simulation training. Currently, teacher and learner time constraints dictate that most simulation training is scheduled irrespective to the timing of the actual procedures.

‘just-in-time’ (JiT) simulation training—training that is done immediately prior to the actual procedure—has been proposed to improve the simulation training process.^7–10 Although the idea of ‘warming up’ or ‘rehearsing’ before a procedure has shown promise in the sports and music literature,^11–13 this technique has not been widely adopted in healthcare. While some studies suggest JiT simulation training may be superior to no JiT training for improving provider performance on clinical procedures, the impact on patient outcomes is less clear and the settings in which it might be of greatest value are not known.^{7 9 10 14–19} Two recent systematic reviews found ‘warm-up’ or ‘rehearsal’ training led to improved technical performance, but one included mostly low-quality observational studies and just two types of complex procedures,¹⁴ and the other included only surgical procedures.²⁰ To date, no comprehensive review has rigorously evaluated the impact of JiT simulation training for procedures on outcomes assessed on actual patients, and across a range of procedures, providers and clinical environments.

Methods

Review protocol

We drafted a protocol outlining our planned approach to addressing our research question. We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement guidelines for reporting our methods and findings.²¹ Our protocol, which remained unchanged, is available on request.

Study eligibility criteria

We included studies that met the following prespecified criteria: (1) the design was a randomised controlled trial, non-randomised controlled trial or before-after study; (2) participants were healthcare providers or trainees who perform clinical procedures; (3) the intervention was JiT simulation training, defined as a technique to replace or amplify real patient experiences with guided experiences²² which occurred within 24 h prior to the anticipated clinical procedure; (4) the control group received no JiT training, which could include simulation training as long as there was no explicit or thoughtful effort to time the training to actual performance of the procedure; and (5) the study included one or more of our prespecified outcomes (see online supplementary appendix S1).

Outcome measures

We prespecified two categories of outcomes—provider performance and patient complications; all outcomes had to be assessed based on procedures performed on actual patients. We chose to exclude the studies measuring performance on simulators (T1—educational laboratory), as our goal was to assess the effects of JiT training on higher levels of the Kirkpatrick model, T2 (patient care practices) and T3 (improved patient and public health).^{23 24} Provider performance outcomes included procedure success rate, defined as full execution of all intended procedural steps, scores on validated performance scales, duration of procedure, number of attempts to success and need for corrective prompts. Patient complication outcomes included rates of procedure-specific adverse events and rates of procedural delay. We prespecified clinical procedure success rate as the primary outcome.

Search methods

We searched MEDLINE (1946–2014), the Cochrane Library (all databases, Issue 10 of 12 October 2014), CINAHL (1984–2014), PsycINFO (1958–2014) and ERIC (1966–2014) using strategies developed with the guidance of two professional librarians. The search themes were ‘Simulation’, ‘Just-in-time’ and ‘Clinical competence’. No limits or language restrictions were used for the MEDLINE, Cochrane Library, CINAHL or PsycINFO searches. Terms were modified as needed for specific databases. Each search was last updated in October 2014 (see online supplementary appendix S2 for full strategies). We searched ClinicalTrials.gov (1997–2014) for unpublished studies. We also reviewed the entire index (since volume 1) of the journals Simulation in Healthcare and Clinical Simulation in Nursing. Reference lists of eligible studies were manually reviewed for additional studies not found by other search strategies.

Study selection, data collection and quality assessment

Two reviewers each screened half of the studies identified by the electronic database searches based on titles and abstracts, excluding only clearly irrelevant records. The remaining studies were distributed among the four members of the study team such that each record underwent independent full-text review by two reviewers. After isolating the eligible studies, two reviewers independently extracted data from each study using a standardised data collection form, which was developed by consensus of four members of the research team (see online supplementary appendix S3 for details) and assessed methodological quality using both the modified Newcastle-Ottawa Scale (mNOS) and the Medical Education Research Study Quality Instrument (MERSQI)^{22 25 26} (see online supplementary appendix S4). While the NOS was originally developed for assessing non-randomised trials/observational studies, the scale we used is a ‘mNOS’ which has been used in other published systematic reviews.^{2 3 5 27} Disagreements were resolved by discussion between primary reviewers; a third investigator was consulted when necessary.

Analysis

Data synthesis and heterogeneity

Owing to the varied nature of the reported results, we summarised each study's findings for our two outcome categories qualitatively. If a study reported more than one measure of provider performance or patient complications, we reviewed the quantitative findings to determine the predominant finding for the outcome category: favours JiT, favours no JiT training or equivocal. When studies reported multiple outcome measures, we used clinical outcomes rather than knowledge measures to determine the predominant finding. If multiple components of a validated performance scale were reported, we used the overall scoring to establish the outcome category. We assigned the label ‘equivocal’ when the reported finding or findings demonstrated no statistically significant difference between study arms, or when the variation among a set of findings was too great to confidently determine a favoured arm. The quantitative findings that informed the qualitative summary judgements for each study are shown in table 3.

Table 3.

Details of provider performance and patient complication outcomes following JiT versus no JiT training

Study	Outcome measure	Effect size		p Value
		JiT	No JiT
Provider performance
Procedure success rate
Kessler et al16
	Successful lumbar puncture by trainee	38%	35%	>0.05
	Number of attempts, mean (SD)	1.4 (±0.6)	2.1 (±1.6)	<0.001*
Nishisaki et al10
	Successful intubation by trainee—first attempt	50.0%	62.5%	0.44
	Successful intubation by trainee—overall	57.5%	75%	0.19
Procedural time
Lee et al9
	Time to complete mobilisation of colon (min)	26.6 (±12.0)	29.4 (±10.1)	0.4
	Time to complete intracorporeal suturing and knot tying to reapproximate the line of Toldt (min)	7.6 (±4.5)	5.8 (±2.9)	0.17
Mucksavage et al18
Laparoscopic radical nephrectomy, mean (SD)
	Total operating room time (min)	334 (±66)	388 (±66)	0.010*
	Surgical time (min)	243 (±70)	290 (±61)	0.021*
Laparoscopic partial nephrectomy, mean (SD)
	Total operating room time (min)	320 (±75)	374 (±62)	0.050*
	Surgical time (min)	227 (±73)	282 (±60)	0.004*
Validated performance scales
Calatayud et al15
	OSATS Global Rating Scale†, median (range)	28.5 (18.5, 32.0)	19.3 (15.0, 31.5)	0.042*
Chen et al7
	Reznick Scale/modified OSATS‡, mean (95%CI)	22.5§ (21, 24§)	19.5§ (18, 21§)	≤0.001*
	Vassiliou Scale¶, mean (95% CI)	18.0§ (19 to 20§)	15.5§ (14 to 17§)	≤0.001*
	Kundhal Scale**, mean (95% CI); lower score=superior performance	9.5§ (8.5 to 10.5§)	11.5§ (10.5 to 12.5§)	≤0.001*
Kessler et al16
	Use of analgesia	68%	19%	<0.001*
	Appropriate stylet removal	69%	54%	0.06*
Lee et al9
	Hand Movement Smoothness Score††, mean (variance)	0.73 (±0.05§)	0.46 (±0.05§)	<0.03*
	Tool Movement Score††, mean (variance)	0.73 (±0.03§)	0.58 (±0.05§)	<0.05*
	Posture Stability Score ††, mean (variance)	0.54 (±0.06§)	0.34 (±0.05§)	<0.05*
	Attention Score ††, mean (variance)	0.8 (±0.02§)	0.64 (±0.03§)	<0.02*
	Distraction Score ††, mean (variance)	0.34 (±0.01§)	0.53 (±0.05§)	<0.001*
	Mental Workload Score ††, mean (variance)	0.68 (±0.03§)	0.87 (±0.05§)	<0.02*
	Mobilisation of colon‡‡, mean (variance)	21.43 (±0.54)	19.86 (±0.51)	0.04*
	Intracorporeal suturing and knot tying to reapproximate the line of Toldt ‡‡, mean (variance)	3.50 (±0.23)	3.54 (±0.25)	0.92
Moldovanu et al17
Respect for tissue scores§§,¶¶—components of OSATS‡, mean (SEM)
	Exposure of biliary region and adhesiolysis	4.71 (±0.18) 4.71 (±0.18)	3.88 (±0.35) 3.88 (±0.29)	0.059 0.037*
	Dissection of cystic pedicle and critical view	4.43 (±0.29) 4.71 (±0.18)	3.38 (±0.18) 3.88 (±0.29)	0.008* 0.037*
	Dissection of gall bladder	4.00 (±0.31) 4.43 (±0.20)	3.00 (±0.18) 3.63 (±0.32)	0.014* 0.058
	Overall tissue score	4.43 (±0.20) 4.57 (±0.20)	3.75 (±0.16) 3.87 (±0.22)	0.021* 0.041*
Time and motion scores§§,¶¶—components of Vassiliou¶, mean (SEM)
	Exposure of biliary region and adhesiolysis	4.29 (±0.18) 4.57 (±0.20)	4.50 (±0.32) 4.13 (±0.35)	0.593 0.308
	Dissection of cystic pedicle and critical view	4.57 (±0.29) 4.71 (±0.18)	4.00 (±0.26) 3.88 (±0.12)	0.175 0.002*
	Dissection of gall bladder	4.57 (±0.20) 4.57 (±0.20)	4.38 (±0.26) 4.00 (0.19)	0.573 0.059
	Overall tissue score	4.71 (±0.18) 4.86 (±0.14)	4.25 (±0.16) 4.00 (±0.19)	0.081 0.004*
Overall impression scores§§,¶¶, mean (SEM)
	Exposure of biliary region and adhesiolysis	4.86 (±0.14) 4.43 (±0.20)	4.50 (±0.27) 4.25 (±0.25)	0.279 0.595
	Dissection of cystic pedicle and critical view	4.71 (±0.18) 4.71 (±0.18)	4.13 (±0.22) 4.25 (±0.25)	0.069 0.169
	Dissection of gall bladder	4.71 (±0.18) 4.71 (±0.18)	4.25 (±0.16) 4.13 (±0.22)	0.081 0.069
	Overall tissue score	4.86 (±0.14) 4.86 (±0.14)	4.37 (±0.18) 4.00 (±0.19)	0.059 0.004*
Other measures
Kessler et al16
	Use of analgesia	68%	19%	<0.001*
	Appropriate stylet removal	69%	54%	0.06*
Scholtz et al19
	Per cent of CVC dressing changes requiring corrective prompts to complete procedure	9%	21%	<0.001*
Patient complications
Mucksavage et al18
Laparoscopic radical nephrectomy
	Estimated blood loss in mL, mean (SD)	74.4 (±41.9)	130.9 (±186.4)	0.226
	Per cent of patients needing patient controlled analgesia	12%	28%	0.119
	Length-of-stay in hours admitted to hospital floor, mean (SD)	65.6 (±31.1)	65.6 (±27.6)	0.995
	Change in Cr from baseline to discharge in mg/dL, mean (SD)	0.46 (±0.34)	1.02 (±2.1)	0.287
Laparoscopic partial nephrectomy
	Estimated blood loss in mL, mean (SD)	102.5 (±75.0)	101.2 (±67.2)	0.960
	Per cent of patients needing patient controlled analgesia	20%	33%	0.662
	Length-of-stay in hours admitted to hospital floor, mean (SD)	58.8 (±30.4)	66.4 (±39.4)	0.562
	Change in Cr from baseline to discharge in mg/dL, mean (SD)	0.13 (±0.22)	0.12 (±0.14)	0.972
	Renal ischaemic time in minutes, mean (SD)	29.4 (±9.76)	32.75 (±11.27)	0.533
Nishisaki et al10
	TIAE—minor (% of cases)	32.5%	20.8%	0.4
	TIAE—major (% of cases)	0%	0%	–
Scholtz et al19
	Central line-associated blood stream infection (per 1000 CVC line days)	2.9	5.3	<0.001*

Favoured arm is bold.

*Statistically significant.

†OSATS is a validated Global Rating Scale based on seven components, scores range 7–35. Higher score=better performance.

‡Reznick Scale OSATS (modified) is a validated Global Rating Scale based on six components each scored on a five-point anchored Likert scale, scores range 6–30. Higher score=better performance. Also contains a pass/fail score.

§Estimated.

¶Vassiliou Scale is a validated Global Rating Scale based on five components each scored on a five-point anchored Likert scale, scores range 5–25. Higher score=better performance.

**Kundhal Scale is a validated Global Rating Scale based on five components each scored on a five-point anchored Likert scale, scores range 5–25. Lower score=better performance.

††Scale ranges from 0 (no) to 1 (yes).

‡‡Modified Vassiliou Scale.

§§First line represents evaluator 1 scores, bottom line indicates evaluator 2 scores.

¶¶Global Rating Scale/Score.

Cr, creatinine; JiT, just-in-time; CVC, central venous catheter; OSATS, Objective Structured Assessment of Technical Skill; TIAE, tracheal intubation-associated events.

Assessment of reporting biases

In order to informally evaluate for publication bias, we ordered the studies by sample size to assess whether the smaller studies had consistently larger effect sizes.

Subgroup analyses

We prespecified three subgroup analyses based on the following variables: (1) provider level of training (resident vs attending vs nurse), hypothesising that earlier learners would benefit more from JiT training; (2) procedure complexity (bedside vs surgical), hypothesising that more complex procedures would be more subject to degradation of skill over time, as well as have increased risk of complications, and thus benefit more from JiT simulation training; and (3) location of procedure (outpatient vs inpatient vs intensive care unit (ICU)), hypothesising that these locations would be proxies for severity of illness and that sicker patients would have more difficult procedures and higher risk of complications.

Sensitivity analyses

We evaluated the impact of methodological quality on our overall summary impressions in three ways. First, for each domain of the two tools, we excluded studies that were deemed high risk of bias. Second, we excluded studies that had total scores less than four on the mNOS or less than the median of the Medical Education Research Quality Instrument previously described in a systematic review.²² Third, we subgrouped the studies according to whether they were randomised trials versus non-randomised study designs.

Results

Description of studies

Results of search

As shown in figure 1, our search strategies yielded 1805 potentially relevant publications. After exclusions based on title and abstract screening, 36 studies required full-text review, of which 8 met our inclusion criteria.^{7 9 10 15–19} See table 1 for study characteristics.

Figure 1.

Study selection flow.

Table 1.

Characteristics of included studies

Study, year, country	Study design	Provider/learner	n	Procedure	n	Description of just-in-time intervention*	MERSQI (maximum 16)	mNOS (maximum 6)
Scholtz et al,19 2013, USA	Before-after study	Nurses	524	Central venous catheter dressing change (inpatient and outpatient)	2469	20 min of simulated central venous catheter dressing change with observation, assessment and debriefing, immediately prior to a dressing change	12.5	4
Nishisaki et al,10 2010, USA	Non-randomised control trial	Paediatric and emergency residents	78	Endotracheal intubation in PICU	401	10 min of hands-on orotracheal intubations on paediatric manikin and 20 min of multidisciplinary team training with high-fidelity infant simulator with debriefing, up to 24 h prior to intubation	12	4
Kessler et al,16 2015, USA	Before-after study	Paediatric and emergency residents	1319	Infant lumbar puncture	436	Standard Training PLUS 5–10 min of hands-on training on an infant lumbar puncture trainer coached by clinical supervisor, immediately prior to LP	14	4
Calatayud et al,15 2010, Denmark, UK, Canada	Randomised cross-over trial	Surgical residents	10	Laparoscopic cholecystectomy	20	15 min of virtual reality training of three tasks, immediately prior to surgery	14.5	6
Lee et al,9 2012, USA	Randomised cross-over trial	Urology residents	7	Laparoscopic radical and partial nephrectomy, dismembered pyeloplasty, renal cyst decortication	28	5 min of electrocautery simulation task training on virtual reality surgical simulator and 15 min of laparoscopic suturing on basic laparoscopic pelvic box trainer, approximately 1 h prior to surgery	14	5
Chen et al,7 2013, USA	Randomised control trial (randomised by surgical case)	Obstetric and gynaecological residents	29	Laparoscopic surgery for supracervical and total hysterectomies and adnexal/tubal surgery	91	Standard Training PLUS 15 min of low fidelity laparoscopic training of three tasks, immediately prior to surgery	15.5	6
Moldovanu et al,17 2011, Romania	Randomised control trial (randomised by surgical case)	Attending surgeon	1	Laparoscopic cholecystectomy	20	15–20 min of training on a virtual reality simulator with seven tasks, less than 15 min prior to surgery	11	5
Mucksavage et al,18 2012, USA	Before-after study	Attending surgeon	1	Laparoscopic partial and radical nephrectomies	75	15–20 min of training using pelvic suturing exercises, prior to surgery	11.5	3

*All studies described control group as receiving ‘no specific warm-up/training’ except Chen et al⁷ (control=low fidelity training of 10 trials of three tasks at beginning of gynaecology rotation) and Kessler et al¹⁶ (control=viewed video and solitary training session on infant lumbar puncture trainer at start of residency until master performance standard achieved).

MERSQI, Medical Education Research Study Quality Instrument; mNOS, modified Newcastle-Ottawa Scale; PICU, paediatric intensive care unit.

Excluded studies

Two relevant studies were excluded from our review based on study design.^{28 29} Willaert et al²⁸ found that patient-specific rehearsal prior to operation may be helpful in preoperative preparation and operative performance, and Desender et al²⁹ found that patient-specific rehearsal prior to endovascular aortic aneurysm repair to be feasible and to have face validity, but neither study design included a formal no JiT training control group.

Methodological quality of included studies

The general methodological quality of the eight included studies was high based on both assessment tools. Just one study scored below 4 of 6 on the mNOS and none scored below 11 of 15 on the MERSQI (table 1).^{17 18} Given the intervention being studied was training, blinding of the participants was not possible. Blinding of the assessor, however, was performed in five of the studies.^{7 9 15–17} Two studies evaluated the intervention on a single surgeon, which limited the representativeness of the findings.^{17 18}

Overall provider performance

All eight studies reported at least one measure of provider performance, but each used unique measures for the specific procedure being studied. Measures included procedure success rate, components of validated performance scales including the Objective Structured Assessment of Technical Skill (OSATS)¹⁵ and the Vassiliou scale (VAS),⁹ surgical time, need for corrective prompts and number of attempts for success. Overall, five^{7 15 17–19} of the eight studies favoured JiT simulation training compared with no JiT training while three^{9 10 16} studies were equivocal (table 2).

Table 2.

Qualitative summary of key provider performance and patient complications for JiT versus no JiT training

Study (year)	Provider/learner	Procedure	Provider performance		Patient complications
			Measure(s)	Favours	Measure(s)	Favours
Scholtz et al19 (2013)	Nurses	Central venous catheter dressing change	Corrective prompts	JiT	Central line-associated blood stream infections	JiT
Nishisaki et al10 (2010)	Paediatric and emergency residents	Endotracheal intubation in PICU	Success rate	Equivocal	Tracheal intubation-associated events	Equivocal
Kessler et al16 (2015)	Paediatric and emergency residents	Infant lumbar puncture	Success rate	Equivocal	–	–
Calatayud et al15 (2010)	Surgical residents	Laparoscopic cholecystectomy	OSATS7	JiT	–	–
Lee et al9 (2012)	Urology residents	Laparoscopic radical and partial nephrectomy, dismembered pyeloplasty, renal cyst decortication	Vassiliou Scale	Equivocal	–	–
Chen et al7 (2013)	Obstetric and gynaecological residents	Laparoscopic hysterectomy and adnexal/tubal surgery	OSATS6, Vassiliou Scale, Kundhal Scale	JiT	–	–
Moldovanu et al17 (2011)	Attending surgeon	Laparoscopic cholecystectomy	OSATS3, Vassiliou Scale2	JiT	–	–
Mucksavage et al18 (2012)	Attending surgeon	Laparoscopic partial and radical nephrectomies	Surgical time	JiT	EBL, Narcotic use, LOS, change in creatinine, ischaemic time	Equivocal

For each study, the preceding provider performance/patient complication measure was used to determine the predominant finding in the favours column (JiT, equivocal, no JiT).

–, not reported; EBL, estimated blood loss; JiT; just-in-time; LOS, length-of-stay; OSATS, Objective Structured Assessment of Technical Skill (subscript indicates dimensions studied); PICU, paediatric intensive care unit; subscripts indicate dimensions studied.

Details of provider performance

Procedure success rate

Only two of the eight studies reported procedure success rates, both were non-operative bedside procedures.^{10 16} Kessler et al,¹⁶ who evaluated JiT versus no JiT training for paediatric and emergency room residents who perform infant lumbar punctures in the emergency room, reported no significant difference in overall success rates (38% vs 35%, p>0.05) but did report a decreased mean number of attempts (1.4±0.6 vs 2.1±1.6, p<0.01). Nishisaki et al¹⁰, who evaluated JiT versus no JiT training for paediatric and emergency room residents who perform endotracheal intubations in paediatric ICU patients, reported no significant difference in intubation success rates based on first attempt (50% vs 62.5%, p=0.44) or overall (57.5% vs 75%, p=0.19; table 3).

Procedural time

Two studies used surgical time as a measure of procedural performance.^{9 18} Lee et al⁹ found no significant difference for the time to complete certain portions of various urological procedures including laparoscopic radical/partial nephrectomy, dismembered pyeloplasty and renal cyst decortication. Mucksavage et al¹⁸reported statistically significant improvements in surgical time comparing JiT versus no JiT training for both partial nephrectomies (227 vs 282 min, p=0.004) and radical nephrectomies (243 vs 290 min, p=0.02; table 3).

Validated performance scales

Four studies, all evaluating operative/surgical procedures, used various validated scales to assess procedural performance.^{7 9 15 17} Two of these studies, Calatayud et al¹⁵evaluating JiT training among surgical residents performing laparoscopic cholecystectomy and Chen et al⁷evaluating JiT training among obstetrics/gynaecology residents performing laparoscopic hysterectomy and adnexal/tubal ligation, used a modified OSATS to evaluate areas such as handling of equipment and tissue, use of assistance, and knowledge of procedure. Both studies found statistically significant differences in final OSATS scores favouring JiT over no JiT training (28.5 vs 19.3, p=0.042 and 22.5 vs 19.5, p≤0.001, respectively, both out of 30 possible points^{7 15}). Two studies used sections of the VAS score.^{7 9} Chen et al⁷used sections related to laparoscopic-specific dexterity, efficiency and tissue handling and found statistically significantly higher mean scores with JiT versus no JiT training (18 vs 15.5 out of 25 points, p≤0.001). Lee et al⁹ evaluated JiT training among urology residents performing laparoscopic radical and simple pyeloplasty and cyst decortication; the study considered only the suturing and knot-tying portion of the scale and found no statistically significant difference between JiT and no JiT training (3.50 vs 3.54, maximum score not provided, p=0.92). Lastly, Moldovanu and colleagues evaluated JiT training for an attending physician performing laparoscopic cholecystectomy and combined elements of the OSATS and VAS to make a Global Rating Scale ranging from 6 to 30. They found improvement in the overall impression scores which ranged from 1 to 5, favouring JiT over no JiT training based on two evaluator assessments (4.86 vs 4.0, p=0.004 and 4.86 vs 4.37, p=0.059)¹⁷ (table 3).

Other provider performance measures

Kessler et al¹⁶ also reported significant improvement in analgesia use prior to spinal needle insertion (68% vs 19%, p<0.001) and a non-significant improvement in early (appropriate) stylet removal (69% vs 54%, p=0.06). Scholtz et al¹⁹ evaluated JiT training for nurses who performed central line dressing changes and reported the percentage of dressing changes that required corrective prompts by observers was significantly lower following JiT versus no JiT training (9% vs 21%, p<0.001; table 3).

Patient complications

Three of the eight studies reported rates of procedural complications, one of which reported improvement after using JiT simulation training and two of which found no difference (table 2).^{10 18 19} Scholtz et al¹⁹ reported significant improvement in central line-associated blood stream infections, which were reduced to 2.9/1000 line days after the implementation of JiT training compared with 5.3/1000 line days at baseline (p<0.001). On the other hand, Mucksavage et al¹⁸ reported non-significant differences between patients whose attending surgeon performed JiT training prior to laparoscopic nephrectomy compared with patients whose attending surgeon did not perform the training for multiple variables including estimated blood loss, narcotic use postoperatively, length of hospital stay, change in creatinine levels and renal ischaemic time. Nishisaki et al¹⁰ reported no significant difference in tracheal intubation associated events, 32.5% vs 20.8% (JiT vs no JiT, respectively, p=0.4).

Subgroup analyses

Subgrouping the studies by provider level found one study evaluated nurses (favoured JiT),¹⁹ five studies evaluated residents (two favoured JiT, three equivocal)^{7 9 10 15 16} and two studies evaluated attending (both favoured JiT).^{17 18} Subgrouping by procedure complexity found three studies were done at the bedside (one favoured JiT, two equivocal)^{10 16 19} and five studies were done in the operating room (four favoured JiT, one equivocal).^{7 9 15 17 18} Subgrouping by location was not possible as location was often unreported. The number of contributing studies was low and variables differed across studies, making it difficult to isolate the effect of a single variable.

Sensitivity analyses

None of our sensitivity analyses based on methodological quality led to changes in our overall qualitative impressions, though the number of contributing studies was low.

Publication bias

After ordering the studies by size, no relationship between sample size and effect size was noted. Just as the varied outcome measures used across different studies precluded quantitative pooling, it did not allow for construction of a funnel plot.

Discussion

Summary of main results

This is the first review to describe the effect of JiT simulation training using a systematic approach, rigorous inclusion criteria and clinical outcomes. Although quantitative meta-analyses were not possible due to the heterogeneity of procedures and outcome measures, several important qualitative findings emerged. Overall, we found that compared with no JiT training, the use of JiT simulation training was an effective means of improving measures of procedural performance across a range of surgical and non-surgical procedures, performed by a variety of provider types. Unexpectedly, subgroup analyses revealed that for residents performing paediatric intubation and infant lumbar puncture—two low-frequency/high-acuity bedside procedures—improvement in success rates was not seen. Only three studies provided data for patient complications, among which just one found improvement with JiT simulation training. No direct harms of JiT simulation training, such as delayed procedure initiation, were mentioned or described in any of the studies.

Our goal was to evaluate the impact of JiT simulation training for procedures on outcomes assessed on actual patients and thus advance JiT simulation-based knowledge. In order to achieve this goal, we limited our inclusion criteria to only T2 and T3 level outcomes, knowing we would vastly decrease the number of eligible studies. We a priori accepted this anticipated decrease in quantity for an increase in the clinical relevance of the included studies. We believe it is useful to know that there are only eight studies, which evaluated the impact of JiT training on clinical outcomes.

Overall completeness and applicability of evidence

Currently available high-quality studies provide a broad perspective on the impact of JiT simulation training on provider and patient outcomes. However, the ability to draw firm conclusions for any one procedure, training approach, or provider group remains challenging. While the findings were generally consistent across the range of interventions and populations studied, we caution against generalising the findings to procedures and participants beyond those represented in our review. With regard to applicability, JiT training seems to be most appropriate as refresher training, with consideration for short training time and readily available relatively low-cost simulation equipment, we speculate that most interventions would be feasible; however, some of the interventions might be difficult to implement at scale due to the time and/or resources required. Although no delays in procedure initiation were described, we understand in high-acuity high-volume settings delays in care could occur as a result of JiT training in a downstream ‘cascade effect’. While this ‘cascade effect’ has not been described as a consequence of JiT training, utilisers of JiT training must be cognisant of potential consequences the utilisation of time and resources may have on other patients.

Quality of the evidence

The quality and rigour of the included studies were generally strong as indicated by MERSQI and mNOS scores. The directional effect of JiT simulation training on provider performance was relatively consistent; however, the varying nature of the procedures made it impossible to aggregate and summarise the findings in a concise, quantitative manner. The lack of assessor blinding in four of the included studies^{10 16 18 19} could have made JiT simulation training appear more favourable, as could have the single-surgeon intervention design in two of the studies.^{17 18} It should be noted that secular trends were likely occurring during the Sholtz study, which may have contributed to the observed reductions in central line-associated blood stream infection rates.¹⁹

Potential biases in the review process

We performed an extensive search of the literature and conducted all of our detailed screenings in duplicate fashion, with at least one experienced clinician in each pair. While using a single reviewer to perform the title and abstract screening could have resulted in missing one or more eligible studies, we sought to prevent this by only excluding obviously irrelevant studies at this step. The language used to describe the use of JiT simulation training is not standardised, which could have introduced bias by obscuring relevant studies. For example, other terms used to describe interventions similar or identical to JiT simulation training are ‘refresher training’, ‘warm-up’ and ‘rehearsal’. JiT simulation training can also entail a variety of technologies, ranging from computer simulations and virtual reality trainers to physical manikins. Additionally, our reliance on qualitative analysis methods necessarily introduced some subjectivity, as it required us to make decisions about what effects were important. Finally, poor reporting of patient complications limited our ability to draw reliable conclusions.

Agreements and disagreements with other studies or reviews

The fact that all of the eight included studies were published in the past 5 years suggests that JiT simulation training is a relatively novel concept, and the generally favourable findings likely forecast a trend towards increasing use over time. Additional factors likely to increase the use of JiT training include demand for improved efficiency and individuality in learner education, increased efforts to improve patient safety, and recognised need for ongoing refresher education.

The unexpected finding of no benefit for the two high-acuity/low-frequency procedure studies may be related to the fact that these studies focused on learners at the stage of establishing initial competency, while other successful studies were evaluating the impact of JiT training as refresher training or warm-up for procedures on which the providers had already established competency. In a recent article by Sawyer et al,³⁰ an evidence-based pedagogical framework for procedural skill training is presented and discussed where learning is broken down into first a cognitive phase which includes ‘learn’ and ‘see’, followed by a psychomotor phase including ‘practice’, ‘prove’, ‘do’ and finally a maintenance phase. Our systematic review suggests that the current JiT training may be beneficial in the maintenance phase as a refresher but not adequate for the cognitive and psychomotor phases of initial procedural learning.³⁰ The included studies fit into the broader body of evidence supporting the idea that mentally visualising tasks and revisiting learned skills prior to performance can be beneficial.¹⁴

Conclusions

We found that JiT simulation training leads to improved procedural performance in a variety of clinical situations, and while a hard link between these improvements and patient outcomes remains elusive, we believe that this evidence supports consideration of JiT simulation programmes and initiatives.

While continued and increased use of JiT training is justified based on procedural performance findings, we strongly advocate for further research including evaluation of patient-related outcomes. More research on the timing and duration of the training is needed to establish benchmarks for training programme design.