We live in a virtual world: Training the trainee using an integrated visual reality simulator curriculum

Abstract

Background

Gynaecology trainees struggle to obtain adequate procedural experience. Training programs integrating virtual reality simulators (VRS) have been suggested as a solution.

Aims

The study aimed to assess if a VRS training program (LapSim^®, Surgical Sciences, Göteborg, 2017) improved live operating performance at six months for novice and experienced trainees. Additional outcomes included the association between LapSim^® logged time and live operating performance at six months, LapSim^® scores and live operating performance at zero and sixmonths and the difference in benefit for novice and experienced gynaecology trainees.

Methods

A prospective intervention study was conducted. Novice and experienced trainees were enrolled, and comparisons made at zero‐ and six‐month time points. The intervention groups were provided with a laparoscopic gynaecology curriculum incorporating VRS. Controls underwent routine training only. Assessment of live operating performance was conducted after six months training.

Results

Thirty‐five trainees participated, and 25 had access to the VRS curriculum (17 novice and eight experienced trainees). Access to the VRS curriculum and time spent training on the LapSim^® made no difference to live operating ability for either intervention group (P > 0.05). The median (interquartile range) hours of VRS usage were 7.9 (4.5–10.8) and 6.0 (4.0–6.8) for novice and experienced trainees respectively. The intervention group provided positive feedback on the utility of VRS in their laparoscopic skill development.

Conclusion

Optimal utilisation of VRS in Australian training paradigms remains incompletely understood. Further research is required to establish the most effective integration of VRS into training models to ensure uptake and transferability to the operating theatre.

INTRODUCTION

Simulation technologies aim to ‘evoke or replicate substantial aspects of the real world in a fully interactive manner’. ¹ Virtual reality simulators (VRS) with in‐built haptics have been added to existing trainee learning tools. ² , ³ , ⁴ In gynaecology, minimally invasive techniques are the standard modality for most pelvic surgeries, but access to surgical case numbers for trainees is an ongoing challenge. ⁴ A widely accepted solution is to supplement live operating experience with simulation technology. ⁵ , ⁶ , ⁷

The benefit of VRS in laparoscopic surgical education has been assessed in both general surgery and gynaecology. Reportedly, VRS training improves trainee scores on the simulator, ⁸ , ⁹ , ¹⁰ demonstrates a reliable proficiency‐based assessment device, ¹¹ , ¹² and improves live operating efficiency and proficiency. ⁷ , ⁸ , ¹³ , ¹⁴ VRS may also be associated with fewer operative errors and potential for reduced patient harm. ³ , ⁷ , ¹⁴ , ¹⁵ , ¹⁶ , ¹⁷ , ¹⁸ Furthermore, evidence suggests a structured training curriculum improves the learning curve, operating time and trainee confidence. ¹⁴ , ¹⁹ , ²⁰

Based on this evidence, a VRS‐integrated curriculum was piloted in a teaching hospital. This study aims to assess if the implementation of a LapSim^® VRS (Surgical Sciences, Göteborg, Sweden, 2017) training program improved live operating performance at six months for novice trainees compared to controls, and improved live operating performance at six months for experienced trainees compared to baseline.

MATERIALS AND METHODS

A prospective cohort study was conducted to assess the efficacy of the curriculum for gynaecology trainees at a tertiary centre in Australia. Trainees employed at the centre from February 2018 through August 2018 were enrolled into the training cohort. These included ‘novice’ trainees – the novice‐trained group (NT), made up of pre‐vocational residents and first and second year registrars – and ‘experienced’ trainees – the experience‐trained group (ET), comprising senior trainees in year three or above of accredited training – who had performed at least ten operative laparoscopies as the primary operator. These trainees were granted access to the VRS‐integrated curriculum. Trainees at other hospitals were the ‘untrained’ controls – novice‐controls (NC) – who over the six‐month trial period received standard gynaecology training, in accordance with national standards (Fig. 1).

Figure 1.

Participant flowchart, indicating the intervention over the six‐month training period. OSA‐LS, Objective Structured Assessment of Laparoscopic Salpingectomy

Curriculum and assessment

The curriculum included didactic teaching on anatomy and principles of safe laparoscopy, two three‐hour workshops, and 24‐hour access to traditional box trainers and LapSim^® VRS. The VRS was programmed with four modules, each with multiple exercises of graded complexity. Basic skills included: camera and instrument navigation, coordination, grasping, cutting, clipping, bowel handling, fine dissection, sealing and cutting, and suturing. Gamification was utilised in a ‘Precision & Speed’ exercise. Procedure simulation included: tubal occlusion, salpingectomy and myoma suturing. In‐built metrics marked each exercise, as were used by Larsen et al. ³ Two investigators (LE and SM) were available for supervision. Trainees signed ‘in’ and ‘out’, verified through electronic logins. The training curriculum was based on a published Delphi consensus. ⁵ In the trial by Larsen et al., ³ the four modules took trainees 8–24 hours to complete. NT and ET were asked to dedicate at least 30 min/week to simulation training.

All trainees completed a baseline questionnaire, incorporating free‐text descriptive responses and visual analogue scales (VAS), and a baseline VRS assessment using the LapSim^® salpingectomy module. ET also performed a video‐recorded laparoscopic salpingectomy in the operating theatre, assessed using the Objective Structured Assessment of Laparoscopic Salpingectomy (OSA‐LS) ²¹ framework marked by blinded assessors (LE, ER, and AP). The mean score from the three assessors was recorded for each domain, after assessing for consistency.

At six months, all participants completed a questionnaire, logbook review, and a single LapSim^® salpingectomy paired with a live laparoscopic salpingectomy assessed using the OSA‐LS. Surgical complexity was standardised with both trainee and supervising surgeon grading the difficulty of the case to allow for adjustment.

Statistical analysis

Data were summarised as mean (SD), median (25th–75th percentiles) and number (%) according to type and distribution. Between‐group differences were tested using Wilcoxon rank sum test. Changes in LapSim^® scores were adjusted for baseline score by including the latter as a covariate in a linear regression model. The significance level was two‐sided and set at 0.05 for all comparisons with raw P‐values adjusted for multiple comparisons using the Holm‐Sidak adjustment. Statistical analysis was performed using Stata v15 (StataCorp. 2017, Release 15, College Station, TX, USA).

Ethics

Approval was obtained from Mercy Health Human Research Ethics Committee (Ref: 2018–002).

RESULTS

Of the 35 study trainees, 25 had access to the VRS‐integrated curriculum: 17 NT and eight ET trainees (Fig. 1).

Baseline data

Thirty‐four participants (97%) completed the baseline questionnaire. All participants completed a baseline LapSim^® salpingectomy, and eight (100%) ET performed a video‐recorded laparoscopic salpingectomy at enrolment. Trainee baseline characteristics are presented in Table 1. On baseline LapSim^® assessment, there was no difference between NT and NC (adjusted P‐values > 0.05). Notably, ET were indifferentiable from novice groups on LapSim^® assessment (adjusted P‐values > 0.05).

Table 1.

Baseline training experience and LapSim® scores

	Group NT ‘novice‐trained’ (N = 17)	Group NC ‘novice control’ (N = 10)	Unadjusted P‐value* (NT vs NC)	Group ET ‘experienced‐trained’ (N = 8)	Unadjusted P‐value* (NT + NC vs ET)
Baseline training characteristics
Level of training, years	1 (0–1)	1 (0–1)	0.42	4.5 (4–5.5)	<0.001 ‡
Total obstetric/gynaecology exposure, years	2 (1–2)	2 (1–3)	0.59	5.5 (4–6.5)	<0.001 ‡
Primary operative laparoscopy experience, n	0 (0–4)	1 (0–8)	0.39	70 (37–107.5)	<0.001 ‡
First assist lap experience, n	2 (1–9)	15 (2–17)	0.18	30 (9–64)	0.003 ‡
Prior laparoscopic salpingectomy experience, n	0 (0–1)	0 (0–3)	0.32	16.5 (9–25)	<0.001 ‡
Prior virtual reality simulator exposure, hours	0 (0–0.75)	0 (0–0)	0.32	1.75 (0.75–3.5)	0.004 ‡
Baseline LapSim® scores
Total time, sec	203 (156–300)	257 (209–357)	0.29	199 (143–262)	0.43
Blood loss, mL	6 (0–26)	18 (2–41)	0.36	1 (0–7)	0.02 ‡
Rate of bleeding, mL/sec	0.2 (0.0–0.2)	0.1 (0.1–0.3)	0.65	0.0 (0.0–0.2)	0.14
Ovarian diathermy damage, sec	0 (0–0)	0 (0–2)	0.27	0 (0–20)	0.32
Tube distance cut, mm	14 (13–14)	13 (13–14)	0.22	13 (12–14)	0.32
Vessel damage, n, median (interquartile range)	0 (0–0)	0 (0–1)	0.72	0 (0–0)	0.11
Left instrument path, cm	90 (70–130)	160 (110–210)	0.10	70 (60–100)	0.15
Right instrument path, cm	290 (240–450)	360 (340–460)	0.28	280 (230–350)	0.22
Left instrument angle, degrees	228 (151–322)	355 (202–592)	0.37	223 (165–279)	0.88
Right instrument angle, degrees	523 (390–1023)	698 (604–999)	0.16	451 (391–627)	0.21
Left instrument out of view, n	0 (0–0)	0 (0–0)	0.74	0 (0–0)	0.60
Right instrument out of view, n	0 (0–2)	2 (0–2)	0.40	1 (0–2)	0.53
Left instrument out of view, sec	0 (0–0)	0 (0–0)	0.74	0 (0–0)	0.60
Right instrument out of view, sec	0 (0–1)	1 (0–1)	0.45	0 (0–1)	0.61
Total score out of 100	93 (84–99)	81 (73–91)	0.10	93 (89–96)	0.61

Data presented as median (25th – 75th percentile).

^*Wilcoxon rank sum P‐value, unadjusted for multiple comparisons.

^‡ Comparisons remain significant (P < 0.05) after adjustment using Holm‐Sidak step‐down (six hypothesis tests).

^‡ Adjusted P‐value using Holm‐Sidak step‐down (15 hypothesis tests) is non‐significant, P > 0.05.

For ET, at the unadjusted significance level (P < 0.05), there was an association between total time to perform a baseline live salpingectomy and total VRS‐salpingectomy score, with each point improvement in baseline VRS score being associated with a six seconds shorter time to perform the salpingectomy (P = 0.024). However, following appropriate adjustment, this association did not remain significant. There were no other associations noted between baseline LapSim^® scores and live operating scores.

Six‐month assessments

Twenty‐eight trainees were retained through the six‐month study period. Seven trainees withdrew participation: four NT and three NC.

Time on the LapSim^® was logged over the study period (Table 2). There was no significant difference in logged VRS time between the training groups (P = 0.29). Nine (69%) NT and six (86%) ET made it to the third LapSim^® training module, and one trainee in each group completed all four modules.

Table 2.

Trainee six‐month logbook

	Group NT ‘novice‐trained’ (N = 14)	Group NC ‘novice control’ (N = 7)	Unadjusted P‐value* (NT vs NC)	Group ET ‘experienced‐trained’ (N = 8)	Unadjusted P‐value* (NT + NC vs ET)
Logged time virtual reality simulators training, h	7.9 (4.5, 10.8)	NA	NA	6.0 (4.0, 6.8)	0.29
Operative laparoscopy, primary operator, n	1 (0, 3)	2 (0, 9)	0.43	20 (20, 31)	0.004 †
Operative laparoscopy, first assistant, n	0 (0, 4)	3 (1, 15)	0.10	16.5 (0, 33)	0.27
Laparoscopic salpingectomy, n	1 (0, 1)	1 (0, 3)	0.93	3.5 (1, 9)	0.02

Data presented as median (25th, 75th percentiles).

^*Wilcoxon rank sum P‐value, unadjusted for multiple comparisons.

^† Comparison remain significant (P < 0.05) after adjustment using Holm‐Sidak step‐down (4 hypothesis tests).

Twenty‐seven trainees completed the six‐month questionnaire and logbook review: 13 NT, seven ET, and seven NC. Trainee logbook data are presented in Table 2. Over the six‐month study period, NT and NC performed fewer operative laparoscopies than ET (unadjusted P = 0.004).

The total time to complete a LapSim^® salpingectomy reduced at six months (mean time improvement 61 sec, 95% CI −116 to −5, P = 0.034) for NT and ET but there were no other associations between LapSim^® usage and other LapSim^® domains at six months (adjusted P‐values > 0.05 for all variables).

Live operating scores are presented in Table 3. A significant difference was seen between NT and ET, with ET demonstrating superior general laparoscopy skills scores (adjusted P = 0.001 for OSA‐LS ‘confidence of instrument handling’ domains). There was a trend toward higher overall OSA‐LS scores (unadjusted P = 0.009) but this did not remain significant after adjustment. Scores for both trained groups remained well below expected3, 15 with even ET scoring a median (IQR) score of 29.3 (26.3–32.0) out of 45. Indications for salpingectomy and case complexity were distributed equally between groups. The amount of assistance (as graded by both trainee and consultant) was higher in novice groups than ET (P = 0.01). Increased consultant input was associated with lower OSA‐LS scores and increased operative time (P < 0.004 for all regression analyses).

Table 3.

Objective Structured Assessment of Laparoscopic Salpingectomy (OSA‐LS) scores at six months

	Group NT ‘novice‐trained’ (N = 14)	Group NC ‘novice control’ (N = 7)	Unadjusted P‐value (NT vs NC)	Group ET ‘experienced‐trained’ (N = 8)	Unadjusted P‐value (NT + NC vs ET)
OSA‐LS Economy of Movements †	1.7 (1.3, 2.2)	2.3 (1.7, 3.3)	0.18	3.2 (2.8, 3.7)	0.006
OSA‐LS Confidence of Instrument Handling †	1.7 (1.2, 2.3)	2.3 (1.7, 3.3)	0.17	3.3 (3.2, 3.3)	0.004 §
OSA‐LS Economy of Time †	2.2 (1.8, 2.7)	2.3 (2.0, 3.3)	0.31	3.7 (2.7, 4.2)	0.02
OSA‐LS Errors & Respect of Tissue †	2.7 (2.2, 3.0)	2.7 (2.7, 3.3)	0.39	3.3 (2.7, 3.7)	0.07
OSA‐LS Flow / Operative Technique †	2.2 (1.8, 2.7)	2.7 (2.3, 3.7)	0.11	3.3 (3.0, 3.7)	0.008
OSA‐LS Presentation of Anatomy †	2.5 (2.0, 2.8)	3.0 (2.7, 3.7)	0.10	3.3 (2.8, 3.5)	0.02
OSA‐LS Use of Diathermy †	2.8 (2.0, 3.0)	2.7 (2.3, 3.7)	0.30	3.5 (2.5, 3.7)	0.09
OSA‐LS Dissection of Fallopian Tube †	2.3 (1.7, 2.7)	3.0 (2.7, 3.7)	0.03	2.7 (2.3,3.0)	0.02
OSA‐LS Care for Ovary/Pelvic Side Wall/Ovarian Artery †	2.7 (2.0, 3.2)	3.0 (2.7, 3.7)	0.09	3.5 (3.0, 3.7)	0.01
OSA‐LS Total Score ‡	21.5 (16.3, 23.2)	21.7 (21.0, 30.7)	0.18	29.3 (26.3, 32.0)	0.009
OSA‐LS Total Time, sec	9.3 (7.8, 11.9)	9.3 (7.2, 12.1)	0.50	5.8 (4.2, 8.1)	0.04

Data presented as median (25th, 75th percentiles).

^† Score /5.

^‡ Score /45.

^§ Only this comparison remains significant (P < 0.05) after adjustment using Holm‐Sidak step‐down (11 hypothesis tests).

For NT, having access to our LapSim^®‐integrated program did not confer a benefit in live performance compared to NC. There was no association between the access to the LapSim^® and six‐month OSA‐LS scores for novice trainees (P > 0.003 for all regression analyses). On paired analysis, ET demonstrated no improvement in OSA‐LS scores over the six‐month training period, irrespective of LapSim^® logged time (all Holm‐Sidak adjusted, P > 0.23). There was no association found between time on the LapSim^® and six‐month OSA‐LS scores for either of the VRS‐trained groups (adjusted P‐values >0.05 for all regression analyses). There was a single association found between LapSim^® scores and OSA‐LS scores at six months; ‘Blood Loss’ (mL) on the VRS was inversely proportional to score for ‘care for the ovary, ovarian artery and pelvic side wall’ on OSA‐LS (adjusted P‐values < 0.0001).

Trainee feedback on curriculum and barriers to training

Only one NT and one ET reported a belief that they had spent adequate time using the VRS. The remaining trainees indicated their reasons for inadequate VRS usage: rostering issues (60%), rotation to other sites (20%), life commitments (10%), leave (5%) and lack of incentive (5%). All trainees reported use of the simulator predominantly after hours and for those who managed in‐hours training, the majority was unprotected with frequent interruptions. Trainees reported inadequate self‐motivation and suggested assessments and defined expectations may increase uptake.

The NT group reported a mean usefulness of the curriculum of 7.7 (1.66) out of ten. ET reported similar usefulness with a mean VAS of 8.4 (0.74) (P = 0.30). Trainees strongly supported the program, with 100% of intervention participants reporting a need for a standardised laparoscopic curriculum.

Trainees suggested key drivers to motivate use of a curriculum: quality of simulation, the use of case scenarios, competitions and rewards, and live operating ‘incentives’ for diligent trainees.

The most reported barrier to operative ‘real‐time’ gynaecology training was obstetric workload, which either eroded into protected teaching time or fatigued trainees. Other barriers included infrequent gynaecology theatre opportunities and length of time between cases. NT reported insufficient ‘basic’ skills to enable participation even when offered the opportunity. ET stressed that inconsistent supervision prevented skill development for more complex tasks.

DISCUSSION

Principal findings

In the context of inadequate utilisation of LapSim^® training, access to the VRS‐integrated curriculum resulted in no improvement in operating skill when using laparoscopic salpingectomy as a proxy for laparoscopic surgical proficiency. NT were unable to be differentiated from NC during live operating assessment, and ET demonstrated no operating skill improvement over time. Moreover, the LapSim^® salpingectomy module was an overall poor predictor of operative performance.

In an environment where simulation training was not prioritised, and protected training time not rostered, uptake of VRS training was poor. Despite a specifically designed training program and incorporation of state‐of‐the‐art technology, many trainees did not complete more than a few hours of simulation training. It is thus problematic to assess our principal outcomes, as utilisation may have been insufficient to see the true impact of VRS training on live operating performance. However, our study demonstrates that trainees will not acquire adequate gynaecological surgical skill using the traditional apprenticeship model.

The results of this study differ from prior simulation studies that demonstrate successful translation to the operating theatre with improved performance after simulation training, reduced operating time, reduced length of stay and fewer errors following simulation training. ³ , ¹³ , ¹⁴ , ²² Unfortunately, the current literature pertaining to transferability of simulation curriculum to the operating theatre is at high risk of bias. ²²

Shore et al. ⁵ , ¹⁵ designed a VRS‐integrated training program for gynaecology residents. Following completion of the curriculum, the trained group showed a higher level of technical proficiency than residents in the conventional group. Despite basing our training curriculum on Shore's²² design, mean trainee OSA‐LS scores in our study were significantly inferior to those previously reported. This is particularly concerning in the ET group, as this group had previously been assessed as ‘competent’ at performing a routine laparoscopic salpingectomy, as part of standard Royal Australian and New Zealand College of Obstetricians and Gynaecologists training.

Our study confirms prior findings of improved VRS scores following use of VRS training programs, with trainees being able to perform the LapSim^® salpingectomy approximately one minute faster after training. An observational study of gynaecology trainees ⁸ showed that trainees improved in both LapSim^® procedure time and LapSim^® instrument pathway. In the live operating environment, this translated to a non‐significant reduction in operating time for post‐LapSim^®‐training salpingectomies, similar to the ET group. Another prospective study of 26 gynaecology trainees reported faster procedure times, reduced blood loss and improved subjective assessments of efficiency and coordination with subsequent training sessions. ⁹ Mannella et al. ¹⁰ demonstrated improved VRS scores for juniors and seniors after training, with improvements for junior trainees being greater. However, unless transferable to the operating theatre, improved VRS scores will not translate to improved patient outcomes.

Clinical implications

VRS use in surgical education is reportedly superior to conventional workplace training. ¹¹ , ¹⁶ , ²³ However, it must first be confirmed that simulation models are a true and useful surrogate for live operating. Unfortunately, our study failed to demonstrate this. Results by Janssens et al. ⁸ in a similar training environment also failed to demonstrate transferability of improved LapSim^® scores to the operating theatre. The randomised controlled trials by Shore et al. ¹⁵ and Larsen et al ..³ reported trainees spending many more hours on the VRS than demonstrated by our trainees, with three‐hour sessions for seven weeks in the Shore et al. ¹⁵ study and a mean of seven hours 15 minutes performing VRS‐salpingectomies in the Larsen et al. ³ paper. Our failure to replicate these results may be explained by the pragmatic study design that was incorporated into real‐life hospital training. Thus, it is questionable whether prior positive results are reproducible in the real‐world training paradigm, unless this paradigm changes.

Research implications

Despite attempts to improve curriculum engagement (such as easy access for opportunistic use, and scheduling supervised sessions), trainees struggled to utilise the curriculum. Unfortunately, the benefits of the LapSim^® cannot be accurately assessed if training is not accessed. Burden et al. ²⁴ also found their simulation curriculum was positively received, although no trainee completed more than two‐thirds of the curriculum. External motivators, such as the live operating rewards, have already been trialled with limited success. ⁸ , ²⁵ The role of VRS training as a pre‐theatre hurdle has been incorporated into several surgical settings. ³ , ²⁶ A review by Gostlow et al. ²⁷ confirmed that providing unrestricted access to simulator equipment is inadequate in motivating trainees to voluntarily participate in laparoscopic skills training. Future research endeavours should tease out these issues.

Strengths and limitations

A strength of this study is its pragmatic design, allowing a feasibility assessment of our simulation program implemented within the constraints of usual service provision. Training was not mandated but rather encouraged, and thus a picture of trainee self‐motivation in our Australian climate is clear. There is extensive literature highlighting issues with non‐mandatory training: a clear disconnect between trainee intentions and reality is reported. ²⁶ Should a training program like ours be recommended for the future (in our institution or others), then assessment of the program in the setting of mandatory VRS training prior to live operating would be required.

The study groups were not randomised; rather the decision was made to run the study as a cohort study, using trainees from hospitals without a VRS as controls. This design assumes that gynaecological training at other sites within the same city is similar. Given training sites are nationally accredited, and analyses controlled for workplace surgical experience, this remains a reasonable assumption. Furthermore, due to ethical concerns, NT were unable to perform a baseline video‐recorded laparoscopic salpingectomy and thus analysis of individual NT benefit in live operating skill following simulation training was not possible.

Baseline trainee data would indicate that novice trainees were significantly different from ET. However, baseline LapSim^® assessments did not differ. The lack of difference may be due, in part, to study design, where trainees had no prior familiarity with the simulator. However, it remains unknown whether the LapSim^® can be used as a reliable differentiator of trainee ability.

CONCLUSIONS

Implementation of a VRS‐integrated training program in a tertiary hospital demonstrated poor participant uptake. In this setting there was no association between access to the curriculum and improved live operating performance at six months. VRS technologies require further investigation to optimise their use and improve understanding of the utility of these as a surrogate for live operating, and how to effectively incorporate them into training.

FUNDING

This project received financial support from the Australasian Gynaecological Endoscopy & Surgery Research Fund (2018 grant recipient) and the Mercy Hospitals Victoria Ltd Small Research Grant Program. Funds were used to support the employment of a research assistant and for statistical analysis and stationery supplies.