Incentivising practice with take-home laparoscopic simulators in two UK Core Surgical Training programmes

Abstract

Introduction

Practice using simulators has been validated as a mean for surgical trainees to improve basic laparoscopic skills and free their attention for higher cognitive functions. However, mere provision of equipment does not result in frequent practice. This study assesses one approach to incentivising practice within core surgical training programmes and leads to further recommendations.

Methods

30 core surgical trainees (CST) starting laparoscopic-based specialties were recruited from East and West of Scotland CST programmes and given take-home laparoscopic simulators, with six training modules. Attainment of target metric scores generated an eCertificate, to be rewarded by progression in the live theatre. Questionnaires assessed confounding variables and explored CSTs’ anxieties about laparoscopy.

Results

27 trainees (90%) agreed to participate (mean age 28 years, range 24–25; 17 males). 13 CSTs (48%) were in the first year of surgical training. 11 (41%) had no previous simulation experience and 7 (32%) CSTs played video games >3 hours/week. 12 of 27 trainees (44%) completed ≥1 task and 7 completed all (26%).

Performances improved in some participants, but overall engagement with the programme was poor. Reasons given included poor internet connectivity, busy rotations and examinations. CSTs who engaged in the study significantly reduced their anxiety (mean 4.96 vs 3.56, p<0.05).

Conclusions

The provision of take-home laparoscopic simulators with accompanying targets did not successfully incentivise CSTs to practise. However, the subgroup who did engage with the project reported performance improvements and significantly reduced anxiety. Proposals to overcome barriers to practising in simulation, including obligatory simulation-based assessments, are discussed.

Introduction

Halsted's time-honoured apprenticeship model of surgical training—with observation, coaching and practice—depended on sheer volume of exposure to caseload, as it arose, often unpredictably, as its cornerstone.¹ Over years, the trainee gradually accumulated knowledge and skills for independent practice. Variability of learning opportunity between posts and rates of learning between trainees was compensated for in most cases by length of time spent in training. However, hours of training have now been strictly controlled by working time legislation, especially in Europe. Whatever benefits this may have brought, there has also been a negative impact on ‘apprenticeship’ learning in surgery: operative training time is at a premium, total training hours have diminished, and duty rosters have become more restrictive.² Even in the USA, where working hours are relatively longer, by the time of completing a general surgery residency (equivalent to core surgical training (CST) in the UK), 30% of trainees were still unable to safely handle tissue laparoscopically, 12% unable to control bleeding and 22% using energy devices inappropriately.³ Recognition of the unreliability of the apprenticeship model along with an increased emphasis on surgical patient safety⁴ has led to increased interest in alternative pedagogic paradigms, notably simulation-based education.

The acquisition of skills to perform complex technical tasks expertly has been described in several models.^{5 6} Fitts and Posner described the process as follows:

1. An initial cognitive phase (attention focused on the technical task, identifying component parts of the task and trying to construct a mental picture of how to complete it).

2. An associative or integrative phase (linking component parts into smooth actions).

3. An autonomous phase in which the motor skills have become intuitive.

These phases are familiar to the surgical trainer. For example, operating lists for training are known to take longer if novice trainees are still in the cognitive phase of learning motor skills. It is well known to successful sports men and women and musicians⁷ that frequent deliberate practice, especially in the associative phase, is essential to gain mastery. The work of K Anders Ericsson famously expounds this principle with respect to craft medical specialties.⁸

In the final autonomous stage, attention is freed for the ‘higher functions’ required performing complex tasks.⁹ In the surgical context, these will include the non-technical skills that are known to be crucial for safe operating, such as situation awareness and decision-making.¹⁰

These observations are nowhere more true than in laparoscopic surgery, a particular discipline demanding spatial awareness, 2D/3D translation and excellent hand–eye coordination. Numerous studies have shown that laparoscopic motor skills can be learnt using portable simulators, and that the benefit of deliberate practice transfers ‘from virtual reality (VR) to operating room (OR)’.^11–13 Number of products are available for this purpose, including basic box trainers, high tech virtual reality simulators and some impressive hybrid devices combining physical tasks with computer-aided metric measures of performance and online learning aids.

However, despite the seemingly obvious benefits of practice using these devices, trainees tend not to avail themselves of the opportunity to practice, even if given ready access to a suitable simulator,^14–17 supervisors still have the experience of watching junior trainees learning basic instrument handling in the operating theatre These findings chime with our own observations within the UK context.

It was this that led us to attempt to incentivise frequent practice on take-home simulators by trainees in the two Scottish core surgical training programmes. We wished to remove the need for basic instrument handling practice from the operating room.

Drawing on the work of Stefanidis et al¹⁸ and van Empel et al,¹⁹ we hypothesised that goal setting linked to practice and the reward of progression in the live operating theatre would incentivise trainees to practise using take-home laparoscopic simulators. Thus, the aim of this feasibility study, the Incentivised Laparoscopy Practice Study (ILPS), was to quantify gains in laparoscopic motor skills of CSTs using take-home simulators and to assess trainee engagement with simulation. Our specific research questions were, first, does incentivising practice by setting metric performance targets and providing an eCertificate to facilitate access to ‘first operator’ tasks in the live theatre improve uptake and effectiveness of frequent practice on a laparoscopic simulator? Our second area of interest was to explore the process of participating in the study with trainees to gather insight into barriers, facilitators and their experiences.

Methods

Design

This was a pragmatic, feasibility study using mixed methods. Quantitative measures and outcomes were important to assess improvement in technical ability, while qualitative methods added vital information regarding the process by which these skills are attained and its acceptability to trainees. Since the ultimate aim of the intervention was to create a component of a simulation programme for all core trainees in Scotland that might require funding and integration into the curriculum, we had to show if it was a method of skill acquisition and an acceptable way of learning for trainees.

Participants

In total, 30 participants were selected from lists of first-year and second-year trainees provided by the East and West of Scotland CST programmes. We included all those who were starting their first post in either general or paediatric surgery (laparoscopy-using specialties). Two trainees declined to take part and one trainee discontinued the study prematurely, expressing a desire for a career change. Of the remaining 27 trainees (15 in the West of Scotland and 12 in the East), 17 were male and 10 female with an average age of 28 years (range 24–35). Out of 27 trainees, 13 were in their first year and 14 in their second year of surgical training.

Equipment

The eoSim model of simulator (eoSurgical, Edinburgh) was selected from a comparison of a number of products (virtual reality and standard box trainers), because it was rated highly by a pilot group of trainees and consultants for portability, ease of use, ergonomics, haptic feedback and cost. It also fulfilled our requirements for instrument tracking software and web connectivity. eoSim is a portable box trainer with a built in HD camera which connects to the user's own laptop computer, allowing the trainee to use their laptop as a screen (online or offline) or to use a smartphone or tablet (offline only). Construct and content validity have previously been well demonstrated using this equipment.^{20 21} The eoSim was used with a computer vision instrument tracking software application called ‘InsTrac’ (eoSurgical, Edinburgh). This maps the movement of instruments within the simulator and generates instrument movement metrics.^{21 22}

Support

Educational supervisors were involved early in the process by letter with information leaflets detailing the project, its aims and what would be expected of them. They were emailed a reminder at the midpoint of the study, to approach their trainees and express their support and offer help where required.

Trainees were recruited by email or at training days. Written consent to participate was taken at face-to-face meetings with a researcher. They were provided with e-instruction packs, a software user guide and questionnaires as described below. These included documenting any potential confounding variables, for example, previous use of simulators, any eoSim courses, previous experience in laparoscopic surgery and posts undertaken to date. On return of questionnaires, they were given the eoSim. Each trainee received an induction session with the researcher, giving particular attention to technical set-up, and some preliminary instrument handling instruction and feedback. They were required to register with a secure website. Technical support was made readily available via a telephone hotline and online.

Simulation content

Once logged in to eoSim, participants had access to a modular series of six validated and standardised tasks to be completed, as listed and illustrated in figure 1, each one with an online instructional video. The metric measurement that was used as the incentive was time spent for task, and a target time was given to be achieved for each task before moving on to the next.²³ The time to complete the tasks was not intended as the best measure of performance but rather an incentive and proof of practice. Other measurements recorded using InsTrac software were motion smoothness, distance between instruments, speed, acceleration, percentage time off screen and handedness. These metrics have been studied in various types of simulator and have demonstrated concurrent and construct validity in studies based on the Fundamentals of Laparoscopic Surgery (FLS) programme in the USA.²⁰ Time taken to complete the task, and the other metric measurements, formed the quantitative outcome measures.

Figure 1.

The tasks.

Target metric scores were stated on the website and the application for running the simulation programmes was also opened here, allowing the investigator to monitor time spent practising as well as the metric measurements recorded by the InsTrac software. Trainees’ scores, uploaded to an online database, were visible only to the trainee and the investigator, in the hope of minimising dishonesty.

To introduce an element of competitive gamification, a Facebook page was created and consenting trainees took part in a picture competition and a league table published there.

On completion of each module, trainees were required to upload via a YouTube channel, an anonymised video performance of the task in order for the researcher to confirm the metric measurements. Achievement of target metric scores resulted in an eCertificate which could then be presented to the trainer, to be rewarded by progression in the live operating theatre, from camera holding to first operator tasks.

Qualitative component

Trainees were asked in online pretest and post-test questionnaires about their previous experiences of laparoscopic surgery, the use of simulation in their training to date, computer game use. Questionnaires included open questions with space for free text comments about feelings regarding laparoscopic surgery. Closed and forced-choice question responses were analysed using descriptive and non-parametric statistics, while open comments were analysed for themes.

Statistical analysis was undertaken using SPSS software (SPSS Statistics, IBM analytics, Armonk, NY):

Self-reported levels of anxiety	Wilcoxon signed rank test
Correlation of video game play and career intention with study engagement	Spearman's rank coefficient
Level of task achievement (skewed from the mean)	Mann-Whitney U test

Results

In total, 12 of 27 trainees (44%) completed one or more tasks, with 7 of these 12 completing all 6 modules (figure 2).

Figure 2.

Number of tasks completed by trainees.

As some of the trainees practised offline using a smartphone or tablet mounted on the simulator, the total time spent practising online and offline was estimated by trainees, and the results are displayed in figure 3. It was clear that trainees chose to practise more offline. The trend in metric scores indicated improvement with practice. However, trainee numbers were too small to be able to draw any meaningful or statistically robust conclusions from these measurements.

Figure 3.

Trainees’ estimates of time spent practising, online and offline.

Participants cited their own reasons for lack of frequent practice. In a minority, these included software incompatibility with older operating systems on personal computers. These trainees (n=3) were provided with a ‘non-tracking’ version of the software, but only one took this up. Further practical reasons for lack of uptake included poor internet connectivity, especially while trainees were moving between cities to take up posts or living in hospital accommodation without Wi-Fi; and other competing professional imperatives, namely mandatory courses, busy rotations and Membership of the Royal College of Surgeons (MRCS) examinations, as barriers to engagement.

One participant stated that, while they felt the incentive was acceptable, s/he had found that the reward (in the form of progression in the live operating theatre) was not readily available in their hospital. Despite written communication with educational supervisors at two points during the project, only five of 25 participants reported that they had been approached by the educational supervisor enquiring about the progress in the study.

Though part of a generic CST programme, trainees were able to declare a subspecialty career intention (table 1). Not all of these intended subspecialties require laparoscopic skills. There was a non-significant association between pursuing a career in General Surgery (including gastrointestinal (GI) surgery) and greater engagement with the process, compared with those with other specialty interests (Spearman's rank coefficient).

Table 1.

Trainees’ career intentions

Intended specialty	No. of trainees	Mean no. of tasks completed per trainee
GI/HPB	11	3.1
Urology	7	1.3
Undecided	4	0
Vascular	2	3
OMFS and ENT	2	0.5
Endocrine	1	0

ENT, ear nose throat; GI/HPB, Gastrointestinal/hepatopancreatobiliary; OMSF, oral and maxillofacial surgery.

A total of 10 trainees had used box trainers previously, either the eoSim itself in one case or similar portable trainers, 2 trainees had some experience with high tech virtual reality simulators, such as the LAP Mentor (Simbionix), and 11 trainees had no previous experience of laparoscopy simulation. There was no correlation between previous simulation experience and engagement in the process.

In total, 7 of the 19 trainees (32%, all male) regularly played video games for >3 hours/week; while, 12 trainees did not (10 male, 2 female). Four trainees chose not to answer this question. A comparison of the task completion rates for gamers and non-gamers is shown in table 2. Although numbers are too small to achieve statistical significance (p=0.45), the trend was for the non-gamers to perform to a higher level in terms of time to task completion.

Table 2.

Career intentions

Intended subspecialty	No. of trainees	Mean no. of tasks completed per trainee
GI/HPB	11	3.1
Urology	7	1.3
Undecided	4	0
Vascular	2	3
OMFS and ENT	2	0.5
Endocrine	1	0

ENT, ear nose throat; GI/HPB, Gastrointestinal/hepatopancreatobiliary; OMSF, oral and maxillofacial surgery.

In the post-test questionnaire, trainees were asked their views on the usefulness of the intervention. The majority of trainees believed, it was a worthwhile use of their time and all believed that the skills would be transferable (figure 4). However, they were divided in the degree to which they thought this might be the case.

Figure 4.

Trainees’ opinions in post-test questionnaire.

Trainees were asked in pretest and post-test questionnaires to rate their anxiety regarding performing laparoscopic surgery on a 10-point Likert scale, where 1 is no anxiety at all and 10 is the most anxiety ever. Pretest trainees reported a mean anxiety score of 4.96. In trainees who engaged with the ILPS, mean post-test anxiety score was 3.56, a statistically significant reduction (p<0.05; table 3). However, free text comments indicated concerns regarding the transferability of skills to the operating theatre remained.

Table 3.

Video game use and levels of engagement and performance

	No. of trainees	Mean no of tasks completed per trainee	Level of task completion
Gamers	7	2.3 (0–6)	Basic	56%
			Advanced	31%
			Elite	13%
Non-gamers	12	1.8 (0–6)	Basic	27%
			Advanced	46%
			Elite	27%

Free text comments

Thematic analysis of participants’ free text comments identified the following themes:

• Technical and follow-through issues

  • For example, incentive of ‘green light’ to progress to first operator tasks in theatre not forthcoming.

• A useful supplement but not the same as real operating.

When asked if they would recommend this programme to other core trainees, participants were generally positive but with the caveat that the incentive being implemented was key. They reported seeing improvement, feeling more confident in their ability to learn motor skills, and noticing change from focus on instrument handling to thinking about what the instruments were doing and anticipating the next steps.

Discussion

There is currently no obligatory simulation-based training in laparoscopy integrated into the UK core surgery training programmes. Most trainees will attend a 2-day basic laparoscopy course with use of box trainers, and some trainees may avail themselves of ad hoc opportunities for further practice.

In this novel, feasibility study, we found that the provision of free home-based laparoscopic simulators with accompanying targets did not successfully incentivise core surgical trainees to practise laparoscopic skills even though participants reported a high level of belief in the value of the intervention. However, the subgroup who did engage with the project reported performance improvements and significantly reduced anxiety. As well as improvement in technical ability, assessing the process by which these skills are attained and the acceptability of the process to trainees is important. Anxiety regarding laparoscopic operating was thought to be a potential driver to frequent practice, and so this was assessed in the qualitative measures before and after the intervention.

The incentive to practice was an eCertificate which would in turn signal trainers to allow the trainee to progress in the live operating theatre, from camera holding to first operator tasks. Thus, the real incentive to practice was the promise of increased laparoscopic operating. Trainees were asked in the questionnaires their opinion towards an eCertificate. The vast majority felt that they would still have participated in the study if an eCertificate was not available.

The training tasks were developed alongside, and optimised for, the eoSim take-home box simulator used in this study. These were designed by surgical trainees with the aim of improving on the tasks used in the FLS programme. They had been previously assessed and shown to have construct, concurrent and content validity in a comparison study with the previously extensively validated FLS task.²⁰ We attempted to address issues associated with non-engagement in laparoscopic skills practice outside the operating theatre by providing set learning goals (time to task completion), giving feedback, ensuring opportunity for practice during working hours using the highly portable equipment, without needing to be online every time and providing the incentive of live operating.^{15 18 19} It was the last of these which was novel and which we felt would make the difference. However, we found that despite our efforts to communicate with trainers across the country, there was not a clear and linear relationship between practice and access.

This opens up several areas of discussion. First, in the assumption that this incentive is appropriate, future studies must ensure full engagement from trainers and their units, so that incentives are forthcoming (or not as the case may be). Exploring the views of trainers towards laparoscopic skills practice outside the operating theatre may be a good first step in planning how to integrate this incentive into future studies. Second, experience in other domains of medical education points to the possibility of additional or alternative incentives. If ‘assessment drives learning’,²⁴ one way to incentivise practise would be to incorporate assessments of task performance in simulation within trainee assessment procedures (in the UK, this refers to competencies reached and recorded in an ePortfolio and reviewed annually). This is not in place currently but could be facilitated via an assessed Objective Structured Assessment of Technical Skill (OSATS).²⁵

However, those who engaged showed objective improvement in performance as well as reduction in anxiety. We cannot explain the difference between ‘engagers’ and ‘non-engagers’ in terms of simple demographics such as gender, stage of training or prior laparoscopic training. It may be that, for future studies, adding measures of self-regulation and self-efficacy, shown to predict engagement with learning and change in other populations, may be useful in order to gain more insight into trainee learning behaviour.

Despite measures such as the use of the most common social media platforms and readily accessible web-based support, participants cited their own technical and logistical reasons for lack of frequent practice. Some of these could be overcome by supplying an integrated simulator and personal computer (PC) system, ready set-up, as is done for example by the SimEndo system. Though somewhat more expensive, it does remove the dependence on trainees’ own computers. Newer versions of the InsTrac software may however be less processor-dependent in the future; and thus, this may become less of an issue in time.

In summary, trainees who engaged with the ILPS process clearly benefited but were in the minority. To improve uptake and effectiveness of practice on take-home simulators, further research should assess the benefits of following:

1. Improving trainer engagement so that achievement of targets is indeed rewarded by progress in the live operating theatre;

2. Supply of an integrated simulator and PC system, if resource allows;

3. Introducing an OSATS-based assessment tool in the ePortfolio of the UK Intercollegiate Surgical Curriculum Project or equivalent and making this or the project eCertificate mandatory at each trainee's Annual Review of Competence and Progression.

Conclusions

Incentivising deliberate practice in simulation is a challenge. Despite provision of free laparoscopic simulators, with defined goals and a clinical reward of progression in the live operating theatre, few trainees took up the opportunity of frequent deliberate practice in their own time. Those who did engage with the project reported improvements in performance and reduction in anxiety. The qualitative component of this mixed methods feasibility study, suggests measures which could be put in place in a revised iteration of the project, notably improving trainer engagement, overcoming technical obstacles, and introducing an obligatory, simulation-based assessment at trainees’ annual reviews.