A Randomized Controlled Trial to Assess the Effects of Competition on the Development of Laparoscopic Surgical Skills

Abstract

Background

Serious games have demonstrated efficacy in improving participation in surgical training activities, but studies have not yet demonstrated the effect of serious gaming on performance. This study investigated whether competitive training affects laparoscopic surgical performance.

Methods

Twenty novices were recruited, and 18 (2 drop-outs) were randomized into control or competitive (CT) groups to perform 10 virtual reality (VR) laparoscopic cholecystectomies (LC). Competitiveness of each participant was assessed. The CT group was informed they were competing to outperform one another for a prize; performance ranking was shown prior to each session. The control group did not compete. Performance was assessed on time, movements, and instrument path length. Quality of performance was assessed with a global rating score (GRS).

Results

There were no significant intergroup differences in baseline skill or measured competitiveness. Time and GRS, at final LC, were not significantly different between groups; however, the CT group was significantly more dexterous than control and had significantly lower variance in number of movements and instrument path length at the final LC (p=0.019). Contentiousness was inversely related to time in the CT group.

Conclusion

This was the first randomized controlled trial to investigate if competitive training can enhance performance in laparoscopic surgery. Competitive training may lead to improved dexterity in laparoscopic surgery but yields otherwise similar performance to standard training in novices. Competition may have different effects on novices versus experienced surgeons, and subsequent research should investigate competitive training in experienced surgeons as well.

Introduction

Surgical training programs are working to adapt their curricula to improve the efficiency of surgical education by augmenting didactic training and intraoperative education with simulation to remain in compliance with Accreditation Council for Graduate Medical Education (ACGME) requirements.¹ Many questions remain, however, about how best to implement simulation into curricula to maximize the efficiency of training.

While prior comparisons to aviation have yielded a fair amount of knowledge regarding the utility of simulation,² other high performance industries, such as sports, may also provide valuable insight into potential training strategies to elicit superior performance. Some pedagogical techniques identified in sports have already been investigated in surgery, including warm-up,³ mental practice,⁴ and deliberate practice.^5,6 Competition has been found to lead to improved performance in sports including golf, weight lifting, and basketball.^7-9 Gamification, the use of game mechanics such as competition, has been successfully utilized to improve motivation to participate in surgical simulation training and to teach and assess clinical decision making^10,11; however, no studies have investigated the effects of competition on technical skills performance in a randomized, controlled manner.

We hypothesized that competition would lead to improved performance in trainees. This study investigated the effects of competition on performance during successive virtual reality (VR) laparoscopic cholecystectomy (LC) cases.

Methods

Participant Selection

Due to the educational nature of the study, this protocol was exempted from further ethics review. Informed consent was obtained from all participants, and participants were informed that their participation, or lack thereof, would in no way impact their medical training or medical care they might receive. Medical students from London hospitals with an interest in surgery were invited to participate in the study. Based on power analysis and cost constraints, twenty (n=20) medical students were recruited. All trainees had limited surgical experience (performed 0 but observed > 1 LCs in the operating room). All participants were offered a certificate of completion in a basic laparoscopic skills course if they completed all sessions of the study. At recruitment, participants were randomized into one of two equal groups – Competitive Training (CT) group or Control group – using a random number generator (STATA, College Station, TX) (Figure 1).

Figure 1.

Flow chart of study protocol with recruited subjects and drop-outs

Baseline Assessment

Each participant underwent a validated baseline skills assessment on the LapMentor VR (Simbionix; Cleveland, OH) laparoscopic simulator on Basic Skills tasks 5 and 6. For Basic Skills task 5, time to completion was assessed. For Basic Skills task 6, time to completion and number of movements was recorded as these metrics have been shown to be construct valid.¹²

Participants were also asked to complete the Revised Competitiveness Index, a psychometric questionnaire designed to assess individuals’ trait of competitiveness along two domains – enjoyment of competition and contentiousness (desire to outperform others).¹³ Each domain is tested on its own subscale within the Revised Competitiveness Index and can be considered as an individual factor that makes up a person’s trait of competitiveness.

Didactic and Proficiency Training

Participants underwent a modified laparoscopic skills training program based on a previously validated curriculum.¹² Participants were trained to proficiency in basic skills and were given video instruction on performing a full procedure LC on the simulator.

Competitive Training Group Sessions

Participants in the CT group underwent 10 training sessions comprising a total of 10 VR LCs. Participants in the CT group were told to perform each procedure as safely and efficiently as possible but were also informed that the top performer after 10 sessions would be awarded a gift card for a flight simulator experience (valued at approximately $150). Each session, participants completed a VR LC on the LapMentor simulator, and their performance was assessed in real time by a trained observer using a previously validated rating scale of surgical technical skill [Objective Structured Assessment of Technical Skills Global Rating Scale (OSATS GRS)].^14,15 They were then given immediate post-procedure feedback on their performance by being shown time to complete the VR LC, number of movements, total path length (cm) of instrument tips, and OSATS GRS. At the conclusion of each session, participants were shown a leader board demonstrating their performance and rank compared to others in the CT group (Figure 2).

Figure 2.

Leader board demonstrating participant performance and rank compared to others in the CT group.

Ranking was based on a formula (Formula 1) that weighted quality of performance (OSATS GRS) greater than time or dexterity (as measured by number of movements and path length) based on the recommendations of surgical educators at Imperial College London. Similar to golf, a lower score was considered to have a higher rank.

$$\left[\text{Time}\right(\sec )+\text{Movements}+\text{Path Length}(\mathrm{cm}\left)\right]∕\mathrm{OSATS}\mathrm{GRS}$$ Formula 1

Before and after each VR LC, participants were asked to complete a short form State Trait Anxiety Inventory (STAI), a validated tool to assess state anxiety.¹⁶ After each VR LC, participants were also asked to complete the NASA Task Load Index (TLX), a validated, subjective multidimensional assessment tool that allows participants to rate perceived workload.¹⁷ TLX was utilized to assess for any increased workload that participants may experience from competition. Participants in the CT group were asked to not disclose their status in the study to prevent a potential effect on motivation in the control group.

Control Group Training Sessions

The control group similarly underwent 10 training sessions comprising a total of 10 VR LCs, but no mention of a prize was made. They were only instructed to perform each procedure as safely and efficiently as possible. Their performance was also assessed using the same metrics as the CT group by the same trained observer. The control group was given immediate post-procedure feedback on their performance by being shown time to complete the VR LC, number of movements, total path length (cm) of instrument tips, and OSATS GRS. Participants in the control group were not ranked against one another and were not shown the performance of other participants.

Control group participants were also asked to complete a short form State Trait Anxiety Inventory before and after each VR LC. After each VR LC, participants were asked to complete the NASA TLX.

Participants in both groups were not allowed to practice laparoscopic skills outside of the scheduled study sessions. Participants were allowed no more than two sessions per day with each trial separated by one hour to prevent fatigue. Scheduling of sessions was controlled to allow for accurate comparison of performance amongst the CT group based on session number.

Statistical Analysis

Statistical analysis was performed using STATA Intercooled 12 (College Station, TX). Shapiro-Wilk test showed the nature of the data to be nonparametric. Mann-Whitney U-test was employed to compare intergroup baseline laparoscopic performance and VR training session performance. Wilcoxon signed-rank test was utilized for intra-group comparison. Data are reported as median (interquartile range). Levene’s test was utilized to compare the consistency in performance of the CT group versus the control group. Multivariate regression was used to assess the effect of competitiveness and contentiousness on surgical performance. Nonlinear regression was utilized to assess the learning curve of participants.¹⁸

In addition to live ratings, videos of VR LCs from both groups were assessed by an independent, blinded rater using the OSATS GRS. Intraclass correlation coefficient (ICC) was calculated to assess inter-rater reliability of the OSATS GRS. A p <0.05 was be considered statistically significant.

Sample size was based on detecting at least a 25% difference in time and dexterity with alpha of 0.05 and beta of 0.8 as based on preliminary data collected prior to the study.

Results

Subjects

Eighteen of the twenty recruited participants completed the study. Two participants dropped out during proficiency training and cited scheduling conflicts for their inability to complete the study. All participants were right handed. Two of the participants in the control group and three in the CT group were female.

Baseline Assessment of Laparoscopic Skill

There were no significant differences in baseline laparoscopic skill between control and the CT groups (Table 1).

Table 1.

Pre-test baseline skills assessment of control and CT groups. Values as median (interquartile range).

	Control	CT	p-value
Task 5
Time (sec)	132.7 (122-142)	134 (123-146)	0.70

	Control	CT	p-value
Task 6
Time (sec)	164.3 (132-177)	170.5 (139-198)	0.31
Movements	208 (169-253)	264 (229-307)	0.10
Path Length (cm)	510 (405-631)	668.5 (525-726)	0.30

Competitiveness Index

Analyzing the results of the Revised Competitiveness Index, there was no significant difference in enjoyment of competition between groups (Control: 27.44 ± 2.51, CT: 27.11 ± 3.78; p=0.3). However, the CT group (16.89 ± 5.39) was significantly more contentious than the Control group (13 ± 2.59; p<0.001).

Virtual Reality LC Performance

Both the control and CT groups improved over the course of 10 LCs in time, movements, path length, and OSATS GRS (Table 2). The intraclass correlation coefficient for OSATS GRS was ICC=0.858.

Table 2.

Comparison of first and final VR LC time, movements, path length, and quality of surgical performance. Comparison of pre- and post-LC mean State Trait Anxiety Index between groups. Values as median (interquartile range).

		Control	CT	p-value
Time (sec)
First LC		871 (797-898)	598 (529-708)	0.01
Final LC		390 (305-405)	319 (208-357)	0.40
	p-value	0.008	0.008
Movements
First LC		816 (684-952)	643 (529-756)	0.04
Final LC		327 (301-373)	253 (230-258)	0.02
	p-value	0.008	0.008
Path Length (cm)
First LC		1397 (1053-1617)	920.7 (810-1057)	0.04
Final LC		518 (509-633)	376.7 (343-404)	0.02
	p-value	0.008	0.008
OSATS GRS
First LC		16 (15-17)	16 (16-18)	0.47
Final LC		20 (18-21)	21 (21-22)	0.16
	p-value	0.013	0.008
STAI
Pre-LC		8 (7-11)	11 (10-12)	<0.001
Post-LC		9 (7-11)	12 (10-14)	<0.001
	p-value	0.118	<0.001

At the first LC, the CT group was significantly faster, made fewer movements, and had lower path length than the control group. There was no significant difference between groups in quality of surgical performance as assessed by the OSATS GRS at first LC (Table 2). By the tenth and final LC, there were no significant differences between groups in time to complete the procedure or OSATS GRS score. With regards to dexterity, the CT group made significantly fewer movements and had lower path length than the control group (Table 2).

The CT group had significantly lower variance in number of movements and instrument path length than the control group at the tenth and final LC (p=0.019).

Virtual Reality LC Learning Curves

After 5 cases, the control group plateaued at an average procedure completion time of 345.8 seconds (p<0.001), total number of movements of 308 (p<0.001), and path length of 482 cm. (p<0.001). The control group plateaued at an average OSATS GRS of 20 after 5 cases (p<0.001) (Figure 3A-D).

Figure 3.

(A-D) Virtual Reality learning curves and performance box-and-whisker plots of control and CT groups for time to complete procedure (A), total number of movements (B), total path length (cm) (C), and OSATS GRS (D).

The CT group plateaued at 365.1 seconds (p<0.001) after 8 cases, total movements of 288 (p<0.001) after 9 cases, and path length of 427.4 cm (p<0.001). The CT group on average plateaued at an OSATS GRS of 21 after 5 cases (p<0.001) (Figure 3A-D). There were no significant differences in the plateau levels of the two groups for any of the metrics.

Virtual Reality LC Performance and Psychometrics

The CT group reported higher mean state anxiety after completing a VR LC compared to state anxiety just prior to performing a VR LC while the control group reported no difference in state anxiety either before or after VR LC performance (Table 2). For both groups, state anxiety as assessed after VR LC had a negative effect on quality of surgical performance (Table 3).

Table 3.

Multiple linear regression model for VR LC quality of performance and state anxiety. State anxiety after VR LC had a negative effect on quality of surgical performance.

Control Group State Anxiety
Variable	Coefficient	SE	95% CI	R2	p-value
Intercept	20.66	0.92
Pre-STAI	0.11	0.09	−0.08 to 0.29	0.13	0.253
Post-STAI	−0.33	0.09	−0.51 to −0.14	0.13	0.001

CT Group State Anxiety
Variable	Coefficient	SE	95% CI	R2	p-value
Intercept	20.61	1.45
Pre-STAI	0.16	0.12	−0.08 to 0.39	0.1	0.19
Post-STAI	−0.24	0.08	−0.4 to −0.09	0.1	0.003

There was no significant difference in perceived workload between groups after each VR LC. There was no effect of perceived workload on either group for any of the performance metrics.

For both groups, there was no effect of competitiveness or contentiousness on quality of performance. For the CT group, contentiousness was inversely related to time to complete a VR LC (β=−4.56 ± 2.23, R²=0.57, p=0.044) and number of movements (β=−5.6 ± 2.41, R²=0.65, p=0.023). There was no relationship between contentiousness and time or movements for the control group.

Discussion

Trainees engaging in competitive training developed greater dexterity when performing VR LC. Although the time taken to complete the procedure and quality of surgical performance were similar between trainees in CT and those in standard training, the decreased movements and instrument path length suggest that the CT group was able to complete a VR LC with greater efficiency. Furthermore, the CT group was more consistent in movements made and instrument path length as suggested by the decreased variance of these metrics in the CT group at the final LC (Figure 3B-C).

Gamification in surgical education has predominantly been rooted in the utilization of serious games.¹⁰ Serious games are interactive, scored computer games that are fun, engaging, yet challenging with the goal of improving skills or knowledge applicable to real world scenarios. While many games investigated in the literature have focused on teaching decision-making skills or cognitive knowledge, surgical skills competitions are often held at various society meetings and within institutions as a fun exercise in skills practice. To our knowledge, this was the first randomized controlled trial investigating the effect of competitive training on the acquisition of surgical skill.

As the assignment of participants to CT versus control training was random, we did not intend to have more contentious people in the CT group. However, the CT group in this study reported being more contentious than the control group, and a regression model of the performance data suggests that contentiousness in the CT group relates to faster and more dexterous performance. While we interpret these findings with caution, one potential explanation is that contentious participants, when placed in a competitive environment, have improved performance in dexterity that may have been driven by their desire to outperform others.

Competition is not the only tool that may exist to promote improved performance in novice surgeons. Delivery of surgical care requires a coordinated team effort involving surgeons, nurses, technicians and other ancillary staff; and cooperation in the team setting may be the more appropriate means of improving performance as suggested in both the surgical and sports science literature.^9,19 However, a study in sports science suggests that in situations where individuals are involved in a structured, fair competition and are able to gauge the progress of opponents, competition can lead to higher levels of individual motor performance.²⁰ Acquisition of skill in the simulation center tends to occur in the individual setting, and a structured competition may promote improved performance in domains such as dexterity as suggested in this study. These skills may then be translated to the team environment to provide the best care for patients. Being able to anonymously gauge the progress of other trainees in a program may have provided a target for which the participants in the CT group could strive to outperform, thus leading to improved dexterity versus the control group.

Analysis of the state anxiety in participants found that high state anxiety after completion of VR LC negatively affected quality of surgical performance as assessed by the OSATS GRS. Due to the design of the study, we are not able to conclude whether increased anxiety during the VR LC resulted in a decreased OSATS GRS score or if the increased anxiety was a result of technical errors that were reflected in a decreased OSATS GRS score. Previous research suggests either explanation may be plausible as the literature has reported technical errors to be a source of stress for surgeons but also that some individuals may be less capable of skilled surgical performance under pressure.^21,22 While there was no difference in state anxiety before or after a VR LC for the control group, the CT group reported significantly higher state anxiety after completing a VR LC. The participants were not surveyed on why they may have felt more anxious; however, competition may have heightened levels of anxiety as the CT group awaited the results of their ranking. While it is certainly not ideal to stress trainees to the point of negatively impacting performance, surgeon stress is present and measurable in an operation.²³ Competition in the simulation environment may provide a safe avenue through which to expose trainees to stress that may be present in a real operative setting.

The results of this study are not without limitations. Although participants received their OSATS GRS scores, they did not receive feedback on specific steps to improve quality of surgical performance. Previous work has demonstrated that specific feedback is necessary to improve performance quality; thus, quality of performance may have been limited in this study.⁵ To control elements such as complexity of the case, the competition was limited to VR in this initial study. As the ultimate goal of simulation training is to improve performance in the operating room, individuals who participate in CT should undergo evaluation in a live operative case. Since this study was conducted with medical students, no attempt was made to assess the participants in a live operating room. However, future work with residents should assess the effect of CT in a real clinical setting. Steps should be taken to ensure that competition remains confined to the simulation environment, as patient care and safety should not be treated as a game.

Implementation of competitive training for programs with pre-existing simulation programs can be low cost. We utilized an in-house programmed webpage; however, a similar study could be conducted by manually placing values into a spreadsheet and sorting values to calculate rank. Future work should survey participants on motivation prior to each session to determine whether participants subjectively report being motivated by competition, and studies can investigate a tailored approach to learning that compares competitive and non-competitive training based on the motivational preferences of the trainees. Such work may help elucidate whether competition motivates trainees to varying levels depending on individual motives.

Conclusion

Surgical skills competitions have been held at surgical meetings and at institutional levels, but competition as a novel training strategy had not previously been investigated. The results of this study suggest that competition in surgical education for medical students may lead to improved dexterity in laparoscopic cholecystectomy but has otherwise equivalent effects as standard, repetition-based training on time and quality of performance. Additional research is needed to determine if similar effects are seen in residents who receive feedback on their performance.

Abbreviations

ACGME

Accreditation Council for Graduate Medical Education

VR

Virtual reality

LC

Laparoscopic cholecystectomy

CT

Competitive Training

OSATS GRS

Objective Structured Assessment of Technical Skills Global Rating Scale

STAI

State Trait Anxiety Inventory

TLX

Task Load Index