Novices Outperform Experienced Laparoscopists on Virtual Reality Laparoscopy Simulator

Abstract

Background and Objectives:

Virtual reality has been poorly studied among gynecologic surgeons. The aim of this study was to evaluate whether performance on the Minimally Invasive Surgery Trainer-Virtual Reality (MISTVR) laparoscopic trainer reflects laparoscopic experience among gynecologic surgeons and trainees.

Methods:

Twenty-six medical students, residents, and attending gynecologic surgeons completed a MIST-VR training program. A new simulated task was then presented to each participant, who repeated the task until proficiency was reached.

Results:

Attending physicians performed poorly when compared with medical students, requiring more than twice the number of attempts to reach proficiency (Mann-Whitney P<0.01). Among medical students and residents, there was an association between years of live laparoscopy experience and poor simulator performance (Spearman r P=0.01).

Conclusion:

Increased operating room experience and age were associated with worsening simulator performance. Several potential explanations for this trend are discussed, including lack of tactile and contextual feedback. Caution should be exercised when considering current virtual reality simulator technology as a measure of experience or ability among gynecologic surgeons.

INTRODUCTION

During the past 10 years, several virtual reality (VR) laparoscopic surgery platforms have been studied as an adjunct to surgical education. Current technology shows promise in 2 important areas. First, laparoscopic VR surgical trainers have been studied as a tool to objectively assess surgical ability, proficiency, and raw psychomotor talent. Second, laparoscopic VR trainers have been used as a training tool to improve surgical performance. By some estimates, as many as a quarter of US general surgery resident training programs use laparoscopic VR technology to train and/or assess residents, and it is reasonable to assume that many of these resident surgical skills labs are shared with gynecologic surgical trainees.¹ The Minimally Invasive Surgical Trainer-Virtual Reality (MISTVR) laparoscopic trainer is the most rigorously studied and widely used virtual reality platform. Two well-designed randomized control trials have shown that training using the MIST-VR platform improves operating room performance among general surgical residents during gallbladder bed dissections.^2,3 The MIST-VR platform has also demonstrated utility in assessing surgical skill and experience among general surgeons and trainees.^4–6 However, all of these studies have been performed among general surgeons and trainees. The purpose of this study was to evaluate the use of the MIST-VR trainer for proficiency-based laparoscopic skills assessment among gynecologic surgeons and trainees not previously exposed to VR laparoscopic simulators.

We hypothesized that those participants with more operating-room laparoscopy experience would perform superiorly on the MIST-VR trainer than those who had less experience (ie, attendings would perform better than residents, etc).

MATERIALS AND METHODS

The MIST-VR trainer consists of 12 laparoscopic tasks in which virtual cubes, spheres, and cylinders are manipulated in 3-dimensional space with virtual graspers, and a mechanical pedal is used to apply virtual diathermy to the objects (Figure 1). The MIST-VR provides visual feedback, such as change of object color, to simulate both success and error, but there is no change in resistance of the instrument handles with object manipulation, as would be encountered in live laparoscopy. Performance is measured based on time taken for task completion, economy of movement, economy of diathermy, and errors, with an overall score that combines individual measurements.

Figure 1.

MIST-VR, Manipulative Diathermy task. A sphere is grasped with virtual graspers and placed precisely inside a transparent cube. When the sphere is properly placed and held still inside the cube, a smaller cube appears on the side of the sphere. The virtual diathermy instrument is then placed inside of the smaller cube, and diathermy is applied with a mechanical pedal for a specified burn time. Each “attempt” consists of multiple repetitions of this task in varying spatial orientations.

Enrollment in the study was voluntary and was open to all medical students, gynecology residents, and attending physicians at our hospital from December 2004 through March 2006. Due to the educational nature of the research, the study received institutional review board exemption. None of the participants had any prior exposure to the MIST-VR platform. Each participant went through a standardized initial orientation to familiarize them with the MIST-VR platform. For consistency, a single instructor oriented all participants to the computer software. After the initial orientation, each participant was then asked to perform and repeat the “manipulative diathermy” task, the most advanced of the trainer tasks, until the predetermined level of proficiency was reached. Scores were automatically recorded by the computer. The manipulative diathermy task was chosen because of its previous validation for transferability into skill improvement.³

The proficiency level we chose was similar to previously published standards,⁷ but slightly more relaxed to include more subjects in our analysis. Proficiency was reached when the participant made 3 attempts in a row where the average weighted score was <200 (lower score is better) and the score for both left- and right-handed economy of diathermy was <4. Seven medical students, 12 residents, and 7 attendings reached proficiency and were included in the analysis.

The primary outcome measure was the number of task repetitions required for the participants to reach minimal proficiency. Years of experience and age were correlated with the number of task repetitions required to reach minimal proficiency (Figures 2 to 4). Because we made no attempt to quantify the level of experience of the attendings beyond verifying that they regularly performed laparoscopic surgery, attendings were excluded from the experience vs proficiency analysis (Figure 2). Attending surgeons performed an average of 30 laparoscopic procedures per year, ranging from 12 to 61 procedures per year. Attendings were included in a second nonparametric correlation analysis in which age was used as a surrogate for experience (Figure 3). There was no overlap in age between the medical student, resident, and attending groups with the exception of a 30-year-old medical student and a 49-year-old resident. Average number of attempts required to reach proficiency in the medical student, resident, and attending groups were compared using a Mann-Whitney test in a post-hoc analysis (Figure 4). Statistical tests were performed using GraphPad InStat statistical software.

Figure 2.

Attempts required to reach proficiency on MIST-VR manipulative diathermy task by years of live laparoscopy experience. Med students=0 years experience, PGY1=1 year experience.. Y4=4 years experience. Attendings not included. Linear regression P=0.01; F=8.4, degrees of freedom=1,17; r²=0.33.

Figure 4.

Average number of attempts required to reach proficiency on the MIST-VR manipulative diathermy task by participant group. *Mann-Whitney test: attending vs. medical student P=0.01, attending vs. resident P=0.14, resident vs. med student P=0.15.

Figure 3.

Attempts required to reach proficiency on MIST-VR manipulative diathermy task by age. Medical students, residents, and attendings included in analysis; 2-tailed t P<0.01, Spearman r=0.53.

RESULTS

Unexpectedly, among those reaching proficiency, the more experienced laparoscopists took a greater number of attempts to become proficient than those who had little or no experience. This was demonstrated in several ways. First, among medical students and residents, those believed to have the most homogenous and incremental exposure to live laparoscopy, there was a strong correlation between increasing experience and worsening performance on the simulator (Figure 2) (P=0.01). Because of the greater heterogeneity in background, experience, and the presence of an outlier in the attending group, the nonparametric Spearman rank test was used for the correlation of age and attempts to proficiency. Similarly, older subjects required a greater number of task repetitions to become proficient (Figure 3) (P<0.01).

Finally, a comparison of average attempts to proficiency between study groups showed the attending group, members having on average more than 15 years of live laparoscopic experience, required almost twice as many task repetitions before reaching proficiency as the medical student group, who had no live laparoscopy experience (Figure 4) (P<0.01, Spearman r=P=0.53). The trend was not significant in comparing residents with medical students (P=0.15) and residents with attendings (P=0.14), though the trend towards incremental exposure to live laparoscopy and worsening performance is evident.

DISCUSSION

In the framework of our study, the MIST-VR laparoscopic trainer was not a valid measure of experience level among our population of gynecologic surgeons and trainees, and even showed that increased operating room experience was associated with a handicap for simulator performance. This trend may suggest that gynecologists with live laparoscopy experience expect certain tactile, visual, and contextual feedback cues not provided by the virtual trainer, and that this expectation may create difficulty adapting to the parameters measured by the MIST-VR. Illustrating this point, the gynecologic laparoscopist with the highest volume of surgery among the attending physicians in our study required 25 iterations to meet minimal proficiency, more than 2 standard deviations above the mean. The trend was actually stronger with his exclusion (P=0.002), and so he was left in all analyses.

Differences in surgical variety and orientation between general surgeons and gynecologic surgeons may explain why several studies have suggested that MIST-VR may be a valid platform for skills assessment among general surgeons but was not in our population of gynecologic surgeons.^4–6 Compared with general surgeons, gynecologic surgeons tend to have less of a variety of port-site and operator/camera/instrument orientations. The greater variety of operating conditions and orientations to which the expert general surgery laparoscopist is exposed may create cognitive and visuo-spatial flexibility, which prepares them to adapt quickly to the new environment of the MIST-VR trainer with its own specific set of rules and idiosyncrasies. Likewise, current laparoscopic VR trainers, including MIST-VR, simulate operator/camera/instrument orientations are common to general surgery but relatively uncommon in pelvic surgery.

One explanation for the observed trend involves the potential confounding variable of age (Figure 3). At least one study suggests that prior experience with video games tends to improve simulator performance.⁸ Though not controlled for, the fact that younger participants likely have had more exposure to video games could be a relevant issue in our study and could contribute to this age-related effect. However, given the fact that the medical students and residents were almost entirely within 10 years of age of each other and still showed worsening performance with increasing experience, it seems that the exposure to live laparoscopy, and not age, is a more likely explanation for the trend.

In our study, the same proficiency standard was applied to all participants, with a highly significant result. Many participants reached proficiency after 5 or 6 iterations. However, at least one study has suggested that a more challenging proficiency standard, one that requires more than 10 iterations for even the expert, may more accurately distinguish the novice from expert laparoscopist.⁷ Notably, one of the strengths of our study is that none of the participants had been previously exposed to virtual reality laparoscopy, which may not be the case with other skills assessment studies.

It must be acknowledged that the published literature on the value of virtual reality platforms, including MIST-VR, describe a range of effectiveness in differentiating novices from skilled laparoscopists.^6,7,9–13 This is probably related to several factors, including platform validity and mathematical weighting of measured parameters, variable exposure to the VR simulator among test groups, the consistency of the user experience (eg, teacher input during simulator sessions, consistency of training conditions), as well as other factors that can be harder to predict such as software stability. This is, however, both the first study of its kind among gynecologic surgeons and trainees, as well as the first study to demonstrate a strong negative correlation between experience and performance.

CONCLUSION

Virtual Reality remains a promising platform for both the assessment and acquisition of laparoscopy skills and will certainly play an increasingly important role in surgical education. This study raises several questions about the current limitations of VR technology, including whether proficiency standards for virtual laparoscopy trainers determined in populations of general surgeons may be generalizable to gynecologic surgeons. More studies are needed to determine which measurable parameters in virtual laparoscopy are reflective of skill level in the operating room among diverse groups of surgeons performing various procedures.