Setting Benchmarks for the New User: Training on the Robotic Simulator

Abstract

Background and Objectives:

Data showing the impact of the robotic simulator on fellowship training are limited. This study was conducted to determine whether simulator scores reflect the experience of the robotic gynecologic surgeon and to develop a simulator curriculum for trainees in gynecologic oncology.

Methods:

All faculty and fellows in the Department of Gynecologic Oncology and Reproductive Medicine were asked to participate. For phase 1, all participants were divided into 2 groups based on robotic surgical experience: beginner (0–50 cases) and experienced (>50 cases). Each participant completed 9 modules 3 times each to establish baseline data. Median module scores for the experienced group defined the benchmarks scores. In phase 2, all trainees who did not meet the benchmark score on a module were asked to repeat the module until they reached the score twice.

Results:

Twenty-four participants were included: 18 beginners and 6 experienced surgeons. For all modules, experienced surgeons received higher median scores than beginners. There was a significant difference between the scores of the 2 groups in the Energy Switching 1 (87.5 vs 92.5; P = .002) and Suture Sponge 2 (75.0 vs 87.3; P = .011) modules. Thirteen trainees participated in phase 2. For 8 of 9 of the modules, >75% of trainees met proficiency, with a median of 3 to 6 attempts (range, 2–24).

Conclusion:

Based on the findings, scores reflected each surgeon's experience. With repetition, most of the trainees were able to reach the benchmark scores. Further study is needed to determine the impact of surgical simulation on true intraoperative performance.

INTRODUCTION

The da Vinci surgical system (Intuitive Surgical, Sunnyvale, California, USA) was initially cleared in 2005 by the Food and Drug Administration for use in gynecology. Since that time, there have been publications evaluating the utility of robotic surgery for endometrial cancer,^1,2 cervical cancer,^3,4 and adnexal masses.⁵ The number of gynecologic oncology cases performed on the robot has continued to increase, but there are limited published data regarding the impact of the robot surgical system on fellowship training.

As with other advances in technology used in the operating room, the general guidelines set forth by the American College of Surgeons, the American College of Obstetrics and Gynecology, and the Society of Gynecologic Surgeons apply to the robotic system. These guidelines include assessing a surgeon's eligibility to use the new technology based on previous training and experience, amount of education required for adequate understanding of the technology, and environment recommended for appropriate use of the new technology.^6,7 This systematic approach is believed to result in improved patient safety when new technology is implemented. Although patient outcomes and safety are the primary concern for surgeons in general, experienced surgeons also have a responsibility to train future surgeons to adequately manage their patients after completion of their training.

The advances in surgical technology are making operating increasingly more complex. Simulation provides an opportunity to break these complex procedures into individual skills that can then be perfected by the learner.⁸ Some randomized controlled trials have shown that, although simulation training is not superior to hands-on training in the operating room, it is superior to no training.⁹

The da Vinci Skills Simulator (Intuitive Surgical) was launched in 2010. Although the goals of the Skills Simulator are to allow surgeons to become more familiar with the da Vinci Surgical System and to practice outside of the operating room, there are limited data validating this robotic surgery simulator. The purpose of phase 1 of this study was to determine whether scores on the surgical simulator reflect the skills of a gynecologic surgeon experienced in the use of a robotic system. The purpose of phase 2 of this study was to determine how much practice would be needed for new users to meet the benchmark scores set in phase 1.

MATERIALS AND METHODS

After institutional review board (IRB) approval, all of the faculty and fellowship trainees (n = 32) in the Department of Gynecologic Oncology and Reproductive Medicine at MD Anderson Cancer Center were invited to participate in the study. In phase 1, the participants were divided into 2 groups based on robotic surgical experience. Beginners were defined as those who had performed fewer than 50 robotic surgeries as the primary surgeon. Experienced surgeons were defined as those who had performed 50 or more operations as the primary surgeon at the console. There were faculty and fellows in both groups. As mandated by the IRB, the research coordinator deidentified the data for the participants in each group before evaluation by the study investigators.

Once enrolled, each participant was asked to complete 9 modules on the simulator, 3 times each. The 9 modules were selected based on their applicability during a hysterectomy and included Camera Targeting 1, Energy Switching 1, Energy Switching 2, Match Board 1, Match Board 3, Ring and Rail 2, Suture Sponge 2, Thread the Ring, and Energy Dissection 2. The description and skills assessed in each module are shown in Table 1. Participants were allowed to complete the modules in multiple sessions and in any order. The first run of each module was considered a dry run and was not used in the statistical evaluation. For each participant, the scores from the second and third runs were averaged for a composite score for that module.

Table 1.

Skills Simulator Module Descriptions

Module	Description
Camera Targeting 1	Assess camera controls when tracking a moving target.
Energy Switching 1	Assess camera controls, clutching, and energy control when using monopolar and bipolar instruments.
Energy Switching 2	Assess camera controls, clutching, and energy control when using monopolar and bipolar instruments.
Match Board 1	Assess EndoWrist manipulation when picking up objects and placing them into their corresponding places.
Match Board 3	Assess EndoWrist manipulation, camera control, clutching, and fourth-arm control when picking up objects and placing them in their corresponding places.
Ring and Rail 2	Assess EndoWrist manipulation, camera control and clutching when picking up 3 colored rings and guiding them along their matching colored railing.
Suture Sponge 2	Assess EndoWrist manipulation, camera control, needle control and needle-driving basics when inserting and extracting a needle through several pairs of targets on the edge of a sponge.
Thread the Ring	Assess EndoWrist manipulation, camera control, and needle control when inserting and extracting a needle though rings on a board.
Energy Dissection 2	Assess EndoWrist manipulation, dissection, and energy control when cauterizing and cutting small blood vessels.

The composite scores for both groups on each of the modules were summarized. The mean scores were compared by t test, and the median scores were compared by Wilcoxon test. A result reaching P < .05 was considered statistically significant. For phase 2, the median scores for the experienced group from phase 1 were defined as the benchmark scores. Trainees who did not meet the benchmark score on a specific module in phase 1 were then asked to repeat that module until they reached the benchmark score twice and were considered proficient. Summary statistics were used to report phase 2 data.

RESULTS

A total of 26 surgeons enrolled in the study over a period of 3 years: 14 fellows and 12 faculty members. Based on their experience in robot-assisted surgery in the operating room, 6 were experienced (all faculty), 18 were beginners (6 faculty, 12 fellows), and 2 were of unknown experience. The scores for the 2 participants with an unknown experience were excluded, leaving scores for 24 surgeons in the analyses. Most participants completed all tasks in 1 session. The mean and median scores for each module are shown in Table 2. The mean and median scores were consistently higher in the experienced group, as compared to the beginner group on all 9 modules (Table 2). There was a statistically significant difference between the median scores of the 2 groups on the Energy Switching 1 (87.5 vs 92.5; P = .0024) and Suture Sponge 2 (75 vs 87.3; P = .0113) modules.

Table 2.

Composite Comparison Between Beginner Group and Experienced Group on Different Modules

Group	Beginner (n = 18)		Experienced (n = 6)		Pa	Pb
Simulator Module	Mean	Median	Mean	Median
Camera Targeting 1	81.4	84.0	89.5	91.8	.1062	.0952
Energy Switching 1	81.1	87.5	93.2	92.5	.0019	.0024
Energy Switching 2	86.2	89.0	91.4	91.8	.2678	.4046
Match Board 1	78.4	80.8	89.3	91.5	.0747	.1024
Match Board 3	55.6	59.0	66.1	70.0	.1681	.1831
Ring And Rail 2	70.3	73.5	78.1	78.0	.2827	.2775
Suture Sponge 2	74.8	75.0	86.7	87.3	.0098	.0113
Thread the Rings	85.8	90.5	91.8	91.8	.3088	.2097
Energy Dissection 2	83.6	86.3	86.6	86.8	.2891	.6306

^aT-test P for comparing means between beginner and experienced groups.

^bWilcoxon test P for comparing medians between beginner and experienced groups.

Thirteen trainees participated in phase 2. The median number of sessions to complete phase 2 was 2 (range, 1–3). Although there was a time gap between sessions (up to 3 months), none of the participants transitioned from the beginner group to the experienced group during this period. The number of fellows who achieved proficiency on each module is shown in Table 3. For 8 of 9 modules, greater than 75% of trainees met proficiency. Suture Sponge 2 proved to be the most challenging with only 69% (9 of 12) trainees meeting proficiency. The number of attempts required to achieve proficiency is shown in Table 4. For all modules, the median number of attempts among those who were proficient was between 3 and 6. Among those who did not meet proficiency, up to 21 attempts on a single module were made.

Table 3.

Number of Fellows Achieving Benchmark Scores on the Different Modules

Simulator Module	Did Not Meet n (%)	Met Once n (%)	Met Twice n (%)
Camera Targeting 1	0 (0.0)	1 (7.7)	12 (92.3)
Energy Dissection 2	0 (0.0)	0 (0.0)	13 (100.0)
Energy Switching 1	0 (0.0)	2 (15.4)	11 (84.6)
Energy Switching 2	0 (0.0)	0 (0.0)	13 (100.0)
Match Board 1	1 (7.7)	0 (0.0)	12 (92.3)
Match Board 3	3 (23.1)	0 (0.0)	10 (76.9)
Ring and Rail 2	0 (0.0)	1 (7.7)	12 (92.3)
Suture Sponge 2	2 (15.4)	2 (15.4)	9 (69.2)
Thread The Rings	0 (0.0)	0 (0.0)	13 (100.0)

Table 4.

Number of Attempts Needed by Fellowship Trainee to Achieve Proficiency

Simulator Module	Median	Range
Camera Targeting 1 (n = 12)	6.5	2–12
Energy Dissection 2 (n = 13)	3.0	2–12
Energy Switching 1 (n = 11)	6.0	3–22
Energy Switching 2 (n = 13)	4.0	2–11
Match Board 1 (n = 12)	5.5	2–24
Match Board 3 (n = 10)	6.0	2–13
Overview of Controls (n = 6)	3.0	3–3
Ring and Rail 2 (n = 12)	4.0	2–8
Suture Sponge 2 (n = 9)	5.0	2–17
Thread the Rings (n = 13)	4.0	2–12

DISCUSSION

In this cohort study, we found that the experience of the robotic surgeon was reflected in their scores on the surgical simulator. Experienced surgeons scored consistently higher in all categories when compared to beginner surgeons. These findings were statistically significant for 2 of the 9 modules tested. In phase 2, we were able to show that, with repetition, trainees were able to reach the benchmark scores set by the experienced surgeons. These benchmark scores are now used in our fellows' simulator curriculum.

With the growing number of gynecologic oncology surgeries being performed with the da Vinci System, training programs administrators are looking at more effective ways to train their future colleagues. The Skills Simulator provides a tool that can be used to train new users outside of the operating room. There are limited data validating this new robotic surgery simulator in gynecologic oncology.

In urology, Hung et al¹⁰ published a study comparing the simulator scores among surgeons who had different levels of experience with the robotic surgical system. This study validated the current output measures of the robotic simulator and showed a clear difference in scores based on the surgeon's experience. Based on our analysis, the scores on the robotic simulator reflect the surgeon's experience and thus validate the current output measures of the robotic simulator in gynecological oncology. These results are important, because the number and complexity of robotic surgeries performed in gynecologic oncology are different from those performed in the field of urology.

Even with validated measures, it is unclear how best to integrate the simulator into the curriculum for trainees. There have been a few proposed curricula in the literature that include simulation outside the operating room into the training and assessment of robotic surgeons.^11–14 A common theme among these studies is the importance of having objective benchmarks in which to monitor trainee's progression. These studies consistently show that with continued use of a simulator, participants' performance improved. In addition, simulators allow assessment of a trainee's skills before performing a procedure. After completion of a trainee's training, simulation could also be considered in robotic credentialing for gynecologic procedures by hospitals.

In this study, we were able to set benchmarks for the new user that we defined as the median score of the experienced group on each of the 9 designated modules. We were then able to identify (in phase 2) how much console time it took each participant to reach the benchmark score on each of the modules. We found that, with repetition, most trainees were able to reach the benchmarks set by the experienced users. The number of attempts required for a trainee to reach proficiency varied greatly, based on the specific module. These ranges may underestimate the amount of console time needed to reach proficiency, as not all trainees were able to achieve the benchmark scores twice for all of the modules. We are currently in the process of incorporating this simulator curriculum into the training of all surgical oncology fellows at our institution. In addition, we hope to determine whether participation in the simulator curriculum will translate into improved performance in the operating room.

The main strength of this study is that participants had a wide range of experience with robotic surgery. All participants performed the same modules and scores could be compared. Trainees were then tracked prospectively as they completed the simulator curriculum and tried to achieve the benchmark scores. Our study provides valuable data on the performance on a simulator of gynecologic oncology trainees and lays a foundation for later studies. The main limitation of this study was the small number of participants. The small numbers may account for the fact that the difference between experienced and beginner surgeons was statistically significant in only 2 modules. In addition, the study spanned a 3-year period. It took a significant amount of time to enroll participants because of the limited access to the simulator and scheduling difficulty, but each participant completed the session in a short time. Also, the participants were faculty and fellows at a large academic center, which may not reflect the general surgical learner. Finally, it is not yet clear whether work on the skills simulator will translate into improved performance in the operating room.

CONCLUSION

In conclusion, scores as measured by the robotic da Vinci Skills Simulator reflected the surgeon's experience. We showed that, with repetition, most of the trainees were able to reach the benchmark scores set by the experienced users. Ultimately, we will evaluate the relationship between performance on the simulator and the performance in the operating room.