A proficiency-based virtual reality endoscopy curriculum improves performance on the Fundamentals of Endoscopic Surgery examination

Structured Abstract

Introduction

The Fundamentals of Endoscopic Surgery (FES) examination is a national test of knowledge and skill in flexible gastrointestinal endoscopy. The skill portion of the examination involves five tasks that assesses the following skills: scope navigation, loop reduction, mucosal inspection, retroflexion, and targeting. This project aimed to assess the efficacy of a proficiency-based virtual reality (VR) curriculum in preparing residents for the FES skills exam.

Methods

Experienced (>100 career colonoscopies) and inexperienced endoscopists (<50 career colonoscopies) were recruited to participate. Six VR modules were identified as reflecting the skills tested in the exam. All participants were asked to perform each of the selected modules twice, and median performance was compared between the two groups. Inexperienced endoscopists were subsequently randomized in matched pairs into a repetition (10 repetitions of each task) or proficiency curriculum. After completion of the respective curriculum, FES scores and pass rates were compared to national data and historical institutional control data (endoscopy-rotation training alone).

Results

Five experienced endoscopists and twenty-three inexperienced endoscopists participated. Construct valid metrics were identified for six modules and proficiency benchmarks were set at the median performance of experienced endoscopists. FES scores of inexperienced endoscopists in the proficiency group had significantly higher FES scores (530 +/− 86) vs. historical control (386.7 +/− 92.2, p=0.0003) and higher pass rate (Proficiency: 100%, Historical Control 61.5%, p=0.01).

Conclusion

Trainee engagement in a VR curriculum yields superior FES performance compared to an endoscopy rotation alone. Compared to the 2012–2016 national resident pass rate of 80%, 100% of trainees in a proficiency-based curriculum passed the FES manual skills examination.

Introduction

The importance of incorporating simulators into surgical training has been recognized by governing bodies such as the Residency Review Committee (RRC) of the Accreditation Council for Graduate Medical Education (ACGME) in the United States. In 2008, the RRC mandated that all American surgical residency programs have access to a simulation laboratory. The goal was to incorporate simulation into training to reduce the length of the learning curve before trainees operate on live patients [1].

Effective use of virtual reality (VR) simulators favors the utilization of proficiency-based curricula, which require the identification of construct valid tasks and performance-based rather than time- or repetition-based criteria for completion [2]. A curriculum for attainment of proficient performance accounts for different rates of learning in individuals and ensures that trainees are truly acquiring an acceptable level of skill prior to performing procedures on real patients [2].

The Fundamentals of Endoscopic Surgery (FES) program, created by the Society of American Gastrointestinal and Endoscopic Surgeons (SAGES), is a national test of knowledge and skill in flexible gastrointestinal endoscopy. The skill portion of the examination involves five tasks on a VR simulator that assess the following skills: scope navigation, loop reduction, mucosal inspection, retroflexion, and targeting. Validation of this examination was based on the performance of experienced endoscopists (defined as those having performed >100 upper and lower endoscopies) and those of novice endoscopists [3].

Although prior studies have demonstrated that VR simulator tasks can differentiate experienced and novice endoscopists, these studies did not specifically assess performance as defined for the FES examination [4, 5]. Thus, there is currently no evidence-based VR curriculum through which to prepare for the technical skills portion of the FES examination. Beginning in 2018, the American Board of Surgery (ABS) has mandated that all surgical residents must pass the FES examination in order to take the ABS Qualifying Exam; therefore, there is a need for an effective and efficient proficiency-based curriculum to prepare trainees.

As such, we conducted a two-phase study. The first phase used the difference in performance between experienced and inexperienced endoscopists to establish a proficiency score. The second phase compared FES examination skills scores and first-time pass rates for residents in 1) a proficiency-based curriculum, 2) a repetition-only group, 3) a national cohort, and 4) historical institutional data when only an endoscopy rotation was taken prior to the exam. We hypothesized that quantitative metrics on a VR simulator could differentiate between experienced and inexperienced endoscopists and that a proficiency-based VR endoscopy curriculum based on those metrics would result in improved performance on the FES examination (as measured by score and pass rate) compared to a repetition-based VR curriculum or an endoscopy rotation alone.

Methods

This study was designed in two phases. Phase I was designed to establish the face and construct validity of VR tasks in discriminating between experienced endoscopists who passed the FES examination and inexperienced endoscopists who had not trained for the FES examination. Phase II utilized the benchmarks established in Phase I as performance targets in a proficiency-based curriculum that were compared to performance in a repetition-based curriculum. The Partners Institutional Review Board deemed this study exempt (IRB#: 2015P000522).

Simulator

The GI Mentor (Simbionix, Airport City, Israel) is a virtual reality endoscopic simulator that is designed to assist in the teaching and practice of both upper and lower gastrointestinal endoscopy. This platform was chosen by SAGES for the administration of the skills portion of the FES examination [3]; therefore, this platform was selected for use in this study.

Participants

Faculty, fellows, and residents within the Department of Surgery and the Division of Gastroenterology at a single institution were eligible to participate in the study. Based on ABS colonoscopy experience requirements for surgical residents [6], experienced and inexperienced endoscopists were recruited to participate in the study. Experienced endoscopists were defined as having performed greater than 100 colonoscopies while inexperienced endoscopists were defined as having performed 50 or fewer colonoscopies, based on modifications to criteria utilized to validate the FES examination [3]. Participants were excluded if they had previous participation in a simulation-based endoscopy training curriculum or had previously taken the FES test.

Due to limitations in residency class size for participant recruitment, retrospective data from a historical control group was collected. The historical control group consisted of postgraduate year 2 (PGY2) surgical residents at our institution who had completed a one-month endoscopy rotation only (no other training) and taken FES between June 2014–June 2016. FES scores and pass rates in a national database of all surgical residents who took FES from July 2012–June 2016 (i.e. national group) were also retrospectively analyzed.

Phase I

One FES-certified experienced endoscopist from the Department of Surgery and one from the Division of Gastroenterology were asked to review all tasks available on the VR simulator and select those which were most reflective of the skills tested on the FES examination. Experienced endoscopists were recruited to take the FES examination. Those who passed were asked to review the selected tasks and agree on whether they should be included in a training curriculum for FES. These experienced endoscopists who had passed FES then completed two repetitions of each of the tasks. Inexperienced endoscopists were also recruited to complete two repetitions of each of the tasks (Figure 1A).

Figure 1.

A) Phase I – Comparison of performance between experienced and inexperienced endoscopists to determine proficiency benchmarks. B) Phase II – Randomized controlled trial to compare effect of repetition versus proficiency-based curricula on FES performance.

Median performance across all available metrics provided by the simulator of experienced endoscopists was compared to that of inexperienced endoscopists. Metrics that demonstrated a statistically significant difference (alpha=0.05) between groups were selected as meaningful, discriminative components for computing a total performance score. Median performance of experienced endoscopists on discriminative metrics for each task was set as the benchmark for proficient performance.

Phase II

From July 2016–February 2017, inexperienced endoscopists were recruited and randomized in modified matched pairs (rank ordered by colonoscopy experience and postgraduate year) into a repetition group or a proficiency group (Figure 1B). The repetition group completed ten repetitions of each VR simulator task identified in Phase I; no benchmark performance was provided to this group. The proficiency group performed repetitions of each VR simulator task identified in Phase I until the proficiency benchmark was met on two consecutive attempts. The proficiency group had no restriction on number of repetitions needed to meet proficiency. For both groups, an FES-certified coach provided coaching and feedback for the first two repetitions of each task [7]. Subsequent repetitions in both groups were self-directed by the participant, and all participants could see numeric feedback (e.g. task time, lesions found) from the simulator after each repetition.

For both proficiency and repetition groups, participants took the FES manual skills exam after completion of their respective curricula. Pass rate and FES score were compared between proficiency and repetition groups as well as against the historical control group and the national group. For the proficiency and repetition groups, total time on the simulator was calculated for each participant, and mean total time on the simulator was then calculated for each group. All participants received modest monetary compensation at the completion of their study participation.

Statistical Analysis

There was no published data regarding the scoring rubric or the distribution of scores for the FES technical skills examination at the conception of this study; thus, sample size estimates were based on best available published research demonstrating discriminative validity of GI Mentor tasks and metrics for distinguishing experienced and novice endoscopists [4]. A priori power analysis (alpha =0.05 and power = 0.80) suggested the need for thirteen participants in each curriculum group to detect a 25% pass rate difference between subjects in the repetition, proficiency, and historical control groups. Post hoc power analysis suggested the need for 27 participants in the proficiency group to detect a 20% pass rate difference versus the national cohort.

Distribution of the data was tested by Shapiro-Wilk test. Nonparametric analysis in the form of Kruskal-Wallis test and Wilcoxon rank-sum was used to compare task times. Student’s two-tailed t-test was used to compare FES scaled scores and total time spent on the simulator. Chi² test was used to compare pass rates. All statistical analysis was performed using STATA 14/IC (STATA Corp., College Station, TX).

Results

Participants

Forty-two participants (experienced n=9, inexperienced n=33) were recruited to participate. Fifteen (n=15) inexperienced endoscopists were randomized into the repetition group and eighteen (n=18) into the proficiency group. Experienced endoscopists included three general surgery faculty and six gastroenterology faculty. All inexperienced endoscopists were general surgery residents. Dropouts included four participants from the experienced group, six from the repetition group, and four from the proficiency group. The historical control group included FES test data from thirteen (n=13) residents while national FES data included results from 342 residents. Distribution of PGY year for residents in each group demonstrated a significant difference in median PGY year (National: PGY4, Repetition: PGY2, Proficiency: PGY3, Historical: PGY2, p=0.0001; Table 1). All residents in the repetition, proficiency, and historical groups were inexperienced endoscopists. Incomplete data from the national cohort prevented determination of whether all residents in that group were inexperienced.

Table 1.

Distribution of resident PGY by group.

	NATIONAL (N=342)	REPETITION (N=9)	PROFICIENCY (N=14)	HISTORICAL (N=13)
INEXPERIENCED	N/A	100%	100%	100%
PGY
1	0	1	3	0
2	59	4*	3	13*
3	95	3	5*	0
4	90*	1	2	0
5	98	0	1	0
ENDOSCOPY NAIVE	N/A	4 (44%)	5 (36%)	0 (0%)
COLONOSCOPY EXPERIENCE MEAN (SD)	N/A	14 (16.3)	17 (19.6)	N/A

^*Median PGY for each group.

N/A denotes incomplete data.

There was no significant difference between repetition and proficiency groups on number of participants who were endoscopy naïve (i.e. never performed any endoscopy or completed an endoscopy rotation) (p=0.78) or in mean number of real patient colonoscopies performed at enrollment (p=0.48).

Phase I

All nine experienced endoscopists passed FES. After dropouts, five reviewed the selected VR simulator tasks, agreed on their use in a VR curriculum, and completed two repetitions of each task. Nine inexperienced endoscopists completed two repetitions of each task. Performance between experienced and inexperienced endoscopists was discriminated largely by time (Table 2). These metrics were utilized to establish benchmark metrics for a proficiency curriculum based on median performance of experienced endoscopists (Table 3).

Table 2.

Comparison of performance of experienced and inexperienced endoscopists on VR simulator tasks.

GI MENTOR MODULE	FES SKILL PRACTICED	METRICS	EXPERIENCED MEDIAN (IQR) (N=5)	INEXPERIENCED MEDIAN (IQR) (N=9)	P
ENDOSCOPIC NAVIGATION	Scope Navigation, Targeting	Total time	559.1 sec (481.9, 600.8)	1169.1 sec (869.4, 1661.1)	0.001
ADVANCED MUCOSAL EVALUATION I	Mucosal Inspection	Total time	671.5 sec (632.5, 715.4)	751.4 sec (624.7, 877.4)	0.44
		% lesions	100% lesions (92, 100)	96% lesions (92, 96)	0.04
COLONOSCOPY MODULE 1, CASE 9	Loop Reduction	Time to cecum	242 sec (180, 324)	373 sec (257, 506)	0.03
COLONOSCOPY MODULE 2, CASE 10	Loop Reduction	Time to cecum	140 sec (119, 161)	237 sec (167, 331)	0.005
BLEEDING MODULE 1, CASE 5	Retroflexion, Targeting	Total time	74 sec (66, 115)	212 sec (85, 212)	0.02
BLEEDING MODULE 1, CASE 6	Retroflexion, Targeting	Total time	81 sec (64, 89)	108 sec (97, 154)	0.03

Table 3.

Proficiency-benchmarks established as the median performance of experienced endoscopists

GI MENTOR MODULE	FES SKILL PRACTICED	METRICS	BENCHMARK
ENDOSCOPIC NAVIGATION	Scope Navigation, Targeting	Total time	9 min 19 sec
ADVANCED MUCOSAL EVALUATION I	Mucosal Inspection	Total time % lesions	11 min 11 sec 100% lesions
COLONOSCOPY MODULE 1, CASE 9	Loop Reduction	Time to cecum	4 min 12 sec
COLONOSCOPY MODULE 2, CASE 10	Loop Reduction	Time to cecum	2 min 30 sec
BLEEDING MODULE 1, CASE 5	Retroflexion, Targeting	Total time	1 min 14 sec
BLEEDING MODULE 1, CASE 6	Retroflexion, Targeting	Total time	1 min 21 sec

Phase II

Nine inexperienced endoscopists completed the repetition curriculum while fourteen completed the proficiency curriculum. Mean FES manual skills exam score for the proficiency group was significantly higher than that of the historical and national cohorts (Figure 2). Only the proficiency group had a 100% pass rate. Pass rate of the proficiency group was significantly higher than that of the historical cohort (100% vs. 61.5%, p=0.01; Table 4).

Figure 2.

Mean +/− SEM scaled scores in the FES manual skills exam across groups. * and ** denote statistical significance.

Table 4.

Pass rates for FES manual skills exam across groups.

^*denotes statistical significance.

The repetition group spent an average of 5.9 (+/− 1.3 SD) hours on the VR simulator compared to the proficiency group’s average of 4.8 (+/− 2.4 SD) hours (p=0.25).

Discussion

A VR curriculum for flexible endoscopy results in a higher pass rate and score on the FES manual skills exam than an endoscopy rotation alone. All 14 proficiency curriculum participants, including endoscopy-naïve interns, and 89% of the comparison group passed the FES manual skills exam on their first attempt. With the FES mandate from the ABS rapidly approaching, this study is the first to have demonstrated the effectiveness of a proficiency-based VR curriculum for preparing general surgery residents for the FES manual skills exam.

Proficiency-based curricula have been demonstrated to be effective in preparing residents for skill certifications such as the Fundamentals of Laparoscopic Surgery (FLS) [8]. Furthermore, Stefanidis et al (2010) suggested performance benchmarks may help improve resident motivation and simulation participation [9]. Our work builds on the growing body of literature that has demonstrated the effectiveness of proficiency-based education [10, 11] and provides performance benchmarks that surgical residents across the country may utilize to prepare for the FES manual skills exam.

After receiving feedback for the first two repetitions of each module, participants in both groups engaged in self-directed learning. Our results demonstrate higher FES scores after engagement in the proficiency curriculum compared to an endoscopy rotation alone and is consistent with prior literature that demonstrated improved performance in gastroenterology fellows who engage in a self-directed VR endoscopy curriculum [12]. Thus, trainees can continue to make improvements in performance toward proficiency benchmarks and a passing performance in FES without the need for time- and cost-intensive longitudinal coaching beyond two repetitions.

Both the repetition and proficiency curricula yielded similar performance though only the proficiency curriculum yielded a 100% pass rate and performance significantly greater than our institutional historical control. Borrowing elements from the expert performance framework, both curricula incorporated coaching and feedback at the beginning of the training period for each task to allow participants to engage in some deliberate practice (DP), i.e. receive immediate and focused feedback to practice specific skills to improve performance, in their VR endoscopic training. Incorporation of DP into both curricula for the first two trials on the simulator may have contributed to general performance improvement for both groups, for DP-based training has previously been shown to result in superior performance in other surgical skills [7, 13]. Thus, performance improvement from DP may have masked differences between the two groups. While there is no statistical difference between the repetition and proficiency groups, the 100% pass rate of the proficiency group is promising and merits further investigation in a larger, multi-institutional study to assess resident performance on the FES examination after proficiency training.

Interestingly, in the repetition group, seven of nine participants (77.8%) reached proficiency within 10 repetitions on four of the six curriculum tasks (Colonoscopy Module 1/Case 9, Colonoscopy Module 2/Case 10, Bleeding Module 1/Cases 5 and 6) despite not being given benchmarks to achieve. While no participant in the repetition group reached proficiency on the Endoscopic Navigation or Advanced Mucosal Evaluation I tasks, only one participant in the repetition group failed the FES manual skills exam. It is possible that the proficiency benchmarks set in this study for tasks such as Endoscopic Navigation or Advanced Mucosal Evaluation I may be too strict, and more relaxed benchmarks may be sufficient for trainees to pass the FES manual exam. However, additional research is necessary to determine whether the benchmarks presented in our proficiency curriculum can be reduced and still result in similar performance and pass rate effectiveness.

This study does have several limitations. This study was performed at a single academic institution with residents from a single general surgery training program. Due to limitations in time, funding, and access to potential study subjects at a single institution, study numbers were small. However, a priori power calculation suggested ten participants were needed in the repetition and proficiency groups to detect a difference in pass rate when compared to our institution’s historical control. Thus, the study was appropriately powered to detect a statistically significant difference in pass rate between the proficiency group and historical control (100% vs. 61.5%, p=0.01) but retrospectively was underpowered to detect a difference between the repetition group and our institutional historical control (88.9% vs. 61.5%, p=0.16). Similar to our comparison of the proficiency group to the national cohort (100% vs. 80.1%, p=0.06), other institutions may not see a statistically significant difference in pass rate; however, post hoc power analysis suggests the proficiency group was underpowered for comparison to the national cohort. A multi-institutional trial with a greater number of participants would allow for a more thorough evaluation of the generalizability of the proficiency curriculum developed in this study.

Another limitation of this study is that no baseline FES testing was conducted for the subjects. The study was designed with a historical control to avoid practice effect amongst our participants that may be unrelated to the intervention curricula in this study. Given that all study participants and historical control subjects were inexperienced endoscopists, we hypothesized that participants in this study would approximate a 60% pass rate (similar to our historical control) if they had not participated in a VR curricula. However, we had no contemporary data for direct comparison.

This curriculum is limited to modules available to the GI Mentor VR simulator as that simulator was available at our institution; therefore, generalizability of this curriculum to other institutions is limited to those with access to the same simulator. However, this simulator is used to administer the FES examination, and there are currently 66 institutions within the United States that are FES test centers and have access to this simulator [14]. To prepare for the FES examination, residents near an FES test center should seek opportunities to engage in this proficiency-based curriculum. Other curricula that do not require a VR simulator may also be in development by other research groups.

In the repetition group, three of four (75%) endoscopy-naïve participants passed FES while all five (100%) endoscopy-naïve participants in the proficiency group passed FES. Thus, proficiency training also yielded passing scores for residents who had no endoscopy experience. A reasonable hypothesis is that by engaging in proficiency training prior to an endoscopy rotation, residents acquire baseline endoscopic skills that may ultimately be improved through clinical training. While increases in FES manual skills score have previously been demonstrated to correlate with improved performance in colonoscopy in real patients [15], it is beyond the scope of this study to determine whether these skills will immediately transfer to clinical practice. Additional research is necessary to determine whether performance gains made through utilization of this proficiency-based VR curriculum will yield improved performance in real clinical settings as measured by the Global Assessment of Gastrointestinal Endoscopic Skill (GAGES) [16].

Conclusion

Resident participation in a VR curriculum that included feedback on the first two repetitions yielded superior performance on the FES manual skills exam compared to those completing an endoscopy rotation alone, and a proficiency-based curriculum resulted in a 100% FES pass rate at a single institution. With the impending ABS mandate for residents to achieve FES certification, validated VR curricula can provide a structured approach for residents to acquire endoscopic skills in preparation for the FES examination. Further research is necessary to determine whether skills gained in VR transfer to improved quality of endoscopy in a live clinical setting.

Figure 3.

Mean +/− SEM of total hours spent on the VR simulator for each curriculum group. p=0.25