Characterizing the Learning Curve of a Virtual Intracorporeal Suturing Simulator VBLaST-SS©

Abstract

Background

The virtual basic laparoscopic skill trainer suturing simulator (VBLaST-SS©) was developed to simulate the intracorporeal suturing task in the FLS program. The purpose of this study was to evaluate the training effectiveness and participants’ learning curves on the VBLaST-SS© and to assess whether the skills were retained after two weeks without training.

Methods

Fourteen medical students participated in the study. Participants were randomly assigned to two training groups (7 per group): VBLaST-SS© or FLS, based on the modality of training. Participants practiced on their assigned system for one session (30 minutes or up to 10 repetitions) a day, five days a week for three consecutive weeks. Their baseline, post-test, and retention (after 2 weeks) performance were also analyzed. Participants’ performance scores were calculated based on the original FLS scoring system. The cumulative summation (CUSUM) method was used to evaluate learning. Two-way mixed factorial ANOVA was used to compare the effects of group, time point (baseline, post-test, and retention), and their interaction on performance.

Results

Six out seven participants in each group reached the predefined proficiency level after 7 days of training. Participants’ performance improved significantly (p<0.001) after training within their assigned group. The CUSUM learning curve shows that one participant in each group achieved 5% failure rate by the end of the training period. Twelve out of fourteen participants’ CUSUM curves showed a negative trend toward achieving the 5% failure rate after further training.

Conclusion

The VBLaST-SS© is effective in training laparoscopic suturing skill. Participants’ performance of intracorporeal suturing was significantly improved after training on both systems and was retained after two weeks of no training.

Introduction

Since its inception, the adoption of minimally invasive surgical techniques has increased dramatically due to improved patient outcomes, including reduced pain, faster recovery, and minimal scarring. However, these techniques have increased the skill requirements for successful and safe surgical procedures. Surgeons must learn to overcome challenges related to a lack of depth perception from the 2D display and decreased haptic feedback from the tools. Surgical skill is one of the primary attributable factors for errors in surgery. A review of 106 medical malpractice cases found 258 with surgical errors that resulted from a “lack of technical competence or knowledge” [1]. Of these, 58% were attributed to a surgeon “lacking experience or skill with the task at hand.” A recently published research agenda from the Society of American Gastrointestinal and Endoscopic Surgeons (SAGES) indicates that the means to “best train, assess, and maintain proficiency of surgeons” remains the top priority among respondents [2].

In order to attain the necessary skills, it has been recognized that the traditional apprentice model is insufficient. Trainees must undergo extensive practice prior to performing procedures in the operating room. This has resulted in the development of a range of simulators and models, including inanimate video box trainers and virtual reality (VR) computer-based simulators. The Fundamentals of Laparoscopic Surgery (FLS) program was developed by SAGES and the American College of Surgeons (ACS). It provides standardized training and evaluation of laparoscopic surgery skills. The FLS Training Simulator (Trainer Box), based on the McGill Inanimate System for Training and Evaluation of Laparoscopic Skills (MISTELS) program, was developed for practicing basic technical skills [3–7]. Training on the FLS trainer box was proven to be effective, however, practicing on FLS trainer box requires a large number of consumables and the performance assessment requires expert evaluation, which is subjective in nature, and manual measurement also requires extra time and effort. Compared to the FLS trainer box, VR simulators can provide unlimited practice and objective feedback immediately after each practice. Without the need for consumables, it can also be more cost-effective in the long term. The FLS trainer box and instruments cost $2,770, plus $4.50 for consumables per intracorporeal suturing practice trial. The VBLaST-SS© system costs about $6,400 for the computer and physical components. With >800 practice repetitions the simulator becomes more cost effective than the FLS trainer box.

The Virtual Basic Laparoscopic Skill Trainer (VBLaST©) was developed to simulate the five tasks of the FLS trainer box: peg transfer, precision cutting, ligating loop, suture with extracorporeal knot, and suture with intracorporeal knot. Of the five tasks, the VBLaST suturing simulator (VBLaST-SS©) simulates the FLS intracorporeal suturing task [8]. Intracorporeal suturing is typically considered the most challenging of the FLS tasks and requires a significant learning curve to master [9, 10]. A 2013 survey of fellowship directors in the United States revealed that 56% of new fellows arriving at different fellowship subspecialties were not proficient in suturing [11]. If shown to be valid and effective, the VBLaST-SS© has the potential to improve the ability of trainees to practice suturing, since this task often requires the most repetitions to reach proficiency, and thus a large number of consumables, including suture and Penrose drain for each repetition.

Previous studies have shown the validity of different tasks in VBLaST© [12–14], and the test content evidence [15] of the VBLaST-SS© was recently reported [16]. In addition to the test content evidence, simulators must be effective for skill training for them to be useful for surgical trainees. The learning curve is the improvement in performance that novice trainees undergo as they become familiar with a task [17, 18]. The documentation of the learning process is an important tool for analyzing a trainee’s learning outcome and monitoring their quantitative scores across trials. It is widely used to measure the competency of surgical skills [19–21] and to assess the training effectiveness of new training systems [22, 23]. By analyzing the features of the learning curve on different training systems, we can compare differences in the learning process to compare and evaluate the effectiveness of the systems in terms of training. The cumulative summation (CUSUM) is a criterion-based evaluation of the learning process [24, 25]. It can be used to analyze the performance over time within a set of criteria. Previous studies have assessed the learning curve of peg transfer and pattern cutting tasks of the VBLaST© simulator [23, 25, 26]. The purpose of this study was, by constructing the CUSUM learning curves for participants’ performance on both systems, to evaluate the training effectiveness of the VBLaST-SS© and to assess whether the skills are retained after two weeks without training.

Methods

Demographics

Fourteen medical students from first to third year (8 male and 6 female) participated in this study. The average age was 23.4 years (range 21–27). All participants were novices in intracorporeal suturing at the start of the study and did not have prior exposure to either the FLS box trainer or the VBLaST-SS© simulator. The study was approved by the University at Buffalo Institutional Review Board and all participants provided written informed consent prior to their participation.

Training task

The standard FLS intracorporeal suturing task was used as the training task on the VBLaST-SS© and the FLS trainer box. Figure 1 shows the view in the virtual environment for the VBLaST-SS© and from the laparoscope for the FLS box trainer. The task requires the participant to close the slit in a Penrose drain intracorporeally, which includes piercing the needle through two black dots on the Penrose drain, tying three knots, and cutting both sides of the suture intracorporeally. The first knot must be a double throw followed by two single-throw knots. Participants were asked to transfer the needle to the alternate hand between each throw.

Figure 1.

Intracorporeal suturing task in VBLaST-SS© (left) and in FLS trainer box (right)

Procedures

Participants were randomly assigned to two training groups upon entering the study (7 per group): VBLaST-SS© training group and FLS training group. At the start of the first session, participants watched a video that instructed them on the intracorporeal suturing task. Then they were asked to perform the task on both systems as a measure of their baseline performance. The presentation of the systems was counterbalanced.

After the baseline test, participants practiced for one session a day, five days a week, for three consecutive weeks. They performed a total of 15 training sessions. For each training session, participants practiced intracorporeal suturing for 30 minutes or up to 10 trials on their assigned system. A schematic of the overall testing protocol is provided in Figure 2. The performance scores on the respective systems were calculated based on the original FLS scoring system [27, 28]. The performance score was calculated by subtracting the normalized penalty score from the timing score. The completion time in seconds was subtracted from the cut off time of 600 seconds as the timing score. The penalty score includes the measurement of accuracy: deviations to the black dots on both sides of the Penrose drain (mm), remaining incision gap (mm), and knot security (0 for secure knot, 10 for slipping knot, and 20 for undone or open knot) [29]. Completion time was measured from the time the needle driver entered the field of view to the cutting of the suture. This was measured by the researcher using a stopwatch. The VBLaST-SS© automatically recorded the deviations, incision gap, and knot security to calculate the performance score. For the physical box trainer group, the researcher present measured each metric for each repetition completed. During the practice session, the participants were informed of the scoring metrics used to measure their performance. They did not receive any feedback except for completion time, as the Penrose drain measurement for errors requires additional time. To maintain consistency with the FLS training group, the VBLaST-SS© group only received feedback on completion time.

Figure 2.

Schematic of the experimental design

The final repetition performed on training day 15 was used to determine post-training performance on the assigned system. Two weeks after the post-test, participants came back to perform the task on both systems for skill transfer and retention assessment.

Data analysis

The proficiency score used for performance data analysis is 408.425, which was the average performance score on the FLS trainer box from FLS certified participants in a previous study [16]. For the VBLaST-SS© simulator, when the participant did not penetrate both sides of the Penrose drain or the software did not recognize their performance, the software provided the maximum value of the deviations (i.e. 40.840706 mm) and incision gap (i.e. 3 mm). These results were considered erroneous results and those data points were excluded. In total, seven erroneous results were excluded from the analysis.

A two-way mixed factor analysis of variance (ANOVA) was used to compare the effect of group, time point (baseline, post-test, and retention), and their interaction on performance. When a significant effect was found, post hoc analyses were completed for paired comparisons. IBM® SPSS® Statistics Version 25 for Windows (IBM Corp., Armonk, New York, USA) was used for statistical analysis.

CUSUM curves were generated for each participant. The intermediate criterion score (280.8) was defined based on previous studies [5, 20]. When a participant’s performance score was equal to or better than the criterion score, the task was considered as success (1), when the performance score was lower than the criterion score it was considered as failure (0). The acceptable failure rate (p₀) was set at 5% and the unacceptable failure rate (p₁) was set at 10% (2 × p₀). Type I and type II errors (α and β) were set as 0.05 and 0.2. Based on these variables, two decision limits (h₀ and h₁) and the s variable were calculated. For each success, the s variable was subtracted from the previous CUSUM score. For each failure, the value 1-s was added to the previous CUSUM score. A negative trend of the CUSUM graph indicates success and a positive trend indicates failure. When the CUSUM score remains above the decision limit h₁, the actual failure rate is significantly greater than the acceptable failure rate, with a probability of type I error equal to α. When the CUSUM score remains below the decision limit h₀, the actual failure rate does not differ significantly from the acceptable failure rate, with a probability of type II error equal to β [23–25, 30, 31]. Table 1 shows the CUSUM variables that were used for data analysis. Table 2 shows the example CUSUM chart of a participant (MS 14).

Table 1.

CUSUM variables

Variable	Value
Success score	280.8
P0	0.05
p1 = 2 × p0	0.10
α	0.05
β	0.20
P = ln(p1/p0)	0.69
Q = ln[(1 − p0)/(l − p1)]	0.05
s = Q/(P + Q)	0.07
1 − s	0.93
a = ln[(1 − β)/α]	2.77
b = ln[(1 − α)/β]	1.56
h0 = −b/(P + Q)	−2.09
h1 = a/(P + Q)	3.71

Table 2.

CUSUM chart calculation for MS 14

Trial Number	Score	Binary score	CUSUM Score
1	0	0	0.93
2	255.10093	0	1.86
3	223.40799	0	2.79
4	184.42589	0	3.72
5	363.88182	1	3.65
6	75.68146	0	4.58
7	*	*	4.58
8	294.46365	1	4.51
9	426.49029	1	4.44
10	383.14467	1	4.37
11	226.94776	0	5.3
12	279.35423	0	6.23
13	308.41935	1	6.16
14	450.04779	1	6.09
15	444.60605	1	6.02
16	287.34268	1	5.95
17	482.70021	1	5.88
18	103.80327	0	6.81
19	420.57151	1	6.74
20	438.96558	1	6.67
21	420.2844	1	6.6
22	333.80592	1	6.53
23	429.96274	1	6.46
24	430.85288	1	6.39
25	479.74901	1	6.32
26	196.61908	0	7.25
27	431.47582	1	7.18
28	473.3345	1	7.11
29	445.29359	1	7.04
30	473.48165	1	6.97
31	468.45398	1	6.9
32	407.98992	1	6.83
33	496.57081	1	6.76
34	473.04472	1	6.69
35	476.82367	1	6.62
36	490.16109	1	6.55
37	506.54347	1	6.48
38	444.93994	1	6.41
39	489.40553	1	6.34
40	420.29998	1	6.27
41	476.05677	1	6.2
42	473.70564	1	6.13
43	473.03559	1	6.06
44	469.00703	1	5.99
45	501.67904	1	5.92
46	383.48288	1	5.85
47	377.69942	1	5.78
48	490.68327	1	5.71
49	481.29092	1	5.64
50	468.56295	1	5.57
51	488.44658	1	5.5
52	436.17753	1	5.43
53	390.6599	1	5.36
54	440.77799	1	5.29
55	491.68815	1	5.22
56	484.18817	1	5.15
57	495.621	1	5.08
58	492.55642	1	5.01
59	490.20632	1	4.94
60	459.90318	1	4.87
61	491.07815	1	4.8
62	507.50502	1	4.73
63	494.91258	1	4.66
64	477.06102	1	4.59
65	464.509	1	4.52
66	465.14537	1	4.45
67	486.83555	1	4.38
68	490.5886	1	4.31
69	496.56814	1	4.24
70	465.01206	1	4.17
71	477.17145	1	4.1
72	503.22105	1	4.03
73	424.09693	1	3.96
74	451.15087	1	3.89
75	455.53983	1	3.82
76	433.3154	1	3.75
77	474.41674	1	3.68
78	448.41011	1	3.61
79	439.98668	1	3.54
80	462.46075	1	3.47
81	490.29651	1	3.4
82	*	*	3.4
83	391.32286	1	3.33
84	498.75015	1	3.26
85	327.67215	1	3.19
86	444.47728	1	3.12
87	507.58928	1	3.05
88	491.34691	1	2.98
89	499.75632	1	2.91
90	516.41457	1	2.84
91	500.50974	1	2.77
92	515.43915	1	2.7
93	507.57441	1	2.63
94	504.712	1	2.56
95	*	*	2.56
96	508.4395	1	2.49
97	500.0481	1	2.42
98	449.89995	1	2.35
99	515.29477	1	2.28
100	479.65249	1	2.21
101	513.19728	1	2.14
102	509.58397	1	2.07
103	464.66536	1	2
104	472.90166	1	1.93
105	508.53848	1	1.86
106	503.31702	1	1.79
107	492.21951	1	1.72
108	500.69461	1	1.65
109	455.00836	1	1.58
110	504.12988	1	1.51
111	485.24949	1	1.44
112	480.73942	1	1.37
113	509.74176	1	1.3
114	461.46976	1	1.23
115	516.89201	1	1.16
116	486.74702	1	1.09
117	501.18901	1	1.02
118	487.24801	1	0.95
119	479.92299	1	0.88
120	505.54657	1	0.81
121	510.38976	1	0.74
122	496.45825	1	0.67
123	506.28711	1	0.6
124	465.4561	1	0.53
125	508.56283	1	0.46
126	498.59422	1	0.39
127	443.04109	1	0.32
128	501.47814	1	0.25
129	517.48509	1	0.18
130	511.42518	1	0.11
131	497.84248	1	0.04
132	511.3456	1	−0.03
133	499.14338	1	−0.1

^*The score was discarded due to system error

Results

Performance

In both training groups, participants’ performance significantly improved after the training sessions. Most of the participants reached the predefined proficiency level (408.425) and retained their performance two weeks after completing the training. Based on the performance curves (Figure 3), six of the participants in each group reached the predefined proficiency level after 7 days of training. During the retention test, all participants in the FLS group and 5 out of 7 participants in the VBLaST-SS© group kept their performance score higher than the proficiency score.

Figure 3.

Average Learning Curves for Participants in Different Groups (proficiency score is 408.425, intermediate score is 280.8, the error bars represent standard deviation)

The average performance scores for both training groups at baseline, post-test and retention within their assigned training group are shown in Figure 4. The two-way mixed factorial ANOVA showed that there was a significant difference between time points (baseline, post-test, and retention) within assigned training groups (p<0.001), however, the effect of group (p=0.482) and group × time interaction (p=0.203) were not significant. Within the participants’ assigned group, there was a significant difference between baseline and post-test (p<0.001), but no significant difference between post-test and retention (p=0.253), which indicates that skill acquired on both the FLS and the VBLaST-SS© was retained for the two-week retention period.

Figure 4.

Performance on assigned training systems (the error bars represent standard deviation)

CUSUM learning curve for the VBLaST-SS© group

Under the intermediate criterion of 280.8, one participant (MS 2) in the VBLaST-SS© group achieved 5% failure rate at trial 117 (Figure 5). The final performance of MS 5, MS 6, MS 12 and MS 14 were between the two decision limits h₀ and h₁. Six out of seven participants demonstrated a transition point (the peak CUSUM score where the slope of the CUSUM line change from positive to negative), which indicates a tendency to achieve the 5% failure rate after further training (MS 2 at trial 16, MS 5 at trial 9, MS 6 at trial 44, MS 10 at trial 30, MS 12 at trial 68, MS 14 at trial 26).

Figure 5.

CUSUM Learning Curve of VBLaST-SS© Group (Intermediate Criterion Score = 280.8, p₀ = 5%)

CUSUM learning curve for the FLS group

Under the criterion of 280.8, one participant (MS 13) in the FLS group achieved 5% failure rate at trial 88 (Figure 6). The final performance of MS 1, MS 4 and MS 9 were between the two decision limits h₀ and h₁. Six out of seven participants demonstrated a transition point (MS 1 at trail 44, MS 3 at trial 32, MS 4 at trial 9, MS 9 at trial 22, MS 11 at trail 80, MS 13 at trial 9).

Figure 6.

CUSUM Learning Curve of FLS Group (Intermediate Criterion Score = 280.8, p₀ = 5%)

Discussion

In this study, we compared the training effectiveness between two different systems while participants completed the same training schedule. Overall, the study showed that the VBLaST-SS© is effective for training laparoscopic suturing skill.

Previous studies have described three main features in the learning curve: the starting point indicate the initial performance level, the slope or the rate of learning represents how quickly the learner’s performance reaches a certain level, and when the performance reaches the plateau or the expert level it stabilizes [17, 18, 32]. For the current study, both groups had an equivalent starting point for the baseline test (Figure 4).

In terms of the rate of learning, most of the participants in both training groups reached the predefined proficiency level on two consecutive attempts and five additional attempts [10]. For the FLS group, the average time spent to reach proficiency was 211 ± 58 minutes and 67 ± 19 repetitions. One participant in the VBLaST group did not reach the proficiency level after two consecutive plus five additional attempts. The average time spent for the six participants in this group who reached the proficiency level was 117 ± 28 minutes and 54 ± 21 repetitions. The learning curves on both systems have similar main features. This indicates that for novice learners the learning process on the VBLaST-SS© simulator is similar to the learning process on the FLS box, which supports the VBLaST-SS© simulator as an effective training tool for intracorporeal suturing skill.

Similar slopes of learning were observed in other training studies for intracorporeal suturing [10, 33]. Stefanidis et al. [10] showed that participants who did not receive basic laparoscopic skill training reached the expert level at 50 ± 16 repetitions (310 ± 98 min). In another study by Vossen et al. [33], most of the participants achieved the training effect after 20–30 repetitions (2-3 hours of training). While similar slopes were observed, each of the studies had slightly different tasks and training schedules, limiting the direct comparability of the findings.

In several previous studies [27, 28, 34, 35], the proficiency score for intracorporeal suturing task was defined as 512 using the same scoring system as in this study. In a study comparing the effect of different feedback given to novice trainees by Stefanidis et al. [34], the group received limited feedback took range 47–213 minutes (21–67 repetitions) to reach proficiency (512). In an earlier study by Stefanidis et al. [35], participants took average 5.6 ± 1.4 hours (336 ± 84 minutes and 54 ± 22 repetitions) to reach proficiency (512). Stefanidis et al. [27] also compared the effect of different training conditions on novice participants. Participants in the group with regular training setting trained 239 ± 69 minutes (59 ± 14 repetitions) to reach proficiency (512). The gap of training time between the previous studies and the current study indicate that there is possibility to reach higher performance score with additional training.

Although the learning curve can be used to analyze the learning process, with limited information contained in the learning curve, it is hard to examine performance over time. Therefore, the CUSUM chart can be used to more closely analyze the performance over time during the learning process, as has been done in previous studies [23–26, 30, 31, 36–40]. For tasks on the VBLaST simulator, the CUSUM has been used to assessing learning curves of peg transfer [25] and pattern cutting [23, 26] tasks. Zhang et al. [25] characterized the learning curve for peg transfer task on the VBLaST and the FLS box. Out of six participants in the VBLaST-PT© training group, 5 achieved junior level, 3 achieved intermediate level, and 2 achieved senior level by 150 trials. For the FLS training group, all 6 participants achieved junior and intermediate level, and 3 achieved senior level. Linsk et al. [23] characterized the learning curve for pattern cutting task on the VBLaST and the FLS box. 5 out of 7 participants in the FLS training group and 2 out 7 participants in the VBLaST-PC© group achieved intermediate criterion by 150 trials, none of the participants in both groups achieved proficiency criterion. Nemani et al. [26] evaluated the CUSUM learning curve for pattern cutting task on the VBLaST and the FLS box. Four out of six participants in the VBLaST group and three out of seven participants in the FLS group achieved the acceptable failure rate during the 12 days training.

In the current study, we evaluated the CUSUM learning curve of intracorporeal suturing task on the VBLaST simulator and the FLS trainer box. Based on the intermediate criterion, although only one participant in each group reached the 5% failure rate on the CUSUM learning curve, six out of seven participants in each group demonstrated a transition point. Also, apart from the participants who reached the 5% failure rate, for four participants in the VBLaST-SS© group and three participants in the FLS group, their final performance fell between the two decision limits, which indicates the requirement for more observation. Given the fact that the intracorporeal suturing task is more difficult than other tasks in the FLS program, with more practice, those participants’ performance is likely to have reached the 5% failure rate. The difficulty of intracorporeal suturing task likely resulted in fewer participants reaching the 5% failure rate compared to those in the peg transfer and pattern cutting studies.

One of the advantages of the CUSUM learning curve is that it is sensitive to variations in each task repetition. Unlike the performance learning curve where outlier performance is masked by the calculation of the daily average, each performance contributes to the CUSUM learning curve. In the CUSUM calculation, for every failure (i.e. performance score is lower than the criterion score), the value 0.93 was added to the previous CUSUM score. But when the performance was a success (score is equal or higher than the criterion score), only 0.07 was subtracted from the previous CUSUM score. Therefore, if an outlier failure performance occurred, the CUSUM score will increase substantially. To bring the score back to the level before the failure performance, will require more than 13 continually success performances. For some participants, even if they reached proficiency in the performance learning curve, they still could not demonstrate a transition point in the CUSUM learning curve because they cannot maintain continuous successful performance. This is important for tasks such as those for FLS certification, since the trainee should be able to consistently perform the task well to be able to pass during the high-stakes examination.

Limitations and Future Work

This study had a few limitations that should be acknowledged. First, the sample size in the study was relatively small, which limited the statistical power. However, the sample size was consistent with other studies on FLS skills learning and the experimental protocol limited the ability to test a large number of subjects [21, 24, 28, 35]. Second, the amount of training may not have been sufficient to reach a true plateau of performance. In this study, the training entailed 10 repetitions or 30 minutes per day, 5 days per week for 3 consecutive weeks. However, because of the difficulty of the intracorporeal suturing task, more training sessions are needed in order to train participants to an expert level. In a study by O’Connor et al. [41] participants have to have 8 hours’ training in order to reach the plateau of laparoscopic suturing. Although participants’ performance stabilized at the end of the training period, more training might be useful to see if there are further improvement. Participants in the current study did not attain the previously published proficiency score of 512, indicating that it is possible to further improve performance with added training. A third limitation is that we only evaluated the skill retention after two weeks. Several studies have shown that skills could be retained after a longer interval [35, 42]. However, in those studies, participants typically underwent a longer training period or trained to a higher performance level. A longer retention interval should be considered in future studies in combination with a longer training period or higher performance level. Finally, we only evaluated the VBLaST simulator’s training effectiveness in novice learners. It could be important to look at the training effectiveness of more advanced suturing skill with more experienced users. More experienced users like surgical residents might benefit differently from practicing on the VBLaST simulator.

Future studies should consider the effectiveness of the VBLaST-SS© for training more advanced skill with experienced users and the transfer of acquired skills from the virtual simulator to actual surgery conditions. As a first step, future research should consider the skill transfer between the VBLaST-SS© and the FLS trainer box, since the current study only considered this at the time of the retention test.