Abstract
Background
Intracorporeal suturing is one of the most important and difficult procedures in laparoscopic surgery. Practicing on a FLS trainer box is effective but requires large number of consumables and the scoring is somewhat subjective and not immediate. A virtual basic laparoscopic skill trainer (VBLaST©) was developed to simulate the five tasks of the FLS Trainer Box. The purpose of this study is to evaluate the face and content validity of the VBLaST suturing simulator (VBLaST-SS©).
Methods
Twenty-five medical students and residents completed an evaluation of the simulator. The participants were asked to perform the standard intracorporeal suturing task on both VBLaST-SS© and the traditional FLS box trainer. The performance scores on each system were calculated based on time (seconds), deviations to the black dots (mm), and incision gap (mm). The participants were then asked to finish a 13-item questionnaire with ratings from 1 (not realistic/useful) to 5 (very realistic/useful) regarding the face validity of the simulator. A Wilcoxon signed rank test was performed to identify differences in performance on the VBLaST-SS© compared to that of the traditional FLS box trainer.
Results
Three questions from the face validity questionnaire were excluded due to lack of response. Ratings to 8 of the remaining 10 questions (80%) averaged above a 3.0 out of 5. Average intracorporeal suturing completion time on the VBLaST-SS© was 421 (standard deviation = 168 sec.) seconds compared to 406 (175 sec.) seconds on the box trainer (p=0.620). There was a significant difference between systems for the incision gap (p=0.048). Deviation in needle insertion from the black dot was smaller for the box trainer than the virtual simulator (1.68 vs. 7.12, p<0.001).
Conclusion
Participants showed comparable performance on the VBLaST-SS© and traditional box trainer. Overall, the VBLaST-SS© system showed face validity and has the potential to support training for the suturing skills.
Keywords: Virtual reality, Suturing simulation, The Fundamentals of Laparoscopic Surgery (FLS), Intracorporeal suturing, Face validity
Introduction
Laparoscopic surgery is a widely employed surgical technique in the operating room. The benefits of laparoscopic surgery include less postoperative pain, quicker recovery, and minimal scarring [1, 2]. Novice surgeons looking to adopt laparoscopic techniques encounter a significant learning curve due to challenges like lack of depth perception, lack of haptic feedback, and limited range of motion [3]. To overcome this obstacle, there is an increasing emphasis on simulation based training to develop the fundamental skills specific to laparoscopic surgery [4, 5].
The Fundamentals of Laparoscopic Surgery (FLS) program was developed to help surgical residents and surgeons develop the psychomotor skills and dexterity required during basic laparoscopic surgery. The FLS program utilizes a Fundamentals of Laparoscopic Surgery Training Simulator (FLS Trainer Box) to teach basic technical skills pertinent to laparoscopic surgery. The FLS Trainer Box was developed based on the McGill Inanimate System for Training and Evaluation of Laparoscopic Skills (MISTELS) program [6, 7] and contains five of the original seven tasks; peg transfer, precision cutting, placement and securing of ligating loop, simple suture with extracorporeal knot, and simple suture with intracorporeal knot. Among the five tasks, intracorporeal suturing is the most difficult technical skill to acquire. Even for experienced surgeons, intracorporeal suturing and knot tying is considered a significant barrier to performing laparoscopic surgery [8].
Practicing on a FLS trainer box is effective but requires large number of consumables (a Penrose drain and suture per practice trial, costing ~$4.50 per trial), and the scoring is somewhat subjective (despite extensive proctor training) and not immediate. In contrast, virtual reality simulation provides unlimited practice without using consumables. Moreover, they can provide instant and objective feedback of the trainee’s performance. Studies have shown that existing virtual reality simulators demonstrated face validity and construct validity [1, 9, 10]. In addition, virtual reality simulator training has also been shown to be effective for acquisition of laparoscopic surgery skills [5, 11–17].
A virtual basic laparoscopic skill trainer (VBLaST©) was developed to simulate the five tasks of the FLS Trainer Box. The purpose of this study is to evaluate the face and content validity of the VBLaST suturing simulator (VBLaST-SS©) which was developed by Qi et al. [18].
Methods
The participants were asked to perform the standard FLS intracorporeal suturing task on each of the VBLaST-SS© (Figure. 1) and the FLS Trainer Box. In the FLS Trainer Box, to keep consistency with the VBLaST-SS©, the needle was placed in the needle driver and placed in the box trainer before the task started. Timing began when the participant started to operate the needle drivers. The task included piercing the needle through two black dots on a Penrose drain, then tying three knots intracorporeally to close the slit in the Penrose drain. The first knot must be a double throw, and the following two knots must be single throws. Participants were asked to transfer the needle to the other hand between each throw. After three knots were finished, participants were asked to cut both sides of the suture inside the box trainer.
Figure 1.
The intracorporeal suturing task in VBLaST-SS©
At the beginning of the session, participants were asked to complete a questionnaire regarding their demographic information and experience on surgery, simulator and videogames. After a video demonstration, participants had 15 minutes or up to 3 times to practice on each system. The presentation order of the systems was balanced. The trial session was 10 minutes on each system, starting with the system used first in the practice session. The trial session was video recorded for future analysis. After the participants completed the session, the VBLaST-SS© automatically calculated the participant’s performance, and the Penrose drain they used in the FLS Trainer Box was collected to be measured. The performance scores on each system were calculated based on the time taken (seconds), deviations to the black dots (mm) - the intended piercing points for the needle, and the incision gap (mm) post-suturing.
After performing the tasks, the participants were then asked to finish a 13-item questionnaire regarding the face and content validity of the simulator. Participants answered questions with ratings from 1 (not realistic/useful) to 5 (very realistic/useful). In addition, they finished a NASA-TLX questionnaire regarding their perceived workload.
Three participants’ performance data, apart from their completion time, on the VBLaST-SS© were excluded in the final analysis due to issues with their data. In these cases, the participants did not penetrate both sides of the Penrose drain or the software did not recognize their performance. The software then provided the maximum value of the deviations (i.e., 40.840706 mm) and the maximum value of the incision gap (i.e., 3mm). A Wilcoxon signed ranks test was performed to identify differences in performance on the VBLaST-SS© compared to the FLS Trainer Box.
Results
Demographics
Twenty medical students (MS, n=20), six residents (n=6) and a research assistant (n=1) voluntarily enrolled in this study. One participant withdrew from the study for scheduling reason. All 25 remaining participants completed the study. The demographic data is shown in Table 1.
Table 1.
Demographic data
| Age, average (range) | 25.8 (21–39) |
| Sex, female:male | 9.16 |
| Hand dominance, left:right | 0:25 |
| Corrected vision, yes:no | 11:14 |
| Current position, medical student:post medical school | 19:6 |
| Surgical experience (observed), yes:no | 13:12 |
| Surgical experience (assisted/performed), yes:no | 11:14 |
| FLS certification, yes:no | 4:21 |
| Simulator experience, yes:no | 6:19 |
| Video game experience, yes:no | 5:20 |
Face validity
Post-task questionnaire results are shown in Table 2. Questions 10 to 12 regarding VBLaST-SS© comparing to animal model were excluded from the analysis due to limited experience of the participants on animal models. Responses to 8 of the 10 questions (80%) averaged above a 3.0 out of 5. Those questions that rated the highest were the degree of realism of the target objects in the VBLaST-SS© compared to the FLS (3.88) and the usefulness of the VBLaST-SS© simulation in learning hand-eye coordination skills compared to the FLS (3.84). Those rated lowest were the quality of the force feedback in the VBLaST-SS© compared to the FLS (2.88) and the degree of realism of the instrument handling in the VBLaST-SS© compared to that in actual laparoscopic surgery (2.93).
Table 2.
Post-task questionnaire
| Questions: ratings from 1 (not realistic/useful) to 5 (very realistic/useful). | Mean Rating | SD | MS | SD | Post medical school | SD |
|---|---|---|---|---|---|---|
| 1. Realism of the target objects (how realistic they look) in the VBLaST-SS© task environment, compared to the corresponding task environment in the FLS | 3.84 | ±0.94 | 3.79 | ±1.03 | 4 | ±0.63 |
| 2. Realism of the instrument handling (how realistic it feels) in the VBLaST-SS©, compared to that in the FLS | 3.34 | ±0.85 | 3.34 | ±0.88 | 3.33 | ±0.82 |
| 3. Realism of the instrument handling (how realistic it feels) in the VBLaST-SS©, compared to that in actual laparoscopic surgery. | 2.93 | ±1.03 | 3 | ±1.12 | 2.83 | ±0.98 |
| 4. Overall realism of the VBLaST-SS© simulation (how it looks AND feels), compared to the corresponding FLS task | 3.44 | ±0.92 | 3.53 | ±0.96 | 3.17 | ±0.75 |
| 5. Quality of the force feedback (sensation of feeling the tools on the target and in the task space) in the VBLaST-SS© compared to the FLS | 2.88 | ±1.17 | 3 | ±1.15 | 2.5 | ±1.22 |
| 6. Usefulness of the force feedback (sensation of feeling the tools on the target and in the task space) in the VBLaST-SS© in helping your performance | 3.48 | ±1.23 | 3.58 | ±1.26 | 3.17 | ±1.17 |
| 7. Usefulness of the VBLaST-SS© simulation in learning hand-eye coordination skills, compared to the FLS | 3.88 | ±0.93 | 3.74 | ±0.99 | 4.33 | ±0.52 |
| 8. Usefulness of the VBLaST-SS© simulation in learning ambidexterity skills, compared to the FLS. | 3.64 | ±0.99 | 3.53 | ±1.07 | 4 | ±0.63 |
| 9. Overall usefulness of VBLaST-SS© in learning the fundamental laparoscopic technical skills compared to the FLS | 3.56 | ±1.00 | 3.42 | ±1.07 | 4 | ±0.63 |
| 13. How trustworthy VBLaST-SS© is to quantify accurate measures of performance. | 3.52 | ±0.92 | 3.63 | ±0.96 | 3.17 | ±0.75 |
Performance
Performance scores are shown in Table 3. Average intracorporeal suturing completion time on the VBLaST-SS© was 421 (standard deviation = 168) seconds compared to 406 (175) seconds on the box trainer (p=0.620). There was a significant difference between systems for the incision gap (p=0.048). Deviation in needle insertion from the black dot was smaller for the box trainer than the virtual simulator (1.68 vs. 7.12, p<0.001).
Table 3.
Comparison of the performance scores between VBLaST-SS© and Real FLS
| VBLaST-SS© | Real FLS | P-value | |
|---|---|---|---|
| Completion time (s) (mean ± SD) | 421.23 ± 167.56 | 406.31 ± 175.48 | 0.620 |
| Incision gap (mm) (mean ± SD) | 1.123 ± 0.856 | 0.757 ± 1.223 | 0.048 |
| Deviations to black dots (mm) (mean ± SD) | 7.121 ± 4.831 | 1.675 ± 2.714 | <0.001 |
Discussion
This study compared the performance on the intracorporeal suturing task for a newly developed virtual reality laparoscopic surgery simulator and on the traditional FLS box trainer. Participants showed comparable performance time on the VBLaST-SS© and traditional box trainer. No significant difference was found in task completion time. However, there were significant differences in performance quality for remaining incision gap and needle insertion deviation. The differences in remaining incision gap may result from the difference of calculation method and the difficulty to distinguish knot tightness in the virtual environment. Although the ability to tighten the knot in the VBLaST-SS© was mentioned during the practice session, participants tended to continue the task without doing so. The differences in needle insertion deviation may result from limited depth perception and the limited mobility of the Penrose drain target in the virtual environment. Some participants mentioned in their comments that the Penrose drain in the VBLaST-SS© made the task harder to perform because of this characteristic.
Based on feedback from participants, the VBLaST-SS© was considered moderately realistic compared to the traditional box trainer (mean rating 3.84 out of 5). However, the quality of force feedback and the realism of the instrument handling received low ratings. Several participants mentioned in their comments that when handling the needle with needle drivers, the force feedback was helpful but too weak. They also suggested that there should be more force feedback on the Penrose drain. This suggests that the force feedback might need to be improved in future iterations of the system, not only for needle handling but also for Penrose drain. Sankaranarayanan et al. [4] evaluated the face and construct validity of peg transfer, pattern cutting, and endoloop on the same simulator system and found that quality of force feedback had the lowest rating. This supports the importance of force feedback in the design of virtual reality simulator trainers.
The main limitation of this study was a small number of participants who were more experienced laparoscopic surgeons. While this limited the ability to evaluate the system in terms of skill differentiation, the individuals who tested the system are those who are currently training on physical suturing trainers and are the target user group for a virtual reality-based skills trainer. Future work is needed to determine whether the VBLaST-SS© intracorporeal task can be used to distinguish between expert and novice surgeons.
Overall, the VBLaST-SS© system showed face validity and has the potential to support training on suturing skills. Due to the limitation of force feedback, as mentioned by the participants, the current system can supplement rather than replace current training methods. However, with further improvement and validation, the VBLaST-SS© can supplant the FLS box trainer as a cost-effective means of practice. The FLS box trainer and instruments have a one-time cost of $2,770, plus $4.50 for consumables per practice trials. By comparison, the VBLaST-SS© system costs about $6,400 for all parts and the computer system. Therefore, if the VBLaST-SS© system is used for more than 820 practice trials (~10 trainees to proficiency), the simulator becomes more cost effective than the box trainer. In the next stage of the study, the training effectiveness of the VBLaST-SS© will be evaluated to compare participants’ performance after training on different systems, and their ability to transfer their skill between systems and to retain the skill over time. Future work will also consider skill transfer from practice on the VBLaST-SS© to the operating room and the addition of advanced suturing tasks for the simulator.
Figure 2.
Participant performing suturing task on VBLaST-SS©
Acknowledgements
This work was supported by the NIBIB/NIH grant #5R44EB019802.
Footnotes
Disclosures
Yaoyu Fu and Drs. Lora Cavuoto, Di Qi, Karthik Panneerselvam, Gene Yang, Venkata Sreekanth Artikala, Andinet Enquobahrie, Suvranu De have no conflicts of interest or financial ties to disclose. Dr Schwaitzberg has no relevant conflicts related to this manuscript and is a consultant for Activ Surgical, Human Extensions, Arch Therapeutics, Acuitiy Bio and Nu View Surgical.
References
- 1.Maithel S, Sierra R, Korndorffer J, Neumann P, Dawson S, Callery M, Jones D, Scott D (2006) Construct and face validity of MIST-VR, Endotower, and CELTS: Are we ready for skills assessment using simulators? Surg Endosc Other Interv Tech 20:104–112. doi: 10.1007/s00464005-0054-4 [DOI] [PubMed] [Google Scholar]
- 2.Chellali A, Zhang L, Sankaranarayanan G, Arikatla VS, Ahn W, Derevianko A, Schwaitzberg SD, Jones DB, DeMoya M, Cao CGL (2014) Validation of the VBLaST peg transfer task: a first step toward an alternate training standard. Surg Endosc Other Interv Tech 28:2856–2862. doi: 10.1007/s00464-014-3538-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Rodrigues SP, Horeman T, Blomjous MSH, Hiemstra E, van den Dobbelsteen JJ, Jansen FW (2016) Laparoscopic suturing learning curve in an open versus closed box trainer. Surg Endosc Other Interv Tech 30:315–322. doi: 10.1007/s00464-015-4211-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Sankaranarayanan G, Lin H, Arikatla VS, Mulcare M, Zhang L, Derevianko A, Lim R, Fobert D, Cao C, Schwaitzberg SD, Jones DB, De S (2010) Preliminary Face and Construct Validation Study of a Virtual Basic Laparoscopic Skill Trainer. J Laparoendosc Adv Surg Tech 20:153–157. doi: 10.1089/lap.2009.0030 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Munz Y, Kumar BD, Moorthy K, Bann S, Darzi A, Munz Y, Kumar BD, Moorthy K (2004) Laparoscopic virtual reality and box trainers: Is one superior to the other? Surg Endosc Other Interv Tech 18:485–494. doi: 10.1007/s00464-003-9043-7 [DOI] [PubMed] [Google Scholar]
- 6.Fried GM, Feldman LS, Vassiliou MC, Fraser SA, Stanbridge D, Ghitulescu G, Andrew CG (2004) Proving the value of simulation in laparoscopic surgery. Ann Surg 240:518–25–8. doi: 10.1097/01.SLA.0000136941.46529.56 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Fried GM, Fried Steinberg-Bernstein GM (2008) FLS Assessment of Competency Using Simulated Laparoscopic Tasks Description of the FLS Manual Skills Component. J Gastrointest Surg 12:210–212. doi: 10.1007/s11605-007-0355-0 [DOI] [PubMed] [Google Scholar]
- 8.Lim S, Ghosh S, Niklewski P, Roy S (2017) Laparoscopic Suturing as a Barrier to Broader Adoption of Laparoscopic Surgery. JSLS J Soc Laparoendosc Surg 21:1–7. doi: 10.4293/JSLS.2017.00021 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Zhang A, Hünerbein M, Dai Y, Schlag PM, Beller S (2008) Construct validity testing of a laparoscopic surgery simulator (Lap Mentor®): Evaluation of surgical skill with a virtual laparoscopic training simulator. Surg Endosc Other Interv Tech 22:1440–1444. doi: 10.1007/s00464-007-9625-x [DOI] [PubMed] [Google Scholar]
- 10.Iwata N, Fujiwara M, Kodera Y, Tanaka C, Ohashi N, Nakayama G, Koike M, Nakao A (2011) Construct validity of the LapVR virtual-reality surgical simulator. Surg Endosc 25:423–428. doi: 10.1007/s00464-010-1184-x [DOI] [PubMed] [Google Scholar]
- 11.Larsen CR, Soerensen JL, Grantcharov TP, Dalsgaard T, Schouenborg L, Ottosen C, Schroeder T V, Ottesen BS (2009) Effect of virtual reality training on laparoscopic surgery: randomised controlled trial. Bmj 338:b1802–b1802. doi: 10.1136/bmj.b1802 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Grantcharov TP, Kristiansen VB, Bendix J, Bardram L, Rosenberg J, Funch-Jensen P (2004) Randomized clinical trial of virtual reality simulation for laparoscopic skills training. Br J Surg 91:146–150. doi: 10.1002/bjs.4407 [DOI] [PubMed] [Google Scholar]
- 13.Seymour NE, Gallagher AG, Roman S a, O’Brien MK, Bansal VK, Andersen DK, Satava RM (2002) Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg 236:458–63–4. doi: 10.1097/01.SLA.0000028969.51489.B4 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Hamilton EC, Scott DJ, Fleming JB, Rege R V., Laycock R, Bergen PC, Tesfay, Jones DB (2002) Comparison of video trainer and virtual reality training systems on acquisition of laparoscopic skills. Surg Endosc Other Interv Tech 16:406–411. doi: 10.1007/s00464-001-8149-z [DOI] [PubMed] [Google Scholar]
- 15.Diesen DL, Erhunmwunsee L, Bennett KM, Ben-David K, Yurcisin B, Ceppa EP, Omotosho PA, Perez A, Pryor A (2011) Effectiveness of laparoscopic computer simulator versus usage of box trainer for endoscopic surgery training of novices. J Surg Educ 68:282–289. doi: 10.1016/j.jsurg.2011.02.007 [DOI] [PubMed] [Google Scholar]
- 16.McGaghie WC, Issenberg SB, Cohen ER, Barsuk JH, Wayne DB (2011) Does Simulation-Based Medical Education With Deliberate Practice Yield Better Results Than Traditional Clinical Education? A Meta-Analytic Comparative Review of the Evidence. Acad Med 86:706–711. doi: 10.1097/ACM.0b013e318217e119 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Gurusamy K, Aggarwal R, Palanivelu L, Davidson BR (2008) Systematic review of randomized controlled trials on the effectiveness of virtual reality training for laparoscopic surgery. Br J Surg 95:1088–1097. doi: 10.1002/bjs.6344 [DOI] [PubMed] [Google Scholar]
- 18.Qi D, Panneerselvam K, Ahn W, Arikatla V, Enquobahrie A, De S (2017) Virtual interactive suturing for the Fundamentals of Laparoscopic Surgery (FLS). J Biomed Inform 75:48–62. doi: 10.1016/J.JBI.2017.09.010 [DOI] [PMC free article] [PubMed] [Google Scholar]