Abstract
Background
Virtual reality simulators allow trainees to perform repeated practice and provide objective dexterity metrics regarding their performance, which means that virtual reality–based surgical training is becoming a vital part of initial learning of basic laparoscopic surgical skills. However, its educational role in learning advanced procedures remains undetermined. We evaluated the validity of the laparoscopic radical nephrectomy module of the LapVision virtual reality simulator.
Methods
Urologists, medical students, and a junior resident voluntarily participated in the present study, and they performed training with a laparoscopic left radical nephrectomy module. For construct validation, dexterity metrics calculated in the simulator and the mean score of Global Operative Assessment of Laparoscopic Skills evaluated by 2 experts' video review were compared according to the certification of Japanese Endoscopic Surgical Skill Qualification or previous surgical experience.
Results
Ten experts (≥ 50 laparoscopic surgeries), 9 intermediates (11-49), and 14 novices (0-10) voluntarily participated in the present study. Regarding the construct validity, there was a significant difference in the total number of errors, blood loss, and Global Operative Assessment of Laparoscopic Skills score among the groups for both the Endoscopic Surgical Skill Qualification status and previous surgical experience.
Conclusion
The present study demonstrated good construct validity for the LapVision nephrectomy module. Furthermore, global skill assessment was possible by experts' reviews, which indicates the usefulness of the virtual reality procedural module as a skill assessment tool. Virtual reality–based procedural simulation has marked potential to become a vital part of integrated laparoscopic training programs.
Highlights
-
•
Good face, content, and construct validity was observed with the LapVision nephrectomy module.
-
•
Global skill assessment was also possible by experts' reviews, which indicate its usefulness as a skill assessment tool.
-
•
VR-based simulation has marked potential to become a vital part of laparoscopic training.
Introduction
To date, laparoscopic surgery has been used in a variety of surgical disciplines, including urology, general surgery, thoracic surgery, and gynecology, based on accumulated evidence of quicker postoperative recovery, less postoperative pain, and fine anatomical visualization facilitated by high-resolution endoscopy. On the other hand, because of the inherent drawbacks associated with laparoscopic surgeries, such as limited haptic feedback, limited depth perception, and a restricted field of view, novice surgeons are required to learn psychomotor skills and dexterity specific for laparoscopic surgery. In addition, because of working-hour restrictions and ethical considerations, there is a growing need for a surgical training curriculum including simulation training outside the operating theater. Especially in urology, compared with general surgery and gynecology in which laparoscopic procedures are frequently used for benign disease treatments such as laparoscopic cholecystectomy or laparoscopic myomectomy, the total number of laparoscopic procedures involving young urological trainees during their training periods may be smaller than those of the aforementioned disciplines. They might start advanced procedures such as laparoscopic radical nephrectomy or laparoscopic radical nephroureterecomy early in their learning periods.
Virtual reality (VR) simulators allow trainees to perform repeated practice, and provide objective dexterity metrics to assess their performance. These outcomes offer direct feedback to trainees and are useful for educators to evaluate trainees' skill achievements, which mean that VR-based surgical training is becoming a vital part of initial learning of basic laparoscopic surgical skills. However, its educational role in learning advanced procedures remains undetermined. Considering the continuous evolution of computer graphics technology, VR-based surgical training has marked potential to also become a vital part of learning advanced procedures, and data on its educational value should be accumulated. Regarding the VR laparoscopic nephrectomy module, 2 previous studies generated conflicting results for construct validity [1,2]. LapVision is a newly developed laparoscopic simulator, and its magnet technology provides haptic feedback, with very realistic and wireless laparoscopic instruments (https://www.medvisiongroup.com/lapvision.html), which could provide improved realism of surgical simulation. Furthermore, the present VR laparoscopic nephrectomy module represents a well-designed operative scenario, including mobilization of the descending colon and spleen, and division of the renal vasculature and left ureter. Therefore, in the present study, we hypothesized that LapVision simulator allows us to assess participants' skill level in laparoscopic surgery and has the potential to play a vital role in surgical education for advanced laparoscopic procedures. To test this hypothesis, we conducted the present study.
Materials and Methods
The present study was performed after institutional review board approval for simulation training in laparoscopic surgery, which was registered with the University hospital Medical Information Network clinical trial registry (UMIN000030874). A total of 33 participants voluntarily joined the study.
LapVision Smart (MedVision, Japan) was used in the present study (Fig 1). LapVision Smart consists of a Windows computer with a software package developed by MedVision, 2 displays, completely extractable laparoscopic instruments and a laparoscope, their haptic ports, and 2 foot pedals for electrocautery devices. Before the start of simulation, one of the investigators (HM or TA) gave verbal instructions on how to manipulate the simulator. After finishing 4 basic VR modules (tasks involving laparoscopic scissors control, electrocoagulation skills, endoclip applicator control, and movement of objects on pins) twice as a warm-up, participants watched a movie demonstrating laparoscopic transperitoneal left radical nephrectomy prebuilt in the simulator, explaining each step of the nephrectomy scenario, including mobilization of the descending colon after dissection of the line of Toldt and splenocolic ligament, hilar dissection and division of the renal artery and renal vein, division of the left ureter, and dissection of the remaining tissues (Fig 2). Participants then performed training with the nephrectomy scenario. During the simulation training, one of the investigators (HM or TA) played the role of a scopist and gave verbal assistance if participants had trouble with manipulation of the simulator or sequence of the nephrectomy scenario. Especially for medical students, each step of the nephrectomy scenario was verbally guided.
Fig 1.
Components of LapVision Smart.
Fig 2.
Summary of procedural steps of laparoscopic left nephrectomy.
Participants are required to perform (A, B) mobilization of descending colon after dissection of the line of Toldt and splenocolic ligament, (C, D) hilar dissection and division of the renal artery and renal vein, and (E) dissection of the remaining tissues and division of the left ureter.
After the session, completed questionnaires were collected including demographic information, experience of laparoscopic surgeries, simulation training, and videogames. In Japan, the Endoscopic Surgical Skill Qualification (ESSQ) system was developed in 2004, in which 2 double-blinded experts (referees) assessed the complete, unedited movie [3,4]. In urology, laparoscopic nephrectomy, adrenalectomy, or pyeloplasty performed by applicants was assessed based on procedural safety and stable maneuvers. This ESSQ qualification status was also collected. Regarding the reality and representability of the simulator, their impressions were collected, in which a 5-point Likert scale was used by subjects to evaluate the simulator's closeness to reality (5: very realistic, 3: average, 1: very poor) based on factors such as graphics, instrument handling, tissue resistance, and procedural steps of nephrectomy. Data from the experts and intermediates were used for assessment of the face and content validities.
Supplementary Table 1 shows performance metrics calculated for LapVision Smart. For construct validation, we focused on the following parameters according to previous studies regarding VR simulations [[5], [6], [7], [8], [9]] and our view that they may be associated with surgical dexterity: total time, total number of errors, time of coagulation application, blood loss, length of right instrument motion trajectory, length of left instrument motion trajectory, ergonomics of manipulations of right instrument, ergonomics of manipulations of left instrument, average motion speed of right instrument, and average motion speed of left instrument.
Furthermore, the surgical quality was also evaluated using the rating scale of Global Operative Assessment of Laparoscopic Skills (GOALS) [10]. Two blinded experts assessed the unedited movies of whole procedures, and the mean scores of participants' first trial were used for analyses.
Data Analysis
The Mann-Whitney U test or Kruskal-Wallis test was used to compare differences among groups. The interclass coefficient (ICC) was evaluated for interrater reliability. All statistical analyses were performed using JMP Pro12.01 (SAS) or SPSS version 21.
Results
Table 1 summarizes the demographics of the participants. Twenty-one urologists, 11 medical students, and 1 junior resident voluntarily joined the present study. Ten participants had experience of more than 50 laparoscopic surgeries performed as a chief surgeon (categorized as an expert group for construct validation), 9 had that of 11–49 surgeries (categorized as an intermediate group), and 14 had that of 0–10 surgeries (categorized as a novice group). Of the 33 participants, 14 had the ESSQ qualification.
Table 1.
Demographics of the participants.
| N = 33 | |
|---|---|
| Age, y | Median 37 (range, 21–56) |
| Sex | Male/female = 25/8 |
| Background | Urologist, n = 21 |
| Medical student, n = 11 | |
| Junior resident, n = 1 | |
| Experience of laparoscopic surgery | Expert (≥ 50 surgeries), n = 10 |
| Intermediate (11–49), n = 9 | |
| Novice (0–10), n = 14 | |
| Endoscopic Surgical Skill Qualification | Yes/no = 14/ 19 |
| Experience of simulation training | Yes/no = 25/ 8 |
| Experience of video games | Yes/no = 25/ 8 |
Figure 3 summarizes posttraining questionnaire responses from the experts and intermediates regarding face and content validity. In terms of the closeness to reality, most factors were rated as above average, except for tissue resistance (mean score: 2.85), and graphics of the pancreas (mean score: 2.85) and renal artery (mean score 2.85). Regarding the educational role, experts and intermediates reported that the current simulator was useful to aid understanding of the basic procedures in laparoscopic nephrectomy (mean score: 3.65).
Fig 3.
Summary of posttraining questionnaire regarding face and content validity.
In terms of closeness to reality, most of the aspects were rated as above average, except for tissue resistance, and graphics of the pancreas and renal artery.
Of the 33 participants, 26 completed the nephrectomy in the first trial, and 4 completed it in the second trial. The remaining 3 participants could not finish the nephrectomy either in their first or second trial because of computer problems. Therefore, performance outcomes using the simulator were available in 30 participants. Table 2A summarizes the performance outcomes according to the ESSQ qualification status. There were significant differences in the total time (Kruskal-Wallis test, P = .0202), total number of errors (P = .0018), time of coagulation application (P = .0251), blood loss (P = .0002), and ergonomics of manipulations of right instrument (P = .0346) between the 2 groups. Table 2B shows the same analysis divided by previous laparoscopic experience (experts/intermediates/novices). Total number of errors (P =.0427) and blood loss (P =.0021) remained significant on 3-group comparison.
Table 2.
Summary of performance metrics.
| A. Performance metrics according to the ESSQ qualification | ||||
|---|---|---|---|---|
| Outcomes | Total, n = 30 Median (range) | Endoscopic Surgical Skill Qualification, yes: n = 13 Median (range) | Endoscopic Surgical Skill Qualification, no: n = 17 Median (range) | P value |
| Total time (min) | 32.5 (17–61) | 27 (17–43) | 34 (25–61) | .0202 |
| Total number of errors (times) | 41.5 (15–198) | 28 (15–54) | 50 (26–198) | .0018 |
| Time of coagulation application (s) | 14.99 (0–270.92) | 59.62 (1.88–270.92) | 9.56 (0–145.1) | .0251 |
| Blood loss (mL) | 165.145 (19.24–1318.66) | 57.57 (19.24–365.22) | 198.48 (74.48–1318.66) | .0002 |
| Length of right instrument motion trajectory | 5516.1 (491.33–9927.97) | 4694.34 (491.33–9249.59) | 5970.14 (1137.08–9927.97) | .0516 |
| Length of left instrument motion trajectory | 3259.25 (1641.37–8972.2) | 2655.89 (1680.75–5640.93) | 3574.13 (1641.37–8972.2) | .2859 |
| Ergonomics of manipulations of right instrument | 8.585 (5.67–45.73) | 9.24 (5.73–43.51) | 7.99 (5.67–45.73) | .0346 |
| Ergonomics of manipulations of left instrument | 36.485 (7.96–57.51) | 42.83 (15.28–57.51) | 28.64 (7.96–54.45) | .0753 |
| Average motion speed of right instrument | 3.44 (1.82–4.71) | 3.66 (2.61–4.71) | 3.41 (1.82–4.35) | .2859 |
| Average motion speed of left instrument | 1.685 (0.8–3.48) | 1.83 (1.35–2.32) | 1.66 (0.8–3.48) | .3681 |
| B. Performance metrics according to previous surgical experiences | ||||
|---|---|---|---|---|
| Outcomes | Expert, n = 10 Median (range) | Intermediate, n = 8 Median (range) | Novice, n = 12 Median (range) | P value |
| Total time (min) | 27 (17–43) | 31 (21–54) | 34.5 (29–61) | .2075 |
| Total number of errors (times) | 31.5 (15–54) | 46.5 (20–81) | 49.5 (26–198) | .0427 |
| Time of coagulation application (s) | 50.465 (1.88–270.92) | 20.375 (2.53–166.81) | 7.87 (0–89.75) | .0705 |
| Blood loss (mL) | 56.115 (19.24–365.22) | 113.17 (43.21–418.45) | 222.065 (149.27–1318.66) | .0021 |
| Length of right instrument motion trajectory | 4696.75 (491.33–9249.59) | 5457.285 (2739.07–9113.61) | 6111.66 (1137.08–9927.97) | .3963 |
| Length of left instrument motion trajectory | 2722.67 (1809.66–5640.93) | 2792.455 (1680.75–5974.53) | 3628.995 (1641.37–8972.2) | .6526 |
| Ergonomics of manipulations of right instrument | 9.085 (5.73–43.51) | 8.805 (5.67–18.69) | 7.79 (6.72–45.73) | .2052 |
| Ergonomics of manipulations of left instrument | 43.94 (15.28–57.51) | 45.085 (26.89–54.71) | 28.495 (7.96–50.91) | .0663 |
| Average motion speed of right instrument | 3.725 (2.87–4.71) | 3.255 (2.61–4.14) | 3.415 (1.82–4.35) | .2622 |
| Average motion speed of left instrument | 1.875 (1.61–2.32) | 1.515 (1.35–2.04) | 1.77 (0.8–3.48) | .1928 |
Regarding the experts' video review, 43 procedures were successfully video-recorded (7 participants performed the module multiple times), and the first trial was available in 32 participants (1 movie was lost because of backup error). For interrater reliability, the score assigned by each rater had an ICC of 0.688 (Fig 4). Figure 5 summarizes the distribution of mean scores of GOALS divided by the ESSQ qualification status or previous experience of laparoscopic surgery. Significant differences were observed in GOALS scores in both comparisons (P < .0001).
Fig 4.
Correlation of the GOALS sores between raters 1 and 2.
For interrater reliability, the score assigned by each rater had an ICC of 0.688.
Fig 5.
Distribution of mean score of GOALS, divided by the ESSQ qualification status or previous experiences of laparoscopic surgery.
Significant difference was observed in GOALS scores in both comparisons (P < .0001).
Discussion
There are several surgical simulators used for surgical education such as box trainers, animal laboratories, and VR simulators. Despite the expensive initial cost, VR simulators have marked advantages in that trainees can easily repeat simulation training with a low running cost, and they receive several forms of feedback including time to complete tasks, path length of both hands, and blood loss when they perform procedural surgical simulations. For example, Seymour et al demonstrated in their randomized, double-blinded study that a VR-training (MIST VR simulator diathermy task) group showed faster and better operative performance with less errors in laparoscopic cholecystectomy compared with a non–VR-trained group. Recently, VR procedure-specific modules also became available, such as for laparoscopic nephrectomy [1,2], laparoscopic colectomy [9], and laparoscopic cholecystectomy [8]. However, evidences to support their educational role or ability to facilitate skill assessment remain limited. Before educators integrate VR simulators into proficiency-based training curriculums, validation studies are necessary to assess, for example, the reality of computer graphics, representability of surgical steps and haptic feedback, and construct validity, which means the ability of the VR simulator to differentiate experienced from inexperienced surgeons.
In the present study, we evaluated so-called face, content, and construct validity of the laparoscopic radical nephrectomy module of the LapVision virtual reality simulator. Based on a previous study showing a shorter operative time over 50 cases [11], an expert opinion [12], and our similar opinion, we used a cutoff point of 50 cases to define the “expert” category in the present study. Regarding the face and content validity, acceptable reality was accomplished in most aspects based on evaluation by the experts and intermediates. Regarding the construct validity, significant differences were observed in the total time (P =.0202), total number of errors (P =.0018), time of coagulation application (P =.0251), blood loss (P =.0002), and ergonomics of manipulations of right instrument (P =.0346) according to the ESSQ qualification status, whereas the total number of errors (P =.0427) and blood loss (P =.0021) were significant on 3-group comparison based on previous laparoscopic experience (experts/intermediates/novices). We consider that it was a reasonable finding that the analysis based on the ESSQ qualification status, representing acknowledgement of a sufficient skill level in laparoscopic surgery, provided more dexterity metrics associated with expertise in laparoscopic surgery compared with that simply based on the number of previous surgical experiences. Because participants were required to perform almost the whole procedure of laparoscopic radical nephrectomy, including mobilization of the descending colon, vascular ligation and division, ureter division, and dissection around the kidney, the excellent representability of the surgical scenario may be associated with good construct validity. When restricted to the 26 participants who completed the nephrectomy in the first trial (excluding 4 participants who finished the nephrectomy in the second trial), the total number of errors and blood loss remained significant on both comparisons (data not shown).
To further confirm the construct validity and identify how to effectively use the VR simulator as a part of integrated surgical training programs, 2 blinded experts assessed the videos according to GOALS. First, regarding the interrater variability, ICC was 0.688, which meant good interrater reliability. The present result was higher than we had expected, and our observation demonstrated that, although it was a virtual surgery, experts could assess performers' surgical expertise according to the global surgical rating scale. In addition, a significant difference was also observed in the mean scores of GOALS on both comparisons of the ESSQ qualification and previous surgical experience, with P values of < .0001. We consider that both dexterity metrics derived from the built-in algorithm and global skill assessments by experts' review (evaluators' perception of surgical performance) help educators to grasp trainees' skill level outside an operative theater, which could minimize patient risk when they perform actual surgery. Because VR simulation offers realistic anatomy, the VR procedural module could provide a level environment for skill assessment and help educators comprehend learning curves during training periods.
We recognize that our study is limited by the small sample size as well as the lack of sample size calculation. We do not have data regarding the learning curves on repeating the present VR module or data on skill retention after training. We could not draw a definitive conclusion on whether VR-based procedural simulation training offers a better way of transferring surgical skills to actual clinical practice. We need to extend the study to confirm learning curves of continual VR-based procedural training and subsequent skill improvement in daily practice (predictive validity). In terms of LapVision Smart, improvement of the computer program is still necessary because, due to computer problems, 3 participants did not finish the nephrectomy in either the first or second trial, which resulted in missing data on 3 participants for dexterity parameter analyses. Nevertheless, we believe that simulation-based training could help trainees overcome the first part of the learning curve, and our observation supports the positive role of VR procedural simulation, especially for skill assessment.
Conclusion
The present study demonstrated good face, content, and construct validity for the LapVision nephrectomy module. Furthermore, global skill assessment was possible by experts' reviews, which indicates the usefulness of the VR procedural module as a skill assessment tool. VR-based procedural simulation has marked potential to become a vital part of integrated laparoscopic training programs.
The following are the supplementary data related to this article.
Performance metrics calculated with LapVision Smart.
Author Contribution
Study conception and design: Haruka Miyata, Takashige Abe. Acquisition of data: Haruka Miyata, Takashige Abe, Madoka Higuchi, Takahiro Osawa, Ryuji Matsumoto, Hiroshi Kikuchi. Analysis and interpretation of data: Haruka Miyata, Takashige Abe, Kiyohiko Hotta, Sachiyo Murai. Drafting of manuscript: Haruka Miyata, Takashige Abe. Critical revision: Yo Kurashima, Nobuo Shinohara.
Conflict of Interest
Haruka Miyata, Takashige Abe, Kiyohiko Hotta, Madoka Higuchi, Takahiro Osawa, Ryuji Matsumoto, Hiroshi Kikuchi, Sachiyo Murai, and Nobuo Shinohara have nothing to disclose. Yo Kurashima is a consultant at Ethicon.
Funding Sources
This work was supported by a Grant-in-Aid for Scientific Research (C) (KAKENHI no. 17K08897) from the Japan Society for the Promotion of Science.
Acknowledgement
The authors would like to thank Kazuyoshi Watanabe for his support of this study.
References
- 1.Brewin J, Nedas T, Challacombe B, et al. Face, content and construct validation of the first virtual reality laparoscopic nephrectomy simulator. BJU Int 2010;106(6):850–4. PubMed PMID: 20128776. [DOI] [PubMed]
- 2.Wijn RP, Persoon MC, Schout BM, et al. Virtual reality laparoscopic nephrectomy simulator is lacking in construct validity. J Endourol 2010;24(1):117–22. PubMed PMID: 19954352. [DOI] [PubMed]
- 3.Matsuda T, Ono Y, Terachi T, et al. The endoscopic surgical skill qualification system in urological laparoscopy: a novel system in Japan. J Urol. 2006;176(5):2168–72; discussion 72. PubMed PMID: 17070285. [DOI] [PubMed]
- 4.Matsuda T, Kanayama H, Ono Y, et al. Reliability of laparoscopic skills assessment on video: 8-year results of the endoscopic surgical skill qualification system in Japan. J Endourol 2014;28(11):1374–8. PubMed PMID: 24819163. [DOI] [PubMed]
- 5.Nayar SK, Musto L, Fernandes R, Bharathan R. Validation of a virtual reality laparoscopic appendicectomy simulator: a novel process using cognitive task analysis. Ir J Med Sci 2018. PubMed PMID: 30456516. [DOI] [PubMed]
- 6.Harrison P, Raison N, Abe T, et al. The validation of a novel robot-assisted radical prostatectomy virtual reality module. J Surg Educ. 2018;75(3):758–66. PubMed PMID: 28974429. [DOI] [PubMed]
- 7.Jensen K, Bjerrum F, Hansen HJ, et al. Using virtual reality simulation to assess competence in video-assisted thoracoscopic surgery (VATS) lobectomy. Surg Endosc 2017;31(6):2520–8. PubMed PMID: 27655381. [DOI] [PubMed]
- 8.Aggarwal R, Crochet P, Dias A, et al. Development of a virtual reality training curriculum for laparoscopic cholecystectomy. Br J Surg 2009;96(9):1086–93. PubMed PMID: 19672934. [DOI] [PubMed]
- 9.Wynn G, Lykoudis P, Berlingieri P. Development and implementation of a virtual reality laparoscopic colorectal training curriculum. Am J Surg 2018;216(3):610–7. PubMed PMID: 29268942. [DOI] [PubMed]
- 10.Vassiliou MC, Feldman LS, Andrew CG, et al. A global assessment tool for evaluation of intraoperative laparoscopic skills. Am J Surg 2005;190(1):107–13. PubMed PMID: 15972181. [DOI] [PubMed]
- 11.Gill IS, Meraney AM, Schweizer DK, et al. Laparoscopic radical nephrectomy in 100 patients: a single center experience from the United States. Cancer. 2001;92(7):1843–55. PubMed PMID: 11745257. [DOI] [PubMed]
- 12.Vallancien G, Cathelineau X, Baumert H, et al. Complications of transperitoneal laparoscopic surgery in urology: review of 1,311 procedures at a single center. J Urol 2002;168(1):23–6. PubMed PMID: 12050484. [PubMed]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Performance metrics calculated with LapVision Smart.