Face and content validation of a Virtual Translumenal Endoscopic Surgery Trainer (VTEST™)

Abstract

Background

Natural Orifice Translumenal Endoscopic Surgery (NOTES) is an emerging surgical paradigm, where peritoneal access is achieved through one of the natural orifices of the body. It is being reported as a safe and feasible surgical technique with significantly reduced external scarring. Virtual Translumenal Endoscopic Surgical Trainer (VTEST™) is the first virtual reality simulator for the NOTES. The VTEST™ simulator was developed to train surgeons in the hybrid transvaginal NOTES cholecystectomy procedure. The initial version of the VTEST™ simulator underwent face validation at the 2013 Natural Orifice Surgery Consortium for Assessment and Research (NOSCAR) summit. Several areas of improvement were identified as a result, and the corresponding modifications were implemented in the simulator. This manuscript outlines the results of the subsequent evaluation study, performed in order to assess the face and content validity of the latest VTEST™ simulator.

Methods

Twelve subjects participated in an Institutional Review Board (IRB) approved study that took place at the 2014 NOSCAR summit. Six of the twelve subjects, that are experts with NOTES experience, were used for face and content validation. The subjects performed the hybrid transvaginal NOTES cholecystectomy procedure on VTEST™ that included identification of the Calot’s triangle, clipping and cutting the cystic duct/artery, and detaching the gallbladder. The subjects then answered 5-point Likert scale feedback questionnaires for face and content validity.

Results

Overall, subjects rated 12/15 questions as 3.0 or greater (60%), for face validity questions regarding the realism of the anatomical features, interface and the tasks. Subjects also highly rated the usefulness of the simulator in learning the fundamental NOTES technical skills (3.50 ± 0.84). Content validity results indicate a high level of usefulness of the VTEST™ for training prior to operating room experience (4.17 ± 0.75).

Introduction

Natural Orifice Translumenal Endoscopic Surgery (NOTES) is a novel surgical paradigm, commonly considered a natural successor of laparoscopic surgery [1]. During a typical NOTES based procedure, internal organs are accessed by perforating the viscera (stomach, colon or vagina) using a flexible endoscope inserted through natural orifices such as the mouth, anus or vagina, without making any incisions on the surface of the body [2]. The initial NOTES concepts were demonstrated in 2004, when the feasibility and safety of an oral transgastric endoscopic approach to the peritoneal cavity were explored by Kaloo et al. [3]. Similar “scarless” procedures have been associated with the enhanced prospects of decreased wound infections and incisional hernia, operative stress, postoperative immobility, and pain, while also leading to better cosmetic results [4]. In 2005, the Natural Orifice Surgery Consortium for Assessment and Research (NOSCAR) was formed in order to realize the full potential of NOTES. Furthermore, members from the Society of American Gastrointestinal and Endoscopic Surgeons (SAGES) and the American Society for Gastrointestinal Endoscopy (ASGE) have identified eight fundamental problems that must be resolved in order to make NOTES a commonly accepted and a widely utilized procedure [1]. One of the identified problems is the urgent need for effective training platforms capable of reducing poor outcomes of the early NOTES trials.

Hybrid NOTES approaches rely on laparoscopic assistance via a multidisciplinary team and can utilize either flexible or rigid scopes, whereas pure NOTES approaches only use flexible scopes without laparoscopic assistance [5]. Consequently, successful NOTES trainings require expertise in both the traditional laparoscopic techniques and interventional gastroenterology and thus complicates the efficacy of NOTES training. Currently, only physical NOTES trainers exists but require constant replenishment of the materials (ex-vivo model-based simulators) and often have complex institutional approval procedures (training with porcine models and cadavers) [6], [7]. Albeit these simulators evaluate performance and surgical expertise, they often rely on subjective metrics such as expert proctor observations. Virtual reality (VR) trainers, however, present highly objective metrics for performance evaluations and eliminate the need for material replenishments. These benefits have enabled the commercialization of VR simulators and are utilized in training labs specializing in traditional laparoscopic and endoscopic procedures [8]–[10]. Since the inception of laparoscopic and endoscopic techniques, significant strides in surgical training research has allowed the surgical community to embrace trainers to ultimately create successful programs such as the Fundamental of Laparoscopic Surgery (FLS) and the Fundamentals of Endoscopic surgery (FES), which are used for boar certification [11], [12]. Similarly, we aim to advance surgical training research for NOTES by presenting the first VR NOTES trainer.

Recognizing the potential of such a simulator, the National Institutes of Health (NIH) have funded a multi-year research project, led by researchers from the Rensselaer Polytechnic Institute (RPI) closely working with the medical experts at Beth Israel Deaconess Medical Center, and Cambridge Health Alliance. Prior to the technical development of a VR NOTES simulator, a thorough needs analysis at the 2011 NOSCAR meeting was conducted to determine critical design parameters [13]. The questionnaire results, along with human and animal studies in literature, indicate that cholecystectomy is the best procedure to be simulated for the VR NOTES trainer [13]–[16]. Transvaginal access was selected, instead of the transgastric and the transrectal approaches, due to the potential risk of intestinal leakages and fecal bacteria contamination [17], [18]. Furthermore, a strong preference for the hybrid NOTES approach over pure NOTES was evident [13]. Ultimately, study results showed that the surgical community deemed that a virtual transvaginal hybrid NOTES cholecystectomy simulator is the most appropriate for development. [13], [19].

As one of the first steps in the development of the simulator, a thorough task analysis was completed to determine key tasks of the NOTES procedures to be modeled within the VR-NOTES simulator. For this purpose, 19 rigid scope transvaginal and 11 laparoscopic cholecystectomies were recorded and analyzed for time and errors. A hierarchical task analysis (HTA) and a time series analysis was performed on the data [20]. Tasks that require the most amount of time and were essential for any cholecystectomy procedure for both NOTES and laparoscopic surgery were identified and considered for implementation into the VR NOTES simulator. These tasks include the removal of fibrous tissue, exposure of the Calot’s Triangle, establishment of the critical view of safety, and the electrosurgical tasks used to separate the gallbladder from the liver. This information was used as the basis for creation of the Virtual Translumenal Endoscopic Surgery Trainer (VTEST™) - the first virtual reality simulator for the NOTES cholecystectomy procedure performed using a hybrid procedure using a rigid endoscope [21].

A preliminary evaluation study was performed to ensure the realism and the usefulness of the initial version of the VTEST™ simulator [22]. For that purpose, ten subjects with varying NOTES and laparoscopic surgery experience were recruited to participate in the study at the 2013 NOSCAR summit. The subjects were asked to perform the procedural NOTES cholecystectomy surgery (identifying the Calot’s triangle, clipping and cutting the cystic artery and the duct, detaching the gallbladder), and then to answer a feedback questionnaire [22]. Several areas of improvement were identified as a result, and the corresponding modifications were implemented in the simulator. This manuscript outlines the results of the follow-up evaluation study performed at the 2014 NOSCAR meeting in order to assess the face and content validity of the latest VTEST™ version.

Methods

The functionality of the VTEST™ simulator is provided by two primary modules - the hardware interface and the custom simulation software. A brief description is provided in this section for each these modules, followed by the face and content validation experiment design description.

Hardware interface

An effective and realistic multi-modal interface is an essential part of a successful surgical simulator. VTEST™ utilizes a novel physical interface, consisting of an insufflated torso model with female external reproductive organs to provide the user with an immersive training experience of the procedure (Fig.1). Interaction with the simulator is performed via a rigid scope, inserted through the vagina, and one of the available laparoscopic tools, inserted through the umbilical port.

Figure 1.

VTEST™ interface

Four laparoscopic surgical instruments can be used in the tasks of exposing the Calot’s triangle and detaching the gallbladder: L-hook electrosurgery tool, dissection forceps, clip applier, and shears. The modular design of the VTEST™ interface allows the user to quickly change tool handles during the simulation, as shown in Figure 2. Rotary potentiometers, in combination with an analog-to-digital converter (USB-6008, National Instruments®), are used to interpret the angular motion of the tool handles as an analog voltage signal and to provide this data to the simulator in digital format.

Figure 2.

VTEST™ laparoscopic tools

The interface provides users with 3-degree of freedom (DOF) translational force feedback by mounting the tip of the tool shaft to the end-effector of a haptic device (Geomagic^® Touch™). The haptic device measures the 6-DOF motion of the tool tip and sends the positional information to simulation software. The connection between the tool and the haptic device is also used to directly read the tool articulation data from the roll angles of the haptic device. Accurate tracking of the position and orientation of the camera at the tip of a rigid scope is achieved with a miniature 6-DOF motion tracking sensor (3D Guidance Trakstar, Ascension Technology Co.). The tip motion inside the torso corresponds to the simulated camera navigation within the virtual peritoneal cavity. Typically, the hybrid NOTES procedure is performed by several individuals. However, it was deemed necessary to ensure that the users would have the ability to practice the required camera navigation and surgical tool manipulation without external assistance. Therefore, an additional input device consisting of three foot pedals (Savant Elite, Kinesis Corporation) is used for controlling the rigid scope functionality, changing the virtual laparoscopic tools within the simulated surgical environment and operating the electrosurgical tools. One of the key features of the hybrid NOTES cholecystectomy procedure implemented in the VTEST™ simulator is the manipulation of the gallbladder via transabdominal sutures rather than laparoscopic graspers. Therefore, the simulation environment is augmented with the virtual version of these sutures, and the users are provided with the ability to control the suture tension via dedicated keyboard shortcuts, effectively repositioning the gallbladder model within the operating space.

Simulation software

VTEST™ simulator software functionality is provided by three main modules: haptics, simulation, and graphics [21]. These modules are built on top of a general purpose virtual reality framework – Software Framework for Multimodal Interactive Simulations (SoFMIS) [23]. Realistic visual representation of the peritoneal cavity and its organs is achieved using advanced computer graphics techniques such as, multi-pass rendering with off-screen buffers, OpenGL vertex and fragment shaders, and bump and normal mapping. The resulting simulated environment exhibits photorealistic depiction of the internal organs, tool shadows, depth-of-field effect, imitation of the particulate suspension in the peritoneal cavity, organ bleeding, as well the smoke and the sparking effects during the simulated electrosurgery (Fig. 3).

Figure 3.

Simulated peritoneal cavity in VTEST™

Realistic interaction with the surgical environment is provided by the innovative simulation software, responsible for numerical computation of physical behavior of the simulated organs, and enabling collision detection and response between these organs and the surgical tools. Furthermore, we effectively model complex tissue mechanics such as tissue handling and tearing. Physical behavior of the internal organs, including the gallbladder, the liver, and the peritoneum, is modeled using the mass-spring method. Each organ is characterized by two distinct mesh structures: a fine surface mesh for realistic rendering of the graphical surface details and a coarse volumetric mesh for efficient physics computation. A spring and a damper are assigned to each edge in the volumetric tetrahedral mesh structure, which is then linked to the corresponding vertices of the surface mesh. During physical interaction with the volumetric mesh the position of each surface vertex is calculated by barycentric interpolation of the four vertices of the linked tetrahedral element. Collision detection and response algorithms prevent unrealistic penetration and overlapping during the interaction between the tools and the models of internal organs. Fast collision detection between the laparoscopic tools and the tubular organs (e.g., cystic duct and cystic artery) is accomplished via integration of their respective geometric skeletons into the simulation. The simulator is also capable of recording performance task metrics such as completion time, errors, force and economy of motion.

Experimental design

Twelve subjects participated in this Institutional Review Board (IRB) approved study. Six of the twelve subjects are experts with NOTES experience that have been used for analysis in this study. The study took place at the 2014 Natural Orifice Surgery Consortium for Assessment and Research (NOSCAR) summit, held at the ASGE Institute for Training and Technology Center, Downers Grove, IL. Before the start of the experiment, subjects were asked to fill out a questionnaire focusing on their demographics and any previous NOTES and/or traditional laparoscopic surgery experience. An instructional video, outlining the procedure and the details of the VTEST™ functionality and its interface operation, was shown to the study participants, followed by the actual simulator experiment.

The subjects were asked to perform the hybrid transvaginal NOTES cholecystectomy procedure using the VTEST™ simulator. The simulator currently offers users the ability to perform the following steps: (1) navigation of the peritoneal cavity in order to reach the operation region with the suture-stabilized gallbladder; (2) identification, clipping, and cutting of the cystic artery and the cystic duct; (3) detachment of the gallbladder from the liver via electrosurgery. The subjects were then asked to answer a 5-point Likert scale feedback questionnaire consisting of fifteen questions and addressing a variety of topics, including realism of the simulated anatomy, overall realism of the gall bladder removal task, usefulness of the force feedback in performing the tasks, trustworthiness of the simulator in quantifying accurate measures of performance, etc. (Table 1). The Likert scale for the face validation questionnaire ranged from: 1- very low, 2- low, 3 – neutral, 4 – high, and 5 – very high. The participants subsequently completed an additional questionnaire, targeted specifically at the content validation of the simulator by the hybrid rigid NOTES experts (Table 2). The Likert scale for the content validation questionnaire ranged from: 1- strongly disagree, 2- disagree, 3 – neutral, 4 – agree, and 5 – strongly agree.

Table 1.

Face validation questionnaire - Subjective Preference of the NOTES Simulator.

Face validation questionnaire
1.	Rate the degree of realism of the anatomy (how realistic it looks) in the simulator
2.	Rate the degree of realism of the anatomy (model and textures) in the simulator for correct identification of the Calot’s triangle
3.	Rate the degree of realism of the appearance of the simulator interface (patient, video display, instrument models, and instrument entry ports)
4.	Rate the degree of realism of the rigid endoscope navigation (camera control)
5.	Rate the degree of realism of the instrument handling (how realistic it feels) on the simulator
6.	Rate the degree of realism of the degree of overall realism of the blunt dissection task
7.	Rate the degree of realism of the degree of overall realism of the gall bladder removal task
8.	Rate the degree of overall realism of the NOTES simulation (how it looks AND feels), compared to the corresponding laparoscopic task.
9.	Rate the quality of the force feedback (sensation of feeling the tools on the target and in the task space) in the NOTES simulator.
10.	Rate the degree of usefulness of the force feedback (sensation of feeling the tools on the target and in the task space) in the NOTES simulator in helping your performance.
11.	Rate the usefulness of the NOTES simulation in learning hand-eye coordination skills, compared to a pig model.
12.	Rate the usefulness of the NOTES simulation in learning ambidexterity skills, compared to a pig model
13.	Rate the degree of overall usefulness of the NOTES simulator in learning the fundamental NOTES technical skills compared to a pig model.
14.	Rate your assessment of how trustworthy the NOTES simulator is to quantify accurate measures of performance.
15.	Rate your assessment of how trustworthy the NOTES simulator is in providing different hand-eye coordination compared to traditional two-port laparoscopic approach.

Table 2.

Feedback questionnaire - Usefulness of the NOTES Simulator (VTEST™).

Feedback questionnaire
1.	Is the VTEST a realistic simulator for the NOTES procedure?
2.	Could the VTEST simulator be a useful trainer for residents/surgeons before their operating room experience?
3.	Could the VTEST simulator be a useful trainer for certification in this field?

Statistical Tests

SPSS 18.0 (IBM Inc.) statistical software was used for all statistical analyses. For both face and content validation, descriptive statistics were obtained where the mean Likert scores of the expert group are reported along with standard deviation for error bars.

Results

The results of the face validation study are provided in the Table 3 and Figure 4. Results showed that subjects rated 12 of the 15 questions above 3.0 or greater (60%), including the realism of the anatomical features, interface and the surgical tasks. The highest score (3.67 ± 0.52) was assigned to the trustworthiness of providing different hand-eye coordination compared to a traditional laparoscopy approach. The lowest score (2.40 ± 0.89) was associated with the realism of the blunt dissection task. Subjects also highly rated the usefulness of the simulator in learning fundamental NOTES technical skills compared to a pig model (3.50 ± 0.84).

Table 3.

Results of the face validation questionnaire

#	Question	Expert
		Mean	SD
1	Anatomy	3.33	0.52
2	Calot's triangle	3.50	0.55
3	Simulator interface	3.50	0.55
4	Rigid endoscope navigation	3.50	1.05
5	Instrument handling	3.00	1.10
6	Blunt dissection task	2.40	0.89
7	Gall bladder removal task	3.33	0.52
8	Overall realism	3.00	0.63
9	Quality of force feedback	2.50	0.55
10	Usefulness of force feedback	2.50	1.05
11	Usefulness in learning hand-eye coordination compared to a pig model	3.50	0.84
12	Usefulness in learning ambidexterity skills compared to a pig model	3.33	0.82
13	Usefulness in learning the fundamental NOTES technical skills compared to a pig model	3.50	0.84
14	Trustworthy to quantify accurate measures of performance	3.00	0.89
15	Trustworthy in providing different hand-eye coordination compared to traditional two-port laparoscopic approach	3.67	0.52

Figure 4.

Simulated face validation results (mean Likert scores with standard deviations)

The results of the content validation study are provided in the Table 4 and Figure 5. The highest mean score (4.17 ± 0.75) was associated with the potential of the VTEST™ simulator to be a useful trainer for residents/surgeons before their operating room experience, indicating a high level of importance associated with the virtual reality NOTES trainers by the experts in this discipline.

Table 4.

Results of the content validation questionnaire

#	Question	Expert
		Mean	SD
1	Realistic trainer for NOTES procedure	3.00	0.63
2	Useful trainer for residents/surgeons before their operating room experience	4.17	0.75
3	Useful trainer for certification in this field	2.80	0.84

Figure 5.

Content validation results (mean Likert scores with standard deviations)

Discussion

Natural Orifice Translumenal Endoscopic Surgery (NOTES) is an emerging paradigm that is commonly credited with being a safe and feasible surgical technique, resulting in significantly reduced external scarring in addition to the prospects of decreased operative stress, postoperative immobility, and pain. It is also recognized, however, that a comprehensive NOTES training frequently necessitates establishing expertise in both the traditional laparoscopic techniques and interventional gastroenterology [5]. The contemporary NOTES trainers require constant replenishment of the materials (ex-vivo model-based simulators) and often have complex institutional approval procedures [6], [7]. These trainers also offer limited or no options for varying the parameters associated with the corresponding surgical procedure (e.g., different patient physiologies, pathologies, etc.). Moreover, there are currently no well-established and widely-recognized credential systems for NOTES, as opposed to the traditional laparoscopic surgery (Fundamental of Laparoscopic Surgery, FLS) [11], and the endoscopic procedures (Fundamentals of Endoscopic Surgery, FES) [12].

Virtual reality trainers remedy many of these problems, by eliminating the reliance on animal cadavers or other costly training material during the training phase, offering instead the option of training in special pathological cases, and improving the long-term cost effectiveness [24]. Such simulators are frequently encountered in the training labs specializing in traditional laparoscopic and endoscopic procedures[8], [10]. Virtual Translumenal Endoscopic Surgical Trainer (VTEST™) is the first virtual reality simulator for the Natural Orifice Translumenal Endoscopic Surgery (NOTES), and is envisioned to offer the same level of convenience and utility as the currently available commercial simulators. Virtual simulators must undergo validation, before they are considered for user performance evaluation. As a part of the validation process, we present the result of face and content validation of the VTEST™ simulator in this manuscript.

The face validity of the latest VTEST™ version was shown on many aspects of the simulation, where all subjects rated 12 of 15 questions with at least 60% of the maximum Likert score. Other validation studies have shown that this is an indication of successful face validation for surgical simulators [25], [26]. Subjects also rated highly the usefulness of the simulator in learning the fundamental NOTES technical skills and improving the hand-eye coordination. Furthermore, the results of the content validity study (exclusive to the hybrid NOTES experts’ feedback) indicate a high level of importance associated with the virtual reality NOTES trainers by the experts in this discipline. While the realism of the simulator can and will be improved, the underlying message is that there is a great need for a NOTES simulator that will help train physicians before gaining experience in the operating room. Although there is a small sample size for face and content validation, due to the short number of experts in the NOTES field [20], this study gives us tremendous insight into the need for NOTES simulators and the important features of the VTEST™ platform. Further refinements based on the face validation feedback are currently taking place, including novel paradigms for the blunt dissection functionality and the quality of the force feedback. Upon completion of the VTEST™ simulator, we plan to perform concurrent and predictive validation studies to ensure that skills learned in the virtual environment transfer to the operating room.