Graphic and Haptic Simulation for Transvaginal Cholecystectomy Training in NOTES

Abstract

Background

Natural Orifice Transluminal Endoscopic Surgery (NOTES) provides an emerging surgical technique which usually needs a long learning curve for surgeons. Virtual reality (VR) medical simulators with vision and haptic feedback can usually offer an efficient and cost-effective alternative without risk to the traditional training approaches. Under this motivation, we developed the first virtual reality simulator for transvaginal cholecystectomy in NOTES (VTEST™).

Methods

This VR-based surgical simulator aims to simulate the hybrid NOTES of cholecystectomy. We use a 6DOF haptic device and a tracking sensor to construct the core hardware component of simulator. For software, an innovative approach based on the inner-spheres is presented to deform the organs in real time. To handle the frequent collision between soft tissue and surgical instruments, an adaptive collision detection method based on GPU is designed and implemented. To give a realistic visual performance of gallbladder fat tissue removal by cautery hook, a multi-layer hexahedral model is presented to simulate the electric dissection of fat tissue.

Results

From the experimental results, trainees can operate in real time with high degree of stability and fidelity. A preliminary study was also performed to evaluate the realism and the usefulness of this hybrid NOTES simulator.

Conclusions

This prototyped simulation system has been verified by surgeons through a pilot study. Some items of its visual performance and the utility were rated fairly high by the participants during testing. It exhibits the potential to improve the surgical skills of trainee and effectively shorten their learning curve.

Graphical abstract

Introduction

Natural Orifice Transluminal Endoscopic Surgery (NOTES) is an emerging technique that allows surgeons to utilize the patient’s native orifices, such as anal, oral, and vaginal openings to gain access to the peritoneal cavity. Once inside the peritoneal cavity, various tools can be placed through trocars at the natural orifice sites without a visible skin incision, to perform specific endoscopic surgeries. Besides the minimization of surface incisions and cosmetic reasons, the NOTES technique has numerous advantages, which include shorter hospital stays compared to traditional laparoscopic procedures due to the reduced need for anesthesia and the use of lower CO2 pressure for establishing pneumoperitneoum [1, 2]. Therefore, NOTES technique reduces complications, such as hernias, adhesions and wound infections. It can also reduces patients' post-operative pain and recovery time [3]. Currently, there are two types of NOTES techniques in clinic. One is the pure NOTES procedure, which uses only a flexible scope without laparoscopic assistance. The other is the hybrid NOTES procedure with laparoscopic assistance using either flexible or rigid scopes. At present, the most NOTES procedure used in clinic is the hybrid. Performing the hybrid NOTES requires a multidisciplinary team with expertise in both endoscopy and laparoscopy. Though there are a number of advantages for NOTES, it suffers from skill difficulties associated with laparoscopic surgeries. These difficulties include poor operative field visibility, scope and tool maneuverability, and increased grasping distances [1, 4]. These challenges become the motivation to develop medical simulators to train surgeons on NOTES procedures.

Basically, training in NOTES procedures can be done through cadavers, animal tissues and ex-vivo model based simulators like the EASIE-R [5]. Cadaveric courses are available but are expensive and, by their nature, do not allow repetitive tasks to be performed. While some animal tissues, such as porcine, can be used for training, they are different in anatomy with humans and require ethical approval. For the case of ex-vivo models, they also need to be constantly replenished. On the other hand, in recent years, virtual reality (VR) medical simulators have become a viable alternative approach, which allows the trainee to hone their skills by operating on virtual patients in a safe and repetitive manner. In the cases of training for laparoscopic, endoscopic and other procedures, several commercial medical simulators have already been successfully applied, such as MIST-VR [6], Simbionix Lap-Mentor [7] and GI-Mentor [8] and so on.

Due to the advantages of VR medical simulator, researchers have made a great deal of efforts in VR-based surgery simulation and related graphic techniques, such as deformation and soft tissue cutting simulation. Pan et al. [9, 10] developed a medical simulator for the virtual laparoscopic rectum surgery. They use triangle meshes and multi-layer mass spring model to represent the intestine and surrounding membrane tissue. To perform real-time graphic displaying, a cosserat rod model had been introduced to handle the deformation of intestine and collision detection. Kim et al. [11] developed a medical simulator for virtual laparoscopic cholecystectomy training. They proposed a new deformable mesh carving technique, which employ the volume-based potential fields and hexahedral finite element approaches. Choi et al. [12] designed a VR training system for laparoscopic oesophagus surgery. They used boundary element method (BEM) to compute the deformation of soft tissue. The mechanical parameters were collected from the experiments of animal tissue and integrated with haptic device. Mitchell et al. [13] presented a medical simulation system for authoring surgical procedures of soft tissue manipulation using physics-based method. This tool can be used by clinical educators to illustrate the intricate maneuvers involved in craniofacial repairs. Pan et al. [14] describes an interactive dissection approach for soft tissue models governed by extended position-based dynamics. It can afford real-time and robust cutting without sacrificing realistic visual performance. Paulus et al. [15] proposed a new re-meshing algorithm which is generic enough to support a variety of graphic applications. It can achieve realistic multiple cutting on a soft object.

In medical field, some researchers also made preliminary work in the development of VR-based NOTES simulation system. Sankaranarayanan et al. [16] analyzed the needs and identified the design parameters for the VR-based simulator development of NOTES. A 30-point questionnaire was distributed at the 2011 National Orifice Surgery Consortium for Assessment and Research (NOSCAR) meeting to obtain the responses from 22 experts. After statistical analysis, their study reinforces the importance of developing a virtual NOTES simulator and clearly presents expert preferences. Schwaitzberg et al. [17] presented a report to evaluate the trend and market of VR simulator for NOTES training in clinic. They distributed a survey contained 32-point questionnaire at the 2012 NOSCAR annual meeting. Based on their study results, a decision was made to focus exclusively on the transvaginal hybrid NOTES cholecystectomy procedure, including both rigid and flexible scope techniques. Ahn et al. [18] introduced their preliminary development work of VR simulator for cholecystectomy procedure in NOTES. They aimed to simulate hybrid NOTES surgery using a rigid scope inserted the vaginal port.

In clinic, the most common procedures in NOTES are cholecystectomy, appendicectomy and peritoneoscopy, which are mainly performed via transvaginal access [1]. It is evident that morbidity appears to be higher when the transgastric route is used. The safety profile of hybrid NOTES transvaginal procedures is beginning to be confirmed as is evident from the large number of procedures presented in practice [19]. Our research focus on the simulation of transvaginal cholecystectomy by hybrid NOTES technique. So this Virtual Translumenal Endoscopic Surgery Trainer (VTEST™) aims to simulate hybrid NOTES cholecystectomy procedure using a rigid scope inserted through the vaginal port and a laparoscopic surgical instrument inserted through the umbilicus port (Figure 1).

Figure 1.

Peritoneal cavity access via the cul de sac in hybrid NOTES.

This paper describes a VR-based simulation system for transvaginal cholecystectomy by hybrid NOTES technique. We use a 6DOF haptic device (Geomagic Touch [20], formerly Sensable Phantom Omni) and a tracking sensor (3D Guidance Trakstar [21]) to construct the core hardware component of simulator. For graphic software interface, we firstly use the hybrid model, which contains both surface mesh and volumetric data, to represent the geometry of major organs and surrounding soft tissues involved in the cholecystectomy. To meet the requirement of real-time graphic performance and realistic deformation of soft tissue, a novel method based on inner-spheres is presented to compute the physical deformation of organs in procedure simulation. And an adaptive GPU-based collision detection method is designed to handle the frequent collision between soft tissue and surgical instruments. Then a multi-layer hexahedral model is proposed to simulate the blunt dissection of gallbladder fat tissue by electric cautery hook. Finally a preliminary pilot study was performed to evaluate the realism and the usefulness of this VR-based NOTES simulator.

Materials and Methods

Before the description of our technical work in VR-based surgical simulation, a brief introduction of cholecystectomy procedure in NOTES will be discussed.

Transvaginal cholecystectomy procedure overview

To start the transvaginal cholecystectomy, the pneumoperitoneum is generally established within the patient at first. The vaginal walls are retracted using speculums and a puncture site is located for trocar entry to gain the access to the peritoneal cavity. Then a single port trocar with a rigid endoscope is placed through the cul de sac and into the peritoneum ultimately [22]. This process is usually guided with an endoscope placed in a laparoscopic port through the patient’s navel. This scope can be used for monitoring the trocar entry into the peritoneal cavity. And the surgical instrument is also inserted into the peritoneal cavity through the navel for the operation afterwards. Figure 1 illustrates a sketch of access the abdomen transvaginally in hybrid NOTES.

A transvaginal cholecystectomy procedure by rigid scope is relatively the same with the traditional laparoscopic cholecystectomy. Firstly, the gallbladder needs to be retracted to expose the critical vasculature (Figure 2). This task can be handled by placing a grasper in one of navel trocar ports and retracting the gallbladder constantly. However, in a hybrid transvaginal cholecystectomy NOTES, there is only one port through the navel. Thus an alternative way is using large suture needles to penetrate the abdominal wall and suspend the gallbladder by sutures (Figure 4b). Once the gallbladder has been retracted and suspended, the following key procedures can be typically divided into four steps (Figure 3): (1) Identification of the cystic duct and cystic artery in the area of Calot’s triangle; (2) Clamping the cystic duct and cystic artery; (3) Dissection of cystic duct and artery; (4) Dissection of connective tissue between the liver and gallbladder. These four steps are also the key transvaginal cholecystectomy training tasks in our simulator.

Figure 2.

Organs and tissues involved in the cholecystectomy.

Figure 4.

The hardware and software interface of transvaginal cholecystectomy NOTES simulator. (a) The hardware interface of VR simulator; (b) Screenshot from the real surgery video; (b) The software interface of our VR simulator.

Figure 3.

Typical cholecystectomy procedures [23]. (a) Identification of the cystic duct and artery; (b) Clamping of cystic duct and artery; (c) Cutting of the cystic duct and artery; (d) Disconnect the liver and gallbladder.

The interface and architecture of VR simulator

The simulator hardware (Figure 4a) consists of a monitor, a desktop, a haptic devices (Geomagic Touch), a tracking sensor (3D Guidance Trakstar), a foot pedal and a plastic frame with pelvic model. The haptic device provides 6-DOF navigating parameters and 3-DOF force feedback. It can function as a grasper, a cautery hook or any other necessary surgical instrument in procedure. A tiny 6-DOF tracking sensor is attached to the tip of a rigid endoscope. It can track the position and orientation of camera in endoscope during surgical simulation. The foot pedal is used to control the electrosurgery for dissection and lock the camera in endoscope navigation.

The graphic software was developed with OpenGL, C++ and CUDA. The anatomy models were initially obtained from a standard 3d human anatomy database [24]. We select the abdominal organs and soft tissues, which involved in the cholecystectomy, with texture information. These models consist of the gallbladder, liver, cystic artery, cystic duct and bile duct. Referenced with the real cholecystectomy videos, an animator manually modified these models to generate the fat tissue covering the cystic duct and artery, and the membrane tissue connected between the liver and gallbladder. Finally these 3d anatomy models can be utilized in the cholecystectomy simulation. The architecture of the system is described as Figure 5. The core modules make up the Virtual Surgery Engine, which has been equipped with the following essential graphic functionalities in surgery training.

• Real-time graphic rendering and deformation of soft tissue.

• The force feedback control in haptic rendering.

• The endoscope navigation and orientation in the virtual abdominal cavity.

• The manipulation of basic MIS instruments, such as grasper, scissor, stapler and cautery hook.

• Multimedia special effects, such as smoke, bleeding and sound generation in electrosurgical dissection.

Figure 5.

The software architecture of our VR-based NOTES simulator.

From the clinical aspect, our most important contribution is to present a comprehensive framework of VR-based transvaginal cholecystectomy NOTES simulator. Compared with previous research in related fields, there are three innovative technical contributions:

1. An inner-spheres based approach is presented to compute the physical deformation of related organs in real-time during surgery simulation.

2. We designed an adaptive GPU-based collision detection method to handle the frequent collision between soft tissue and surgical instruments in procedure.

3. A multi-layer hexahedral model is designed to simulate the gallbladder fat tissue removal by electric cautery hook.

Inner-spheres based deformation of organs

Physical modeling and deformation of soft tissue is an essential task in virtual surgery. Generally human abdominal organs, such as liver and gallbladder, have continuous, blobby-like surfaces which can be expressed by a set of overlapping spheres with different radii. These spheres fill inside the polygon mesh of organ model. so we also name these spheres "inner-spheres". Here we simplify the physical structure of organs as a set of inner-spheres linked by springs. These spheres and springs can directly participate in the dynamic simulation to compute the deformation of organs. Besides, the surface mesh with high-precision geometric information and texture, is also employed to represent the boundary structure of organs. Then we treat the inner-spheres as the "interior skeleton" of organs and the surface mesh as the "skin". An automatic skinning algorithm based on distance field is applied to transform the deformation of inner-spheres to the exterior surface of organs finally. The simplified geometry of spheres affords far less details for the interior structure of organs model. Hence, it could effectively reduce the computational cost of deformation simulation.

Construction of the inner-spheres

The collection of overlapping spheres with different radii are extremely appropriate for geometrical modeling of organs, as it can express continuous, blobby-like surfaces. And spheres based approaches are also wildly used in collision detection due to this unique advantage. Many researchers have been carried out in this area. Bradshaw et al. [25] extended Hubbard’s theory [26] in collision detection and proposed an easy-to-use algorithm to pack spheres in polygon mesh. Here we use Sphere Tree Construction Toolkit in [25] to construct inner-spheres in the triangular mesh of organ initially. The key strategy of this toolkit is to find the medial surface of a 3d object using the Voronoi Diagram and pack spheres from the medial surface to approximate the object roughly. Figure 6b illustrates the initial spheres packing result of a liver. However, the initial inner-spheres model cannot match the mesh boundary very well. There are a large parts of spheres outside the surface of the object (Figure 6c). It will influence the accuracy of collision detection and deformation afterwards. So an optimization process, which contains radius reduction and vacant space filling with spheres, is needed to drive the inner-spheres model fit the mesh boundary.

Figure 6.

Construction of inner-spheres for liver model. (a) The original triangular mesh of liver; (b) Initial inner-spheres model of liver after spheres packing; (c) Liver model contained both mesh and inner-spheres; (d) Inner-spheres of liver after optimization. (e) Liver model contained both mesh and inner-spheres after optimization.

For sphere radius reduction, a distance field is constructed for each sphere center. We compute its corresponding shortest distance to the surface. Since we use a fine triangular mesh for the exterior structure of an organ model, the exact distance can be computed by Eq (1):

$$D(\mathbf{c})=\text{min }d(\mathbf{c},{\mathbf{T}}_i),$$ (1)

where c is the sphere center, T_i is the triangle in index i, and d(c, T_i) indicates the shortest distance between c and triangle T_i. Then we adjust the sphere radius according to the distance D(c), to ensure that the exterior parts of the sphere shrink inwards and contact with the surface boundary accurately.

The strategy of vacant space filling is the ”interpolation”, which is based on K-nearest neighbors algorithm. We search the vacant space around a sphere and fill it by adding new spheres (Figure 7). The position of new sphere can be determined by Eq (2):

$${\mathbf{c}}_{\mathit{\text{new}}}=\frac{d({\mathbf{c}}_1,{\mathbf{c}}_2)+r_2-r_1}{2d({\mathbf{c}}_1,{\mathbf{c}}_2)}{\mathbf{c}}_1+\frac{d({\mathbf{c}}_1,{\mathbf{c}}_2)+r_1-r_2}{2d({\mathbf{c}}_1,{\mathbf{c}}_2)}{\mathbf{c}}_2,$$ (2)

where c₁, c₂ are the coordinates of center for spheres 1 and 2, and r₁, r₂ are radii for spheres 1 and 2. d(c₁, c₂) is the distance between c₁ and c₂. The radius of new sphere is the shortest distance between c_new and mesh surface. r_new can be computed by Eq (1). Figure 6d illustrates the final result of inner-spheres model of liver after radius reduction and vacant space filling. From Figure 6e, we can find the inner-spheres model fit the mesh boundary very well.

Figure 7.

The illustration of vacant space filling.

Deformation of inner-spheres

To deform the inner-spheres of a given organ model, we firstly build the dynamic system (physical model) by connecting these spheres with springs. For each sphere, we select its nearest adjacent spheres among all its overlapping spheres. The number of adjacent spheres is limited (Maximum number is 6) to simplify the topology of inner-spheres. Figure 8 illustrates the springs connection of a liver inner-spheres model.

Figure 8.

The topology construction of the liver inner-spheres model. (a) The original inner-spheres model; (b) The inner-spheres model with connectivity.

After springs connection, we use position based dynamics (PBD) to compute the position of spheres during simulation, due to its robustness and position based manipulation feature. These advantages make PBD very popular in the game industry and VR medical simulator development [27]. The technical detail of PBD algorithm can be found in [28]. Here we apply two types of constraints: stretching constraint and Laplacian coordinates constraint, to the dynamic system. Considering the irregular topology of inner-spheres model, we cannot treat it as the polygon or polyhedron mesh. Only the stretching constraints can be used to deform the inner-spheres. Figure 9 gives an example of stretching constraint for two connected spheres. The constraint function can be described as Eq (3).

$$C_{\mathit{\text{streching}}}({\mathbf{p}}_1,{\mathbf{p}}_2)=|{\mathbf{p}}_1-{\mathbf{p}}_2|-d,$$ (3)

where d is the initial distance between sphere centers p₁ and p₂ before deformation. w_i is the reciprocal of the mass of sphere i. The displacement in each iteration can be computed by Eq (4) and (5):

$$\mathrm{\Delta }{\mathbf{p}}_1=-\frac{w_1}{w_1+w_2}(|{\mathbf{p}}_1-{\mathbf{p}}_2|-d)\frac{{\mathbf{p}}_1-{\mathbf{p}}_2}{|{\mathbf{p}}_1-{\mathbf{p}}_2|},$$ (4)

$$\mathrm{\Delta }{\mathbf{p}}_2=\frac{w_2}{w_1+w_2}(|{\mathbf{p}}_1-{\mathbf{p}}_2|-d)\frac{{\mathbf{p}}_1-{\mathbf{p}}_2}{|{\mathbf{p}}_1-{\mathbf{p}}_2|}.$$ (5)

Figure 9.

The illustration of stretching constraint for two connected spheres.

Stretching constraints only makes 2d constraints and lacks of constraints for 3d space. Here we introduce the Laplacian coordinates constraint [29] to preserve the local detail of the inner-spheres shape. The method can be described like this: For any sphere m whose center is c_m, c_i indicates the center of spheres which are connected with the sphere m in topology. The number of these connected spheres is n. So the center coordinates of these adjacent spheres center can be computed by Eq (6).

$${\mathbf{c}}_{\mathit{\text{center}}}=\sum _i^n\frac{{\mathbf{c}}_i}{n},$$ (6)

Before the simulation iteration, we can initialize the Laplacian coordinate of c^m by (7):

$${\mathbf{L}}_m={\mathbf{c}}_m-{\mathbf{c}}_{\mathit{\text{center}}}={\mathbf{c}}_m-\sum _i^n\frac{{\mathbf{c}}_i}{n},$$ (7)

During iteration, we treat L_m as a fixed vector constraint. ${\mathbf{c}}_{\mathit{\text{center}}}^{\prime }$ is the updated c_center by (6). Then the updated position of ${\mathbf{c}}_m^{\prime }$ can be computed by Eq (8):

$${\mathbf{c}}_m^{\prime }={\mathbf{L}}_m+{\mathbf{c}}_{\mathit{\text{center}}}^{\prime }.$$ (8)

Deformation of surface mesh by skinning

After the deformation of inner-spheres, the final task is transforming this deformation to the exterior surface. Firstly we need to build the mapping between the inner-spheres model and surface mesh of organs. This process is very similar to the skinning technique used in the skeleton driven animation [30]. Here we treat the spheres as the "interior skeleton" and the polygon mesh as the "skin". And an automatic algorithm based on distance field function is designed to assign the weights for each vertex in surface mesh. It can be described as follows:

For a sphere, its field strength of weighting can be computed by a function related with its radius, the distance between the vertex on mesh and its sphere center. Here we use the non-linear field strength of weighting, which can be described as a Gaussian function:

$$f(d,r)=c·e^{-\frac{{(d-r)}^2}{r^2}},$$ (9)

where r stands for the sphere radius. d is the distance between a point and the sphere center. c is a constant coefficient.

For a vertex v on the surface mesh, we can find its attached spheres i(i = 1,2,…n) among all the inner-spheres by this condition: The attached spheres satisfy f (d_v,ci,r_i) ≤ T, where, d_v,ci is the distance between v and sphere center c_i. T stands for a threshold. So for v, each attached spheres i has a different influence weight on v. We set a weighted center coordinate of these attached spheres. And it can be computed by Eq (10).

$${\mathbf{c}}_{\mathit{\text{center}}}=\sum _i^n\frac{f(d_{\mathbf{v},{\mathbf{c}}_i},r_i)}{\sum _i^{n}f(d_{\mathbf{v},{\mathbf{c}}_i},r_i)}{\mathbf{c}}_i,$$ (10)

where c_i is the center of sphere i. Before simulation, the initial displacement between v and c_center will be computed by Eq (11):

$$\mathbf{d}=\mathbf{v}-{\mathbf{c}}_{\mathit{\text{center}}}.$$ (11)

During simulation, we can compute the updated v′ using Eq (12) in each frame.

$$\mathbf{v}\prime ={\mathbf{c}}_{\mathit{\text{center}}}^{\prime }+\mathbf{d}$$ (12)

where ${\mathbf{c}}_{\mathit{\text{center}}}^{\prime }$ is the updated weighted center coordinate of attached spheres. Figure 10(b) illustrates the final deformation result of liver model by this skinning method. The liver model is interacted with a grasper.

Figure 10.

The deformation of liver model by our inner-spheres based method. (a) The inner-spheres model; (b) The surface mesh with texture.

Adaptive Cylinder Based Collision Detection by GPU

In transvaginal NOTES cholecystectomy, there is frequent collision among abdominal organs and also between organs and surgical instruments. For the collision among multiple organs and self-collision of a single organ, The use of spheres in collision detection is not new and some similar approaches were implemented [31, 32]. Here we use a straightforward geometric projection method to handle the collision. It directly corrects the positions of spheres after a contact was detected. In this projection method, when intersection happens, a correction was directly applied to move the sphere to clear the occlusion. The direction of the movement is the normal of the intersection plane. As there is no need to compute the penalty forces, the solving process can be speeded up. Figure 11 outlines this collision detection process.

Figure 11.

Illustration of collision handling for multiple organs.

Most interactions in virtual surgery happen at the collision and manipulation between surgical instruments and soft tissues. Therefore the collision detection of rigid surgical instrument and soft tissues takes the most computation time for interaction [10]. In a real-time VR system, the display rate of graphic rendering is about 30 frames per second. However, in a multi-modal virtual environment with haptic interactions, such as virtual surgery system, a much higher update rate of about 1 kHz is achieved to ensure smooth force feedback and continuous interactions. Theoretically, the force from the haptic device applies to only one contact point during manipulation. The most efficient interaction method is to represent the proxy (haptic cursor) as a point [32]. However, in many cases, the instruments used in minimally invasive surgery (MIS) are long slender tools (such as grasper, cautery hook) whose contact surface is an area or a line. In our simulator, as shown in Figure 12, we treat the surgical instrument as the combination of a set of cylinders with different radii and orientations. An innovative adaptive cylinder based collision detection algorithm is designed for haptic rendering. It computes the distance between each sphere and each cylinder to test whether there is a collision. This process can be described as Figure 13 and Algorithm 1.

Figure 12.

The simplification of surgical instruments (cautery hook and grasper) with a set of cylinders.

Figure 13.

The illustration of collision detection between a sphere and a cylinder.

Since the collision detection for each sphere and cylinder is independent, Algorithm 1 is very suitable for parallel processing by GPU acceleration. Algorithm 2 illustrates the kernel in CUDA of intersections between a tool and inner-spheres model. We use each thread to compute the distance from each sphere to the tool respectively. In transvaginal cholecystectomy, surgeons basically manipulates two instruments: grasper and cautery hook (Figure 12). Here both of them are approximated by three cylinders in our tests. In Algorithm 2, each thread computes the distance to these cylinders in order. If there is a distance less than zero, we believe that the corresponding cylinder is intersected with the sphere. And a vector array is used to record the index of spheres intersected with the instrument and the 3d coordinates of contact point.

Another advantage of our approach is that the accuracy of collision can be adaptively controlled by the number of cylinders simplified for the surgical instrument. For a surgical instrument with irregular shape, we can represent it with more cylinders to achieve higher accuracy. For the force feedback in haptic rendering, we compute the value and orientation of output force by the straightforward method based on Hook's law [33, 34].

Algorithm 1.

Compute the distance from a cylinder to a sphere

Input: The two end points of the cylinder medial axis p1,p2 and the radius r0 of the cylinder. The sphere center position c and radius r.
Output: Whether the sphere and the cylinder are intersected
1:	p0 ← (p1 + p2)/2
2:	h ← distance(p1, p2)
3:	u ← getNormalizedVector(p1, p2)
4:	x ← |dot(u, cp0)|
5:	n ← distance(c, p0)
6:	$d\leftarrow \sqrt{n^2-x^2}$
7:	if x < h/2 then
8:	if d < r0 + r then
9:	intersection ← true
10:	else
11:	intersection ← false
12:	end if
13:	else
14:	if d < r0 then
15:	if x − h/2 < r then
16:	intersection ← true
17:	else
18:	intersection ← false
19:	end if
20:	else
21:	if $\sqrt{{(d-r_0)}^2+{(x-h/2)}^2}<r$ then
22:	intersection ← true
23:	else
24:	intersection ← false
25:	end if
26:	end if
27:	end if

Algorithm 2.

Kernel: Intersection between tool and the meta-balls

1:	tid ← getThreadIndex()
2:	if sphere[tid] then
3:	dist ← shortestDistance(upperblade, sphere[tid], point)
4:	if dist < 0 then
5:	collisionSign ← true
6:	collisionSpherelndex ← tid
7:	collisionPlace ← point
8:	end if
9:	dist ← shortestDistance(lowerblade, sphere[tid], point)
10:	if dist < 0 and collisionSign = false then
11:	collisionSign ← true
12	collisionSpherelndex ← tid
13:	collisionPlace ← point
14:	end if
15:	dist ← shortestDistance(handle, sphere[tid], point)
I6:	if dist < 0 and collisionSign = false then
17:	collisionSign ← true
I8:	collisionSpherelndex ← tid
19:	collisionPlace ← point
20:	end if
21:	end if

The simulation of fat tissue dissection by multi-layer hexahedral model

Besides abdominal organs, such as liver and gallbladder, cholecystectomy also involves cystic duct, cystic artery and the fat tissue covering them. Identification of the cystic duct, cystic artery and removal of the fat tissues is an essential training task in the cholecystectomy procedure. Surgeons need to carefully dissect and remove the fat tissues using a cautery hook without damaging the cystic duct and artery. To simulate this dissection process, we proposed a innovative multi-layer hexahedron model to handle the fat tissue removal.

Figure 14 illustrates the construction process of the multi-layer hexahedron model. Our task is to fill the vacant space inside the fat tissue mesh with hexahedra. Unlike the inner-spheres which are used to deform the major organs, hexahedron is treated as the basic volumetric element for fat tissue removal. Figure 14(a) shows the area of gallbladder with fat tissue for electric dissection in simulator software interface. From Figure 14(b), we can find the fat tissue structure is located between two layer of meshes. Then we segment the two layer meshes according to the texture distribution of fat tissue in Figure 14(a). Figure 14(c) shows the mesh segmentation result. The interior mesh consist of gallbladder, cystic duct and cystic artery (the red mesh). The exterior mesh is the surface of fat tissue. Then a pile of silhouettes around the fat tissue are detected along the direction of cystic duct (Figure 14(d)). In each cross section, we generate the intermediate layers between the silhouette of fat tissue surface and cystic duct, artery by interpolation (Figure 14(e)). To create quadrangles, we make uniform sampling on each layer and connect the adjacent points on different layers. Figure 14(e) shows the geometry structure of a cross section in the multi-layer hexahedral model. In the end, we connect all the points with their adjacent points in the neighbouring cross sections to generate the whole hexahedral model (Figure 14(f)).

Figure 14.

The construction process of the multi-layer hexahedron model for fat tissue. (a) The exterior gallbladder and fat tissue model with texture; (b) Exterior and interior mesh (two layers) of gallbladder and fat tissue; (c) Exterior and interior mesh in the area of fat tissue; (d) Silhouettes for the exterior and interior mesh; (e) Silhouette and intermediate layers in one cross section; (f) Multi-layer hexahedral model.

During dissection simulation, the deformation of interior and exterior mesh (two layers) for fat tissue can be computed by our inner-spheres based method respectively. Hence, after the construction of multi-layer hexahedral model, the position of each point in the intermediate layers can be interpolated according to the shape of interior and exterior mesh (two layers) in very frame. For the removal of fat tissue, we simplify this process as the hexahedra deleting. Once the cautery hook contacts the surface of fat tissue, the nearest hexahedron in the contacted area will be deleted and the surface mesh of fat tissue will shrink inwards. To prevent damaging the cystic duct and artery in training, we also increase the transparency of hexahedral model when a hexahedron is deleted, to help the trainer notice the change of cystic duct and artery inside the fat tissue. Figure 18(a) illustrates a screenshot of fat tissue removal in electric dissection.

Figure 18.

Screenshots of the transvaginal cholecystectomy simulation. (a) Identification of the cystic duct and artery; (b) Clamping of cystic duct and artery; (c) Cutting of the cystic duct and artery; (d) Disconnect the liver and gallbladder.

With the techniques above, a prototyped VR-based simulation system (VTEST™) is developed for the transvaginal NOTES cholecystectomy. When the system starts, the simulator generates a continuous loop with the force feedback. The haptic devices send the orientation and position of surgical instruments to the computer. The collisions detection between the surgical instruments and soft tissue are processed according to these data. Then the signal of feedback force is sent to the haptic devices for haptic rendering. The deformation of organs are computed by the inner-spheres based method. For the fat tissue dissection, the multi-layer hexahedral model is used to simulate and control the fat tissue removal process. Our system is running on a desktop with NVIDIA GeForce GTX 460, Intel(R) Core(TM)2 Quad CPU (2.66GHz, 4 cores) and 4G RAM. The force refreshing frequency is 1000Hz and the graphic refreshing frequency is about 55Hz in average.

To evaluate the effectiveness of above techniques used in this simulator, we have designed three experiments.

Experiment 1 is to test the visual performance and computation cost of the inner-spheres based deformation method. We construct the inner-spheres model for four abdominal organs: liver, gall-bladder, intestine and stomach. These organs are also interacted and deformed by a grasper. We also compare our method with three typical approaches (FEM, mass-spring and PBD) by deforming a liver model.

Experiment 2 is to compare our adaptive GPU-based collision detection method with other two typical approaches [10, 35], in interacting with four different abdominal organs: liver, gall-bladder, intestine and stomach.

Experiment 3 is to test the visual performance of our simulator in four essential training tasks (Figure 3) in transvaginal cholecystectomy.

In addition, this NOTES simulator had been demonstrated at 2013 National Orifice Surgery Consortium for Assessment and Research (NOSCAR) summit (ASGE IT&T Center, Downers Grove, IL). In face validation, we invited ten subjects with varying NOTES and laparoscopic surgery experiences to verify and evaluate our simulator. The subjects were asked to perform the transvaginal cholecystectomy NOTES simulation including four major training tasks (Figure 3), and then to answer a questionnaire consisting of fourteen questions: (1) realism of the anatomy; (2) realism in identification of the Calot's triangle; (3) realism of the appearance of the simulator interface; (4) realism of the instrument handling; (5) overall realism of the electric dissection task; (6) overall realism of the gall bladder removal task; (7) overall realism compared to the traditional laparoscopic tasks; (8) quality of the force feedback; (9) usefulness of the force feedback in performing the tasks; (10) usefulness in learning hand-eye coordination; (11) usefulness in learning ambidexterity skills; (12) overall usefulness in learning the fundamental NOTES skills; (13) trustworthiness of the simulator in quantifying accurate measures of performance; (14) trustworthiness of the simulator in providing different hand-eye coordination compared to traditional two-port laparoscopic approach. Each question can be answered on a 5-point Likert scale with 1 for very poor and 5 for very good satisfactory.

Results

Experiment 1

Figure 15 illustrates the deformation result of three abdominal organs (gall-bladder, intestine and stomach) by inner-spheres based method. The deformation result of liver can be found in Figure 8 and 10. Table 1 documents the data size and the computation cost of the inner-spheres construction and deformation in Experiment 1.

Figure 15.

The inner-spheres model and deformation result of different abdominal organs (From left to right: mesh and the inner-spheres model; the topology connection for the inner-spheres; deformation result 1; deformation result 2). (a) Gall-bladder, (b) Intestine, (c) Stomach.

Table 1.

Data size and computation cost of our inner-spheres based deformation method for liver and gallbladder.

Model	Number of vertices	Number of spheres	Number of springs	Time for inner-spheres construction (second)	Time for deformation (ms)
Liver	8128	581	3588	2207	7.5
Gallbladder	1461	475	3293	1179	2.9
Intestine	887	383	2625	754	1.2
Stomach	3349	452	3161	2050	8.1

Table 2 is the numerical comparison with three typical approaches (FEM, mass-spring and PBD) in deforming a liver model. Hexahedra are used in the FEM tests. PBD tests uses stretch constraints and volume conservation constraints. Figure 16shows the deformation result. Here we choose the Explicit Euler solver for mass-spring method.

Table 2.

Comparison our method with three typical deformation approaches (FEM, mass-spring and PBD).

	FEM	Mass-spring method	PBD	Our method
Elements + Constraints	967	6427	6427+4079	3588+581
Deformation (ms)	48.6	2.5	2.2	7.6

Figure 16.

The comparison between our method and three typical approaches in the liver deformation. (a) FEM based method; (b) Mass-spring method; (c) PBD based method; (d) Our method.

Experiment 2

Table 3 gives the data size of four abdominal models and the response speed of our collision detection method. Figure 17 illustrates the comparison result with other methods.

Table 3.

Data size and time cost of our collision detection method based on inner-spheres for four abdominal organs.

Model	Number of vertices	Number of triangular faces	Number of spheres	Time for collision detection by our method (ms)
Liver	8128	12252	581	1.25
Gallbladder	1461	2918	475	1.21
Intestine	887	1770	383	1.16
Stomach	3349	6694	452	1.24

Figure 17.

The comparison of computation efficiency for collision detection between soft tissue and surgical instrument among three different approaches.

Experiment 3

Figure 18 illustrates the screenshot of simulator software interface during four key training tasks (Figure 3).

For the validation of our simulator in 2013 NOSCAR summit, the results from the questionnaire study are given as Table 4.

Table 4.

Descriptive statistics obtained from the questionnaire study.

Question	1	2	3	4	5	6	7	8	9	10	11	12	13	14
Mean	3.1	2.7	3.5	3.4	2.6	2.8	3	2.3	3.8	3.1	3.1	3.1	3.3	3.6
SD	0.88	0.67	0.97	0.84	1.17	1.03	0.94	0.82	0.78	0.99	0.87	0.99	1.41	0.97

Discussion

For the result in Experiment 1, the inner-spheres construction take the most computation time in Table 1. But this task is pre-processed in initialization, it will not influence the real-time performance in surgical simulation. In Table 2, the computation time of our method is at the same level with mass-spring and PBD, much lower than the FEM. Due to the local detail and volume preservation, the deformed liver surface by our method is much more realistic than PBD and mass-spring method (Figure 16). Therefore, among all these approaches, the cost performance of our inner-spheres based deformation method is the best.

In Experiment 2, the complexity of our method is O(log(nm)) when running on the CPU platform. n is the number of spheres in the inner-spheres model. m is the number of cylinders approximated for the surgical tool (etc. grasper). The complexity of method in [10] is O(pm), p is the number of vertices in organ mesh. m is the number of cylinders approximated for the surgical tool. The complexity of method in [35] is O(log(q)), q is the number of faces in organ mesh. Generally, the number of spheres in the inner-spheres model is much smaller than the number of vertices and faces in a mesh model. And when the number of vertices and faces increase, our adaptive inner-spheres construction method can maintain the number of spheres as a nearly constant in low level. Therefore our method is specially efficient to handle the collision of organs with large mesh data.

From the result in Experiment 3, we found that the deformation of gallbladder is realistic. In the end, the gallbladder and liver tissue are clearly separated in Figure 18 (d). Moreover, our simulator can provide special visual effects, such as smoke and bleeding. The function of notification is also included in the "Evaluation of training" module of our simulator (Figure. 5). When the trainee touch the incorrect or dangerous anatomy area, the system will give a alarm and lower the evaluation score of the "correct operation" for this trainee. If the trainee cut the cystic duct or artery without clamping, the system will close the training procedure and let this trainee fail directly in this task. In the face validation process, the result from the preliminary pilot study (Table 4) shows the average scores for all the questions. It also indicates which aspects need to be improved for more realistic and useful simulation. In Table 4, the lowest score (2.3) are "quality of the force feedback". Currently, we use the same computation model (Hook's law based method) to output the force feedback for the organs and soft tissue (e.g. membrane, fat) in surgical simulation. In future, we can use different physical model to compute the force for the organs, fat and membrane tissue. And we can also set more accurate physical parameters, such as elastic coefficient, through vivo measurement in animal experiments.

Conclusion and Future Work

We have developed a prototype VR-based NOTES simulator for transvaginal cholecystectomy. Several technical innovations have been described in this article, including the inner-spheres based deformation method for the organs, an adaptive GPU-based collision detection, a multi-layer hexahedral model to simulate fat tissue removal. Real-time visual performances is achieved. The surgeon collaborators are satisfied with the realistic visual quality of our simulator.

Despite successful implementation of the prototype, this simulator also has a few limitations. The first is the force feedback in haptic rendering. Currently, we use the same computation model to output the force for the organs, membrane, fat and other issue with different material properties. In future we can use different methods to compute the haptic force and set more accurate physical parameters by animal experiments. The second is the torque output in grasping and cutting. Since Geomagic Touch provides only a 3-DOF force feedback, the torque is not considered. In real procedure, proper grasping or cutting torque is required. So we plan to choose more sophisticated haptic devices coupled with torque feedbacks to enhance the tactile realism. In future, more trials and study will also be made in the evaluation of our simulator. With the collected and analyzed data, this VR-based NOTES simulator will be compared quantitatively with other approaches, such as physically based training and real human surgery training.

Highlights.

A prototyped VR-based transvaginal cholecystectomy NOTES simulator is developed.

The Inner-spheres based method is presented to compute the deformation of organs.

An adaptive GPU-based collision detection approach is presented.

A multi-layer hexahedral model is designed to simulate the fat tissue removal.