Real-time computed tomography-based augmented reality for natural orifice transluminal endoscopic surgery navigation

Abstract

Background

Natural orifice transluminal endoscopic surgery (NOTES) is technically challenging owing to endoscopic short-sighted visualization, excessive scope flexibility and lack of adequate instrumentation. Augmented reality may overcome these difficulties. This study tested whether an image registration system for NOTES procedures (IR-NOTES) can facilitate navigation.

Methods

In three human cadavers 15 intra-abdominal organs were targeted endoscopically with and without IR-NOTES via both transgastric and transcolonic routes, by three endoscopists with different levels of expertise. Ease of navigation was evaluated objectively by kinematic analysis, and navigation complexity was determined by creating an organ access complexity score based on the same data.

Results

Without IR-NOTES, 21 (11·7 per cent) of 180 targets were not reached (expert endoscopist 3, advanced 7, intermediate 11), compared with one (1 per cent) of 90 with IR-NOTES (intermediate endoscopist) (P = 0·002). Endoscope movements were significantly less complex in eight of the 15 listed organs when using IR-NOTES. The most complex areas to access were the pelvis and left upper quadrant, independently of the access route. The most difficult organs to access were the spleen (5 failed attempts; 3 of 7 kinematic variables significantly improved) and rectum (4 failed attempts; 5 of 7 kinematic variables significantly improved). The time needed to access the rectum through a transgastric approach was 206·3 s without and 54·9 s with IR-NOTES (P = 0·027).

Conclusion

The IR-NOTES system enhanced both navigation efficacy and ease of intra-abdominal NOTES exploration for operators of all levels. The system rendered some organs accessible to non-expert operators, thereby reducing one impediment to NOTES procedures.

Introduction

Natural orifice transluminal endoscopic surgery (NOTES) has the potential to offer advances in abdominal exploration¹. Technical difficulties to be addressed include reliable access to the peritoneal cavity, closure systems, prevention of infection, means of controlling intraperitoneal haemorrhage, adequate instrumentation (suturing and anastomotic devices) and effortless spatial orientation². Using endoluminal instruments designed for intra-abdominal applications has demonstrated inherent impediments to NOTES development: short-sighted visualization precluding easy identification of anatomical land-marks, and overflexibility causing ‘looping’ or preventing intuitive spatial awareness of scope position.

Augmented reality has been proposed to overcome navigational challenges^3-6. To this end, the authors have investigated an image registration system for NOTES procedures (IR-NOTES). Previous iterations of this system have been used to improve efficiency and structure identification in endoscopic ultrasonography (EUS)⁷. The system is based on a three-dimensional (3D) reconstruction generated from a computed tomography (CT) image acquired before the procedure. A sensor is attached at the tip of the endoscope, providing real-time 3D positioning. The resulting output is real-time 3D visualization of the abdominal organs and live tracking of the tip of the endoscope (Fig. 1). Few human studies are available to date, but recent publications have confirmed the technical complexity of intra-abdominal NOTES navigation in humans⁸. The system described in the present study is intended to allow visualization of otherwise difficult or in accessible organs during human NOTES procedures, even in the hands of non-expert operators.

Fig. 1.

Natural orifice transluminal endoscopic surgery with image registration system: view with endoscope. The screen shows real-time tracking of the endoscope tip in relation to a three-dimensional reconstruction of relevant anatomy. The insert shows laparoscopic and endoscopic views

Methods

Experiments were conducted at the minimally invasive surgery centre at Harvard Medical School. Three adult human cadavers were used for this study, one female and two male. A list of 15 predetermined intra-abdominal organs were identified as target points to be reached endoscopically via both a transgastric (upper) and transcolonic (lower) route (Table 1). Transgastric access sites were created in the anterior mid-body of the stomach and transcolonic access sites at the rectosigmoid junction. These targets were chosen for their clinical or anatomical significance. A total of three rounds of organ targeting were designed (two without and one with IR-NOTES) by three endoscopists with different levels of experience (intermediate, 2 years’ experience; advanced, 4 years’ experience, expert, over 10 years’ experience). This provided 90 target points with IR-NOTES (15 organs × 3 operators × 2 routes) and 180 points without.

Table 1.

Intra-abdominal target organs

Right upper quadrant	Left upper quadrant	Right lower quadrant	Left lower quadrant	Centre	Pelvis
Left liver lobe	Gastric antrum	Appendix	Sigmoid colon	Anterior abdominal wall	Bladder
Right liver lobe	Gastric fundus	Right inguinal ring		Transverse colon	Rectum
Right diaphragm	Splenic hilum			Small bowel
Gallbladder

The IR-NOTES system was derived from the system used to improve efficiency and structure identification in EUS⁷ and designed specifically for flexible endoscopy⁹. The data for the system were collected using a standard CT scanner (Sensation; Siemens Medical Solutions, Malvern, Pennsylvania, USA). The 3D anatomical models were generated using the open-source software package 3D Slicer (http://www.slicer.org). The models were generated using a semiautomatic approach.

The synthetic images have no perceptible lag when the endoscope is moved. The IR-NOTES system uses established techniques for the visualization of probe position and image registration, but implements them in real time by using recent advances in miniaturized position-tracking technology (microBIRD; Ascension Technology, Burlington, Vermont, USA). The tracking sensors are small (0·3 mm diameter, 1·8 mm length) and have been tested to meet International Electrotechnical Commission (IEC) 60601-01 standards. Such a sensor is attached to the tip of a standard colonoscope (Olympus CFQ-160; Olympus America, Center Valley, Pennsylvania, USA). The system relies on standard desktop computer software, the sensor, and a sensor tracker that provides the reference on which the sensor position is based. All components (tracker system, interfaces, personal computer with displays) are commercially available, with a total cost well below US $20 000, depending on the size of the displays, and the software is written as a module of the open-source 3D Slicer environment. The sensor tracker is a metal plate placed under the subject. The camera’s field of view is first calibrated with respect to the sensor placed on the scope. Then, the subject CT volume is registered to the tracker coordinate system using points collected over the skin. This provides accurate, 3D, real-time positioning of the tip of the endoscope at all times.

Analysis of performance

The efficacy of IR-NOTES was defined by the number of anatomical targets located by an operator in a fixed time period. It was measured as the number of target organs not acquired after 15 min of navigation (failed targets).

Ease of navigation with or without IR-NOTES was evaluated objectively by precisely analysing movements of the endoscope in 3D using the sensor utilized for IR-NOTES for all arms. The sensor also provides kinematic (movement analysis) data of the scope. The kinematic analysis measures the following parameters: path length (in millimetres), time elapsed (in seconds), linear jerk (in millimetres), rotation 1 and 2 (in degrees), and angular jerk 1 and 2 (in degrees). Path length records the total distance travelled by the tip of the endoscope to reach a target; movements in any direction are added to calculate total path length. Elapsed time is measured from entrance into the abdominal cavity to target acquisition. Linear jerk is the time derivative of acceleration. This was computed from the position measurements obtained from the sensors on the endoscope. Rotation 1 and 2 measure the total angular rotation of the endoscope on its longitudinal and transverse axes respectively. Angular jerk 1 and 2 measure the angular smoothness of motion on the longitudinal and transverse axes respectively. When IR-NOTES was not used for navigation, the sensors were still active to record data in order to provide the kinematic analysis. Laparoscopic monitoring (blinded to the endoscopist) was used to confirm target acquisition during the procedure and video logs were kept in the same fashion.

Navigation complexity for each target organ was determined by creating an organ access complexity score. This score was created to translate the difficulty of accessing a specific organ during NOTES procedures. Points were attributed using failure rate (failed to access organ once or twice, 1 point; failed to access organ three or four times, 2 points; failed to access organ five times or more, 3 points) and kinematic data (one or two variables statistically significant when accessing the organ with IR-NOTES, 1 point; three or four variables statistically significant, 2 points; five or more variables statistically significant, 3 points). Organs were then categorized according to the total score: easy to access (0 points), intermediate (1 and 2 points) and complex (3 points or more).

Statistical analysis

Comparisons for the two groups were performed using a two-sample t test. P < 0·050 was considered statistically significant.

Results

Use of image registration system

IR-NOTES including calibration was set up by two people in less than 15 min. Recalibration was needed once during the study, because a sensor had to be replaced. Visually, there was no difference between the position on the 3D model by IR-NOTES and that on the laparoscopic screen.

The stomach was emptied before the procedure in order to proceed with gastric insufflation and transgastric access (variations in gastric distension or emptying between time of acquisition and procedure can generate minor inconsistencies in navigation around that organ).

Efficacy of target acquisition

Without IR-NOTES, 21 (11·7 per cent) of 180 total target points were not reached with standard endoscopic visualization. When IR-NOTES was used, the failure rate decreased significantly to only 1 per cent (1 of 90 failed targets) (P = 0·002). Without IR-NOTES, the expert endoscopist had three failed targets, the advanced endoscopist had seven and the intermediate endoscopist had 11 failed targets. The only failure with IR-NOTES involved the intermediate endoscopist.

Failed target acquisition without IR-NOTES occurred for the following organs (number of failures for a total of 12 attempts per organ): right diaphragm (1), splenic hilum (5), stomach (5), sigmoid colon (3), right inguinal ring (1), rectum (4) and bladder (2) (Table 2). The target not acquired with IR-NOTES was the splenic hilum.

Table 2.

Failed target organs when image registration system was not used for natural orifice transluminal endoscopic surgery

	Transgastric	Transcolonic
Right upper quadrant
Right diaphragm	1	0
Left upper quadrant
Gastric antrum	2	1
Gastric fundus	1	1
Spleen	3	2
Right lower quadrant
Right inguinal ring	1	0
Left lower quadrant
Sigmoid colon	2	1
Pelvis
Bladder	1	1
Rectum	0	4

Ease of navigation

IR-NOTES results were statistically significantly better in at least one kinematic measurement for eight of the 15 organs: left liver lobe, splenic hilum, gastric antrum, transverse colon, small bowel, anterior abdominal wall, rectum and bladder (Table 3). Nineteen (18·1 per cent) of the 105 kinematic variables measured showed significant improvement with IR-NOTES. The organs for which most parameters were significant were the rectum and bladder.

Table 3.

Statistical difference in kinematic analysis between target acquisition in natural orifice transluminal endoscopic surgery with versus without an image registration system

	P *
	Time	Distance	Linear jerk	Rotation 1	Rotation 2	Angular jerk 1	Angular jerk 2
Gallbladder	NS	NS	NS	NS	NS	NS	NS
Left liver lobe	NS	NS	NS	NS	NS	0.007	NS
Right liver lobe	NS	NS	NS	NS	NS	NS	NS
Right diaphragm	NS	NS	NS	NS	NS	NS	NS
Splenic hilum	NS	NS	0.014	NS	NS	0.005	0.008
Gastric antrum	NS	NS	0.024	NS	NS	NS	0.010
Gastric fundus	NS	NS	NS	NS	NS	NS	NS
Small bowel	NS	NS	NS	0.041	NS	NS	NS
Appendix	NS	NS	NS	NS	NS	NS	NS
Transverse colon	NS	NS	NS	0.045	NS	0.043	NS
Sigmoid colon	NS	NS	NS	NS	NS	NS	NS
Rectum	0.002	0.009	NS	<0.001	<0.001	NS	0.011
Anterior abdominal wall	NS	NS	NS	NS	NS	0.028	NS
Right inguinal ring	NS	NS	NS	NS	NS	NS	NS
Bladder	0.043	0001	NS	0.001	0.001	NS	NS

^*Two-sample t test. NS, not significant.

The ease of navigation of IR-NOTES was also analysed according to the access route used. With a transcolonic approach a total of six kinematic variables (in 4 organs: right diaphragm, spleen, small bowel, right inguinal ring) were statistically significantly improved with the use of IR-NOTES. This compared with 17 variables (in 6 organs: spleen, transverse colon, rectum, anterior abdominal wall, right inguinal ring and bladder) improved when a transgastric approach was used. It should be noted that for the rectum the large number of failed attempts through a transcolonic approach reduced the number of variables achieving statistical significance via that route.

Use of IR-NOTES allowed the intermediate endoscopist to significantly improve ten kinematic variables in six organs. The advanced endoscopist improved 13 metrics in five organs, and the expert improved one metric.

Organ access complexity score

The complexity score for each target organ ranged from 0 (easily accessed) to 5 (complex access) (Table 4). When these organs were grouped by location in the abdomen, the most complex areas to access were the pelvis and left upper quadrant (Table 5).

Table 4.

Organ access complexity score

	No.of statistically significant variables	No.of failures	Total no. of points	Category
Gallbladder	0	0	0	Easy
Right liver lobe	0	0	0	Easy
Small bowel	1	0	0	Easy
Appendix	0	0	0	Easy
Anterior abdominal wall	1	0	0	Easy
Left liver lobe	1	0	0	Easy
Right diaphragm	0	1	1	Intermediate
Gastric fundus	0	2	1	Intermediate
Transverse colon	2	0	1	Intermediate
Right inguinal ring	0	1	1	Intermediate
Sigmoid colon	0	3	2	Intermediate
Gastric antrum	2	3	3	Complex
Rectum (transcolonic)	1	4	3	Complex
Rectum (transgastric)	5	0	3	Complex
Bladder	4	2	3	Complex
Splenic hilum	3	5	5	Complex

Score indicates level of difficulty in accessing labelled organ with natural orifice transluminal endoscopic surgery.

Table 5.

Organ access complexity aggregated score by location

	Organ access complexity score
Right upper quadrant
Left liver lobe	0
Right liver lobe	0
Right diaphragm	1
Gallbladder	0
Aggregate score	1
Left upper quadrant
Gastric fundus	1
Gastric antrum	3
Splenic hilum	5
Aggregate score	9
Right lower quadrant
Appendix	0
Right inguinal ring	1
Aggregate score	1
Left lower quadrant
Sigmoid colon	2
Aggregate score	2
Centre
Anterior abdominal wall	0
Transverse colon	1
Small bowel	0
Aggregate score	1
Pelvis
Bladder	3
Rectum	6
Aggregate score	9

The complexity of organ access was independent of access route, either transgastric or transcolonic, for all organs, except the rectum. For the rectum most of the points were from target failure when the transcolonic route was used. However, analysis of the metrics also showed statistically significant improvement in five variables from the transgastric route, showing that the rectum was a complex organ to reach via both approaches.

Figs 2-4 show the detailed results for one organ in each category: easy (appendix; Fig. 2), intermediate (right inguinal ring; Fig. 3) and complex (rectum; Fig. 4).

Fig. 2.

Comparison of kinematic data obtained with natural orifice transluminal endoscopic surgery with (IR-NOTES) versus without (NOTES) image registration system for navigation to the appendix: a elapsed time to target, b path length to target, c linear jerk, d rotation 1, e rotation 2, f angular jerk 1 and g angular jerk 2. Median values (horizontal line within box), interquartile range (box) and error bars (range) are shown. a P = 0·269, b P = 0·281, c P = 0·684, d P = 0·208, e P = 0·280, f P = 0·280, g P = 0·866 (two-sample t test)

Fig. 4.

Comparison of kinematic data obtained with natural orifice transluminal endoscopic surgery with (IR-NOTES) versus without (NOTES) image registration system for navigation to the rectum: a elapsed time to target, b path length to target, c linear jerk, d rotation 1, e rotation 2, f angular jerk 1 and g angular jerk 2. Median values (horizontal line within box), interquartile range (box) and error bars (range) are shown. a P = 0·002, b P = 0·009, c P = 0·308, d P < 0·001, e P < 0·001, f P = 0·391, g P = 0·011 (two-sample t test)

Fig. 3.

Comparison of kinematic data obtained with natural orifice transluminal endoscopic surgery with (IR-NOTES) versus without (NOTES) image registration system for navigation to the right inguinal ring: a elapsed time to target, b path length to target, c linear jerk, d rotation 1, e rotation 2, f angular jerk 1 and g angular jerk 2. Median values (horizontal line within box), interquartile range (box) and error bars (range) are shown. a P = 0·196, b P = 0·116, c P = 0·293, d P = 0·079, e P = 0·165, f P = 0·319, g P = 0·267 (two-sample t test)

Discussion

The IR-NOTES system provided the operator with an intuitive device to enhance NOTES navigation without requiring previous training or knowledge of the system. Use of standard and commercially available components for the device provided the means for an inexpensive and reproducible system.

IR-NOTES allowed a significant increase in intra-abdominal organ target acquisition for all operators. As expected, the efficacy of IR-NOTES was maximized when targeting difficult quadrants. The organ access complexity score based on the kinematic and efficacy data provided an objective means of identifying these challenging areas. This allowed confirmation of assumptions about these locations based previously on intuitive knowledge^1,8. The left upper quadrant and pelvis were the highest scoring regions, each with 9 points. IR-NOTES provided the most efficient support for some organs in these quadrants, reducing failed visualization of the splenic hilum by more than half, from five of 12 attempts to one of six.

The analysis of different approach routes, transgastric versus transcolonic, also provided an interesting insight; the difficulty in accessing a particular organ depended on the position of the organ itself more than the approach used to access it. An organ that is difficult to access will be hard to access regardless of the route. The spleen and the rectum provided good examples; failure to access the rectum occurred four times through a transcolonic approach, but never through a transgastric one. However, analysis of the kinematic data showed that access to the rectum from the stomach was challenging nonetheless: three kinematic variables were statistically significant when IR-NOTES was compared with NOTES without the image registration system, including a time to target of 54·9 versus 206·3 s (P = 0·027). This provided a more detailed analysis, reflecting the difficulties of accessing challenging intra-abdominal organs.

Beyond its efficacy in achieving organ visualization, IR-NOTES facilitated intra-abdominal navigation as assessed by kinematic analysis. These measures of movement complexity provided an insight into the multiplicity of manoeuvres required for organ acquisition. Although visually unnoticeable to an observer and performed unconsciously by the operator, these movements are largely responsible for the limitations of non-expert endoscopists. As a reflection of this complexity, the distance covered by the endoscope tip to reach the splenic hilum was on average 232 cm, approximately tenfold greater than the actual distance in a straight line. The kinematic results allowed precise analysis of these movements and the impact of the image registration system on them. This analysis demonstrated that in half of the target organs the amplitude of at least one of these fine-tuning movements was statistically significantly reduced by using the IR-NOTES system for target acquisition. This was most noticeable for the splenic hilum where three of seven fine-tuning parameters were statistically significantly reduced when the system was used. IR-NOTES therefore rendered this and other difficult organs accessible even to non-expert operators.

This study was not designed to assess the degree of precision provided by IR-NOTES. However, compared with a standard laparoscopic view available during the study (blinded to the operators), the location provided by IR-NOTES was visually identical to the live laparoscopic image. The exception was the stomach, which had been emptied of food after the preoperative CT image had been acquired. This illustrates a potential limitation of use of the image registration system in NOTES; the IR-NOTES system may lack precision owing to movement of intra-abdominal organs between the time of scan acquisition and the procedure (whether due to respiration or to body movement and position). This limitation had no impact in the present study as the degree of precision required by this experiment was significantly below the threshold created by organ movement, as illustrated by the laparoscopic confirmation of each target acquisition.

Future studies of the system should include an evaluation of the exact degree of precision obtained in the abdominal cavity, but also potential applications to mediastinal or thoracic exploration. Evaluation of the image registration system in a sterile environment would also allow future use in standard laparoscopic surgery to provide virtual 3D visualization. Further miniaturization of the system could also render it applicable to intravascular and other minimally invasive procedures such as intrahepatic tumour chemoembolization. In its current form the system has already been used successfully in endoscopic procedures in humans, specifically in association with EUS for localizing pancreatic lesions¹⁰.

As some of the initial enthusiasm for NOTES subsides¹¹, the human applications of totally NOTES procedures have been limited mostly to a transvaginal approach for cholecystectomy¹². The advantages of NOTES over laparoscopy for simple intra-abdominal procedures need to be well defined for this approach to be used to its full advantage in the appropriate setting¹³.

The indications for future NOTES procedures should take advantage of its intrinsic benefits, such as inherent flexibility and potential to be performed outside of operating rooms. For example, peritoneal exploration is an interesting indication for NOTES^5,14,15 and has led to research investigating the feasibility of transgastric, transvaginal or transcolonic peritoneoscopy¹⁶, its efficacy, and even human studies^1,8,17. The limitations of this approach include the lack of broad overview and ability to exert force and lift organs. Some of the complexities could, however, be easily addressed by the system described here, allowing easier navigation and mapping capabilities to ensure that no area is left unexplored. Benefits would include the possibility of performing bedside diagnostics in intensive care units¹⁸, including those for non-oncological indications.