A Real-Time Endoscope Motion Tracker

Abstract

Objective: In colonoscopy, it is desirable to accurately localize the position of the endoscope’s distal tip. Current tip localization techniques are not sufficient for recording the position and movement of the tip, nor is its rotation measured. We hypothesize that integration of multiple tracking modalities can effectively record the endoscope’s motion in real time and continuously corrects cumulative errors. Methods: A dual modality tracking method is developed to measure the motion of the endoscope’s insertion tube in real time, including insertion length, rotation angle, and their velocities. Optical trackballs were used to measure the endoscope insertion tube’s motion and cameras were used to correct cumulative errors. Results: The accuracy of insertion length and rotational angle were measured. For speeds ≤ 10 mm/s, the median and 90th percentile insertion position errors were 0.88 mm and 2.2 mm, respectively. The insertion position error increases with the speed, reaching a maximum of 10 mm for speeds < 40 mm/s. 11° and 21° were the median and 90th percentile rotation angle errors for angular speeds < 40°/s. Cumulative errors are sufficiently reduced by the imaging modality. Conclusion: The prototype device can precisely measure an unmodified endoscope’s position, rotation, and motion in real time without significant accumulative error. The prototype device is small and compatible with existing commercial endoscopes as an add-on accessory, which could be used for reporting, localizing the lesions in follow up procedures, operational guidance, quality assurance, and training. Clinical and Translational Impact Statement—This preclinical research develops an endoscope tracker that can be integrated into colonoscopy training, automatically record endoscope motion, and be further developed to improve polyp and tumor localization during colonoscopy.

I. Introduction

Colonoscopy is an essential procedure for all colorectal cancer (CRC) screening programs [1], [2], [3], [4]. In the United States, CRC is the third most commonly diagnosed cancer for both men and women with an estimated 151,030 new cases occurring in 2022 [5], [6]. Besides equipment costs, access to qualified endoscopists is a key barrier to rolling out systematic screening world-wide [1], [7]. During screening and surveillance colonoscopy, an endoscope is used to detect and remove precancerous lesions, such as polyps, to prevent them from developing into malignant CRC [8], [9], and large tumors are biopsied for histological testing. Many developed countries have implemented organized CRC screening — initial screening ages vary from 40 to 60 years old — and the outcome data shows that it is effective [1].

In colonoscopy, accurate localization of lesions identified during a procedure is critical for guiding inspection in follow up procedures. Typically, polyp and CRC locations are reported by dividing the colon into segments, and recording the colon segment that includes each site [10], [11]. The scale of colonoscopy’s localization problem is obvious during CRC tumor removal. A recent study found that the tumor location recorded during preoperative colonoscopy was inaccurate for 16.7 % of patients [11] and a meta-analysis of 15 good-quality studies found incorrect tumor localization in 13.7 ± 3.6 % of patients [12]. This location inaccuracy forces changes to the surgical plan for 5.3–11 % of tumors [13], [14], [15].

Currently, the location of abnormal tissue is estimated by the endoscopist based on anatomical landmarks observed in the endoscope images and on the insertion length — which is related to the relative position of the endoscope’s camera. Coarse (every 5 cm) scale markings on the endoscope’s insertion tube can be used to determine the insertion length manually during the procedure when a landmark is identified, or an intervention is performed. This method is time-consuming (relative to the procedure), does not provide rotational information, and is only recorded manually at sporadic locations.

As a result, several endoscope tip location technologies have been developed including commercial products using magnetic sensors (e.g., Olympus ScopeGuide^®) and integration with robotics (e.g., Auris MONARCH^® and Intuitive Surgical ION^®). Most of these technologies are designed to measure the 3D location of the tip — within the patient or within a preoperative computed tomography (CT) volume — instead of insertion length. Yet, due to the mobile and contractile nature of the colon, such methods produce poor results for colonoscopy [16]. The commercial systems are also proprietary to their manufacturers.

Our research objective was to build a device that could automatically record the insertion length and record the endoscopic technique of the endoscopist, particularly during training. As such, in addition to the absolute 3D position of the tip, the back-and-forth motion is also important. Many of the motion tracking methods suffer from accumulative errors, which should be corrected. We hypothesize that a multimodality approach to measure the insertion length and rotation with cumulative error correction will have less error and be more suitable for colonoscopy than existing solutions. Such devices could find applications in many catheter-based endoscopic procedures. For example, it can be used in the upper gastrointestinal tract to measure the extent and progression of Barrett’s esophagus which is a pre-malignant condition that predisposes patients to esophageal cancer.

In this work, we present a real-time endoscope tracker that continuously records the position, rotation, and the velocities, of the endoscope’s insertion tube. The tracker combines optical trackballs that record the endoscope’s motion, with machine vision to correct cumulative errors. The system is designed to be compatible with most existing endoscopes as a low cost, add-on device without the need to modify the endoscope itself. This approach should simplify the regulatory approval process and reduce barriers when translating the technology into clinical practice.

II. Materials and Methods

A. Device Design

The prototype endoscope tracker consists of a sensor cuff as well as software that runs on two Raspberry Pi (RasPi) single-board computers. The sensor cuff holds two trackballs and two cameras that are arranged as shown in Fig. 1. The sensor cuff is designed to be positioned outside the patient (or training phantom) with the endoscope passing through its center. The endoscope is manually inserted through the entrance and exit holes in the cuff. This places its insertion tube between the top and bottom trackballs, and between the cameras at each end of the cuff.

FIGURE 1.

Side view schematic of the sensor cuff and endoscope. Positive position is directed into the page and positive rotation is clockwise.

The endoscope is in direct contact with both trackballs, so they rotate in place as it moves. The endoscope’s position (insertion length) is measured using the trackballs’ y-axes while rotation is measured using their x-axes. Each trackball’s measurement uncertainty is proportional to their rotation in each axis. The resulting uncertainty in the endoscope’s position and rotation can be corrected by using the cameras to detect the scale markings which occur at 5 cm intervals along the insertion tube. The sensor cuff’s two cameras observe the markings from opposite sides of the endoscope to allow its rotation to be corrected every 180°.

The diameters of the entrance and exit and the distances between the two trackballs are customized for a specific type of endoscope based on its shaft diameter. This allows good contact without a significant amount of slipping.

All the endoscope tracker calibration and results presented in this paper used a 1 m long gastroscope (GIF-XQ10, Olympus Canada) but the design can be adjusted to work with different endoscope models that indicate the insertion length with scale markings. We have tested the prototype device on three common, commercial, diameters of endoscope insertion tube shafts with similar results to those reported here. An annotated photograph of the endoscope tracker prototype with the gastroscope inserted is shown in Fig. 2.

FIGURE 2.

Photograph of the endoscope motion tracker prototype with a gastroscope inserted. The endoscope’s position (insertion length) increases when it is moved towards the right as it passes through the sensor cuff.

The two RasPis (Model 3B+, RasPi Trading; Cambridge, England, UK) read the trackballs’ motion, capture and process images, and compute information about the scope’s motion in real time. The tracker’s output can be displayed in a separate monitor (current setup) and/or integrated with the endoscopy controller (future). Each RasPi is connected to a RasPi Camera (Version 1.3, RasPi Trading) and a white light emitting diodes (LEDs) ring light (Pi-Light, Mindsensors.com; Henrico, VA, USA) for illumination. One of the RasPis is designated as the ‘leader’ and the other as the ‘follower’. The leader RasPi is connected to one camera and two USB trackballs. The follower RasPi only controls the second camera. The leader RasPi combines all the sensor data and then calculates the endoscope’s position, rotation, and motion.

Fig. 3 shows the interior of the sensor cuff as well as where the trackballs and cameras are mounted.

FIGURE 3.

Interior photograph of the sensor cuff with the top housing appearing above the bottom housing. The color-coded circles show how the corners correspond to each other and to Fig. 2. The positive position axis for the bottom trackball is upwards in this figure.

1). Trackball Setup

Two laser trackballs (X13, Cursor Controls; Newark, Nottinghamshire, UK) are connected to the leader RasPi and convert the endoscope’s motion into trackball rotation. The trackball’s rotation is measured optically without direct contact between the trackballs and the sensors. Each trackball has a diameter of 12.7 mm and should output 300 ± 10 % counts per revolution in linear mode [17]. This should result in a linear resolution ( ) of

This is equal to approximately 380 counts measured for every 5 cm that the endoscope is inserted.

The angular resolution of the endoscope tracker depends on the endoscope’s insertion tube diameter which is 9.8 mm for the GIF-XQ10. Thus, the angular resolution ( ) is

Each trackball should produce about 231 counts per 360°.

2). Camera Setup

Two cameras are used to detect markings that appear every 5 cm along the endoscope’s insertion tube. The images of these markings are used to correct the accumulative positional and rotational error from the trackball measurements. The cameras have a fixed field of view (FOV) that includes a region of interest (ROI) where the insertion tube appears. A 2.7 cm length of the insertion tube is visible in each image. The leader and follower cameras view opposite sides of the endoscope’s insertion tube as shown in Fig. 4.

FIGURE 4.

Still images taken by the leader and follower cameras. Both images show the white line (green box) marking the 50 cm position. The black gap in the 50 cm line (orange box) is also shown. The gap is located at a positive (CW) angle relative to the center of the leader image. The images also partially show the two ‘50’ number markings on the gastroscope.

We modified the open-source AVA RaspiCam code [18] to configure the RasPi cameras and capture images at 90 frames per second. OpenCV (C++) was used for fast image processing. The settings used for each camera are summarized in Table 1. In addition, the stock focal length of each camera lens is shortened to match the 3.5 cm working distance and a ring light is mounted onto each camera to provide illumination.

TABLE 1. Raspberry Pi Camera Settings.

Camera Parameter	Value
RasPi Camera Version (Sensor)	1.3 (OmniVision OV5647)
Resolution and Frame Rate	@ 90 Hz [19]
Endoscope ROI Size
Rolling Shutter Speed	@ 90 Hz
Greyscale Pixel Range	0–255 (8-bit)
Analog and Digital Gain	4.0 (“ISO 400”)
White Balance Gains	Red (V) = 1.5, Blue (U) = 1
Image Rotation	Leader = 180°, Follower = 0°

Each white line on the gastroscope has one black line gap. The line gaps are all aligned along one side of the insertion tube. The real-time motion tracking program uses the white scale lines to correct accumulating errors in the insertion length and uses the black line gaps to correct rotation angle. Using two cameras, the length position (a.k.a. insertion length) can be corrected every 5 cm and the rotation angle can be corrected every 180°. Fig. 4 shows an example image of a white scale line and the black line gap.

The real-time motion tracking program detects two types of image features: white scale lines and black line gaps. The origin is in the images’ upper left corner. A rotation setting of 180° is used for the leader camera because this aligns the positive image x-axis with the positive endoscope insertion direction. The follower camera does not require rotation. With this setup, image features move right as the endoscope is inserted through the sensor cuff and line gap features move downward during a clockwise (CW) endoscope rotation. Next, both images are cropped to display the endoscope ROI before image features are detected. Fig. 4 shows the cropped images captured and image features detected by the leader and follower cameras.

3). Data Acquisition

The real-time endoscope motion tracker acquires data using a computer program that runs on two RasPis. The leader RasPi is connected to both trackballs and polls them every 8 ms to update the endoscope’s motion. Additionally, both the leader and follower Raspis use their cameras to acquire images and detect image features in real time.

The follower RasPi runs a remote procedure call (RPC) server that passes the image features it detects to the leader RasPi. The leader RasPi polls the follower RasPi for updates at regular intervals. To improve the connection reliability, the pair of RasPis are connected using wired Ethernet.

The leader RasPi combines all the information to update its real-time estimate of the endoscope’s motion. The positional error is corrected each time the center of the white line (image feature) crosses the center of the leader camera’s image in the x-axis. The rotational error is corrected each time the center of the black line gap (image feature) crosses the center of the cropped endoscope ROI in the y-axis for either of the cameras. Thus, the position and rotation can be corrected every 5 cm and 180° respectively.

B. Sensor Calibration and Machine Vision

1). Trackball Calibration

The trackballs were calibrated by recording their output during motions including insertion, removal, CW rotation and counter-CW rotation. This calibration process was completed several times during the tracker’s development. The calibration presented in Table 2 and Table 3 is for the current prototype; earlier prototypes had similar results.

TABLE 2. Endoscope Motion Tracker Position Calibration. All Uncertainty Values are Quoted at One Standard Deviation.

	Bottom Trackball	Top Trackball	Reconciled Position
Counts per ± 50 mm (% Uncertainty)	413 ± 27 (6.5 %)	403 ± 21 (5.1 %)	419 ± 12 (2.9 %)
Linear Distance in mm per Count	0.121 ± 0.008 mm	0.124 ± 0.006 mm	0.119 ± 0.003 mm

TABLE 3. Endoscope Motion Tracker Rotation Calibration. All Uncertainty Values are Quoted at One Standard Deviation.

	Bottom Trackball	Top Trackball	Reconciled Rotation
Counts per ± 360° (% Uncertainty)	264 ± 15 (5.9 %)	261 ± 12 (4.6 %)	286 ± 10 (3.6 %)
Angular Distance in ° per Count	1.37° ± 0.08°	1.38° ± 0.07°	1.26° ± 0.05°

Position calibration was achieved using four insertion motions and four removal motions. All insertion motions started at an insertion length of 20 cm and ended at 100 cm. The GIF-XQ10 endoscope has a maximum insertion length of 102.5 cm. Removal motions were identical but reversed. Ground truth points were manually recorded every ± 5 cm for a total of 16 points during each ± 80 cm motion. Referring to Fig. 2, the starting point, ending point, and ground truth points are recorded when their corresponding white line is aligned with the left edge of the sensor cuff.

The position calibration presented in Table 2 is the combined result for both insertion and removal since there was no significant difference between them. The results include 124 data points (8 motions translations −4) since 4 of the ± 5 cm translations were outliers that did not accurately track the endoscope’s motion.

Table 2 includes distance conversion factors for the bottom trackball, top trackball, and the reconciled position. The top trackball recorded less motion than the bottom trackball for most of the ± 5 cm translations. This occurs because the forces exerted on the two trackballs are different as the endoscope is moved through the sensor cuff. If the endoscope’s friction on a trackball becomes too low, it may fail to record part of the endoscope’s motion.

It was experimentally determined that the most accurate tracking was achieved by periodically comparing the motion measured by the two trackballs — finding the maximum of the two readings — which is used to update the real-time estimate of the endoscope’s motion. This allows the endoscope tracker to be calibrated based on the number of reconciled trackball counts produced during a known calibration motion. This approach is justified since the reconciled position in Table 2 has a smaller relative uncertainty (~3%) than the >5 % for one trackball.

Rotation calibration was achieved by completing one motion of +2880° (8 CW revolutions) and one motion of −2880° (8 counter-CW revolutions). Ground truth points were recorded after each revolution (± 360°) by aligning temporary arrows on the insertion tube and sensor cuff. There was no significant difference between CW and counter-CW rotation. The results are presented in Table 3.

2). Camera Calibration

The leader and follower cameras were calibrated by taking several images while using the camera settings summarized in Table 1. To remove noise, the camera calibration program saves the mean image of a series of 256 images that are cropped to the endoscope ROI. Three main images were acquired during the calibration of each camera: a background reference image, an image showing a white line feature and an image showing a white line feature with its black line gap feature facing the camera.

The background reference image shows the black endoscope insertion tube and the glare from the LED light reflecting off its surface. The glare is similar to the glare that appears in Fig. 4. is captured by inserting the endoscope to a point where no white lines or numbers can be seen in the camera’s FOV inside the sensor cuff. Later, these images are loaded by the endoscope tracker to perform background subtraction during real-time feature detection. This enhances the signal from the white line feature and helps mitigate the glare.

Images showing white line and black line gap features were manually examined. The white line features were selected, and a pixel histogram was used to choose a value of 50 as the threshold to distinguish the white line from the black background ( ). The same threshold of 50 was chosen to detect the edges of the black line gap ( ). A glare threshold ( ) of 180 was chosen to remove glare when its location has changed within the image.

Finally, the process of imaging the features is repeated with the line positioned at various places in the camera’s FOV. This step measures how the white line’s apparent width and the black line gap’s height change as they move within the FOV. The values are used to check if the image features are a valid size to reduce false positives.

3). Camera Feature Detection

The endoscope tracker’s software is multithreaded C++ code and feature detection uses its own dedicated thread. This means that, to achieve real-time performance at 90 Hz, the software must process each image within 11.1 ms. The algorithm is designed to detect valid white line futures and black line gap features if they appear in the camera’s FOV. Recall that black line gap features are always found within a larger white line feature. At this point, image rotation and endoscope ROI cropping have already been completed. The feature detection algorithm has five phases: background subtraction, glare correction, white line detection, black line gap detection, and image feature validation.

Background subtraction is used to enhance the signal from the white scale lines on the endoscope and to correct most of the non-uniformity in illumination across the camera’s FOV. To perform background subtraction, the image pixels ( ) are converted to signed 16-bit integers and then the reference image, , is subtracted from the result. Most of the remaining glare is corrected next by setting pixels with extreme greyscale values to a value of 0.

Next, the algorithm attempts to detect the white line feature if it appears in the image. First, the average of the pixels in each column of the image ( ) is calculated by

Then, the value for each column is compared with the threshold value, , to detect white image features. Finally, a 1-dimensional (1D) distance transform is used to find the center of a white scale line if one is present in the frame and the line’s width is tested to decide if it is valid.

A similar approach is used to detect a possible black line gap feature if a white scale line feature has been found. First, the image is cropped to the white line feature’s ROI. Then, the pixel row average is calculated and compared with , to detect black regions. Finally, a 1D distance transform is used to locate the black line gap and validate its height.

C. Testing Procedure

Many trials were completed to test the performance of the endoscope tracker throughout its development. This section describes the trials run on the current prototype tracker whose results are included in this paper. These trials and their results are an accurate representation of the endoscope trackers’ overall performance. Other trials that have been omitted showed similar results, with very few white line features and black line gap features being missed.

Three position tracking trials were completed to test the endoscope tracker’s positional error and how the error relates to the average velocity during the motion. The positional error is defined as the difference between the measured position and the ground truth position. Each trial includes an 80 cm insertion motion and an 80 cm removal motion. These motions are very similar to the calibration motions except that: the speed of motion was changed between trials (slow, medium, fast), and 15 translations (each ± 5 cm) are recorded by the camera for each motion. One less translation is recorded per motion because the camera corrects error at the center of the box which is 19 mm from the sensor cuff’s left edge.

The position tracking trials’ data can be organized into three groups by considering each translation individually and categorizing them based on their average velocity. Group 1 consists of 31 slow translations with velocities between −10 mm/s and +10 mm/s. Group 2 includes fast withdrawal translations with velocities between −40 mm/s and −10 mm/s, and Group 3 includes fast insertion translations with velocities greater than +10 mm/s.

Two trials were completed to test the device’s rotational error and its ability to measure rotational velocity. The rotational error is defined as the difference between the measured rotation angle and the ground truth. Both trials begin with a motion of +1440° (4 CW rotations) followed by a motion of −1440° (four counter-CW rotations). The motions are half the size compared to during calibration and the two cameras attempt to correct the accumulating error after every ± 180° of rotation.

III. Results

The position tracking results are presented in Fig. 5 and Fig. 6. Fig. 5 shows the results for (a) slow insertion, (b) slow removal, (c) fast insertion, and (d) fast removal. In Fig. 5, the white line image features are defined as the ground truth, and they are successfully detected by the camera every 5 cm to reset the positional error to zero. In the graphs, each 5 cm translation appears as a sloped line and error correction appears as the vertical lines.

FIGURE 5.

Positional error (blue line) and average velocity (orange x) results. The arrows show the direction of insertion or withdrawal in each plot. Zero white line features were missed during these trials.

FIGURE 6.

Positional error is plotted versus average velocity for each translation in Groups 1–3. The piecewise trend line is shown in purple.

The relationship between positional error and average velocity is illustrated in Fig. 5. The positional error is small and distributed around zero for both (a) insertion and (b) withdrawal at speeds less than 10 mm/s. When the speed exceeds 10 mm/s, the positional error usually has the same direction as the direction of travel — i.e. (c) positive during insertion and (d) negative during withdrawal. This shows that the positional error is positively correlated with the average velocity at speeds >10 mm/s.

The same trend can be observed in Fig. 6, which plots positional error versus velocity for all three trials. In Fig. 6, the three groups are clearly separated along the x axis. For Group 1, the slope of the piecewise trendline was set to 0 since the line of best fit through the origin had a slope of −0.05 ± 0.04s (p-value = 18 %). The positional error of the translations in Group 1 is not correlated with their velocities. This shows that the device is well calibrated for speeds under 10 mm/s. Group 2’s trendline passes through the point (−10, 0) and has a slope of 0.46 ± 0.05 s. The trendline for Group 3 passes through (10, 0) and has a slope of 0.61 ± 0.06 s. The quoted uncertainty in the slope for each group’s trendline is the standard error.

Next, the translations in Group 1 were used to calculate the absolute positional error statistics for the endoscope tracker since it is well calibrated for speeds ≤10 mm/s. The median absolute error (AE) for position tracking was 0.88 mm — or 1.8 % of the 50 mm translation. The 10^th percentile AE was 0.19 mm and the 90^th percentile AE was 2.2 mm. The positional error of the faster translations in Group 2 and Group 3 is positively correlated with their velocities. The maximum positional error measured was approximately 10 mm (20 %) at speeds < 40 mm/s. The device did not function perfectly during the rotation tracking trials. It missed detecting the black line gap image features on three occasions.

Fig. 7 presents (a) the CW rotation tracking data from the first trial, and (b) the counter-CW rotation data from the second trial. In Fig. 7, the black line gap image features are defined as the ground truth. In (a), the line gaps are all detected by the camera (every 180°) to reset the rotational error to zero. This means that the line gaps were correctly tracked every time they appeared in front of the leader or follower cameras. The corrected errors are usually about ± 10° and occasionally reach a maximum of ± 30°. In (b), the cameras failed to accurately track the line gap and correct the rotational error on three occasions, at 1420°, 1240°, and 700°. Although the device is not perfect in (b), this situation shows that it can self-correct once the next line gap is detected after a rotation of 360°. The rotational error in this case is still less than 20° over a 360° rotation.

FIGURE 7.

Rotational error and average rotational velocity are plotted versus the rotational ground truth. The arrows show the direction of CW or CCW rotation in each plot. In (a), zero black line gap features were missed. Whereas in (b), three black line gap features were not correctly detected when they crossed the center (vertically) of the leader or follower images.

The device’s AE statistics for rotation tracking were calculated based on 18 segments, each 180° in size, collected during the two trials. The median AE for rotation was 11° (6.8 %). The 10^th and 90^th percentile AE were 1.8° and 21°, respectively. The device was tested at angular speeds < 40 °/s.

IV. Discussion

The goal of this work is to develop an endoscope tracker capable of measuring the tube’s insertion length (position) and orientation (rotation) as well as their motion in real time. Such information can be recorded continuously and correlated with the video, which may improve procedure documentation, relocalization of biopsy sites and lesions, and navigation. Such technology can also be used in physician training as well as quality assurance.

Our endoscope tracker uses a dual modality approach with trackballs that measure the movement of the endoscope’s tube shaft while cameras image the scale markings to correct accumulative error. Our prototype device achieved a median AE of 0.88 mm for position and 11° for rotation with a 90^th percentile AE of 2.2 mm and 21°. Our results are encouraging because a recent paper that localized points within a colon phantom using electromagnetic tracking (EMT) and computed tomography (CT) had a greater than 10 mm position error for ~8 out of 12 points [16]. Before discussing our results in more detail, we must first explain how existing solutions localize the endoscope’s distal tip.

Although locating an endoscope’s distal tip is a common problem in endoscopy, the methods used vary dramatically depending on the application [16], [20], [21], [22], [23]. A number of solutions exist for measuring the location of an endoscope’s distal tip. External imaging modalities including ultrasound [21], x-ray fluoroscopy [22], and CT [16], [20], [23] have been reported. These modalities generally add significant cost to the procedure whilst fluoroscopy and CT may carry additional radiation risk.

A large study of colonoscopy outcomes found that the prevalence of large (> 9 mm) tumors/polyps in average risk screening was 6.6% [24]. Thus, while bronchoscopic lung cancer removal can justify a CT scan, a CT is unlikely to benefit patients undergoing screening or surveillance colonoscopy.

EMT methods [16], and hybrid methods that combine EMT with endoscopic video or data from an inertial measurement unit (IMU) [20], [25], also exist. Pentax and Olympus offer EMT products to detect coils, integrated at regular intervals along the endoscope’s insertion tube, and locate them in 3D space. The coils provide a basis for showing a 3D image of the insertion tube and any loops or bends that may have formed in the colon. The information about loops and bends is useful to navigate the endoscope inside the colon, particularly in training. EMT typically requires modifying the endoscope itself (i.e., buying a new scope or accessory) and each manufacturer uses their own proprietary technology [12], [26], [27].

Moreover, these techniques primarily focus on measuring the 3D absolute position of the endoscope tip. While such information is important, the ability of these techniques to account for insertion length and orientation of the lesion from the orifice is limited. In endoscopy, particularly in the colon, the absolute 3D position of the colon (and its segments) changes inside the abdominal cavity [16]. For example, even during the same procedure, when a patient changes their body position, the 3D position of the scope tip changes with the colon movement, while the tip may not move inside the colon itself. Likewise, such 3D positions are not useful when compared to follow up procedures.

Finally, wireless capsule endoscopes can be tracked by combining IMU data with cameras that use visual odometry to estimate the distance traveled [28]. However, the accumulative error is significant and requires an external device to provide correction.

Our hypothesis was that the dual modality approach to locating the endoscope’s tip would have less error and be more suitable for colonoscopy than existing solutions. The prototype endoscope tracker directly measures the motion of the endoscope’s insertion tube outside the patient, which is correlated with the insertion and rotational motion of its distal tip. This correlation should occur for the majority of commercial gastro-intestinal endoscopes because they have a rotationally rigid insertion tube with a fixed length.

Existing solutions are primarily focused on localizing the endoscope’s tip in an absolute 3D space with little consideration of their relative position in the colon. A hybrid tracking method for bronchoscopy — using CT, EMT, and endoscopic video — achieved errors of 2.5 mm and 4.7° [20], which is sufficient in clinical practice. Our device has 90^th percentile error of 2.2 mm and 21°. The positional accuracy is similar, and the rotational error is adequate for colonoscopy. For example, if a polyp’s location in an image is described using a 12-hour clock face, then the 20° precision of this device is lower than the typical 1hr/30° error used in clinical practice.

While the prototype can be used on its own, it also can be combined with other tracking solutions to create a more precise hybrid solution that measures the endoscope’s motion both outside the patient and at its distal tip camera.

The rotation (or orientation) of the endoscope tip provides additional location information of the biopsy/lesion sites and is useful for detecting and/or avoiding looping [29] as well as improve navigation. Current commercial 3D positioning techniques are not able to measure rotation. Bernhardt et al, reported rotational movement tracking with an intraoperative CT image [23]. For most endoscopic procedures, radiation from CT is to be avoided if possible. Other previously reported techniques used endoscopic video [16], [23] and IMU motion sensors [16], [25], [29]. These approaches are limited to bronchoscopy applications where the endoscopic images have significant more landmark features than colonoscopy [16] or have significant more accumulative error [25], [29].

Second, our results (e.g., Fig. 5 & Fig. 6) also demonstrated that the insertion velocity can be calculated and recorded from the real time insertion length measurement. Although the insertion speed is generally not measured in clinical practices, it is an important parameter in training new endoscopists, as well as for quality assurance. Such applications generally do not require regulatory approval hence opening faster translation pathways.

Third, gastrointestinal endoscopy screening and surveillance, especially colonoscopy, is a whole population scale program with high volume. Both cost in equipment/facilities and access to qualified endoscopists should be considered when new technologies are developed [7]. The reported endoscope tracker is designed to be an add-on auxiliary device that is compatible with most endoscopes on the market. This means that it can be used with existing endoscopes after going through a separate regulatory approval process. This add-on approach could significantly reduce the barriers and cost in technology translation.

V. Conclusion

Colonoscopy is the gold standard for detecting CRC. However, the tumor location (colon segment) reported by preoperative colonoscopy is still inaccurate for 10–17 % of patients [12]. It is currently a challenge for endoscopists to accurately record the location of tumors and polyps found during colonoscopy. Colon polyp surveillance monitors patients with a high risk of developing polyps using periodic colonoscopies. Improved polyp localization would allow sites of polyp removal to be examined for recurrent polyps during a subsequent colonoscopy.

In clinical practice, the insertion and rotation speeds vary significantly between different phases of the endoscopy procedure and vary even more between different endoscopists. However, these observations are anecdotal and qualitative since there is no existing technology that can objectively measure the speed of such motion.

In collaboration with an expert gastroenterologist, we have developed a real-time endoscope motion tracker. The externally mounted device records the motion of the endoscope in real time with the goal of improving lesion localization accuracy during, for example, preoperative colonoscopy. The device uses trackballs to measure the endoscope’s position and rotation, and camera-based feature detection to periodically correct the accumulated uncertainty in those measurements. The related velocity is calculated each time camera-based feature detection corrects the endoscope’s position or rotation. The device precisely measures the motion of the endoscope’s shaft with a median AE of 0.88 mm for position and 11° for rotation. The real-time motion records produced by the endoscope tracker can be a valuable tool for both clinical endoscopy procedures as well as answering some research questions. For example, it can be synchronized with the video recording and allow more precise recording of the locations of lesions during colonoscopy. The recorded position and tip orientation (rotation) can be combined with the video to create a 3D reconstruction of the colon or an unwrapped map of its luminal surface [30], [31]. The technology can also provide an objective and quantitative measure of endoscopic procedures for new residents undergoing training.

The motion tracker can still be improved. The trackballs are in direct contact with the endoscope and so the tracker must be thoroughly cleaned and sanitized after it is used. A non-contact, fully camera-based tracker could remove the sanitization requirement and is worth exploring in the future. Such work is ongoing by our group and others.

Funding Statement

The work was supported in part by the Canada Cancer Society Research Institute under Grant 701734 and the Natural Sciences and Engineering Research Council of Canada (NSERC) under Grant RGPIN-2019-07127 and an Engage Grant where MDA is a partner.