Accurate three-dimensional virtual reconstruction of surgical field using calibrated trajectories of an image-guided medical robot

Abstract.

Brain tumor margin removal is challenging because diseased tissue is often visually indistinguishable from healthy tissue. Leaving residual tumor leads to decreased survival, and removing normal tissue causes life-long neurological deficits. Thus, a surgical robotics system with a high degree of dexterity, accurate navigation, and highly precise resection is an ideal candidate for image-guided removal of fluorescently labeled brain tumor cells. To image, we developed a scanning fiber endoscope (SFE) which acquires concurrent reflectance and fluorescence wide-field images at a high resolution. This miniature flexible endoscope was affixed to the arm of a RAVEN II surgical robot providing programmable motion with feedback control using stereo-pair surveillance cameras. To verify the accuracy of the three-dimensional (3-D) reconstructed surgical field, a multimodal physical-sized model of debulked brain tumor was used to obtain the 3-D locations of residual tumor for robotic path planning to remove fluorescent cells. Such reconstruction is repeated intraoperatively during margin clean-up so the algorithm efficiency and accuracy are important to the robotically assisted surgery. Experimental results indicate that the time for creating this 3-D surface can be reduced to one-third by using known trajectories of a robot arm, and the error from the reconstructed phantom is within 0.67 mm in average compared to the model design.

1. Introduction

Complete surgical resection of tumor tissue remains one of the most important factors for survival in patients with cancer.¹ Tumor resection in the brain is exceptionally difficult because leaving residual tumor leads to recurrence and decreased survival, and removing healthy brain tissue can impair patients’ memory, mobility, and other vital functions. Unfortunately, tumor margins are often indistinct under direct (optical) visualization. Although MRI and CT imaging can locate tumors, their use intraoperatively during surgery is very difficult and costly.^2,3 Ultrasound imaging has a low resolution, requires tissue contact, and lacks tissue-specific contrast.^4,5 Optical imaging provides a high-resolution noncontact imaging that is familiar to the surgeon, but the penetration depth is limited. To enhance the optical contrast during intraoperative tumor removal, fluorescence contrast is often used to label brain tumor margins at high resolution and contrast.^6,7,8 This technique is especially useful when brain tumors and their margins are exposed to the surgeon, where the inherent shallow penetration depth (in white brain matter) of visible (0.5 to 0.83 mm) and near-infrared (NIR) light (0.9–1.0 mm)⁹ is not limiting. The mechanisms for providing fluorescence contrast can range from increased leakiness of the capillaries feeding a tumor,^10,11 to photodynamic therapy sensitizers accumulating in tumor tissue,¹² and to molecularly specific fluorescence probes.^13,14

The devices used to image fluorescently labeled brain tumors range from large surgical microscopes,¹⁵ to light-emitting diode (LED) flashing illumination with large camera detection,¹⁶ to hand-held cameras systems,¹⁷ and to goggles worn by the surgeon.¹⁸ The main criterion is having a wide-field video imaging that can provide both reflectance and tumor-specific fluorescence image contrast. Another key criterion is the ability to detect at a very high sensitivity, typically down to nanoMolar concentrations of fluorescence dye. One more key criterion is the capability to use biocompatible dyes for translation to human clinical procedures, which limits the choice of visible-NIR dyes to fluorescein and simple derivatives, 5-ALA and other chemical precursors to PpIX,¹⁹ methylene blue, and derivatives of interactive computer graphics (ICG).⁷ In the future, a criterion will be flexibility and small size to enable brain surgeries through less-invasive keyhole openings in the skull.²⁰ All of these criteria are met by a new forward viewing, laser based, flexible endoscope, called the scanning fiber endoscope (SFE).²¹

1.1. Scanning Fiber Endoscope for Multimodal Imaging

The SFE attains an unconventionally high resolution for a flexible endoscope of only 1.2 mm in diameter (Fig. 1). Instead of using a single optical fiber or a charge-coupled device sensor element for each pixel in the endoscope image, the SFE generates an image from scanning a single beam of red (R), green (G), and blue (B) laser light from a submillimeter fiber scanner. The fiber scanner consists of a cantilevered optical fiber driven near its mechanical resonance of lateral vibration (12 KHz) by a 0.4-mm tubular piezoelectric actuator. This spiral-scanned illumination fiber is single mode for deep-red to blue laser wavelengths. By collecting the laser reflectance backscattered from the tissue, full-color reflectance images are formed; by collecting wavelength-shifted fluorescence emission, multiple channels of wide-field fluorescence images are formed. To create video rate imaging, the collected light is concurrently sampled at 25 MHz on 4 channels to form $>500$-line images at 30 fps.²¹ By collecting light nonconfocally, such as through a ring of multimode optical fibers (0.6 numerical aperture) surrounding the lensed fiber scanner, a 2-mm diameter scope is constructed with 5 to 10 nM fluorescence dye detection sensitivity to sodium fluorescein and Cy5.5 at a typical working distance of 2 cm.⁸ Simultaneous reflectance and fluorescence imaging have been demonstrated using two techniques, (1) by replacing one reflectance channel (e.g., green) for fluorescence, while providing color reflectance imaging from the other two reflectance channels (e.g., red and blue), fluorescence-guided procedures can be conducted in vitro.^22–24 (2) A new multispectral fluorescence SFE was constructed with RGB lasers exciting three separate fluorescence channels, and one reflectance channel provides a grayscale reflectance imaging for fluorescence-guided biopsy.⁸ Furthermore, this grayscale reflectance channel can be used to provide distance compensation to the fluorescence imaging channels, which allows a more quantitative fluorescence imaging feature of the SFE.²² The multiple fluorescence channels also allow autofluorescence to be measured at every pixel and eliminated from the fluorescence image, providing a more quantitative fluorescence image, especially at the low dye concentrations that will be encountered at the margins of fluorescence-guided brain surgery.⁸

Fig. 1.

The schematic diagram of the scanning fiber endoscope (SFE). A white optical beam consisting of RGB laser light is being scanned by a single-mode optical fiber in a spiral pattern and focused onto an object at low optical power by 2 or 3 lenses. The backscattered and fluorescence light is collected by a ring of multimode optical fibers that are wrapped around the 1-mm diameter fiber scanner assembly. Color video stream is generated with $>500$ line resolution at 30 Hz frame rate.

1.2. Robotic Image-Guided Interventions

With the characteristics of flexibility and ultrathin size, the SFE is advantageous as a guidance scope in robotic image-guided interventions (IGI), which allow for less pain and smaller incisions, quicker recovery, and fewer side effects.²⁵ Since the SFE is so small, lightweight, and flexible, the scope can be attached to a larger surgical tool which can be used for delicate microsurgeries. Thus, the SFE affords a new concept in IGI which is placing a forward-viewing eye directly on a tool rather than the traditional method of putting the tools through a much larger endoscope for surgical procedures such as NOTES^26,27 (natural orifice transluminal endoscopic surgery). Flexible endoscopes can be used to inspect the internal surgical surface through a keyhole,²⁸ but the drawback of conventional endoscopic surgery is the lack of depth information,²⁹ which becomes a highly significant barrier to robotically assisted brain surgery. This impaired depth perception issue is overcome in the central field of view (FOV) by stereopsis, such as within daVinci® surgical robot series.^30–32 But the rigid laparoscopes on commercial daVinci® systems are restrictive to straight lines of sight and relatively large surgical fields. Another disadvantage of endoscopic surgery is the inability of providing the surgeon with an overview of the entire surgical field in three-dimensions (3-D) due to the limited FOV of the endoscope. Surgeons and their assistants must redraw a mental 3-D mosaic of the surgical field as they move the endoscope within the larger surgical field.³³ Thus, 3-D panorama reconstruction software of the surgical field is demanded, which can also facilitate the autonomous surgery robot to better control the process of image-guided tasks, such as tumor margin removal.³⁴

1.3. Three-Dimensional Virtual Panoramic Views from Endoscopic Video

To overcome the limitations of lack of depth and limited FOV, 3-D reconstruction of an endoscopic video has been implemented using the computer vision algorithms of structure from motion. Typically, images are stitched into a panoramic image that simply extends the FOV laterally without providing depth information.^35,36 Previous studies achieved 3-D reconstruction of a whole bladder from an endoscopic surveillance video manually moved or with the assistance of simple robotics.^24,37,38 Basically, the algorithm applied scale-invariant feature transform (SIFT) to find feature points from SFE images, random sample consensus (RANSAC) to figure out matching frames, and bundle adjustment to create a 3-D panoramic surface by simultaneously refining the positions and orientations of the camera (SFE) and the 3-D coordinates of features. Finally, a thin plate spline algorithm was applied to generate a smooth 3-D surface by fitting these feature points.

When applied to surgical robotics, the position of a robot tool can be tracked and recorded within the robot’s motion control software.^39–41 By placing SFE directly on a surgical tool, the poses of SFEs (camera) can be obtained from the robot system. Unlike previous studies that were solely based on images, this 3-D model can be generated in a more efficient and accurate way by simplifying the procedure of bundle adjustment.^42,43

1.4. Novel System using Surgical Robot RAVEN II

In our study, the SFE imaging system was directly attached to one robotic arm of RAVEN II⁴⁴ as an eye on the tool to guide the removal of residual tumor during simulated robotically assisted brain surgery. The RAVEN II surgical robot is an open source hardware and software robotic system for research on robotic surgical assistants. RAVEN was designed for use in either laparoscopy or open surgery. The robot consists of two arms which can be tele-operated separately by the user via a remote master. Each arm has seven degrees of freedom providing a very high degree of dexterity, precise cutting, and accurate navigation with surgeon-assisted visual feedback.⁴⁵

The goal of this study is to merge three different technologies, performing 3-D reconstruction intraoperatively from two-dimensional (2-D) SFE multimodal images on the RAVEN platform as an image-guided surgical robotics system in brain tumor margin removal surgery. By knowing all the SFE poses, this software will generate a highly efficient and accurate real-scale multimodal 3-D model of surgical field that can be rapidly generated and repeated during a surgical procedure. Creating this 3-D reconstruction of the surgical field in real time is advantageous to robotically assisted brain surgery due to tissue deformation and shifting that occurs during surgical resection,⁴⁶ caused by gravity, cerebrospinal fluid, hyperosmotic drugs and etc.,^47–49 and, of course, tissue removal.

In the preliminary study,⁴³ the experiment on the micropositioning stage provided the feasibility of our proposed algorithm development for reconstructing the surgical field in 3-D. This work was extended onto the RAVEN II surgical robotic research platform to validate the feasibility of our proposed image-guided surgical robot system that generates and later relies on the 3-D surface reconstruction of the surgical field. The purpose of this work is to validate the accuracy of this 3-D reconstruction model using calibrated trajectories of the robotically moved endoscope to the original 3-D phantom. Thus, in this study, an accurate-shape, near-realistic phantom was predesigned and 3-D printed to mimic the debulked brain tumor without matching the optical properties of the brain tissue other than surface color and target fluorescence.

The RAVEN has been designed only for a visual guidance control and not autonomous computer control. The RAVEN uses a cable-driven mechanism. Thus, programmable positioning errors are unpreventable and time variant, so that the direct positioning control of the robot under the autonomous motion mode is inherently imprecise.⁵⁰ This unreliability of the camera pose information from the RAVEN system readings leads to unsuccessful 3-D reconstruction.⁴³ This paper will show that an external optical tracking system for the RAVEN provides the needed camera pose information; therefore, the two major goals of this project can be attained. These two goals are demonstrating (1) accurate 3-D virtual reconstruction of the geometry of the surgical field phantom with fluorescence features and (2) algorithmic approach to providing these reconstructions at near-real time for repeated implementation in a robotic-assisted surgical procedure.

The paper is structured as follows. In Sec. 2, a new approach is proposed to generate a multimodal 3-D mosaic with higher algorithm efficiency and robustness using a calibrated robot arm. In Sec. 3, an experiment of a scanning near-realistic phantom of a debulked brain tumor cavity with SFE on the RAVEN II robot platform is described. Section 4 gives the 3-D surface reconstruction results of the experiment, analyzing speed and accuracy. Section 5 discusses the advantages and disadvantages of this 3-D surface mosaic software with feature position prediction and the new stereo surveillance system of RAVEN II. Implementation of the demonstrated goals in a more minimally invasive environment with real-time computer processing is considered as future work.

2. Method

2.1. Multimodal Scanning Fiber Endoscope

Endoscopic visualization in brain tumor removal is challenging because tumor tissue is often visually indistinguishable from healthy tissue. Fluorescence biomarkers provide an opportunity to improve tumor delineation. In this study, a multimodal SFE imaging system was used to capture reflectance and fluorescence images. Laser light combined with red (635 nm), green (532 nm), and blue (444 nm) was projected collectively or selectively at the proximal end, transmitted to the distal end using a single-mode optical fiber, and scanned to illuminate the surgical field. The RGB laser light has a full width at half maximum of 70 to $90\mu \mathrm{m}$ for the standard imaging distance of 5 mm.⁵¹ The reflected RGB signals are simultaneously captured by a ring of multimodal returning fibers and colors separated by filtering, then separately amplified by three photomultiplier tubes (PMTs). In the standard reflectance imaging mode, the intensity of RGB laser source was set to generate white light, see Fig. 2(a). Typically, the fluorescence biomarkers are not visible in the reflectance image since the emission signal is very weak.

Fig. 2.

Reflectance and multimodal (reflectance and fluorescence) imaging diagrams for the SFE using a green fluorescence biomarker. (a) Standard SFE RGB imaging. Each RGB laser source is filtered and amplified using a color-specific photomultiplier tube (PMT) channel; (b) SFE fluorescence imaging, with green and red laser sources inactive. Reflectance images are formed using blue laser illumination only, while fluorescence is detected within the green color channel.

Since our example fluorescence targets emit green light under blue laser excitation, the standard green reception channel is used to detect the fluorescence emission signals after the green laser source was powered down. In this study, we turned off the red and green laser sources. The captured fluorescence image only contains the signal from the green channel, see Fig. 2(b). Compared with reflectance imaging, fluorescence imaging channel is sensitive to the background light in the filter channel bandpass. In our study, reflectance image was recorded for each pose of the SFE; therefore, it was the corresponding fluorescence image. The fluorescence emission signal could be all black when all tumor tissues are removed. Thus, reflectance images were used to reconstruct the 3-D surface of the debulked tumor phantom since they contain more features.²⁴ The corresponding fluorescence images were mapped to the 3-D model to improve visualization; also the locations of tumors in the 3-D space could be read out for robotic path planning.

2.2. Three-Dimensional Reconstruction Algorithm with Feature Position Prediction

In previous studies, a surface mosaic software was developed to produce a 3-D internal panoramic surface solely based on a bladder surveillance video.^37,38 By attaching SFE directly to an accurate RAVEN robot arm, the positions and orientation of SFE can be read out from the RAVEN system. The efficiency and accuracy of the 3-D reconstruction can be improved by applying these values of SFE pose as preknowledge.⁴² To take advantage of the programmable surgical robot RAVEN, we set the distances of each neighboring SFE position as equal in this study to boost the efficiency of the reconstruction algorithm.

With equidistant SFE positions, the homography of one pair of neighbor SFE frames can be estimated and would then be applied to other pairs of neighbor frames to approximately predict the positions of feature points; we call this feature position prediction in this paper. Thus, instead of searching the whole image to find the matching feature, we reduce our search to a small region around the prediction position, which makes the algorithm more efficient.

2.3. Stereo System on RAVEN II Platform

Numerous tests of RAVEN accuracy were carried out by sending commands of different pure translations in the $x$, $y$, $z$ directions, and compared the deviation between command and real movement. The conclusion was drawn that the accuracy of RAVEN was changing based on different cases, such as the operation time and initial location of the RAVEN arm.

To address this inaccuracy issue, one possible solution includes adding an external tracking system to obtain the actual pose information of the robot end-effector. Previous tracking systems are optical-based systems which are sensitive to the lighting condition. For example, Kehoe et al.⁵⁰ used two high resolution industrial cameras for the construction of the stereo tracking system in their autonomous multilateral debridement task on the RAVEN surgical research platform. Mahler et al.⁵² used a PhaseSpace LED-based motion capture that consists of four cameras to record the true pose of the robot end-effector. These trackers can provide accurate pose information of the robot arm but are expensive. An alternative solution to improve the control precision of a cable-driven robot is to install encoders for each joint directly in the joint space. In this case, the encoders reflect the true joint angles, but the robot could become very bulky due to installation limitations and it might be difficult to sterilize due to electric cable wiring.

In our study, a custom optical tracking system was utilized to track color markers in a pair of stereo images which were provided by two low-cost Logitech QuickCam® Communicate Delux webcams. Both webcams were snapped into a 3-D-printed base that can be mounted on an aluminum bar with an adjustable base distance as shown in Fig. 3(a). The optical axes of the two webcams were approximately aligned parallel in this setup to provide a reasonable FOV based on the robot working area. The standard USB video class driver was used for image streaming. The synchronization of the two webcams was implemented on the software level using the approximate time ROS message filter. The openCV library was utilized for image processing to segment the markers from the image. For the purpose of robust tracking, a Kalman filter with four internal states was implemented for de-noising in each image frame. The workflow of the stereo tracking system is illustrated in Fig. 3(b). The average performance of the stereo tracking system using a $4\times 4\mathrm{mm}$ color marker in a $20\times 15\mathrm{mm}$ workspace is listed in Table 1. The accuracy of the $x$ and $y$ directions is $\pm 0.2\mathrm{mm}$, which is higher than the accuracy in the $z$ direction.

Fig. 3.

The stereo system. (a) The hardware of stereo system with two webcams mounting on an aluminum bar. (b) The workflow of the stereo tracking algorithm; a Kalman filter was utilized for robust tracking.

Table 1.

The performance evaluation of the custom optical tracking system.

Camera resolution:	$1280\times 960\text{pixels}$
Tracking frequency	7 Hz
Accuracy	$x$ axis: $\pm 0.2\mathrm{mm}$$y$ axis: $\pm 0.2\mathrm{mm}$$z$ axis: $\pm 0.5\mathrm{mm}$

3. Experiment

In the preliminary study,⁴³ the 3-D reconstruction of a debulked brain tumor phantom was generated with a known SFE (camera) position and orientation on the micropositioning stage. The reconstruction algorithm with feature position prediction took one-third of the computational time of a conventional algorithm. The general shape of the 3-D virtual reconstruction appears similar to the phantom, demonstrating the feasibility and reliability of the new algorithm, but there was no dimensional comparative analysis between the real phantom and the virtual 3-D model. To accurately navigate the robot and resect tumors, quantification was analyzed in this paper with an iterative closest points (ICP) algorithm.

All the codes were MATLAB® implemented, and were run on a workstation Dell Precision M4700 with 2.7 GHz Intel i7-3740QM CPUs, 20.0 GB memory in a 64-bit windows operating system.

3.1. Phantom Fabrication

A cup-shaped latex phantom ($3\times 4\mathrm{cm}$) was manually made to mimic the debulked tumor cavity in the preliminary study.⁴³ With an irregular shape and multiple scattered weak fluorescence spots to imitate the practical surgical field, it is representative of what a surgeon would typically encounter. Different from the previous phantom, a new phantom was made to validate and quantify the reconstruction results with computer-aided design (CAD) designed geometry and to provide a more realistic coloration.

In this study, a 3-D model of the spherical dome was designed using OpenSCAD with a maximum radius of 17.5 mm and a depth of 10 mm, see Fig. 4(a). This designed model was saved as a STereoLithography (.stl) file and read by the 3-D printer MakerBot Replicator (MakerBot Industries, Brooklyn, New York). The phantom Fig. 4(b) was printed using PLA (Polylactic Acid) material. In order to make a phantom for dimensional tests that also allowed multiple image contrast enhancement on the surface of the 3-D surface, a near-realistic phantom was made without adding the complexity of light penetration deeply into the substrate. Specifically, colored paper from a photograph of actual brain surgery⁵³ (featuring cortex, blood vessels and even electrocorticography numbered electrodes) was glued over the PLA substrate, see Fig. 4(c). This limited the laser light penetration to approximate the thickness of the colored paper and reduced the uncertainty in our 3-D measurement.

Fig. 4.

Phantom fabrication. (a) The CAD-designed phantom model of a spherical dome with radium 17.5 mm and depth 10 mm. (b) The three-dimensional (3-D)-printed model of debulked brain tumor cavity with 4.0-cm square PLA material. (c) The phantom with near-realistic surgical features and multiple scattered fluorescence targets representing the tumor residual.

Also for quantification purposes, eight fluorescence targets of known shapes and sizes were scattered on the surface of the spherical dome phantom, representing the residual tumor at the margins. These targets are fabricated and cropped from a dye-in-polymer material that is formed by dissolving Fluorol dye (Exciton Inc., Dayton, Ohio) powder into polyurethane resin (AquaClear Resin, ArtMolds, Summit, New Jersey).²² By illumination with a 444-nm laser (blue), these fluorescence targets emit green light but are nearly clear and colorless in RGB reflectance imaging.

3.2. Experiment Setup and Procedure

In the experiment, SFE was affixed onto one robotic arm and pointed downward toward the tumor phantom that was placed on the horizontal workspace of RAVEN, as shown in Fig. 5(a). A triangular marker plate with three distinguished color dots in each corner was attached to the SFE probe, see Fig. 5(b). The triangular height of the marker plate was approximately aligned with the optical axis of the SFE to simplify the transform estimation between the marker and the SFE. The end-effector of the robot arm was programmed to automatically travel along a 2-D $5\times 6$ grid map parallel to the workspace. All grid nodes were equally spaced at a distance of 5 mm. The SFE has a depth of focus of 4 to 40 mm that matches conventional flexible cystoscopies.²⁴ A bright image with plenty fine details can be captured when the SFE is close to the object; a relatively dark image within a larger scene is obtained as the SFE moves further away from the object. In this experiment, the front end of the SFE is located about 15 mm away from the tumor phantom, where we can capture images with sufficient features and also guarantee more than 50% image overlap among neighbor frames for the reconstruction algorithm.

Fig. 5.

The 3-D reconstruction experiment with SFE and RAVEN II system. (a) The experiment setup. RAVEN arm would hold SFE to scan the tumor phantom along planned trajectory with stereo system as closed-loop feedback. (b) Front and back view of the layout among SFE, marker plate and spherical-dome-shaped phantom; the SFE was aligned with the triangle height to simplify the transformation matrix between SFE and marker frames.

With the stereo system as a closed-loop feedback, the RAVEN arm can move along the grid nodes very accurately. At each grid node, reflectance and fluorescence images were taken, respectively. Since the fluorescence signal in vivo is expected to be weak from the residual tumor tissue, the SFE is expected to remain stationary at these preprogrammed locations of the robotic arm to increase the signal to noise by video-frame averaging. Meanwhile, the locations of the colorful markers were recorded for 3 s. The average value of each location was calculated to minimize the noise issue of the stereo tracking system, so that a better estimation of the SFE pose (position and orientation) can be achieved.

3.3. Multimodal Imaging

As fluorescence imaging can enhance tumor delineation, a flexible, wide-field calibrated multimodal SFE of 1.6 mm outer diameter was utilized to capture both reflectance and fluorescence images in real time at a high resolution. Moreover, the reflectance contrast of the vascularity can be increased with the enhanced spectral imaging (ESI) mode of the SFE scope by enhancing the blue channel over the red channel. This shares the same concept as the narrow band imaging⁵⁴ that is widely used to improve the visibility of blood vessels. The 1.6-mm SFE has larger multimode optical fibers surrounding the fiber scanner to improve fluorescence sensitivity compared to the 1.2-mm SFE. As shown in Figs. 6(a) to 6(c), there are many more features in the reflectance and ESI images versus the fluorescence images. A grayscale threshold was set to highlight the residual tumor based on a pixel grayscale value in the fluorescence image. The white points/regions represent tumor in Fig. 6(d), which may look noisy due to the high PMT gain to capture the weak fluorescence signal. The tumor region was filtered and segmented with an image morphology technique of image opening and closing, as shown in Fig. 6(e). A multimodal image can be generated by merging the reflectance with a green fluorescence overlay, see Fig. 6(f).

Fig. 6.

The reflectance and fluorescence SFE image of the tumor phantom. (a) The standard reflectance imaging of SFE; (b) the enhanced spectral imaging of SFE; (c) the fluorescence imaging by deactivating red and green laser sources. The fluorescence targets represent tumor residual, emitting fluorescence signals (green) under blue laser activation; (d) the white points or regions represent tumor cells by filtering with a specific grayscale threshold; (e) the segmentation of tumor regions with image morphology techniques; (f) multimodal 2-D image that is generated by merging color reflectance and fluorescence. The blue curvatures represent the outlines of tumors.

3.4. Three-Dimensional Reconstruction

In the first step of 3-D reconstruction, SIFT features were detected and characterized with an open source VLFeat toolbox.⁵⁵ In each pair of frames I and II, shown in Fig. 7, the feature space distances were calculated for each feature of frame I to every single feature of frame II in a conventional algorithm and the features of frame II with the nearest and the second nearest distances were determined. To maintain good (real) matches, the ratio of nearest and second nearest distances was preserved to be less than a threshold.³⁷ We set 0.8 as the threshold in this study to eliminate unreliable matching.

Fig. 7.

Matching features in a pair of frames I and II were detected with and without feature position prediction and lined up with colorful lines. (a) Fewer matching pairs were found by the conventional algorithm. (b) More matches were detected with the prediction of feature points.

With accurate position and orientation of the SFE, the homography between image pairs can be estimated. Thus, the positions of the matching features can be roughly predicted. A search window was set around the prediction position so that the real matching feature can be detected in a more robust and more efficient way compared with the previous algorithm. A search window with a radium of 100 pixels was applied in our study.

In addition to the improvement of computational efficiency, our new algorithm was advantageous to the robustness of the reconstruction, especially for insufficient image overlap ($<50\%$). Considering that an image contains several similar features, the ratio of the nearest and second nearest distances may not satisfy the ratio threshold. By searching a small region around the feature prediction position, we can reduce the interference from similar features that are located in other parts of the image. In this way, our new algorithm can determine more matches than the previous work.³⁷ Figure 7 shows the matching feature pairs with/without a feature position prediction approach. Taking the two images in Fig. 7 as an example, 216 pairs of matching points were found by the conventional method, Fig. 7(a), while 278 pairs were detected with the feature position prediction, see Fig 7(b). One-twentieth of the matching points were randomly picked and lined up for better illustration in Fig. 7.

Bundle adjustment was implemented after matching the feature points. By giving the known positions and orientation of the endoscope for each frame, bundle adjustment greatly simplifies the computations by only solving the 3-D coordinates of each feature. More robust matching features were provided by narrowing down the search region. These improvements make the 3-D reconstruction algorithm more efficient and robust.

4. Result

The 3-D panoramic mapping procedure of the spherical dome fluorescence tumor phantom is illustrated in Fig. 8. Figure 8(a) represents the reconstructed 3-D map of the phantom with 30 positions $5\times 6$ of the SFE (camera) above and the features as blue points used for mosaicking the images together into the panorama. Figure 8(b) shows the overview of the 3-D panoramic image in reflectance color contrast only, in which the eight tumor targets are difficult to see. Figure 8(c) is the corresponding depth map of the reconstruction in pseudocolor. Figure 8(d) is the 3-D surface reconstruction of the entire phantom with multimodal color contrast, having bright green targets representing the fluorescence residual tumor. An image morphology method was utilized to segment the tumors which were encircled with blue curves and marked with series number labels. Eight tumors were detected and labeled in series, coincident with the total number of fluorescence targets used in phantom fabrication. The shape and size of the tumor targets were also well recovered in 3-D virtual reconstruction. Taking the star tumor (#4) as an example, the average length of the edges was 6.2 mm in the reconstruction. The error was 3.3% compared with the actual fluorescent star (6 mm edge) in the phantom. The 3-D virtual overview with a residual green tumor may be visualized in both reflectance (showing complete vascularity) and multimodal views with pseudocolor replacing reflectance contrast at tumor locations by slowly flashing between visualization modes to maintain the highest information content according to the surgeon’s requirements.

Fig. 8.

The 3-D reconstruction of a spherical-dome tumor cavity phantom with surgical image and residual fluorescence tumor targets. (a) Image sampling grid $5\times 6$ of SFE above phantom with blue points as the features and small pyramids as SFE locations. (b) Reconstructed a map from the SFE reflectance images. (c) The corresponding depth map with pseudocolor scale. (d) Overview of multimodal 3-D reconstruction showing residual tumor with outlines and labels (#13 is circled with arrow). All of eight tumor targets were detected. (e) The comparison of reconstructed 3-D model to the CAD designed model, shown as blue point cloud.

To quantify the reconstruction result, ICP⁵⁶ was employed to align and to compare the reconstructed virtual model to the CAD designed model (ground truth model),⁵⁷ Fig. 8(e). The CAD model is illustrated by the blue dots as a reference point cloud, while the reconstructed one was translated and rotated to best match the CAD design by minimizing the mean-squared error cost function of the distance from each reconstructed feature point to the closest point in the reference point cloud. To demonstrate that the ICP result provides a reliable alignment, eight different positions and orientations around the CAD model were chosen as the initial poses of the reconstructed model.⁵⁸ The deviation between the 3-D virtual reconstruction and CAD model was calculated after alignment. The obtained ICP error values were in the range of 0.6736 to 0.6762 mm with an average of 0.6742 mm and a standard deviation of $0.874\mu \mathrm{m}$, which indicated that the fine alignment does not rely on the initial poses. The mosaic image was a little blurred compared with the original SFE image, as shown in Fig. 8(d). But the numbered electrode patches are still readable in the 3-D mosaicked image, such as #13 patch with a yellow arrow in Fig. 8(d).

To better quantify the error of the reconstructed and CAD models, one alignment was chosen for further error analysis. The error distribution map is shown in Fig 9(a). Most of the error values were less than 1mm, and locating in the central region. Most of the high-value errors ($>1.0\mathrm{mm}$) located along the phantom flat periphery were due to the insufficient density of the features. These features with a large error can be easily eliminated based on their spatial locations. Figure 9(b) showed the histogram of the error distribution which indicated that the average error is about 0.6 mm. Error values in the range of 0 to 1 mm occupied 96.3% of all the data.

Fig. 9.

Error distribution between reconstructed and CAD models. (a) The error distribution map overview with main low-value error points locate in the spherical dome area. (b) The error histogram showing the average error is about 0.6 mm.

This new 3-D reconstruction process based on known SFE (camera) poses takes about 280 s for processing all the 30 reflectance images, while the conventional algorithm takes around 980 s. The new approach is 3.5 times faster because the search range is reduced during the SIFT and the bundle adjustment iteration is decreased. However, the drawback is that this new algorithm is highly sensitive to the accuracy of the camera position and orientation. The conventional algorithm does not rely on knowing the camera position and orientation,³⁷ but is much slower ($>3\times$) and requires a large number of features in all the images.⁴³

5. Discussion

Compared with the previous 3-D surface mosaic algorithm,^37,38 the new reconstruction algorithm proposed in this paper: (1) is more efficient, as $>3\times$ efficiency was achieved since the matching features can be found more quickly within a smaller search region and the iteration can be reduced in the bundle adjustment because of a better initial shape estimation; (2) is more robust, because the smaller search region reduces the interference of similar features in the same image. More matches can be found, which lead to a better initial estimation to bundle adjustment; (3) yields a multimodal 3-D map of the tumor region that can improve visualization and robotic path planning; and 4) generates a physical scale 3-D model instead of relative dimensions, with tumor targets of actual shape and size recovered. This real size reconstruction is useful for locating the physical coordinates of the brain tumor residual in the removal procedure and for providing quantitative characteristics such as size and morphology. But the reconstruction algorithm is very sensitive to the input of the SFE pose data. Inaccurate SFE position and orientation may cause a clearly different geometry away from the phantom.³⁷ To clearly demonstrate this dimensional error when the stereo surveillance system does not provide accurate camera poses, additional tests with flat square paper with our surgery scene were performed, as shown in Fig. 10. The reconstruction with and without accurate data varies dramatically as the square planar paper scene is highly distorted in Fig. 10(a). The motion control of the robot arm with stereo vision augmentation significantly facilitates the reconstruction process, see Fig. 10(b).

Fig. 10.

The 3-D reconstruction of a flat printed paper of a surgical scene at low resolution. (a) The 3-D reconstruction with the data from RAVEN autonomous mode, showing unreasonable shape $40\times 10\times 3\mathrm{mm}$ and no recognizable features in a highly distorted mosaic image. (b) The 3-D reconstruction with the data from stereo surveillance system, exhibiting as a square planar surface and recognizable features such as vessels.

Precise control of the cable-driven surgical robot under the autonomous mode is challenging due to nonlinear factors such as mechanism backlash, cable stretch under tension, and the imperfection of the available kinematic model. One possible solution includes the application of an external tracking system to get accurate pose data for the robot arm as demonstrated in this work. Another approach is to mount encoders on each joint to directly measure the real joint angles, but the robot could become very costly and bulky. In our study, an economical and effective stereo tracking system was built to improve the precision positioning of the SFE and to provide high accurate pose data to the 3-D reconstruction algorithm. The accuracy of the stereo system depends on the webcam image resolution, the distance between stereo to markers and lighting condition. A homogeneous light environment greatly decreases the tracking noise, which quickly stabilizes the robot control.

Experiments showed that the environmental lighting plays a more significant role than the camera resolution in marker tracking, which will be an issue for practical surgery since lighting is not homogeneous. However, we can take advantage of other information to enhance the robustness, such as the shape and size of the markers. Moreover, another drawback of this external stereo tracking system is that it may not be feasible within an enclosed surgical space, such as minimally invasive surgery (MIS) with flexible tool shafts.

Fluorescence imaging is also influenced by the environment lighting since the emission signal is very weak. We darkened the environment and raised the PMT detection sensitivity to capture the fluorescence signals in this study. However, in the future, the SFE may be used in an enclosed keyhole surgical fieldMIS, which can be sheltered from the interference of environmental lighting, like the daVinci® Si robot with a FireFly fluorescence imaging system.

With the combination of multimodal SFE, stereo-assisted RAVEN robot, 3-D reconstruction with feature position prediction, and a spherical-dome shape phantom containing fluorescence targets, an accurate 3-D virtual model representing a debulked brain tumor cavity can be generated with $>3\times$ efficiency with all the tumor targets outlined and labeled. For future work, a more advanced reconstruction algorithm is required to eliminate the feature blurring in the stitched image, which is caused by imprecise calibration of SFE internal parameters, such as focal length, FOV, and distortion. As we know, any measurement system has its own accuracy, which is typically a range such as $\pm 0.2\mathrm{mm}$ for our stereo system. Instead of directly using the known SFE position and orientation as fixed values, the bundle adjustment needs to be improved to optimize the problem of box-constrained parameters (3-D scene, SFE pose, focal length, FOV, distortion, and etc.). In this way, we still can narrow down the search region and speed up the algorithm with a good initial shape estimation, from which a more accurate and a precise 3-D reconstruction model with fewer blurred features can be generated.

Future work will also require a more realistic phantom that can mimic the optical properties and even allow for material resection to mimic image-guided surgery. This initial phantom was simplified with the main property of dimensional accuracy of the 3-D-printed PLA model and glued photographic paper and fluorescence surface targets for multimodal contrast enhancement because this work focused on validating the dimensional accuracy of our 3-D reconstruction algorithm on the surface of this phantom. Future phantoms will require more translucent materials that provide multimodal contrast such as dyed gels containing scattering particles.⁵⁹ The versatile SFE can support multiple modes of imaging, such as reflectance using enhanced spectral imaging which enhances the contrast of the blood vessels and vascularized tissues,²¹ and wide-field fluorescence imaging of NIR dye molecules that are conjugated to molecular specific probes that bind to brain tumor cells.⁸ Additional SFE imaging modes that are molecular specific can also be included in a future phantom or ex vivo tissue model, such as autofluorescence imaging,⁵¹ time-resolved fluorescence lifetime imaging and spectroscopy,⁶⁰ and coherent Raman imaging and spectroscopy.⁶¹

Finally, a systematic analysis of error between the reconstructed and CAD-designed models needs to be performed in the future, including geometry, size, and location of each tumor target. Moreover, the computation procedure of the new algorithm was not fast enough ($>200\mathrm{s}$ for 30 frames although the efficiency rose to $>3\times$), since tissue deformation and shift occur during surgical resection. Also, the 3-D reconstruction of the surgical field needs to perform in an iterative fashion to remove all clearly visualized residual tumor tissue. Thus, faster computation can reduce the surgery time and result in a more precise virtual model. In the future, the algorithm can be boosted by being written in C++ and partial implementation with $z$ parallel computing platform CUDA.

Biographies

Yuanzheng Gong is a PhD student in mechanical engineering at the University of Washington. He received his BS and ME degrees in mechanical engineering from University of Science and Technology of China (USTC) in 2009 and Iowa State University (ISU) in 2011, respectively. His main research interests include three-dimensional (3-D) reconstruction, optical metrology, stereo vision, and machine vision. He is a member of SPIE.

Danying Hu is a PhD student in the BioRobotics Laboratory at the University of Washington. Her research interests include, but are not limited to, medical robotics, robotic surgery, machine learning, control, and automation. She received her BS and MS degrees in mechanical engineering from Tongji University and RWTH Aachen University, respectively. She is a member of VDI.

Blake Hannaford is a professor of electrical engineering at the University of Washington. Before he joined the University of Washington in 1989, he received his BS degree in engineering and applied science from Yale University in 1977, and his MS and PhD degrees in electrical engineering from the University of California, Berkeley, in 1982 and 1985, respectively. Haptic interfaces and medical robotics are his main researches. He is an IEEE fellow.

Eric J. Seibel is a research professor of mechanical engineering, adjunct in bioengineering and electrical engineering, and director of the Human Photonics Laboratory at the University of Washington. He received his BS and MS degree in mechanical engineering from Cornell University and University of California, Berkeley, respectively, and his PhD degree in bioengineering from the University of Washington, Seattle. He invented the scanning fiber endoscopy and develops novel instrumentation techniques for early disease diagnosis and image-guided interventions. He is a member of SPIE.