Real-Time 3-D Video Reconstruction for Guidance of Transventricular Neurosurgery

Abstract

Neuroendoscopic approach to deep-brain targets imparts deformation of the ventricles and adjacent parenchyma, limiting the accuracy of conventional neuronavigation. We report a method for 3D endoscopic reconstruction and registration via simultaneous localization and mapping (SLAM) for real-time guidance with or without robotic assistance. The aim is to permit augmented video overlay of structures registered from preoperative or intraoperative 3D images within and beyond the endoscopic field of view for more accurate targeting in the presence of deep-brain deformation. Phantom studies were performed to evaluate geometric accuracy and uncertainty in distinct scenarios of limited data (feature sparsity and scene occlusion), demonstrating performance over a broad range of challenges to endoscopic data. Reconstruction and registration accuracy were maintained even with up to 40% loss in feature density or 120° of the visual scene occluded. Overall, the method achieved a high degree of geometric accuracy, with target registration error of 1.02 mm and runtime supporting real-time guidance (3.45 Hz, representing a >16× speedup with SLAM approach compared to previous work). The studies establish essential quantitative performance characteristics and validation that are essential to future translation to clinical studies.

I. Introduction

NEUROENDOSCOPY enables minimally invasive approach with direct visualization of deep brain targets about the cerebral ventricles for a variety of diagnostic and therapeutic procedures, including biopsy, cyst resection/drainage, ventriculostomy for hydrocephalus, and emerging approaches for deep-brain stimulation (DBS) [1], [2], [3], [4]. Transventricular neuroendoscopy is particularly helpful in accessing structures in proximity to the lateral and third ventricles, which can be difficult to reach via lateral or transsphenoidal endoscopy [5]. An important consideration for high-precision neuronavigation, however, is geometric deformation due to brain shift and change in cerebrospinal fluid (CSF) volume in the course of endoscopic approach. Brain shift is associated primarily with deformation of the cortical surface and peripheral regions of brain parenchyma (e.g., ~5–24 mm [6] upon incision of the dura) and deformation in periventricular structures (~4–10 mm [7] due to CSF volume changes). Precise target localization can be challenging even with direct neuroendoscopic visualization due, for example, to inconspicuous appearance, occlusion in hard-to-reach recesses of the ventricles, and presentation beyond the visible surface of the ventricles.

Conventional neuronavigation relies on rigid registration to preoperative imaging and is limited in its ability to resolve such deformations. Traditional approaches employ a stereotactic frame [8] to co-register the patient coordinates with 3D image data. Both framed and frameless navigation systems [9] using optical or electromagnetic (EM) tracking similarly employ rigid registration with preoperative imaging via correspondence of fiducial markers [10] or surfaces [11]. The same is true of robotic systems for tool stabilization, cooperative control, and trajectory positioning in neurosurgery and neuroendoscopy [12] – for example, systems such as the Neuromate^® (Renishaw, Wotton-under-Edge, United Kingdom) for placement of DBS electrodes [13], the Excelsius^® (Globus Medical, Audubon, PA) for spine and brain surgery [14], the Galen Surgical Platform (Galen Robotics, Baltimore, MD) for ENT microsurgery [15], and the ROSA ONE^® Brain (Zimmer Biomet, Warsaw, IN) for DBS and stereoelectroencephalography (SEEG) [10]. Such tracking and robotic assistance technologies have provided major strides for confident localization of intracranial targets; however, the reliance on rigid registration to preoperative images fundamentally limits the accuracy with which such systems can be expected to perform in the presence of brain deformation.

Intraoperative magnetic resonance (MR) imaging can provide exquisite depiction of intracranial structures and has demonstrated benefit in a variety of neurosurgical procedures [16]; however, limited field strength (associated with long scan times and/or poor image quality), relatively high operational costs, workflow disruptions, and the need for MR-compatible instrumentation and site requirements challenge its widespread use [17], [18]. Intraoperative computed tomography (CT) [19], [20] and cone-beam CT (CBCT) [21] offer high spatial resolution and clear visualization of bone and intravascular/intraarterial contrast. Recent developments in 3D image reconstruction and artifact correction algorithms [22] aim to improve the soft-tissue contrast resolution in CBCT to a level supporting visualization of gray/white matter parenchyma, the cerebral ventricles, and hemorrhage [23]. Intraoperative ultrasound imaging is an effective means to visualize intracranial structures, including cysts, tumors, and hemorrhage [24]. While it offers advantages of real-time visualization and versatility, the technique is not without challenges, including operator variability and difficulties with image quality, artifacts, and image interpretation [25]. Despite these obstacles, ultrasound image guidance remains a valuable tool for neurosurgeons in the intraoperative setting.

Alternatively, investigators have sought to address challenges of brain shift via biomechanical modeling to predict tissue deformation and provide more accurate, model-based guidance. Early work simulated brain shift due to CSF loss [6], and later work enriched the deformation models by accounting for the material characteristics of various tissues [24]. Recent work employs intraoperative measurements of geometric shift to guide the biomechanical models [25], as investigated using laser range scanners [26] and stereo cameras [27]. More generally, multi-modality image-based deformable registration is an active area of ongoing research, including physics-based [28] and neural network-based [29] solutions aiming to geometrically resolve preoperative and intraoperative imaging in the presence of brain shift.

Computer vision-based registration methods pertinent to neurosurgery have been used to support tracking and guidance in a variety of interventional procedures. Previous work reported video and tracker augmented guidance [30] in bronchoscopy for lung biopsy, wedge resection, and lymph node staging via registration to preoperative CT. Similarly, in laparoscopy, video tracking of deformable tissue surfaces has been demonstrated in liver surgery [31], with some using models for motion compensation and tissue biomechanics [32]. Other work examines whole stomach 3D reconstruction using monocular chromoendoscopic imaging [33] for gastric navigation and lesion localization. Endoscopic video registered to preoperative CT or MR images has also been used to guide trans-sphenoid skull-base surgery [34] for treatment of skull base tumors. Other work demonstrates video-to-CT registration for navigation in rhinoscopy [35] to better localize nerves and arteries and thus monitor deformation over the course of the surgery.

Robot-assisted ventriculoscopy (RAV) is a promising technology that combines the precision of robotic positioning with vision-based guidance to address the challenges posed by deep-brain deformation. In current clinical implementation, a RAV system can provide stable alignment and positioning of the ventriculoscope [36] along a planned trajectory relative to the cranium. More advanced RAV implementations aim to reconstruct and register the neuroendoscopic video scene with 3D imaging and planning data for up-to-date guidance [37]. Registration of the video scene with intraoperative imaging (MR, CT, CBCT, or ultrasound) – in turn deformably registered to preoperative MR and planning data – provides the potential to overcome geometric errors associated with deep-brain deformation. Augmentation of the video scene with deformably registered planning data could thereby provide accurate guidance and visualization beyond the visible endoscopic scene and assist in localizing inconspicuous targets about the lateral and third ventricles even in the presence of deformation.

3D reconstruction of the video scene can be performed via two prevalent classes of computer vision algorithms – structure-from-motion (SfM), and simultaneous localization and mapping (SLAM). SfM uses a collection of unstructured sets of images to estimate camera poses and reconstruct a point-cloud representation by backprojection of covisible features across multiple views. Due to the lack of structure during image acquisition, such an approach is best suited to applications for which a one-time, offline reconstruction is sufficient. SLAM, on the other hand, uses a sequence of video frames to estimate camera poses in a local window, followed by simultaneous reconstruction for tracking. The constrained problem of frame-to-frame camera localization makes SLAM potentially suitable to online, real-time application. Previous work demonstrated the capability of both SfM and SLAM for 3D video reconstruction and visual tracking in endoscopy [38]. Moreover, recent advances in deep learning methods promise to extend such capability further – e.g., neural networks for dense depth estimation and reconstruction in sinus endoscopy [39] along with appearance and/or geometric priors [40].

Previous work by the authors [37] demonstrated the initial development of a RAV prototype for intracranial neurosurgery, including geometric calibration and the ability to use the robot pose for improved 3D video reconstruction using SfM. The system demonstrated accuracy of 1.2 mm in 3D reconstruction and registration but was limited in its runtime to 4.9 s per frame. The work reported below aims to develop SLAM for 3D video reconstruction more suitable to practical application in neuroendoscopy by providing real-time guidance, following early work describing parameter selection [41]. As a precursor to future clinical studies, a series of experiments is reported in rigorous, systematic development and evaluation of the method in laboratory studies. Such work provides essential guidance and validation of the algorithm, rigorous analysis of parameter selection, geometric accuracy, runtime capability, and characterization of uncertainties in system performance that are requisite to clinical translation. Specific contributions of the work include first reporting of: (1) development of a preclinical mobile platform for real-time RAV with SLAM video reconstruction; (2) investigation of SLAM parameter sensitivity and optimal/nominal parameter selection; (3) experimental characterization of reconstruction and registration uncertainty associated with limited data (viz., feature sparsity and regional/spatial ablation); and (4) evaluation of geometric accuracy and runtime in comparison to SfM.

II. Methods

A. System for Robot-Assisted Ventriculoscopy

1). System Components:

The preclinical RAV prototype illustrated in Fig. 1 was designed as a platform for 3D video reconstruction and registration integrated with intraoperative imaging and navigation. The RAV was implemented on a mobile cart for tableside positioning superior to the patient as shown in Fig. 1b. The ventriculoscope (Lotta 28164LA 6°, Karl Storz, Tuttlingen, Germany) was coupled to a camera head and console (AIM1588, Stryker, Kalamazoo MI, USA) with the video acquired at fixed (1X) zoom and focus at 1920 × 1080 pixel resolution and 10 frames-per-second (fps). Video was captured with a frame grabber (USB2DVIE 3.0, Epiphan systems, Ottawa, Canada) on a computer workstation (Dell Precision T7600 with Intel Xeon E5–2620, 32GB memory and NVIDIA GTX 960Ti GPU). The ventriculoscope was rigidly coupled to a robotic arm (UR3e, Universal Robots, Denmark) with a pose repeatability of ± 0.03 mm (per UR3e specifications) to ensure repeatable and consistent data acquisition and ease of experimentation. Previous work [37] demonstrated advantages in geometric accuracy and runtime by exploiting robot pose information in 3D point-cloud reconstruction, but robot pose was not used in the current work, which focused on aspects of a reconstruction algorithm that could translate either to robot-assisted or freehand use. The O-arm^™ (Medtronic, Littleton MA, USA) was used for intraoperative CBCT with standard cranial scan protocols and algorithms for high-quality 3D image reconstruction sufficient for visualization of the ventricles [42]. A surgical tracker (StealthStation^™, Medtronic, Boulder CO, USA) was used to track the ventriculoscope using a rigidly attached dynamic reference frame (DRF, SureTrak^™) and a reference marker attached to the cranium (shown in Fig. 1b on a head phantom; LiquiPhil, The Phantom Lab, Greenwich NY, USA). A custom 3D-printed anthropomorphic ventricle phantom with simulated neurovasculature is shown inside the cranial phantom in Fig. 1b and in the example video frame in Fig. 1c.

Fig. 1.

Preclinical system for robot-assisted 3D neuroendoscopy. (a) Systems for intraoperative CBCT imaging, surgical tracking, and robot-assisted ventriculoscopy. (b) Positioning of the robot end effector and neuroendoscope in relation to the head phantom. (c) Neuroendoscopic view of the anthropomorphic ventricle phantom, showing lateral ventricles, Foramen of Monro, simulated vasculature, and BB target points.

2). Coordinate Frames, Transforms, and Calibration:

Coordinate frames and transformations pertinent to the RAV system are shown in Figure 2, taking the surgical tracker as the world frame ($W$). The camera frame is denoted $C$ with origin at the center of the endoscope lens. The DRF (frame $D$) attached to the endoscope is tracked relative to the reference marker (frame $R$). Following CBCT acquisition, the ventricular surface was segmented (denoted $S$ in CBCT coordinates). Notation $T_A^B$ represents the homogenous rigid transformation locating frame $B$ with respect to frame $A$. Intraoperatively, the tracking system yields $T_W^D$ and $T_W^R$ and consequently $T_R^D={\left(T_W^R\right)}^{-1}{T}_W^D$. The endoscope is first localized in the tracker frame using one-time, rigid hand-eye calibration ($T_D^C$) using $T_W^C=T_W^D{T}_D^C$. The CBCT image is registered with the reference frame (via $T_R^S$) automatically by the StealthStation^™ tracker, which observes the reference marker during CBCT acquisition. $T_R^S$ is used as a one-time initialization to localize the CBCT surface with the camera via $T_C^S={\left(T_D^C\right)}^{-1}{\left(T_R^D\right)}^{-1}{T}_R^S$. Preoperative imaging (CT and/or MR) can be registered to intraoperative CBCT via image registration (CBCT-to-CT [43] or CBCT-to-MR [29]) or co-registered fiducial points. As detailed below, SLAM (alternatively, SfM) was used to reconstruct a 3D point-cloud representation of the ventricles from endoscopic video, with the point-cloud coordinate frame denoted $P$. Real-time camera tracking relative to the phantom is achieved via $T_C^P$, and the point cloud is registered with the CBCT surface to give $T_P^S$. The transformation chain $T_C^S=T_C^P{T}_P^S$ thereby yields guidance and augmented visualization of 3D structures from the CBCT reference frame directly in the endoscopic view.

Fig. 2.

Coordinate frames and transformations for video reconstruction and registration. The hand-eye transformation ($T_D^C$, in gray) is a one-time calibration performed preoperatively. The reference-to-CBCT transform ($T_R^S$, in gray) is automatically obtained using StealthStation^™, and used for a one-time initialization. All other transformations are dynamically updated for real-time registration of the 3D endoscopic point-cloud and intraoperative CBCT.

B. Point-Cloud Reconstruction and Registration

A SLAM approach was used for real-time video tracking of the interior ventricular surface, with algorithmic components summarized in Fig. 3. Section II–B1 details the data acquisition and distortion correction procedure, followed by simultaneous tracking (feature correspondence and localization) and mapping (sparse reconstruction), summarized as follows. First, a one-time system calibration is performed to estimate the camera intrinsics and tracker-to-camera transformation ($T_D^C$). Ventriculoscopic video data ($𝒥$) are input and processed on a per frame basis after correction for lens distortion. For each image, features ($ℱ$) are extracted and matched (with feature matches denoted by $𝒬$) with features in consecutive frames to yield an approximate relative camera pose (${𝒫}^{\prime }$) and thus track the ventriculoscope. Salient frames observed during tracking are designated as keyframes according to duration elapsed since last keyframe and minimum visual change conditions [44] for further mapping. Simultaneously, the mapping thread maintains a pose graph [45], [46] whose nodes correspond to the keyframe pose estimates and whose edges represent relative camera pose constraints. Triangulation of matched features yields a point-cloud representation (${𝒳}^{\prime \prime }$) of the scene, and a local bundle adjustment is performed to yield the optimized point cloud (${𝒳}^{\prime }$) along with further improved relative poses ($𝒫$) to minimize reprojection error. Finally, as described in Section II–B2, the resulting point cloud is filtered and registered with ground truth to estimate $T_P^S$, enabling real-time overlay of target structures in the ventriculoscopic view.

Fig. 3.

Algorithmic pipeline illustrating preprocessing along with simultaneous feature detection/pose prediction (“Tracking” – blue) and 3D point-cloud reconstruction (“Mapping” – green). The resulting point cloud is registered to intraoperative image coordinates for overlay of CT or MR image/planning data in the video scene. The transform $T_P^S$ registers the 3D point cloud to the CBCT surface (ventricle segmentation), and $T_C^S$ maps the data defined in CBCT coordinates directly to the endoscopic view – alternatively ${\left(T_C^S\right)}^{-1}$ maps the camera pose to CBCT coordinates for 3D navigation.

1). Real-Time Point-Cloud Reconstruction via SLAM:

SLAM-based methods offer potential advantages for continuous, real-time tracking and reconstruction, as opposed to processing unstructured images as reported previously with SfM methods, which may be more appropriate to one-time, offline reconstruction of the scene. The SLAM pipeline summarized in Fig. 3 was developed to enable real-time tracking of camera pose along with an iteratively updated 3D point-cloud reconstruction of the visual scene. In the current work, the terms ‘localization’ and ‘mapping’ are used interchangeably with ‘tracking’ and ‘reconstruction’, respectively. A parallel architecture [47] combined with use of keyframes for keypoint backprojection and mapping presents an important advantage in computational cost for real-time tracking. Similarly, bundle adjustment subroutines performed in a local window of frames gives a substantial reduction in computational cost. The PySLAM v2 library [48] along with OpenCV [49] and TREK [50] were used for the reconstruction pipeline.

Neuroendoscopic video was acquired as a series of video frames denoted $𝒥=\left\{I_i:i=1\dots N_I\right\}$. The RAV setup of Fig. 1 was used to avoid freehand tremor and motion artifacts and reproducibly focus the analysis on the geometric accuracy of the SLAM vs. SfM algorithms. Linear motion along the axis of the endoscope was executed via the robot in a trajectory analogous to clinical approach between the lateral and third ventricles via the Foramen of Monro. Previous work showed the ability to perform 3D video reconstruction for alternative motion trajectories and described the preoperative calibration procedure for estimation of camera intrinsics ($K$) and correction of lens distortion effects [51]. Hand-eye calibration was used to estimate the rigid transform ($T_D^C$) to localize the camera coordinate frame with DRF.

As illustrated in the upper arm (blue) of Fig. 3, the tracking thread performs the tasks of feature detection and pose prediction in sequential camera frames. For each video frame, Speeded Up Robust Features (SURF) features [52] were used to establish correspondences between sequential images and estimate homography. For a gray-scale converted video frame, $I_i$, SURF features designated as ${ℱ}_i=\left\{f_{ij},x_{ij}:j=1,\dots ,N_{iF}\right\}$ were extracted with $f_{ij}$ denoting a feature descriptor at pixel location, $x_{ij}$. A Hessian blob detector over precomputed integral images in scale space [53] was used to identify interest points. The determinant of the Hessian matrix at interest points was then thresholded to limit the maximum number of keypoints to $N_K$. For this purpose, a series of response maps were obtained by convolving the input image with increasing filter sizes to construct the scale space, divided into $N_O$ octaves and $N_L$ scale level subdivisions. The corresponding feature detection runtime is designated as $t_f$.The SURF-128 extended feature descriptor at each keypoint location was obtained by computing 4D Haar wavelet response along 4 × 4 subregions of the image. The descriptors are computed separately for wavelet responses <0 and ≥0, resulting in a 128D descriptor for each keypoint location.

The resulting features were matched to refine tracked camera poses and assess visual saliency for keyframe decision. First, a constant velocity motion model was used to predict relative camera poses, designated ${𝒫}^{\prime }=\{P_i^{\prime }=T_{C_i}^{C_{i+1}}\}$, defining $P_1^{\prime }=I_3$ (identity matrix). Features (${ℱ}_i$) were then projected using the approximate pose $P_i^{\prime }$ over image $I_{i+1}$ to perform a guided search for matches. A Fast Library for Approximate Nearest Neighbors (FLANN) matcher [54] with $L^2$ norm distance was used to establish a match matrix of features, $𝒬=\left\{I_a,I_b,L_{ab}:I_a,I_b\in 𝒥\right\}$, with $L_{ab}\subset {ℱ}_a\times {ℱ}_b$ describing the strength of correspondence between features in images $I_a$ and $I_b$. A pair of interest points is matched if the descriptor distance is closer than the distance ratio ($r_{ab}$) of the second nearest neighbor, resulting in $N_M$ successful matches. Finally, if the current frame tracks less than a fraction ($r_X$) of keypoints compared to the previous keyframe, and at least $N_{\text{insert}}$ frames have passed since the last keyframe insertion, the frame is designated a keyframe and inserted into a pose graph for the mapping thread.

In parallel with the tracking thread, a mapping thread (lower arm in Fig. 3 marked in green) operates on the pose graph to observe inserted keyframes and reconstruct the 3D scene. For each inserted keyframe, matched features are triangulated using the approximate camera poses (${𝒫}^{\prime }$) obtained by tracking to estimate a coarse point-cloud reconstruction (${𝒳}^{\prime \prime }$). A local bundle adjustment routine is then used to optimize the current keyframe pose along with other keyframes sharing matches in a local window. The bundle adjustment routine is then used to minimize the net reprojection error (RPE), jointly refining approximate camera poses (${𝒫}^{\prime }$) and coarse point cloud (${𝒳}^{\prime \prime }=\{{\overrightarrow{X}}_k^{\prime \prime }:i=1,\dots ,N_X\}$), defined as:

$${𝒳}^{\prime },𝒫=\underset{\left({𝒳}^{\prime \prime },{𝒫}^{\prime }\right)}{\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{i}\mathrm{n}}‖\mathrm{R}\mathrm{P}\mathrm{E}\left({𝒳}^{\prime \prime },{𝒫}^{\prime }\right)‖$$ (1)

The reprojection error (RPE) for a 3D point ($X_k^{\prime \prime }$) obtained by triangulation of corresponding feature pixel locations $\left(x_{ij}\right)$ is:

$$\mathrm{R}\mathrm{P}\mathrm{E}\left({𝒳}^{\prime \prime },{𝒫}^{\prime }\right)=\frac{1}{N_I}\sum _{i=1}^{N_I}\sum _{j=1}^{N_{iF}}{\left(x_{ij}-K{P}_i^{\prime }{\overrightarrow{X}}_k^{\prime \prime }\right)}^2$$ (2)

where the feature, $x_{ij}$, is detected in an image with estimated pose, $P_i^{\prime }$, using the calibrated camera intrinsics matrix, $K$. The resulting refined point cloud (denoted ${𝒳}^{\prime }$) and precise camera poses (denoted $𝒫$) are obtained within a scale factor for registration. To reduce computational load, redundant keyframes are detected and culled, and all keyframes in which more than a fraction, $r_X$, of keypoints have been observed in at least 3 other keyframes are marked as redundant and removed. Further, outlier points in the reconstruction are detected and removed based on thresholding of the triangulation angle and focal length within realistic values supported by the system geometry and anatomical context. Finally, a statistical outlier filter with $k_{nn}$ neighbors and ${\sigma }_{nn}$ standard deviation was used to trim outlier points, yielding the reconstructed sparse point cloud, $𝒳$ (also within a scale factor).

2). Point-Cloud to Surface Registration:

The pose of the reconstructed point cloud ($𝒳$) was initialized to the CBCT surface segmentation, $S=\{{\overrightarrow{S}}_j:j=1,\dots ,N_S\}$ (or preoperative CT or MR images registered to CBCT) via the initial affine transform ($T_P^S=[R_P^S,{\overrightarrow{t}}_P^S]$) using the tracked endoscope pose ($T_P^C$) with a manual initialization of scale ($r$). The point cloud ($𝒳$) was then registered to the surface ($S$) using iterative optimization via the trimmed iterative closest point (ICP) method [55] to estimate the similarity transform $\left(T_P^S\right)$ with scale. The Levenberg-Marquardt optimizer was used with a point-to-plane distance metric optimized for $m=1,\dots ,N_M$ iterations. The ICP registration routine is then defined to minimize normal distance to mesh ($d_k$) as:

$$d_k={\left(R_P^S{\overrightarrow{X}}_k+r{\overrightarrow{t}}_P^S-{\overrightarrow{S}}_j\right)}^T\cdot {\overrightarrow{n}}_j$$ (3)

and

$$r,T_P^S=\underset{{\left(r,T_P^S\right)}_m}{\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{i}\mathrm{n}}\left(\sum _{N_{\mathrm{i}\mathrm{n}}}{\left(d_k\right)}^2\right)$$ (4)

for a given point (${\overrightarrow{X}}_k$) closest to surface point (${\overrightarrow{S}}_j$) in $S$ with face normal (${\overrightarrow{n}}_j$). The sum-of-squared residual distances was calculated for all points and sorted by distance. A fraction $\left(\gamma \right)$ of inlier points denoted $N_{\text{in}}=\gamma N_X$, with lowest residual distance were chosen for minimization, yielding an incremental update ${\left(r,T_P^S\right)}_m$ converging to the solution $r$ and $T_P^S$. CloudCompare [56] was used for filtering and visualization along with ICP routines from MATLAB 2020a (MathWorks, Natick MA, USA) for registration. The resulting registration then enables visualization and overlay of target structures directly in the video frame along with conventional mapping of the endoscope trajectory with respect to the ventricle surface, $S$, and other data registered to CBCT coordinates. Blender [57] was used to render target structures as viewed from camera poses, and OpenCV [49] was used for alpha blending and overlay of rendered views in endoscopic video.

C. Experimental Evaluation and Validation

A series of three experiments was performed for quantitative evaluation of the accuracy, uncertainty, and runtime in 3D point-cloud reconstruction and registration: (1) the sensitivity of feature detection performance to SLAM parameter selection; (2) the uncertainty of 3D point-cloud reconstruction and registration under conditions of limited data – viz., as a function of feature sparsity (loss in density of features) and spatial ablation (regional loss of features); and (3) evaluation of the geometric accuracy in point-cloud reconstruction and registration along with system runtime in comparison to prior work based on SfM.

1). Experiment 1(Selection of SLAM Reconstruction Parameters):

The first experiment was designed to inform optimal and nominal selection of parameters affecting feature detection and reconstruction using an anthropomorphic ventricle phantom [37]. The ventricle phantom [Fig. 1(c)] presents semirealistic structures corresponding to the lateral and third ventricles, Foramen of Monro, and neurovasculature such as thalamostriate vein and smaller vessels. The walls of ventricles were embedded with radiopaque Teflon BBs (green, Fig. 1c) for measurement of target registration error (TRE). High-resolution CBCT of the phantom was acquired using the O-arm^™ with a high-definition cranial scan protocol: 745 X-ray projections; 100 kV and 150 mAs; 360° scan orbit; and 0.3 × 0.3×0.3 mm³ voxel size reconstructed using 3D filtered backprojection. The reconstructed volume was segmented using a fast-marching algorithm [58] to extract a triangulated surface mesh ($S$) along with surface normal, used as ground truth for registration. Target fiducials (Teflon BBs) were manually localized for analysis of TRE.

Ventriculoscopic video was acquired by translating the RAV end-effector in a linear trajectory, traversing the lateral ventricles and entering the third ventricle through the Foramen of Monro. The trajectory mimics clinical ventriculoscopic motion through a sheath inserted via a Kocher point on the cranium, with a length of 50 mm traversed at approximately 2 mm/sec to avoid motion artifacts. Reconstruction in each experiment was based on a single video sequence capturing $N_I=300$ images (e.g., spanning a 50 mm path length for the linear trajectory). Each trajectory was highly reproducible due to rigid mounting of the ventriculoscope to the robot end effector and high degree of precision in robot positioning (± 0.03 mm).

The performance of feature detection and reconstruction /registration accuracy was investigated as a function of algorithm parameter settings. Fixed parameters in these experiments were set according to previous and other work not shown for brevity: the feature descriptor ratio for successful matches was set ($r_{ab}$) to 0.7 ; for keyframe selection, the fraction $r_X$ was set to 0.9, and the minimum number of frames ($N_{\text{insert}}$) was set to 20 ; a local window of 10 frames was used for bundle adjustment, and global bundle adjustment was not performed; the reconstructed point cloud was filtered using $k_{nn}=5$ neighbors and ${\sigma }_{nn}=1$, filtered for 3 iterations; and the trimmed ICP fraction of overlap ($\gamma$) was set to 0.95, and the registration was assumed to be converged for tolerance limits of ${\mathrm{\Delta }}_r<{0.05}^{\circ }$ in rotation and ${\mathrm{\Delta }}_t<0.01\mathrm{m}\mathrm{m}$ in translation between two successive iterations. The window size for bundle adjustment did not have a strong effect on reconstruction quality – viz, less than 10% increase in the number of points and minimal influence on track length for window size up to 40 frames – attributed to the relatively small frame-to-frame camera motion and static phantom.

Primary experimental variables in the sensitivity analysis were the number of scale space octaves ($N_O$), number of scale levels per octave ($N_L$), and the number of keypoints ($N_K$). A complete bivariate analysis was conducted with $N_L$ and $N_O$ as experimental variables to measure the effect on the number of 2D image features extracted ($N_F$) and the feature detection runtime $\left(t_f\right).N_O$ was varied from 3 to 8, and for each $N_O,N_L$ was varied from 3 to 17. The corresponding $N_F$ and $t_f$ were then measured ignoring data transfer overheads. Sensitivity to the strength (saliency) of extracted keypoints was investigated by selecting the $N_K$ strongest keypoints from candidate interest points via thresholding of the Hessian detector. The effect of $N_K$ on geometric accuracy was evaluated in terms of projected error (PE):

$$\mathrm{P}\mathrm{E}={\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{j}}_S\left(d_k\right)$$ (5)

where ${\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{j}}_S$ is the projection of residual error on mesh surface. A $k$-nearest neighbor interpolation for PE was then performed over the surface with $k_{nn}=5$. To reduce extrapolation artifacts, the ground truth surface was first truncated to the region that was within a convex hull of the reconstructed point cloud’. The projected error (PE) was subsequently measured over this truncated region of the ground truth surface.

The system was also evaluated in terms of overall runtime (termed $t_{\mathit{sys}}$) along with its individual components, tracking runtime ($t_{tr}$) and mapping runtime ($t_m$). For SfM-based methods, the offline, one-time execution results in a net duration $t_{\mathit{sys}}=t_{tr}+t_m$. For applications in neuroendoscopy, however, the procedure is performed in two stages. First, the endoscope is oriented and the surrounding anatomy is mapped using views of the lateral ventricles. In the second stage, the endoscope is advanced through the Foramen of Monro into third ventricles to locate critical target structures, during which only tracking is following registration with the ground truth surface, resulting in $t_{sys}=t_{tr}$. The per-frame system runtime was also measured, defined as $t_I=\frac{t_{\mathit{sys}}}{N_I}$ for a one-to-one comparison of SLAM and SfM methods.

2). Experiment 2(Uncertainty in Video Reconstruction and Registration):

The second experiment was designed to quantify the uncertainty in reconstruction and registration using two distinct ablation models of limited data, motivated by two challenges observed clinically in neuroendoscopy. In each case, only matched features were ablated (cf, random features that may or may not contribute to the point cloud reconstruction). The first model emulates the scenario of anatomy with a lower density of features throughout the scene (e.g., a relatively “featureless” ventricular surface with less conspicuous neurovasculature or other salient structures). The second model approximates a regional loss of visibility (e.g., an air bubble on the endoscope lens or large-scale glare obscuring a relatively large spatial extent). For the second model and a linear trajectory with relatively small incremental motion between frames, the image occluded approximately the same part of the 3D scene and thereby is a reasonable approximation of large-scale artifacts such as glare and specular reflection. Each experiment employed the phantom as a typical best-case presentation of feature density and visibility (e.g., neurovasculature on the ventricular surface) – the first model evaluating the sensitivity of SLAM to feature density by systematically reducing the density of salient feature matches in the video data (feature sparsity), and the second model evaluating performance as a function of regional spatial ablation in the image domain (spatial ablation). 3D reconstructions were computed from a single video sequence acquired using the linear trajectory, with 20 trials conducted over various degree of ablation for both models.

The first model (feature sparsity) was developed to emulate loss of salient features observed in video frames, whereas SURF features may or may not be pertinent to reconstruction due to spurious detections. Note again that only successfully matched features which would be subsequently triangulated and thereby contribute to the reconstructed 3D point cloud were ablated. The fraction of matched features was reduced from 1.0 to 0.3. A fraction of successful matches between consecutive frames was then randomly ablated, with the ablation fraction varied from 0.0 (representing un-ablated reconstructions) to 0.7 (where 70% of successfully matched features were randomly ablated) with step size of 0.1. For each ablation fraction, 20 independent trials were performed, resulting in assessment of uncertainty in 160 3D point-cloud reconstructions.

The second model (spatial ablation) targeted regional removal of observed features, where all features within an angular sector from the image center were discarded. Sectors spanning a particular extent (wedge-shaped sector) of the video image were selected, and the extent of sector was varied from 0° (representing un-ablated reconstruction) to 240° (for which features observed in 2/3 of image were ablated) in 20° increments. For each ablation extent, the orientation of the ablated sector was rotated from 0° to 360° in increments of 18° to obtain 20 trials, yielding 260 point-cloud reconstructions that sampled a broad range of regional ablations.

The uncertainty in point-cloud reconstruction was assessed in terms of PE in the registered point-cloud, the number of points in the 3D reconstruction ($N_X$, which reflects the density of the point-cloud reconstruction), and the precision $\left(P\right)$ and recall $\left(R\right)$ of points therein. The median registration error was anticipated to degrade with increased feature sparsity or spatial ablation, and the variability in registration error across the distribution of ablations was analyzed to characterize the uncertainty associated with the registration result. Analysis of precision and recall metrics as defined in [59] further quantified the accuracy and completeness of the reconstruction. For a reconstructed point, ${\overrightarrow{X}}_k\in 𝒳$, the distance to ground truth ($S$) is:

$$e_{𝒳\to s}=\underset{s_j\in s}{\mathrm{m}\mathrm{i}\mathrm{n}}||{\overrightarrow{X}}_k-{\overrightarrow{S}}_j||$$ (6)

Precision is defined as the fraction of points from reconstruction ($𝒳$) within a distance threshold ($d_t$) to ground truth $\left(S\right)$:

$$P\left(d_t\right)=\frac{1}{N_X}\sum _{i=1}^{N_X}\left[e_{𝒳\to S}<d_t\right]$$ (7)

where $[\cdot ]$ is the Iverson bracket binarizing outcome, and $P\left(d_t\right)$ is defined to lie in range [0, 1]. Conversely, for a ground truth point $S_i\in S$, the distance to reconstruction ($𝒳$) is defined as:

$$e_{S\to 𝒳}=\underset{X_k\in 𝒳}{\mathrm{m}\mathrm{i}\mathrm{n}}||{\overrightarrow{S}}_j-{\overrightarrow{X}}_k||$$ (8)

and recall is thus defined as the fraction of points from ground truth ($S$) within a distance threshold ($d_t$) to reconstruction ($𝒳$):

$$R\left(d_t\right)=\frac{1}{N_S}\sum _{i=1}^{N_S}\left[e_{S\to 𝒳}<d_t\right]$$ (9)

where $R\left(d_t\right)$ is also in the range [0, 1]. Low recall is associated with a lower scene coverage in that a smaller number of ground truth points have a corresponding point in the reconstruction within the distance threshold. Precision and recall curves were computed by varying the threshold $\left(d_t\right)$ to characterize the operating curves for a particular level of geometric accuracy.

3). Experiment 3(Evaluation of Geometric Accuracy):

Following the parameter selection of Experiment 1 and the uncertainty assessment of Experiment 2, a third experiment assessed the geometric accuracy of SLAM reconstruction compared to previously reported SfM methods using the experimental setup of Fig. 1. The COLMAP library [60], [61] was used for SfM reconstruction, with the Scale-invariant feature transform (SIFT) [53] for feature detection and description. Linear motion of the endoscope from the lateral to third ventricle via the Foramen of Monro was used to obtain reconstruction without use of end-effector pose information to facilitate direct comparison of SLAM and SfM methods. Shared parameters were commonly fixed between methods – for example, statistical outlier filtering with $k_{nn}=5$ neighbors with $\sigma =1$ for 3 iterations prior to trimmed ICP registration $(\gamma =0.95)$ with the ground truth surface.

The reconstructed point clouds for both methods were assessed in terms of geometric accuracy and system runtime. In addition to analysis of PE, the BBs embedded in the phantom were manually segmented, with the 3D position of BBs denoted ${\overrightarrow{X}}_i\left(i=1,\dots ,N_B\right)$ in the reconstructed point cloud following registration along with the position of corresponding BBs in the ground truth image (${\widehat{X}}_i$). The TRE was calculated as the RMS distance between BBs:

$$\mathrm{T}\mathrm{R}\mathrm{E}=\sqrt{\frac{1}{N_B}\sum _{i=1}^{N_B}{\left({\overrightarrow{X}}_i-{\widehat{X}}_i\right)}^2}$$ (10)

A Mann-Whitney U test was used to evaluate statistical significance in observed differences for both the PE and TRE measurements under the null hypothesis that two independent samples draw from identical distributions with equal median. This form of test is appropriate in comparison of distributions without prior assumptions on the shape of the error distribution (e.g., no assumption of a Gaussian distribution as with Student t-tests). The tracking and mapping runtimes for both SLAM and SfM methods were measured, along with system runtime ($t_{\mathit{sys}}$) and per-frame runtime ($t_I$). The system runtimes were assessed as a function of the number of images used in forming the reconstruction, $N_I$, ranging from 10 to 300 images. Finally, target structures annotated in CBCT (or preoperative CT or MR registered to CBCT – e.g., simulated target lesions outside the ventricular wall) were rendered using the resulting registration and overlaid to qualitatively depict an augmented view of the endoscopic scene.

III. Results

A. Experiment 1: Selection of SLAM Reconstruction Parameters

Figure 4 summarizes the effect of SLAM tracking and reconstruction parameters on geometric accuracy and runtime. As shown in Figure 4a by isocontours of $N_F$, the mean $N_F$ obtained is a function of both $N_O$ and $N_L$. For any $N_L$, an increase in $N_O$ results in a sharp increase in the number of features detected for small values of $N_O$ (~3–5), followed by saturation at $N_O~6$, beyond which the mean number of features is roughly unchanged. Similarly, an increase in $N_L$ results in larger $N_F$ sharply up to $N_L~10$, beyond which a gradual saturation is reached for $N_L>11$. A mean $N_F~500$ was selected, with $N_O=6$ and $N_L=10$ ensuring sufficient features for tracking.

Fig. 4.

Parameter selection and sensitivity analysis for feature detection. The number of octaves ($N_O$) and number of scale levels ($N_L$) were evaluated in terms of (a) the number of features detected ($N_F$) and (b) feature detection runtime ($t_f$), depicted as isocontours. (c-d) Effect of the number of candidate keypoints used for feature detection ($N_K$) on the projected error (PE) and tracking runtime ($t_{tr}$).

Figure 4b shows the effect of $N_O$ and $N_L$ on feature detection runtime ($t_f$), demonstrating a larger sensitivity to increases in $N_L$ compared to $N_O$. The increase in $t_f$ for constant $N_O$ was approximately linear, with mean $t_f$ increasing from 0.01 s to ~0.08 s over the range of scale levels investigated. To achieve runtime consistent with clinical application (at least ~10 fps desired for real-time overlay in freehand endoscopy) while maintaining a sufficient number of features, a nominal value of $N_L=10$ was selected, resulting in a mean $t_f$ of 0.07 s.

Figure 4(c–d) demonstrates the sensitivity of PE and tracking runtime ($t_{tr}$) with respect to the number of candidate keypoints $\left(N_K\right)$. Selecting fewer $N_K$ (<500) results in poor feature detection and tracking, causing steep increases in median PE of ~0.85 mm. Increasing $N_K$ up to ~500 keypoints steeply reduces geometric error, beyond which mean PE saturates at ~0.2 mm with no further improvement over an order of magnitude in $N_K$. The tracking runtime ($t_{tr}$) increased sharply from $t_{tr}=0.26\mathrm{s}$ up to 1.06 s over the corresponding range in $N_K$. Therefore, to balance tradeoffs between runtime and geometric accuracy, $N_K=500$ was selected for further experiments.

B. Experiment 2: Assessment of Uncertainty in Reconstruction and Registration

Figure 5 summarizes the system performance for varying levels of feature sparsity. With the fraction of features ablated ranging from 0.0 to 0.7, the median PE increased from ~0.5 mm to ~1.1 mm. The variation in PE also increased for a lower density of features, with the interquartile range, IQR (and 95% confidence interval, ${\mathrm{C}\mathrm{I}}_{95}$) in PE varying from 0.05 mm (${\mathrm{C}\mathrm{I}}_{95}=0.67\mathrm{m}\mathrm{m}$) to 0.31 mm (${\mathrm{C}\mathrm{I}}_{95}=1.34\mathrm{m}\mathrm{m}$), signifying an increase in the uncertainty of reconstruction and registration for higher ablation fraction (i.e., lower feature density). Figure 5b shows the distribution in the number of 3D points ($N_X$) and PE over the range of resulting point-cloud reconstructions, with $N_X$ as well as the mean and uncertainty in PE observed to increase with higher ablation.

Fig. 5.

Uncertainty in reconstruction associated with the sparsity of visual features. (a) Projected error (|PE|) measured for various match ablation fractions demonstrating an increase in error and uncertainty of prediction as ablation fraction increases. (b) |PE| measured as a function of the number of reconstructed 3D points ($N_X$). (c) Precision and Recall for a nominal threshold of $d_t=2\mathrm{m}\mathrm{m}$ (d) Precision and Recall as measured for various distance thresholds ($d_t$) from 0.5 mm to 2 mm. (e-f) PE from reconstructed point cloud (with BBs in green and vasculature in red) with an ablation fraction of 0.5 visualized as colorwash indicating an unbiased distribution of errors across 3D space.

The effect of feature density on the precision and recall of point-cloud reconstructions is shown in Figure 5c, taking $d_t=2\mathrm{m}\mathrm{m}$ as a nominal level of desired geometric accuracy. Precision and recall are reduced from ~0.95 and ~0.81, respectively, in best-case (un-ablated) reconstructions to ~0.85 and ~0.55, respectively for an ablation fraction of 0.4. These metrics quantify the quality of point-cloud reconstruction in that a high precision requires a large spatial coverage. For ablation fraction > 0.4, the reconstruction process is seen to breakdown, evident in the distribution of points with nearly zero recall. Such large ablations (i.e., relatively featureless scenes) result in failed reconstructions with insufficient coverage and degenerate registration solutions (i.e., diminishing recall and a broad range in precision). Figure 5d shows the (median) precision-recall curves for various settings of the distance threshold $\left(d_t\right)$, with ablation fraction increasing from right to left along each curve. These operating curves quantify the expected precision and recall for any level of feature density (down to ablation fraction ~0.4) and any requirement in distance accuracy (from 0.5 mm to 2.0 mm, corresponding approximately to the best-case accuracy of surgical navigation systems [8], [9]). Precision is observed to reduce more slowly than recall, showing a stable, accurate operating curve for $d_t=2\mathrm{m}\mathrm{m}$ even with a loss in 40% of salient features.

Figure 5(e–f) depicts the spatial distribution in (signed) PE visualized with respect to the ground truth surface within the convex hull of the reconstructed point cloud for an ablation fraction of 0.5, where positive values represent distance along surface normal “outward” from the surface. Little or no directional bias is observed, and a relatively even distribution of error is observed across the surface, consistent with the spatially random nature of ablation in this experiment.

The effects of spatial/regional feature ablation are summarized in Figure 6. As shown in Figure 6a, the median PE over a broad range of spatial ablation (angular extent of the ablated sector) was observed to increase only slightly for larger ablations, varying from ~0.82 mm for no occlusion to ~0.95 mm for 220° ablation, beyond which the reconstruction and registration collapsed due to degeneracy. However, the uncertainty in PE increased substantially over the same range, with IQR increasing from 0.10 mm (${\mathrm{C}\mathrm{I}}_{95}=1.45\mathrm{m}\mathrm{m}$) to 0.56 mm (${\mathrm{C}\mathrm{I}}_{95}=2.78\mathrm{m}\mathrm{m}$). The increase in variability signifies the more uncertain reconstruction and registration resulting from ablation of larger regions of the image. Figure 6b shows the number of 3D points ($N_X$) in the resulting reconstructions with respect to PE and ablation fraction, where a larger extent of ablation is seen to increase the magnitude and range (uncertainty) of PE.

Fig. 6.

Uncertainty in reconstruction associated with spatial/regional ablation of the video scene. (a) Projected Error (|PE|) measured as a function of the spatial ablation sector angle, demonstrating an increase in registration error and uncertainty of prediction as the extent of ablation increases. (b) |PE| measured as a function of the number of reconstructed 3D points. (c) Precision and recall for a nominal threshold of $d_t=2\mathrm{m}\mathrm{m}$. (d) Precision and recall measured for various distance thresholds ($d_t$) from 0.5 mm to 2 mm. (e-f) Distance from ground truth visualized in an example point-cloud reconstruction with 180° spatial ablation, with larger errors seen in the ablated (anterior) region and corresponding reduction in recall.

Precision and recall for reconstructions resulting from regionally ablated scenes are shown in Figure 6(c–d). Taking $d_t=2\mathrm{m}\mathrm{m}$ as a nominal level of desired geometric accuracy, Fig. 6c shows a relatively stable level of median precision (up to ~0.95) over a broad range of ablation, whereas recall reduced from ~0.87 to ~0.56 for 120° of ablation. The range (uncertainty) in precision, however, similarly worsened for high levels of ablation, with IQR ~0.07 (CI₉₅ = 0.18) at ablation angle = 120°. Furthermore, large spatial ablations (≥120°) resulted in noisy reconstructions and poor recall and/or degenerate registrations leading to low recall (and arbitrarily high precision), evident in the cluster of points with recall ≤ 0.2. Figure 6d shows the operating characteristic in precision-recall for various settings of $d_t$. For each curve, a lower median recall and median precision was observed for increased ablation extent, and a precision was seen to decrease more slowly than recall for all $d_t$. Compared to the characteristic curves of Fig. 4d for various levels of feature density (i.e., spatially random ablations), the operating curves exhibit similar areas under the curves, suggesting robustness against either model of ablation.

Finally, an example reconstruction is shown in Figure 6(e–f) for a spatial ablation of 180° (across the anterior horn of the ventricles in this example – e.g., glare or an air bubble occluding the “upper” half the visual scene). The colorwash overlay shows the distance to ground truth ($e_{S\to x}$). The reconstruction is seen to be precise (median precision = 0.92); however, owing to the relatively strong ablation (50% of the image), the recall was poor (~0.42) due to poor coverage.

C. Experiment 3: Evaluation of Geometric Accuracy

Guided by the experiments described above for optimal /nominal parameter selection and observed uncertainty in point-cloud reconstruction, the SLAM method was evaluated in comparison to a previously reported SfM method in terms of geometric accuracy and system runtime. As shown in Fig. 7a, the median (IQR) PE for the SfM method was 0.33 mm (0.38 mm). The SLAM method achieved median (IQR) PE = 0.35 mm (0.35 mm) with no statistically significant difference measured by the test ($p=0.39$). The above was evaluated over the 4422 ground truth surface points for both methods, and while the large p-value does not prove that the PE distributions are the same, it shows that the current experiments did not detect a meaningful difference between the median values.

Fig. 7.

Evaluation of geometric accuracy of registration when using SLAM based reconstruction compared to Structure-from-Motion. (a) Projected error, |PE| (b) target registration error, and (c) System runtime per frame (for $N_I=300$ images) across reconstruction method (d) System runtime as a function of number of images, with SLAM demonstrating a major improvement in runtime (e) Colorwash overlay of projected error on the ground truth surface along with reconstructed point cloud. (f) Augmented view of the neuroendoscopic scene with vasculature and Foramen of Monro (FoM) overlaid with target BBs (yellow), green-scale depth cue, anatomical region (orange) and simulated target (purple). A video version of (f) is available as supplemental material.

Fig. 7b shows the measured TRE for both methods with either method achieving sub 2 mm error, a typical error observed in stereotactic navigation systems. The median (IQR) TRE for SfM method was measured as 0.73 mm (0.77 mm) from 30 BBs visible within the reconstructions, and the SLAM method achieving similar median (and IQR) TRE of 1.02 mm (1.16 mm) based on 18 BBs visible in the reconstruction. Again, there was no statistically significant difference between the measurements ($p=0.45$), and while this does not prove equivalence, it suggests no difference detected by these experiments. SfM and SLAM methods yielded $N_X=5933$ and $N_X=1984$ 3D points respectively, achieving a recall of 0.90 and 0.87 for a $d_t$ threshold of 2 mm.

However, as shown in Fig. 7c, a major improvement in system runtime per frame (for $N_I=300$ images) is achieved due to the parallelized architecture of SLAM method. The offline, sequential SfM method required 4.79 s for tracking and mapping a single frame on an average, while the SLAM method required only 0.29 s for tracking $\left(t_{\mathrm{t}\mathrm{r}}\right)$, and 0.91 s for mapping $\left(t_m\right)$. Vision-based tracking is pertinent to neuroendoscopic procedures, since mapping can be performed in non-critical regions of anatomy before guidance is required for targeting. The system achieved more than 16 × speedup in tracking runtime compared to SfM method.

Fig. 7d shows a similar trend of drastic reduction in system runtime ($t_{\mathit{sys}}$), observed across different $N_I$ used for reconstruction and tracking. Across all $N_I$, the SLAM method demonstrated a lower runtime compared to SfM, with over an order of magnitude of improvement measured. While SfM method has a runtime complexity of $O\left(N^3\right)$ with respect to $N_I$ [61], SLAM methods suggest an asymptotic increase in runtime due to the use of a local window for tracking and mapping.

The PE of SLAM method is visualized in Fig. 7e (along with reconstructed point cloud and BBs in green) overlaid as a colorwash on the ground truth surface. The error was observed to be unbiased across both lateral and third ventricles, with the reconstructed BBs in strong visual agreement with ground truth positions.

Fig. 7f illustrates the augmented overlay view of the endoscopic scene during tracking, with the ground truth surface in green and a target region rendered in purple. Other anatomical annotations such as the mamillary bodies on the floor of third ventricle is overlaid in orange. The BBs embedded within surface of the phantom are seen as yellow spheres, demonstrating overlap with BBs visible in green in the endoscopic view. The method enables not only tracking in the scene but permits localization of structures beyond the current endoscopic field of view. A video version of Fig. 7f is attached as supplemental material.

Finally, Table I presents the various components of the tracking thread along with breakdown of each runtime step for $N_I=300$ frames. The pose prediction step required the most time at 0.18 s (0.15 s) [median (IQR)], followed by feature detection and matching steps, requiring 0.07 s (0.01 s) and 0.04 s (0.03 s), respectively. The SLAM method thereby achieved an update rate of (1/0.29 sec) = 3.45 Hz during tracking, demonstrating the capability for real-time navigation with intermittent updates to reconstruction via the mapping thread.

TABLE I.

Tracking Runtime (t_tr) for Slam (for N_I = 300 Images)

Step	Time/frame [Median (IQR)l
Feature Detection	0.07 s (0.01 s)
Feature Matching	0.04 s (0.03 s)
Pose Prediction	0.16 s (0.13 s)
Keyframe Decision	0.02 s (0.01 s)
Runtime per Frame (tI)	0.29 s (0.17 s)
Update Rate (Hz)	1/0.29 = 3.45 Hz

IV. Discussion

A series of laboratory studies was reported using an anthropomorphic phantom of the cerebral ventricles to rigorously evaluate the performance and variability of 3D video reconstruction and to provide a quantitative basis for translation to future clinical studies. Experiment 1 informed the selection of parameters that affect feature extraction and SLAM reconstruction. Key findings showed that accurate reconstruction was obtained even with relatively small values of 6 octaves, 10 scale levels, and 500 keypoints ($N_O,N_L$ and $N_K$, respectively). A distribution in $N_F$ was observed (varying from 100 to 800 features) as a function of parameter selection, consistent with the fact that some regions of the phantom (and for that matter, real anatomy) are more feature rich than others. Extraction of larger $N_F$ was found to assist in tracking feature-sparse regions with a tradeoff of increased $t_f$ by 0.1 s for an increase in $N_O$ and $N_L$. The observed saturation in $N_F$ with increase in $N_L$ suggests that the maximal number of features required for accurate reconstruction was indeed extracted from the image. Similarly, minimal improvement in registration accuracy was observed beyond $N_K=500$ keypoints, indicating that the salient features for reconstruction and registration were captured, and further increase in $N_K$ does not improve precision. To maintain reasonable runtime that is potentially suitable to clinical application, parameter values were selected to balance $N_F$ and runtime ($t_f$ and $t_{tr}$) to achieve runtime >1 frame/sec. Other work [62], [63] also compared various feature descriptors and matching methods, including hand-crafted [53] and deep-learning based descriptors [64], and achieved $t_f$ similar to that reported above.

Experiment 2 provided information pertinent to translation of laboratory studies to clinical evaluation, where video of natural human anatomy in vivo may present different forms of limited data. The system performance in sparsely textured regions of the ventricles was evaluated by ablating the number of matched feature points, demonstrating robustness even with a loss of up to 40% of the number of features ($N_F~500$) present in the phantom studies. The system demonstrated 3D reconstruction and registration with PE <1 mm under such conditions. The uncertainty in PE was also observed to increase with a lower $N_X$, quantified by the increased IQR in PE measured as a function of ablation. The precision and recall analysis quantified the effect further, showing the extent in ablation for which a high recall (i.e., point-cloud coverage) was achieved – also consistent with the expected feature density in clinical neuroendoscopy. By contrast, noisy 3D reconstructions demonstrated lower recall and arbitrarily high precision due to degenerate registration. The precision-recall analysis quantified the accuracy achieved with various settings of threshold $\left(d_t\right)$. For a given $d_t$, the analysis showed the changes in precision and recall expected across a broad range in ablation. A relatively smoother trend was observed for $d_t=1\mathrm{m}\mathrm{m}$, suggesting that the system was robust to ablation (up to 40% of features) before suffering a substantial loss in precision and recall. The steep falloff in performance beyond $d_t=1\mathrm{m}\mathrm{m}$ indicated that feature points salient to 3D reconstruction tended to be within a threshold of 1 mm, also confirmed by measurements of PE for various ablation fractions.

Experiment 2 further evaluated the uncertainty in 3D reconstruction accuracy as a function of spatial ablation – i.e., angular sectors wholly ablated of feature content, analogous to a region of the video image being obscured by glare or other form of visual occlusion. Uncertainty increased with the extent of ablation, evident as an increase in the IQR and 95^th percentile of PE. In observational studies of clinical neuroendoscopy, the view can be observed to be commonly occluded over a range of ~30° – 90°. The system developed in this work demonstrated robust performance for ablation angles of up to ~120° (one-third of the endoscopic field of view). Larger variability (and PE) was observed with increasing ablation angle due to a reduction in $N_F$, and corresponding reduction in $N_X$. Compared to findings in the feature sparsity experiment (above), performance with spatial ablation exhibited robustness such that a ~70% loss in precision and recall was tolerated before data failed to reliably reconstruct and register. Precision-recall curves showed a smoother trend for $d_t\ge 1\mathrm{m}\mathrm{m}$, demonstrating a large operating region and gradual decline of precision with reduced recall. The experiments thus quantify the precision and recall over a range that is expected to capture challenging, realistic clinical scenarios in which feature density and/or spatial distribution are lost, and the system consistently demonstrated sub-mm registration accuracy.

Experiment 3 involved a detailed analysis of video tracking and reconstruction based on SLAM with a quantitative comparison to a previously reported SfM-based method. A similar magnitude and distribution of geometric error – both PE and TRE – were observed for each reconstruction method, and there was no statistically significant difference in mean geometric accuracy ($p>0.05$) between the methods. That both methods achieved similar geometric accuracy suggests that both SLAM and SfM yield a sufficient number and distribution of features for accurate reconstruction of the 3D scene. The current experiments involved a linear trajectory of the endoscope, which is pertinent to the anticipated clinical trajectory of a ventriculoscope traversing the lateral ventricles to the third ventricles via the Foramen of Monro along a straight path, and previous work [37] showed this trajectory to perform similarly to circle or arc trajectories. Further, the experiments were conducted using a robot for stable, reproducible positioning in quantitative performance characterization, and the method is applicable in principle either to robot-assisted or freehand motion of the endoscope. Vitally important to eventual clinical translation, a major improvement in system runtime was demonstrated with the SLAM approach, yielding up to 16x speedup compared to SfM and an update rate of 3.45 Hz. Recently, end-to-end differentiable deep-learning based methods [40] have emerged for SLAM with the inclusion of appearance and geometric priors, demonstrating similar real-time update rates. Augmented overlay of the video scene with virtual structures registered from 3D CBCT or MRI illustrated accurate alignment with visual features and the ability to visualize virtual target structures not only in the direct field of view, but beyond the ventricular walls.

Overall, the experiments provided quantitative validation in support of translation of the approach to clinical studies. Of course, such work is not without limitations. First, the ventricle phantom did not contain CSF, which varies in optical quality on a case-by-case basis in clinical situations and could cause optical distortion that could dynamically perturb camera calibration. Although the phantom is based on a segmentation model derived from real ventricular anatomy and thereby gives a fairly realistic representation of shape, it may not reflect, tissue-specific specularity or other photometric properties of the innreller ventricular wall, causing an unknown degree of mismatch between the density and spatial distribution of features in the phantom studies versus what can be expected in clinical use. The experimental studies addressed this question as a function of “limited data” - i.e., loss of feature density and regional ablation over a broad range of such variations. Furthermore, some surgical procedures (e.g., endoscopic third ventriculostomy, which involves creation of a stoma on the third ventricular floor) could introduce floating debris that could challenge reliable feature detection for 3D video reconstruction. Addressing visual drift (i.e., compounding of errors in localization and mapping over time) in SLAM also remains a topic of active research [65], which could potentially occur in procedures with longer duration. Visual drift could also be caused by tissue deformation induced after acquisition of the intraoperative CBCT or MR images, thus motivating a deformable point-cloud-to-surface registration for accurate tracking. In the current studies, the robotic arm was used to provide steady positioning and motion of the ventriculoscope for rigorous, reproducible measurements, and the suitability of the methods to true freehand motion invites future investigation. Translation to clinical studies benefits from the rigorous evaluation of key system performance characteristics, and future work will address additional, high-level clinical challenges, such as the ease in maneuvering the ventriculoscope in proximity to sensitive anatomical structures and compatibility with the overall operating setup in procedures such as ETV and transventricular DBS. Further, deep-learning based methods will be investigated for real-time feature description and end-to-end SLAM based navigation with translation to clinical studies.

V. Conclusion

The first real-time SLAM method for 3D video tracking and reconstruction in ventriculoscopy was presented. The system is integral to a RAV prototype being developed for translation to clinical studies, and the quantitative evaluation of geometric accuracy, robustness, and runtime are essential steps in such development. The parallel algorithm architecture and real-time tracking intrinsic to SLAM enabled ~16× speedup with an update rate of 3.45 Hz compared to a previously reported SfM implementation without loss in geometric accuracy. Studies conducted using an anthropomorphic ventricle phantom identified the nominal/optimal parameters for real-time guidance using SLAM, and uncertainty in geometric performance was measured using two limited data models – feature sparsity and spatial ablation, demonstrating robust performance (≤ 1 mm PE) even for large degrees of ablation (e.g., up to 40% loss of features or 120° of spatial ablation). Precision-recall curves quantify the geometric accuracy and lack of bias in 3D reconstruction/registration, with PE = 0.35 mm and TRE = 1.0 mm and no statistically significant difference compared to SfM. These studies establish a quantitative characterization of algorithm settings and performance that supports translation of the system to clinical studies.