Self-Supervised Feature Detection and 3D Reconstruction for Real-Time Neuroendoscopic Guidance

Abstract

Objective:

Transventricular approach to deep-brain targets offers direct visualization but also imparts deformation that challenges accurate neuronavigation. 3D reconstruction and registration of the endoscopic view could provide up-to-date, real-time guidance. We develop and evaluate a self-supervised feature detection method for 3D reconstruction and navigation in neuroendoscopy.

Methods:

Unlabeled neuroendoscopic video data from 15 clinical cases yielding 11,527 video frames yielding 11,527 video frames were used to train a self-supervised learning method (R2D2-E) with 5-fold cross validation integrated into a simultaneous localization and mapping (SLAM) pipeline for 3D reconstruction. A series of experiments guided nominal hyperparameters selection and evaluated performance in comparison to SIFT, SURF and SuperPoint in terms of the accuracy of feature matching and 3D reconstruction.

Results:

R2D2-E demonstrated a superior performance in feature matching and 3D reconstruction. R2D2-E features achieved a median projected error of 0.64 mm compared to 0.90 mm, 0.99 mm and 0.83 mm error for SIFT, SURF and SuperPoint, respectively. The method also improved F1 score by 14%, 25% and 22% compared to SIFT, SURF and SuperPoint, respectively.

Conclusion:

The proposed feature detection approach enables accurate, real-time 3D reconstruction in neuroendoscopy, offering robust feature detection in the presence of endoscopic artifacts and provides up-to-date navigation following soft-tissue deformation.

Significance:

The self-supervised feature detection method advances capabilities for vision-based guidance and augmented visualization of target structures in neuroendoscopic procedures. The approach could enhance the accuracy and precision of neurosurgery to improve patient outcomes.

I. Introduction

Neuroendoscopic procedures are emerging as a valuable approach for accessing deep-brain targets, offering direct visualization of structures that are typically challenging to reach through conventional methods [1], [2]. Clinical applications of the technique include endoscopic third ventriculostomy (ETV), choroid plexus cauterization, cyst fenestration or biopsy, intraventricular tumor resection, basal cistern exploration, and potentially deep-brain stimulation [3], [4], [5]. A transventricular approach is useful for accessing the targets in close proximity to lateral or third ventricles that are difficult to reach by other procedures. However, brain shift and egress of cerebrospinal fluid (CSF) during neuroendoscopic procedures induces geometric deformation of deep-brain structures, challenging accurate neuronavigation [6]. Brain shift induces deformation ranging 5-24 mm upon incision of the dura [7], primarily in regions close to the cortical surface, whereas CSF egress causes deformation about deep-brain structures in proximity to the ventricles by 4-10 mm [8]. Even with direct visualization, targets can present with inconspicuous appearance, occlusion, or location beyond the visible surface of the ventricles, leading to challenging target localization.

Conventional neuronavigation approaches for these procedures rely on rigid registration with preoperative magnetic resonance (MR) or computed tomography (CT) imaging. Both framed and frameless navigation systems utilizing optical [9] or electromagnetic tracking [10] employ fiducial markers [11] or surface delineation [12] for registration. While emerging robotic systems, such as ROSA (Zimmer Biomet, Warsaw, IN), Neuromate (Renishaw, Wotton-under-Edge, United Kingdom), and Excelsius (Globus Medical, Audubon, PA) offer advanced capabilities like tool stabilization and cooperative control, they also rely on rigid registration of the cranium, thereby limiting accuracy in the presence of deep-brain deformation. Intraoperative 3D imaging, including MR, CT, and cone-beam CT (CBCT), have been explored as means to capture accurate representations of deformed anatomy [13]. Intraoperative 3D imaging can provide clear visualization of intracranial structure, but workflow disruptions and high operational cost may limit broad utilization [14], [15]. Intraoperative ultrasound imaging presents another promising approach, offering real-time visualization of intracranial structures, but may be challenged by factors such as operator variability and poor image quality [16], [17].

Vision-based navigation has emerged as a promising method to support intraoperative tracking and guidance in various medical procedures [18]. Previous studies reported its application in bronchoscopy [19], transnasal skull-base surgery [20], and laparoscopy [21]. In these contexts, navigation was facilitated through 3D reconstruction of anatomy [22], which served as a means for registration with prior anatomical data. Two primary classes of algorithms were employed for this purpose: structure-from-motion (SfM) and simultaneous localization and mapping (SLAM). While SfM is best suited for one-time offline reconstruction, SLAM utilizes a sequence of video frames for pose estimation and reconstruction, making it potentially suitable for online, real-time navigation [23]. Previous work demonstrated development of a prototype Robot Assisted Ventriculostomy (RAV) system with 3D reconstruction and registration using SfM [24] and SLAM [25]. Recent advances in deep learning methods have also demonstrated significant promise in reconstructing 3D anatomy through dense visual tracking [26], depth estimation [20], and end-to-end volumetric reconstruction [27], with or without geometric priors [28].

Feature detection is crucial step for camera localization in both SfM and SLAM, involving extraction of distinctive features across a sequence of image frames that may be challenged by variations in lighting, viewpoint, occlusions, and other artifacts observed in endoscopy. Classical feature detection algorithms, such as SIFT [29], SURF [30], and ORB [31], are hand-crafted, deterministic representations to identify unique features represented as vector descriptors of the local image context of a detected feature location. Matching similar descriptor vectors establishes correspondence between images across frames, enabling the tracking of detected features in successive frames for estimating the camera pose.

Deep learning methods present an alternative means of feature detection that could address the limitations of classical detectors, including robustness to glare and occlusion. The development of robust feature detection methods for endoscopic imaging, particularly in neuroendoscopy, is limited by the scarcity of annotated ground truth data due to the labor-intensive process of labeling individual keypoints and establishing their correlations across images. Traditional supervised learning approaches, which typically required large amounts of labeled data, were thus challenging to implement in this context.

Self-supervised methods have emerged as a promising alternative [32], [33], [34], [35] offering the potential to leverage abundant unlabeled data available from various sources. Recent advancements in self-supervised feature detection techniques [34], [35] demonstrated significant potential for application in vision-based navigation endoscopy without the need for extensive manual annotation.

The work reported below aims to adapt self-supervised feature detection methods for navigation using 3D video reconstruction in neuroendoscopy. Specific contributions of the work include: (1) finetuning and adaptation of a data-driven, self-supervised approach for feature detection and description based on R2D2 [32] to a neuroendoscopy context; (2) the first report of clinical translation of the system for 3D reconstruction in a pilot study with 15 patients; (3) experiments evaluating nominal hyperparameter selection for feature detection and performance in image matching tasks; and (4) evaluation of geometric accuracy of 3D reconstruction in clinical data using the novel feature detector with SLAM. The robustness of feature detection against typical artifacts observed in endoscopy was also investigated. Augmentation of neuroendoscopic video with color overlay of structures registered from MR images was also implemented to demonstrate visualization of 3D structures beyond the visible surface directly for assistance in target localization.

II. Methods

A. Prototype Neuroendoscopy System for Clinical Studies

The neuroendoscopy prototype consists of components illustrated in Fig. 1. The system provides the basis for an IRB-approved clinical pilot study (IRB No.: IRB00283437) involving 15 patients undergoing left or right ETV involving neuroendoscopy of the lateral ventricles within their treatment under the standard of care. Each of the 15 cases accrued under the clinical pilot study yielded video segments of 10 s – 47 s acquired at 30 frames per second (fps), resulting in a total of 11,527 ventriculoscopic video frames. Reconstruction was based on a continuous video segment acquired in sweep of the lateral ventricles in an early phase of the procedure. The large number of frames were further augmented (style and geometric perturbations) to form a rigorous, quantitative basis for validation. 5-fold cross validation was performed for evaluating the geometric accuracy, where each of the five folds comprised 12 cases for training and 3 cases held out for validation. The held-out cases served as ground truth such that all cases were eventually analyzed for validation. Pre-operative T1-weighted MR images were collected from each patient in the study (along with intraoperative CBCT when available), and features of interest were segmented as detailed below [Fig. 1(c)]. The ventriculoscope was calibrated intraoperatively using a checkerboard pattern mounted on a calibration jig [36] as shown in Fig. 1(b). With the patient positioned supine – optionally with a radiolucent frame or headboard compatible with intraoperative CBCT – a straight 6° ventriculoscope (Lotta 28164LA, Karl Storz, Tuttlingen, Germany) was coupled with the camera (Image1HD H3-Z TH100, Karl Storz, Tuttlingen, Germany) for intraoperative neuroendoscopy. The endoscopic view was displayed on the integrated monitor (Image1 Hub, Karl Storz, Tuttlingen, Germany) – e.g., as illustrated in Fig. 1(d) – and the video stream was recorded using the Advanced Image and Data Acquisition/Archiving System (AIDA, Karl Storz, Tuttlingen, Germany) for offline analysis.

Fig. 1.

System and data acquisition for clinical neuroendoscopic procedures (a) Patient setup, ventriculoscope, and ventriculoscopic display. (b) Intraoperative ventriculoscope calibration setup and the custom calibration jig. (c) Pre-operative MR image with corresponding deep-brain structures segmented for the right hemisphere. (d) Endoscopic image showing the Foramen of Monro, choroid plexus, and adjacent neurovasculature. The ability to accurately overlay structures from (c) within real-time venticulostomy (d) could improve targeting in an emerging range of neuroendoscopic interventions.

An overview of the SLAM reconstruction pipeline with the proposed self-supervised feature detection network is shown with notation in Fig. 2(a). Preoperative MR images were denoted by the coordinate frame $M$, including segmentation of anatomical structures of interest, such as surfaces of the lateral and third ventricles, denoted $S$. When available, intraoperative CBCT imaging was denoted by the frame $CT$. Neuroendoscopic video data $𝒥=\{I_i:i=1\dots N_i\}$ comprised individual frames $I_i$ acquired over $N_i$ frames and taken as input to the feature detection and 3D point-cloud reconstruction methods described below. The resulting 3D point-cloud was denoted $P=\{X_k:k=1\dots n_k\}$, where $X_k\in {\mathbb{R}}^3$. The notation $T_A^B$ denotes the homogenous rigid transformation locating frame $B$ with respect to frame $A$. $T_M^S$ was obtained by segmentation of the lateral ventricles in the MR image, forming the ground truth surface in the MR frame. The transform of each video frame with respect to the point-cloud reconstruction was similarly denoted $T_{I_i}^P$. The 3D point cloud was registered to surface segmentations via $T_P^S$, and 3D annotations of target structures were fused with the endoscopic video via $T_M^{I_i}=T_M^S{(T_P^S)}^{-1}{(T_{I_i}^P)}^{-1}$. Alternatively, the registration of preoperative MR to intraoperative CBCT [37] (when available) yields the transform $T_{CT}^M$. The ventricular surface and segmented structures were then localized in the CBCT frame as $T_{CT}^S=T_{CT}^M{T}_M^S$. Target annotations were then fused with endoscopic video via $T_{CT}^{I_i}=T_{CT}^S{(T_P^S)}^{-1}{(T_{I_i}^P)}^{-1}$.

Fig. 2.

3D reconstruction and video fusion using self-supervised keypoint detection and description. (a) Flowchart depicting the steps for data acquisition, feature detection and 3D reconstruction. Integration of ventriculoscopic video with reconstruction methods provides direct fusion of planned targets with the live endoscopic video. (b) Dual-branch self-supervised network architecture for feature detection, with the upper branch computing descriptors for input image, and the lower branch detecting keypoints at the locations of maximal repeatability and reliability (c) Keypoint error estimation in feature matching using simulated warping of images.

As shown in Fig. 2(b), the network extracts features which are then matched across image pairs, $I_i$ and $I_{i+1}$, to obtain the correspondences between two images. These matches were filtered using MAGSAC [38] with a homography model to estimate the relative pose of the camera frame between two images, $T_{I_i}^{I_{i+1}}$. The triangulation of these features (with camera intrinsics obtained from calibration) yields the sparse 3D point cloud $(P)$ with bundle adjustment and statistical outlier filtering [39] to reduce noise from mismatched features. The point-cloud was then registered using trimmed iterative closest point registration (ICP) for video fusion and visualization of the target structures directly in the endoscopic view.

B. Feature Detection and Description

Classic feature detection via SIFT and SURF, along with a learned feature detection method [40] served as a basis of comparison to the proposed approach detailed below. For both SIFT and SURF, keypoint localization was performed by identifying local extrema (minima and maxima) in the scale space, ensuring that detected features were stable across different scales. Each keypoint was assigned an orientation based on the local gradient directions in its neighborhood, providing invariance to image rotation. A 128-dimensional descriptor was obtained for each SIFT keypoint by computing the histograms of gradient magnitude and orientations in a local window. Similarly, 128-dimensional SURF descriptors were computed according to their Haar wavelet responses in the local window centered on the keypoint location. The descriptors from different images were matched using L2 distance metric to establish correspondences between detected features.

Deep learning methods present an alternative means of feature detection that could address known limitations of classical detectors, including robustness to glare and occlusion. The recently reported learning-based feature detection approach, SuperPoint [40], employs a fully convolutional neural network architecture that jointly performs interest point detection and description in a single forward pass. The network consists of a shared encoder that processes a grayscale input image, followed by two decoder heads - one for interest point detection and another for descriptor computation. The detector head outputs a tensor of feature probabilities for each pixel, while the descriptor head generates a semi-dense grid of 256-dimensional descriptors. The training procedure involves (1) basic corner detection on a synthetic dataset of simple geometric shapes and (2) self-supervised homographic adaptation applied to adapt the method to real-world images. Step 2 generates multiple random homographic warps of each training image as pseudo-ground truth labels. SuperPoint was adapted to the neuroendoscopy following context Step 2 using the clinical dataset described above and the same partitioning of 12 cases held for training, 3 cases for validation, and 5-fold cross validation over all cases. The same camera calibration, image extraction, and cropping steps were followed, and as described below, data augmentation entailed spatial variations (randomized resizing, skew transformation, and rotation) and intensity variations (randomized glare, noise, brightness, hue, contrast, and saturation) applied for each image during training.

The R2D2 [32] fully convolutional neural network was adapted in this work to endoscopic video, referred to as “R2D2-E” and developed to jointly learn keypoint detection, local descriptors, and descriptor reliability from endoscopic training data. The architecture is similar to R2D2, as shown in Fig. 2(b), and predicts three outputs: dense descriptors, a repeatability heatmap for keypoint detection, and a reliability heatmap for descriptor discriminability. The network consists of a shared backbone of weights that couples the tasks of keypoint detection and description. The R2D2-E architecture used an L2Net backbone [41] with 7 convolutional blocks, with downsampling achieved by stride 2 convolution. The last 8×8 convolutional layer was replaced by three 2×2 convolutional layers to reduce the number of weights. Batch normalization was used after each convolutional layer along with ReLU activation. The shared backbone forms a 128-dimensional tensor, serving as input to a detector head. The detector head results in two branches following a 1×1 convolution and a softmax operation to form the repeatability heatmap denoted $R_P\in [0,1]:\{\mathrm{W}\times \mathrm{H}\}$, and the reliability heatmap denoted by $R_L\in [0,1]:\{\mathrm{W}\times \mathrm{H}\}$. Simultaneously, the backbone serves as an input to a ${\mathrm{L}}_2$ normalization layer resulting in the per-pixel descriptors tensor denoted $D\in [0,1]:\{H\times W\times 128\}$. The loss function for the descriptor branch is based on a differentiable Average Precision (AP) metric [42] that learns to rank descriptors for matching. The repeatability head was trained to encourage consistent feature localization at local image maxima across viewpoints, whereas the reliability heatmap was trained jointly with the descriptor loss to avoid ambiguous or repetitive regions across different images.

R2D2-E was trained using self-supervision across pairs of endoscopic images, augmented to impart a broad variety of image characteristics in the training set, including blur, illumination glare, and perspective distortion. Given an image $I_1:\{\mathrm{W}\times \mathrm{H}\}$, the region outside the endoscopic circular view is masked, and a spatial transformation was applied (denoted $T_{12}$, applying perspective transformations), followed by an image domain transformation (denoted $H_{12}$, imparting effects of blur, illumination glare, and color shift). The resulting image, $I_2$, has a known ground truth transformation mapping the pixels from the original image. Corresponding image patches denoted $[p]\in P:\{N\times N\mathrm{px}\}$, were sampled from both images for training the network. Prior work [32] details the individual loss terms along with weighting factors between losses.

During inference, the input image was masked to the endoscopic circular field and processed by the network to obtain the repeatability, reliability and descriptors scores in a single forward pass. The feature confidence score was computed as $R_P\times R_L$ and thresholded, followed by non-maximum suppression to select a single feature located at pixel $(x,y)$ from a patch $p$, thus allowing for choice in the density of features by varying the patch size. The corresponding ${\mathrm{L}}_2$-normalized feature descriptors were extracted at the feature locations, yielding $N_f$ features denoted ${ℱ}_i=\{(f_j(x,y),D_j):j=1:N_f\}$. Feature matching was performed between a pair of images with features $({ℱ}_1,{ℱ}_2)$ using L2 descriptor vector distance as the matching criterion, resulting in $N_M$ one-to-one matches. These matches were further filtered using MAGSAC [38] with a confidence threshold of 0.99, a reprojection error threshold of 1.5 px, and 1000 maximum iterations. The resulting filtered matches were used to obtain a subset of $N_{FM}$ matches consistent with the image homography.

C. 3D Surface Reconstruction and Registration

The pipeline in Fig. 2(a) was developed for real-time 3D video reconstruction and video fusion, consisting of two major components: feature detection and matching, followed by 3D reconstruction and registration – the latter using PySLAM v2 [43], OpenCV, and TREK [44] in a parallel multithreaded architecture for tracking and mapping. As part of the tracking thread, the image frame was taken as input to R2D2-E to compute features and corresponding descriptors $({ℱ}_i)$ for each image $(I_i)$. Matching was then performed for features detected in consecutive frames, followed by filtering. Successfully matched features were then designated as keypoints, and based on the fraction of keypoints tracked in the current frame (compared to the previous frame), the frame was designated a keyframe and inserted into a pose graph for mapping.

In parallel with the tracking thread, the mapping thread observes the inserted keyframes in the pose graph to reconstruct the 3D point cloud. After obtaining matched feature correspondences between image pairs, epipolar geometry constraints were applied to estimate the relative camera motion and 3D structure. For a known set of corresponding features, $f_1$ and $f_2$ in two images, the essential matrix $(E)$ was estimated using the five-point algorithm with MAGSAC [38], requiring at least five point-to-point correspondences using the epipolar constraint: ${f_2}^{\mathrm{T}}{K}^{-\mathrm{T}}E{K}^{-1}{f}_1=0$, where $K_1$ and $K_2$ are the camera intrinsic matrices obtained from calibration. The essential matrix encodes the relative rotation $(R)$ and translation $(t)$ between the two camera positions $(T_{I_1}^{I_2})$ decomposed as $E={[t]}_{\mathrm{x}}R$, where ${[t]}_{\mathrm{x}}$ represents the skew-symmetric matrix of the translation vector. For each point triangulated with homogeneous 3D coordinates denoted $X$, a linear system of equations was solved: $[f_1{P}_1]X=0$, and $[f_2{P}_2]X=0$, where $P_1$ and $P_2$ are the projection matrices for the two views. This system was solved using Direct Linear Transformation (DLT) [45] to yield the triangulated points, followed by nonlinear optimization to minimize the geometric reprojection error:

$$\text{min}{\sum }_{I_i}{\Vert f_{1i}-\pi (P_1{X}_i)\Vert }^2+{\Vert f_{2i}-\pi (P_2{X}_i)\Vert }^2$$ (1)

where $\pi$ represents the perspective projection. The resulting 3D points and camera poses were further refined through bundle adjustment in a local window of keyframes to minimize the total reprojection error across multiple views simultaneously. Outlier points were filtered according to realistic system geometry (e.g., minimum or maximum focal length and triangulation angles). A local window of 10 frames was used for bundle adjustment, and global bundle adjustment was not performed. Keyframes in which more than a 0.9 fraction of keypoints were observed in at least 3 other keyframes were termed redundant and removed to reduce computational load. Finally, the resulting point cloud was refined using statistical filtering to remove outliers, returning a sparse point cloud consisting of $N_X$ 3D points, denoted $P=\{X_i:i=1:N_X\}$. R2D2-E was integrated with PySLAM [43] for feature detection.

The 3D point-cloud was registered using trimmed ICP [46] with structures segmented from MRI via a point-to-plane metric. The resulting transformation $(T_P^S)$ precisely maps the point-cloud to the surface and the MR imaging ($T_M^P=T_M^S{(T_P^S)}^{-1}$). The individual camera pose and video frames were then localized with respect to MR surface as $T_M^{I_i}=T_M^P{(T_{I_i}^P)}^{-1}$, enabling augmentation of the video scene with the surgical plan, target structures and any critical anatomy as annotated in the MR imaging.

III. Experimental Methods

A. Training Data Generation

The dataset has diverse anatomy and different endoscopic imaging artifacts. MR images were automatically segmented using Fastsurfer [47], [48] to annotate target structures such as the ventricular surfaces, amygdala, thalamus, and nucleus accumbens. Vasculature such as the distal basilar artery and proximal posterior cerebral arteries were manually segmented. Video data were corrected for distortion according to intraoperative geometric calibration [Fig. 1(b)] and cropped to the endoscopic field of view. Video frames were randomly resized between 384 and 1024 px length maintaining aspect ratio, followed by cropping to $(W,H)$ = (256 px, 256 px). Spatial augmentations $(T_{12})$ included a perspective skew transform ranging randomly over the range $[0,0.4]\ast W$. Images were further rotated randomly over ±180°. Image domain (intensity) augmentations included the addition of random distributions in glare: [0, 0.8], pixel noise: [0, 30 px], brightness: [0, 0.2], hue: [0, 0.1], contrast: [0, 0.3], and saturation: [0, 0.3]. The applied spatial augmentations were chosen to reflect a range of commonly observed variations in endoscopic procedures. Endoscopic artifacts such as glare, partial occlusion, bleeding and smoke from cauterization were partially reflected in augmentation via local intensity perturbations to create a more robust training routine.

Each fold of the network was trained on a workstation with a single NVIDIA Titan RTX GPU with 24 GB memory. 3D reconstruction (SLAM) was performed on a separate workstation with NVIDIA GTX 980Ti GPU with 6 GB memory.

B. Experimental Evaluation and Validation

A series of experiments was conducted to optimize and evaluate the performance of the R2D2-E feature detector. The first experiment focused on hyperparameter selection and validation in the image domain. The second experiment evaluated the accuracy of keypoint detection and matching. The final experiment assessed the 3D geometric accuracy of reconstruction and registration.

1). Selection of Network Hyperparameters

The first experiment was designed to inform nominal selection of hyperparameters affecting feature detection and matching. The network was first pretrained on natural images of indoor and outdoor scenery as done by [32]. The loss terms for repeatability and reliability were equally weighted, and other image sampling parameters were also set to the recommended values as in prior work [32]. A fixed batch size of 6 was selected according to GPU memory constraints, with the network being trained for 30 epochs. The network was trained using the Adam optimizer with $\lambda =5\times {10}^{-4}$, ${\beta }_1=0.9$, ${\beta }_2=0.999$, and $\epsilon =1\times {10}^{-8}$. To adapt the network to endoscopic imaging, the descriptor loss weighting factor $\kappa$ was set to 0.2 for epoch 1, with an increase of 0.01 per epoch. Two univariate studies were conducted with: 1) the learning rate (LR) for the network varied as a parameter over the range [10⁻⁵, 10⁻²]; and 2) the patch size $[p]$ varied as a parameter over the range [4, 128] to assess the effect of feature density on error.

Performance of the feature detector was evaluated in terms of the following metrics. The median validation loss across epochs 20-30 was used to assess the convergence of network optimization. The number of features obtained for each patch size was noted as a measure of feature density, and the fraction of matches $(F_{match})$ for a pair of images $(I_1,I_2)$ and a corresponding set of features presenting with $N_{1f}$ and $N_{2f}$ features in each image respectively, resulting in $N_M$ successful matches was computed as:

$$F_{match}(\%)=\frac{N_M}{\text{min}(N_{1f},N_{2f})}\times 100$$ (2)

The number of inlier matches (following MAGSAC filtering) was denoted by $N_{FM}$, and the filtered match percentage $(F_{match-f})$ was computed as:

$$F_{match-f}(\%)=\frac{N_{FM}}{N_M}\times 100$$ (3)

Finally, the keypoint error for features $f_j$ and $f_k$ in $I_1$ and $I_2$ respectively, as shown in Fig. 2(c), was defined as the error between successfully matched and filtered keypoints on the image:

$$KPE=\Vert T_{12}{f}_j(x_j,y_j)-f_k(x_k,y_k)\Vert$$ (4)

2). Evaluation of Feature Matching Accuracy

With parameter selections determined by the previous experiment, the performance in feature matching was evaluated for SIFT, SURF, and R2D2-E approaches. The number of features extracted from the image was chosen as the primary experimental variable, measuring the effect on the fraction of keypoints that were successfully matched across a series of video frames, defined as:

$$K_{match}(\%)=\frac{N_{FM}}{\text{min}(N_{1f},N_{2f})}\times 100$$ (5)

The resulting fraction of matched keypoints (before and after filtering) was compared while varying parameters that affect the number of detected keypoints, namely: 1) the contrast threshold for SIFT; 2) the Hessian threshold for SURF; and 3) the confidence score threshold for R2D2-E. For each method, the number of features extracted was varied over the range [500, 5000] in increments of 500 by varying the above thresholds. R2D2-E was trained with 5-fold cross validation, retaining 3 cases in each fold for validation. The KPE was quantitatively evaluated with these cases, resulting in performance validation for all cases individually. Finally, the distribution in track lengths was evaluated for each method. The accuracy of feature matching was evaluated in terms of KPE as well as qualitative assessment across a range of endoscopic views varying in quality (e.g., scenes with blur or glare) and content (scenes varying in feature density from relatively sparse to dense).

3). Evaluation of 3D Geometric Accuracy

The third experiment assessed the geometric accuracy of 3D point cloud reconstruction with the various feature detection methods as the primary experimental variable. For all methods, the reconstructed point cloud was filtered using $k_{nn}=5$ neighbors and ${\sigma }_{nn}=0.1\text{mm}$, filtered for 3 iterations, and the trimmed ICP fraction of overlap $(\gamma )$ was set to 0.95. Registration was taken to be converged for tolerance limits in rotation $({\Delta }_r<{0.05}^{°})$ and translation $({\Delta }_t<0.01\text{mm})$ between successive iterations. CloudCompare [49] was used for point-cloud filtering and registration.

The geometric accuracy of registration was evaluated in terms of projected error (PE). The right lateral ventricular surface was segmented from MR imaging using previously established methods [50], [51] to provide the ground truth surface, denoted $S$. For a given point $X_i$ in the cloud and closest point $S_i$ on $S$, PE is:

$$\text{PE}={\text{proj}}_S(X_i-S_i)$$ (6)

where ${\text{proj}}_S$ is the projection of error on $S$, followed by $k$-nearest neighbor interpolation over the surface with $k=5$. To reduce extrapolation artifacts, $S$ was truncated to the region within a convex hull of the reconstructed point cloud. In a similar manner as noted in Section III.B.2, the PE was evaluated for all methods across the same 15 cases using 5 fold cross validation, with 3 cases held out for each fold.

Further assessment of the point cloud reconstruction to quantify its accuracy and completeness was performed in terms of precision $P(d_t)$ and recall $R(d_t)$ as described in [52] with respect to a variable distance threshold $(d_t)$. For a reconstructed point $X_i\in 𝒳$, the distance to $S$ was:

$$e_{X\to S}=\underset{S_i\in S}{\text{min}}\Vert X_i-S_i\Vert$$ (7)

Precision is therefore the fraction of points from the point-cloud reconstruction $(𝒳)$ to the $S$ within the distance threshold $(d_t)$:

$$P(d_t)=\frac{1}{N_X}{\sum }_{i=1}^{N_X}[e_{X\to S}<d_t]$$ (8)

where the square brackets $[]$ represent the Iverson binarizing operator, giving $P(d_t)$ in the range [0, 1]. Similarly, for a ground truth point $S_i\in S$, the distance to the point-cloud reconstruction $(𝒳)$ is:

$$e_{S\to 𝒳}=\underset{X_i\in 𝒳}{\text{min}}\Vert S_i-X_i\Vert$$ (9)

yielding recall as the fraction of points from $S$ to $𝒳$ within threshold $d_t$:

$$R(d_t)=\frac{1}{N_S}{\sum }_{i=1}^{N_S}[e_{S\to 𝒳}<d_t]$$ (10)

with $R(d_t)$ in the range [0, 1]. Precision and recall therefore form a basis for assessing system performance under an accuracy requirement $(d_t)$, similar to an operating curve. F1 score was also computed as the harmonic mean of precision and recall for each distance threshold, as derived in prior work [53].

Finally, visualization of the resulting registration in endoscopic video was achieved by overlay of target structures annotated in preoperative MR images (such as the amygdala, thalamus and nucleus accumbens) transformed to the camera frame. Blender [54] was used to define the 3D scene and camera poses, and segmented structures were rendered in alpha-blended overlay for direct visualization in the endoscopic field of view.

IV. Results

A. Selection of Network Hyperparameters

Figure 3 summarizes the effect of network hyperparameters on validation loss and image matching accuracy, with 5-fold cross-validation during training resulting in 15 data points for each boxplot shown. The validation loss achieved for different learning rates (LRs) was evaluated, using the default N=16 px as done in [32], as shown in Figure 3(a-c). A slight decrease in loss was observed with increasing learning rates, with a loss of 1.06 achieved for LR=10⁻³. However, for larger LRs, the network failed to converge due to gradient explosion, resulting in final losses of 1.56 and 1.79 for LRs of 5 × 10⁻³ and 10⁻², respectively. Figure 3(b) shows the effect of LR on the match fraction, showing an increase in both $F_{match-f}$ and $F_{match}$ up to a maximum of 84% and 41%, respectively, with an increase in LR up to 10⁻³. Beyond this value, the network failed to reliably predict features, resulting in no matches. Similarly, larger LRs resulted in lower median KPE up to an LR of 10⁻³, as shown in Fig. 3(c), for which a KPE of 1.01 px was achieved. Notably, for the successfully trained networks, a KPE of less than 2 px was consistently achieved. Based on these findings, a nominal LR of 10⁻³ was selected for further experiments.

Fig. 3.

Hyperparameter selection and sensitivity analysis for feature detection and description by evaluation of (a) validation loss (b) successful matches (c) keypoint error as a function of learning rate $(\alpha )$ and (d, e, f) as a function of patch size (N) for keypoint detection.

Figure 3(d-f) demonstrates the sensitivity of validation loss and match accuracy with respect to the patch size. Selecting smaller patch sizes, such as 4 px and 8 px, resulted in a larger number of features; however, the corresponding networks did not converge sufficiently, with final losses of 2.08 and 1.51, respectively. Conversely, larger patch sizes led to lower final losses and a reduced number of features, as shown in Fig. 3(d). Figure 3(e) further illustrates the variation of $F_{match-f}$ with increasing patch sizes. While the $F_{match}$ values were comparably similar for the evaluated patch sizes, a sudden increase in $F_{match-f}$ was observed for $\mathrm{N}>8$ px. A maximum $F_{match-f}$ of 94.5% was achieved for a patch size of 128 px, albeit with a tradeoff of obtaining only 167 features. The KPE was observed to reduce minimally with larger patch sizes, varying from a maximum of 1.03 px for N=8 px to 0.94 px for N=128 px. To achieve a feature density and accuracy consistent with clinical applications, a nominal value of N=32 px was selected for further experiments, yielding a median of 3142 features and a KPE of 0.97 px.

B. Evaluation of Feature Matching Accuracy

A quantitative evaluation of R2D2-E for feature matching tasks was performed and compared against SIFT and SURF, as shown in Figure 4. The sensitivity of matched keypoints $(K_{match})$ with respect to the number of extracted features $(N_F)$ was demonstrated in Figure 4(a). A sharp reduction in matched keypoints fraction $(K_{match})$ was observed for both SIFT and SURF with an increase in $N_F$, varying from 18% and 16% respectively for 500 features to 5.8% for 5,000 features. $K_{match}$ using SuperPoint reduced gradually in comparison, varying from 12.1% for 500 features to 7.0% for 5000 features. Classical feature detectors outperformed SuperPoint in scenarios with $N_F<3000$, beyond which SuperPoint was marginally better. In contrast, R2D2-E successfully matched a higher number of features across all $N_F>1000$, with or without filtering. R2D2-E achieved a higher and exhibited a gradual decrease from 13.55% for 500 features to 8.3% for 5,000 features.

Fig. 4.

Assessment of keypoint detection and matching accuracy: Median matching accuracy (validation) measured as a function of epochs compared to classical feature detectors (a) before and after filtering, demonstrating a superior performance following training. (b) KPE of self-supervised method compared to classical feature detectors across 5 folds. (c) Track lengths (no. of images) following feature matching for each feature detector.

Figure 4(b) showed that SIFT, SURF and SuperPoint achieved median KPEs of 2.20 px, 1.70 px, and 1.46 px respectively, while R2D2-E achieved a superior, sub-pixel median KPE of 0.83 px with fewer outliers.

The performance during feature tracking was quantified in Figure 4(c), showing the track lengths measured in the number of images, where a larger track length signified better performance. The median (IQR) track length for SIFT, SURF and SuperPoint methods was measured as 9 images (6 images), 11 images (9 images) and 13 images (7 images) respectively. R2D2-E demonstrated a major improvement in track length, with a median (IQR) of 19 images (7 images).

The qualitative performance of the methods was visualized in Fig. 5, with (a) showing a feature-dense endoscopic scene viewing the ventricular adhesions and (b) depicting a feature-sparse lateral ventricle surface with additional simulated artifacts. Successfully matched features with a KPE of less than 1 px were shown in green, while matches with a KPE greater than 1 px were shown in red. For the same number of extracted features, R2D2-E demonstrated a higher fraction of correct matches $(F_{match-f})$ and a lower KPE, with features evenly distributed across the image.

Fig. 5.

Qualitative assessment of keypoint detection on original and simulated images for (a) feature dense and (b) feature sparse regions of lateral ventricles. Successful matches (KPE of <1 px) are depicted in green, whereas matches with higher errors (KPE of >1 px) are marked in red. Blue arrows mark example areas of interest. SIFT and SURF tended to cluster features around high contrast structures (including artifacts). R2D2-E tended to compute more evenly distributed features associated with anatomical landmarks and vasculature (and better avoiding artifacts), resulting in successful more matches across the entire image.

SIFT and SURF methods typically detected high contrast structures and edges as features, shown in Fig. 5(c) by blue arrows, resulting in feature clustering around tissues such as the choroid plexus. In contrast, R2D2-E detected more evenly distributed features that appear more clearly associated with anatomical landmarks such as vasculature and other intraventricular anatomy, and assigned low confidence to regions affected by artifacts.

C. Evaluation of 3D Geometric Accuracy

The geometric accuracy of R2D2-E for 3D reconstruction was evaluated in comparison to other methods as shown in Fig. 6. Projected Error (PE) following reconstruction is shown in Fig. 6(a), with all methods achieving sub 2 mm accuracy, a typical error seen in stereotactic navigation systems. The median (IQR) PE for SIFT and SURF were 0.90 mm (1.45 mm) and 0.99 mm (1.60 mm) as evaluated on 7 and 13 cases respectively, with 6 cases which failed to track and reconstruct for SIFT implying a success rate of 54%. SuperPoint achieved a relatively lower PE of 0.83 mm (1.26 mm) as evaluated on 10 cases implying a success rate of 77%. R2D2-E achieved a lower PE and reduced variability across all cases, with a median of 0.64 mm (IQR: 0.84 mm) and successful reconstructions for all 13 cases. The median (IQR) runtime for feature detection with R2D2-E was 0.39 s (0.01 s), with a tracking runtime of 1.11 s (0.06 s) per frame, permitting tracking at 0.90 fps in the current implementation.

Fig. 6.

Evaluation of geometric accuracy of registration when using SLAM based reconstruction with R2D2-E feature detection, compared to other feature detectors. (a) Projected error (PE) for different reconstructions (b) F1 score (c) Precision (d) Recall for point-cloud reconstructions.

The F1 score was used to further evaluate the point-cloud reconstructions as shown in Fig. 6(b). As the distance threshold $(d_t)$ increased, the F1 score gradually improved for all methods. At $d_t$ values of 1 mm and 2 mm, SIFT achieved F1 scores of 0.63 and 0.83, while SURF achieved scores of 0.48 and 0.75, respectively. SuperPoint achieved scores of 0.59 and 0.91 at the same distance thresholds, indicating better performance than SURF. R2D2-E outperformed both methods for all distance thresholds, consistently maintaining a smaller IQR and achieving scores of 0.72 and 0.94 for $d_t$ values of 1 mm and 2 mm, respectively. This represents an improvement of 14%, 25%, and 22% in F1 scores compared to SIFT, SURF, and SuperPoint, respectively, for $d_t=1\mathrm{mm}$.

The precision and recall breakdown is shown in Fig. 6(c-d). Precision gradually improved for all methods with an increase in $d_t$. SIFT and R2D2-E achieved similar median precision up to $d_t=1.5\mathrm{mm}$, while SURF and SuperPoint exhibited lower overall precision. R2D2-E achieved a precision of 0.66 (0.26) and 0.91 (0.10) for $d_t$ values of 1 mm and 2 mm, respectively, with a consistently lower IQR compared to other methods. In terms of recall, R2D2-E consistently outperformed the other methods for $d_t\le 1.5\mathrm{mm}$, demonstrating a drastic increase between $d_t$ of 0 to 1 mm and achieving a recall of 0.81 (0.14) and 0.97 (0.06) at 1 mm and 2 mm, respectively.

The segmentation and annotation of typical neuroendoscopic targets in MR imaging, along with the augmented overlay view of the right lateral ventricle, are illustrated in Figure 7. Following point-cloud registration, target structures surrounding the lateral ventricles including accumbens (cyan), amygdala (yellow), pallidum (green) and thalamus (magenta) were rendered and overlaid with depth cues as intensity. The augmented visualizations are demonstrated for a feature dense scene (Fig. 7a) along with a feature sparse scene (Fig. 7b).

Fig. 7.

Augmented neuroendoscopic guidance with corresponding structures shown in MR imaging as demonstrated with a (a) feature-dense scene and (b) feature-sparse scene. Deep-brain targets such as accumbens (cyan), amygdala (yellow), pallidum (green) and thalamus (magenta) were augmented with depth cues as intensity for overlay. (c) Critical structures in neuroendoscopy such as the distal basilar artery and proximal posterior cerebral arteries are shown in red, providing spatial context and depth cues in endoscopic third ventriculostomy.

Critical structures in neuroendoscopy such as the distal basilar artery and proximal posterior cerebral arteries were also augmented with the neuroendoscopic view as shown in Fig. 7(c), demonstrating localization of targets both within and beyond the neuroendoscopic field of view to provide valuable spatial context for the surgeon. A video version of the augmented overlay is provided as supplemental material to demonstrate the dynamic nature of the visualization and its potential to enhance intraoperative guidance during neuroendoscopic procedures.

V. Discussion

A series of experiments were conducted to quantitatively evaluate the performance of R2D2-E for feature matching and 3D video reconstruction in clinical data. The first experiments informed the selection of nominal parameters for feature detection and matching in neuroendoscopic images. The findings showed that accurate feature detection was achieved with $\mathrm{LR}\le 1\times {10}^{-3}$, and higher LR resulted in failure to converge, lower $F_{match}$, and KPE >1 px. Analysis of patch size further showed an inverse relationship between $N_F$ and patch size. A strong drop in $F_{match-f}$ was observed for patch size <16 px, associated with noisy feature sets that were unreliable for image matching. Larger patch size similarly reduced KPE at the cost of reduced $N_F$, and a nominal patch size of 32 px was determined to best balance the tradeoffs between quantity and quality of keypoints for robust scene mapping and 3D reconstruction. Previous work reported a similar tradeoff between accuracy and $N_F$ [32] in outdoor video scenes and achieved similar matching performance.

The feature mapping performance of R2D2-E was superior to both: classical methods like SIFT and SURF and SuperPoint, the learning based method, as evidenced in the second set of experiments. R2D2-E extracted strong features across a range of $N_F$, while other methods showed a sharp decline in $K_{match}$ with larger $N_F$, indicating unreliable features. R2D2-E also demonstrated a higher and more stable matching rate and achieved lower median KPE with reduced variance. The resulting improvement in homography and camera pose estimation following matching by R2D2-E could potentially reduce error propagation associated with scale/camera drift compounded over time – a common problem in traditional endoscopic SLAM systems [55]. If longer video segments are available, prior loop detection and closure methods [56], [57] could further mitigate localization errors and correct for deviations with a continuous map update computed over a separate thread. As shown by arrows in Fig. 5(c), unlike classical methods that tend to cluster keypoints in high-contrast regions such as the choroid plexus, R2D2-E exhibited a more even distribution of keypoints over clinically relevant structures such as vasculature and enabled robust tracking in the presence of occlusions. The method intrinsically assigned low reliability and repeatability to regions with image artifacts, thereby rejecting keypoints in areas of glare or blur. The longer track length distribution and higher median also demonstrated robust tracking of features across a series of video frames, which is crucial for consistent localization and reconstruction in SLAM. These findings suggest that R2D2-E not only outperformed other evaluated methods in terms of accuracy but also provided a more robust and clinically practical approach to feature detection and matching in neuroendoscopic applications.

3D reconstructions performed with R2D2-E features consistently outperformed those from SIFT and SURF, with lower median and IQR in PE. All feature detectors resulted in registration accuracy of PE < 2 mm, which may be clinically acceptable, but only in cases where the reconstruction contained a large number of points and registration was successful. R2D2-E demonstrated more consistent tracking and stable reconstructions with all reconstructions successfully completed compared to other methods. R2D2-E yield in a slight improvement in PE and, more importantly, was much less susceptible to failure in real data. The augmented frame shown in Fig. 5(c) compares the detected features and track lengths improvement with R2D2-E. Prior work reports similar values of error and tracking runtime as reported here [22], and as evaluated in phantom studies of varying anatomy using SLAM. The robustness of R2D2-E was evident in higher median F1 score and reduced IQR across all tested distance thresholds. While reconstructions with classical features demonstrated a high level of precision in some instances, they were subject to larger variability and failure rate compared to R2D2-E. The superior recall achieved with learning-based methods (SuperPoint and R2D2-E) was attributed to a more even distribution of keypoints across the entire surface of the lateral ventricle. The superior precision of R2D2-E was attributed to its network architecture and augmentation techniques, and superior recall was attributed to the more uniform distribution of keypoints across the image.

The present study has several limitations. First, the dataset size of 15 cases, while sufficient for early validation studies, may not fully represent the spectrum of anatomical variations and surgical scenarios encountered for testing in clinical practice. Although the cases exhibited diverse patient-specific anatomical features in both endoscopic and MR imaging, thereby partially mitigating this limitation, robustness of the approach warrants evaluation in future work with larger, diverse datasets featuring more anatomical variations. While the method appears well suited to procedures involving direct visualization of the lateral ventricles (e.g., ETV, CPC, intraventricular tumor resection, and deep brain stimulation), further investigation could also address the generalizability of the trained network across other neuroendoscopic procedures. By comparison, classical feature detectors without reliance on learned exemplars may be intrinsically more generalizable across different anatomical regions. 5-fold cross-validation during training improved the robustness in the challenging scenario investigated in this work, and future studies could incorporate additional data to improve generalizability. The specific parameter values for image augmentation could also be tuned for generalizability across other endoscopic procedures. While our overlay visualization approach demonstrated feasibility, the current implementation lacks sophisticated depth cues such as motion parallax or depth contours, which could enhance surgeons’ perception of subsurface structures. Future work could further improve augmented visualization using an interactive display and viewpoint rendering to improve depth perception of the overlaid 3D structures. Other methods to improve depth perception could use stereoscopic endoscopes and displays or use haptic depth feedback.

Several strategies employed in this work for implementation of the R2D2-E pipeline may prove valuable for future researchers. Careful management of PySLAM dependencies and version control was crucial, particularly when integrating multiple libraries. Interfacing between Python-based deep learning frameworks and C++-based computer vision libraries employed custom binding wrappers to ensure interoperability. Precise calibration of the endoscopic camera, along with masking the image area per frame to avoid the sheath and other instrumentation was essential for accurate tracking and reconstruction. Additionally, a modular architecture for individual components of reconstruction, registration, and visualization tools facilitated debugging within the complex pipeline. System runtime could be further improved by optimization in compiled libraries (cf. scripted languages like Python) and more performant GPU hardware. To aid reproducibility, detailed documentation of specific software versions, configuration files, and custom scripts developed for data preprocessing or validation were maintained within a local git repository.

Looking ahead to potential clinical application, the ability of R2D2-E to provide accurate 3D reconstruction with a tracking runtime of 0.90 fps is potentially suitable for application in clinical neuroendoscopy. Augmented overlay of 3D structures segmented in MR, CT, or CBCT and registered directly to the endoscopic view could improve visualization and targeting of deep-brain structures in neuroendoscopic interventions and warrants further clinical investigation to demonstrate its clinical value.

VI. Conclusion

The study presented R2D2-E, a self-supervised approach to robust feature detection for real-time 3D video reconstruction and navigation in clinical neuroendoscopy. Experiments identified nominal hyperparameters and demonstrated superior performance in image matching tasks compared to SIFT and SURF, for example: median KPE = 0.83 px (compared to 2.20 px and 1.70 px, respectively); median PE = 0.64 mm (compared to 0.90 mm 0.99 mm, respectively; and F1 = 0.72 (at 1 mm distance threshold) and 0.94 (at 2 mm threshold), representing a 14% and 25% improvement over SIFT and SURF, respectively. The method achieved tracking runtime of 0.90 fps, demonstrating its potential for real-time reconstruction in clinical neuroendoscopy. A clinical pilot study conducted in the context of endoscopic third ventriculoscopy showed the potential value of augmenting real-time endoscopy with spatial context depicting critical structures beyond the visible field of view. These experiments established a rigorous quantitation of performance in clinical neuroendoscopy and supports future integration with clinical systems.