Deformable 3D-2D registration for high-precision guidance and verification of neuroelectrode placement

Abstract

Purpose.

Accurate neuroelectrode placement is essential to effective monitoring or stimulation of neurosurgery targets. This work presents and evaluates a method that combines deep learning and model-based deformable 3D-2D registration to guide and verify neuroelectrode placement using intraoperative imaging.

Methods.

The registration method consists of three stages: (1) detection of neuroelectrodes in a pair of fluoroscopy images using a deep learning approach; (2) determination of correspondence and initial 3D localization among neuroelectrode detections in the two projection images; and (3) deformable 3D-2D registration of neuroelectrodes according to a physical device model. The method was evaluated in phantom, cadaver, and clinical studies in terms of (a) the accuracy of neuroelectrode registration and (b) the quality of metal artifact reduction (MAR) in conebeam CT (CBCT) in which the deformably registered neuroelectrode models are taken as input to the MAR.

Results.

The combined deep learning and model-based deformable 3D-2D registration approach achieved $0.2\pm 0.1\text{mm}$ accuracy in cadaver studies and $0.6\pm 0.3\text{mm}$ accuracy in clinical studies. The detection network and 3D correspondence provided initialization of 3D-2D registration within $2\text{mm}$, which facilitated end-to-end registration runtime within 10 s. Metal artifacts, quantified as the standard deviation in voxel values in tissue adjacent to neuroelectrodes, were reduced by 72% in phantom studies and by 60% in first clinical studies.

Conclusions.

The method combines the speed and generalizability of deep learning (for initialization) with the precision and reliability of physical model-based registration to achieve accurate deformable 3D-2D registration and MAR in functional neurosurgery. Accurate 3D-2D guidance from fluoroscopy could overcome limitations associated with deformation in conventional navigation, and improved MAR could improve CBCT verification of neuroelectrode placement.

1. Introduction

Neuroelectrode placement is an important task in deep brain stimulation (DBS) treatment of Parkinson’s disease (Deuschl et al 2006), and emerging DBS therapies using novel electrode designs (Willsie and Dorval 2015, Steigerwald et al 2016) offer potential breakthroughs in a growing spectrum of neuro-pathologies, including Alzheimer’s disease, autism, and depression (Laxton et al 2010). Accurate electrode placement is essential to targeting functional nuclei (Richardson et al 2009) and avoiding adverse effects (Seijo et al 2014, Vakharia et al 2017). Although there is no consensus as to what constitutes a sufficient level of accuracy, most studies consider electrodes placed farther than $2-3\text{mm}$ from the target as misplacement and recommend reimplantation (Burchiel et al 2013, McClelland et al 2005). Similarly, electrode placement is a crucial aspect of stereoelectroencephalography (sEEG) (Cardinale et al 2013) for epilepsy localization and monitoring (Girgis et al 2020). In each case, accurate guidance and verification of electrode placement are important to minimizing targeting error and understanding variations in outcome.

Existing standards of care for guiding electrode placement include intraoperative cone-beam CT (CBCT) or magnetic resonance imaging (MRI), frame-based stereotaxy (Fitzpatrick et al 2005), surgical navigation with infrared (IR) or electromagnetic (EM) trackers (Nowell et al 2014), and (more recently) surgical robotics (Dorfer et al 2017). With the exception of direct visualization in real-time MRI, such systems largely assume rigidity with respect to the skull and are susceptible to geometric errors resulting from intracranial tissue deformation and bending of the electrode array (delivered via a long, thin wire). Kerezoudis et al (2020) for example, demonstrated electrode array deformations due to cerebrospinal fluid loss during transventricular approaches. Such deformations can confound the accuracy of robot-assisted sEEG placement, which provides a high-precision (e.g. 0.9 mm using NeuroMate [Renishaw, UK]) (von Langsdorff et al 2015), tremor-free platform that can streamline procedural workflow (González-Martínez et al 2016), but assume a rigid relation among frames, trackers, and intracranial targets. EM tracking can resolve device deformation by localizing the tip of a flexible instrument (Burchiel et al 2020), but does not account for intracranial deformation and can be susceptible to interference from metal instruments.

Intraoperative imaging, such as MRI, CT, and CBCT can provide up-to-date visualization that helps to address discrepancies between the preoperative 3D images and surgical plan and the intraoperative scene. For example, intraoperative CBCT acquired at the beginning of a case can be used to register navigation systems to the cranium, although image artifacts and low-contrast resolution can confound visualization of intracranial deformation or verification of the surgical product. Despite such image quality challenges, intraoperative CBCT has proven to improve the accuracy of stereotactic electrode placement (Smith and Bakay 2011, Holloway and Docef 2013, Carlson et al 2017). Repeat CBCT during the case (e.g. CBCT images acquired for planning, registration, guidance, and/or verification), however, must be managed judiciously with respect to radiation dose. Recognizing that CBCT systems commonly permit 2D fluoroscopy (with one fluoroscopic frame amounting to $\sim 1/400\text{th}$ the dose of a CBCT scan), the method described below was developed to guide and verify neuroelectrode placement via deformable 3D-2D registration of CBCT and fluoroscopy.

3D-2D registration of intraoperative fluoroscopy images for guiding neuroelectrode placement was first proposed by Hunsche et al (2017) using an intensity-based algorithm to rigidly map electrode locations to the preoperative CT. Prior work established a model-based 3D-2D registration method that demonstrated utility in spinal pedicle screw placement to: (1) guide instruments by providing 3D localization from biplane fluoroscopy (Uneri et al 2017); and (2) verify implant placement via 3D imaging with metal artifact reduction (MAR) (Uneri et al 2019). While the accuracy of such model-based approaches is well established (Uneri et al 2015, 2017), the underlying iterative optimization can suffer from limited capture range and require accurate initialization to reach a stable global solution with minimal iterations (i.e. short runtime). This is especially true in localizing neuroelectrode arrays, where the object is prone to deformation and is small with respect to the field-of-view. Solving such registrations with complex object models and high-dimensional (deformable) transformations within a large search space are subject to local minima and can result in long runtimes that may not be suitable to intraoperative workflow. Modern approaches using convolutional neural networks (CNN) have seen a rapid growth fueled by improvements in parallel computation and GPU hardware and offer to address many of these challenges in a fast and context-aware manner. Pinte et al (2021) for example, used a U-Net for segmenting EEG electrodes in MRI scans, followed by template-based registration to refine the accuracy of the network output. The solution by Pei et al (2017) similarly addresses challenges in network generalization by using the CNN within an iterative framework to regress deformation parameters that map a 3D CBCT image to 2D radiographs.

The deformable 3D-2D registration method presented below is novel in its joint utilization of data-driven techniques based on CNNs with physical model-based techniques that exercise a principled approach with respect to knowledge of the device and imaging system geometry. The CNN-based detection absolves the need for manual initialization, prior surgical planning, or time-consuming search strategies that challenged previous conventional methods using numerical optimization (Hunsche et al 2017). The subsequent model-based registration takes advantage of a-priori knowledge (i.e. C-arm geometry and parametric description of the implant shape) to achieve high accuracy while circumventing challenges common to current deep learning approaches, such as generalizing learned models to new datasets (Pei et al 2017). The resulting deformable registration facilitates two important tasks, to our knowledge demonstrated for the first time in DBS surgery: (1) guiding flexible neuroelectrodes to anatomical targets using 2D fluoroscopy (see, repeat 3D imaging and/or navigation); and (2) reducing metal artifacts in reconstructed images to verify implant placement in 3D CBCT imaging. The work builds on our recent success in detecting pedicle screws in radiographs (Doerr et al 2020) and preliminary algorithm development for deformable neuroelectrode registration with evaluation of minimum dose in phantom studies (Uneri et al 2021). Current work extends the previous approach by using a CNN trained for object detection combined with the projective geometry of the imaging system for reliable pose initialization of the registration process. The subsequent deformable registration of a 3D neuroelectrode model leverages the CNN-based initialization to reach a solution with fewer iterations and reduced susceptibility to local minima. The performance of instrument detection, registration, and MAR were experimentally evaluated in phantom, cadaver, and first clinical studies.

2. Methods

2.1. Algorithm for 3D-2D localization of neuroelectrodes

As illustrated in figure 1, the algorithm operates on a pair of fluoroscopic images containing one or more neuroelectrodes in the patient and comprises three main stages: (1) keypoint detection of the neuroelectrode using Mask R-CNN (He et al 2020), (2) establishing 3D correspondence of detections from multiple views via backprojection and rank-ordering of identified points; and (3) deformable 3D-2D image registration of the patient and electrodes using the known-component registration (KC-Reg) (Uneri et al 2017) algorithm. The method outputs the deformable shape and pose parameters of the neuroelectrode with respect to the patient anatomy, which in turn is used to guide instrument placement (section 2.2.1) and to mitigate metal artifacts in CBCT acquired at the end of the case to verify placement (section 2.2.2).

Figure 1.

Algorithm flowchart for 3D-2D registration of flexible DBS neuroelectrodes. (a) Fluoroscopic images are input to (b) keypoint detection based on Mask R-CNN. (c) Correspondence among multiple keypoint detections is established by backprojection, which in turn initializes the 3D component model. (d) 3D-2D registration is computed using the KC-Reg algorithm with a rigid motion model for the cranium and a deformable model of the neuroelectrode(s) therein.

2.1.1. Detection and keypoint localization of neuroelectrodes

The first stage of the approach detects neuroelectrodes in the fluoroscopic images and extracts 2D locations. An extended version of Mask R-CNN was adapted for concurrent object detection (tip of the electrode array) and keypoint localization (individual electrodes) (He et al 2020). The network uses a ResNet-50 backbone for feature extraction pre-trained on the COCO dataset (Lin et al 2014), and the last 3 layers were retrained on our training dataset (below). The region proposal network identifies rectangular regions-of-interest (ROIs) containing neuroelectrodes, which are then pooled and aligned on the image pixel grid, yielding the output bounding boxes containing all four electrodes with associated confidence scores. A separate head performs one-hot binary segmentation on each bounding box, producing single-pixel keypoint detection of individual electrode centers. A threshold on confidence scores $\left(t_s\right)$ is commonly used to reduce the incidence of false detections.

The model was trained on patient images acquired under an IRB-approved clinical study involving 35 CBCT scans from 23 patients undergoing cranial neurosurgery. Training dataset images were drawn from projections underlying 26 of these CBCT scans that did not contain DBS electrodes, although most included other instrumentation, such as microelectrode arrays or sEEG electrodes. Projection data were sampled from each scan at 4^∘ intervals, yielding a total of 2314 images, which was split 10:1 for training and validation. For each image, 1–2 electrodes were simulated using the parametric device model illustrated in figure 2(a) and detailed in section 2.1.3. Model parameters were randomly sampled from uniform distributions to capture variations in design, including: distance between electrodes $\left({\delta }_e=2.5\pm 0.5\text{mm}\right)$; electrode diameter $\left(d_e=1.4\pm 0.2\text{mm}\right)$; length $\left(l_e=1.5\pm 0.1\text{mm}\right)$; attenuation $\left({\mu }_e=0.5\pm 0.1{\text{mm}}^{-1}\right)$; and cable diameter $\left(d_c=1.4\pm 0.2\text{mm}\right)$. The electrode arrays were randomly positioned in $3\text{D}\left(\pm 6\text{cm}\right.$ and $\pm {30}^{\circ }$, with respect to the $\text{C}$-arm isocenter), and random deformations were applied by displacement of the $B$-spline control points $\left({\alpha }_j\right)$. The 3D model was forward-projected according to the projective geometry $\left(P_{\theta }\right)$ associated with the projection view at gantry angle $\theta$. The true bounding box and keypoint labels were automatically defined from the forward projection model. The model was trained using the Adam optimizer with a learning rate of 10⁻⁴ and batch size of 10 . The multi-task loss function was defined over each ROI as:

$$ℒ={ℒ}_{\text{class }}+{ℒ}_{\text{box }}+{ℒ}_{\text{keypoint,}}$$ (1)

where ${ℒ}_{\text{class}}$ represents the combined log loss over 2 classes (i.e. electrode or not),${ℒ}_{\text{box}}$ is the box regression using smooth $L_1$ loss, and${ℒ}_{\text{keypoint}}$ is the binary cross-entropy. Data augmentation during training included random horizontal and vertical flips and small perturbations of the pixel values (by adding 0%−1% uniform noise) to help prevent overfitting by ensuring no image is observed more than once.

Figure 2.

Neuroelectrode model and fluoroscopic projections. (a) Deformable $B$-spline component model of a 4-electrode array, showing the parameters (electrode diameter $\left[d_e\right]$, length $\left[l_e\right]$, separation $\left[{\delta }_e\right]$, cable diameter $\left[d_c\right]$, and $B$-spline control points $\left[{\alpha }_j\right]$) defining the ‘known-component’ (KC) model. (b)-(e) Example projections of various 4-electrode designs $\left(E_{\{A\dots D\}}\right)$ acquired on an O-arm TM imaging system.

2.1.2. Correspondence of detected electrodes

The correspondence among keypoint detections on multiple fluoroscopic images—i.e. the determination of which keypoint detections correspond to the same electrode—is established by backprojection of 2D keypoint locations into a 3D coordinate frame according to the imaging system geometry. Each detected keypoint $\left(k_{\theta ,i}\right)$ is backprojected according to the projective transform $\left(P_{\theta }\right)$ of its associated fluoroscopic view (Galigekere et al 2003), yielding a ray from the x-ray source to the detector plane, where the source point $\left(s\right)$ and direction vector $\left(\overrightarrow{v}\right)$ for each ray are:

$$s=-P_{[1\dots 3]}^{-1}{P}_{\left[4\right]},$$ (2a)

$$\overrightarrow{v}=P_{[1\dots 3]}^{-1}[k,1{]}^T,$$ (2b)

where $P_{[1\dots 3]}$ signify the first 3 columns of $P$, and $P_{\left[4\right]}$ is the last. Fluoroscopic views (each with $K_{\theta }$ detections) yield the backprojected rays:

$$r_{\theta }=\left\{s_{\theta },{\overrightarrow{v}}_{\theta ,i}:i=1\dots K_{\theta }\right\}.$$ (3)

Rays from multiple views are intersected in a combinatorial fashion (e.g. the Cartesian product $r_{\text{AP}}\times r_{\text{LAT}}$) to compute their intersections in 3D. Exact intersections are not expected due to small variations in system geometry and keypoint localization; therefore the intersection is approximated as the 3D point $x_i$ that minimize the sum of squared distances to rays:

$${\hat{x}}_i=\underset{x_i}{\text{arg} \text{min}}\sum _{\theta }{\delta }_{\theta ,i}^2\left(x_i\right),$$ (4)

where ${\delta }_{\theta ,i}$ is the distance from $x_i$ to a ray, such that

$${\delta }_{\theta ,i}^2\left(x_i\right)={\left(s_{\theta }-x_i\right)}^T\left(I-{\overrightarrow{v}}_{\theta ,i}{\overrightarrow{v}}_{\theta ,i}^T\right)\left(s_{\theta }-x_i\right).$$ (5)

The least-squares minimization in equation (4) can be solved algebraically by rearranging the terms as $A={\sum }_{\theta }\left(I-{\overrightarrow{v}}_{\theta ,i}{\overrightarrow{v}}_{\theta ,i}^T\right)$ and $b={\sum }_{\theta }\left(I-{\overrightarrow{v}}_{\theta ,i}{\overrightarrow{v}}_{\theta ,i}^T\right)s_{\theta }$, and solving $Ax=b$ using Moore–Penrose pseudoinverse. $A$ is expected to be full rank, except for instances in which ray directions from multiple views are parallel. Such cases, however, would either indicate failure to rotate the gantry for the second view or require joint translation of the source and detector along the same axis, which is not possible with C-arm geometries considered here.

The points that represent true electrode positions are then identified as the set that result in the smallest average distance ${\Delta }_i=\frac{1}{N}{\sum }_{\theta }{\delta }_{\theta ,i}\left({\hat{x}}_i\right)$, filtering intersections with ${\Delta }_i>10\text{mm}$ as outliers due to false detections. All such possible sets are evaluated in a permutative fashion, ensuring that the mapping from 2D keypoints to 3D points is bijective (i.e. each detection is uniquely associated with a point). Establishing correspondences in this manner is inherently robust to false positives but requires a minimum of false-negative detections. The preceding detection stage can therefore be tuned to take advantage of this bias as presented in section 3.1 below.

2.1.3. Deformable known-component 3D-2D registration (KC-Reg)

The third stage of the approach leverages a parametric form of the KC-Reg algorithm (Uneri et al 2017) to compute the 3D pose of flexible neuroelectrodes from a pair of fluoroscopy images. The 3D model of a neuroelectrode $\left(E\right)$ was a $B$-spline (Goerres et al 2017):

$$E(\alpha ,u)=\sum _{j=0}^M{\alpha }_j{B}_{j,k}\left(u\right)$$ (6)

chosen due to its locality in manipulation of individual control points $\left({\alpha }_j\right)$ without changing the shape of the entire curve. A cubic spline $(k=3)$ was used and the knots were clamped $\left(t_0=t_1\cdots =t_k\right)$. Using $M=4$ control points, the deformable transformation of an electrode array amounts to 12 degrees-of-freedom (DOF).

As illustrated in figure 2(a), a cylindrical cable of diameter $d_c$ contains $N=4$ electrodes separated by ${\delta }_e$, where each electrode was modeled as a rigid cylinder of diameter $d_e$ and length $l_e$. The combined transformation and shape parameters,$\lambda =\left\{{\alpha }_{1\dots 4},d_c,l_e,d_e,{\delta }_e\right\}$, result in a component model with $\left|\lambda \right|=16$ DOF. Such parametric modeling circumvents the need for exact implant models (e.g. manufacturer CAD specifications) and supports a broad variety of designs, such as those used in phantom and cadaver studies, shown in figures 2(b)–(e).

The KC-Reg algorithm (Uneri et al 2015, Uneri et al 2016) iteratively optimizes component model parameters $\left(\lambda \right)$ to maximize the gradient similarity (viz. gradient correlation, GC) (Penney et al 1998) between the acquired images $\left(I_{\theta }\right)$ and the component forward projections as:

$$\hat{\lambda }=\underset{\lambda }{\text{arg} \text{max}}\sum _N^{\theta }\text{GC} \left(I_{\theta },P_{\theta }\circ E\left(\lambda \right)\right)$$ (7)

where the parametric component is rendered as a 3D triangular mesh and forward-projected (using trilinear interpolation) according to the projective transform $P_{\theta }$.

The output from the CNN-based electrode detection stage helps the registration process in two ways. First, correspondent keypoint detections at 3D location $x_i$ are used to initialize ${\alpha }_j$, and their mean separation is used as an initial value for ${\delta }_e$. Second, the bounding box locations are used to crop projections to ROIs that limit the search space, allow more efficient use of images at their original resolution (without downsampling), and absolve the need for a multi-resolution pyramid (commonly used to increase capture range).

2.1.4. Known-component metal artifact reduction (KC-MAR)

The precise 3D localization offered by KC-Reg forms the basis of a MAR strategy, referred to as KC-MAR. Previous studies (Uneri et al 2019) demonstrated the approach in the context of MAR in CBCT images containing spinal fixation hardware (pedicle screws and rods), where accurate 3D-2D registration was shown to overcome common pitfalls associated with metal segmentation.

The KC-MAR approach forward-projects the registered component models according to the imaging system geometry to obtain precise 2D segmentations. Specifically, pixels associated with metal objects were identified as the region where $P_{\theta }\circ E\left(\hat{\lambda }\right)>0$. A bilinear interpolation-based sinogram inpainting method was used for projection data correction. As the localized region does not constitute a regular grid, Delaunay triangulation was used to partition the 2D space (Barber et al 1996), followed by barycentric interpolation over each triangle. To further mitigate effects due to image noise, the surrounding metal-free region was first median-filtered with a $3\times 3$ pixel kernel prior to interpolation. The filtered image was only used to obtain a noise-reduced interpolation, and the original measurements in the metal-free region were used during reconstruction.

The corrected 2D projection data were reconstructed using 3D filtered backprojection on a $710\times 710\times 534$ voxel grid with $0.3\times 0.3\times 0.3{\text{mm}}^3$ voxel size, using a 2D Hann apodization filter with cutoff at 50% of Nyquist frequency. The metal implant was reintroduced to the 3D image using a conventional voxel replacement approach in which the reconstructed voxels that correspond to the metal were replaced by the expected attenuation coefficients.

2.2. Experimental validation in image-guided neurosurgery

2.2.1. Guidance of electrode placement using KC-Reg

The accuracy of 3D-2D registration of neuroelectrodes from two fluoroscopic views was first evaluated in a cadaver study. Neuroelectrodes $\left(E_{\{A\dots D\}}\right)$ were first placed in the head via Kocher’s point burr holes to approximately 6 cm depth—first unilaterally, then bilaterally. Following each electrode placement, CBCT scans were acquired on the ${\text{O-arm}}^{\text{TM}}$ imaging system ($120\text{kV},186\text{mAs},745$ projections, $\left.33.6\text{mGy}\right)$. Two scans were consecutively acquired to assess the impact of pixel size on registration accuracy: first using the standard clinical $2\times 4$ pixel binning mode (giving $0.388\times 0.776{\text{mm}}^2$ pixel size); and the second scan using a higher-resolution $2\times 2$ pixel binning mode (with isotropic $0.388\times 0.388{\text{mm}}^2$ pixel size).

Registration accuracy was quantified in terms of the 3D target registration error (TRE) measured for each electrode as follows:

$${\text{TRE}}_i={∥E\left(\hat{\lambda },i{\delta }_e\right)-{\mathbb{x}}_i∥}_2,$$ (8)

where the registered location of electrode $i$ was calculated by evaluating the spline $\left(E\right)$ at $i{\delta }_e$ distance from the tip. True electrode locations $\left({\mathbb{x}}_i\right)$ were manually identified in 3D images, reconstructed from $2\times 2$ binned projection data within a $2\times 2\times 2{\text{cm}}^3$ ROI reconstruction with a finely oversampled $(0.05\times 0.05\times 0.05{\text{mm}}^3$) voxel grid to mitigate partial volume effects.

2.2.2. Verification of electrode placement using KC-MAR

The performance of deformable 3D-2D registration was also evaluated in terms of the reduction in metal artifacts yielded by the KC-MAR algorithm in CBCT images of a head phantom. As shown in figure 3 , the phantom was a water-tight 3.2 mm thick cranium shell (LiquiPhil, Phantom Laboratory, Greenwich NY) modified to include entry ports for low-contrast simulated intracranial structures and neuroelectrodes. A set of cylindrical rods emulating low-contrast intracranial soft tissues were inserted in the head phantom, including plastics of varying density with $-150,-135,-105,+40$, and $+200\text{HU}$ contrast to water. Neuroelectrodes were introduced from positions approximating Kocher’s point. CBCT images were acquired using the same scan protocols described above. The analysis involved implant detection (via CNN) and deformable registration (via KC-Reg), followed by projection data correction and 3D image reconstruction (via KC-MAR). Artifact magnitude was quantified as the standard deviation of voxel values within homogeneous regions of interest adjacent to the electrode.

Figure 3.

Experimental setup. (a) The O-arm ^™ imaging system (Medtronic, Littleton MA) used for phantom and cadaver studies. (b) Photograph of the water-filled phantom with neuroelectrode array and low-contrast intracranial inserts. (c) Example PA radiograph of a cadaver head with bilateral neuroelectrodes.

2.2.3. Clinical study of electrode guidance and verification

The performance of 3D-2D neuroelectrode guidance and verification was tested on clinical images obtained under an IRB-approved protocol. CBCT scans were acquired with a clinical $\text{O-arm}{}^{\text{TM}}$ imaging system (120 kV, 75–240 mAs) operated either in a rotated-detector mode with 40 cm axial FOV (23/35 scans) or a centered-detector mode with 20 cm axial FOV (12/35 scans). Images from 26 cases were used as training data as described in section 2.1.1. The KC-Reg and KC-MAR methods were tested on 100 images from 9 cases containing a total of 15 neuroelectrodes.

Electrode detection accuracy was evaluated on the clinical dataset in terms of recall $(\text{TP}/(\text{TP}+\text{FN}\left)\right)$ and precision (TP/(TP + FP)). Projection distance error (PDE) was measured as the 2D distance (on the detector plane) between the true and predicted electrode position, with the true 2D location estimated by forward-projecting manually identified true 3D locations $\left({\mathbb{X}}_i\right)$. The TRE was computed for each electrode after 3D-2D registration, and artifact reduction was measured in KC-MAR images in terms of the standard deviation in voxel values in regions adjacent to the implant.

3. Results

3.1. Electrode detection with keypoint localization

Figure 4 summarizes the accuracy of electrode detection in 100 intraoperative fluoroscopy images obtained from various perspectives in 9 clinical cases, demonstrating the ability of the network to generalize from the (simulated) training set to real electrode images. Example images from the training dataset are shown in figures 4(a)–(b), which highlight the degree of variations in curvature, electrode interval distances, and the complexity of the underlying radiographic scene with instruments and other implants. Training loss curves in figure 4(c) indicate a limit of 35 epochs, beyond which an increase in validation loss suggests overfitting. Keypoints with $<10\text{mm}$ PDE were considered as correct (true positive) detections, based on an empirical observation of figure 4(d), which shows that values of 8–15 mm do not affect recall or precision. While a $10\text{mm}$ PDE is significant from a clinical perspective, it is well within the capture range of the subsequent 3D-2D registration and is sufficient for purposes of initialization.

Figure 4.

Detection of neuroelectrodes in clinical studies. (a) Example images and (b) zoomed insets from the training dataset showing simulated electrodes superimposed on clinical radiographs. (c) Training and validation loss curves. Detection accuracy evaluated in terms of recall and precision (d) as a function of PDE threshold and (e) as a function of score threshold.

Figure 4(e) highlights the tradeoff in recall and precision with the detection score threshold $\left(t_s\right)$. A typical selection criterion balances performance—e.g. $t_s=80\%$ giving recall $=94.3\%$ and precision $=93.1\%$. As noted above, while the multi-stage detection/correspondence approach intrinsically filters false-positive detections, the process requires a minimum of false-negative detections; therefore, $t_s=0\%$ was adopted (all detections considered valid), thereby eliminating false negatives (100% recall) at the expense of precision (60.0%)

3.2. Correspondence of detected electrodes

The accuracy of correspondence was evaluated on pairs of fluoroscopic images with $\Delta \theta ={90}^{\circ }$ angular separation (e.g. AP and LAT) obtained from the 100 images. The method demonstrated 98.8% precision in correspondence (see 60% precision for detection), reflecting the intrinsic rejection of false detections. Figure 5(d) (top row) shows a false detection related to the connector end of the electrode, which nominally has 12 terminals, but were truncated to capture only 4 in the image. Figure 5(d) (bottom row) shows a failure mode in which 4 threads of a screw were identified as electrodes. Such false detections are typically not consistent between radiographic views and are therefore automatically rejected via 3D correspondence. The small number of correspondence failures were cases for which detections were narrowly separated in the $z$ direction (i.e. when multiple detections lie on the epipolar plane). Since such instances can be automatically detected, a third projection view can be used to resolve the registration (which improved correspondence recall and precision to 100%).

Figure 5.

3D correspondence of detected neuroelectrodes. (a) Example detections on patient images from the clinical study highlight the ability to detect the neuroelectrode arrays in the presence of other surgical instruments (e.g. surgical retractors and cables). (b) Zoomed insets show example keypoint (individual electrode) detections. (c)-(d) Example images highlighting false-positive detections; most failures are not consistent across all views and are thereby filtered via 3D correspondence. (e) 2D PDE of detected neuroelectrodes (distance in the projection domain) and (f) 3D TRE (registration error in the 3D backprojection domain of the patient CBCT image).

As illustrated in figures 5(e)–(f), detection of electrodes with $\text{PDE}=3.7\pm 2.3\text{mm}$ resulted in $\text{TRE}=1.4\pm 1.0\text{mm}\left(3.4\text{mm}{\text{CI}}_{95\%}\right)$ after 3D correspondence—well within the expected capture range of the subsequent 3D-2D registration stage. Note that the relation between the 2D PDE and 3D TRE was governed both by the system magnification factor (1.8 for the $\text{O-arm}{}^{\text{TM}}$ imaging system) as well as the rejection of false detections.

3.3. Deformable 3D-2D registration for guiding electrode placement

The geometric accuracy of KC-Reg was first evaluated in a cadaver specimen, using pairs of radiographs separated by $\Delta \theta ={90}^{\circ }$. The random perturbation of component parameters $\left(\lambda \right)$ gave initial TRE $=2.7\pm 0.3\text{mm}$. As shown in figure 6(a), KC-Regachieved TRE $=0.2\pm 0.1\text{mm}\left(0.3\text{mm}{\text{CI}}_{95\%}\right)$. The higher-resolution imaging protocol $(2\times 2$ detector pixel binning) gave modest improvement in mean TRE $(0.29\text{mm}$ for $2\times 4$ binning, compared to $0.17\text{mm}$ for $2\times 2$ binning) and resolved an additional $2\%(8/392)$ of outliers. The dependence of TRE on angular view separation shown in figure 6(c) was consistent with results in previous

Figure 6.

Geometric accuracy of deformable 3D-2D registration. (a) Registration accuracy in cadaver experiments with randomly perturbed initializations. (b) The higher-resolution detector readout mode ($2\times 2$ binning) showed modest reduction in TRE and outliers. (c) Angular separation of fluoroscopic views with $\Delta \theta ={25}^{\circ }-{30}^{\circ }$ provided TRE $<0.5$ mm with $95\%$ confidence. (d) Accuracy of deformable 3D-2D registration in the clinical study. ‘Initialization’ refers to the TRE following CNN-based detection and correspondence, and ‘Registered’ refers to the accuracy following 3D-2D registration.

studies, (Uneri et al 2014) showing that $\Delta \theta ={10}^{\circ }-{15}^{\circ }$ provided median TRE similar to that for $\Delta \theta ={90}^{\circ }$, and higher angular separation (e.g. $\Delta \theta ={25}^{\circ }-{30}^{\circ }$) ensured TRE $⩽0.5\text{mm}$ with $95\%$ confidence.

Figure 6(d) shows the end-to-end performance of the detection, correspondence, and deformable 3D-2D registration approach on images from the clinical study. With the robust initialization provided by the CNN-based detection and correspondence stage (initial TRE $=1.4\pm 1.0\text{mm}$), deformable KC-Reg achieved $\text{TRE}=0.6\pm 0.3\text{mm}\left({\text{CI}}_{95\%}=1.0\text{mm}\right)$.

3.4. KC-MAR for verifying neuroelectrode placement

Axial and coronal slices in figures 7(a)–(b) show the reduction in metal artifacts achieved with KC-MAR in phantom studies. The registered 3D component model is overlaid in yellow. Artifact magnitude was reduced from $58\pm 36\text{HU}$ without correction to $16\pm 5\text{HU}$ following KC-MAR, approaching the level of variation in the un-instrumented reference image ($13\pm 1\text{HU}$, attributable to quantum noise).

Figure 7.

Metal artifact reduction in the phantom. Example CBCT axial and sagittal slices (a) without and (b) with KC-MAR. (c) Artifact magnitude measured in homogeneous regions adjacent to the electrodes, demonstrating 72% reduction using KC-MAR.

Evaluation of KC-MAR in clinical images is summarized in figure 8. Although artifacts from the electrode were the dominant source of information loss in the region of target anatomy central to the cranial vault, the images suffered from a number of other artifacts not addressed in the current work (including truncation, scatter, beam hardening, and other metal instruments—e.g. the Leksell-frame and robotic arm). Despite these additional confounding factors, KC-MAR reduced the magnitude of metal artifact within the region of interest about the electrode from $206\pm 105\text{HU}$ to $83\pm 39\text{HU}(60\%$ reduction) across all 9 cases.

Figure 8.

Metal artifact reduction in the clinical study. (a) CBCT axial slice showing a number of artifacts attributable to scatter, truncation, and metal. Regions of interest about the neuroelectrodes are shown (b) without and (c) with KC-MAR. (d) Artifact magnitude measured in the region about the electrodes, demonstrating 60% reduction with KC-MAR.

4. Discussion

This work presented an approach for image-based guidance and verification of neuroelectrode placement in functional neurosurgery, consisting of three main elements: (1) deep-learning-based detection of neuroelectrodes in biplane fluoroscopy; (2) deformable 3D-2D registration according to a physical model of the neuroelectrodes; and (3) MAR in CBCT. The detection method (1) provided fast, robust initialization of the registration method (2), which in turn provided accurate device segmentation for the MAR method (3). The approach is novel in its combination of the strengths of deep learning for object detection with physically principled approaches to registration and image reconstruction, which helps circumvent common problems such as model generalization in deep learning and slow runtimes for conventional approaches.

1. Mask R-CNN trained on images with simulated electrodes provided accurate detection with no false negatives (100% recall) in a clinical test dataset. The method for establishing correspondence among electrodes successfully rejected false detections, achieving 98.8% recall and precision from 2 fluoroscopic views (and 100% with 3 views). The method provided 3D localization with $\text{TRE}=1.4\pm 1.0\text{mm}$, providing good initialization for the subsequent deformable registration. The detection runtime for a single image was $0.11\text{s}$, and correspondence from a pair of images took $12\text{ms}$. Using a broad variety of component models in simulating the training dataset was key to obtaining the large, labeled dataset necessary for supervised learning. Generalization of the trained model to real images was consistent with other efforts in literature (Gherardini et al 2020). While similar methods commonly use a segmentation approach (e.g. a U-Net architecture) (Ambrosini et al 2017), keypoint detection provided direct localization of relevant features (electrodes) without additional post-processing. The parametric model employed (and therefore the CNN) is expected to generalize well to a number of existing 4-electrode device models from various vendors. The method can be easily adapted to detect other/new electrode designs (e.g. $>4$-electrode leads) by extending the parametric model and retraining the network. For such cases, the number of classes could also be increased $(>2)$ to differentiate between designs. For such cases, the number of classes could also be increased $(>2)$ to differentiate between designs.

2. Deformable 3D-2D registration informed by a physical model of the neuroelectrode achieved TRE = $0.2\text{mm}$ in cadaver and $0.6\text{mm}$ in clinical studies $\left({\text{CI}}_{95\%}=1.0\text{mm}\right)$. The high level of registration accuracy was attributable to: (i) excellent initialization provided by the CNN-based detection and correspondence method; (ii) relative simplicity of the instrument (i.e. simple cylindrical shapes—see, pedicle screws) (Uneri et al 2015); and (iii) use of a relatively high-resolution detector readout mode ($2\times 2$ detector pixel binning). Automated initialization via CNN-based detection improved runtime by an order of magnitude compared to earlier implementations using multi-resolution pyramids (1–3 min) (Goerres et al 2017) and reduced the total number of iterations—e.g. an average of 36 iterations required with $0.15\text{s}/$ iteration, resulting in a mean runtime of $5.2\pm 2.4\text{s}$. The runtime is consistent with the time required to tilt the $\text{C}$-arm gantry and acquire two views, and should be kept at a minimum to avoid a delay after the second view. While the current runtime does not qualify as real-time, it permits rapid spot-checks of 3D implant poses and trajectories as electrodes are advanced toward targets.

3. Using the deformable registration results as input to the KC-MAR algorithm demonstrated MAR of 60%–72% in phantom and clinical studies, supporting intraoperative 3D verification of implant placement. While KC-MAR consistently improved image quality in regions local to the electrode, other artifacts limited overall CBCT image quality (e.g. scatter, truncation, and other metal objects) and are the subject of other work. Example work underway includes sinogram+image domain networks for MAR (Ketcha et al 2021) as well as more complete approaches that target soft tissue contrast resolution in the brain (Wu et al 2021).

A key benefit of this approach is the reduction in radiation exposure, since solving registrations from two radiographs amounts to $\sim 1/200$ th the dose of a CBCT scan. While the registered implants can be visualized in the context of a preoperative scan (e.g. CT), the robustness of the solution is subject to the multiple stages of detection and registration. The risk of registration error can be mitigated via successive shots as the implants are advanced towards the target, taking advantage of low-dose and rapid acquisition of fluoroscopic frames. 3D CBCT imaging provides definitive visualization of the implants and surrounding structures and can potentially obviate the need for a postoperative scan at $1/3-1/2$ the dose of a CT. Detecting and determining the pose of implants is simpler in 3D images, and the conventional challenges of reconstructing structures around metal implants can be avoided by the proposed method. Considering these benefits and tradeoffs, the decision for 2D versus 3D imaging (and when to use each) ultimately falls to the operator to weigh the benefits of guidance via rapid, low-dose fluoroscopy compared to decisive verification using 3D imaging at the cost of time and dose.

The approach assumes that the acquired radiographs capture pertinent features of the electrode implants. These requirements are consistent with current clinical practice in which the surgeons acquire lateral views of the electrodes to judge depth with respect to other anatomical features. In DBS surgery, the electrodes are implanted with a superior approach and are therefore approximately aligned with the C-arm rotation gantry, which allows clear delineation of individual electrodes. Oblique gantry tilts or electrode arrays with severe deformations may lead to radiographs with an ‘end-on’ view of the implant, in which individual electrodes may be superimposed. While the detection stage is robust to a certain level of error (e.g. missing keypoints), such views are prone to failures and may necessitate a third view to provide the missing features for detection and registration. The method also relies on prior calibration of the imaging system geometry to identify corresponding detections. While this information is commonly available for CBCT-capable C-arms, simpler C-arms for conventional fluoroscopy may not support a reliable calibration (i.e. gantry motion may not be repeatable). Previous work included a method for self-calibration of the geometry that may be used for this purpose (Ouadah et al 2016). More recent approaches include application of a CNN that directly localizes instruments from multiple 2D views, (Schaffert et al 2020a, 2020b, Wu et al 2020) assuming a fixed relation between the views (e.g. consistently using AP and LAT views).

Data from the ongoing clinical study will be used in future work to increase training dataset size to better handle variations in anatomy and imaging protocols. Alternative electrode designs will also be considered, including sEEG arrays with variable number of electrodes and DBS electrodes with novel designs. Work is also underway to extend the 3D component model to support detection and registration of electrodes with radio-opaque markers near the electrode tip to help resolve the 3D orientation of the electrode array.

5. Conclusions

Image-based guidance and verification of neuroelectrode placement was demonstrated in phantom, cadaver, and first clinical studies. Guidance using a novel joint deep learning and model-based deformable 3D-2D registration method achieved $0.2\pm 0.1\text{mm}$ accuracy in cadaver studies and $0.6\pm 0.3\text{mm}$ accuracy in clinical studies. The deep learning stage provided initialization of 3D-2D registration within $2\text{mm}$, which facilitated end-to-end registration runtime within $10\text{s}$. Verification of electrode placement in CBCT images, enabled by 3D-2D registration and MAR, demonstrated 72% artifact reduction in phantom studies and 60% reduction in first clinical studies. The deformable 3D-2D registration method could improve the accuracy and precision of image-guided neurosurgery, helping to minimize errors in geometric targeting and reduce variations in functional outcomes.