Nonrigid 3D Brain Registration Using Intensity/Feature Information

Abstract

The brain deforms non-rigidly during neurosurgery, preventing preoperatively acquired images from accurately depicting the intraoperative brain. If the deformed brain surface can be detected, biomechanical models can be applied to calculate the resulting volumetric deformation. The reliability of this volumetric calculation is dependent on the accuracy of the surface detection. This work presents a surface tracking algorithm which relies on Bayesian analysis to track cortical surface movement. The inputs to the model are 3D preoperative brain images and intraoperative stereo camera images. The addition of a camera calibration optimization term creates a more robust model, capable of tracking the cortical surface in the presence of camera calibration error.

1 Introduction

The use of Surgical Navigation Systems (SNS) has greatly aided neurosurgical procedures. These systems provide real-time visualization of surgical tool positions relative to preoperative patient brain images, allowing surgeons to locate pathologic structures and other surgical targets. As SNS use becomes more prevalent in varied surgical procedures, the accuracy and reliability of these systems must be demonstrated. Soft tissue movement due to gravity, loss of blood and cerebrospinal fluid (CSF), and the action of certain medications can lead to misalignment between preoperative brain scans and the intraoperative brain [4, 9]. Therefore, accounting for this brain shift is necessary to realize the effectiveness of these navigation systems and to allow surgeons to confidently excise tissue, especially near functionally eloquent areas.

Volumetric images revealing the location and amount of brain shift can be taken intraoperatively. However, surgical constraints limit both the time per acquisition as well as prevent multi-modality imaging and use of certain modalities such as functional magnetic resonance imaging (fMRI) and positron emission tomography (PET). The effect of these restrictions is decreased spatial resolution, lower contrast and the loss of meaningful information [14, 15]. Additionally, intraoperative magnetic resonance imaging (iMRI) and computed tomography (iCT) are prohibitively expensive [15] for most hospitals.

Since the multitude of factors affecting brain shift, coupled with patient variability, make it impossible to predict brain shift a priori, it becomes necessary to characterize brain movement using sparse intraoperative data to update the rich preoperative information. Warfield et al. [3, 14] warped preoperative images containing projected atlas information through the use of surface data acquired by iMRI and a volumetric biomechanical model. Their study focused on the visualization of the corticospinal tract, which is not observable with conventional MRI, and indicated visually appealing results can be obtained in real time. Several groups use commercially available laser range scanners (LRS) to detect the cortical surface. Registration can then be performed between the acquired depth maps. Audette et al. [1] used a non-rigid iterative closest point (ICP) algorithm for this registration in which the initial alignment must be done manually. This work did not take advantage of any surface features such as blood vessels or sulci. In [10], the LRS acquired both a range and texture map, allowing the non-rigid registration to be performed in the 2D texture space, which uniquely corresponds to a depth map.

Nakajima et al. [7] used cortical surface vessels for registration of 2D intraoperative images with 3D postcontrast MRI images. Cortical vessels were chosen as landmarks due to their similarity with sulcal and gyral patterns and ease of identification in phase contrast MRI. Sun et. al [12] also used cortical surface vessels to guide surface registration in a phantom study. Vessels were simulated by metal wires and imaged using CT. Intraoperative surface information was acquired using a single charge-coupled device (ccd) camera mounted on an operating microscope. To perform stereo reconstructions of the surface, two images were acquired at different microscope positions, which were tracked optically. Sun et al. [13] later replaced this with a stereo camera mounted microscope. S̆krinjar [11] et al. used a similar stereo camera setup. His work combined the model deformation and stereo reconstruction. However, neither study used feature matching with the stereo reconstruction.

This work extends that of previous authors by performing both intensity and feature based nonrigid registration of cortical brain surface using a stereo camera setup. Since the reliability of a system based on cameras is highly dependent on the accuracy of the camera calibration, this work seeks to correct for any calibration inaccuracies and update the calibration matrices with the surface deformation calculation. This framework is both convenient and versatile. (Digital cameras are either present or can easily be transported to the operating room.) By using sulcal grooves and not cortical vessels as features, this algorithm does not require patients to ingest contrast or endure more than the standard preoperative imaging. And, it takes full advantage of the feature information to improve the deformation prediction while also correcting for any calibration inaccuracies, which would limit the reliability of the result. In this way, many of the advantages of the above-proposed models are captured in one algorithm.

2 Method

A surface detection algorithm which calculates the most likely intraoperative brain surface deformation using Bayesian analysis was outlined in [2]. This work developed a model predicting the dense displacement field of the cortical surface, given intraoperative images and surface feature locations. This displacement field, Û_dense, was found to be:

$$\underset{\_}{{\widehat{U}}_{\mathit{\text{dense}}}}=arg\underset{\underset{\_}{U_{\mathit{\text{dense}}}}}{max}p(\underset{\_}{U_{\mathit{\text{dense}}}}|\underset{\_}{\mathbf{\text{I}}},\underset{\_}{\mathbf{\text{K}}},\underset{\_}{C},S^U,\underset{\_}{\mathbf{\text{A}}})$$ (1)

where U_dense is a possible dense displacement field. A = [A₀,A₁], where A₀,A₁ are the camera calibration matrices acquired by the left (0) and right (1) cameras. I = [I₀, I₁], where I₀, I₁ are the intraoperative images acquired by the left (0) and right (1) cameras. K = [K₀, K₁], where K₀, K₁ are matrices in which each column is the vector representation of a 2D sulcal groove traced by an expert user from images I₀ and I₁, respectively. C is a matrix in which each column is the vector representation of a 3D preoperative sulcus position, and S^U is the original (preoperative) cortical surface, extracted from the preoperative MRI.

This method yielded surface deformation predictions with submillimeter mean accuracy both in simulation and phantom experiments. The camera calibration matrices were critical for taking advantage of both the surface intensities, as acquired from the stereo cameras, and the feature alignment, which requires a projection from 3D space to the image plane. However, ideal calibrations are not feasible in many real world situations [6], especially in a busy operating room where time is of the essence. Therefore, we propose updating the camera calibration calculation in the region of interest using the extra scene information obtained from cortical feature information obtained during neurosurgery.

The updated model can be written as follows:

$$(\underset{\_}{{\widehat{U}}_{\mathit{\text{dense}}}},\underset{\_}{\widehat{\mathbf{\text{A}}}})=arg\underset{\underset{\_}{U_{\mathit{\text{dense}}}},\underset{\_}{\mathbf{\text{A}}}}{max}p(\underset{\_}{U_{\mathit{\text{dense}}}},\underset{\_}{\mathbf{\text{A}}}|\underset{\_}{\mathbf{\text{I}}},\underset{\_}{\mathbf{\text{K}}},\underset{\_}{C},S^U)$$ (2)

Equation (2) can be solved using an iterative scheme that updates both the camera calibration and the dense displacement field calculation at every iteration.

$$\underset{\_}{{\widehat{U}}_{\mathit{\text{dense}}}^k}=arg\underset{\underset{\_}{U_{\mathit{\text{dense}}}}}{max}p(\underset{\_}{U_{\mathit{\text{dense}}}}|\underset{\_}{\mathbf{\text{I}}},\underset{\_}{\mathbf{\text{K}}},\underset{\_}{C},S^U,{\underset{\_}{\mathbf{\text{A}}}}^{k-1})$$ (3)

$$\underset{\_}{{\widehat{\mathbf{\text{A}}}}^k}=arg\underset{\underset{\_}{\mathbf{\text{A}}}}{max}p(\underset{\_}{\mathbf{\text{A}}}|\underset{\_}{\mathbf{\text{I}}},\underset{\_}{\mathbf{\text{K}}},\underset{\_}{\mathit{\text{C}}},{\mathit{\text{S}}}^U,{\underset{\_}{U_{\mathit{\text{dense}}}}}^{k-1})$$ (4)

where k is the iteration number.

Maximizing the expressions in equations (3) and (4) is equivalent to maximizing the log of the posterior probabilities. The expansion of equation (3) can be found in [2].

2.1 Camera Calibration Optimization

Looking at equation (4), the most likely camera calibration, Â, will best match 3D cortical surface points to their corresponding 2D image locations. The imaged sulci provide accurate feature information from which correspondence can be reliably determined. Since the brain deformation is small relative to the brain volume, the sulcal positions should not vary greatly between frames. Because of this, the projection of sulci from the 3D preoperative brain surface, C, should be in close proximity to the intraoperatively imaged sulci, K. Using this information and further simplification, the determination of Â^k can be reduced to the maximum argument over A of:

$$log[p(\underset{\_}{\mathbf{\text{A}}}|\underset{\_}{\mathbf{\text{K}}},\underset{\_}{C},{\underset{\_}{U_{\mathit{\text{dense}}}}}^{k-1})]$$ (5)

Assuming independence of the stereo cameras, equation (5) can be written as:

$$log[p(A_0|\underset{\_}{K_0},\underset{\_}{C},{\underset{\_}{U_{\mathit{\text{dense}}}}}^{k-1})]+log[p(A_1|\underset{\_}{K_1},\underset{\_}{C},{\underset{\_}{U_{\mathit{\text{dense}}}}}^{k-1})]$$ (6)

To solve equation (6), it is assumed that the camera calibration matrices from the the previous iteration are known. For iteration k = 1, the k-1 calibration matrices are the initial matrices obtained from calibration procedure described in [2]. According to [5], assuming the measurement error in calculating the calibration matrices is Gaussian, the reconstruction error will be Gaussian. Therefore, equation (6) reduces to:

$$log[p(\underset{\_}{\mathbf{\text{A}}}|\underset{\_}{\mathbf{\text{K}}},\underset{\_}{C},{\underset{\_}{U_{\mathit{\text{dense}}}}}^{k-1})]=-{\eta }_{0,1}\int d[\underset{\_}{K_{0,1}}-(A_{0,1}\cdot (\underset{\_}{C}+\underset{\_}{U_{\mathit{\text{dense}}}^C})]dS$$ (7)

where d is a distance metric, $\underset{\_}{U_{\mathit{\text{dense}}}^C}$ represents the deformation field restricted to the sulci, and η_0,1 are constants.

Whereas the search for the displacement field can take advantage of the image intensities, it would be difficult to incorporate these intensities into a search for the optimum camera calibration matrix. This is why the use of features is critical to this process.

2.2 Data Acquisition

Noncontrast preoperative MRI images were obtained on a 1.5 T MR scanner. The preoperative cortical surface was extracted and 3D sulci positions were located manually. The camera calibration was performed using the method outlined in [2]. Images (1024 × 768) were acquired from a mounted ccd camera in the OR, which could be repositioned by the surgeon. For validation purposes, the 3D locations of various points on the exposed cortical surface were selected with a BrainLAB pointer tool (BrainLAB AG, Heimstetten, Germany), and saved using the Vector Vision Cranial software [2, 8].

3 Results

Seven patients undergoing stage I epilepsy surgery, in which intracranial electrodes are inserted to monitor the patient during and between seizures, were chosen for brain shift tracking. Though the craniotomies varied in position, the size of the skull opening, as well as the surgical time, were similar across all patients. One sample case is explored below.

Figure 1 shows the extracted preoperative brain surface and the 3D point locations selected during surgery for validation. The 3D manually-selected preoperative sulcal locations, smoothed extracted surface and intraoperative pictures with outlined sulci (Figure 2) were inputs to the surface detection algorithm. The algorithm took 23.5 minutes to run on a 3 GHz Pentium 4 computer. As seen qualitatively in Figure 3, the algorithm closely predicts the deformed cortical surface position. The coloring on the 3D patches represents greylevel intensities projected on the surface from one of the intraoperative images. Due to the misalignment caused by the soft tissue deformation, most of the initial surface patch is not in the camera view. However, the actual and predicted surface show the correct intensities, reflecting their accurate location relative to the OR camera.

Fig. 1.

The left image depicts the extracted 3D brain surface. The craniotomy was performed in the highlighted section. The right image shows initial (dark spheres) and final (light spheres) brain surface positions as acquired in the OR for algorithm validation. The hardware and software required for point acquisition are explained in [8].

Fig. 2.

Stereo camera intraoperative images acquired approximately three hours after surgery began. The sulci used as the feature information are outlined.

Fig. 3.

The color of the surfaces patches was found by backprojecting the image intensities to the surface using the camera calibration matrices. The cortical surface sinks during surgery from its initial position, out of the intraoperative camera view, to the deformed position. The surface tracking algorithm takes advantage of both the image intensities and the feature information to closely determine the final cortical position.

The mean reconstruction error was 1.01 mm when the calibration optimization was used. As can be seen in Table 1, without the camera calibration correction, the surface matching result is much worse. This makes intuitive sense because calibration accuracy determines the effectiveness of both the image intensities and the projected feature information. In this case, the optimized calibration matrices were less than 2% different from the initial matrices. This small difference, which could have been the result of fitting error or slight camera movement, created a significant difference in the algorithm result.

Table 1.

Surface Deformation Tracking Results. The actual surface displacement was measured by the BrainLAB VV Cranial software and compared to the displacement predicted by the algorithm, with and without camera calibration optimization. When the camera calibration matrices are inaccurate, the intensity and feature information cannot be reliably used to track the surface.

	mean (mm)	max (mm)	std (mm)
surface displacement	7.49	8.38	0.61
algorithm error, no calibration correction	6.80	11.01	1.98
algorithm error, calibration correction	1.01	1.97	0.60

4 Discussion

During neurosurgery, it is essential to accurately locate pathologic structures within the brain. Due to the similarity in appearance between these pathologic regions and healthy tissue, neurosurgeons often rely on SNS, though these systems cannot currently compensate for intraoperative brain shift. Biomechanical models can be used to predict this nonrigid brain shift, if driven by accurate intraoperative information, such as cortical surface deformation. This work made use of a stereo camera setup to track this cortical surface displacement. Future model refinement will focus on application of the model to a greater number of surgical cases, updating model parameters for optimum accuracy, as well as using the acquired surface results with biomechanical models to track the volume deformation.

Though it has been previously shown that intraoperative images can be used to drive surface tracking models, the results have always been susceptible to camera calibration error. This is especially true in patient studies in which OR time, and therefore camera calibration time, is limited. As seen in the above test case, the surface tracking algorithm would have failed without the calibration optimization, though the calibration matrices were within two percent of their correct values. The addition of the camera calibration optimization term, however, yielded significantly better results. The combined algorithm predicted the surface displacement with an error less than, or within the range of, previously published surface deformation studies [3, 10, 13]. This work shows that by taking advantage of the cortical feature information, the surface displacement can be tracked in the presence of calibration inaccuracies.