Non-rigid alignment of preoperative MRI, fMRI, and DT-MRI with intra-operative MRI for enhanced visualization and navigation in image-guided neurosurgery

Abstract

Objective

The usefulness of neurosurgical navigation with current visualizations is seriously compromised by brain shift, which inevitably occurs during the course of the operation, significantly degrading the precise alignment between the preoperative MR data and the intra-operative shape of the brain. Our objectives were (i) to evaluate the feasibility of non-rigid registration that compensates for the brain deformations within the time constraints imposed by neurosurgery, and (ii) create augmented reality visualizations of critical structural and functional brain regions during neurosurgery using pre-operatively acquired fMRI and DT-MRI.

Materials and Methods

Eleven consecutive patients with supratentorial gliomas were included in our study. All underwent surgery at our intra-operative MR imaging–guided therapy facility and have tumors in eloquent brain areas (e.g. precentral gyrus and cortico-spinal tract). Functional MRI and DT-MRI, together with MPRAGE and T2w structural MRI were acquired at 3T prior to surgery. SPGR and T2w images were acquired with a 0.5T magnet during each procedure. Quantitative assessment of the alignment accuracy was carried out and compared with current state-of the-art systems based only on rigid-registration.

Results

Alignment between preoperative and intra-operative datasets was successfully carried out during surgery for all patients. Overall, the mean residual displacement remaining after non-rigid registration was 1.82 mm. There is a statistically significant improvement in alignment accuracy utilizing our non-rigid registration in comparison to the currently used technology (p<0.001).

Conclusions

We were able to achieve intra-operative rigid and non-rigid registration of (1) pre-operative structural MRI with intra-operative T1w MRI; (2) pre-operative FMRI with intra-operative T1w MRI, and (3) pre-operative DT-MRI with intra-operative T1w MRI. The registration algorithms as implemented were sufficiently robust and rapid to meet the hard real-time constraints of intra-operative surgical decision making. The validation experiments demonstrate that we can accurately compensate for the deformation of the brain and thus can construct an augmented reality visualization to aid the surgeon.

Introduction

The American Cancer Society estimates 18,820 new brain tumors will be diagnosed in 2006 in the United States, with an estimated 12,820 deaths. Low-grade gliomas account for 25% of all primary brain tumors.

One of the principle causes of death among patients with low-grade glioma (LGG) is progression of the tumor to a malignant form (ie, anaplastic degeneration). Surgical resection of low-grade gliomas may decrease the rate of recurrence and increase the time to tumor progression (Piepmeier et al. 1996, Berger et al. 1994). Additionally, a maximal resection and a smaller volume of postoperative residual tumor are associated with an improved prognosis for the patient (Philippon et al. 1993, Piepmeier et al. 1996, Janny et al. 1994, Healy et al. 1991). But increased resection margins can increase the risk for postoperative neurologic deficits, due to possible damage of eloquent brain areas, such as the precentral gyrus and cortico-spinal tract, which concern motor function. Therefore, the principle challenge and objective of surgical intervention is to maximize the resection of tumor, while also minimizing the potential for neurological deficit by preserving critical tissue.

One of the challenges for neurosurgeons is to preserve the function during surgery for lesions in the central region. Cortical stimulation is commonly used to localize motor function and has become the “gold standard” when performing surgery in and adjacent to the motor cortex. However, it does not allow for ascertaining the risk of a new postoperative motor deficit before surgery. Moreover, cortical stimulation is a demanding, time consuming, and costly procedure and as such is often not possible during surgery. Therefore, because fMRI of motor and language tasks is feasible in patients with cerebral tumors (Mueller et al. 1996), several groups have proposed the integration of functional data into the neuronavigation system in recent years (Krishnan et al. 2004, Nimsky et al. 1999, Ganslandt et al. 1999, Gralla et al. 2003, Jannin et al. 2002, Roessler et al. 2005, Reithmeier et al. 2003, Talos et al. 2003, O'Shea et al. 2006). And strong evidence that a more radical tumor resection may be achieved by using fMRI information during neurosurgery has been demonstrated by Krishnan et al. 2004, Haberg et al. 2004.

Diffusion tensor imaging (DTI) has recently emerged as a potentially valuable tool for preoperative planning (Tummala et al. 2004, Field et al. 2004, Mori et al. 2002, Clark et al. 2003, Wieshmann et al. 2000, Holodny et al. 2001, Holodny et al. 2001, Witwer et al. 2002, Moller-Hartmann et al. 2002, Coenen et al. 2003.) and postoperative follow-up (Alexander et al. 2003) of surgically treated brain tumors and vascular malformations. DTI provides information about the normal course, displacement, or interruption of white matter tracts in and around a tumor, as well as detecting the widening of fiber bundles due to edema or tumor infiltration (Beppu et al., 2003; Clark et al., 2003; Hendler et al., 2003; Lu et al., 2003; Price et al., 2003; Tummala et al., 2003; Wieshmann et al., 1999; Witwer et al., 2002; Yamada et al., 2003). Consequently, efforts have been made in recent years to integrate DTI data with neurosurgical navigation systems (Nimsky et al. 2005, Coenen et al. 2003, Nimsky et al 2005, Talos et al 2003, Berman et al. 2004, Shinoura et al. 2005, Nimsky et al. 2006). Such a study on the role of diffusion tensor imaging of the corticospinal tract (CST) before and after mass resection, and the correlations with clinical motor findings, was recently published by Laundre et al. 2005.

Interventional MRI (iMRI) has proven to be an effective tool for improving the completeness of low grade glioma resection (Claus et al. 2005, Bradley 2002, Schwartz et al 1999, Schneider et al. 2001, Schneider et al. 2005, Knauth et al. 1999, Wirtz et al. 2000). However, brain deformations typically occur during the neurosurgical procedure, which results in a misalignment between the pre-operatively acquired datasets and the intraoperative brain position. Commonly, commercial systems (such as those developed by Medtronic or BrainLab) only use rigid registration algorithms to project the pre-operatively acquired fMRI and DTI into the navigational system. Intraoperative re-acquisition of fMRI and DTI with iMRI is impractical at present due to long image acquisition and processing times. Non-rigid registration algorithms are therefore necessary to preserve the accuracy of the preoperative fMRI and DTI data.

Intraoperative changes in the shape of the target anatomy impose a stringent requirement upon navigation systems. In order to capture such shape changes it is often necessary to make use of nonrigid registration techniques, which are characterized by a capacity to estimate transformations that model not only affine parameters (global translation, rotation, scale and shear), but also local deformations. This typically requires higher order transformation models, with increased numbers of parameters, and is usually more computationally expensive.

Modeling the behavior of the brain remains a key issue in providing navigation in image-guided neurosurgery. The biomechanical property experiments of Miller (2002) have contributed significantly in the understanding of the physics of brain tissue. He and his colleagues have explored and evaluated several constitutive models (Miller and Chinzei 1997; Miller et al. 2000; Miller and Chinzei 2000; Chinzei and Miller 2001; Miller and Chinzei 2002), which have shown very good concordance of the hyper-viscoelastic constitutive equation with in vivo and in vitro experiments (Miller et al. 2000).

Additionally, Miga, Paulsen and their collaborators (Miga et al. 1999; Miga et al. 1999; Miga et al. 2000, 2000; Miga et al. 2000; Miga et al. 2001; Paulsen et al. 1999; Roberts et al. 1998; Roberts et al. 1999; Roberts et al. 2001) have developed a sophisticated model of brain tissue while undergoing surgery, incorporating simulation of retraction, resection and local stress associated with tumor tissue. Careful validation experiments indicate their model is capable of closely matching observed deformations (Platenik et al. 2002). Their experiments also indicate further improvements in accuracy will be possible by incorporating sparse data from inexpensive intraoperative imaging devices. This work has demonstrated that computer aided updating of preoperative brain images can restore close correspondence between the preoperative data and the intraoperative configuration of the subject. But a practical difficulty of these models is the extensive time necessary to mesh the brain and solve the problem, which is takes too much time for intra-operative purposes. Davatzikos et al. (2001) proposed a statistical framework consisting of pre-computing the main mode of deformation of the brain using a biomechanical model. And recent extensions of this framework show promising results for intra-operative surgical guidance based on manually extracted data (Lunn et al. 2003).

Simple biomechanical models have been used to interpolate the full brain deformation based on sparse measured displacements. Audette (2003) and Miga et al. (2003) measured the visible intra-operative cortex shift using a laser range scanner. The displacement of deep brain structures was then obtained by applying these displacements as boundary conditions to the brain mesh. A similar surface based approach was proposed by Skrinjar et al. (2002) and Roberts et al. (2003), whereby they imaged the brain surface with a stereo vision system.

Previously, we created a full, three-dimensional non-rigid registration implementation using the mean square intensity difference in local regions as the similarity metric, constrained by a linear elastic material (Ferrant et al., 1999). In practice, the method was successful in clinical applications where an assumption of constant image intensities for corresponding structures held true. Our most recent work has built upon our earlier efforts and explorations in non-rigid registration for segmentation, preoperative planning, and enhanced visualization in support of image-guided surgery, and has been described previously (Warfield, Jolesz, and Kikinis 1998; Warfield et al. 2000; Ferrant et al. 2000; Warfield et al. 2000; Ferrant et al. 2000; Warfield et al. 2001; Rexilius 2001; Ferrant et al. 2001; Guimond et al. 2002).

A robust volumetric non-rigid registration scheme for brain deformations has been introduced by our group (Clatz et al. 2005). These studies, using intra-operative brain non-rigid registration, were demonstrated using only retrospective data. To our knowledge, there is no published prospective study on non-rigid registration of pre-operative imaging (T1, fMRI, DTI) with intra-operative images (T1). Five major contributions are presented in our manuscript: (i) our study is prospective, with 11 patients enrolled over one year; (ii) the alignments are achieved in near real-time during the neurosurgical procedure; (iii) we aligned both the fMRI and DTI using non-rigid registration; (iv) we presented quantitative results assessing the accuracy of the rigid and the non-rigid registration accuracy; (v) and we prospectively utilized a new volumetric non-rigid registration scheme, that has previously been assessed only retrospectively (Clatz et al. 2005), offline and outside of the operating room.

Materials and Methods

Patient population

Eleven consecutive patients (6 female, 5 male, age range: 28–62 years; mean: 45.2 years) with supratentorial gliomas (World Health Organization (WHO) grading: II: 5, III: 4, IV: 2) were included in our study. All patients underwent surgery at our institution's intraoperative MR image–guided therapy facility between April 2005-January 2006 for tumors in and adjacent to eloquent brain areas (such as the precentral gyrus and cortico-spinal tract, for motor function; and Broca's and Wernicke's areas, for language function). For these patients, DTI was judged necessary for preoperative surgical planning. The study was carried out with Institutional Review Board approval.

Pre-operative imaging

After providing informed consent, patients underwent the following MR imaging protocol on a General Electric (Milwaukee, WI) 3T Signa scanner several days before their scheduled surgery:

1. Anatomic Imaging: (1) whole brain sagittal 3D-SPGR (slice thickness 1.3mm, TE/TR=6/35 msec, FA=75°, FOV = 24cm, matrix=256×256); (2) axial T2-weighted fast-spin-echo (slice thickness 5mm, TE/TR 100/3000 msec, FOV=22cm, matrix=512×512).

2. Functional MRI: Functional MRI was performed using a GE (Milwaukee, WI) 3 Tesla Signa scanner. Whole-brain functional images were acquired with a T2*-weighted echo-planar (EPI) sequence sensitive to the blood oxygen-level dependent signal (TR, 2000 ms; TE, 30 ms; matrix, 64 ×64 mm; FOV, 240 mm; imaging 24 contiguous slices of 5 mm thickness). Visual stimuli were presented using a PC laptop (Dell, Inc., Austin,TX) running either E-prime (Psychology Software Tools, Pittsburgh, PA) or the Presentation (Neurobehavioral Systems Inc., Davis, CA) software package, on an MR-compatible goggle system (Resonance Technology, Northridge, CA). Auditory stimuli were presented with MR-compatible headphones (Avotec Inc., Stuart, FL).

   Patients performed a clinically-relevant set of tasks from a battery of motor, language, and visual paradigms. All fMRI stimuli were visually-presented (except one language task as noted below). Motor mapping tasks were block paradigms of twelve alternating 20-second blocks of active finger-tapping or hand-clenching and rest. The visual mapping task was a block paradigm of twenty 20-second blocks of pattern-reversal (2-Hz) checkerboards of right, left, and whole visual field stimulation, and rest, whereby patients were instructed to maintain fixation on a point in the center of the screen. And language mapping tasks consisted of: a silent, block paradigm of twelve alternating 20-second blocks of antonym generation and rest; a silent, block paradigm of eighteen alternating 20-second blocks of word generation (i.e., verbal fluency) and rest; a vocalized, event-related antonym generation paradigm (average ISI=8.3 sec., 50 stimuli); a silent, event-related Stroop task, whereby the patient performed congruent and incongruent color naming of color words (2-4 sec. pseudo-random ISI, 90 stimuli); and a vocalized, event-related, aurally-presented, semantic concrete words paradigm (“alive” or “not alive” response; average ISI=8.3 sec., 50 stimuli). Functional images were motion-corrected, smoothed (8 mm Gaussian kernel), and analyzed with SPM2 (Wellcome Department of Cognitive Neurology, London, UK), using the general linear model (Friston et al. 1995). Differences between stimulus and baseline conditions, convolved with the hemodynamic response function, were examined using analysis of co-variance with global signal and low frequency components treated as nuisance covariates. Correction for multiple comparisons was performed using the theory of Gaussian random fields (Friston et al. 1996).

3. Diffusion Tensor Imaging: axial line scan diffusion images (LSDI) (Slice thickness=5mm, matrix=512×512, FOV=24cm) and EPI DTI (128×128, Phase FOV=1.0, FOV=25.6, Slice thickness=3, Bvalue=800, 31 directions, number of T2=1) were acquired, covering the entire region of interest as well as “landmark” regions, i.e. areas where the relevant fiber tracts show high density (e.g. ventral brain stem for the corticospinal tract, lateral geniculate body for the optic radiation).

4. MR-Spectroscopy from the tumor was also acquired for three patients.

Intra-operative imaging

After the patients were positioned for craniotomy and their heads were fixated by using an MR-compatible carbon fiber Mayfield clamp (Ohio Medical Instruments, Cincinnati, Ohio), they were examined with the following initial imaging protocol by way of a vertically open 0.5-T iMR imaging unit (SignaSP; GE Medical Systems, Milwaukee, W) with the following parameters: (a) for transverse, sagittal, and coronal T1-weighted fast spin-echo imaging, 700/29 (repetition time msec/echo time msec), a 22-cm field of view, a 256 × 256 matrix, one signal acquired, a 3-mm section thickness, and a 1-mm intersection gap; (b) for transverse T2-weighted fast spin-echo imaging, 5000/99, a 22-cm field of view, a 256 × 256 matrix, two signals acquired, a 3-mm section thickness, and a 1-mm intersection gap; and (c) for transverse three-dimensional spoiled gradient-echo (SPGR) imaging, 15.5/5.2, a 45° flip angle, a 22-cm field of view, a 256 × 256 matrix, one signal acquired, a 2.5-mm section thickness, and a 0-mm intersection gap. During the surgical intervention, T2-weighted fast spin-echo and three-dimensional SPGR MR image updates were obtained after the dural opening, and at any time the neurosurgeon believed that a brain shift had occurred or a substantial amount of tumor tissue had been removed. The final imaging protocol, which was performed after the dura mater was closed, included the same sequences that were performed at initial imaging. The initial and final image datasets were used to determine the initial and residual tumor volumes, respectively.

Rigid registration with the first acquired intra-operative dataset

As surgery began, before opening the dura mater, a first intra-operative T1 scan was always acquired as part of the current protocol. Because there is no brain shift at this stage, a rigid registration between the pre- and first intra-operative T1 is sufficient.

The rigid alignment is performed using software tools developed by our research group, using an approach based on Mattes mutual information metric. This method is automatic and requires no human interaction. Registration between two datasets takes approximately 30 seconds on a conventional workstation, and is quite satisfactory given the time constraints of neurosurgery.

Non-rigid registration

As the tumor resection progressed after opening the dura mater, brain deformation would inevitably occur. Brain shift is influenced by tissue characteristics, intraoperative patient positioning, opening of the ventricular system, craniotomy size, and resected volume.

Intra-operative T1 and T2 scans are acquired to asses the patient's status and the intraoperative shape of the brain. These intraoperative scans now allowed us to update the preoperative high-resolution structural images, by non-rigid registration with the intra-operative T1 dataset.

Non-rigid registration algorithms are time consuming, and therefore, to date several groups have reported these algorithms as being impractical for real-time use in neurosurgical procedures (Nimsky et al., 2006). Therefore, only rigid registration methods (which does not account for brain deformations) are used in commercial navigational systems (such as ones developed by Medtronic and BrainLab). To address this inconvenient issue, our group has developed a software tool that has proven to be sufficiently robust for volumetric, non-rigid registration between two 3D MR images (Clatz et al., 2005), using a high-performance computational infrastructure. Despite being computational expensive, the full biomechanical model has previously shown to be robust and accurate, which is one of the essential requirements in image-guided neurosurgery.

The image-guided neurosurgery architecture is presented in Figure 1. It uses a patient-specific registration algorithm that estimates the brain deformation between two 3D volumetric MRI images (presented in Figure 2). The algorithm can be decomposed into three main parts. The first part consists of building the patient-specific model utilizing intra-operative MRI. The second part is comprised of a block matching computation for selected blocks which estimates a set of displacements across the volume. The third part is an iterative hybrid solver that estimates the 3D volumetric deformation field, utilizing the anisotropic information provided by the structure tensor computed for each block.

Figure 1.

The current system for the image guided neurosurgery that integrates our novel non-rigid registration technology.

Figure 2.

Two Dimensional View of Brain Shift. (a) Pre-operative Image. (b) Intra-operative Image (after the dura has been open and part of the tumor removed).

One advantage of our proposed scheme is that a large part of the computation can be performed before the intra-operative MR images acquisition. In the following section, we describe the sequence of actions of the algorithm and distinguish between the preoperative and intra-operative calculations. Since the pre-operative images are available hours or days before surgery, we can use robust, accurate, time consuming pre-processing algorithms to 1) segment the brain, 2) generate the patient-specific biomechanical model of the brain, 3) select blocks in the pre-operative images with relevant information, and compute the structure tensor in the selected blocks.

1. Pre-operative data processing

Following the pre-operative imaging acquisition, a biomechanical model of the brain is built. The following steps are performed prior the surgery:

• mage segmentation: The delineation of brain in the preoperative data is achieved using segmentation strategies optimized for the particular type of acquisition (Warfield et al. 2000). Recently we have also successfully used a method based on a deformable model which evolves to fit the brain's surface by the application of a set of locally adaptive model forces (Smith 2002).

• Mesh generation: Tetrahedral discretization (volume mesh) of the segmented intra-cranial cavity serves as the basis for Finite Element Method (FEM) modeling of the physical tissue deformation, and serves as the function of regularization on the estimated displacements obtained as the result of block matching step of non-rigid registration. The technique used for tetrahedral mesh generation is described in (Fedorov et al. 2006). This is a fast and effective mesh generation algorithm that generates elements of high quality.

• Biomechanical model: As in our past work (Ferrant et al. 2001; Ferrant et al. 2002; Warfield et al. 2002) we discretized the partial differential equation describing the behavior of the brain under deformation by utilizing the finite element method. We model the brain material using an incompressible linear elastic constitutive equation to characterize the mechanical behavior of the brain parenchyma. Because the ventricles and subarachnoid space are connected, the CSF is free to flow from one to another. We thus assume very soft and compressible tissue for the ventricles. The skull is implicitly modeled by preventing the brain mesh vertices from moving outside the brain segmentation. Biomechanical model is used to bring additional information mostly in the vicinity of the tumor resection. It is also used in areas where outliers are identified and rejected. The outliers typically occur in homogenous regions of the brain structural MRI.

• Block Selection: As we described previously (Ruiz-Alzola et al. 2002), we select blocks with high variance of signal intensity to ensure sufficient information is present to compute matches. Additionally, the ill-posed nature of finding correspondences in the resection cavity is anticipated, by performing the block selection in a mask corresponding to the segmentation of the brain without the tumor segmentation.

2. Intra-operative data processing

Extension of Block Matching Algorithm with Outlier Rejection

Also known as template or window matching, the block matching algorithm is a simple method used for decades in computer vision (Dengler and Schmidt 1988; Ruiz-Alzola et al. 2002). It makes the assumption that a global deformation results in translation for small parts of the image. Then the complex global optimization problem can be decomposed into many simple ones: considering a block B(O_k) in the reference image centered in O_k, and a similarity metric between two blocks M(B_a,B_b), the block matching algorithm consists in finding the positions $O_k^{\prime }$ that maximize the similarity:

$$\underset{O_k^{\prime }}{\text{arg max}}\left[M\left(B(O_k),B(O_k^{\prime })\right)\right]$$

That is, we seek to identify the smallest block center displacement that maximizes the similarity between the reference image and the target image.

Performing this operation on the selected blocks in the pre-operative image produces a sparse estimate of the displacement between the two images. The choice of the similarity function has largely been debated in the literature, we will refer the reader to the article of Roche (Roche, Malandain, and Ayache 2000) for a detailed comparison of them. In our case, the mono-modal (MR T1 weighted) nature of the registration problem authorizes us to make the strong assumption of an affine relationship between the two image intensity distributions, and so we use the correlation coefficient:

$$c=\frac{{\sum }_{X\in B}\left(B_F(X)-\overline{B_F}\right)\left(B_T(X)-\overline{B_T}\right)}{{\sum }_{X\in B}\left(B_F(X)B_T(X)-\overline{B_F}\overline{B_T}\right)}$$

where B_F and B_T respectively denote the block in the floating and in the reference image, and B̄ is the average intensity in block B. In addition, the value of the correlation coefficient for two matching blocks is normalized between 0 and 1 and reflects the quality of the matching: a value close to 1 indicates two blocks very similar while a value close to 0 for two blocks very different. We use this value as a confidence in the displacement measured by the block matching algorithm.

Estimation of Displacement Field from Block Displacements

The remaining step is to compute a volumetric displacement field from the block displacements. In this section, we summarize o ur concept of approaching the classical energy minimization formulation of this problem by moving from first an approximation problem to interpolating after robust outlier rejection. The reason for this new formulation is that the approximation formulation, which balances match quality against a regularization term, performs well in the presence of noise, but suffers from a systematic error, whereas the exact interpolation problem gives an undesirable solution when it is based on noisy or erroneous data.

We propose an algorithm that takes advantage of both formulations to iteratively estimate the deformation from the approximation to the interpolation formulation, while rejecting outliers. Let K be the stiffness matrix representing the brain mechanical properties, H be a linear interpolation matrix and S the structure tensor. We seek the displacement field represented by U and denote the block displacements by D. The gradual convergence to the interpolation solution is achieved through the use of an external force added to the approximation formulation:

$$\left[K+H^T\mathit{\text{SH}}\right]U=H^T\mathit{\text{SD}}+F$$

This force F_i is computed at each iteration i to balance the mesh internal mechanical energy U^T_iKU_i. This leads to the iterative scheme:

$$\begin{array}{l}{F}_i\leftarrow K{U}_i\\ U_{i+1}\leftarrow {\left[K+H^{T}SH\right]}^{-1}\left[H^{T}SD+F_i\right]\end{array}$$

The transformation is then estimated in a coarse to fine approach, from large deformations to small details up to the interpolation. This scheme is guaranteed to converge toward the interpolation formulation. The above equations are iterated to convergence, incorporating outlier rejection as described below. This new formulation combines the advantages of robustness to noise at the beginning of the algorithm and accuracy when reaching convergence. Because some of the measured displacements are outliers, we propose to introduce a robust block-rejection step based on a least-trimmed squares algorithm (Rousseeuw and Van Driessen 1999). This algorithm rejects a fraction of the total blocks based on an error function measuring for block at O_k the error between the current mesh displacement and the matching target. The error measure is thus simply the estimated displacement error weighted according to the direction of the intensity gradient (structure tensor) of the block.

The mechanical system was solved using the conjugate gradient (Luenberger 1969) method with the GMM++ sparse linear system solver with a maximum error set to 10⁻⁴mm. The rejected block fraction for 1 iteration was set to 2.5% and the number of rejection steps to 10.

During the neurosurgical procedures, a three dimensional displacement field between the pre-operative and intra-operative T1 images was estimated using the strategy described above.

High performance computational architecture

Non-rigid registration algorithms are typically computationally expensive and often proven to be impractical for solving clinical problems. In a previous retrospective study, we employed a cluster of computers to achieve near-real time performance. Our implementation addresses three aspects: (1) load balancing, (2) fault-tolerance and (3) ease-of-use for parallel and distributed registration procedures. With dynamic load balancing we improved by 50% the performance of the most computational intensive part, parallel block matching. Our 2-level fault-tolerance introduced a moderate 6% overhead due to additional communication. With web-services and by hiding pre-processing overheads, we developed faster and easier to use remotely registration procedure. Details of the novel technology can be found in Chrisochoides et al. (2006). The system architecture is presented in Figure 3.

Figure 3.

The architecture used to perform the non-rigid registration computations required by the neurosurgery. The data is transferred from the operating room (MRT) at our institution (BWH). The computations are performed during the case, and then are transferred back to BWH in the OR. The complete process takes approximately 5 minutes.

The datasets are transferred during the surgery from the operating room to the supercomputing facility. The computations are performed and subsequently, the results are transferred back to our institution. And finally, the registration results are displayed in the OR.

Augmented Reality Visualization with FMRI

The fMRI, after processing, is stored as a 3D dataset. The displacement fields estimated with the non-rigid registration algorithm are used to deform the pre-operative image and match the intra-operative anatomy. Regions of activation are rendered with color coding on cross-sectional slices, and as triangle models that illustrate the 3D region of activation.

Augmented Reality Visualization with DT-MRI

The fiber tracts are reconstructed on the pre-operative images, before the surgery starts. For the present study, we have used the 3D tractography method described by Westin et al. (2002 and Park et al. (2004). For initialization of the 3D tractography algorithm, seeding points were placed in the landmark areas. Alternatively, seeding points were placed in the subcortical white matter subjacent to areas of cortical activation seen on fMRI. The motor fibers are identified as described by Mori et al. (2002) and Nimsky et al. (2006).

During surgery, the displacement field between T1w MR images is estimated with the non-rigid registration algorithm. The same displacement field is applied to the pre-operative DTI data, using the method described in Sierra (2001) and Ruiz-Alzola et al. (2002). Then the seeds used in the pre-operative phase for tractography are deformed using the displacement field, so they match the brain anatomy during tumor resection. The novel seeds are used to re-generate the tracts of interests. On a standard workstation it takes less then 30 seconds; therefore, this step is not a concern with regard to the execution time within the context of neurosurgery.

The patient coordinate system is extracted from the DICOM images acquired with the intra-operative MRI. The augmented reality visualization can then be presented in the patient space, and available for navigation with surgical instruments, based on 3D Slicer (www.slicer.org) our open source software.

Estimation of registration accuracy

The non-rigid registration algorithm estimates a displacement field that warps the pre-operative structural MRI to the intra-operative T1w imaging, accounting for brain deformations occurring during tumor resection. The displacement field is subsequently applied to the pre-operative image. This new obtained image must match the intra-operative image of the brain.

We measured the accuracy of alignment between these two images, by extracting the edges from the images using a Canny edge detector, as in Figure 9. The 95% Hausdorff metric (Hausdorff 1962) is used to estimate the registration accuracy. The Hausdorff distance is the maximum distance of a set to the nearest point in the other set. More formally, Hausdorff distance from set A to set B is a maxi-min function, defined as:

Figure 9.

Validation of the non-rigid registration alignment. (a) Non-rigid registered pre-operative T1 3T. (b) Intra-operative image T1 0.5T. (c) Contours extracted from (a) with the Canny edge detector. Contours extracted from (b) with the Canny edge detector. The accuracy of alignment is computed between the points on (c) and (d).

$$h(A,B)=\text{max}{}_{a\in A}{}^{}\left\{\text{min}{}_{b\in B}{}^{}\left\{d(a,b)\right\}\right\}$$

where a and b are points of sets A and B respectively, and d(a, b) is Euclidian metric between these points. The 95% Hausdorff distance is then calculated between the two set of points representing the edges, and it represents the accuracy of the alignment. The ideal case with perfect alignment is when the Hausdorff distance is equal to zero.

The non-rigid registration is compared with the rigid registration only, which represents the current state of the art. Therefore, two Hausdorff distances are computed between (1) the deformed registered image, and the intra-operative image, and (2) the preoperative data are rigidly aligned with the intra-operative image. The ratio between the (1) and (2) is computed and represents the factor of improvement of our novel technology.

Results

The system has been evaluated during 11 consecutive neurosurgery cases. The data has been transferred, processed and displayed in the OR during the neurosurgery. A total of 19 non-rigid registrations have been performed. All the alignments for all the datasets acquired during the tumor resection have been successfully estimated.

Examples of meshes used in the brain biomechanical model are presented in the Figure 4. For all the cases, meshes have been successfully generated. The rigid alignment of the datasets (T1, T2, fMRI, DTI) and the segmentation of anatomical structures of interest are performed. A complex scene is built and used for surgical planning. An example is illustrated in the Figure 5.

Figure 4.

Meshes of the brain used to generate the biomechanical model.

Figure 5.

Alignment of preoperative imaging is performed. Tumor segmentation is also carried out, prior the surgery. The images show the T1, DT-MRI tractography, and fMRI alignment together with the 3D reconstruction of the tumor. Neurosurgeons have complex information available to decide the best strategy to adopt for the craniotomy.

For each patient, we measured the alignment accuracy between (1) the deformed registered image, and the intra-operative image, and (2) the preoperative data rigid aligned with the intra-operative image. The ideal case is when the distance in (1) is zero. The (2) is used as a comparison, as only rigid registration is performed in current clinical procedures. Overall, the minimum error for our novel registration technique is 0.85mm, the mean 1.82mm, and the max 3.57mm. The mean ratio between (2) and (1) is 4.58; which reveals that our results can be at least 4.58 times more precise than current technology. There is a statistically significant difference between the accuracy of the alignment of pre-with intra-operative images, with and without non-rigid registration (p<0.001). The complete results are presented in Table 1. The resected tumor can be a maximum 3 voxels off its contour, as seen on the MR imaging. Typically, the voxel size on the axial plane is 0.8 mm. Therefore, the registration accuracy should be < 2.4 mm.

Table 1.

Numerical accuracy results for the non-rigid registration algorithm and its comparison with conventional rigid-registration.

	Tumor position	Sex/Age	Registration between preoperative and intraoperative scans (95% Haussdorf distance)
			Max Displacement measured (mm)	Rigid registration accuracy–preop to intraop (mm)	Non-Rigid registration accuracy–preop to intraop (mm)	Ratio Rigid/Non-Rigid
Case 1	right posterior frontal	F/29	10.68	5.95	1.90	3.13
Case 2	left posterior temporal	M/54	21.03	10.7	2.90	3.69
Case 3	left medial temporal	F/57	15.27	1 7.65	1.70	4.50
Case 4	left temporal	M/54	10.00	6.80	0.85	8.00
Case 5	right frontal	F/33	9.87	5.10	1.27	4.01
Case 6	left frontal	M/62	17.48	10.20	3.57	2.85
Case 7	right medial temporal	F/50	19.96	9.35	2.55	3.66
Case 8	right frontal	M/40	17.44	8.33	1.19	7.00
Case 9	right frontotemporal	M/28	15.08	7.14	1.87	3.81
Case 10	right occipital	F/56	9.48	5.95	1.44	4.13
Case 11	left frontotemporal	M/34	10.74	4.76	0.85	5.60
AVG			14.27	7.44	1.82	4.58

Alignment results of the preoperative with intra-operative datasets are presented in Figure 6. Fiber tracts that were deformed due to brain shift were calculated and made available for the neurosurgeon during the tumor resection (Figure 7). The fMRI activation areas were updated, thereby matching the deformed brain, and used to seed the fiber tracts.

Figure 6.

Non-rigid registration of preoperative imaging (T1, fMRI, DTI) with intraoperative imaging. We enhance the current procedure, by aligning preoperative imaging with intraoperative imaging. The DTI, fMRI, T1 images are displayed during tumor resection. Damage of critical structures can be avoided, while achieving gross tumor resection.

Figure 7.

White matter fiber tracts deformation during the neurosurgery. The pre-operative data is shown in the images (a) and (b), while the intra-operative imaging is shown in (c) and (d). A significant displacement of the fiber tracts can be noticed.

One of the hypotheses investigated is that relevant white matter fiber tracts pass through the fMRI activation areas. An example of such a scene created within the time constraints imposed by the neurosurgery procedure is illustrated in the Figure 8.

Figure 8.

Significant brain deformations require precise re-alignment of pre-operative fMRI and DTI datasets. (a) Illustrates the pre-operative images aligned (T1, fMRI) and several fiber tracts located in the vicinity of the tumor. Some of them are also crossing the tumor. Tumor is represented with green, while the fMRI in blue, and fiber tracts in magenta. (b) The fMRI is realigned with the intra-operative images, while compensating for the brain shift. The preoperative fiber tracts are also realigned and displayed during tumor resection (in yellow). (c) Shows the fMRI and fiber tracts before and after craniotomy. It is important to notice the deformation occurred.

The non-rigid registration algorithm between T1 images (including the loading and saving the output data) is executed in 179 seconds using 240 processors. The execution time for the algorithm itself (once data loaded from the hard-drive) is about 1 minute.

The transfer of images between the two sites takes aproximatelly 1 minute. The orientation of each of the pre-operative dataset is achieved in less than o ne minute. Overall, this is satisfactory given the time constraints imposed by neurosurgeries. Illustrative examples of the validation algorithm are shown in the Figures 9 and 10.

Figure 10.

(a) 95% Hausdorff distance between the points on the edges of the registered image and intra-operative image. (b) presents the 95% Hausdorff between points on the edges on the rigid aligned preoperative image and the edges on the intra-operative data. The number of points on the extracted edges is different in the two cases, since typically MR 3T images have higher contrast.

The non-rigid registration algorithm was also tested on a standard workstation (2-way Dell Precision Workstation 650, Intel(R) Xeon(TM) 3.06GHz with 1MB Level 3 cache 512KB L2 cache and 3.5GB RAM). Between 15 and 40 minutes are needed to run the non-rigid registration algorithm on a workstation with 4 processors. Therefore, complex facilities involving high performance computing are currently necessary for use during neurosurgery. However, the performance of the standalone workstations is continuously improving. And, based on the dramatic increase in computer performance over the past decade, we envision in the near future a marked reduction in execution time for this novel technology on standard workstations. The current study establishes its feasibility in a clinical context. The neurosurgeons involved in our study have also reviewed the registration results. Anatomical landmarks (such as ventricles, putamen, gyrus, midline), visible on both pre-operative and intra-operative imaging, were used. For all the patients enrolled in our study, these anatomical landmarks matched o ur non-rigid registration results with intra-operative findings. Therefore, the results were considered clinically useful.

Discussion

Enhanced navigation using pre-operative multi-modal images (fMRI, MRI-DTI) can provide useful information during tumor resection. However, brain shift can produce non-linear changes in the anatomy that will induce inaccuracies of conventional rigid registration techniques (used by current commercial neurosurgery navigation systems).

Modeling the behavior of the brain remains a key issue in providing a priori knowledge for image-guided surgery. Sophisticated models of brain tissue undergoing surgery are presented and validated in Miga et al. (2000) and Platenik et al. (2002). However, a practical difficulty of these models is the extensive time necessary to mesh the brain and solve the problem, which is too long for intra-operative purposes. Similar approaches have investigated the use of brain biomechanical models updated during neurosurgery, using brain surface measurements based on laser range scanner (Miga et al. 2003) or stereo vision system based (Skrinjar, Nabavi, and Duncan 2002).

The use of intraoperative ultrasound in order to provide data that could be used to update pre-operative models to account for brain shift has been also investigated. (Dey et al. 2002; Gobbi and Peters 2003) have utilized a tracked, free-hand ultrasound probe. Pennec, Cachier, and Ayache (2003) demonstrated a system that utilizes full-volume, intensity-based registration and 3D ultrasound, rather than the landmark-based methods discussed above.

These studies show that several intraoperative imaging modalities have the potential to accurately measure brain deformation, but that further study is needed. Methods that require the identification of landmarks in the intraoperative image need to overcome the difficulty of robust landmark detection in those images. Also, further evaluation in the operating room is needed to determine how well these methods can capture, and adjust for, true brain deformation, as opposed to the mechanically-induced simulations that have been investigated thus far.

An interesting aspect is how our mathematical model for DTI reorientation compares with the intra-operative DTI acquired at 0.5T. Our group has demonstrated the feasibility of acquiring intra-operative diffusion weighted imaging in the 0.5 T MRI (Mamata et al. 2001). However, for neurosurgical procedures, this is impractical due to the long acquisition time (scan time per slice 94 seconds for diffusion tensor imaging, and 46 seconds for diffusion trace imaging), and poor spatial resolution of the diffusion weighted images (rectangular FOV = 260×195 mm; effective slice thickness = 7 mm; slice gap = 3mm; matrix size = 128 × 48). Therefore, for our prospective study we did not have diffusion weighted intra-operative images of the patients. Nevertheless, future studies will be performed on animals and phantoms to compare the results of intra-operative DTI with our mathematical estimation.

The similarity between the DTI datasets will be computed based on the metrics recently introduced by Fletcher et al. (2004), Moakher et al. (2005), Arsigny et al. (2006), and Schwartzman (2006).

An important aspect is the way multi-modal imaging information is presented during tumor resection to neurosurgeon. Relevant fiber tracts for each patient are displayed during tumor resection, along with information about the fMRI activation areas. Because the DTI data is also non-rigidly registered with the brain anatomy during the tumor resection, extracted fiber tracts correspond precisely to the brain changes induced by the surgery. And our system works effectively even if some parts of a fiber tract system (such as visual pathway) were removed during the surgical procedure, or in a case where fiber tracts become visible after resection that were not before the resection (due to compression).

The proposed solution fits well within the time constraints imposed by neurosurgery at the MRT. An initial SPGR scan is acquired before the craniotomy (scanning time is approximately 12 minutes). The first rigid registration between the pre-operative T1w MRI and the first intra-operative SPGR is performed, and an affine transformation is estimated. Less than 30 seconds are needed to perform this task on a conventional workstation. The other pre-operative images (T2, fMRI, DTI) are also aligned with the intra-operative SPGR based on the estimated affine transformation (in approximately 30 seconds on a conventional workstation). At this point, neurosurgeons will express an interest in having the fMRI and DTI information during the navigation, to decide about the best strategy for the craniotomy.

As the surgery progresses (depending on the tumor histology and location), brain deformation inevitably occurs, and the initial alignment becomes inaccurate. A new SPGR scan is acquired, typically after 45 minutes. The pre-operative T1w MRI scan is non-rigidly registered with the intra-operative scan, using the grid-computing architecture previously presented, and the results are available in less than 5 minutes. The other pre-operative datasets (T2w, fMRI, DTI) are also aligned with the intra-operative brain anatomy in less than 1 minute on a conventional workstation. The multi-modal images are registered and available for navigation in less than 7 minutes from the scan acquisition time. At this point, the neurosurgeons are interested in the fiber tracts that can be found in the vicinity of the tumor. Towards the end of neurosurgery, additional intra-operative SPGR scans are often acquired, typically every 30 minutes. The non-rigid registration process is performed for each new SPGR scan, and hence, the time constraints are extremely significant.

An interesting question is whether the introduced technology could potentially improve the patients outcome, by reducing the tumor residual, while avoiding clinical deficits. Our group has recently published a study that assesses the main variables that affect the complete magnetic resonance (MR) imaging–guided resection of supratentorial low-grade gliomas (Talos et al. 2006). Of all the variables assessed individually in the univariate analyses, 11 were found to be significantly associated with incomplete tumor resection (Table 1). Among the tumor characteristics, an ill defined tumor margin on T2-weighted MR images, LGO or low-grade mixed oligoastrocytoma histopathologic tumor type (ie, both types appear to be more difficult to resect than LGA), and large tumor volume were found to be associated with incomplete resection. Furthermore, tumor involvement of the following functionally critical structures led to incomplete resection: corpus callosum, CST, insular lobe, middle cerebral artery, primary motor cortex, optic radiation, visual cortex, and basal ganglia (one-sided P < .05 for all correlations).Therefore, in a future prospective study, we will employ our technology on a large number of patients that have predictors of potentially incomplete tumor resection. The intra-operative information from DTI, as provided with our novel system, can contribute to improved patients' outcome.

Conclusions

The utility of enhanced visualization during neurosurgery procedures is clearly demonstrated in the literature. To date, commercial image-guidance neurosurgery systems can only rigidly align the pre-operative, high resolution imaging data with the intra-operative imaging data. However, rigid alignment can induce errors as high as 20 mm, as measured in our study. To date, there is no fully volumetric, non-rigid registration of brain deformations demonstrated in a clinical environment during a neurosurgical procedure.

We demonstrated the feasibility of our system utilizing volumetric non-rigid registration, and quantitatively measured its value with respect to the commercially available state-of-the-art technology, based on rigid registration methods only. We found that a statistically significant increase in alignment accuracy was achieved using non-rigid registration.

Five major contributions are presented in our manuscript: (i) our study is prospective, with patients enrolled over 1 year; (ii) the alignments are achieved during the neurosurgery; (iii) we aligned both fMRI and DTI using non-rigid registration; (iv) we presented quantitative results assessing the accuracy of the rigid and the non-rigid registration; (v) we utilized prospectively a new volumetric non-rigid registration scheme, that has previously been assessed only retrospectively.