Automatic tracking of intraoperative brain surface displacements in brain tumor surgery

Abstract

In brain tumor surgery, soft-tissue deformation, known as brain shift, introduces inaccuracies in the application of the preoperative surgical plan and impedes the advancement of image-guided surgical (IGS) systems. Considerable progress in using patient-specific biomechanical models to update the preoperative images intraoperatively has been made. These model-update methods rely on accurate intraoperative 3D brain surface displacements. In this work, we investigate and develop a fully automatic method to compute these 3D displacements for lengthy (~15 minutes) stereo-pair video sequences acquired during neurosurgery. The first part of the method finds homologous points temporally in the video and the second part computes the nonrigid transformation between these homologous points. Our results, based on parts of 2 clinical cases, show that this speedy and promising method can robustly provide 3D brain surface measurements for use with model-based updating frameworks.

I. Introduction

In brain tumor surgery, brain tissue deformations, commonly referred to as brain shift, can produce inaccuracies of 1–2.5cm in the preoperative plan and within image-guided surgery (IGS) systems [1]. Furthermore, the process of soft-tissue resection can also compound the challenge of accounting for soft-tissue changes during IGS. As a result, establishing accurate correspondences between the patient’s physical state and their images is a challenging problem that potentially limits the scope of IGS systems. A method to account for the volumetric soft-tissue shift and deformation is to employ surface data and measurements to drive a patient-specific computational biomechanical model to intraoperatively update the IGS system [2–3]. The textured laser range scanner (tLRS) [3–4] and stereovision systems are a few modalities researched for obtaining organ surface measurements intraoperatively [6–9]. We believe that persistent delivery of digitized 3D organ measurements to drive the model-update framework is sufficient to realize an active and superior IGS system. To achieve this, correspondences between intraoperative 3D brain surfaces need to be established.

The work in [4] proposed a semi-automatic method for establishing these correspondences in pre- and post-resection tLRS surfaces. Previous work in [5] involved performing nonrigid registration on the 2D surgical tLRS images before and after deformation and then finding the full 3D displacements by relating each 2D tLRS image to its corresponding acquired depth measurement. This type of approach resulted in smaller registration errors than performing fully 3D-to-3D nonrigid registration on the point clouds. We take a similar approach here but apply it on the stereovision system we have developed.

Stereovision systems in [6–7] digitized the cortical surface in 3D and established manually delineated correspondences for short sequences of stereo-pair video. Analysis of brain tumor surgery video sequences has been proposed in [9–10] for the tracking of the cortical surface. In [9], the proposed method tracked manually selected points at the bifurcations of vessels in short stereovision video sequences. The method in [10] developed a nonrigid registration algorithm for tracking entire vessels in short monocular video sequences. Using registration between the tLRS and the monocular video, brain shift was estimated.

Though acceptable error were achieved in tracking the cortical surface in brain tumor surgery videos [9–10], the proposed methods were limited in scope when determining homologous points robustly and tracking was effective in short video sequences only. Furthermore, both approaches required frequent manual initializations and interventions. In this paper, we develop an algorithm that robustly determines homologous points in the highly dynamic and lengthy brain surgery video. These homologous points are used in a nonrigid registration framework to compute 2D deformations. With the stereovision methodology we developed for neurosurgery in [8], we are able to digitize these computed deformations to 3D yielding cortical surface displacements persistently throughout the surgery. We present our 2D registration errors on 13- and 15-minute video sequences from 2 clinical cases.

II. Methods

A. Data acquisition

Brain tumor surgery stereo-pair videos were acquired for the clinical cases under Vanderbilt University’s IRB approval. The videos were acquired at 30 frames per second using the OPMI® Pentero^™ operating microscope (Carl Zeiss, Inc., Oberkochen, Germany) equipped with two internal CCD cameras (Zeiss’ MediLive® Trio^™), and have NTSC (720×540) image resolution.

B. Stereovision point clouds

Our stereovision work is based on our previous work presented in [8], where stereo calibration is achieved using a planar chessboard pattern shown in 10–15 different poses to the operating microscope’s stereo-pair cameras. Stereo calibration accuracy between 0.67–0.81 pixel² is achieved. After stereo rectification, stereo correspondence is performed using the block-matching algorithm, with a window size of n_BM, to find disparities between left and right camera images. These disparities are used in projecting the pixels in left and right images to 3D. It should be noted that each left-right image pair has an associated disparity image, which is used for finding 3D points.

C. Homologous points

We use the left camera video of the neurosurgery as input into our algorithm in Sections C and D. To robustly detect scale- and rotation-invariant salient features in the surgical video, we use the SURF detector because of its fast computation time [11]. The image location, in pixels, of these salient features is called a keypoint. The SURF feature detector yields a 128-float feature descriptor per keypoint in the image. Let φⁱ be the set of keypoints detected at t_i and φ^j be the set of keypoints detected at t_j, where t_i < t_j and are within a temporal range of a few seconds. A hessian threshold, th_SURF, determines the number of SURF keypoints detected per image frame [11].

A matching stage establishes correspondences between φⁱ and φ^j sets of SURF keypoints. The putative matching between φⁱ and φ^j are determined using an approximate nearest neighbor approach on the 128-float SURF feature descriptors of the keypoints using k-d trees. We use the computationally fast implementation of k-d trees from the FLANN library to get putative matches [12]. Though this approach results in correspondence mapping, it does so with several mismatches or outliers. To remove spurious matches, a homography matrix, H, is computed using the RANSAC method. Homography preserves the fact that if three keypoints lie on the same line in one image, then these keypoints will be collinear in the other image as well, as shown in (1a) [13–14]. The RANSAC-based estimation of H, Ĥ, maximizes the number of inliers, n, of all the putative correspondences, (φⁱ, φ^j), subject to the reprojection error of (1b), ε_H. These n inliers are homologous points. Note that this procedure determines a different n for different image pairs of t_i and t_j. More details related to this method can be found in [8, 13]. This standard technique yields the homologous points, (pⁱ, q^j), where pⁱ ∈ φⁱ and q^j ∈ φ^j. Homologous points from this fully automatic method steer the registration stages. Figure 1 shows the determined homologous points between two image frames for two different clinical cases.

Figure 1.

The left and right columns are of different brain tumor surgery cases. Row (a) of both cases shows the results of nearest-neighbor matching between SURF keypoints at t_i and t_j time points. Row (b) shows the results of the homography procedure for cleaning up mismatches to find the homologous points between t_i and t_j.

$${\phi }^j=H{\phi }^i$$ (1a)

$${\text{argmax}}_{\widehat{H}}\sum \left({\Vert {\phi }^j-\widehat{H}{\phi }^i\Vert }_2<{\epsilon }_H\right)$$ (1b)

D. Registration

The registration stage is split into rigid and nonrigid registration steps. The rigid registration accounts for global movement changes in the FOV of the surgery video. These types of global movements are attributed to movement of the operating microscope’s head and small movements introduced by the neurosurgeon while performing tumor resection. The rigid-body transformation, T_r, is computed between pⁱ to q^j, where pⁱ are the source points and q^j are the target points [15].

To account for local movement, a nonrigid registration refinement step is used on the rigidly transformed homologous points and is based on Thin Plate Splines [16]. The number of TPS control points used is equal to the n homologous SURF keypoints obtained from Section C. The smoothness of the resulting deformation field is controlled by a regularization parameter, λ, [17], which is annealed over n_nr iterations using γ as the annealing parameter. This is described in (2), where a unique minimizer f solves the variational problem between the source and target points. The regularized TPS finds the deformation field between successive time points, t_i and t_j, as F_ij.

$$\begin{array}{c}E(f)={min}_f{\sum }_{i=1}^n{\Vert q_i-f(T_r(p_i))\Vert }^2+{\lambda }_{\gamma }\iint \left[\frac{{\partial }^{2}f}{\partial x^2}+\frac{{\partial }^{2}f}{\partial y^2}+2\left(\frac{{\partial }^{2}f}{\partial x\partial y}\right)\right];\\ {\lambda }_{\gamma }={\lambda }_0\ast {\gamma }^k;k=(1\dots n_{nr})\end{array}$$ (2)

E. Computing 3D displacements

Let φ₀ be the set of keypoints detected at t_i=0 or t₀ of the left camera video. These φ₀ keypoints have associated disparities, Δ⁰, and using the stereovision method of Section B, 3D points, Z⁰, are determined. The computed deformation field, F_ij, is used for deforming φ₀ to t_j and we denote this as ${\phi }_0^j$. Since F_ij is computed at every successive pair of time points, t_i and t_j, φ₀ is deformed in a temporal manner akin to the progression of the surgery. With ${\phi }_0^j$ and associated stereo disparities, Δ^j, 3D points of deformed ${\phi }_0^j$ can be estimated at t_j as Z^j. Now, displacements of the initial φ₀ keypoints from t₀ deformed to t_j and ${\phi }_i^j$ keypoints from t_i deformed to t_j are computed as shown in (3).

$$d_0^j=Z^j-Z^0;d_i^j=Z^j-Z^i$$ (3)

III. Experiments and Results

In this section, we present our results for the image registration algorithm outlined in Section II. Table 1 shows the different parameters and their values used in the presented algorithm. The algorithm was executed on a Windows 7 Dell Precision Desktop T1500 with Intel Core i7 2.80 GHz Processor and 12GB RAM. The 3D point cloud computation took 0.5 seconds per stereo image pair. The homologous features computation and registration steps of Section D took 0.5 seconds. The deformation field computation, F_ij, for a 720×540 image size took 1.5 seconds. Overall, the estimation of 3D displacements on the cortical surface using the fully automatic algorithm can be computed in 2.5 seconds per stereo image pair. To the best of our knowledge, this kind of speed in estimating 3D displacements of soft tissue has not been achieved.

Figure 2 shows the registration of SURF keypoints between image frames of t_i and t_j, where t_i and t_j are 1 second apart, for clinical case 1. Note the different SURF keypoints selected per pair of images. From Figure 2, if the registration between the determined homologous points were accurate, then the blue (target points, t_j) circles and green (source points, t_i) circles would fully overlap. Figure 2(b) shows φ₀ deformed using F_ij computed from t_i to t_j for the image frames shown in Figure 2(a). It is apparent that the keypoints in ${\phi }_0^j$, marked as green circles, are tracked accurately using the registration algorithm over time.

Figure 2.

Row (a) shows the result of registration from clinical case 1’s image frames t_i (source) to t_j (target), accurate registration (F_ij) will make the target (blue) and source (green) circles overlap to form cyan (columns 3 and 4). Row (b) shows the keypoints, φ₀, at t₀ (69.1 seconds) deformed continuously in time by computed F_ij. The locations of the blue keypoints, φ₀, ${\mathit{\phi }}_{\mathbf{0}}^{\mathbf{188}},{\mathit{\phi }}_{\mathbf{0}}^{\mathbf{469}}$, and ${\mathit{\phi }}_{\mathbf{0}}^{\mathbf{667}}$ illustrate the tracking of points on the brain for the duration of surgery.

To compute fiducial registration errors (FRE) and target registration errors (TRE) [15], ${\phi }_i^j$ are considered “fiducial” points that register image frames from t_i to t_j. On 13- and 15-minute video sequences from 2 clinical cases, we achieve a mean FRE of 1.42 ± 0.32 pixels with approximately 670 fiducial keypoints on average per t_i-t_j image-pair. The FRE is computed automatically and indicates how well F_ij aligns frames t_i and t_j. To compute TRE, a set of points have been marked manually at frames positioned at 25%, 50%, 75% and 100% of the clinical video sequences. The TRE points marked at the frame t=0 of each clinical video sequence forms the φ₀, which is continuously tracked by the proposed algorithm throughout the video sequence. The algorithm’s estimated locations of ${\phi }_0^{25\%},{\phi }_0^{50\%},{\phi }_0^{75\%}$, and ${\phi }_0^{100\%}$ are compared with the manually delineated locations for computing the TRE. It is possible that the some of the many manually marked TRE points may not be visible in the later frames of the video (for example, at frame position of 75%) because of neuro patties, blood, and tumor resection. These points are not used in the TRE analysis. 14 and 21 target points were used in the TRE analysis for the 13- and 15-minute video sequences. We achieve a mean TRE of 6.71 ± 1.95 pixels for the two video sequences. Table II shows the TRE values in greater detail. It should be noted that the locations of points marked manually for TRE analysis can be prone to operator error and bias. Therefore, the presented TRE results can be considered an upper bound on the TRE. Figure 3 illustrates TRE analysis for the 13-minute video.

TABLE II.

Target Registration Errors (PIXELS)

Video	25%	50%	75%	100%	μ ± σ
13 min	5.78	5.74	8.45	9.74	7.43 ± 1.73
15 mins	3.77	4.87	6.56	8.78	5.99 ± 1.89

Figure 3.

TRE in the 13-minute video sequence. Green dots are targets that were manually delineated, blue dots are the algorithm’s estimations of the targets that are deformed continuously in time, and cyan indicates overlap between the green dots and the blue dots.

IV. Discussion

In this paper, we propose a method to track points on the cortical surface in brain tumor surgery videos acquired from the operating microscope. When combined with the stereovision methodology developed by our research group in [8], this speedy method furnishes the required intraoperative 3D cortical surface displacements to drive the model-update pipeline for brain shift compensation. Though this promising method has been tested only on 2 clinical datasets, we are in the process of acquiring and validating this algorithm on more clinical cases. Furthermore, this presented algorithm has been tested on clinical video sequences of 13–15 minutes and will be tested on the complete hour-long surgical sequences. A study where a comparison between the 3D displacements obtained from this stereovision system and the displacements obtained from the pre- and post-resection tLRS is underway at the moment. A classification of the FOV into brain and non-brain regions can drive the registration stage in a more refined manner. This can help deliver 3D cortical surface displacements more accurately. With such an operating microscope-based system capable of persistent delivery of 3D displacements of the cortical surface, a strong, novel, surgical workflow friendly, and functional intraoperative IGS platform capable of real-time soft-tissue surgical guidance is quite achievable in the future.

TABLE I.

Parameters and Values

Variable	Description	Value
nBM	Block matching window size for stereo correspondence	21
thSURF	Hessian threshold for SURF detector	100.0
εH	Homography matrix estimation’s reprojection error for RANSAC	10.0
λ0	Initial regularization parameter for thin plate splines	106
γ	Annealing parameter for TPS	0.93
nnr	Annealing iterations for TPS	3