Superficial vessel reconstruction with a multiview camera system

Abstract.

We aim at reconstructing superficial vessels of the brain. Ultimately, they will serve to guide the deformation methods to compensate for the brain shift. A pipeline for three-dimensional (3-D) vessel reconstruction using three mono-complementary metal-oxide semiconductor cameras has been developed. Vessel centerlines are manually selected in the images. Using the properties of the Hessian matrix, the centerline points are assigned direction information. For correspondence matching, a combination of methods was used. The process starts with epipolar and spatial coherence constraints (geometrical constraints), followed by relaxation labeling and an iterative filtering where the 3-D points are compared to surfaces obtained using the thin-plate spline with decreasing relaxation parameter. Finally, the points are shifted to their local centroid position. Evaluation in virtual, phantom, and experimental images, including intraoperative data from patient experiments, shows that, with appropriate camera positions, the error estimates (root-mean square error and mean error) are $\sim 1\mathrm{mm}$.

1. Introduction

Reconstruction of anatomical structures is a critical component of surgical navigation systems. These structures can be used to guide the surgeon to the correct locations of interest. In particular, blood vessels can be used as landmarks.¹

One important situation that requires proper reconstruction of anatomical structures in neurosurgery is compensation for brain shift, i.e., the deformation of the exposed brain after the craniotomy opening. The brain can bulge or sink at the craniotomy due to a number of reasons, such as gravity, loss of cerebrospinal fluid, tumor and peri-tumoral edema pressing the brain, hyper- or hypoventilation, and anesthetic drugs. Due to brain shift, a preoperative magnetic resonance (MR) image no longer corresponds to the current anatomy of the patient. Ideally, for image-guided surgery, the MR image needs to be virtually deformed to the current state, and that requires information of how the anatomy is changing. The use of the MR image without compensation of brain shift is a limitation of current neuronavigation systems.

In this paper, we present a technique to reconstruct blood vessels, more precisely their centerlines. Our final goal is to use this information to compensate for brain shift. Here, we focus on the initial stages of such a framework, in particular on the computer vision issues related to the accurate reconstruction of the centerlines. To evaluate the methods, we use virtual and phantom brain images as well as experimental images of a patient brain.

The system is intended for use only in the initial stages of surgery, before any resection has been performed. The larger brain shift (before any brain resection) occurs after the opening of the dura mater; thus, it is important to reconstruct the vessel centerlines accurately at this time point. Once the brain is penetrated, complex deformations occur that require complex models and tracking of surgical tools. Ultimately, the purpose is to use this system information in combination with neuronavigation systems to present the initial volume (preoperative MRI) with correction of large nonrigid deformations.

Although registration is not the focus of this work, the overall goal drives the choices in methodology, e.g., the same structures (vessel centerlines) are reconstructed/registered, and the accuracy of the centerline reconstruction is directly related to the accuracy of the nonrigid registration. If the reconstruction accuracy is poor, it might even cause the registration to fail. The structures chosen (vessel centerlines—curves) are also related to registration. In comparison to surface registration, a curve registration has less degrees of freedom (DOF); thus, a more constrained and accurate registration can be obtained. For these reasons, a new method for vessel centerline reconstruction is proposed and evaluated.

1.1. Related Work

The problem of brain shift tracking has been approached using different landmarks and computer vision systems. D’Apuzzo and Verius² used a three-dimensional (3-D) monitoring system (three cameras) to track points in a brain phantom, with reference points painted on the surface. Sun et al.,³ Paul et al.,⁴ DeLorenzo et al.,⁵ Hsieh et al.,⁶ Vivanti et al.,⁷ and Kumar et al.⁸ (two cameras) also tracked points on the brain surface, but in this case automatically extracted.

As an alternative to the previous methods, which all used multiview computer vision systems (two or three cameras), the work of Sinha et al.⁹ utilized laser range scanners to obtain textured point-clouds describing the surface geometry/intensity pattern of the brain surface. Roberts et al.¹⁰ also used lasers; a pair of laser beams was placed within the optics of an operating microscope, which can be used to track individual points on the surface of the brain to analyze cortical displacement. Finally, it is also possible to use intraoperative MR imaging^11,12 or intraoperative computed tomography (CT)¹³ to track the current form of the brain.

In this work, we focus on the reconstruction of superficial vessels of the brain and the overall system description. Using the vessels for correction of brain shift has already been proposed using ultrasound.^14–17 Ding et al.^18,19 proposed the use of vessels in intraoperative microscope video sequences for cortical displacement estimation. Vessel segments are identified by the user in the first frame of a video sequence. Then, these are tested for similarity in the subsequent frames to track the vessels. Ding et al.^20,21 also made use of vessels extracted semiautomatically and automatically for registration of pre- and postbrain tumor resection using laser range scanner data.

Recently, Ji et al.²² used intraoperative microscope sequences and stereovision for cortical displacement estimation between two intraoperative images. The goal is similar to the work of Ding et al.,¹⁹ but in this case, optical flow motion tracking was used. Finally, Berkels et al.²³ used sulci information to coregister intraoperative brain surface photographs and preoperative MR images.

In a completely different clinical context, similar vessel reconstruction problems also arise when vessels, e.g., the coronary arteries, are reconstructed from multiple angiographic views.^24–26

Our first attempt to track the vessel centerlines used normalized cross correlation as presented in Ref. 27, the results of which were used as input data to compensate for brain shift. Due to the limitations of this method, a geometrical rather than an intensity-based approach was used here to increase the accuracy of the system.

Our reconstruction methodology is inspired in this work by Li et al.²⁸ Spatial coherence is explored to exclude incorrect centerline points. The main difference is that Li et al. used an increasing number of cameras to enforce the coherence and exclude incorrect points. In our case, three cameras are used, and therefore an extraprocessing step was introduced to improve the results. It is based on the knowledge that the vessels are on a surface (cortical brain surface). Further details can be found in Sec. 2.

The data obtained with our method are similar to the aforementioned vessel reconstruction methods (3-D points). The main differences are the underlying methods and the purpose. The compensation for brain shift using this data is presented in our work,²⁹ where we focus on the final stages of the overall brain shift pipeline using superficial blood vessels.

2. Materials and Methods

The blocks of our computer vision pipeline for reconstruction of tube-like structures are

• •image acquisition and undistortion (three images),
• •centerline manual selection, and
• •correspondence matching and triangulation (reconstruction).

Before using the pipeline, we need to calibrate the three cameras. For this task, we developed software that makes use of OpenCV.³⁰ OpenCV uses the pinhole camera model, and with a calibration pattern, the intrinsic and extrinsic camera parameters can be obtained. The intrinsic parameters are internal parameters of each camera, and the extrinsic parameters give us the spatial relations (rotation and translation) between pairs of cameras. Details on the camera calibration and 3-D reconstruction are given in the OpenCV documentation,³¹ which provides a clear description of all the matrices involved.

From these parameters, one can derive the fundamental matrix for each pair of cameras. The fundamental matrix is used to establish relationships between the images of the different cameras, or more precisely: if we select one point in one image, we know that it has to correspond to a point that is on a line (epipolar line) in the image of an associated camera. This is referred to as epipolar constraints. Once these calibration parameters have been obtained, we can proceed and use the pipeline described below.

2.1. Image Acquisition and Undistortion

Three images were acquired with mono-complementary metal-oxide semiconductor cameras that can capture the visible and near-infrared (NIR) spectrum (Point Grey, FireFly, Point Grey Research Inc., Richmond, Canada) at each time point. Figure 1 shows the camera arrangement used and the calibration pattern. The calibration process is manual; it consists of placing the calibration object at different locations (visible by the three cameras). The calibration checkerboard vertices points are extracted on the three images as shown in Fig. 1 and used as input for the calibration algorithm. Usually, 30 acquisitions are performed for the calibration, taking approximately 2 to 5 min. This process can be performed prior to surgery.

Fig. 1.

(a) Camera arrangement and (b) calibration pattern.

A triangular camera configuration was used in all the experiments (Fig. 1). For the patient setup, the distance of the object of interest to the camera was $\sim 30\mathrm{cm}$, whereas for the remaining data sets, several distances were tested, and the resolution of the images was $640\times 480\text{pixels}$. The camera programming interface was used to acquire the images of the several cameras at a synchronized rate. These were then saved for further offline processing.

Due to the properties of the cameras, the images need to be undistorted. The radial and tangential coefficients can be obtained in the calibration stage using OpenCV, which also has routines to use these parameters to correctly undistort the images.

2.2. Centerline Manual Selection

With the undistorted images, we can proceed to select the centerlines. The vessels are clearly visible in the images and easy to select. Each vessel centerline was individually selected using an image processing tool (ImageJ³²) with the segmented line tool and the selection points fitted to a spline. This process was performed to the three images, taking $\sim 1\min /\text{image}$; the results can be seen in Fig. 2.

Fig. 2.

(a) Views of the patient brain, (b) brain phantom, and (c) virtual segmented brain. The vessel centerlines manually selected for evaluation are highlighted (yellow).

In addition to the position information, the image centerlines will also contain directional information (lines perpendicular to the vessel direction), which will be exploited by the next pipeline block. To obtain the directions, the vesselness approach proposed in Ref. 33 for the two-dimensional (2-D) case was used. Just like the vesselness concept proposed in Ref. 34, it uses the Hessian matrix as the core of the method. In this case, only one scale was used (as the vessel centerlines are 1 pixel wide). The eigenvalues of the Hessian matrix provide an estimate of the vesselness; for the 2-D case, we look for the eigenvalue of largest absolute value. If this eigenvalue is negative, we are in a tube-like or blob-like structure for images that have bright vessels. The larger absolute eigenvalue is associated with a line passing through the pixel being tested, perpendicular to the vessel direction.

2.3. Correspondence Matching and Triangulation

One of the most difficult challenges in a computer vision system is correspondence matching, i.e., how one feature point in an image corresponds to a point in a different image.

2.3.1. Geometrical constraints

For multiview systems, a well-known method is epipolar geometry or epipolar constraints.³⁵ As mentioned already, the fundamental matrix is used to establish relationships between the images of the different cameras. That is, if we select one point in one image, we know that it has to correspond to a point that is on a line (epipolar line) in the image of an associated camera.

The initial possible correspondences are calculated individually per view, i.e., if camera 1 is set as the reference, then the corresponding points of camera 2 and 3 must be near the epipolar lines, and the same procedure is performed taking the remaining cameras as reference. A point is considered a possible correspondence point if the distance from the point to the epipolar line is less than $10\times \sqrt{2}\text{pixel}$. This initial process will provide lists of possible pairs of correspondences, in our case, six lists, two for each reference view. The main reason to use three cameras and the triangular arrangement is related to the epipolar geometry and the features we are using. If only two cameras are used, it is possible that a vessel coincides with an epipolar line, and this creates high ambiguity. To alleviate this problem, a camera arrangement where at least one epipolar line is not coincident with the vessel centerline must be used. This can be obtained with a triangular camera arrangement as shown in Fig. 1.

Considering view 1 as reference, the lists associated to it are $l_{\mathrm{1,2}}(p_1,p_2)$ and $l_{\mathrm{1,3}}(p_1,p_3)$. To produce a triplet list, with all possible correspondences $l_{\mathrm{1,2},3}(p_1,p_2,p_3)$, the spatial coherence between views is tested. Each pair $l_{\mathrm{1,2}}$ and $l_{\mathrm{1,3}}$ is triangulated; if the 3-D point generated is $<200\mathrm{mm}$ or $>700\mathrm{mm}$, the pair can be discarded. Because the cameras are placed $\sim 300\mathrm{mm}$ from the head, and the maximum diameter of the head is $\sim 250\mathrm{mm}$, points $<200\mathrm{mm}$ or $>700\mathrm{mm}$ should not occur. The triangulation was performed using the midpoint method. For all valid pairs, if a 3-D point of $l_{\mathrm{1,2}}(p_1,p_2)$ is at a distance $<3\mathrm{mm}$ of a 3-D point of $l_{\mathrm{1,3}}(p_1,p_3)$, then the corresponding triplet is added to $l_{\mathrm{1,2},3}(p_1,p_2,p_3)$. The same procedure is performed for the remaining reference views.

Using the triplet correspondences, we can further test epipolar constraints. For three points, one per view, six corresponding epipolar lines exist. A correspondence point has two epipolar lines associated to it. For the triplet to be considered a possible correspondence, all points must have a distance to the epipolar lines less than $10\times \sqrt{2}\text{pixel}$.

Spatial coherence can be further explored using the remaining triplets. This fact was also explored by Li et al.,²⁸ which served as inspiration for our work. Two types of spatial coherence are used, the first relies only on the triplet points and the second uses the tangents of the triplet points.

Let us now consider the case when the 3-D point of $l_{\mathrm{1,2},3}(p_1,p_2,p_3)$ is to be calculated. To do so, three triangulations, $P_{\mathrm{1,2}}=\text{triang}(p_1,p_2)$, $P_{\mathrm{1,3}}=\text{triang}(p_1,p_3)$, and $P_{\mathrm{2,3}}=\text{triang}(p_2,p_3)$, are required. The final result is given by the centroid of these points, i.e., $P_{\mathrm{1,2},3}=(P_{\mathrm{1,2}}+P_{\mathrm{1,3}}+P_{\mathrm{2,3}})/3$. If the distance of the centroid point to any of the intermediate 3-D point is larger than 3 mm, then the triplet is discarded. Please note that in order to evaluate the spatial coherence, at least three cameras are required. If two cameras are used, only one 3-D point is obtained, and thus no spatial coherence measure can be performed. This type of spatial coherence can be calculated by

$$\mathrm{sC}=\frac{\parallel P_{\mathrm{1,2},3}-P_{\mathrm{1,2}}\parallel +\parallel P_{\mathrm{1,2},3}-P_{\mathrm{1,3}}\parallel +\parallel P_{\mathrm{1,2},3}-P_{\mathrm{2,3}}\parallel }{3\times {\max }_{\mathrm{ED}}},$$ (1)

where the term ${\max }_{\mathrm{ED}}$ (maximum estimated error distance) was set to 3 mm, in order to normalize the values to the interval [0, 1].

The second type of spatial coherence makes use of the 2-D centerline tangents computed in the previous section. As mentioned previously, the larger absolute eigenvalue is associated with a line passing through the pixel being tested and perpendicular to the vessel direction. To obtain the tangent direction, a simple 2-D rotation of 90 deg is performed. The reason why the tangent direction is used and not the perpendicular direction can be explained geometrically. Figure 3 showed the geometry of a tube projection onto the three view planes. In 3-D, the tube has three eigenvectors (${\lambda }_1$, ${\lambda }_2$, and ${\lambda }_3$), in which the eigenvector with the smallest magnitude is ${\lambda }_1$³⁴ and its projections on the image planes correspond to the 2-D tangents. The eigenvector ${\lambda }_1$ can be estimated by calculating the intersection lines of the planes that pass by the cameras’ focal points and the tangent lines. Three estimates of ${\lambda }_1$ are possible to obtain using the three plane combinations. The angles formed by the three vectors are then tested for coherence. A problem may occur if all planes are coplanar; in this case, the planes intersections do not produce lines. With the triangular camera configuration used in this work, this is in general not a problem. Still a test is performed: if two planes have a plane normal with an angle $<10\mathrm{deg}$, then the resulting line cannot be used. At least two lines must exist, or else the triplet is discarded. The minimum angle formed by the possible line pairs is used as the coherence measure, and if the minimum angle is $>45\mathrm{deg}$ the triplet is discarded. This initial stage was performed using parallel processing in the CPU, using eight threads, two threads for each double-threaded core of an Intel Core i7, 3.8 GHz, Quad-Core.

Fig. 3.

Geometry of a tube projection onto three view planes. The projection of the three-dimensional smallest magnitude eigenvector ${\lambda }_1$ corresponds to the two-dimensional centerline tangents.

Unfortunately, even applying all these constraints, a high degree of ambiguity in the correspondence selection exists. For each pixel, we have limited the number of possible correspondences to 100 points with the best point spatial coherence, due to the observation that the correct match is usually found within the first most coherent points.

2.3.2. Relaxation labeling

To further reduce ambiguity, relaxation labeling was used. Here, we follow a similar methodology as in Barnard and Thompson.³⁶ In their case, stereo correspondence matching was performed using as initial estimates the disparity. Also, Li et al.²⁸ used relaxation labeling in their work with initial estimates of spatial coherence (point and tangent). In this work, only spatial coherence of points was used for the relaxation.

Considering view 1 as the reference, the method works as follows. For each pixel of view 1, a list $L_i$ of possible matches exists (labels —$l$), with higher ambiguity if a larger number of labels are present. To each label, a probability ($p_i(l)$) of the point being the correct match is computed. Furthermore, as the first element of the list is placed as an element (referred to as $l^{*}$) that represents the probability that a point in view 1 has no match. Thus, the sum of the labels in the $L_i$ lists is equal to 1, and the labels have values in the interval [0,1]. To start the process, initial probabilities are calculated for the $L_i$ labels ($p_i^0(l)$), and to do so a weight associated to the label is first computed. Let

$$w_i(l)=1-\mathrm{sC},l\ne l^{*},$$ (2)

where $\mathrm{sC}$ is the spatial coherence. $w_i(l)$ cannot be directly used as the initial probability because $w_i(l^{*})$ is undefined and $\sum w_i(l)\ne 1$. Still, $w_i(l)$ has quantitative values that will be associated with the probabilities. As in Ref. 36, the probability of $l^{*}$ is first calculated

$$p_i^0(l^{*})=1-\underset{l\ne l^{*}}{\max }[w_i(l)].$$ (3)

Then, Bayes’ rule can be applied to obtain the remaining initial probabilities

$$p_i^0(l)=p_i(l|i)\times [1-p_i^0(l^{*})],l\ne l^{*},$$ (4)

where $p_i(l|i)$ is the conditional probability that a particular label associated to pixel $i$ is matchable, and $[1-p_i^0(l^{*})]$ is the probability that pixel $i$ is matchable. The conditional probability can be estimated with

$$p_i(l|i)=\frac{w_i(l)}{\sum _{l^{\prime }\ne l^{*}}{w}_i(l^{\prime })}.$$ (5)

Once all the labels are assigned an initial probability, the method can iteratively proceed to calculate better estimates of the probabilities (relaxation). The estimates can be improved using the coherence property. In our case, the coherence is calculated using 3-D points as opposed to Ref. 36 which used disparity. Since each label is associated to a 3-D point ($P_i(l)$), we use the points in the local neighborhood to update the probability. If the label is consistent, it should have neighbors that also have a high probability. Furthermore, since we are tracking blood vessels, we can limit the neighborhood to a sphere with the maximum vessel size. For all cases, $\mathrm{\Theta }=2.5\mathrm{mm}$, approximately equal to the mean maximum diameter of the human vessels considered.³⁷ Furthermore, for each view only the neighbors that do not share the reference image pixel are considered. Let

$$q_i^k(l)=\underset{\parallel P_i(l)-P_j(l^{\prime })\parallel \le \mathrm{\Theta }}{\sum _{j\ne i}}{p}_j^k(l^{\prime }),l\ne l^{*},l^{\prime }\ne l^{*}.$$ (6)

In all cases, $q_i^k(l)\ge 0$. This quantity will be larger if many neighbors have a high probability, and is used to compute the probabilities of the next iterations

$${\widehat{p}}_i^{k+1}(l)=p_i^k(l)\times [A+B\times q_i^k(l)],l\ne l^{*}$$ (7)

and

$${\widehat{p}}_i^{k+1}(l^{*})=p_i^k(l^{*}).$$ (8)

The $\widehat{p}$’s must be normalized

$$p_i^{k+1}(l)=\frac{{\widehat{p}}_i^{k+1}(l)}{\sum _{l^{\prime }{inL}_i}{\widehat{p}}_i^{k+1}(l^{\prime })}.$$ (9)

For the parameters $A$ and $B$, the same values as in Ref. 36 were used ($A=0.3$ and $B=3$). If the new probability value is $\le 0.00001$, its value is set to 0. The relaxation process is terminated after 10 iterations and it is performed individually per view. Finally in each view, the 3-D point associated to each $L_i$ (except $l^{*}$) with the higher probability is chosen, unless all labels in $L_i$ have probability 0. This will result in three sets of 3-D points. It may occur that even after the relaxation some correspondences will be incorrect, but the probability of the correct match exist in at least one of the three sets is rather high. These points are then all added into one point set for the last stages. The relaxation was implemented using the graphics hardware and NVIDIA Compute Unified Device Architecture (CUDA) with each thread computing one pixel, i.e., updating one list $L_i$.

2.3.3. Thin-plate spline filtering

The next stage of our method uses a surface approximation approach. Since the vessels that we are tracking are positioned on the brain surface, a surface exists that passes by the vessel points. To approximate this surface, we use the thin-plate spline (TPS), but due to the incorrect matches it cannot be used directly. An iterative approach was developed to deal with the outliers. In our case, the depth map of the surface is calculated, similar to the approach of Carr et al.³⁸ The result of the function is a depth value $z$

$$z=f(x,y)=a_1+a_{2}x+a_{3}y+\sum _{i=1}^n{w}_i\phi [|P_i^{2-\mathrm{D}}-(x,y)|],$$ (10)

where $a_1$, $a_2$, and $a_3$ are the coefficients of an affine transformation, $n$ is the number of 2-D points, $w_i$ are the weights, $P_i^{2-\mathrm{D}}=(x_i,y_i)$ are the 2-D point corresponding to the first two coordinates of a 3-D point $P_i=(x_i,y_i,z_i)$ obtained by triangulation, and $\phi$ is the radial basis function (in our case, the TPS).

The TPS function for the depth map is given by the following equation:

$$\begin{array}{ll}\phi (r)=r^2\log (r),& r>0,\\ \phi (r)=0,& \text{otherwise}.\end{array}$$ (11)

To obtain $a_1$, $a_2$, $a_3$ and $w_i$, it is necessary to solve a linear system. We used a C++ linear algebra library named Armadillo³⁹ in CPU to find the solution. The system has the form

$$\left[\begin{array}{cc}K& Q\\ Q^{\mathrm{T}}& 0\end{array}\right]\left[\begin{array}{c}w\\ a\end{array}\right]=\left[\begin{array}{c}v\\ 0\end{array}\right],$$ (12)

where

$$K=(k_{ij})=[\phi (r_{ij})],r_{ij}=\parallel P_i^{2-\mathrm{D}}-P_j^{2-\mathrm{D}}\parallel ,$$ (13)

$$Q=\left[\begin{array}{ccc}1& x_1& y_1\\ 1& x_2& y_2\\ ⋮& ⋮& ⋮\\ 1& x_n& y_n\end{array}\right],$$ (14)

$$w={(w_1,w_2,\dots ,w_n)}^{\mathrm{T}},$$ (15)

$$a={(a_1,a_2,a_3)}^{\mathrm{T}}.$$ (16)

The vector $v$ will contain the values of the $z$ coordinate of the 3-D points used in the system. In this way, it is possible to solve the system.

If the 3-D points obtained in the relaxation labeling stage were to be used directly, the result would be an overfit surface, with many oscillations, due to location errors of the points. Instead, it is ideal to allow some degree of relaxation. This can be accomplished by introducing the so-called relaxation parameter $\lambda$^40–42 in the linear system, by replacing $K$ by $K+\lambda I$, where $I$ is the $n\times n$ identity matrix. The lower the value, the closer the surface passes to the 3-D points (overfit), whereas with high values the surface will have less oscillations (underfit). This parameter is a key to our iterative method.

The parameter $\lambda$ is depended on the scale of the linear system. In order to make the parameter scale invariant,⁴³ $\lambda$ is replaced by ${\alpha }^2{\lambda }^{\prime }$, where

$$\alpha =\frac{1}{n^2}\sum _{i=1}^n\sum _{j=1}^n{r}_{ij}.$$ (17)

The method starts by considering all 3-D points in the linear system and with a high-scaled relaxation parameter (in all cases ${\lambda }^{\prime }=1$). Then, the values obtained by the TPS are compared to the corresponding 3-D point, i.e., if $P_i=(x_i,y_i,z_i)$ then $z_i^{\mathrm{TPS}}=f(x_i,y_i)$ and the comparison: $|z_i-z_i^{\mathrm{TPS}}|\le \mathrm{\Theta }$ (in all cases $\mathrm{\Theta }=2.5\mathrm{mm}$). The 3-D points that satisfy this condition are used in the linear system of the next iteration, where ${\lambda }^{\prime }$ is reduced. Once the new linear system is computed, the TPS function is again evaluated against the original 3-D points, and the points near the surface used in the next iteration. This is a top-down approach that includes more points the higher the number of iterations (tighter the fit). Ten iterations were used in all cases, the maximum ${\lambda }^{\prime }=1$ and the minimum ${\lambda }^{\prime }=0.1$, with equally spaced values in between (0.1). With the resulting points (after 10 iterations and ${\lambda }^{\prime }=0.1$), we compute a final TPS to test if the points are still close to the final TPS surface; if this is not the case, they are discarded. The remaining points are updated to the value obtained by the TPS, i.e., if $P_i=(x_i,y_i,z_i)$ and $z_i^{\mathrm{TPS}}=f(x_i,y_i)$ then $P_i^{\mathrm{new}}=(x_i,y_i,z_i^{\mathrm{TPS}})$.

2.3.4. Mean shift

Finally, the points are shifted to centroid positions, computed using the points in a local neighborhood, similar to the mean shift algorithm,⁴⁴ but in our case only one iteration is performed, with a neighborhood radius of 1.5 mm. In the worst case, the computation time of the previous methods was 27 s.

2.4. Vessel Reconstruction Evaluation

Figure 4 shows the diagram of the vessel centerline reconstruction evaluation. Three different depth ranges ($Z$ coordinate) were tested in order to evaluate the influence of depth in the system error. Furthermore, the impact of the several blocks of the processing pipeline was also evaluated. Three different cases were studied:

Fig. 4.

Diagram of the vessel centerline reconstruction evaluation.

• •RL: uses relaxation labeling without TPS filtering and mean shift;
• •RL + TPS: uses relaxation labeling and TPS filtering without mean shift; and
• •RL + TPS + MS: uses relaxation labeling, TPS filtering, and mean shift.

The root-mean square error (RMSE) and mean error⁴⁵ were computed for each test. The median and the 95th percentile were also calculated to express the distribution of the errors. The metrics use the distance from centerline points of our method to the closest MRI/CT centerlines point (aligned by a rigid transformation, if applicable).

2.5. Virtual Experiment

In order to evaluate the errors, we used virtual views of an MR angiography dataset of the human head. The brain was segmented from the dataset using a model-based level set method as described in Ref. 46. Three views were generated using a ray-casting direct volume rendering algorithm. An appropriate transfer function was selected manually in order to emphasize the vessels on the brain surface (similar to the approach found in Ref. 47). Superficial brain veins were chosen for this purpose, since they are easily identified in the MRI and eventually at the brain surface on craniotomy. Care has to be taken to avoid using superficial veins at their entrance in the superior sagittal sinus (so-called bridge veins), since they are extracerebral and their deformation may not reflect the brain shift properly. Figure 2(c) shows an image generated for one view. The virtual camera positions were obtained using a real camera setup appropriate to view the surgical region of interest. The only parameters not considered are the radial and tangential distortions.

3-D MRI vessel centerlines were obtained using the iterative vessel segmentation and centerline tracking algorithm presented in Ref. 48. These are used for comparison with 3-D vessel centerlines obtained with our reconstruction method, as shown in Fig. 4.

2.6. Phantom Experiment

To further evaluate the system errors, a brain phantom was used. The phantom was constructed using two-component silicone⁴⁹ and a brain gelatin mold. Both products are available online from several vendors. By filling the mold with the silicone, it is possible to construct the phantom. The mold is not an accurate anatomical representation of the human brain, but its shape contains structures that are similar to the sulci and gyri of the brain. For the vessels, we used small black tubes with 3.0-mm diameter. The tubes were attached to the silicon with small clips [Fig. 2(b)].

The phantom was scanned with a CT producing a volume with voxel spacing: $0.32\times 0.32\times 0.25\mathrm{mm}$. The CT 3-D vessels centerlines were obtained similarly to the virtual experiment. In this case, the rigid registration was not known. To align the CT centerlines to the centerlines obtained by the system, fiducial points were selected at locations that are possible to identify in the camera images and in the CT. The CT fiducial locations are obtained by first generating a triangular surface using the CT data and then this surface was loaded to MeshLab⁵⁰ where it is possible to pick points. To obtain the homologous points in the camera’s coordinate system, on each image, the correspondent pixel positions were selected and triangulated. With these two sets of points, the rigid transformation was computed using the MATLAB implementation “estimateRigidTransform” of the method for rigid transformation estimation presented in Ref. 51. The rigid transformation is then used to align the CT centerlines and the camera centerlines. This registration process introduces an error that can be quantified by the fiducial registration error (FRE), which is the RMSE of the registered fiducial points.⁵² Similarly, the target registration error, which is the distance between corresponding points other than the fiducial points after registration, is typically expressed by its RMSE value. For this reason, we only mention the RMSE in the evaluation. The error evaluation is similar to the virtual experiment.

2.7. Patient Experiment

The experimental procedures were performed after approval by the regional research ethics committee in Linköping, Sweden (decision No. 2015/71-31, dated March 25, 2015). The experiment was performed in one patient.

The image surgical procedure was performed after craniotomy and dura mater opening, and data were recorded with the exposed brain. The camera acquisition was synchronized, and the postprocessing performed, using the images from the different cameras with the same timeframe. Figure 2(a) shows one view with manually selected centerlines. The evaluation methodology used for the previous cases is inapplicable here; due to the brain shift, it is impossible to use the MRI vessel centerlines for comparison. Hence, only a visual qualitative inspection was performed.

3. Results

The results of the centerlines using the virtual data are shown in Fig. 5, where the MRI centerlines (green) and the centerline points using the virtual cameras (white spheres) for the depth 2 are visible. The geometrical errors for the three depth ranges tested are shown in Fig. 6. The depth ranges are depth 1 [321 mm, 332 mm], depth 2 [371 mm, 382 mm], and depth 3 [420 mm, 432 mm]. The range represents the minimum and maximum $Z$ coordinate value of the 3-D ground truth points. The values of the different measures were computed in MATLAB.

Fig. 5.

Virtual centerlines for the depth 2 [371 mm, 382 mm]. The MR centerlines are presented in green, and the centerline points obtained using the virtual cameras are presented as white spheres.

Fig. 6.

RMSE, mean error, median error, and 95 percentile calculated using the virtual data for three depth ranges and several blocks of the reconstruction pipeline.

The results of the centerlines using the phantom data are shown in Fig. 7, where the CT centerlines (green) and the centerline points of the phantom obtained using our method (white spheres) for depth 2 [371 mm, 382 mm] are visible. The geometrical errors for the three depth ranges tested are shown in Fig. 8. The depth ranges are depth 1 [334 mm, 366 mm], depth 2 [347 mm, 376 mm], and depth 3 [410 mm, 436 mm]. The FRE of the phantom fiducial points used to calculate the rigid registration is shown in Table 1, for the three depths.

Fig. 7.

Phantom centerlines for the depth 2 [347 mm, 376 mm]. The CT centerlines are presented in green, and the centerline points obtained using the cameras presented as white spheres.

Fig. 8.

RMSE, mean error, median error, and 95 percentile calculated using the phantom data for three depth ranges and several blocks of the reconstruction pipeline.

Table 1.

Fiducial registration error (FRE) of the phantom fiducial points used to calculate the rigid registration for the three depths.

	Depth 1 [334 mm, 366 mm]	Depth 2 [347 mm, 376 mm]	Depth 3 [410 mm, 436 mm]
FRE	0.49 mm	0.53 mm	0.49 mm

It is clear that the results for the phantom data are inferior compared to the virtual, due to camera image noise and distortions not considered with the virtual data. Furthermore, the rigid registration introduces an error of $\sim 0.5\mathrm{mm}$ as measured by the FRE. In both the experiments and all depths considered, the accuracy increased when using more pipeline blocks. When using all the pipeline blocks (RL + TPS + MS), the results were similar for the depths considered, and in all cases, both the RMSE and mean error were $\sim 1\mathrm{mm}$.

The results of the patient experiment are shown in Fig. 9, where the centerline points are presented as white spheres. As mentioned before, just a qualitative visual inspection is possible. The complete selected centerlines were possible to reconstruct and the shape seems to correspond to the shape of the patient’s vessels.

Fig. 9.

Patient vessel centerlines.

4. Discussion and Conclusion

The multiview vessel reconstruction method presented in this work is able to obtain 3-D point vessel centerlines accurately ($\mathrm{RMSE}\sim 1\mathrm{mm}$ as can be seen in Figs. 5 and 7). The results show that our method is feasible for virtual as well as real images, including intraoperative data. For appropriate camera positions (depth ranges), the error estimates are $\sim 1\mathrm{mm}$, which is reasonable for the surgical applications we are considering, i.e., brain shift compensation for brain tumor removal.

The virtual and phantom experiments exemplify the impact of the rigid registration error. In the virtual case, the registration is perfect reflecting the true reconstruction errors. All the depth ranges tested in this study produced similar errors, but it is expected that larger distances may produce worse results, since depth and proximity to the edges of the field of view have been reported as the major sources of error.⁵³ If the object of interest is located $<400\mathrm{mm}$ from the cameras, and the rigid transformation is performed, the RMSE and mean error are $\sim 1.4$ and 1.3 mm, respectively. This value is slightly better than for approaches using ultrasound, which have a clinical accuracy of 2 mm.⁵⁴ Sun et al.³ used a brain-shaped phantom from agar gel with copper wires (representing cortical vessels) embedded into the gel. This phantom was CT scanned, and the surface was compared with stereovision results. The mean distance error obtained was 1.16 mm, a value similar to the one obtained by our method. It should be noted that to align the stereovision reconstructed surface with the CT scanned, the closest iterative closest point was used.⁵⁵ Effectively this approach minimizes the rigid registration error. Also, when obtaining 3-D vessel positions from laser range scanners,²⁰ typical errors are close to 1 mm. A common problem seems to be that the stereovision and laser range methods share the fact that the vessel points obtained are not centerline points, but rather points on the outer surface of the vessel. If the vessel is small enough ($\le 1\mathrm{mm}$), then this error can be neglected.

It should be noted that the centerlines obtained using the MRI and CT data (used as ground truth) might themselves have small errors (not exactly at the vessel centerline). The reconstructed centerlines can be used for matching with centerlines of vessels extracted from preoperative imaging. Once the correspondences are known, a nonrigid transformation can be applied to the preoperative image to compensate for the brain shift.²⁹

The TPS process has some similarity to the work of Vivanti et al.,⁷ who used the random sample consensus⁵⁶ and a parabolic model fitting to obtain points on the brain surface. The parabolic model has much fewer DOF than the TPS, and for complex curves a low degree model may not be the most adequate to fit the data. In fact, splines have been used before for stereo matching, as in the work of Chen and Boult,⁵⁷ but in this case the method relies on the fact that initially a few matching points are known and in each iteration more points are added. Such an assumption is not made in our approach.

From the experiments using the phantom and patient brain, we have observed that the images can be sensitive to light conditions. The main problem regarding illumination is specular highlights, which can be observed in Fig. 2, since the surface of the brain is highly reflective. This is a known problem that has been addressed by several authors.^58–60 Controlled diffuse illumination seems to yield superior results. Polarization filters were used for the phantom and patient data to try to reduce the specular highlights, but still a high number of highlights occur, since silicone and the wet brain surface are highly reflective.

We also realized that a fully automatic pipeline was not feasible due to the high number of tube-like structures in the images. Manual selection of vessels of interest, which proved very effective in our case, has to be further tested in a clinical setting to assess its viability. A benefit of the manual segmentation is the fact that a human can easily interpolate small regions of specular highlights, thus reducing this problem.

For future clinical applications, it would certainly be desirable to replace the current manual selection of vessels with a fully automated procedure. Unfortunately, for the type of images we are using, this is a difficult task, although efforts have been made by Ding et al.²¹ to produce an automatic method for vessel centerline extraction. Furthermore, several methods for detection of vessel centerlines exist for the eye (fundus images).⁶¹ Those images are rather different and do not suffer from specular highlights. One possibility to produce images similar to fundus images might be to use indocyanine green⁶² fluorescence and NIR cameras. This approach should be tested in future studies.

One may notice that some vessels can be seen even below the surface of the brain. However, the penetration is limited, and it is more difficult to reconstruct the blood vessels correctly at increasing depths. This is explained by refraction, as the light rays change direction at the interface between brain and air, making the camera settings unviable. For these reasons, these vessels are avoided.

To further improve the results, one could use cameras with higher resolution than the $640\times 480\text{pixels}$. If cameras with higher resolution were to be used, we would expect better results, but the computation time would be considerably higher.

The evaluation results were only performed with the virtual and phantom setup, since the patient data are deformed by the brain shift; thus, it is not possible to compare the reconstructed vessels directly to the MRI ground truth. The patient experiment was mainly used to demonstrate the viability of the method, using real camera data (surface vessels of the patient brain).

One of the main constraints used in our work is spatial coherence (points and tangents). From a conceptual point of view, this provides a geometrical measure of similarity between views. The main difference between our approach and stereovision approaches³ is the fact that we use a geometrical similarity measure instead of an intensity correlation similarity measure.

For the relaxation labeling stage, we have only used point spatial coherence to compute the initial probabilities. Since the tangent calculation is more sensitive to noise and image resolution, the tangents were only used before relaxation with a rather high threshold (45 deg). As the results show, high accuracy is still achieved without considering the tangents in the relaxation.

Some of the parameter values in the pipeline were selected empirically, in particular, the sizes of the vessels. In a realistic clinical scenario, we do not expect that the size of the brain vessels will change considerably between subjects. If the cameras are positioned at approximately the same distance for each subject, similar values for the parameters of the pipeline are thus expected.

The patient and phantom images used were monochromatic, but since in our case the vessel selection is performed manually this is not a relevant factor. Provided that a clear visualization of the vessels is available, the reconstruction can be performed with color or even fluorescence images.

The main reason to use methods that rely on geometry rather than texture is due to our previous experience using such intensity-based methods. These can be rather sensitive, for example, to the curvature of the object of interest because the similarity is based on images that are projections of the object. If the object is rather round it produces projections that are less similar, degrading the results of the method. Since the brain is rather round, it is likely that the image similarity is affected. Ultimately, each technique has its strengths and weaknesses, and potentially this problem might in the future be solved by some advanced machine learning technique.

In conclusion, we have presented a method for vessel tracking in multiview images using three cameras and methods that use the geometrical properties of the vessel structures. Our pipeline is able to obtain accurately 3-D point vessel centerlines. In future work, we plan to apply it to the detection and quantification of brain shift in neurosurgery in order to enable intraoperative augmented reality visualization of deformed (brain shift corrected) preoperative images.

Biographies

Filipe M. M. Marreirosis a PhD student at the Center for Medical Image Science and Visualization, Linköping University, Sweden. He received his BS degree in systems and computation engineering from University of Algarve in 2002, and his MS degree in computer graphics and virtual environments from University of Minho in 2006. He has worked as a researcher at the Fraunhofer IGD, Darmstadt, Germany, in 2003; Computer Graphics Center, Guimarães, Portugal, from 2003 to 2005; and Innsbruck Medical University, Austria, from 2008 to 2010. His current research interests include computer graphics, scientific visualization, computer vision, medical image analysis, and computer-assisted surgery.

Sandro Rossitti is certified as a specialist in neurological surgery and neuroradiology by the Swedish National Board of Health and Welfare. He is an associate professor of neurosurgery at Göteborg University. Currently, he is a consulting endovascular neurosurgeon at the Linköping University Hospital, Sweden.

Per M. Karlsson is a senior consultant neurosurgeon and assistant head of the Department of Neurosurgery, Linköping University Hospital, Sweden. He received university medical degree from Linköping University, Sweden, in 1988. He has been a specialist in neurological surgery in 1997.

Chunliang Wang is a researcher at the School of Technology and Health, KTH Royal Institute of Technology, Sweden. He received his medical training in Tianjin Medical University, China, from 1998 to 2005, and got his PhD degree in medical science from Linköping University, Sweden, in 2011. He also worked as a software engineer at Sectra AB, Linköping, Sweden, during 2011 to 2015. His current research interests are medical image analysis, image segmentation, machine learning, and statistical shape modeling.

Örjan Smedby graduated from Medical School at Uppsala University in 1983 and defended his doctoral thesis there in 1992. From 1999 to 2014, he has served as a professor of medical radiology at the Linköping University, Faculty of Health Sciences, and since 2004, he has been a professor of medical image science at the Faculty of Engineering, LiU. He was one of the founders of the Center for Medical Image Science and Visualization. Since 2015, he has served as a professor in medical image processing and visualization at the KTH Royal Institute of Technology.

Biographies for the other authors are not available.