4D interventional device reconstruction from biplane fluoroscopy

Abstract

Purpose:

Biplane angiography systems provide time resolved 2D fluoroscopic images from two different angles, which can be used for the positioning of interventional devices such as guidewires and catheters. The purpose of this work is to provide a novel algorithm framework, which allows the 3D reconstruction of these curvilinear devices from the 2D projection images for each time frame. This would allow creating virtual projection images from arbitrary view angles without changing the position of the gantries, as well as virtual endoscopic 3D renderings.

Methods:

The first frame of each time sequence is registered to and subtracted from the following frame using an elastic grid registration technique. The images are then preprocessed by a noise reduction algorithm using directional adaptive filter kernels and a ridgeness filter that emphasizes curvilinear structures. A threshold based segmentation of the device is then performed, followed by a flux driven topology preserving thinning algorithm to extract the segments of the device centerline. The exact device path is determined using Dijkstra’s algorithm to minimize the curvature and distance between adjacent segments as well as the difference to the device path of the previous frame. The 3D device centerline is then reconstructed using epipolar geometry.

Results:

The accuracy of the reconstruction was measured in a vascular head phantom as well as in a cadaver head and a canine study. The device reconstructions are compared to rotational 3D acquisitions. In the phantom experiments, an average device tip accuracy of 0.35 ± 0.09 mm, a Hausdorff distance of 0.65 ± 0.32 mm, and a mean device distance of 0.54 ± 0.33 mm were achieved. In the cadaver head and canine experiments, the device tip was reconstructed with an average accuracy of 0.26 ± 0.20 mm, a Hausdorff distance of 0.62 ± 0.08 mm, and a mean device distance of 0.41 ± 0.08 mm. Additionally, retrospective reconstruction results of real patient data are presented.

Conclusions:

The presented algorithm is a novel approach for the time resolved 3D reconstruction of interventional devices from biplane fluoroscopic images, thus allowing the creation of virtual projection images from arbitrary view angles as well as virtual endoscopic 3D renderings. Availability of this technique would enhance the ability to accurately position devices in minimally invasive endovascular procedures.

1. INTRODUCTION

Fluoroscopy is established as a valuable tool in today’s clinical practice for the guidance of minimally invasive procedures. Therefore, an angiography system acquires a series of x-ray images from a fixed position that shows the movement of the device over time. Biplane systems offer a second gantry, which allows acquiring two simultaneous fluoroscopic image sequences from different viewing angles. This imaging technique is used for the treatment of a wide range of vascular conditions including arteriovenous malformations, aneurysms and strokes.¹ An exact location of the object of interest, however, can be difficult especially in areas with a complex vascular structure, because of vascular overlap and the lack of depth information. The range of viewing angles that can be reached with both planes is limited by the system geometry, the table, and the patient so that the ideal viewing position cannot always be found. In some cases, even if the desired position can be reached, the detector would block the access to the patient, which requires a compromise between viewing position and patient accessibility.

Recently, efforts have been made to use biplane fluoroscopy acquisitions to reconstruct the 3D shape of the device.^2–7 This would allow creating virtual images from arbitrary viewing angles without moving the gantry or 3D renderings of the object within the vascular system, like virtual endoscopic views. The steps necessary to perform this reconstruction vary with the used algorithm but usually include the estimation of the projection matrices, motion compensation, segmentation of the device, search for corresponding points, and 3D reconstruction. Several algorithms have been published in order to solve parts of the described problem. The accuracy of the 3D point localization from two views with known point correspondence was investigated by Brost et al.⁸ They achieved localization errors of less than 1 mm for sphere-shaped objects with a diameter of 2 mm using a floor mounted C-arm CT system. Baert et al.⁹ proposed a method for tracking a guidewire in a 2D image sequence using third order B-spline fitting. This is done in two steps. First, a rough registration is applied to the result of the previous frame using a rigid transform and then an energy term is minimized considering the length and the curvature of the estimated spline as well as features extracted from the image using a Hessian filter and different preprocessing methods. Based on this algorithm, an extended segmentation and reconstruction method is provided in Ref. 3 where a statistical end point detection is performed additionally to the B-spline detection. The corresponding points for the 3D reconstruction are found using the epipolar constraint. If multiple potential corresponding points are found, the one closest to the previous point along the segmented device is used. A semiautomatic reconstruction method was published by Hoffmann et al.,^4,5 which requires the manual selection of a seed point along the device. The image is then binarized and the object pixels are combined to a graph where edges represent point sequences and nodes are the respective start or end points. The graph is used to create a 2D spline representation of the device. The point correspondences for the epipolar reconstruction are determined by finding a monotonic function that maps the points along the spline representation in the first image to the second image.

Attempts have also been made to reconstruct the 3D device shape using monoplane angiography systems. Since no depth information is available using only a single projection image, additional information has to be included to determine the 3D shape of the device. In Refs. ^10–12, a previously acquired 3D volume of the vascular structure is used to decrease the number of possible solutions. Brückner et al.¹⁰ use a probabilistic approach to find a B-spline representing the most likely device path within the vessel volume. In Ref. 11, a feature image is created from the 3D vessel volume, which is then used for a snake optimization to find the 3D path. Petković et al.¹² use the skeleton of the vessel tree to determine the topology of possible segments and then use a greedy algorithm to extract feasible 3D reconstructions. The downside of most monoplane reconstruction methods is that the exact position of the device within the vessel cannot be determined and shifting or transformation of vessels during the procedure might make the reconstruction impossible. A different approach for the 3D location tracking of the tip of an ablation catheter, which does not rely on a previously acquired 3D image, is proposed in Ref. 13. It calculates the depth of the device tip using the change of its size in the 2D image due to perspective projection. Model based reconstruction of ablation catheters from 2D projections has also been investigated.^14,15 The arrangement of the radio opaque electrodes in the tip of the catheter is used to create a dynamic 3D model, which can then be optimized to represent the 3D device reconstruction. For the reconstruction of more generic devices, such as guidewires, however, this method would not be suitable since it requires distinguishable parts like radio opaque electrodes, whose positions define the shape of the device. Finally, reconstruction methods based on iterative optimization of 3D B-snakes have been published.^6,7 In this case, no explicit device segmentation in the 2D planes is required. Instead, an energy functional is minimized representing the curvature of the device and the conformity with detected image features.

The main contributions of the proposed work are that it provides a complete algorithmic framework for the time resolved 3D reconstruction of interventional devices from biplane fluoroscopy sequences, that it is able to extract the device path in the 2D projection images without relying on iterative optimization of 2D splines or 3D snakes^3,6,7,9 and that it is performed fully automatically and does not require manual seed point selection.^4,5 Instead, topology preserving thinning is used to extract the centerline of curvilinear segments, which are then connected based on spatial distance and directional difference between endpoints. This method has the advantage that arbitrary device shapes can be described as a list of connected pixels and it is not limited by the number of parameters used to describe splines or snakes. Although these methods have been used successfully for the reconstruction of curvilinear devices, the iterative optimization procedure might increase the computation time, which could make a potential real time implementation of the algorithm difficult. Furthermore, good initialization is required for the optimization in each frame, which might not be given for fast device movement between adjacent frames. Finally, certain device shapes might cause the fitting procedure to fail.³ The proposed algorithm uses the first time frame of each sequence as mask, which is registered to and subtracted from the following frames. The images are then preprocessed by a noise reduction algorithm using directional adaptive filter kernels and a ridgeness filter that emphasizes curvilinear structures. A threshold based segmentation of the device is then performed, followed by a flux driven topology preserving thinning algorithm to extract the segments of the device centerline. The exact device path is determined using Dijkstra’s algorithm¹⁶ to minimize the curvature and distance between adjacent segments as well as the difference to the device path of the previous frame. The 3D device shape is then reconstructed using epipolar geometry and a monotonic function mapping the corresponding points of each 2D device centerline.

2. RECONSTRUCTION PIPELINE

Figure 1 shows an overview of the reconstruction pipeline. First, an offline calibration step is performed to estimate the projection matrices for the desired C-arm positions. The first frame of the fluoroscopic image sequence is used as mask and is subtracted from the subsequent frames to isolate the device. In case of patient motion, the subtraction might cause artifacts. Therefore, a registration step is introduced to adjust the mask to the current frame. The subtracted frames are then preprocessed to reduce the noise and emphasize curvilinear structures in the image followed by an automatic threshold segmentation. A topology preserving thinning algorithm is then applied to extract the segments of the 2D device centerline. The exact device path connecting the segments is found using Dijkstra’s algorithm,¹⁶ minimizing the curvature and distance between adjacent segments as well as the difference to the device path of the previous frame. Finally, the best point correspondences for each line segment of the centerline are found and an epipolar reconstruction is performed. Sections 2.A–2.F give detailed descriptions of each step.

FIG. 1.

Flow diagram of the reconstruction pipeline. The first frame of both planes is used as a mask image and registered to the current frame. A noise reduction and ridgeness filter is then applied to the subtraction of current frame and mask. Subsequently, the image is binarized to separate the device from the background and the centerline is extracted using topology preserving thinning and Dijkstra’s path search algorithm. Finally, an epipolar reconstruction is performed using the projection matrices from an offline calibration step to obtain the 3D shape of the device.

2.A. Calibration

As a first step, a calibration of the desired C-arm positions of the angiography system has to be performed to estimate the projection matrices P_a|b for both image planes A and B, respectively, which describe the transformation of a 3D point into the 2D image space of each plane. The existing system calibration used to reconstruct rotational 3D acquisitions cannot be used, since it is performed for dynamic C-arm trajectories and would not be accurate enough for static C-arm positions. Furthermore, only plane A is calibrated to perform 3D acquisitions. The calibration for the biplane fluoroscopy reconstruction can be done in advance and does not have to be repeated for every case, as long as the system geometry does not change. Small variations of the C-arm positions, however, are expected depending on the system accuracy. A cylindrical calibration phantom with metal beads in two different sizes, arranged in a helical pattern, is used. A rotational 3D acquisition of the phantom is performed as well as 2D fluoroscopic acquisitions from the desired view angles (anteroposterior and lateral). The coordinates of the center of the metal beads are segmented both in the 3D and 2D images and an iterative optimization is performed that minimizes the mean squared distance between the forward projected 3D coordinates and the 2D coordinates to estimate the projection matrix. A more detailed description on camera calibration and projection matrices can be found in Refs. ^{8, 17}, and ¹⁸.

2.B. Registration

For most neuroradiological interventions in the head, little motion can be expected, which allows subtracting previously acquired mask frames to remove the background and facilitates the segmentation. However, in some cases, breathing motion near the neck or small variations of the head position can be observed, which causes curvilinear structures along the border of high contrast objects such as the skull. To avoid interference of this artifacts with the device segmentation, an elastic grid registration technique is used, based on the method presented in Ref. 19. The image is partitioned into distinct blocks, where the center of each block serves as node of the elastic grid as demonstrated in Fig. 2. The registration method estimates a translation vector for each of these nodes. The image transformation is then determined by interpolating the translation vector for each pixel using a spline interpolation. The optimization of the node translation vectors is done using a gradient descent method, where the gradient can be estimated by calculating the image metric within the surrounding block of the node only. In this case, the mean squared error (MSE) is used as image metric,

$${\displaystyle \mathrm{MSE}=\sum _{\forall \overrightarrow{x}\in \mathcal{I}}{\left[I_F\left(\overrightarrow{x}\right)-I_M\left(\overrightarrow{x}+\mathrm{\Delta }\left(\overrightarrow{x}\right)\right)\right]}^2/\left|\mathcal{I}\right|,}$$ (1)

where $\mathcal{I}$ is the set of all $\overrightarrow{x}$ in the image and $\mathrm{\Delta }\left(\overrightarrow{x}\right)$ is the translation of the pixel $\overrightarrow{x}$ calculated by

$${\displaystyle \mathrm{\Delta }\left(\overrightarrow{x}\right)=\sum _{i=0}^3\sum _{j=0}^3{a}_{ij}\left(⌊\frac{x_0}{m_0}⌋,⌊\frac{x_1}{m_1}⌋\right)\cdot {\left(\frac{x_0-⌊\frac{x_0}{m_0}⌋\cdot m_0}{m_0}\right)}^i\cdot {\left(\frac{x_1-⌊\frac{x_1}{m_1}⌋\cdot m_1}{m_1}\right)}^j.}$$ (2)

The block size in this equation is given by m₀ and m₁, respectively, and a_ij(b₀, b₁) represents the cubic spline coefficients for the image block (b₀, b₁).

FIG. 2.

Illustration of the elastic grid registration. The solid line represents the image blocks. Their centers are the node for the elastic grid, shown as dashed line. A translation vector is estimated for each node, which is used to calculate the final image transformation by spline interpolation.

2.C. Preprocessing

The subtracted image frames are preprocessed to reduce the noise and emphasize the curvilinear structures of the device. The noise reduction algorithm uses binary masks to determine a directional filter kernel for pixels close to line structures and an isotropic kernel for background pixels.²⁰ Which type of kernel is used for each pixel is determined by a line conformity measure (LCM) calculated using the mean squared error in the local neighborhood along different directions. A detailed description of the algorithm can be found in Ref. 20. Additionally, a ridgeness filter is applied based on a method described in Ref. 21 for a fixed scale to emphasize curvilinear structures (ridges) in the image. The ridgeness values R for image I are calculated by

$${\displaystyle R=-0.5\cdot \left(\frac{{\partial }^{2}I}{\partial x_0^2}+\frac{{\partial }^{2}I}{\partial x_1^2}\right)-\sqrt{{\left(\frac{{\partial }^{2}I}{\partial x_0^2}-\frac{{\partial }^{2}I}{\partial x_1^2}\right)}^2+4\cdot {\left(\frac{{\partial }^{2}I}{\partial x_0\partial x_1}\right)}^2}.}$$ (3)

2.D. Segmentation

The segmentation step is based on two thresholds τ_h and τ_l with τ_h > τ_l, where each pixel with a gray value above the threshold τ_h is considered to be part of the object, herein, referred to as initial device pixel. Candidate pixels with a value within the interval $\left[{\tau }_l,{\tau }_h\right]$ are only considered part of the device, if they are connected to an initial device pixel. Here, the term connected refers to each candidate pixel which has at least one neighboring initial device pixel or candidate pixel within the four-connected neighborhood, which is itself connected to an initial device pixel. The threshold values are determined automatically using the image histogram as illustrated in Fig. 3. Since the device only covers a small number of pixels compared to the background area, a Gaussian function can be fitted to the histogram in order to estimate the probability distribution G_B of the background pixels. A maximum likelihood estimation²² is used to determine the parameters of the Gaussian function. The upper and lower thresholds are then determined by calculating the values where the probability for the pixel to be part of the background area is lower than 0.1% and 10⁻⁵%, respectively.

FIG. 3.

Automatic threshold detection by fitting a Gaussian function to the image histogram. The lower and upper thresholds are then determined by the points where the estimated probability distribution reaches 0.1% and 10⁻⁵%, respectively.

2.E. Centerline extraction

Giving the binarized images, a topology preserving thinning algorithm is applied to extract the device centerline. The algorithm is based on the average outward flux (AOF) as proposed in Ref. 23. The distance map D is calculated which contains for each device pixel the Euclidean distance to the closest background pixel. The AOF is then calculated by

$${\displaystyle \mathrm{AOF}\left(\overrightarrow{x}\right)=\frac{\sum _{\forall \overrightarrow{\mathrm{\Delta }}\in \underset{x}{N}}\left[\frac{\partial D}{\partial x_1}(\overrightarrow{x}+\overrightarrow{\mathrm{\Delta }}),\frac{\partial D}{\partial x_2}(\overrightarrow{x}+\overrightarrow{\mathrm{\Delta }})\right]\cdot \frac{\overrightarrow{\mathrm{\Delta }}}{∥\overrightarrow{\mathrm{\Delta }}\hspace{0.17em}∥}}{\left|N_x\right|},}$$ (4)

where N_x is the eight-connected neighborhood of $\overrightarrow{x}$. The segmented device is then converted to a one pixel thin centerline by subsequent removal of simple boundary pixels of the segmented area in descending order of the AOF value. A boundary pixel is called simple according to the definition in Ref. 24, if its removal does not change the topology of the object or the background. For 2D images, this is true if the Hilditch crossing number²⁵ equals one. Following the leftover device pixels after applying the thinning algorithm, the centerline can be extracted, represented by one or more curvilinear segments. Each segment is described by a list of connected pixels. Multiple segments might be necessary to describe a single device in cases where the centerline is interrupted due to bad image contrast or noise or if the centerline overlaps itself. In the latter case, the path from each intersection point or end point to the next is described by a separate line segment.

The device segments S₁–S_n are connected to a single centerline using Dijkstra’s algorithm¹⁶ to find a path that minimizes the distance and direction change between adjacent ridges as well as the difference to the device path of the previous time frame. The costs c_con for the connection of two segments is calculated by

$${\displaystyle c_{\mathrm{con}}(S_i,S_j)={\left(\eta +\left|{\overrightarrow{v}}_{i,j}\right|\right)}^{\varphi \cdot \left[2-{\dot{\overrightarrow{s}}}_{i,m_i}^{\top }\cdot {\overrightarrow{v}}_{i,j}-{\overrightarrow{v}}_{i,j}^{\top }\cdot \left(-{\dot{\overrightarrow{s}}}_{j,1}\right)\right]+1},}$$ (5)

with

$${\displaystyle {\overrightarrow{v}}_{i,j}={\overrightarrow{s}}_{i,m_i}-{\overrightarrow{s}}_{j,1},}$$ (6)

where each segment S_i is a list of connected points ${\overrightarrow{s}}_{i,1}$ to ${\overrightarrow{s}}_{i,m_i}$ and the direction vector at each point is given by ${\dot{\overrightarrow{s}}}_{i,j}$. The parameters η > 1 and ϕ > 0 determine the weighting of the direction change between segments. For the purpose of this work, η = 1.5 and ϕ = 0.5 were chosen empirically in previous 2D tracking experiments. The difference between the extracted device path and the device path of the previous time frame $D_{(t-1)}={\left({\overrightarrow{d}}_{(t-1),1},\dots ,{\overrightarrow{d}}_{(t-1),n\left(D_{(t-1)}\right)}\right)}^{\top }$ is calculated using

$${\displaystyle c_{\mathrm{dis}}({\overrightarrow{s}}_{i,j},{\overrightarrow{d}}_{(t-1),k})={\left(\eta +\left|{\overrightarrow{s}}_{i,j}-{\overrightarrow{d}}_{(t-1),k}\right|\right)}^{\varphi \cdot \left(1-{\dot{\overrightarrow{s}}}_{i,j}^{\top }\cdot {\dot{\overrightarrow{d}}}_{t-1,k}\right)+1}.}$$ (7)

The overall cost function for a series of segments $S=\left(S_1,S_2,\dots ,S_{n\left(S\right)}\right)$ is defined by

$${\displaystyle c\left(S\right)=\sum _{i=2}^{n\left(S\right)}{c}_{\mathrm{con}}\left(S_{i-1},S_i\right)+\sum _{k=1}^{n\left(D_{t-1}\right)}{c}_{\mathrm{dis}}\left(\psi \left(S,k\right),{\overrightarrow{d}}_{t-1,k}\right)+\sum _{\forall S_k\notin S}n\left(S_k\right)\cdot ϵ,}$$ (8)

where $\psi \left(S,k\right)$ is the monotonic function, which maps ${\overrightarrow{d}}_{(t-1),k}$ to its corresponding point in the sequence S and the parameter ϵ determines the cost of unused line segments. For the purpose of this work, ϵ = 0.5 was used.

2.F. Reconstruction

Given a pair of corresponding points $({\overrightarrow{q}}_a,{\overrightarrow{q}}_b)$ from both image planes that represent the same point ${\overrightarrow{q}}_3$ in 3D, its coordinates can be easily calculated by finding the intersection between the two projection rays of the 2D points, which connect ${\overrightarrow{q}}_a$ and ${\overrightarrow{q}}_b$ with the respective view points of the image planes. The viewpoints ${\overrightarrow{v}}_{a|b}$ and the direction vectors ${\overrightarrow{r}}_{a|b}$ of the projection rays can be calculated using the respective projection matrices. The direction vectors are calculated by taking the inverse of the upper left 3 × 3 matrix of the projection matrix,

$${\displaystyle {\overrightarrow{r}}_{a|b}={\left(P_{a|b}^{(3\times 3)}\right)}^{-1}\cdot {\overrightarrow{q}}_{a|b}={\left(\begin{array}{ccc}\hfill p_{11}\hfill & \hfill p_{12}\hfill & \hfill p_{13}\hfill \\ \hfill p_{21}\hfill & \hfill p_{22}\hfill & \hfill p_{23}\hfill \\ \hfill p_{31}\hfill & \hfill p_{32}\hfill & \hfill p_{33}\hfill \end{array}\right)}^{-1}\cdot {\overrightarrow{q}}_{a|b}.}$$ (9)

The viewpoints can be calculated using

$${\displaystyle {\overrightarrow{v}}_{a|b}={\left(P_{a|b}^{(3\times 3)}\right)}^{-1}\cdot \left(\begin{array}{c}\hfill p_{14}\hfill \\ \hfill p_{24}\hfill \\ \hfill p_{34}\hfill \end{array}\right).}$$ (10)

Due to aliasing, segmentation inaccuracies, or imperfect calibration, the two lines might not intersect; therefore, the closest point to both lines is calculated instead by solving the linear equation,

$${\displaystyle \underset{M}{\underbrace{\left(\begin{array}{c}\hfill {\overrightarrow{n}}_{a1}\hfill \\ \hfill {\overrightarrow{n}}_{a2}\hfill \\ \hfill {\overrightarrow{n}}_{b1}\hfill \\ \hfill {\overrightarrow{n}}_{b2}\hfill \end{array}\right)}}\cdot {\overrightarrow{q}}_3=\underset{\overrightarrow{c}}{\underbrace{\left(\begin{array}{c}\hfill \left(\begin{array}{c}\hfill {\overrightarrow{n}}_{a1}\hfill \\ \hfill {\overrightarrow{n}}_{a2}\hfill \end{array}\right)\cdot {\overrightarrow{v}}_a\hfill \\ \hfill \left(\begin{array}{c}\hfill {\overrightarrow{n}}_{b1}\hfill \\ \hfill {\overrightarrow{n}}_{b2}\hfill \end{array}\right)\cdot {\overrightarrow{v}}_b\hfill \end{array}\right)}},}$$ (11)

where ${\overrightarrow{n}}_{a1}$ and ${\overrightarrow{n}}_{b1}$ are normal vectors perpendicular to ${\overrightarrow{r}}_a$ and ${\overrightarrow{r}}_b$, respectively, with ${\left|{\overrightarrow{n}}_{\left(a\right|b)1}\right|}_2=1$ and

$${\displaystyle {\overrightarrow{n}}_{\left(a\right|b)2}={\overrightarrow{r}}_{a|b}\times {\overrightarrow{n}}_{\left(a\right|b)1}.}$$ (12)

This can be solved using the Moore–Penrose inverse,

$${\displaystyle {\overrightarrow{q}}_3={\left(M^{\top }M\right)}^{-1}{M}^{\top }\cdot \overrightarrow{c}.}$$ (13)

The corresponding point q_b in plane B for a point q_a in plane A can be found by calculating the epipolar line E_qa of q_a in plane B.¹⁷ Every intersection of E_qa with the device centerline in plane B is a candidate for the corresponding point. However, only for very simple device shapes, this would result in a unique solution. In general, more than one candidate might exist for every point q_a. In this case, additional information is required to find the correct corresponding point. As suggested in Ref. 5, a monotonic constraint can be used for the function that maps points on the centerline of the device in plane A to their respective corresponding points. This means that the points in plane A have to appear in the same order on the centerline of the device then their counterparts in plane B. To determine the point correspondences, Dijkstra’s algorithm¹⁶ is used to find the monotonic function that minimizes the distance between the projection rays of each pair of corresponding points.

3. IMAGE DATA

All images are acquired using a biplane angiography system (Artis Zee, Siemens Healthcare AG, Forchheim, Germany). As a first step, a 3D image of the vasculature is acquired using 3D digital subtraction angiography. Therefore, two rotational acquisitions are performed with and without contrast agent injection, respectively. The subtraction of both images shows only the contrast medium within the vascular system. The C-arms are then moved to one of the precalibrated positions, where 2D fluoroscopic projection images of the moving device are acquired from two different angles, with a constant frame rate of 15 fps. For the quantitative evaluation of the 3D reconstruction accuracy, a guidewire was imaged within a vascular head phantom (Replicator, Vascular Simulations, Stony Brook, NY, USA). Additionally, a cadaver head study and canine study were performed to verify the results. The vascular head phantom is shown in Fig. 4; the cadaver head and canine images are shown in Fig. 6. The guidewire was moved under fluoroscopic guidance to five different positions in the head phantom, one position in the cadaver heads and four positions in the canine. For each position, a rotational 3D acquisition of static device was performed as a reference of the actual device shape. Reconstruction results of real patients will also be presented but are not included in the quantitative evaluation since no reference positions for the device are available.

FIG. 4.

Vascular head phantom: (a) surface rendering of the vascular structure within the head phantom, created by 3D digital subtraction angiography; (b) fluoroscopic image of the catheter acquired with plane A; (c) fluoroscopic image of the catheter acquired with plane B; and (d) photo of the vascular head phantom with pump to simulate blood flow.

FIG. 6.

Original fluoroscopic images and reconstruction results of the cadaver head and animal study. Rows 1 and 2 show the A and B planes of the original fluoroscopic images, respectively. Row 3 shows a semitransparent overlay of the 3D reference device (blue) with the reconstructed device (red). The cadaver head study is displayed in column 1. Columns 2–5 are the results of the animal study (see color online version).

An anterior–posterior (AP) position for plane A and a lateral projection angle for plane B are used for all acquisitions. Since in practice other gantry positions might also be of interest, an additional repeatability study is performed to investigate the influence of other projection angles on the reconstruction accuracy. Therefore, a series of ten fluoroscopy sequences of a calibration phantom was acquired for five different positions of each gantry. Between two adjacent acquisitions, each gantry was moved to arbitrary locations and then moved back to one of the five positions. The accuracy, each position could be reached with, was measured in terms of the standard deviation of the calibration phantom beads positions. Table I lists the investigated gantry positions for both planes.

TABLE I.

Gantry positions for plane A and plane B (primary and secondary angles) for each of the five positions evaluated in the repeatability study. The primary angle is rotated around the longitudinal axis of the patient and the secondary angle around the normal vector of the plane described by the C-arm.

Position No.	1	2	3	4	5
Primary A (deg)	0	30	45	0	10
Secondary A (deg)	0	0	0	−20	−10
Primary B (deg)	−90	−60	−45	−90	−80
Secondary B (deg)	0	0	0	20	10

4. STATISTICS FOR EVALUATION

The quantitative evaluation of the 3D reconstruction accuracy is performed in terms of three different measures. First, the tip accuracy e_t measured in terms of the Euclidean distance between the reconstructed device tip ${\overrightarrow{d}}_{t,0}$ and the reference position ${\overrightarrow{d}}_{t,0}^{\ast }$ gathered from the rotational 3D acquisition of the device,

$${\displaystyle e_t=∥{\overrightarrow{d}}_{t,0}-{\overrightarrow{d}}_{t,0}^{\ast }∥.}$$ (14)

The worst conformity between the reconstructed and the reference device shape can be measured in terms of the Hausdorff distance e_h,²⁶ which can be defined as the maximum of the longest Euclidean distance from any point ${\overrightarrow{d}}_{t,k_1}$ on the reconstructed device centerline to the closest point ${\overrightarrow{d}}_{t,k_2}^{\ast }$ on the reference centerline and the longest distance from any point on the reference centerline to the closest point on the reconstructed device centerline,

$${\displaystyle e_h=max\left\{\underset{\forall k_1}{max}\hspace{0.17em}\underset{\forall k_2}{min}∥{\overrightarrow{d}}_{t,k_1}-{\overrightarrow{d}}_{t,k_2}^{\ast }∥,\underset{\forall k_2}{max}\hspace{0.17em}\underset{\forall k_1}{min}∥{\overrightarrow{d}}_{t,k_1}-{\overrightarrow{d}}_{t,k_2}^{\ast }∥\right\}.}$$ (15)

Finally, the overall reconstruction accuracy of the reconstructed device is measured in terms of the mean distance e_m, where each point ${\overrightarrow{d}}_{t,k_1}$ on the reconstructed centerline is compared to the point ${\overrightarrow{d}}_{t,k_2}^{\ast }$ on the reference centerline with the same distance to the device tip,

$${\displaystyle e_m=\sum _{\forall k_2}∥{\overrightarrow{d}}_{t,k_1}-{\overrightarrow{d}}_{t,k_2}^{\ast }∥/∥T∥,}$$ (16)

with

$${\displaystyle k_1=\underset{k_1}{\arg \hspace{0.17em}\min }\sum _{k=1}^{k_1}∥{\overrightarrow{d}}_{t,k_1-1}-{\overrightarrow{d}}_{t,k_1}∥-\sum _{k=1}^{k_2}∥{\overrightarrow{d}}_{t,k_2-1}^{\ast }-{\overrightarrow{d}}_{t,k_2}^{\ast }∥.}$$ (17)

5. RESULTS

Table II lists the accuracy results of the device reconstruction from the vascular phantom experiments, measured in terms of the tip accuracy as well as the Hausdorff distance of the device centerlines and the mean distance. For the tip of the device, an average reconstruction accuracy of 0.35 ± 0.09 mm could be achieved, while the overall Hausdorff distance, and therefore, the largest distance between the reconstructed device centerline and the reference centerline was 0.65 ± 0.32 mm on average. The mean point by point device distance of the centerlines averaged to 0.54 ± 0.33 mm. Figure 5 shows a comparison of the reconstructed device and the reference device within the vascular system of the head phantom. A similar evaluation has been performed for the cadaver head study. The quantitative results for the cadaver head and canine study are listed in Table III. The device tip was reconstructed with an average accuracy of 0.36 ± 0.20 mm, whereas the largest distance between the reconstructed device centerline and the reference centerline was 0.62 ± 0.08 mm on average. The mean device distance averaged to 0.41 ± 0.08 mm. A visual comparison between the reconstructed and the reference devices is shown in Fig. 6. A clinical data set of a pipeline embolization procedure in a patient suffering a cerebral aneurysm was processed retrospectively. Figure 7 shows selected time frames of the 3D reconstruction, rendered as virtual endoscopic, and glass pipe views showing the guidewire near the middle cerebral artery (MCA) trifurcation. Additional clinical examples are shown in Fig. 10. The first example is a reconstruction of a guidewire, where additional to the reconstructed device other objects like a stent, leftover contrast medium, and a previous guidewire are visible in the subtracted fluoroscopy frames. The intermediate reconstruction steps are shown in Fig. 10 (A1–A4) including the subtracted fluoroscopic frame, the binarized image overlaid with the extracted curvilinear segments, the final 2D path, and finally the 3D reconstruction. The second example shows the coiling of an aneurysm, where the first 1.5 loops of the coil inside of the aneurysm can be successfully reconstructed; however, after multiple loops, the extraction of the correct 2D path is not possible anymore. The results of the repeatability study are shown in Fig. 8. For plane A, the highest accuracy was achieved for the 0°/0° position with an average accuracy of 0.45 ± 0.28 pixel (0.14 ± 0.09 mm) and the worst repeatability for position 10°/ − 10° with an accuracy of 0.98 ± 0.92 pixels (0.30 ± 0.28 mm). For plane B, the best results were achieved for position −90°/20° with an accuracy of 0.36 ± 0.33 pixels (0.11 ± 0.10 mm) and the worst for position −80°/10° with 1.48 ± 1.37 pixels (0.46 ± 0.42 mm). To give an impression on the influence of the registration part, Fig. 9 shows a comparison of the subtracted fluoroscopy frames before and after applying the registration algorithm for two clinical acquisitions, where the head moved between the mask frame and the current frame.

TABLE II.

Reconstruction accuracy results for the phantom experiments listing tip accuracy e_t, Hausdorff distance e_h, and mean distance e_m for each case. The last row shows the average results for all cases. All measurements are given in millimeters.

Case No.	et	eh	em
Case 1	0.30	0.33	0.31 ± 0.14
Case 2	0.31	0.49	0.32 ± 0.13
Case 3	0.43	0.80	0.70 ± 0.21
Case 4	0.47	0.47	0.31 ± 0.09
Case 5	0.26	1.13	0.91 ± 0.32
Average	0.35 ± 0.09	0.65 ± 0.32	0.54 ± 0.33

FIG. 5.

3D rendering of the reference device (blue) compared to the reconstructed device (red) within the vascular system of the head phantom shown from two different viewing angles (see color online version).

TABLE III.

Reconstruction accuracy results for the cadaver head and canine experiments listing tip accuracy e_t, Hausdorff distance e_h, and mean distance e_m for each case. All measurements are given in millimeters.

Case No.	et	eh	em
Cadaver 1	0.21	0.59	0.23 ± 0.11
Canine 1	0.33	0.57	0.39 ± 0.12
Canine 2	0.58	0.58	0.34 ± 0.03
Canine 3	0.53	0.77	0.68 ± 0.08
Canine 4	0.14	0.59	0.39 ± 0.08
Average	0.36 ± 0.20	0.62 ± 0.08	0.41 ± 0.08

FIG. 7.

Clinical data set of a patient suffering a cerebral aneurysm. First two rows show the original fluoroscopic images of planes A and B, respectively. The third row shows the virtual endoscopic renderings of a guidewire near the MCA trifurcation. The green line represents the path to the predetermined target location, calculated from the vessel centerlines. The last row displays an overview of the guidewire position in a 3D rendering with a semitransparent vessel surface (see color online version).

FIG. 10.

Example of clinical cases. (A) Guidewire in aneurysm with additional objects in the image (stent, previous guidewire, leftover contrast medium, and registration artifacts). (1) Original fluoroscopy frame. (2) Binarized image overlaid with extracted line segments. (3) Extracted 2D path overlaid with original fluoroscopy image. (4) 3D reconstruction of the guidewire within the aneurysm. (B) Coiling of aneurysm. (1) Extracted 2D path after 1.5 loops. (2) 3D reconstruction after 1.5 loops. (3) After multiple loops, 2D path cannot be extracted correctly. (4) Failed 3D reconstruction after multiple loops.

FIG. 8.

Results of the repeatability study measured in terms of the mean error in pixels at five different positions for both planes. First five bars show the accuracy of plane A; the remaining bars give the accuracy of plane B. Oblique angles, where the secondary angle is not zero are shaded gray; other positions are shaded white.

FIG. 9.

Comparison of subtracted fluoroscopy frames for two clinical acquisitions (A) and (B) before and after application of the registration algorithm. Motion artifacts can be seen near the orbital and nasal cavities and along the contour of the skull in the unregistered subtractions of (A) and (B).

6. DISCUSSION

The presented algorithm offers a novel perspective for intervascular procedures allowing to render virtual endoscopic views of the device within the vascular system as well as the option to generate virtual projection images from arbitrary views without moving the gantry. In order for this technique to provide useful information to physicians, an accurate reconstruction of the 3D device shape is crucial. In particular, the position of the device tip is important for many procedures, for example, if the device tip is placed within an aneurysm or stenosis in order to deploy coils or stents. The accuracy of the reconstruction depends on various factors, including the segmentation of the device centerline, finding the correct point correspondences as well as the calibration of the C-arm positions and the angle between the two views. Most manufactures provide calibration phantoms for their angiography systems to allow an accurate estimation of the projection matrices. However, current angiography systems can reach the calibrated C-arm positions only up to a certain accuracy depending on the viewing angle, which might decrease the alignment between the estimated projection matrices and the actual C-arm positions. The presented results have demonstrated that a reconstruction with high accuracy is possible using anterior–posterior and lateral C-arm projection angles. In some cases, however, different C-arm positions might be required to allow better access to the patient. The C-arm positions primarily affect the reconstruction accuracy in two ways. First, the angle between the two projections of plane A and plane B determines how well conditioned the reconstruction of a 3D point from its 2D projections is. The best result is achieved for orthogonal projection angles. A thorough investigation on this topic was performed in Ref. 8. Since the calibration of the system is performed ahead of the procedure, the accuracy of the reconstruction also depends on the accuracy with which the system can go back to a previously calibrated position. The results of the repeatability study (Fig. 8) show a considerable decrease in the repeatability for angles other than AP and lateral for planes A and B. Oblique angles, where the primary as well as the secondary angle is modified, show the lowest accuracy. In general, less accurate results can be expected for other gantry positions than AP and lateral. However, depending on the system, other positions with similar repeatability might be possible as can be seen for plane B and the −90°/20° position. Another limitation regarding the accuracy of the projection matrices is that the patient table position may not be altered between the acquisition of the 3D vasculature and the fluoroscopic images, e.g., to adjust the field of view, since the measurement accuracy of the table position may not be accurate enough to adjust the projection matrices. To solve this issue, a 2D/3D registration step could be included whenever the table position or the position of the C-arms is changed. This would also allow the use of uncalibrated C-arm positions and might resolve variations in the C-arm positions due to system inaccuracies for calibrated positions. It should also be mentioned that all accuracy measurements are calculated compared to a reference device centerline calculated from a voxelized 3D volume. Therefore, the accuracy of the reference is limited by the voxel size of 0.47 mm. The imaging system allows reconstructions with smaller voxel sizes only for reduced volume sizes. This, however, would truncate the device in the reconstructions. The calculated accuracy measures are close to the voxel size limit, which shows that a device reconstruction with high accuracy is possible. The reconstructions of the clinical data set shown in Fig. 7 provide an impression of the possibilities and advantages of using this technology compared to conventional fluoroscopic imaging. The virtual endoscopic view allows an easier localization of the guidewire near the trifurcation, where the complex vessel structure makes it hard to identify the current device position in the original fluoroscopy images. The additional clinical examples presented in Fig. 10 also show the abilities as well as the limits of the presented algorithm. In general, additional objects like stents present in the mask as well as the reconstructed frames do not affect the reconstruction, since they are subtracted out during the segmentation. However, if the position of these objects changes during the procedure, they could be visible even in the subtracted frames as shown in Fig. 10 (A1). But, due to the lower radio opacity of the stent compared to the guidewire, the correct path can still be identified. Previously inserted objects that are present in the mask frame, but have been removed afterward, show up as dark structures in the subtracted frame, which makes it easy to distinguish them from the current device. However, if a previous device overlays with the curvilinear structure of the reconstructed device, it might cancel out parts of it in the subtracted image frame. However, in the presented cases, the algorithm is still able to connect the extracted line segments correctly. The aneurysm coiling case in Fig. 10 (B1–B4) demonstrates that the algorithm is able to identify the device path of a coil despite overlaps after about 1.5 loops. However, when the device shape gets more complex, no feasible solution can be found. Although a real time implementation of the proposed algorithm is not available to this time, it is planned for future work. In order to further reduce the computational effort, the number of iterations for the registration step could be limited, since only little motion is expected for most neuroradiological applications in the head. Furthermore, the presence of patient motion could be detected based on simple image metrics, and the registration algorithm could be applied only to frames, where motion was detected. Further work is also necessary to investigate other applications of the reconstruction algorithm besides neuroradiological interventions in the head region. These include thoracic procedures in the lung or liver as well as cardiac applications. In contrast to the cases presented here, these applications usually exhibit increased motion artifacts caused by breathing and cardiac motion, which might not be compensated by a 2D registration as it is proposed here. This might affect the segmentation of the device based on subtracted image frames. Additionally, in order to create a meaningful overlay of the reconstructed device with a previously acquired 3D vascular volume, the latter has to be adapted to reflect the patient motion.

7. CONCLUSION

The proposed algorithm provides a novel framework to reconstruct the 3D shape of an interventional device from two fluoroscopic images recorded from different angles, thus allowing the rendering of virtual fluoroscopic images from arbitrary angles or virtual endoscopic views. This algorithm consists of 2D/2D image registration, device segmentation, and epipolar reconstruction. Future work has to be done to implement the algorithm into a real time reconstruction system, which allows viewing of virtual endoscopic views and virtual fluoroscopic images online, while manipulating the interventional device. Furthermore, the implementation of a 2D/3D registration could help to further increase the reconstruction accuracy and could provide a more flexible use of the C-arm CT system. The presented results show that an accurate reconstruction of the device is possible. This technique could considerably improve the workflow of endovascular minimally invasive procedures.