Alternating dual updates algorithm for X-ray CT reconstruction on the GPU

Abstract

Model-based image reconstruction (MBIR) for X-ray computed tomography (CT) offers improved image quality and potential low-dose operation, but has yet to reach ubiquity in the clinic. MBIR methods form an image by solving a large statistically motivated optimization problem, and the long time it takes to numerically solve this problem has hampered MBIR’s adoption. We present a new optimization algorithm for X-ray CT MBIR based on duality and group coordinate ascent that may converge even with approximate updates and can handle a wide range of regularizers, including total variation (TV). The algorithm iteratively updates groups of dual variables corresponding to terms in the cost function; these updates are highly parallel and map well onto the GPU. Although the algorithm stores a large number of variables, the “working size” for each of the algorithm’s steps is small and can be efficiently streamed to the GPU while other calculations are being performed. The proposed algorithm converges rapidly on both real and simulated data and shows promising parallelization over multiple devices.

I. Introduction

X-ray computed tomography (CT) model-based image reconstruction (MBIR) combines information about system physics, measurement statistics, and prior knowledge about images into a high-dimensional cost function [1]. The variate of this function is an image; the image that minimizes this cost function can contain less noise and fewer artifacts than those produced with conventional analytical techniques, especially at reduced doses [1]–[3].

The primary drawback of MBIR methods is how long it takes to find this minimizer. In addition to general optimization algorithms like conjugate gradients with specialized preconditioners [4], [5], a wide range of CT-specialized algorithms have been proposed to accelerate the optimization. One popular approach uses iterated coordinate descent (ICD) to sequentially update pixels (or groups of pixels) of the image [6], [7]. Since it is a sequential algorithm, ICD faces challenges from stagnating processor clock speeds and cannot exploit the increasing parallelization in modern computing hardware.

Another family of algorithms uses variable splitting and alternating minimization techniques to separate challenging parts of the cost function into more easily solved subproblems [8]–[11]. When used with the ordered subsets (OS) approximation [10], [12], these algorithms can converge very rapidly. Unfortunately, without relaxation, OS-based algorithms have uncertain convergence properties. Nonetheless, combining OS with accelerated first-order methods [13], [14] has produced simple algorithms with state of the art convergence speeds.

This paper proposes an algorithm that shares some properties with prior works. Like some variable splitting methods, our proposed algorithm consists of steps that consider parts of the cost function in isolation. Separating jointly challenging parts of the cost function from one another allows us to use specialized and fast solvers for each part. Our algorithm also uses a group coordinate optimization scheme, somewhat like ICD, but the variables it updates are in a dual domain; updating a small group of dual variables can simultaneously update many image pixels. Like OS algorithms, our algorithm need not visit all the measured data to update the image, but unlike OS algorithms without relaxation, the proposed algorithm has some convergence guarantees.

The next section sets up our MBIR CT reconstruction problem. Section II introduces the mathematics of the proposed algorithm, and Section III describes our single- and multiple-device implementations. Section IV provides some experimental results and Section V gives some conclusions and directions for future work.

A. Model-based image reconstruction

Consider the following X-ray CT reconstruction problem [1]:

$$\widehat{\mathbf{x}}\in \underset{\mathbf{x}\ge \mathbf{0}}{\text{argmin}}\mathrm{L}(\mathbf{Ax})+\mathrm{R}(\mathbf{Cx}).$$ (1)

There are M measurements and N pixels, and we constrain all the pixels of the image x ∈ ℝ^N to be nonnegative. The CT projection matrix A ∈ ℝ^M×N models system physics and geometry, and the finite differencing matrix C ∈ ℝ^K×N computes the differences between each pixel and its neighbors. The number of differences penalized in the image, K, is a multiple of N. For example, penalizing differences along the three cardinal 3D axes would set K = 3N. The matrices A and C are too large to store in memory, so multiplication with these matrices and their adjoints is implemented “on the fly”.

Both L and R are separable sums of convex functions:

$$\mathrm{L}(\mathbf{p})=\sum _{i=1}^M{l}_i(p_i),\mathrm{R}(\mathbf{d})=\sum _{k=1}^K{r}_k(d_k).$$ (2)

We call L the data-fit term because it penalizes discrepancies between the measured data y ∈ ℝ^M and the CT projection of x. A common choice for L is a weighted sum of quadratic functions; i.e.,

$$l_i(p_i)=\frac{w_i}{2}{(p_i-y_i)}^2,$$ (3)

with the weight w_i > 0. Traditionally, the weight is the inverse of the variance of the ith measurement, $w_i=1/{\sigma }_i^2$.

Similarly, R encourages regularity of the reconstructed image x̂ by penalizing the differences between neighboring pixels. R is a weighted sum of penalized differences,

$$r_k(d_k)={\beta }_k\psi (d_k).$$ (4)

The potential function ψ is convex, even, usually nonquadratic, and coercive. The quadratic penalty function, $\psi (t)=\frac{1}{2}{t}^2$, while analytically tractable, tends to favor reconstructed images x̂ with blurry edges because it penalizes large differences between neighboring pixels (i.e., edges) aggressively. Potential functions ψ(t) that have a smaller rate of growth as |t| → ∞ are called edge-preserving because they penalize these large differences less aggressively. Examples include the absolute value function ψ(t) = |t| from total variation (TV) regularization, the Fair potential, and the q-generalized Gaussian. The positive weights {β_k} are fixed and encourage certain resolution or noise properties in the image [15], [16].

The functions L and R have opposing effects on the reconstructed image x̂: L encourages data fidelity and can lift the noise from the data y into the reconstructed image, whereas R encourages smoothness at the cost of producing an image x that does not fit the measurements as well. Combining L and R complicates the task of finding a minimizer x̂. Without the regularizer R, the reconstruction problem (1) could possibly be solved using a fast quadratic solver. Conversely, without the data-fit term L (without CT system matrix A), (1) becomes a denoising problem for which many fast algorithms exist, including methods suitable for the GPU [17].

Variable splitting and alternating minimization provide a framework for separating L and R into different subproblems [18]. The ADMM algorithm in [8] used a circulant approximation to the Gram matrix A^⊤A to provide rapid convergence rates for 2D CT problems. Unfortunately, the circulant approximation is less useful in 3D CT. We partially overcame these difficulties in [19] by using a duality-based approach to solving problems involving the CT system matrix, but the resulting algorithm still used ADMM, which has difficult-to-tune penalty parameters and relatively high memory use. Gradient-based algorithms like OS [12] with acceleration [13] and the linearized augmented Lagrangian method with ordered subsets [10] (OS-LALM), can produce rapid convergence rates but use an approximation to the gradient of the data-fit term and have uncertain convergence properties. Some of these algorithms require generalizations to handle non-smooth regularizers.

This paper describes an extension of the algorithms in [19], [20]. The proposed algorithm uses duality, group coordinate ascent with carefully chosen groups, and the majorize-minimize framework to rapidly solve the reconstruction problem (1). We extend [20] by also considering the nonnegativity constraint x ≥ 0 in (1). Our algorithm is designed for the GPU: while it uses many variables, the “working set” for each of the algorithm’s steps is small and easily fits in GPU memory. We stream these groups of variables to the GPU and hide the latency of these transfers by performing other, less memory-intensive computations. We show that the proposed algorithm can be implemented with multiple GPUs for additional acceleration.

II. Reconstruction algorithm

At a high level, the proposed algorithm approximately performs the following iteration:

$${\mathbf{x}}^{(n+1)}=\underset{\mathbf{x}}{\text{argmin}}{\mathrm{J}}^{(n)}(\mathbf{x}),\text{where}$$ (5)

$${\mathrm{J}}^{(n)}(\mathbf{x})=\mathrm{L}(\mathbf{Ax})+\mathrm{R}(\mathbf{Cx})+\mathrm{N}(\mathbf{x})+\frac{\mu }{2}{\Vert \mathbf{x}-{\mathbf{x}}^{(n)}\Vert }^2,$$ (6)

with μ > 0. We have expressed the nonnegativity constraint x ≥ 0 as the characteristic function N:

$$\mathrm{N}(\mathbf{x})=\sum _{j=1}^N{\iota }_j(x_j),\text{where}$$ (7)

$${\iota }_k(x)=\{\begin{array}{ll}0,\hfill & x\ge 0;\hfill \\ \infty ,\hfill & \text{else}.\hfill \end{array}$$ (8)

Although ι_k is discontinuous, it is convex. Iteratively solving (5) exactly would produce a sequence {x⁽ⁿ⁾} that converges to a minimizer x̂ regardless of the choice of μ.

The function J⁽ⁿ⁾(x) appears to be as difficult to minimize as the original cost function (1). Even if J⁽ⁿ⁾(x) could be minimized exactly, x⁽ⁿ⁺¹⁾ is likely not a solution to the original reconstruction problem (1). However, the additional proximal term provides structure that allows us to approximately solve J⁽ⁿ⁾ efficiently using the following duality-based technique. The parameter μ controls the tradeoff between how efficiently we can approximately minimize J⁽ⁿ⁾ and how close the minimizer of J⁽ⁿ⁾ is to a minimizer of the original cost function (1).

Let $l_i^{\ast },r_k^{\ast }$ and ${\iota }_j^{\ast }$ denote the convex conjugates of l_i, r_k and ι_j, respectively, e.g., :

$$l_i^{\ast }(u_i)=\underset{p_i}{sup}{p}_i{u}_i-l_i(p_i).$$ (9)

Because l_i, r_k and ι_j are convex, they are equal to the convex conjugates of $l_i^{\ast },r_k^{\ast }$ and ${\iota }_j^{\ast }$, respectively. We use this biconjugacy property to write

$$l_i(p_i)=\underset{u_i}{sup}{p}_i{u}_i-l_i^{\ast }(u_i).$$ (10)

By summing over the indices i, j and k, we write L, R and N implicitly as the suprema of sums of one-dimensional dual functions:

$$\mathrm{L}(\mathbf{p})=\underset{\mathbf{u}}{sup}\sum _{i=1}^M{p}_i{u}_i-l_i^{\ast }(u_i)=\underset{\mathbf{u}}{sup}{\mathbf{u}}^{\top }\mathbf{p}-{\mathrm{L}}^{\ast }(\mathbf{u}),$$ (11)

$$\mathrm{R}(\mathbf{d})=\underset{\mathbf{v}}{sup}\sum _{k=1}^K{v}_k{d}_k-r_k^{\ast }(d_k)=\underset{\mathbf{v}}{sup}{\mathbf{v}}^{\top }\mathbf{d}-{\mathrm{R}}^{\ast }(\mathbf{v}),$$ (12)

$$\mathrm{N}(\mathbf{x})=\underset{\mathbf{z}}{sup}\sum _{j=1}^N{z}_j{x}_j-{\iota }_j^{\ast }(z_j)=\underset{\mathbf{z}}{sup}{\mathbf{z}}^{\top }\mathbf{x}-{\mathrm{N}}^{\ast }(\mathbf{z}).$$ (13)

With (11)–(13), we rewrite the update problem (5) as

$${\mathbf{x}}^{(n+1)}=\underset{\mathbf{x}}{\text{argmin}}\underset{\mathbf{u},\mathbf{v},\mathbf{z}}{sup}{\mathrm{S}}^{(n)}(\mathbf{x},\mathbf{u},\mathbf{v},\mathbf{z}),$$ (14)

$${\mathrm{S}}^{(n)}(\mathbf{x},\mathbf{u},\mathbf{v},\mathbf{z})\triangleq \frac{\mu }{2}{\Vert \mathbf{x}-{\mathbf{x}}^{(n)}\Vert }^2+{({\mathbf{A}}^{\top }\mathbf{u}+{\mathbf{C}}^{\top }\mathbf{v}+\mathbf{z})}^{\top }\mathbf{x}-{\mathrm{L}}^{\ast }(\mathbf{u})-{\mathrm{R}}^{\ast }(\mathbf{v})-{\mathrm{N}}^{\ast }(\mathbf{z}).$$ (15)

Reversing the order of minimization and maximization¹ yields:

$$\underset{\mathbf{x}}{min}\underset{\mathbf{u},\mathbf{v},\mathbf{z}}{sup}{\mathrm{S}}^{(n)}(\mathbf{x},\mathbf{u},\mathbf{v},\mathbf{z})=\underset{\mathbf{u},\mathbf{v},\mathbf{z}}{sup}\underset{\mathbf{x}}{min}{\mathrm{S}}^{(n)}(\mathbf{x},\mathbf{u},\mathbf{v},\mathbf{z}).$$ (16)

The now inner minimization over x is trivial to perform. We solve for the minimizer x and write it in terms of the dual variables u, v and z:

$${\stackrel{\sim }{\mathbf{x}}}^{(n+1)}(\mathbf{u},\mathbf{v},\mathbf{z})={\mathbf{x}}^{(n)}-\frac{1}{\mu }({\mathbf{A}}^{\top }\mathbf{u}+{\mathbf{C}}^{\top }\mathbf{v}+\mathbf{z}).$$ (17)

The image induced by the dual variables, x̃⁽ⁿ⁺¹⁾(u, v, z), minimizes the update cost function (5) when u, v, and z maximize the following dual function²:

$${\mathrm{D}}^{(n)}(\mathbf{u},\mathbf{v},\mathbf{z})\triangleq {\mathrm{S}}^{(n)}\left({\stackrel{\sim }{\mathbf{x}}}^{(n+1)}(\mathbf{u},\mathbf{v},\mathbf{z},),\mathbf{u},\mathbf{v},\mathbf{z}\right)$$ (18)

$$=-\frac{1}{2\mu }{\Vert {\mathbf{A}}^{\top }\mathbf{u}+{\mathbf{C}}^{\top }\mathbf{v}+\mathbf{z}\Vert }^2+{({\mathbf{A}}^{\top }\mathbf{u}+{\mathbf{C}}^{\top }\mathbf{v}+\mathbf{z})}^{\top }{\mathbf{x}}^{(n)}-{\mathrm{L}}^{\ast }(\mathbf{u})-{\mathrm{R}}^{\ast }(\mathbf{v})-{\mathrm{N}}^{\ast }(\mathbf{z}).$$ (19)

Maximizing (19), i.e., solving the dual problem (16), induces an image (17) that minimizes the update problem (5). We maximize (19) approximately using a stochastic group coordinate ascent algorithm described in the next section. Under conditions similar to other alternating minimization algorithms like ADMM [21], the proposed algorithm may converge even with these approximate updates; see Appendix C.

At a high level, our proposed algorithm iteratively performs the following steps:

1. form the dual function D⁽ⁿ⁾ in (19) using x⁽ⁿ⁾,

2. find u⁽ⁿ⁺¹⁾, v⁽ⁿ⁺¹⁾ and z⁽ⁿ⁺¹⁾ by running iterations of the SGCA algorithm detailed in the following sections:

   $${\mathbf{u}}^{(n+1)},{\mathbf{v}}^{(n+1)},{\mathbf{z}}^{(n+1)}\approx \underset{\mathbf{u},\mathbf{v},z}{\text{argmax}}{\mathrm{D}}^{(n)}(\mathbf{u},\mathbf{v},\mathbf{z}),$$ (20)
3. and update x⁽ⁿ⁺¹⁾:

   $${\mathbf{x}}^{(n+1)}={\stackrel{\sim }{\mathbf{x}}}^{(n+1)}\left({\mathbf{u}}^{(n+1)},{\mathbf{v}}^{(n+1)},{\mathbf{z}}^{(n+1)}\right).$$ (21)

A. Stochastic group coordinate ascent

We propose an SGCA algorithm to perform the dual maximization (20). The algorithm iteratively selects a group of variables (in our case, a set of the elements of the dual variables u, v and z) via a random process and updates them to increase the value of the dual function D⁽ⁿ⁾. Because SGCA is convergent [22], enough iterations of the algorithm in this section will produce dual solutions u⁽ⁿ⁺¹⁾, v⁽ⁿ⁺¹⁾ and z⁽ⁿ⁺¹⁾ that are arbitrarily close to true maximizers û⁽ⁿ⁺¹⁾, v̂⁽ⁿ⁺¹⁾ and ẑ⁽ⁿ⁺¹⁾. However, the solution accuracy of more interest is how well the induced image x⁽ⁿ⁺¹⁾ = x̃⁽ⁿ⁺¹⁾ (u⁽ⁿ⁺¹⁾, v⁽ⁿ⁺¹⁾, z⁽ⁿ⁺¹⁾) approximates the exact minimizer of (5). The data-fit and regularizer dual variables u and v affect the induced image x̃⁽ⁿ⁺¹⁾, per (17), through the linear operators A^⊤ and C^⊤ respectively. These linear operators propagate the influence of a possibly small group of dual variables to many pixels: e.g., the elements of u corresponding to a single projection view are backprojected over a large portion of the image. Consequently, performing a just a few dual group updates can significantly improve the image x̃⁽ⁿ⁺¹⁾(u, v, z).

An SGCA algorithm updates one group of variables at a time. We can form these groups arbitrarily, and as long as each group is visited “often enough” the algorithm converges to a solution [22]. To exploit the structure of D⁽ⁿ⁾, we choose each group so that it contains elements from only u, v or z; i.e., no group contains elements from different variables. Sections II-B, II-C, and II-D describe the updates for each of these groups.

Because the SGCA algorithm updates elements of the dual variables in random order, conventional iteration notation becomes cumbersome. Instead, mirroring the algorithm’s implementation, we describe the updates as occurring “in-place.” The “new” value of a variable is written with a superscripted plus, e.g., u⁺. To refer to the “current” value of a dual variable in an update problem, we use a superscripted minus; e.g., u⁻. For example to update u to maximize D⁽ⁿ⁾ while holding the other variables constant, we write u⁺ = arg max_u D(n)(u, v⁻, z⁻). That is, we replace the contents of u in memory with the maximizing value u⁺.

We rewrite the quadratic and linear terms in D⁽ⁿ⁾ using this notation and (17) by rewriting the quadratic and linear terms in (19):

$${\mathrm{D}}^{(n)}(\mathbf{u},\mathbf{v},\mathbf{z})=c-\frac{1}{2\mu }{\Vert {\mathbf{A}}^{\top }(\mathbf{u}-{\mathbf{u}}^{-})+{\mathbf{C}}^{\top }(\mathbf{v}-{\mathbf{v}}^{-})+\mathbf{z}-{\mathbf{z}}^{-}\Vert }^2+{({\mathbf{A}}^{\top }\mathbf{u}+{\mathbf{C}}^{\top }\mathbf{v}+\mathbf{z})}^{\top }\stackrel{\sim }{\mathbf{x}}-{\mathrm{L}}^{\ast }(\mathbf{u})-{\mathrm{R}}^{\ast }(\mathbf{v})-{\mathrm{N}}^{\ast }(\mathbf{z}),$$ (22)

where the buffer x̃= x̃⁽ⁿ⁺¹⁾(u⁻, v⁻, z⁻) and the constant c = D⁽ⁿ⁾(u⁻, v⁻, z⁻) − (A^⊤u⁻ + C^⊤v⁻ + z⁻)^⊤x̃ is independent of u, v and z. After any group of elements of the dual variables is updated, we update x̃ to reflect the changed dual variables (17). The following sections detail these dual variable updates.

B. Tomography (u) updates

Consider maximizing (22) with respect to some subset of the elements of u,

$${\mathbf{u}}_g^{+}=\underset{{\mathbf{u}}_g}{\text{argmax}}{\mathrm{D}}^{(n)}(\mathbf{u},{\mathbf{v}}^{-},{\mathbf{z}}^{-})$$ (23)

$$=\underset{{\mathbf{u}}_g}{\text{argmax}}-\frac{1}{2\mu }{\Vert {\mathbf{u}}_g-{\mathbf{u}}_g^{-}\Vert }_{{\mathbf{A}}_g{\mathbf{A}}_g^{\top }}^2-{\mathrm{L}}_g^{\ast }({\mathbf{u}}_g)+{\mathbf{u}}_g^{\top }{\mathbf{A}}_g\stackrel{\sim }{\mathbf{x}},$$ (24)

where u_g is a subset of the elements of u. The elements of u_g are coupled in (24) by the matrix ${\mathbf{A}}_g{\mathbf{A}}_g^{\top }$, where A_g contains the rows of A corresponding to the group u_g.

If ${\mathbf{A}}_g{\mathbf{A}}_g^{\top }$ were diagonal, i.e., if the rays corresponding to elements of u_g were nonoverlapping, then solving (24) would be trivial. (Of course this is also the case when u_g is a single element of u). However, updating u_g using only nonoverlapping rays would limit the algorithm’s parallelizability. Existing CT projector software may also not be able to efficiently compute the projection and backprojection of individual rays instead of e.g., a projection view at a time. If we allow u_g to contain overlapping rays, then the coupling induced by ${\mathbf{A}}_g{\mathbf{A}}_g^{\top }$ makes (24) expensive to solve exactly. Instead of pursuing the exact solution to (24) or using a line search with GPU-unfriendly inner products, we use a minorize-maximize technique [23], [24] to find an approximate solution that still increases the dual function D⁽ⁿ⁾.

Let the diagonal matrix M_g majorize ${\mathbf{A}}_g{\mathbf{A}}_g^{\top }$, i.e., the matrix ${\mathbf{M}}_{\mathbf{g}}-{\mathbf{A}}_g{\mathbf{A}}_g^{\top }$ has no negative eigenvalues. Solving the following separable problem produces an updated ${\mathbf{u}}_g^{+}$ that increases the dual function D⁽ⁿ⁾:

$${\mathbf{u}}_g^{+}=\underset{{\mathbf{u}}_g}{\text{argmax}}-\frac{1}{2\mu }{\Vert {\mathbf{u}}_g-{\mathbf{u}}_g^{-}\Vert }_{{\mathrm{M}}_g}^2-{\mathrm{L}}_g^{\ast }({\mathbf{u}}_g)+{\mathbf{u}}_g^{\top }{\mathbf{A}}_g\stackrel{\sim }{\mathbf{x}}.$$ (25)

In the common case that $\mathrm{L}(\mathbf{Ax})=\frac{1}{2}{\Vert \mathbf{Ax}-\mathbf{y}\Vert }_{\mathbf{W}}^2$ i.e., $l_i(p_i)=\frac{w_i}{2}{(p_i-y_i)}^2$, the conjugate ${\mathrm{L}}_g^{\ast }$ is

$${\mathrm{L}}_g^{\ast }({\mathbf{u}}_g)=\frac{1}{2}{\Vert {\mathbf{u}}_g\Vert }_{{\mathbf{W}}_g^{-1}}^2+{\mathbf{u}}_g^{\top }{\mathbf{y}}_g,$$ (26)

and the solution to (25) is

$${\mathbf{u}}_g^{+}={({\mathbf{W}}_g{\mathbf{M}}_g+\mu \mathbf{I})}^{-1}{\mathbf{W}}_g(\mu ({\mathbf{A}}_g\stackrel{\sim }{\mathbf{x}}-{\mathbf{y}}_g)+{\mathbf{M}}_g{\mathbf{u}}_g^{-}).$$ (27)

It is computationally challenging to find an “optimal” diagonal majorizing matrix ${\mathbf{M}}_g\succcurlyeq {\mathbf{A}}_g{\mathbf{A}}_g^{\top }$, but the following matrix majorizes ${\mathbf{A}}_g{\mathbf{A}}_g^{\top }$ and is easy to compute [12]:

$${\mathbf{M}}_g=\underset{i}{\text{diag}}\left\{{[{\mathbf{A}}_g{\mathbf{A}}_g^{\top }\mathbf{1}]}_i\right\}.$$ (28)

This choice of M_g depends only on the system geometry through A_g and not on any patient-specific data. Provided the groups and geometry are determined beforehand, these majorizers can be precomputed. This was the case for our experiments, and we used one group per view. Storing the diagonals of all the majorizers {M_g} took the same amount of memory as the noisy projections y.

After updating the group u_g (27), we “backproject” the update into the buffer x̃ (17):

$$\stackrel{\sim }{\mathbf{x}}\leftarrow {\stackrel{\sim }{\mathbf{x}}}^{-}-\frac{1}{\mu }{\mathbf{A}}_g^{\top }({\mathbf{u}}_g^{+}-{\mathbf{u}}_g^{-}).$$ (29)

Altogether, updating u_g and x̃ requires a forward projection and backprojection for the rays in group g and a few vector operations (27).

Running one iteration of the minorize-maximize (MM) operation (27) will not exactly maximize D⁽ⁿ⁾ with respect to u_g. One could run more MM iterations to further increase D⁽ⁿ⁾, but we have not found this necessary in our experiments. In contrast, the updates for the denoising and nonnegativity variables below are exact.

C. Denoising (v) updates

The regularizer R penalizes the differences between each pixel and its neighbors along a predetermined set of N_r directions, e.g., the three cardinal 3D directions or all thirteen 3D directions around a pixel. The finite differencing matrix C ∈ ℝ^K×N computes these differences, and each of the K = N · N_r elements of the dual variable v is associated with one of these differences. We update a subset v_g of the elements of v:

$${\mathbf{v}}_g^{+}=\underset{{\mathbf{v}}_g}{\text{argmax}}-\frac{1}{2\mu }{\Vert {\mathbf{v}}_g-{\mathbf{v}}_g^{-}\Vert }_{{\mathbf{C}}_g{\mathbf{C}}_g^{\top }}^2-{\mathrm{R}}_g^{\ast }({\mathbf{v}}_g)+{\mathbf{v}}_g^{\top }{\mathbf{C}}_g\stackrel{\sim }{\mathbf{x}}.$$ (30)

The dual vector v is enormous: in our experiments, v is as large as thirteen images. Storing a significant fraction of v on the GPU is impractical, so we update only a fraction of v at a time. To make that update efficient, we would like the group update problem (30) to decouple into set of independent one-dimensional update problems.

1) Group design

The elements of v_g are coupled in (30) only by the matrix ${\mathbf{C}}_g{\mathbf{C}}_g^{\top }$. This matrix is banded and sparse: it couples differences together that involve shared pixels. This coupling is very local; Figure 1 illustrates groups that contain only uncoupled elements of v. Updating each of these groups of differences has a “denoising” effect on almost all the pixels (up to edge conditions) in the image and involves solving set of independent one-dimensional subproblems.

Fig. 1.

Illustration of groups of elements of the dual variable v for a two-dimensional denoising case. Elements of v are updated in groups such that none of the groups affect overlapping pixels. For examples, the horizontal differences {v₁, v₃, v₅, v₇} are one group and {v₂, v₄, v₆, v₈} are another.

There are many ways to form these “covering but not overlapping” groups of differences: our implementation uses the following simple “half-direction” groups. Every element of v corresponds to a pixel location i = (i_x, i_y, i_z) and an offset o = (o_x, o_y, o_z) ∈ {±1}³. The difference that v_k represents is between the pixels located at (i_x, i_y, i_z) and (i_x + o_x, i_y + o_y, i_z + o_z). The elements of v corresponding to a single direction all share the same offset and differ only in their pixel locations.

For each difference direction r = 1, …, N_r, we form two groups, v_r,e and v_r,o. We assign every other difference “along the direction r” to each group. For example, if r indicates vertical differences along the y axis then we assign to v_r,e differences with even i_y and those with odd i_y to v_r,o. In Figure 1, the cyan group {v₉, v₁₀, v₁₁, v₁₅, v₁₆, v₁₇} and the green group {v₁₂, v₁₃, v₁₄} partition the vertical differences in this way.

More generally, let o_r = (o_x, o_y, o_z) be the offset corresponding to direction r. Let c_r ∈ {0, 1}³ contain a single “1” in the coordinate corresponding to the first nonzero element of o_r. For example,

$${\mathbf{o}}_r=(0,1,-1)\to {\mathbf{c}}_r=(0,1,0),$$ (31)

$${\mathbf{o}}_r=(0,0,1)\to {\mathbf{c}}_r=(0,0,1).$$ (32)

Recall that i_k is the location associated with the difference v_k. We assign to v_r,e those differences v_k along the direction r such that ${\mathbf{i}}_k^{\top }{\mathbf{c}}_d$ is even.

2) One-dimensional subproblems

Having chosen v_g so that its elements are uncoupled by ${\mathbf{C}}_g{\mathbf{C}}_g^{\top }$, the group update problem (30) decomposes into a set of one-dimensional difference update problems for each k in the group:

$$v_k^{+}=\underset{v_k}{\text{argmax}}-\frac{1}{\mu }{(v_k-\gamma )}^2-{\beta }_k{\psi }^{\ast }\left(\frac{v_k}{{\beta }_k}\right)$$ (33)

$$\gamma \triangleq v_k^{-}+\frac{\mu }{2}{[\mathbf{C}\stackrel{\sim }{\mathbf{x}}]}_k,$$ (34)

where ψ* is the convex conjugate of the potential function ψ. Some potential functions ψ have convenient convex conjugates that make (33) easy to directly solve:

• Absolute value: If ψ(d) = |d|, then ψ^* is the characteristic function of [−1, 1]. The solution to (33) is

  $$v_k^{+}={[\gamma ]}_{[-{\beta }_k,{\beta }_k]}.$$ (35)

  i.e., the projection of γ onto the closed interval [−β_k, β_k].

• Huber function: If ψ(d) is the Huber function,

  $$\psi (d)=\{\begin{array}{ll}\frac{1}{2}{d}^2,\hfill & \mid d\mid \le \delta \hfill \\ \delta (\mid d\mid -\frac{1}{2}\delta ),\hfill & \text{else},\hfill \end{array}$$ (36)
  then its conjugate is

  $${\psi }^{\ast }(v)=\frac{1}{2}{v}^2+{\iota }_{[-\delta ,\delta ]}(v).$$ (37)
  The solution to (33) is

  $$v_k^{+}={\left[\frac{2{\beta }_k\gamma }{2{\beta }_k+\mu }\right]}_{[-{\beta }_k\delta ,{\beta }_k\delta ]}.$$ (38)

In other cases, the convex conjugate ψ^* (33) more difficult to work with analytically. For example, the Fair potential,

$$\psi (d)={\delta }^2\left(\left|\frac{d}{\delta }\right|-log\left(1+\left|\frac{d}{\delta }\right|\right)\right),$$ (39)

is easier to work with in the primal domain, where it has a closed-form “shrinkage” operator, than the dual domain. To exploit potential functions with convenient shrinkage operators but inconvenient convex conjugates, we exploit the convexity of ψ^* and invoke biconjugacy:

$${\beta }_k{\psi }^{\ast }\left(\frac{v_k}{{\beta }_k}\right)=\underset{q_k}{sup}{q}_k{v}_k-{\beta }_k\psi (q_k).$$ (40)

Combining (40) and (33),

$$v_k^{+}=\underset{v_k}{\text{argmax}}\underset{q_k}{inf}-\frac{1}{\mu }{(v_k-\gamma )}^2-v_k{q}_k+{\beta }_k\psi (q_k).$$ (41)

By a similar Fenchel duality argument to (16), we reverse the “max” and “inf” in (41). The resulting expression involves ψ only through its “shrinkage” operator:

$$v_k^{+}=\gamma -\frac{\mu }{2}{q}_k^{+},\text{where}$$ (42)

$$q_k^{+}=\underset{q_k}{\text{argmin}}\frac{\mu }{4}{\left(q_k-\frac{2}{\mu }\gamma \right)}^2+{\beta }_k\psi (q_k).$$ (43)

After updating a group of differences v_g, we update the buffer x̃ (17):

$$\stackrel{\sim }{\mathbf{x}}\leftarrow {\stackrel{\sim }{\mathbf{x}}}^{-}-\frac{1}{\mu }{\mathbf{C}}_g^{\top }({\mathbf{v}}_g^{+}-{\mathbf{v}}_g^{-}).$$ (44)

Because v_g contains variables corresponding to nonoverlapping differences, each element of v_g updates two pixels in x̃, and each pixel in x̃ is updated by at most one difference in v_g.

D. Nonnegativity (z) updates

Updating each element of the image-sized dual variable z helps enforce the nonnegativity constraint on the corresponding pixel of x̃. The dual function D⁽ⁿ⁾ is separable in the elements of z, and the z update is

$${\mathbf{z}}^{+}=\underset{\mathbf{z}}{\text{argmax}}-\frac{1}{2\mu }{\Vert \mathbf{z}-{\mathbf{z}}^{-}\Vert }^2+{\mathbf{z}}^{\top }\stackrel{\sim }{\mathbf{x}}-{\mathrm{N}}^{\ast }(\mathbf{z})$$ (45)

$$=\sum _k-\frac{1}{2\mu }{(z_k-{\eta }_k)}^2-{\iota }_k^{\ast }(z_k),\text{where}$$ (46)

$${\eta }_k\triangleq z_k^{-}+\mu {[\stackrel{\sim }{\mathbf{x}}]}_k.$$ (47)

The dual characteristic function ${\iota }_k^{\ast }$ is also a characteristic function, but on the nonpositive numbers:

$${\iota }_k^{\ast }(z_k)=\{\begin{array}{ll}\infty ,\hfill & z_k>0\hfill \\ 0,\hfill & \text{else}.\hfill \end{array}$$ (48)

We solve (46) by clamping η_k to the nonpositive numbers:

$$z_k^{+}={[{\eta }_k]}_{(-\infty ,0]}.$$ (49)

After updating z (46) we update the buffer x̃:

$$\stackrel{\sim }{\mathbf{x}}\leftarrow {\stackrel{\sim }{\mathbf{x}}}^{-}-\frac{1}{\mu }({\mathbf{z}}^{+}-{\mathbf{z}}^{-}).$$ (50)

E. Warm starting

The dual variable updates in Sections II-B, II-C and II-D find values for the dual variables, u⁽ⁿ⁺¹⁾, v⁽ⁿ⁺¹⁾ and z⁽ⁿ⁺¹⁾ that approximately maximize the dual update problem (19). We use these dual variables and the induced solution x̃⁽ⁿ⁺¹⁾(u⁽ⁿ⁺¹⁾, v⁽ⁿ⁺¹⁾, z⁽ⁿ⁺¹⁾), stored in the buffer x̃, to determine x⁽ⁿ⁺¹⁾:

$${\mathbf{x}}^{(n+1)}={\stackrel{\sim }{\mathbf{x}}}^{(n+1)}\left({\mathbf{u}}^{(n+1)},{\mathbf{v}}^{(n+1)},{\mathbf{z}}^{(n+1)}\right)=\stackrel{\sim }{\mathbf{x}},$$ (51)

then re-form the outer update problem (5) and repeat.

We initialize our algorithm with all the dual variables set to zero. We could also reset the dual variables to zero every outer iteration, but this empirically led to slow convergence. Instead, mirroring a practice in other alternating directions algorithms, we warm-start each outer iteration with the previous iteration’s dual variable values. This has an extrapolation-like effect on the buffer x̃:

$$\stackrel{\sim }{\mathbf{x}}\leftarrow {\stackrel{\sim }{\mathbf{x}}}^{(n+2)}\left({\mathbf{u}}^{(n+1)},{\mathbf{v}}^{(n+1)},{\mathbf{z}}^{(n+1)}\right)$$ (52)

$$={\mathbf{x}}^{(n+1)}-\frac{1}{\mu }\left({\mathbf{A}}^{\top }{\mathbf{u}}^{(n+1)}+{\mathbf{C}}^{\top }{\mathbf{v}}^{(n+1)}+{\mathbf{z}}^{(n+1)}\right)$$ (53)

$$={\mathbf{x}}^{(n)}-\frac{2}{\mu }\left({\mathbf{A}}^{\top }{\mathbf{u}}^{(n+1)}+{\mathbf{C}}^{\top }{\mathbf{v}}^{(n+1)}+{\mathbf{z}}^{(n+1)}\right)$$ (54)

$$={\stackrel{\sim }{\mathbf{x}}}^{-}+\left({\mathbf{x}}^{(n+1)}-{\mathbf{x}}^{(n)}\right).$$ (55)

After initializing x̃ with this “extrapolated” value, subsequent iterated dual updates refine the update. This extrapolation is just an initial condition for the iterative algorithm solving the dual problem (19). If the dual function D⁽ⁿ⁺¹⁾ were maximized exactly then this extrapolation would have no effect on x⁽ⁿ⁺²⁾.

This section outlined the mathematical framework of our proposed CT reconstruction algorithm. Using duality and group coordinate ascent, we decomposed the process of solving the original reconstruction problem (1) into an iterated series of optimization steps, each considering only a portion of the original cost function. The next section describes how we implemented these operations on the GPU.

III. Implementation

For implementing the algorithm described in Section II, GPUs have two important properties:

• GPUs can provide impressive speedups for highly parallel workloads, and;

• GPUs often have much less memory than their host computers.

The first property means that algorithm designers should favor independent operations with regular memory accesses. Our proposed algorithm consists of five operations, each of which has an efficient GPU implementation:

• Tomography update (27): Updating the tomography dual variables corresponding to a group of projection views, u_g, consists of projecting those views, a few vector operations, and then backprojecting those views. Our algorithm relies on projections and backprojections of subsets of the data, and it should be usable with any system model that is suitable for OS methods. Implementing an efficient CT system model on the GPU is nontrivial, and we rely on previous work [25]–[27]. In our experiments, we use the separable footprints CT system model [28]. Our implementation uses thousands of threads for both projection and backprojection to exploit the GPU’s parallelism: we use one thread per detector cell in projection and one thread per voxel in backprojection.

• Denoising update (30): Updating a “half-difference” of elements of v is also highly parallel. We assign one thread to each element dual variable being updated; each thread updates two neighboring pixels of the image x̃. The workload for each thread is independent, and memory accesses are both local and regular.

• The nonnegativity update (49) and warm starting operation (55) both consist entirely of separable, parallelizable vector operations.

The GPU’s memory constraints are very relevant for large imaging problems. We performed the experiments in Section IV on a machine with four NVIDIA Tesla C2050s having 3 GB of memory apiece. The wide-cone axial experiment in Section IV-D requires about 894 MB each for the noisy data y and the statistical weights W when stored in single-precision 32-bit floating point. Storing the regularizer parameters {β_k} and a single image x would take an additional 907 MB apiece. Altogether, storing one image and the parameters of the reconstruction problem would take about 2.7 GB, leaving no room for the algorithm to store any additional state on a single GPU!

Because the X-ray CT reconstruction problem (1) is so memory-intensive, many algorithms will need to periodically transfer some data between the GPU and the host computer. If performed naïvely, these transfers can have a significant effect on algorithm speed. Fortunately, modern GPUs can perform calculations and memory transfers simultaneously, so we can “hide” the latency of these transfers to some degree.

A. Streaming

Our algorithm has many variables: the dual variable v alone is often as large as 13 image volumes. This is far too much to fit simultaneously on the GPU for many realistic problem sizes. Fortunately, each of proposed algorithm’s operations requires a comparatively small subset of the data. For example, performing a tomography update requires only x̃, and the data, weights and dual variables corresponding to the view being updated.

The algorithm in Figure 2 allocates on the GPU only

Fig. 2.

Pseudocode for the proposed algorithm. The buffer x̃_GPU is updated on the GPU using (17) in every step as the dual variables are updated, and the buffer b_GPU stores other variables as they are needed for computation on the GPU. Updating each view of u involves a one-view projection and backprojection and transferring a small amount of memory. The view weights w_g, data y_g, dual variables u_g, and majorizer weights m_g are transferred to the GPU prior to updating u_g. Only the updated u_g needs to be transferred back to the host afterwards.

• a buffer containing x̃,

• an image-sized buffer for storing z or a subset of v,

• the regularizer parameters {β_k},

and several negligibly small view-sized buffers on the GPU. The dual variables are stored primarily on the host computer and transferred to and from the GPU as needed.

The tomography update requires a relatively small amount of data: several view-sized buffers. Even for the wide-cone reconstruction in Section IV-D, each tomography update requires less than 4 MB of projection-domain data. The projection and backprojection involved in the tomography update take much longer to perform than it takes to transfer the dual variables and weights to and from the GPU. Therefore, the tomography update is computation-bound. On the other hand, the nonnegativity, denoising, and warm-start operations require whole images to be transferred to and from the GPU with relatively small amounts of computation. The speed of these operations is bounded by the latency of data transfers between the host computer and the GPU.

Modern GPUs can perform computations and transfer memory concurrently. This allows us to “hide” some of the cost of latency-bound operations by performing computation-bound operations and vice versa. The pseudocode in Figure 2 interleaves computation-bound and transfer-bound operations. After each large memory transfer is begun, the algorithm performs N_tomo tomography updates. These tomography updates serve to “hide” the latency of the large memory transfer by performing useful work instead of waiting for the relatively slow memory transfer to finish. Section III-C discusses selecting N_tomo and other algorithm parameters.

B. Multiple-device implementation

Besides providing more computational resources, implementing a CT reconstruction algorithm on multiple GPUs can reduce the memory burden on each device. Many “distributed” algorithms either store additional variables on each node and/or perform redundant calculations to avoid very expensive inter-node communications [29]–[32]. These designs assume that communication between devices is extremely expensive. It may be tempting to view multiple-GPU programming as a “distributed” setting, but at least in CT reconstruction, frequent communication between the host computer and the GPU(s) seems necessary due to GPU memory constraints. Adding additional GPUs that regularly communicate to the host need not significantly increase the total amount of communication the algorithm performs. Instead of using a more sophisticated “distributed” algorithm framework [30], we distribute the memory and computation of the single-GPU algorithm over multiple devices in a straightforward way.

Let x and y be the transaxial axes of the image volume and z be the axial direction; i.e., z is parallel to the CT scanner’s axis of rotation. Similar to [33, Appendix E], we divide all image-sized buffers into N_device chunks transaxially, e.g., along the y direction. This approach differs from [30], [31], where the image is divided axially along z. Each device also stores two N_xN_z-pixel padding buffers. Because the image-sized buffers are the largest buffers our proposed algorithm stores on the GPU, this decomposition effectively reduces the memory burden on each device by a factor of almost N_device.

1) Tomography update

The buffer x̃ is distributed across multiple GPUs. Fortunately, the tomography update (27) is linear in x̃. When updating the group of dual variables u_g, each device projects its chunk of x̃ and sends the projection to the host computer. The host accumulates these projections, performs the update (27), and transmits the dual update ${\mathbf{u}}_g^{+}-{\mathbf{u}}_g^{-}$ back to each device. Each device backprojects the update into its chunk of the volume, updating the distributed x̃.

2) Denoising update

Every element of the dual variable v couples two pixels together. Most of these pairs of pixels lie on only one device; in these cases, the denoising update is separable and requires no additional communication between the GPUs. However, some of the elements of v couple pixels that are stored on different GPUs. Prior to performing the update for these elements, the algorithm must communicate the pixels on the GPU boundaries between devices.

Fortunately, such communication is needed for only roughly a quarter of the denoising updates. Most of the “half-difference” groups in which v is updated require no communication. For example, in Figure 1 suppose that {x₁,…, x₆} were stored on one device and {x₇,…, x₁₂} are stored on another. Updating the green group of differences {v₁₂, v₁₃, v₁₄} would require communication between the devices, but updating the cyan group {v₉, v₁₀, v₁₁, v₁₅, v₁₆, v₁₇} would not.

3) Nonnegativity update and warm- starting

The nonnegativity update and warm-stating operation are both separable in the elements of the dual variables and x̃, so implementing these operations on multiple devices is straightforward.

C. Parameter selection

There are three parameters in the algorithm listed in Figure 2: N_denoise, N_tomo and μ. This section gives the heuristics we used to set the values of these parameters.

Similar to how OS algorithms compute an approximate gradient using only a subset of the projection views, the proposed algorithm performs an outer update (i.e., increments the iteration n) after updating about N_view/N_subset views of u, chosen randomly with replacement. In an outer iteration, the algorithm also performs N_denoise half-difference denoising updates. We heuristically set N_denoise to be large enough that the expected number of outer iterations between updating an element of v is about one. Because each denoising update modifies half of the elements of v corresponding to a single direction, this means N_denoise ≈ 2N_r, where N_r is the number of directions along which the regularizer penalizes differences. For the common case of penalizing differences in 13 directions around each pixel (as in our experiments), we set N_denoise ≈ 26.

This yields the following relationship:

$$N_{\text{tomo}}=\frac{N_{\text{view}}}{2N_{\text{denoise}}{N}_{\text{subset}}}.$$ (56)

In the shoulder case below, N_view = 6852. We set N_subset = 18 and N_denoise = 23, yielding N_tomo = 8. The wide cone case had N_view = 984; we used N_denoise = 27 and N_subset = 6, thus, N_tomo = 3. A more principled method to select these parameters is future work.

We chose μ using the mean of the entries of the diagonal matrix MW, where W contains the weights in the data-fit term and M contains the entries of all the diagonal majorizers for the tomography update (28):

$$\mu =\frac{{\sum }_{i=1}^M{[\mathbf{MW}]}_{ii}}{4M}.$$ (57)

This heuristic is intended to yield a well-conditioned tomography update (27). Smaller μ would make the outer proximal majorization tighter (5) at the cost of making the dual problem (19) possibly more ill-conditioned.

IV. Experiments

This section compares the proposed algorithm to several state of the art accelerated versions of the popular OS algorithm [12]–[14]. All algorithms were implemented with the OpenCL GPU programming interface and the Rust programming language. Experiments were run on a machine with 48 GB of RAM and four aging NVIDIA Tesla C2050s with 3 GB of memory apiece. To measure how well the algorithms performed on multiple devices we ran each algorithm using 1, 2 and 4 GPUs. Preliminary experiments on an NVIDIA Kepler K5200 (see the supplementary material) indicate that all the algorithms run faster on newer hardware, but their relative speeds are unchanged.

A. Ordered subsets

OS algorithms are first-order methods that approximate the computationally expensive gradient of the data-fit term ∇L(Ax) using a subset of the data [12]–[14]. Without relaxation, this approximation can lead to divergence, but in our experiments we chose parameters that empirically led to limit cycles near the solution.

Our implementation of the OS algorithms stored the following variables on each GPU:

• the current image x,

• the coefficients of the diagonal majorizer D ⪰ A^⊤WA:

  $$\mathbf{D}=\underset{j}{\text{diag}}\left\{{[{\mathbf{A}}^{\top }\mathbf{WA}\mathbf{1}]}_j\right\},$$ (58)

• an accumulator for the current search direction,

• the regularizer parameters {β_k}, and

• an additional image-sized variable to store the momentum state, if applicable [13].

The OS methods require more image-sized buffers than our proposed algorithm. We divided these image-sized volumes across multiple GPUs transaxially, so the memory burden for each device decreases almost linearly with the number of devices. The devices must communicate pixels that lie on an edge between devices before computing a regularizer gradient; this happens N_subset times per iteration.

The OS methods also must compute the majorizer D in (58) from the patient-dependent statistical weights W before beginning to iterate. This requires a CT projection and a backprojection that takes about as much time as an iteration. We do not count this time in our experiments. On the other hand, the majorizers used by the proposed algorithm are nominally independent of patient data and depend only on the scanner geometry through ${\mathbf{A}}_g{\mathbf{A}}_g^{\top }$. Because the majorizers {M_g} in (28) can be precomputed before the scan, but the OS algorithms must compute D (58) after the scan but before beginning to iterate, the proposed algorithm could be considered be an additional iteration faster than the OS-based methods.

B. Figures of merit

We ran experiments using two datasets: a helical shoulder scan using real patient data (Section IV-C) and a wide-cone axial scan using simulated data in (Section IV-D). For both cases we measured the progress of all algorithms tested towards a converged reference x̂ using the root mean squared difference (RMSD) over an N_ROI-pixel region of interest (ROI):

$$\text{RMSD}\left({\mathbf{x}}^{(n)}\right)=\sqrt{\frac{{\Vert {\mathbf{x}}^{(n)}-\widehat{\mathbf{x}}\Vert }_{{\mathrm{M}}_{\text{ROI}}}^2}{N_{\text{ROI}}}}.$$ (59)

The diagonal binary masking matrix M_ROI discards “end slices” that are needed due to the “long object problem” in cone-beam CT reconstruction but not used clinically [34].

We compared the proposed algorithm and the OS algorithms using wall-clock time and “equivalent iterations,” or equits [7]. Because the most computationally intensive part of these algorithms is projection and backprojection, one equit corresponds to computing roughly the same number of forward and backprojections:

• For OS, one equit corresponds to a loop through all the data.

• For the proposed algorithm, one equit corresponds to N_subset iterations; e.g., x^(N_subset) and x^(2N_subset) are the outputs of successive equits. Over this time, our algorithm computes about N_view forward and backprojections.

The proposed algorithm performs more denoising updates than OS and transfers more memory between the host and GPU in an equit, so each equit of the proposed algorithm takes longer to perform. Nonetheless, as the following experiments show, the proposed algorithm converges considerably more quickly than the OS algorithms in terms of runtime.

C. Helical shoulder scan

Our first reconstruction experiment uses data from a helical scan provided by GE Healthcare. The data were 6852 views with 888 channels and 32 rows. We reconstructed the image on a 512 × 512 × 109-pixel grid. We used a weighted squared ℓ₂-norm data-fit term with weights proportional to estimates of the inverses of the measurements’ variances [1]. The regularizer penalized differences along all 13 directions (i.e., 26 3D neighbors) with the Fair potential function (39) with δ = 10 Hounsfield units (HU), and the {β_k} were provided by GE Healthcare. All iterative algorithms were initialized using the filtered backprojection image in Figure 3a. Figure 3b shows an essentially converged reference, generated by running thousands of iterations of momentum-accelerated separable quadratic surrogates [13] (i.e., OS with one subset).

Fig. 3.

Top row: initial FBP and reference images for the helical shoulder case in Section IV-C. Bottom row: images from the proposed algorithm and the state of the art OS-OGM algorithm on 4 GPUs after about 5 minutes; yellow ovals indicate regions with significant differences. Images were trimmed and displayed in a [800, 1200] modified Hounsfield unit window. Each panel shows transaxial, coronal, and sagittal slices through the 3D volume.

We ran the proposed algorithm with N_denoise = 23 and N_tomo = 8. We also ran ordered subsets with 12 subsets (OS) [12], OS with Nesterov’s momentum [13] (FGM), and OS with a faster acceleration [14] (OGM) on one, two and four GPUs. Figure 4 shows RMSD in Hounsfield units against time and equivalent iteration.

Fig. 4.

Convergence plots for the helical shoulder case in Section IV-C. Markers are placed every five equits. The stars in Figure 4c correspond to the images shown in Figures 3c and 3d. The proposed algorithm on one device converges about as quickly as the state of the art OS-OGM algorithm does on four devices. Additional devices provide further acceleration.

Figure 4a shows the proposed algorithm converging considerably faster than the OS-type algorithms in terms of equits, and unlike the OS-type algorithms will converge to a solution x̂ if the conditions in Appendix C are satisfied. Figure 4c shows that the proposed algorithm on one GPUs achieves early-iteration speed comparable to the fastest OS algorithm with four GPUs.

Table I lists several timings for the algorithms in this experiment. Although the OS algorithms achieved more dramatic speedups using multiple devices than the proposed algorithm, additional devices did help accelerate convergence. Figures 3c and 3d show images from both algorithms on four devices after about five minutes of computation. The proposed algorithm produced an image that much more closely matches the converged reference.

TABLE I.

Timings for the helical case in Section IV-C.

Algorithm	Per equit	Time to converge within
		5 HU RMSD	2 HU RMSD
OGM-1	114 sec.	21.5 min.	58.6 min.
OGM-2	62 sec.	11.7 min.	29.8 min.
OGM-4	38 sec.	7.2 min.	18.2 min.
Proposed-1	130 sec.	9.4 min.	16.2 min.
Proposed-2	82 sec.	5.5 min.	9.8 min.
Proposed-4	57 sec.	3.8 min.	6.8 min.

D. Wide-cone axial simulation

Our second experiment is a wide-cone axial reconstruction with a simulated phantom. We simulated a noisy scan of the XCAT phantom [35] taken by a scanner with 888 channels and 256 rows over a single rotation of 984 views. Images were reconstructed onto a 718 × 718 × 440-pixel grid. As in Section IV-C, we used a quadratic data-fit term, the regularizer used the Fair potential and penalized differences along all 13 neighboring directions, and the regularizer weights {β_k} were computed using [16]. All iterative algorithms were initialized with the filtered backprojection image in Figure 5a. An essentially converged reference image is shown in Figure 5b.

Fig. 5.

Top row: initial FBP and reference images for the wide-cone axial case in Section IV-D. Bottom row: images from the proposed algorithm and the state of the art OS-OGM algorithm on 4 GPUs after about 5 minutes; yellow ovals indicate regions with significant differences. Images were trimmed and are displayed in a [800, 1200] modified Hounsfield unit window. Each panel shows transaxial, coronal, and sagittal slices through the 3D volume.

This problem was too large to fit on one 3 GB GPU, so we present results for two and four GPUs. We ran the same set of OS algorithms as the previous experiment with 12 subsets. The proposed algorithm used N_denoise = 27 and N_tomo = 3.

Figures 6a and 6c show the progress of the tested algorithms towards the converged reference. The proposed algorithm running on two devices is about as fast as OS-OGM running on four devices, and additional devices accelerate convergence even more. Figures 5c and 5d show outputs from OS-OGM and the proposed algorithm after about five minutes. After five minutes, the OS algorithm still contains noticeable streaks that the dual algorithm has already removed. At this point, both algorithms have significant distance to the reference at the end slices of the image.

Fig. 6.

Convergence plots for the wide-cone axial case in Section IV-D. Markers are placed every five equits. The stars in Figure 6c correspond to the images shown in Figures 5c and 5d. The proposed algorithm converges about as quickly on two devices as OS-OGM does on four. Additional devices accelerate convergence.

Table II lists timings for OS-OGM and the proposed algorithm. The trends are similar to the smaller helical case in Table I. The OS algorithms scale better (1.7× faster) than the proposed algorithm (1.2× faster) from two to four GPUs, but the acceleration provided by the proposed algorithm is enough to compensate for lower multi-device parallelizability.

TABLE II.

Timings for the axial case in Section IV-D.

Algorithm	Per equit	Time to converge within
		5 HU RMSD	2 HU RMSD
OGM-2	85 sec.	16.3 min.	–
OGM-4	50 sec.	9.5 min.	19.9 min.
Proposed-2	126 sec.	8.6 min.	15.6 min.
Proposed-4	98 sec.	6.3 min.	11.0 min.

V. Conclusions and future work

We presented a novel CT reconstruction algorithm that uses alternating updates in the dual domain. The proposed algorithm is fast in terms of per-iteration speed and “wall clock” runtime, and it converges more quickly than state of the art OS algorithms. If the inner updates are performed with sufficient accuracy the algorithm converges to the true solution of the statistical reconstruction problem, and it can handle a wide range of regularizers including the nondifferentiable total variation regularizer.

The algorithm also maps well onto the GPU. Many of its steps are highly parallelizable and perform regular memory accesses. Although the algorithm stores many variables in the host computer’s memory, the amount of memory required for each update is relatively small, and we hide the latency of transferring variables to and from the GPU by performing computation-bounded operations. Finally, the proposed algorithm is easily adapted for multiple GPUs, providing further acceleration and decreasing the memory burden on each GPU.

Due to communication overhead, the acceleration provided by adding additional GPUs showed diminishing returns. To achieve further acceleration, multiple computers (or groups of GPUs on a single node) may need to be combined using a “distributed” algorithm framework [29], [30]. How to best adapt the proposed algorithm to these frameworks is future work.

The proposed algorithm introduces a dual variable for each difference penalized by the edge-preserving regularizer R. While this memory cost is not too great for a modern computer when regularizing the 13 neighbors around each pixel, increasing the number of differences computed may make the proposed approach infeasible. Consequently, adapting the proposed algorithm for patch-based or nonlocal regularizers may be challenging.

The random process we use for choosing which groups of the tomography dual variable u and denoising dual variable v to update is basic and almost certainly suboptimal. A more sophisticated strategy may provide additional acceleration. Different majorizers M_g for the tomography update (27) and more sophisticated methods to select the algorithm parameters N_tomo and N_denoise are other interesting areas for future work.

Appendix A. Fenchel duality for GPU-based reconstruction algorithm

Proving (16) involves a straightforward application of Fenchel’s duality theorem, see e.g., [36, Theorem 4.4.3]. Define

$$f(\mathbf{x})=\frac{\mu }{2}{\Vert \mathbf{x}-{\mathbf{x}}^{(n)}\Vert }_2^2,$$ (60)

$$\mathbf{K}=\left[\begin{array}{c}\mathbf{A}\\ \mathbf{C}\\ \mathbf{I}\end{array}\right].$$ (61)

We write the blocks of elements of Kx as [Kx]_u, [Kx]_v and [Kx]_z. Define

$$g(\mathbf{Kx})=\mathrm{L}({[\mathbf{Kx}]}_{\mathbf{u}})+\mathrm{R}({[\mathbf{Kx}]}_{\mathbf{v}})+\mathrm{N}({[\mathbf{Kx}]}_{\mathbf{z}}).$$ (62)

The value attained by the primal update problem (5), can be written in this terminology as

$$p=\underset{\mathbf{x}}{min}f(\mathbf{x})+g(\mathbf{Kx})=\underset{\mathbf{x}}{min}{\mathrm{J}}^{(n)}(\mathbf{x}).$$ (63)

The convex conjugates of f and g are [37, pg. 95]

$$f^{\ast }({\mathbf{x}}^{\ast })=\frac{1}{2\mu }{\Vert {\mathbf{x}}^{\ast }\Vert }_2^2+{({\mathbf{x}}^{\ast })}^{\top }{\mathbf{x}}^{(n)},$$ (64)

$$g^{\ast }({\mathbf{q}}^{\ast })={\mathrm{L}}^{\ast }({\mathbf{q}}_{\mathbf{u}}^{\ast })+{\mathrm{R}}^{\ast }({\mathbf{q}}_{\mathbf{v}}^{\ast })+{\mathrm{N}}^{\ast }({\mathbf{q}}_{\mathbf{z}}^{\ast }).$$ (65)

The value attained by maximizing the dual function (19) is

$$d=\underset{{\mathbf{q}}^{\ast }}{sup}-f^{\ast }(-{\mathbf{K}}^{\top }{\mathbf{q}}^{\ast })-g^{\ast }({\mathbf{q}}^{\ast })=\underset{{\mathbf{q}}^{\ast }}{sup}{\mathrm{D}}^{(n)}({\mathbf{q}}_{\mathbf{u}}^{\ast },{\mathbf{q}}_{\mathbf{v}}^{\ast },{\mathbf{q}}_{\mathbf{z}}^{\ast }).$$ (66)

Note that although (66) apparently differs from the statement in [36, Theorem 4.4.2] by a sign, the expressions are equivalent.

The domain of f is dom f = ℝ^N, and the image of dom f under K is K dom f = range K. The set over which g is continuous is cont g = {θ: θ_z > 0}.

Finally, by the Fenchel duality theorem, because

$$\mathbf{K}\text{dom}f\cap \text{cont}g\ne \varnothing ,$$ (67)

and f and g are both convex functions, p = d.

Appendix B. Equivalence of primal- and dual-based solutions

Let the value of x⁽ⁿ⁺¹⁾ produced by solving the primal update problem (5) be

$${\mathbf{x}}_p=\underset{\mathbf{x}}{\text{argmin}}\underset{\mathbf{u},\mathbf{v},\mathbf{z}}{sup}{\mathrm{S}}^{(n)}(\mathbf{x},\mathbf{u},\mathbf{v},\mathbf{z}).$$ (68)

The value of x⁽ⁿ⁺¹⁾ induced by solving the dual problem (19) is

$${\mathbf{x}}_d={\stackrel{\sim }{\mathbf{x}}}^{(n+1)}\left({\widehat{\mathbf{u}}}^{(n+1)},{\widehat{\mathbf{v}}}^{(n+1)},{\widehat{\mathbf{z}}}^{(n+1)}\right),$$ (69)

$${\stackrel{\sim }{\mathbf{x}}}^{(n+1)}(\mathbf{u},\mathbf{v},\mathbf{z})=\underset{\mathbf{x}}{\text{argmin}}{\mathrm{S}}^{(n)}(\mathbf{x},\mathbf{u},\mathbf{v},\mathbf{z}),$$ (70)

where

$$\begin{array}{l}{\widehat{\mathbf{u}}}^{(n+1)},{\widehat{\mathbf{v}}}^{(n+1)},{\widehat{\mathbf{z}}}^{(n+1)}=\underset{\mathbf{u},\mathbf{v},\mathbf{z}}{\text{argmax}}{\mathrm{D}}^{(n)}(\mathbf{u},\mathbf{v},\mathbf{z})\\ ={\mathrm{S}}^{(n)}\left({\stackrel{\sim }{\mathbf{x}}}^{(n+1)}(\mathbf{u},\mathbf{v},\mathbf{z}),\mathbf{u},\mathbf{v},\mathbf{z}\right).\end{array}$$ (71)

Our goal is to show x_p = x_d.

We proceed by contradiction. Suppose x_p ≠ x_d. Because S⁽ⁿ⁾ is strongly convex and x_d minimizes S⁽ⁿ⁾ when the dual variables are fixed at (û⁽ⁿ⁺¹⁾, v̂⁽ⁿ⁺¹⁾, z⁽ⁿ⁺¹⁾) (70),

$$\begin{array}{l}d={\mathrm{S}}^{(n)}\left({\mathbf{x}}_d,{\widehat{\mathbf{u}}}^{(n+1)},{\widehat{\mathbf{v}}}^{(n+1)},{\widehat{\mathbf{z}}}^{(n+1)}\right)\\ <{\mathrm{S}}^{(n)}\left({\mathbf{x}}_p,{\widehat{\mathbf{u}}}^{(n+1)},{\widehat{\mathbf{v}}}^{(n+1)},{\widehat{\mathbf{z}}}^{(n+1)}\right)\\ \le \underset{\mathbf{u},\mathbf{v},\mathbf{z}}{sup}{\mathrm{S}}^{(n)}({\mathbf{x}}_p,\mathbf{u},\mathbf{v},\mathbf{z})=p,\end{array}$$ (72)

contradicting p = d (see Appendix A). Thus, x_p = x_d.

Appendix C. Convergence for GPU-based reconstruction algorithm with approximate updates

If the maximizing dual variables are found exactly (i.e., if (20) holds with equality), then the proposed algorithm is a simple majorize-minimize procedure (5) and (x⁽ⁿ⁾) converges to a minimizer of the cost function [24]. Finding the exact maximizers of D⁽ⁿ⁾ is too computationally expensive, so we settle for approximate optimization. Fortunately, under conditions similar to those for other approximate-update algorithms like ADMM [21], the proposed algorithm can converge even with inexact maximization of D⁽ⁿ⁾.

Let ${\epsilon }_{\mathbf{u}}^{(n+1)},{\epsilon }_{\mathbf{v}}^{(n+1)}$ and ${\epsilon }_{\mathbf{z}}^{(n+1)}$ be the weighted error between the approximate maximizers u⁽ⁿ⁺¹⁾, v⁽ⁿ⁺¹⁾ and z⁽ⁿ⁺¹⁾ of D⁽ⁿ⁾ and the true maximizers û⁽ⁿ⁺¹⁾, v̂⁽ⁿ⁺¹⁾, ẑ⁽ⁿ⁺¹⁾:

$$\begin{array}{ll}{\epsilon }_{\mathbf{u}}^{(n)}={\Vert {\widehat{\mathbf{u}}}^{(n)}-{\mathbf{u}}^{(n)}\Vert }_{{\mathbf{AA}}^{\top }},\hfill & {\epsilon }_{\mathbf{v}}^{(n)}={\Vert {\widehat{\mathbf{v}}}^{(n)}-{\mathbf{v}}^{(n)}\Vert }_{{\mathbf{CC}}^{\top }},\hfill \\ {\epsilon }_{\mathbf{z}}^{(n)}=\Vert {\widehat{\mathbf{z}}}^{(n)}-{\mathbf{z}}^{(n)}\Vert .\hfill & \hfill \end{array}$$ (73)

Assume that we solve the dual maximization subproblem (20) well enough that these errors are summable:

$$\sum _{n=1}^{\infty }{\epsilon }_{\mathbf{v}}^{(n)}<\infty ,\sum _{n=1}^{\infty }{\epsilon }_{\mathbf{u}}^{(n)}<\infty ,\sum _{n=1}^{\infty }{\epsilon }_{\mathbf{z}}^{(n)}<\infty .$$ (74)

Let x̂⁽ⁿ⁺¹⁾ be the exact solution to the primal update problem (5). The error between the approximate update x⁽ⁿ⁺¹⁾ and the exact update, x̂⁽ⁿ⁺¹⁾, is

$$\begin{array}{l}{\epsilon }_{\mathbf{x}}^{(n)}=\Vert {\mathbf{x}}^{(n+1)}-{\widehat{\mathbf{x}}}^{(n+1)}\Vert \\ =\Vert {\stackrel{\sim }{\mathbf{x}}}^{(n+1)}\left({\mathbf{u}}^{(n+1)},{\mathbf{v}}^{(n+1)},{\mathbf{z}}^{(n+1)}\right)-{\stackrel{\sim }{\mathbf{x}}}^{(n+1)}\left({\widehat{\mathbf{u}}}^{(n+1)},{\widehat{\mathbf{v}}}^{(n+1)},{\widehat{\mathbf{z}}}^{(n+1)}\right)\Vert \\ \le \frac{1}{\mu }\left({\epsilon }_v^{(n)}+{\epsilon }_u^{(n)}+{\epsilon }_{\mathbf{z}}^{(n)}\right),\end{array}$$ (75)

using the form of the dual-induced primal solution (17) and the triangle inequality. Because the dual update errors are summable (74), the primal update errors $\left\{{\epsilon }_x^{(n)}\right\}$ are also summable. Then, by [21, Theorem 3], the proposed algorithm is a convergent “generalized proximal point algorithm” and produces a sequence of iterates {x⁽ⁿ⁾} that converge to a minimizer x̂.

In practice, it may be difficult to verify numerically that the conditions (74) hold, but at least this analysis provides some sufficient conditions for convergence. In contrast, OS algorithms [12] have no convergence theory (and can diverge even for well-conditioned problems).