A Direct Algorithm for Optimization Problems with the Huber Penalty

Abstract

We present a direct (noniterative) algorithm for one dimensional (1-D) quadratic data fitting with neighboring intensity differences penalized by the Huber function. Applications of such an algorithm include 1-D processing of medical signals, such as smoothing of tissue time concentration curves in kinetic data analysis or sinogram preprocessing, and using it as a subproblem solver for 2-D or 3-D image restoration and reconstruction. Dynamic programming (DP) was used to develop the direct algorithm. The problem was reformulated as a sequence of univariate optimization problems, for k = 1, ···, N, where N is the number of data points. The solution to the univariate problem at index k is parameterized by the solution at k + 1, except at k = N. Solving the univariate optimization problem at k = N yields the solution to each problem in the sequence using backtracking. Computational issues and memory cost are discussed in detail. Two numerical studies, tissue concentration curve smoothing and sinogram preprocessing for image reconstruction, are used to validate the direct algorithm and illustrate its practical applications. In the example of 1-D curve smoothing, the efficiency of the direct algorithm is compared with four iterative methods: the iterative coordinate descent, Nesterov’s accelerated gradient descent algorithm, FISTA, and an off-the-shelf second order method. The first two methods were applied to the primal problem, the others to the dual problem. The comparisons show that the direct algorithm outperforms all other methods by a significant factor, which rapidly grows with the curvature of the Huber function. The second example, sinogram preprocessing, showed that robustness and speed of the direct algorithm are maintained over a wide range of signal variations, and that noise and streaking artifacts could be reduced with almost no increase in computation time. We also outline the application of the proposed 1-D solver for imaging applications.

I. Introduction

Smoothing or denoising is a ubiquitous problem in one-dimensional (1-D) signal processing. This problem also finds applications in 2-D or 3-D as signal smoothing can be used as subproblem solvers for image restoration and reconstruction. Novel results on signal smoothing were recently published [1]–[4]: direct, noniterative solutions based on dynamic programming (DP) were proposed for 1-D signal smoothing with quadratic data fitting and total variation (TV) penalty. These DP-based solutions are effective for 1-D signal smoothing [1], [2]. When used as subproblem solvers, they also enable fast 2-D image processing [4], [5].

The TV penalty encourages piecewise constant signals and may not be suitable for some applications. One example is smoothing of tissue time concentration curves obtained in dynamic contrast-enhanced imaging. These signals, by their kinetic material exchange nature, are not piecewise constant but smooth with sharp transitions. Another example is sinogram smoothing [6], which preprocesses the sinogram before image reconstruction. The object can often be assumed to be piecewise constant, then the sinogram is smooth with sharp transitions at organ boundaries. Thus there is a need to consider alternative regularization functions that encourage such signal properties. We propose extensions of the previous results [1]–[4] for this purpose. Specifically, we consider quadratic data fitting with the Huber function instead of TV to penalize neighboring intensity differences.

The utility of the Huber penalty in imaging applications is extensive. It can serve as a generic edge preserving penalty [7]–[10], or as an approximation to the nondifferentiable TV penalty [11]–[13] to deal with the computation challenge, or as an effective means of reducing staircasing or the patchy artifacts of the TV penalty [14], [15]. These published works motivate the search for a dedicated solver for problems involving the Huber penalty. Our development is a first step in this direction.

The objective function we seek to minimize is:

$$\mathrm{\Theta }(\mathit{x})\triangleq \sum _{k=1}^N\frac{w_k}{2}{(y_k-x_k)}^2+\sum _{k=1}^{N-1}{\beta }_{k}H(x_k-x_{k+1},{\delta }_k),$$ (1)

where x ≜ {x₁, ···, x_N} is the unknown, N is the signal length, and {y₁, ···, y_N} is the noisy input signal. The weight, w_k > 0, in the data fitting term is typically the inverse variance of the data y_k. The penalty term involves the Huber function with parameter δ ≥ 0, namely

$$H(s,\delta )=\{\begin{array}{ll}\frac{s^2}{2\delta }\hfill & \mid s\mid \le \delta \hfill \\ \mid s\mid -\frac{\delta }{2}\hfill & \mid s\mid >\delta \hfill \end{array}$$ (2)

The parameter δ controls the transition between the quadratic region (|s| < δ) and the linear region (|s| ≥ δ). As δ tends to zero, H(·, δ) approaches the absolute value, |·|, so that problem (1) becomes 1-D quadratic fitting with TV penalty, which was discussed in previous publications [1]–[4], except that here we further allow pixel-dependent weights in the data fitting term as well as in the penalty term, denoted as w_k and β_k > 0, respectively.

Iterative algorithms can minimize (1). Indeed, the performance of the direct TV algorithm [1] was compared with FISTA [16] as applied to the dual problem of 1-D TV smoothing. It is simple to derive that the dual problem to (1), for δ_k ≠ 0 as well as δ_k = 0, is a least squares problem with interval constraints. Then not only are the first order methods such as FISTA applicable, but also the second-order methods such as the active-set method [17]. Also, unlike the TV special case, the Huber penalty makes the (primal) objective (1) a smooth convex function, thus both the gradient descent and the coordinate descent algorithms can be applied. In fact, it can be shown that the Huber penalty is a special case of the smoothing technique in [18] for dealing with the nonsmooth TV term, so that the accelerated gradient descent (AGD) method [19] can be used to solve (1).

We emphasize that the existing algorithms to solve (1) are all iterative in nature. The number of iterations will depend on the parameters of the problem, i.e., β_k, δ_k, and the required accuracy of a solution. In contrast, we propose in this work a direct, noniterative algorithm to compute the exact solution to (1), and we demonstrate using extensive numerical comparisons the algorithm’s high computational efficiency.

The rest of the paper is organized as follows. Sec II describes the solution to problem (1). In Sec III we introduce four methods (two for solving the primal problem, two for the dual) used for efficiency comparison. Sec IV presents numerical results of applying the direct algorithm to 1-D curve smoothing and sinogram smoothing. Sec V discusses potential future extensions of the 1-D algorithm, including pathways to 2-D (3-D) imaging applications. Sec VI concludes the paper.

II. A DP-based direct algorithm

Since the objective function (1) is strong convex, the existence and uniqueness of a global minimum is guaranteed. As mentioned earlier, we adopted a dynamic programming (DP) approach to find the minimum of (1). An overview of this approach is first given for readers who are not familiar with DP. Next, the solution and the associated algorithm are given in detail, including an analysis of the relationship with 1-D quadratic fitting with TV penalty,

A. Overview

DP transforms the problem of minimizing (1) into a sequence of (simpler) minimization problems [20], [21]. First, we introduce some short hand notation for convenience. Let the unary terms in (1) be E_k(x_k) ≜ w_k(y_k − x_k)²/2, and H_k(x_k − x_k+1) ≜ H(x_k − x_k+1, δ_k), Expanding all terms in (1), we have

$$\begin{gathered} \underset{\mathit{x}}{min}\mathrm{\Theta }(\mathit{x})\equiv \underset{x_1,x_2,\cdots x_N}{min}\{E_1(x_1)+{\beta }_1{H}_1(x_1-x_2) \\ +E_2(x_2)+{\beta }_2{H}_2(x_2-x_3)+\cdots \\ +E_{N-1}(x_{N-1})+{\beta }_{N-1}{H}_{N-1}(x_{N-1}-x_N) \\ +E_N(x_N)\}. \end{gathered}$$ (3)

Since x₁ only appears in the first two terms of (3), its influence on the solution can be summarized by defining a new function, Φ₂, where

$${\mathrm{\Phi }}_2(x_2)\triangleq \underset{x_1}{min}\{E_1(x_1)+{\beta }_1{H}_1(x_1-x_2)\}.$$ (4)

Due to the interaction between x₁ and x₂ from the Huber function, the solution to (4), $x_1^{\ast }$, depends on x₂ and so does the function value Φ₂. We can use (4) to eliminate all occurrences of x₁ in (3), which becomes

$$\underset{\mathit{x}}{min}\mathrm{\Theta }(\mathit{x})\equiv \underset{x_2,\cdots x_N}{min}\{{\mathrm{\Phi }}_2(x_2)+E_2(x_2)+{\beta }_2{H}_2(x_2-x_3)+\cdots +E_{N-1}(x_{N-1})+{\beta }_{N-1}{H}_{N-1}(x_{N-1}-x_N)+E_N(x_N)\},$$ (5)

Doing so, we have reduced the original N-variable minimization problem of equation (3) to a minimization problem with N − 1 variables. This procedure can be repeated by introducing

$${\mathrm{\Phi }}_{k+1}(x_{k+1})=\underset{x_k}{min}\{{\mathrm{\Phi }}_k(x_k)+E_k(x_k)+{\beta }_k{H}_k(x_k-x_{k+1})\}$$ (6)

for k = 1, ···, N − 1, each time reducing the problem size by one. For consistency, we let Φ₁(x₁) ≡ 0 in (4). At the last step, we have the following univariate minimization problem

$$\underset{\mathit{x}}{min}\mathrm{\Theta }(\mathit{x})\equiv \underset{x_N}{min}\{{\mathrm{\Phi }}_N(x_N)+E_N(x_N)\}.$$ (7)

Thus the solution to (3) is equivalent to solving (a) the sequence of univariate minimization problems in (6), whose solution depends on the “future” variable x_k+1, and (b) the terminal problem, (7). Both (a) and (b) admit closed-form solutions because the minimization problems only involve piecewise quadratic convex functions. This claim can be established by induction after studying the solution to (6).

B. Solution to (6)

To reduce notation clutter, we first rewrite (6) as the following generic problem, which omits dependence on k:

$$\mathrm{\Psi }(t)=\underset{s}{min}\mathrm{\Omega }(s,t)$$ (8a)

$$\begin{gathered} \text{where}\mathrm{\Omega }(s,t)\triangleq \underset{\_}{\mathrm{\Phi }(s)+E(s)}+\beta H(s-t) \\ =\underset{\_}{Q(s)}+\beta H(s-t) \end{gathered}$$ (8b)

The unary function E(s) ≜ w(y − s)²/2 is quadratic. We also define Q(s) ≜ Φ(s) + E(s), and implicitly H(·) ≡ H(·, δ).

It is easy to see that if Φ(s) is convex, then Ω(s, t) is (strongly) convex in (s, t). From convex analysis, Ψ(t) in (8a) is convex in t; see e.g. [22, page 51] or [23, page 87]. Also, there exists a unique minimizer s^* such that

$$s^{\ast }=arg\underset{s}{min}\{Q(s)+\beta H(s-t)\}$$ (9)

Note that s^* is a function of t, but this dependence is not written explicitly.

Assume for now that Φ is continuously differentiable, convex, and piecewise quadratic with a finite number, B, of knots. Its gradient, ϕ, is then piecewise linear, continuous, and non-decreasing, which implies that ϕ is fully characterized by its knots: ( $s_i^b,\varphi (s_i^b)$), i = 1, ···, B. That is,

$$\varphi (s)=\frac{s_{i+1}^b-s}{s_{i+1}^b-s_i^b}\varphi (s_i^b)+\frac{s-s_i^b}{s_{i+1}^b-s_i^b}\varphi (s_{i+1}^b)s_i^b\le s<s_{i+1}^b$$

Let the gradient of Q(s) be q(s), then q(s) is also piecewise linear and continuous with knots at locations $s_i^b$ and $q(s_i^b)=\varphi (s_i^b)+w(s_i^b-y)$, i = 1 ··· B. Moreover, q(s) is increasing and unbounded, i.e., q(s) → −∞ as s → −∞, and q(s) → ∞ as s → ∞. Similar to ϕ(s), the value of q(s) at any s can be obtained using a linear interpolation formula.

The solution (9) satisfies

$$0=q(s^{\ast })+\beta h(s^{\ast }-t,\delta )$$ (10)

where h(·) is the gradient of the Huber function (2), i.e.,

$$h(s)=\{\begin{array}{ll}1\hfill & s>\delta \hfill \\ s/\delta \hfill & \mid s\mid \le \delta \hfill \\ -1\hfill & s<-\delta \hfill \end{array}$$

Note that, as with H(·), the dependence of h on δ is not written explicitly in this section. Since h is bounded between ±1, q(s^*) ∈ [−β, β] in (10). As q(s) is piecewise linear, strictly increasing, and unbounded, there exists s⁻ and s⁺, and s⁻ < s⁺, such that

$$q(s^{+})=\beta ,$$ (11a)

$$q(s^{-})=-\beta$$ (11b)

and s^* ∈ [s⁻, s⁺]. Whereas s^* depends on t, s⁻ and s⁺ are independent of t since they are determined by q(s) only.

A closed-form expression for s^* can be obtained by considering t in three cases as illustrated in Fig. 1.

Fig. 1.

The three cases for the solution s^* as a function of t. The dashed (brown) ramp represents βh(s − t, δ). The solid piecewise linear function represents q(s). The solution s^* (vertical dash-dot blue line) is such that q(s^*) + βh(s^* −t, δ) = 0. The point s₀ is where q(s₀) = 0.

1. If t − δ ≥ s⁺, then s^* = s⁺. As s^* remains a constant, and s⁺ − t ≤ −δ, Ψ(t) = Q(s⁺) + βH(s⁺ − t) is a linear function of t with gradient equal to β.

2. If t + δ ≤ s⁻, s^* = s⁻ and Ψ(t) = Q(s⁻) + βH(s⁻ − t) is linear in t, with gradient equal to −β.

3. If s⁻ − δ < t < δ + s⁺, then s^* ∈ (t − δ, t + δ), and q(s^*) ∈ (−β, β). Thus, the Huber function operates in the quadratic region, and s^* satisfies [Fig. 1(c)]

   $$q(s^{\ast })+\beta (s^{\ast }-t)/\delta =0,$$ (12)
   which is equivalent to

   $$t=s^{\ast }+\frac{\delta }{\beta }q(s^{\ast }),$$ (13a)

   $$s^{\ast }={\left[1+\frac{\delta }{\beta }q\right]}^{-1}t$$ (13b)
   Given the properties of q, s^* is a piecewise linear, continuous and increasing function of t; and the knots of s^* are at positions

   $$t_i^b=s_i^b+\frac{\delta }{\beta }q(s_i^b)\forall i\text{s.t.}{s}^{-}<s_i^b<s^{+}.$$ (14)
   Now we evaluate

   $$\mathrm{\Psi }(t)=Q(s^{\ast })+\beta {(s^{\ast }-t)}^2/(2\delta ),$$ (15)
   using its gradient ψ(t). By the chain rule,

   $$\psi (t)=\frac{\partial \mathrm{\Psi }}{\partial s^{\ast }}\frac{d{s}^{\ast }}{dt}+\frac{\partial \mathrm{\Psi }}{\partial t}.$$ (16)
   Since ∂Ψ/∂s^* = 0 from (12), we have

   $$\psi (t)=-\frac{\beta }{\delta }(s^{\ast }-t)$$ (17a)

   $$\stackrel{(12)}{=}q(s^{\ast }).$$ (17b)
   Thus ψ(t) is a piecewise linear, continuous and increasing function of t with knots at $t_i^b$ (14). As t increases from s⁻ − δ to s⁺ + δ, ψ(t) increases from −β to β. Combining (17a) with (14), we get the gradient value at the knots:

   $$\psi (t_i^b)=-\frac{\beta }{\delta }(s_i^b-t_i^b)=q(s_i^b).$$ (18)

   (18) and (14) characterize the piecewise linear ψ(t).

Summarizing key points from all three cases, we have s^* = s⁻ when t ≤ s⁻ − δ and s^* = s⁺ when t ≥ s⁺ + δ; otherwise, s^* is a piecewise linear function of t. Correspondingly, Ψ(t) is linear for t ≤ s⁻ − δ and t ≥ s⁺ + δ, and piecewise quadratic otherwise. Also, Ψ(t) is continuously differentiable and convex. Its gradient ψ(t) is bounded below by −β and above by β; otherwise it is piecewise linear and continuous. Combine (13) and (17), we have

$$\psi {(t)\mid }_{t=s+\frac{\delta }{\beta }q(s)}=q(s),s^{-}<s<s^{+}$$ (19)

Thus ψ(t) is obtained by clipping q(s) at ±β, and relabeling the axis (and the knots) according to (19), a stretching operation anchored at the point s₀ where q(s₀) = 0. These key points are depicted in Fig. 2. Further, If ϕ(s) has B knots, then ψ(t) has at most B + 2 knots, two of which are at (s⁻ −δ, −β) and (s⁺ + δ, β), whereas the others, ( $t_i^b,\psi (t_i^b)$), are transferred from the knots ( $s_i^b,q(s_i^b)$) by (14) and (18).

Fig. 2.

(a) The solution s^*(t) is a piecewise linear, non-decreasing function of t. The vertical distance between the diagonal line s = t and s^*(t) is equal to δq(s^*)/β or δψ(t)/β. See (13) and (18). The point s₀ is such that q(s₀) = 0. As δ → 0, the distance between the diagonal line s = t and the multi-linear section between s⁻ − δ and s⁺ + δ would reduce to 0. (b) The function ψ(t) is obtained from q(s) by (i) clipping it at (s^±, ±β) and (ii) relabeling the axis; ψ(t) is bounded between ±β.

C. Algorithm description

The analysis of Sec II-B can be repeatedly applied to (6), by making the substitutions for each k = 1, ··· N − 1.

$$\begin{array}{rr}\hfill s:=x_k,s^{\ast }:=x_k^{\ast },& \hfill t:=x_{k+1},\\ \hfill \mathrm{\Phi }(s):={\mathrm{\Phi }}_k(x_k),& \hfill \mathrm{\Psi }(t):={\mathrm{\Phi }}_{k+1}(x_{k+1}),\\ \hfill \varphi (s):={\varphi }_k(x_k),& \hfill \psi (t):={\varphi }_{k+1}(x_{k+1}),\end{array}$$

starting from Φ₁(·) ≡ 0 which is continuously differentiable, convex, and piecewise quadratic with a finite number of knots. We also define $x_k^{\pm }$ and q_k(x_k) such that

$$\begin{gathered} q_k(x_k^{+})={\beta }_k,q_k(x_k^{-})=-{\beta }_k, \\ {q}_k(x_k)={\varphi }_k(x_k)+w_k(x_k-y_k). \end{gathered}$$

Thus, for all k = 1, ···, N, Φ_k is a continuous, convex, and piecewise quadratic function, whose gradient is bounded between [−β_k, β_k] and piecewise linear with knots ( $x_k^{b,i},{\varphi }_k(x_k^{b,i})$).

The DP-based algorithm consists of two sweeps, a forward sweep that builds the list of knots ( $x_k^{b,i},{\varphi }_k(x_k^{b,i})$), k = 1, ··· N, as the sequence continues, and a backward sweep to trace back values of $x_k^{\ast }$, k < N once $x_N^{\ast }$ is obtained at the end of the forward sweep. Now we discuss in more detail the two sweeps.

Assume we know ( $x_k^{b,i},{\varphi }_k(x_k^{b,i})$), which are the knots for ϕ_k(x_k). We obtain ( $x_{k+1}^{b,i},{\varphi }_{k+1}(x_{k+1}^{b,i})$) by applying two steps to the knots of ϕ_k(x_k), (1) trimming and (2) transfer, and by inserting two knots coming from q_k(s) = ±β_k. In the trimming step, we identify and retain the knots of ϕ_k(x_k) for which $s=x_k^{b,i}$ yields q_k(s) = ϕ_k(s) + w_k (s − y_k) ∈ (−β_k, β_k), see Fig. 3. In the transfer step, we use (14) and (18) to transfer the retained knots $x_k^{b,i}$ to $x_{k+1}^{b,i}$, and $x_k^{\pm }$ to $x_k^{\pm }\pm {\delta }_k$, and also copy the knot’s function value $q_k(x_k^{b,i})$ to ${\varphi }_{k+1}(x_{k+1}^{b,i})$.

Fig. 3.

We assume that ϕ_k has five knots (filled red diamond markers). Depending on the data related term w_k(x_k − y_k), some knots may be removed (trimmed) from q_k(x_k) since we are only concerned in the range of x_k such that |q_k(x_k)| ≤ β_k. In this example, the two knots at $x_k^{b,1}$ and $x_k^{b,2}$ are removed (open face red diamond markers); two new knots (filled green square markers) are inserted at the boundary ±β_k.

At k = N, we need the solution to (7), i.e., $x_N^{\ast }$ such that $q_N(x_N^{\ast })=0={\varphi }_N(x_N^{\ast })+w_N(x_N^{\ast }-y_N)$. Since q_N(·) is piecewise linear, continuous and increasing, we can obtain the exact $x_N^{\ast }$ by searching for the zero-crossing linear section of q_N(x_N) and linear interpolation. We will also need ${\varphi }_N(x_N^{\ast })$ to initiate the backtracking process, which is obtained as ${\varphi }_N(x_N^{\ast })=-w_N(x_N^{\ast }-y_N)$.

Once $x_N^{\ast }$ is obtained, we trace back values $x_k^{\ast }$, k = N − 1, ··· 1 following the three cases in Sec II-B.

1. If $x_{k+1}^{\ast }-{\delta }_k\ge x_k^{+}$, then $x_k^{\ast }=x_k^{+}$, and since ${\beta }_k=q_k(x_k^{+})={\varphi }_k(x_k^{+})+w_k(x_k^{+}-y_k)$ we have

   $${\varphi }_k(x_k^{\ast })={\beta }_k-w_k(x_k^{\ast }-y_k).$$ (20)
2. If $x_{k+1}^{\ast }+{\delta }_k\le x_k^{-}$, then $x_k^{\ast }=x_k^{-}$, and since $-{\beta }_k=q_k(x_k^{-})={\varphi }_k(x_k^{-})+w_k(x_k^{-}-y_k)$, we have

   $${\varphi }_k(x_k^{\ast })=-{\beta }_k-w_k(x_k^{\ast }-y_k).$$ (21)
3. If $x_k^{-}-{\delta }_k<x_{k+1}^{\ast }<{\delta }_k+x_k^{+}$, then from (17)

   $$x_k^{\ast }\stackrel{(17\mathrm{a})}{=}{x}_{k+1}^{\ast }-\frac{{\delta }_k}{{\beta }_k}{\varphi }_{k+1}(x_{k+1}^{\ast })$$ (22)

   $$\begin{gathered} {\varphi }_k(x_k^{\ast })=q_k(x_k^{\ast })-w_k(x_k^{\ast }-y_k) \\ \stackrel{(17\mathrm{b})}{=}{\varphi }_{k+1}(x_{k+1}^{\ast })-w_k(x_k^{\ast }-y_k) \end{gathered}$$ (23)

To summarize, the backtracking of $x_k^{\ast }$ is

$$x_k^{\ast }=\{\begin{array}{ll}{x}_k^{-},\hfill & x_{k+1}^{\ast }<x_k^{-}-{\delta }_k\hfill \\ F_{k+1}(x_{k+1}^{\ast })\hfill & x_k^{-}-{\delta }_k\le x_{k+1}^{\ast }\le x_k^{+}+{\delta }_k\hfill \\ x_k^{+},\hfill & x_k^{+}+{\delta }_k<x_{k+1}^{\ast }\hfill \end{array}$$ (24)

where $F_{k+1}(u)=u-\frac{{\delta }_k}{{\beta }_k}{\varphi }_{k+1}(u)$. We recursively apply (24) until k = 1. At the same time, we also need to calculate ${\varphi }_k(x_k^{\ast })$.

D. A special case: TV smoothing

When δ_k = 0, (1) reduces to 1-D smoothing with quadratic data fitting and a TV penalty. This special case simplifies the analysis and algorithm implementation as there is a simpler relation for the knots transfer. Considering δ_k = 0 and Fig. 1, we need to make some adjustment to the analysis in Sec II-B as follows.

1. If $x_{k+1}\ge x_k^{+}$, then $x_k^{\ast }=x_k^{+}$,

   $${\mathrm{\Phi }}_{k+1}(x)=Q(x_k^{+})+{\beta }_k(x_{k+1}-x_k^{+})$$

   which is linear with slope β_k.

2. If $x_{k+1}\le x_k^{-}$, then $x_k^{\ast }=x_k^{-}$, and

   $${\mathrm{\Phi }}_{k+1}(x)=Q(x_k^{-})+{\beta }_k(x_k^{-}-x_{k+1})$$

   which is linear with slope −β_k.

3. If $x_k^{-}<x_{k+1}<x_k^{+}$, then $x_k^{\ast }=x_{k+1}$ as this value gives $q_k(x_k^{\ast })\in (-{\beta }_k,{\beta }_k)$ and $q_k(x_k^{\ast })+{\beta }_k\partial \mid x_k^{\ast }-x_{k+1}\mid =0$, where ∂|· −x_k+1| is the subgradient of the absolute value function evaluated at $x_k^{\ast }$. As $x_k^{\ast }=x_{k+1}$, we have

   $${\mathrm{\Phi }}_{k+1}(x_{k+1})={\mathrm{\Phi }}_k(x_{k+1})+E_k(x_{k+1}),$$ (25)

   From (25), the knot locations of Φ_k+1(x_k+1) are exactly those of Φ_k(x_k), i.e., $x_{k+1}^{b,i}=x_k^{b,i}$.

Summarizing all three cases, the solution $x_k^{\ast }$ is again a piecewise linear function of x_k+1, but it contains exactly 3 linear sections: (1) $x_k^{\ast }=x_k^{-}$, (2) $x_k^{\ast }=x_{k+1}$, and (3) $x_k^{\ast }=x_k^{+}$. Using Fig. 2(a) as a reference, the complicated multi-section middle portion is compressed to overlap with the diagonal line s = t as δ → 0. This observation has implications for the difference in the worst case memory cost and computational complexity between the Huber penalty and the TV penalty.

During the forward sweep, we build the list of knots by

$$x_{k+1}^{b,i}=x_k^{b,i},\text{where}{x}_k^{-}<x_k^{b,i}<x_k^{+}$$ (26)

$${\varphi }_{k+1}(x_{k+1}^{b,i})={\varphi }_k(x_k^{b,i})+w_k(x_k^{b,i}-y_k)=q_k(x_k^{b,i})$$ (27)

and by inserting the two incoming knots at $x_k^{-}$ and $x_k^{+}$. Note that (26) can be obtained from (14) by setting δ_k = 0, and (27) is the same as (18).

To obtain the solution, we need all knots of ϕ_N (x_N) to search for $x_N^{\ast }$ such that $q_N(x_N^{\ast })={\varphi }_N(x_N^{\ast })+w_N(x_N^{\ast }-y_N)=0$. Afterwards, tracing back values $x_k^{\ast }$, for k = N − 1, ···, 1 is much simpler as we have

$$x_k^{\ast }=\{\begin{array}{ll}{x}_k^{-},\hfill & x_{k+1}^{\ast }<x_k^{-}\hfill \\ x_{k+1}^{\ast },\hfill & x_k^{-}<x_{k+1}^{\ast }\le x_k^{+}\hfill \\ x_k^{+},\hfill & x_k^{+}<x_{k+1}^{\ast }\hfill \end{array}$$ (28)

This special case discussion is in essence Algorithm-1 in [2], except that here we consider weighted data fitting terms and possibly pixel-dependent TV parameter β_k.

Comparing (28) and (24), we notice that by setting δ_k = 0 in (24), the DP algorithm for the Huber penalty immediately reduces to the DP algorithm for TV penalty. This was used to generate the numerical results that are based on TV in Sec IV.

E. Implementation, memory and computational cost

One way to implement (24) is to combine it with (20),(21) and (23). To do this, we need to keep track of (a) the set of knots ( $x_k^{b,i},{\varphi }_k(x_k^{b,i})$) at k = N for searching $x_N^{\ast }$, and (b) the boundary values, $x_k^{-}$ and $x_k^{+}$, for k = 1, ··· N − 1. In the worst case, the number of knots grows linearly in k. Thus the worst case memory requirement for implementing (24) is O(N), the same as for the TV penalty [2], [4].

However, we have observed that the implementation just described is not always numerically stable, especially for large values of δ_k. A possible reason can be inferred from the recursion (23) as follows. Combine (22) with (23) to eliminate $x_k^{\ast }$, then (23) is equivalent to

$${\varphi }_k(x_k^{\ast })=(1+{\delta }_k\frac{w_k}{{\beta }_k}){\varphi }_{k+1}(x_{k+1}^{\ast })-w_k(x_{k+1}^{\ast }-y_k).$$ (29)

In (29), the multiplicative factor of ${\varphi }_{k+1}(x_{k+1}^{\ast })$ is larger than 1 for δ_k > 0, thus any round-off error of ${\varphi }_{k+1}(x_{k+1}^{\ast })$ will grow as k decreases, which appears consistent with our observations.

To avoid the numerical stability issue, we did not use (23) to back-track ${\varphi }_k(x_k^{\ast })$. Instead, after obtaining $x_k^{\ast }$ using (22), we use linear interpolation to calculate ${\varphi }_k(x_k^{\ast })$ from the knots locations ( $x_k^{b,i},{\varphi }_k^{b,i}(x_k^{b,i})$). This method was numerically validated to be stable and was used to produce all results in this paper. But this numerical stability comes at a memory cost. Instead of the worst case O(N) memory cost, we now have a worst case O(N²) memory cost, corresponding to keeping all knots in memory: ( $x_k^{b,i},{\varphi }_k^{b,i}(x_k^{b,i})$) for all k = 1, ··· N. For TV smoothing, since δ_k = 0, and since the recursion (28) does not need to keep track of ${\varphi }_k(x_k^{\ast })$, this issue of numerical instability does not exist.

In practice, as we only need to consider the range $x_k\in [x_k^{-},x_k^{+}]$ for which q_k(x_k) ∈ [−β_k, β_k], the number of knots does not necessarily increase with the sequence length (the trimming step, Fig 3). Thus, the empirical memory cost will be usually much smaller than O(N²). For instance, the maximum number of knots (for all k) in the example of Fig. 4 was 4 even though N = 400.

Fig. 4.

Tissue concentration curve smooching. (a) Huber smoothing, β_k = 0.1 and δ_k = 0.01 for all k = 1, ··· N − 1. (b) TV smoothing (δ_k = 0) generated a piecewise constant result, which is not physiologically reasonable. The insets in (a) and (b) zoom in to the time range [4, 7] min. (c) and (d) are comparisons of accuracy versus computation cost in terms of (c) iteration numbers and (d) wall clock time for the four methods, ICD, FISTA applied to the dual problem (FISTA-d), accelerated gradient descent (AGD), and Matlab’s active set applied to the dual problem. The vertical axis, maximum absolute difference (ε), is measured with respect to the DP solution. The three methods, ICD, FISTA-d, and AGD, were stopped at ε = 10⁻¹⁰, thus they all are located near the top of the figure. Each cluster of symbols represents 10 noise realizations. The dashed line in (c) is a symbolic representation of DP at one iteration; and the dashed line in (d) marks the average runtime of DP (0.012 sec).

Now we discuss computational cost. Defining an operation as either a multiplication, an addition, or a comparison, we estimate the computational cost to be O(Σ_k B_k), where B_k is the number of knots at each index k = 1, …, N. This is independent of the method used to back-track ${\varphi }_k(x_k^{\ast })$. In the worst case when B_k grows linearly, the cost can thus be O(N²). However, just as with the memory cost discussion, in practice, B_k may often remain small and essentially independent of N, as observed in our numerical studies, in which case the effort becomes O(N).

III. Methods for comparison

In Sec IV-A, our direct algorithm is compared with four iterative methods for efficiency evaluation. Two methods were used to solve the primal problem, namely iterative coordinate descent (ICD) and Nesterov’s accelerated gradient descent (AGD) [19], and two for the dual problem. When δ_k > 0, ∀k, the objective function (1) is convex and smooth, thus both ICD and accelerated gradient algorithms converge to the global minimum [24], [25]. To apply Nesterov’s method, the Lipschitz constant of the gradient of the objective (1) was estimated as L_p = ||{w}_d + D^T {β/δ}_dD||₂, where

$${\{\mathit{w}\}}_d=\text{diag}\{w_1,\cdots ,w_N\}$$ (30)

$${\{\mathit{\beta }/\mathit{\delta }\}}_d=\text{diag}\{{\beta }_1/{\delta }_1,\cdots ,{\beta }_{N-1}/{\delta }_{N-1}\}$$ (31)

and D is the finite difference operator, i.e.,

$$D\mathit{x}={[x_1-x_2,\cdots ,x_{N-1}-x_N]}^T$$ (32)

for x = {x₁, ···, x_N}. We point out that the smaller the smoothing constant δ (if δ_k = δ > 0), the larger the Lipschitz constant L_p, which reduces the step size and also the convergence speed.

To obtain the dual problem to (1), we start from the following equivalent expression for the Huber function (2),

$$H(s,\delta )=\underset{t}{inf}\left\{\mid t\mid +\frac{{(s-t)}^2}{2\delta }\right\}=\underset{\mid t\mid \le 1}{max}\{st-\frac{\delta }{2}{t}^2\}.$$ (33)

The first equality in (33) can be obtained by direct evaluation from the minimizer (a soft-thresholding on s), see, e.g., [26]; the second equality in (33) can be derived using the conjugate function of the Moreau envelope (the Moreau-Yosida regularization) [27, page 135] of the function |·|.

We plug (33) into the objective function (1), switch the order of minimization and maximization, and upon minimizing with respect to x, we obtain the following dual minimization problem:¹

$$min\{\mathrm{\Psi }(\mathit{u})=\frac{1}{2}{\mathit{u}}^T({\{\mathit{\beta }/\mathit{\delta }\}}_d^{-1}+D{\{\mathit{w}\}}_d^{-1}{D}^T)\mathit{u}-{\mathit{u}}^{T}D\mathit{y}\}\text{subject}\text{to}\mid u_k\mid \le {\beta }_k,\forall k=1,\cdots ,N-1.$$ (34)

and u = {u₁, ···, u_N−1}, y = {y₁, ···, y_N}. The primal optimal x^* and the dual optimal u^* are related by

$${\mathit{x}}^{\ast }=\mathit{y}-{\{w\}}_d^{-1}{D}^T{\mathit{u}}^{\ast }$$ (35)

The problem (34) is a least squares problem with simple interval constraints. Thus, second order algorithms such as those in quadratic programming are applicable. For the numerical examples in IV-A, we applied Matlab implementation (R2015b) of lsqlin, a built-in function for constrained least squares minimization. We compared the three available algorithm options trust region, interior point and active set and found that the active set option (with analytical gradient and Hessian) consistently performed the best. This option executes the quadratic programming active-set algorithm [17], [28] that at each iteration solves an unconstrained least squares problem. In addition to applying this off-the-shelf algorithm, we also implemented FISTA [16] to solve (34). The required Lipschitz constant was estimated as $L_d={\Vert {\{\mathit{\beta }/\mathit{\delta }\}}_d^{-1}+D{\{w\}}_d^{-1}{D}^T\Vert }_2$.

To initialize the two primal algorithms (ICD and Nesterov’s AGD), we set x₀ = y, i.e., the initial value was the same as the input; to initialize the dual algorithms, we set u₀ = 0, a vector of all zeros which is a feasible solution to (34). From (35), these two settings are compatible.

IV. Numerical results

We consider two example applications of the direct 1-D algorithm, tissue concentration curve smoothing in dynamic contrast-enhanced MR studies and sinogram smoothing, which are both effective preprocessing steps to improve subsequent kinetic data analysis or image reconstruction.

A. Smoothing of 1-D signal

The noisy tissue curves were generated using a parametric model for brain kinetic data analysis [29]. The signal length was N = 400. The curves were denoised using w_k = 1, β_k = 0.1, and δ_k = δ for all k. We experimented with four δ values, 0.1, 10⁻², 10⁻³, and 10⁻⁴. For performance evaluation, we generated 10 noise realizations for each δ.

Fig. 4 (a) shows a sample denoising result from one noise realization when δ = 0.01. Comparing with the TV smoothing result in Fig 4(b), i.e., δ = 0, the characteristic staircase artifact in Fig. 4(b) is absent in Fig. 4(a), which is smooth and physiologically more reasonable.

We use the direct DP solution to assess convergence speed of the four methods in Sec III. For the three in-house methods, ICD, Nesterov’s AGD, FISTA applied to the dual problem (FISTA-d), we used the maximum absolute difference (MAD) over all data points ε between the iterative solution and the direct DP solution as a stopping criterion, and recorded the number of iterations and computation time when ε reached below 10⁻¹⁰. For Matlab implementation of the lsqlin algorithm with active set option, we set maximum iteration number to be 100, and other tolerance settings to default values. We then calculated the MAD between the output of the algorithm and the direct DP solution and report the computation time and iteration numbers.

Fig. 4(c) and (d) show the MAD versus iteration numbers (c) and wall clock time (d) for each combination of δ and the iterative method, with clustered symbols representing the 10 noise realizations. The wall clock time results take into account the complexity of the four algorithms, which is critical here since second order methods are computationally more costly per iteration than first order methods.

Fig. 4(c) shows that the second order active set method requires far fewer iterations than the other three methods to reach an accuracy of 10⁻¹⁵. For δ = 0.1, there are problem instances such that the active set method converged in one iteration. Moreover, the convergence of all four methods is affected by δ: the smaller the value of δ, the more iterations it takes to converge.

As shown in Fig. 4(d), the advantage of the active set method is much diminished once the computational complexity is taken into account, as one iteration of the active set method (0.1 sec) is already as costly as hundreds of iterations of FISTA. For reference, the vertical dashed line in Fig. 4(d) marks the runtime for the DP algorithm (0.012 sec), which is more than 8 times faster than one iteration of active set. For δ = 0.01, which was used to create the result in (a), the fastest method out of the four turns out to be FISTA-d. Depending on the noise realization, this method is still 9–15 times slower than the DP algorithm. It is also obvious from Fig. 4(c) and (d) that ICD is competitive with FISTA or AGD if measured by number of iterations, but its sequential update nature makes it unfavorable in terms of runtime.²

As a measure of the memory cost of the DP algorithm, we looked at how the number of knots B_k varied with the sequence index k. In general, the number of knots depends on the parameter values, i.e., δ and β. For the Huber smoothing result in Fig. 4(a), the progression of B_k as a function of the sequence index k is plotted in Fig. 5. Due to the trimming and knots removal, the number of knots does not necessarily increase as the sequence progresses. The maximum number of knots in this example was 4.

Fig. 5.

The number of knots B_k as a function of the sequence index k for the DP result in Fig. 4(a). The maximum number of knots is 4. In general, the number of knots depends on the values of δ and β.

B. Sinogram smoothing

The second example was sinogram smoothing followed by image reconstruction. The fan-beam sinograms of a physical phantom obtained for a previous study [30], [31] were reused here. The data were acquired on a Siemens SOMATOM^® Sensation^™ 64 CT scanner with the x-ray tube settings of 25 mAs and 120 kVp. The acquisition was repeated to obtain 186 noise realizations. More details of the acquisitions and of the phantom can be found in our previous publications [30], [31].

The phantom data were reconstructed using a fanbeam filtered-backprojection (FFBP) algorithm with and without sinogram smoothing. Internally, the FFBP algorithm performed fan-to-parallel beam rebinning, filtering of the parallel beam data, and backprojection in the parallel geometry. We applied sinogram smoothing to the ramp-filtered parallel beam data (no apodization) before backprojection.

Note that fan-to-parallel beam rebinning and ramp filtering induce slight correlations. Here, we assume these correlations can be neglected and apply objective function (1) to such preprocessed data, acknowledging that such an approach could potentially be sub-optimal.

Conventionally, sinogram smoothing is applied to the original data rather than the filtered data. Our approach is justified for objects that can be reasonably modeled as a combination of pill-box like elemental objects. Indeed, the filtered parallel-beam projection of such an elemental object is constant over the object support and slowly-varying outside, with sharp transitions between the two regions. Such signal features are well amenable to using the Huber penalty as in (1).

To estimate the statistical weights w_k in (1), the empirical variance of the 186 filtered sinograms was calculated. The inverse variance was smoothed by a Gaussian function, then the result was normalized to have a mean value of one and used as the statistical weights.

An example of FFBP reconstruction obtained after sinogram preprocessing is shown side-by-side with the original FFBP image in Fig. 6(a). There is a substantial reduction in noise magnitude and in the horizontal noise streaks connecting the arms.

Fig. 6.

(a) A side-by-side comparison of FFBP reconstruction with (left) and without (right) sinogram smoothing. The horizontal streaks due to photon starvation are greatly reduced. The parameters used in sinogram smoothing were β= 0.005, δ = 0.02. The yellow and green boxes mark the locations of the ROIs for RMSE and resolution calculation. (b) RMSE versus resolution tradeoffs for different values of δ in the Huber penalty and different apodization windows of FFBP. For each δ, the range of β includes the point with the lowest RMSE. The green arrow points to the parameter choice that produced the image on the left in (a). (C, W) = (0, 300) HU.

To quantitatively compare image quality, we calculated task performance versus resolution tradeoffs using (1) different parameter settings for the Huber penalty, and (2) different apodization windows in the FFBP algorithm. The task was estimation of the mean intensity of a 7 × 7 region-of-interest (ROI). The task performance metric was the ensemble RMSE for this mean intensity estimated from the 186 noise realizations. The average of RMSE results obtained for the 16 ROIs shown as yellow boxes in Fig. 6(a) was used as final figure of merit. The ground truth (needed for the bias component of RMSE) was estimated by applying the original FFBP reconstruction with no apodization to the average of the 186 noise realizations. For image resolution, we extracted and averaged radial profiles across the 200 HU insert (Fig. 6(a), green box) to obtain the modulation transfer function (MTF). The frequency where the MTF is at 50% of its maximum was used as a metric for resolution.

Fig. 6(b) compares the tradeoff curves generated by varying β for different values of δ, from δ = 0 to δ = 1.³ The comparison includes the tradeoff curve for variations in the FFBP apodization window with no sinogram preprocessing. At high resolution (region A), we see that the Huber penalty with large δ provides higher task performance (lower RMSE). At moderate resolution (region B), using a smaller edge-preserving δ, around 0.02, is more favorable. Additional data (not shown) demonstrate that the tradeoff curve changes continuously as δ varies. For the parameters used for the image in Fig 6(a), β= 0.005 and δ = 0.02, the gain in RMSE over FFBP is about 15%.

The original sinogram (before smoothing) contains a wide range of signal variations. Fig. 7 plots the maximum number of knots max_k B_k calculated for each projection view; in the worse case, this number was 7. Again, the number of knots depends on the β and δ values. The small number of knots makes sinogram smoothing using the direct algorithm very efficient. The total time for processing all 1160 views was 600 msec, thus negligible compared to the reconstruction time of analytic FFBP (17 sec), using a single CPU implementation (Intel Xeon^® CPU, 2.4 GHz, 48 GB RAM).⁴

Fig. 7.

The maximum number of knots MAX_k B_k for each projection view when β= 0.005 and δ = 0.02.

V. Discussion

The worst case computational complexity of our algorithm is O(N²). In the literature, the direct TV algorithm of Condat [1] has a worst case complexity of O(N²). But the direct algorithms presented in Johnson [2] and Kolmogorov et. al. [4] both have a worst case complexity of O(N); these algorithms are based on a reparameterization of the knots. Direct adoption of such reparameterization will not reduce the computational complexity for the Huber penalty, because the location of the knots will change, and relabeling all knots at each index k, with the worst case linear increase of knots, will still incur an O(N²) computation. Condat [1] gave a numerical example with the worst case complexity for his algorithm. We empirically tested that, when δ_k’s are small, signals with similar characteristics (a linear signal with a slope of O(1/N²)) will have O(N) knots, hence O(N²) complexity. Such signals are obviously contrived. In our numerical studies, the number of knots was not a function of the signal length, and the numerical complexity was O(N).

The several existing DP-based algorithms for TV penalized quadratic fitting can be divided into the primal approach, as in this work, or the dual approach [1], [3]. A dual approach using the objective function (34) can be developed similar to [3]. An interesting topic for future research could be to compare the numerical properties of both primal and dual algorithms.

There are a number of interesting aspects to our development. First, our direct algorithm calculates the exact solution, unlike any other existing ones which are iterative in nature. Researchers can use the output of the direct algorithm as a benchmark for iterative algorithms. Our comparisons demonstrate strong computational advantages over iterative counterparts. Note that these advantages will remain if an alternative penalty is used, such as $\sqrt{{\epsilon }^2+s^2}$, which is twice-differentiable and thus enables the use of second-order methods. We have conducted a number of experiments (not reported here) that support this claim.

Notably, the Huber function is the Moreau envelope of the absolute value function. The Huber penalty indeed comes out as a special case of applying the smoothing technique of Nesterov [18, eqn. (4.17)] to the TV penalty. Such a connection between Huber and TV penalties does not exist for other mollified versions of TV, e.g., $\sqrt{{\epsilon }^2+s^2}$.

The examples we presented, namely tissue concentration curve smoothing and sinogram smoothing, are two promising applications of the proposed algorithm. For each application, more systematic and task-performance based parameter tuning is warranted to exploit the benefit in image (signal) quality. Our focus in this paper was algorithm development, not evaluation. Regarding sinogram smoothing, we do not currently have evidence showing that denoising the filtered data rather than the original data provides better results. It would be interesting to compare the performance of our approach to that in [6], where the image quality benefits of sinogram smoothing with Huber penalty were thoroughly demonstrated.

Another important application for our algorithm is as a subproblem solver in an iterative algorithm for 2-D or 3-D applications such as image restoration or reconstruction. Such an approach has been suggested for 2-D denoising using the TV penalty [4], [5]; a substantial speed advantage was achieved over the primal-dual algorithm [4], [32].

Here we outline the idea using x-ray CT image reconstruction. The objective function can be written as

$$\frac{1}{2}{\Vert \mathit{y}-A\mathit{x}\Vert }^2+\beta \sum _{i,j\in N_i}H(x_i-x_j).$$ (36)

where x = {x_i}, x_i is the unknown linear attenuation coefficient at pixel i, i = 1, ···, N_xN_y is the linear index of the pixel, and N_x and N_y are the number of rows and columns of a 2-D image. We use 2-D reconstruction as an example for its notational convenience; extension to 3-D problems is straightforward. The input y is the set of line integrals, and A is the forward projection operator. The Huber penalty H penalizes intensity differences within a pixel’s neighborhood N_i. For 2D problems with a first-order neighborhood, the Huber penalty can be expanded as

$$\sum _{i,j\in N_i}H(x_i-x_j)=\sum _{i^{\prime },j^{\prime }}[H(x_{i^{\prime }+1,j^{\prime }}-x_{i^{\prime },j^{\prime }})+H(x_{i^{\prime },j^{\prime }+1}-x_{i^{\prime },j^{\prime }})]$$ (37)

where we switched from the linear indexing to the row, column indexing i′, j′, i′ = 1, ···, N_x, j′ = 1, ···, N_y. To apply our direct algorithm as a subproblem solver, we introduce auxiliary variables x¹, x² together with the following constraints

$$x=x^1,x=x^2.$$ (38)

Then minimizing (36) is equivalent to

$$\underset{x,x^1,x^2}{min}\frac{1}{2}{\Vert y-Ax\Vert }^2+\beta \sum _{i^{\prime },j^{\prime }}[H(x_{i^{\prime }+1,j^{\prime }}^1-x_{i^{\prime },j^{\prime }}^1)+H(x_{i^{\prime },j^{\prime }+1}^2-x_{i^{\prime },j^{\prime }}^2)],$$ (39)

subject to (38).

Note that the Huber penalty in the row (i′) and column (j′) directions is applied to each individual auxiliary variable x¹, x², respectively. If we apply the alternating direction method of multipliers (ADMM) to the above constrained problem, then the subproblems involve 1-D Huber smoothing that are column-wise (j′) or row-wise (i′) separable, to which our proposed algorithm can be directly applied.

We implemented this idea for a special case of (36) where A is the identity matrix, which means we considered an image denoising problem. We ran the algorithm with 1000 iterations to define the converged solution. Then, we calculated the RMSE for the first 100 iterations. For comparison, we also ran the accelerated gradient descent (AGD) algorithm on the same image. The denoised image obtained with δ = 5 HU and β= 6.7 is shown together with the original image in Fig. 8. The convergence plot is shown in Fig. 9: it respectively takes 6 and 44 iterations for the proposed method and AGD to reach an accuracy of 0.01 HU. For higher accuracy requirements, the difference is even bigger.

Fig. 8.

Image denoising. (a) A tessellation of the original (quadrants I, IV) and the denoised patient image (quadrants II, III). (b) The ROI around the liver, pancreas, and the kidney of the original (top) and the denoised image (bottom). (C, W) = (10, 400) HU.

Fig. 9.

The root mean squared error (RMSE) in HU as a function of the iteration number for the accelerated gradient descent (AGD) and the proposed method outlined in (39).

Further extensions of the DP-based algorithms could consider generalized unary terms, such as piecewise linear or piecewise quadratic functions. For example, the quadratic unary terms could be replaced by the Huber function to achieve robust data fitting while rejecting outliers. Moreover, hard constraints, such as non-negativity or finite bounds on the variables, which are relevant for both tissue curve smoothing and sinogram or image smoothing, could be incorporated into the problem formulation. These extensions are of high interest to us and are also being pursued.

VI. Conclusions

We developed a direct (noniterative) algorithm for 1-D signal smoothing using quadratic data fitting with a penalty term based on the Huber function. The direct algorithm was derived from a dynamic programming reformulation of the objective function. We discussed in detail the solution method and computational cost. Two example applications, tissue concentration curve smoothing and sinogram smoothing, were used to demonstrate the effectiveness of the direct algorithm. Further applications such as using it as subproblem solvers in 2-D or 3-D image restoration and reconstruction were discussed, as well as possible extensions of the algorithm, such as including interval constraints.