A sequential solution for anisotropic total variation image denoising with interval constraints

Abstract

We show that two problems involving the anisotropic total variation (TV) and interval constraints on the unknown variables admit, under some conditions, a simple sequential solution. Problem 1 is a constrained TV penalized image denoising problem; Problem 2 is a constrained fused lasso signal approximator. The sequential solution entails finding first the solution to the unconstrained problem, and then applying a thresholding to satisfy the constraints. If the interval constraints are uniform, this sequential solution solves problem 1. If the interval constraints furthermore contain zero, the sequential solution solves problem 2. Here uniform interval constraints refer to all unknowns being constrained to the same interval. A typical example of application is image denoising in x-ray CT, where the image intensities are non-negative as they physically represent linear attenuation coefficient in the patient body. Our results are simple yet seem unknown; we establish them using the Karush-Kuhn-Tucker conditions for constrained convex optimization.

1. Introduction

Total variation (TV) is widely used in signal processing and in image denoising, restoration, and reconstruction, as an edge-preserving penalty to cope with ill-posedness in the problem. The TV penalty, both the anisotropic model and the isotropic model, is convex but nonsmooth, thus causes difficulty for numerical optimization. Some numerical algorithms replace the TV by a smoothed version of TV so that conventional gradient-descent type algorithms can be applied. These algorithms introduce a nuisance parameter ε > 0 to the TV penalty; the value of ε can be used to trade off computational accuracy and algorithm efficiency [1]. Other methods, such as the primal-dual type [2], the operator splitting type [3], and numerous others [4–6], have been developed to deal with the nonsmooth TV directly without introducing any nuisance parameters.

In many imaging applications, the image intensity values are known to be non-negative. For example, in x-ray CT, the pixel values can only be non-negative as they represent the x-ray linear attenuation coefficient of human tissues. In nuclear medicine, the pixel values represent radioactivity uptakes, which are also non-negative. Incorporating this prior knowledge not only produces more interpretable results [7–9], it also improves image quality; see e.g., [10–14]. In SPECT and CT, it is also often possible to identify an upper bound on the image intensity values. Such interval constraints have been shown to reduce reconstruction time and generate more accurate attenuation maps in the case of truncated SPECT transmission data [15, 16].

TV problems with interval constraints have also received attention from a numerical optimization perspective, see, e.g., [10, 17–19]. The existing algorithms for the constrained problems often consider a generic image restoration problem with either the anisotropic or the isotropic TV model. In many practical implementations, the interval constraints are simply handled in the following ad-hoc manner. First, the solution to the unconstrained TV problem is obtained. Second, the solution from the first step is thresholded to satisfy the constraints [13,20]. In this work, we show that this seemingly naive solution is indeed accurate for a subclass of constrained TV problems, namely, when the problem is a denoising problem using the anisotropic TV and when the interval constraints are uniform, i.e., all unknowns have the same interval constraint [l, u], l < u. Note that this subclass includes TV denoising with the non-negativity constraint as a special case, i.e., l = 0 and u = ∞.

The rest of the paper is organized as follows. In Sec. 2 we define Problem 1 and establish that the two-step solution mentioned above is exact. Note that this exactness is true for nonsmooth TV minimization, i.e., without introducing any nuisance parameter ε. Counter examples with smooth TV or with nonuniform interval constraints are given in Appendix. The analysis in Sec 2 is more detailed and serves as a model for Sec. 3, which discusses a generalization of Problem 1, a constrained fused lasso signal approximator. The constrained fused lasso also admits a sequential solution when the interval constraint is uniform and contains zero. We discuss some related development in the literature in Sec 4 and conclude in Sec. 5.

2. TV-penalized denoising with uniform interval constraints

Consider the following problem

$$\underset{\{x_i\}}{\text{min}}\{{\mathrm{\Theta }}_1(\gamma )=\sum _{i=1}^N\frac{w_i}{2}{(x_i-y_i)}^2+\gamma \sum _i{\iota }_{[l,u]}(x_i)+\sum _i\sum _{j\in N_i}{\beta }_{\mathit{\text{ij}}}|x_i-x_j|\}$$ (1)

where x_i, y_i, i = 1, ⋯, N are the unknowns^‡ and the noisy input data, respectively, and where N_i is a fixed set of neighbors for each i. The weight w_i > 0 is often chosen to be inversely proportional to the variance of y_i, and β_ij > 0 is the penalty weight.

Note that we use ι_C to denote the indicator function of the convex set C, i.e., ι_C(x) = 0 if x ∈ C, and ι_C(x) = ∞ when x ∉ C. In (1), the parameter γ of the indicator function is either 0 or 1, so that the objective function is unconstrained Θ₁(0) or constrained Θ₁(1), respectively. When γ = 1, the unknowns in (1) are all uniformly constrained to the interval [l, u]. We assume l < u and we allow l = −∞ or u = ∞, which means the constraint can be one-sided. Problem (1) is a general smoothing or denoising problem.

Since the objective function (1) is strongly convex, there is a unique global minimum to Θ₁(γ) for γ = {0, 1}. To simplify notation, we write z ≡ {z_i}, where z is a generic vector variable. The solution x* to the constrained problem Θ₁(1) can be obtained from the Karush-Kuhn-Tucker (KKT) conditions for convex nonsmooth problems [21, Theorem 3.34]: if the triple x*, λ¹*, λ²* satisfies the following conditions

$$w_i(x_i^{\ast }-y_i)+\sum _{j\in N_i}{\beta }_{\mathit{\text{ij}}}{t}_{\mathit{\text{ij}}}(x^{\ast })-\sum _{j:i\in N_j}{\beta }_{\mathit{\text{ji}}}{t}_{\mathit{\text{ji}}}(x^{\ast })-{\lambda }_i^{1\ast }+{\lambda }_i^{2\ast }∍0,\forall i$$ (2a)

$$x_i^{\ast }\le u,{\lambda }_i^{2\ast }\ge 0,{\lambda }_i^{2\ast }(x_i^{\ast }-u)=0,\forall i$$ (2b)

$$x_i^{\ast }\ge l,{\lambda }_i^{1\ast }\ge 0,{\lambda }_i^{1\ast }(x_i^{\ast }-l)=0,\forall i$$ (2c)

where t_mn(x) is the subdifferential of |x_m − x_n| at x, then x* is the minimizer of Θ₁(1). By definition, t_mn(x) = sign(x_m − x_n) if x_m ≠ x_n, and t_mn(x) = [−1, 1] if x_m = x_n. The auxiliary variables ${\lambda }_i^{1\ast },{\lambda }_i^{2\ast }$, i = 1, ⋯, N, are the (optimal) Lagrange multipliers introduced for the constraints.

Denote by x̃* the solution to minimizing Θ₁(0), i.e., the difference between x̃* and x* is that x* satisfies the constraints $l\le x_i^{\ast }\le u$, ∀i. We establish that $x_i^{\ast }=\text{median}\{{\stackrel{\sim }{x}}_i^{\ast },l,u\}$, which expands to

$$x_i^{\ast }=\{\begin{array}{cc}l\hfill & {\stackrel{\sim }{x}}_i^{\ast }\le l\hfill \\ {\stackrel{\sim }{x}}_i^{\ast }\hfill & l<{\stackrel{\sim }{x}}_i^{\ast }<u\hfill \\ u\hfill & {\stackrel{\sim }{x}}_i^{\ast }\ge u\hfill \end{array}.$$ (3)

In other words, x* can be obtained in a two-step manner. First we solve (1) without considering the constraints; second, we perform pixelwise thresholding.

The solution x̃* from the first step satisfies the first order optimality condition, i.e.,

$$w_i({\stackrel{\sim }{x}}_i^{\ast }-y_i)+\sum _{j\in N_i}{\beta }_{\mathit{\text{ij}}}{t}_{\mathit{\text{ij}}}({\stackrel{\sim }{x}}^{\ast })-\sum _{j:i\in N_j}{\beta }_{\mathit{\text{ji}}}{t}_{\mathit{\text{ji}}}({\stackrel{\sim }{x}}^{\ast })∍0,\forall i$$ (4)

and the second (thresholding) step in (3) is the solution to the following constrained minimization problem

$$\underset{{\hat{x}}_i\in [l,u]}{\text{min}}\frac{1}{2}\sum _i{({\stackrel{\sim }{x}}_i^{\ast }-{\hat{x}}_i)}^2.$$ (5)

By applying the KKT conditions to (5), we find that the solution x̂ satisfies

$$({\hat{x}}_i-{\stackrel{\sim }{x}}_i^{\ast })-{\hat{\lambda }}_i^1+{\hat{\lambda }}_i^2=0,\forall i$$ (6a)

$${\hat{x}}_i\le u,{\hat{\lambda }}_i^2\ge 0,({\hat{x}}_i-u){\hat{\lambda }}_i^2=0,\forall i$$ (6b)

$${\hat{x}}_i\ge l,{\hat{\lambda }}_i^1\ge 0,({\hat{x}}_i-l){\hat{\lambda }}_i^1=0,\forall i$$ (6c)

for some $\{{\hat{\lambda }}_i^1\},\{{\hat{\lambda }}_i^2\}$. We show that {x̂_i}, $\{w_i{\hat{\lambda }}_i^1\},\{w_i{\hat{\lambda }}_i^2\}$ is a triple that satisfies the KKT conditions (2). Hence, x̂ = x*, the unique minimizer of Θ₁(1).

Given the relation in (6) between x̃* and x̂, we must have

$$t_{\mathit{\text{ij}}}({\stackrel{\sim }{x}}^{\ast })\subseteq t_{\mathit{\text{ij}}}(\hat{x})$$ (7)

This is because the thresholding from x̃* to x̂ is order-preserving when all pixels are subject to the same constraints [l, u]. In other words, if ${\stackrel{\sim }{x}}_i^{\ast }\ge {\stackrel{\sim }{x}}_j^{\ast }$, then x̂_i ≥ x̂_j. More specifically,

• If x̂_i ≠ x̂_j, then t_ij(x̂) = sign(x̂_i − x̂_j) = t_ij(x̃*);

• If x̂_i = x̂_j, then t_ij(x̃*) ⊆ t_ij(x̂) = [−1, 1].

If we multiply (6a) by w_i and add it to (4), then

$$w_i({\hat{x}}_i-y_i)+\sum _{j\in N_i}{\beta }_{\mathit{\text{ij}}}{t}_{\mathit{\text{ij}}}({\stackrel{\sim }{x}}^{\ast })-\sum _{j:i\in N_j}{\beta }_{\mathit{\text{ji}}}{t}_{\mathit{\text{ji}}}({\stackrel{\sim }{x}}^{\ast })-(w_i{\hat{\lambda }}_i^1)+(w_i{\hat{\lambda }}_i^2)∍0$$ (8)

which is almost (2a), except that in (8) t_ij are dependent on x̃*. The subset relation in (7) ensures that we can replace t_ij(x̃*) by t_ij(x̂) in (8) and the relation still holds. Then our claim is established as the triple {x̂_i}, $\{w_i{\hat{\lambda }}_i^1\},\{w_i{\hat{\lambda }}_i^2\}$ obviously satisfies (2b) and (2c).

3. Denoising with the fused lasso penalty subject to uniform intervals

We define the following minimization problem

$$\underset{\{x_i\}}{\text{min}}\{{\mathrm{\Theta }}_2(\gamma ,\eta )=\sum _{i=1}^N\frac{w_i}{2}{(x_i-y_i)}^2+\gamma \sum _i{\iota }_{[l,u]}(x_i)+\eta \sum _i|x_i|+\sum _i\sum _{j\in N_i}{\beta }_{\mathit{\text{ij}}}|x_i-x_j|\}.$$ (9)

The l₁ penalty encourages sparse coefficients in the unknown x, with hyperparameter η ≥ 0. When w_i = w for all i, Θ₂(0, η) is the objective function of the fused lasso signal approximator (FLSA) [22]. When η = 0, we have Θ₂(γ, 0) ≡ Θ₁(γ), thus (9) includes (1) as a special case. Again, since the objective function is strongly convex, there is a unique global minimizer to Θ₂(γ, η) for γ = {0, 1} and all η ≥ 0.

Applying the KKT conditions for convex nonsmooth problems [21, Theorem 3.34], the solution to (9) satisfies

$$w_i(x_i^{\ast }-y_i)+\eta s(x_i^{\ast })+\sum _{j\in N_i}{\beta }_{\mathit{\text{ij}}}{t}_{\mathit{\text{ij}}}(x^{\ast })-\sum _{j,i\in N_j}{\beta }_{\mathit{\text{ji}}}{t}_{\mathit{\text{ji}}}(x^{\ast })-{\lambda }_i^{1\ast }+{\lambda }_i^{2\ast }∍0,\forall i$$ (10a)

$$x_i^{\ast }\le u,{\lambda }_i^{2\ast }\ge 0,{\lambda }_i^{2\ast }(x_i^{\ast }-u)=0,\forall i$$ (10b)

$$x_i^{\ast }\ge l,{\lambda }_i^{1\ast }\ge 0,{\lambda }_i^{1\ast }(x_i^{\ast }-l)=0,\forall i$$ (10c)

Comparing (10) with (2), the only addition is the term related to η, where s(x_i) is the subdifferential of |x_i|, namely s(x_i) = sign(x_i) if x_i ≠ 0, and s(x_i) = [−1, 1] otherwise.

On the other hand, the solution to the unconstrained counterpart satisfies the first order optimality condition:

$$w_i({\stackrel{\sim }{x}}_i^{\ast }-y_i)+\eta s({\stackrel{\sim }{x}}_i^{\ast })+\sum _{j\in N_i}{\beta }_{\mathit{\text{ij}}}{t}_{\mathit{\text{ij}}}({\stackrel{\sim }{x}}^{\ast })-\sum _{j:i\in N_j}{\beta }_{\mathit{\text{ji}}}{t}_{\mathit{\text{ji}}}({\stackrel{\sim }{x}}^{\ast })∍0,\forall i$$ (11)

The second (thresholding) step, which produces the output x̂_i from ${\stackrel{\sim }{x}}_i^{\ast }$ in (11), is the same as in (6) and will not be repeated. We claim that {x̂_i}, $\{w_i{\hat{\lambda }}_i^1\},\{w_i{\hat{\lambda }}_i^2\}$ is a triple that satisfies the KKT conditions (10). Hence, we must have x̂ = x*, the unique solution of Θ₂(1, η). To prove this, we multiply both sides of (6) by w_i and add it to (11) to obtain

$$w_i({\hat{x}}_i-y_i)+\eta s({\stackrel{\sim }{x}}_i^{\ast })+\sum _{j\in N_i}{\beta }_{\mathit{\text{ij}}}{t}_{\mathit{\text{ij}}}({\stackrel{\sim }{x}}^{\ast })-\sum _{j:i\in N_j}{\beta }_{\mathit{\text{ji}}}{t}_{\mathit{\text{ji}}}({\stackrel{\sim }{x}}^{\ast })-(w_i{\hat{\lambda }}_i^1)+(w_i{\hat{\lambda }}_i^2)∍0$$ (12)

which is almost (10a) except that the subdifferentials are evaluated at ${\stackrel{\sim }{x}}_i^{\ast }$ and not x̂_i. As discussed in Sec 2, since all pixels are subject to the same constraint [l, u] and since the thresholding step is order-preserving, we can replace t_ij(x̃*) by t_ij(x̂) without affecting the validity of (12). If in addition, the interval [l, u] contains zero, i.e., 0 ∈ [l, u], then the thresholding step (3) is sign-preserving as well. Therefore we have $s({\hat{x}}_i)\supseteq s({\stackrel{\sim }{x}}_i^{\ast })$ for all i. Indeed, if x̂_i > 0 then ${\stackrel{\sim }{x}}_i^{\ast }>0$ and thus $s({\hat{x}}_i)=s({\stackrel{\sim }{x}}_i^{\ast })$; if x̂_i = 0, then ${\stackrel{\sim }{x}}_i^{\ast }$ can be nonzero if l = 0 or u = 0, but we always have $s({\stackrel{\sim }{x}}_i^{\ast })\subseteq s({\hat{x}}_i)=[-1,1]$; if x̂_i < 0, then ${\stackrel{\sim }{x}}_i^{\ast }<0$ and thus $s({\hat{x}}_i)=s({\stackrel{\sim }{x}}_i^{\ast })$. In summary, replacing x̃* by x̂ maintains the correctness of (12). Hence our claim is established as the triple {x̂_i}, $\{w_i{\hat{\lambda }}_i^1\},\{w_i{\hat{\lambda }}_i^2\}$ obviously satisfies (10b) and (10c).

It is interesting to point out that the FLSA itself, i.e., the problem of minimizing Θ₂(0, η) for η > 0, admits a two-step solution when all weights w_i = w are equal. First, the solution to the problem of minimizing Θ₂(0, 0) is obtained, then pixelwise soft-thresholding can be applied to the first step solution to obtain the solution to Θ₂(0, η) [23]. Combining this observation with our result, the solution to minimizing Θ₂(1, η) can be obtained in a sequential manner. Starting from the solution to Θ₂(0, 0), we apply pixelwise soft-thresholding (to minimize Θ₂(0, η)) and then another thresholding to satisfy the interval constraints, i.e., to solve Θ₂(1, η).

4. Discussion

Throughout the text, we have assumed that a solver for the unconstrained TV penalized problem is available. Our contribution has shown that applying thresholding to the result from this solver yields the solution to the problem with uniform interval constraints. Any solver for the nonsmooth TV problem can be used. When the problem has additional structures, e.g., if the unknowns x_i form a chain [14, 24], a tree or an acyclic graph [25], a direct (noniterative) solution to the unconstrained problem can be obtained. Applying our result then produces a direct solution to the constrained counterpart.

Some problem instances in sparse reconstruction and machine learning are known to admit a sequential solution, see e.g., [26–28]. These solutions can be addressed in a general setting using proximal decomposition [29]. We did not take the approach of proximal decomposition as that would limit us to uniform data weighting w_i = w > 0. Our sequential characterization of the solution to (1) and (9) works under general nonuniform data weighting.

5. Conclusions

We have discussed constrained image denoising problems that involve the anisotropic total variation (TV) penalty. Two specific problems were considered: (i) the constrained TV denoising problem, and (ii) the constrained fused lasso problem. When all unknowns are subject to the same interval constraint, and when the interval contains 0 for problem (ii), we have shown that the solution to the constrained problem is the thresholded version of the solution to the unconstrained problem. Our work thus provides justification for this approach, which is often adopted in an ad-hoc manner. However, note that the sequential approach is in general not correct for non-uniform interval constraints, nor is it correct in general for smooth TV using the Huber potential (see the counter examples in Appendix). Whether or not the sequential approach works with isotropic TV remains an open question. We also do not know if the sequential solution is valid for a generic image reconstruction problem. However, an image reconstruction algorithm often contains image denoising in an inner loop. Within such an inner loop, our sequential solution is applicable to solve the constrained image denoising problem.

Appendix

Examples

We provide two 1-D counter-examples showing that the two-step approach is incorrect for (i) non-uniform interval constraints and (ii) a smooth TV using the Huber function. We consider the following objective function,

$$\sum _{i=1}^N\frac{1}{2}{w}_i{(x_i-y_i)}^2+\beta \sum _{i=1}^{N-1}\varphi (x_{i+1}-x_i),$$ (13)

where ϕ(·) is a generic penalty function penalizing neighboring differences. The followings are common parameters for both examples.

• Number of data points, N = 3;

• Input y = {y₁, y₂, y₃} = {−1, 0, 1};

• Data weight w₁ = w₂ = w₃ = 1;

• Penalty weight β = 0.5;

Example (i)

Here ϕ(s) = |s|, then (13) is a special case of (1) in Sec 2. The unknowns x₁, x₂, x₃ are subject to the nonuniform constraints: x₁ ≥ 0.5, x₂ ≥ 0, and x₃ ≥ 0.5. That is, we let the upper bound u = ∞ in this example. The unconstrained solution is x̃* = {−0.5, 0, 0.5}, as can be verified by checking the first order optimality condition. To satisfy the nonuniform constraints, we need to threshold ${\stackrel{\sim }{x}}_1^{\ast }$ and obtain x̂* = {0.5, 0, 0.5} which has an objective function value of 1.75. However, this is not optimal as another candidate {0.5, 0.5, 0.5} achieves a lower objective value of 1.375. Thus, the 2-step solution is incorrect when the interval constraints are not the same for all unknowns.

Example (ii)

We replace the TV penalty by the Huber function, given by ϕ(s) = inf_t{|t| + (s − t)²/(2ε)}, where ε > 0 is a nuisance parameter. As ε ↓ 0, ϕ(s) ↑ |s|. Alternatively, we can write ϕ(s) = s²/(2ε) if |s| ≤ ε, and ϕ(s) = |s| − ε/2 otherwise. We set ε = 0.1 in this example. The unknown x = {x₁, x₂, x₃} is subject to the non-negativity constraint, i.e., x_i ≥ 0 for i = 1, 2, 3. It can be verified that the unconstrained solution x̃* = {−0.5, 0, 0.5} which is then thresholded to give x̂ = {0, 0, 0.5} and the objective function at x̂ is 0.85. This solution is not optimal as another candidate {0, 1/12, 0.5} achieves a lower objective value of ≈ 0.83. Thus the 2-step solution is not correct for smoothed TV in general, even with uniform interval constraints.