A STRICTLY CONTRACTIVE PEACEMAN–RACHFORD SPLITTING METHOD FOR CONVEX PROGRAMMING

Abstract

In this paper, we focus on the application of the Peaceman–Rachford splitting method (PRSM) to a convex minimization model with linear constraints and a separable objective function. Compared to the Douglas–Rachford splitting method (DRSM), another splitting method from which the alternating direction method of multipliers originates, PRSM requires more restrictive assumptions to ensure its convergence, while it is always faster whenever it is convergent. We first illustrate that the reason for this difference is that the iterative sequence generated by DRSM is strictly contractive, while that generated by PRSM is only contractive with respect to the solution set of the model. With only the convexity assumption on the objective function of the model under consideration, the convergence of PRSM is not guaranteed. But for this case, we show that the first t iterations of PRSM still enable us to find an approximate solution with an accuracy of O(1/t). A worst-case O(1/t) convergence rate of PRSM in the ergodic sense is thus established under mild assumptions. After that, we suggest attaching an underdetermined relaxation factor with PRSM to guarantee the strict contraction of its iterative sequence and thus propose a strictly contractive PRSM. A worst-case O(1/t) convergence rate of this strictly contractive PRSM in a nonergodic sense is established. We show the numerical efficiency of the strictly contractive PRSM by some applications in statistical learning and image processing.

1. Introduction

We consider the following convex minimization model with linear constraints and a separable objective function:

$$min\{{\theta }_1(x)+{\theta }_2(y)\mid Ax+By=b,x\in \mathcal{X},y\in \mathcal{Y}\},$$ (1.1)

where A ∈ ℜ^m×n₁, B ∈ ℜ^m×n₂, b ∈ ℜ^m, ⊂ ℜ^n₁ and ⊂ ℜ^n₂ are closed convex sets, and θ₁ : → ℜ and θ₂ : → ℜ are convex functions. Note that both θ₁ and θ₂ could be nonsmooth functions. Throughout, the solution set of (1.1) (denoted by S^*) is assumed to be nonempty.

When A = I_m, B = −I_m, b = 0, X = ℜ^m, and Y = ℜ^m, the model (1.1) reduces to

$$min\{{\theta }_1(x)+{\theta }_2(x)\mid x\in {\Re }^m\},$$ (1.2)

where θ₁ and θ₂ can be explained, respectively, as data-fidelity and regularization terms for some ill-posed inverse problems arising widely in statistical learning and image processing areas. Because of the separable structure in its objective function, the majority of efficient solvers for (1.1) in the literature belong to the category of the splitting method, which is effective for taking advantage of the properties of the functions θ₁ and θ₂ individually in the algorithmic design. The resulting subproblems are usually easy enough to have closed-form solutions or can be solved easily up to high precisions; therefore, the implementation of such an algorithm is extremely easy and its convergence is fast. A benchmark is the alternating direction method of multipliers (ADMM) proposed in [24] (see also [9, 21]), which has received tremendous attention from a number of areas recently (see, e.g., [6, 17, 22] for review papers). As analyzed in [20], ADMM is just an application of the Douglas–Rachford splitting method (DRSM) in [15, 39] to the dual problem of (1.1), and its iterative scheme reads as

$$\{\begin{array}{l}{x}^{k+1}=argmin\{{\theta }_1(x)-{({\lambda }^k)}^T(Ax+B{y}^k-b)+\frac{\beta }{2}{\Vert Ax+B{y}^k-b\Vert }^2\mid x\in \mathcal{X}\},\hfill \\ y^{k+1}=argmin\{{\theta }_2(y)-{({\lambda }^k)}^T(A{x}^{k+1}+By-b)+\frac{\beta }{2}{\Vert A{x}^{k+1}+By-b\Vert }^2\mid y\in \mathcal{Y}\},\hfill \\ {\lambda }^{k+1}={\lambda }^k-\beta (A{x}^{k+1}+B{y}^{k+1}-b),\hfill \end{array}$$ (1.3)

where λ ∈ ℜ^m is the Lagrange multiplier associated with the linear constraints in (1.1) and β > 0 is a penalty parameter.

In this paper, we focus on the application of the Peaceman–Rachford splitting method (PRSM) in [39, 46] to (1.1). As elaborated on in [20], applying PRSM to the dual of (1.1), we obtain the iterative scheme of PRSM for (1.1),

$$\{\begin{array}{l}{x}^{k+1}=argmin\{{\theta }_1(x)-{({\lambda }^k)}^T(Ax+B{y}^k-b)+\frac{\beta }{2}{\Vert Ax+B{y}^k-b\Vert }^2\mid x\in \mathcal{X}\},\hfill \\ {\lambda }^{k+\frac{1}{2}}={\lambda }^k-\beta (A{x}^{k+1}+B{y}^k-b),\hfill \\ y^{k+1}=argmin\{{\theta }_2(y)-{({\lambda }^{k+\frac{1}{2}})}^T(A{x}^{k+1}+By-b)+\frac{\beta }{2}{\Vert A{x}^{k+1}+By-b\Vert }^2\mid y\in \mathcal{Y}\},\hfill \\ {\lambda }^{k+1}={\lambda }^{k+\frac{1}{2}}-\beta (A{x}^{k+1}+B{y}^{k+1}-b),\hfill \end{array}$$ (1.4)

where λ ∈ ℜ^m and β have the same meaning as (1.3). As analyzed in [20], the PRSM scheme (1.4) differs from ADMM “only through the addition of the intermediate update of the multipliers (i.e., ${\lambda }^{k+\frac{1}{2}}$); it thus offers the same set of advantages.” The PRSM scheme (1.4), however, according to [20] again (see also [23]), “is less ‘robust’ in that it converges under more restrictive assumptions than ADMM.” Also, it was remarked in [20] that the PRSM scheme (1.4) with optimal parameters converges on the linear rate if Lipschitz continuity and coercivity of $\partial {\theta }_2^{\ast }$ (where ${\theta }_2^{\ast }$ denotes the conjugate function of θ₂) are assumed. We refer the reader to [3, 25] for some numerical verification of the efficiency of PRSM.

We first show that the difference between DRSM and PRSM in convergence can be illustrated by the contraction property of their iterative sequences.¹ More specifically, the iterative sequence generated by DRSM is strictly contractive with respect to the solution set of (1.1), as proved in [29], while this does not hold for the iterative sequence generated by PRSM (see (3.20)). This difference is also the reason a worst-case O(1/t) convergence rate of ADMM for (1.1) in a nonergodic sense can be established in [32], while we can only establish the same convergence rate in the ergodic sense for the PRSM scheme (1.4);² see Theorem 4.1. Also inspired by the failure of strict contraction of PRSM’s iterative sequence, we find that when an underdetermined relaxation factor α ∈ (0, 1) is attached to the penalty parameter β in the steps of Lagrange multiplier updating in (1.4), the resulting sequence becomes strictly contractive with respect to the solution set of (1.1). This strict contraction property makes it possible to establish a worst-case O(1/t) convergence rate in a non-ergodic sense for the PRSM (1.4) with an underdetermined relaxation factor, which will be named as a strictly contractive PRSM from now on. Let us specify the iterative scheme of the strictly contractive PRSM for (1.1):

$$\{\begin{array}{l}{x}^{k+1}=argmin\{{\theta }_1(x)-{({\lambda }^k)}^T(Ax+B{y}^k-b)+\frac{\beta }{2}{\Vert Ax+B{y}^k-b\Vert }^2\mid x\in \mathcal{X}\},\hfill \\ {\lambda }^{k+\frac{1}{2}}={\lambda }^k-\alpha \beta (A{x}^{k+1}+B{y}^k-b),\hfill \\ y^{k+1}=argmin\{{\theta }_2(y)-{({\lambda }^{k+\frac{1}{2}})}^T(A{x}^{k+1}+By-b)+\frac{\beta }{2}{\Vert A{x}^{k+1}+By-b\Vert }^2\mid y\in \mathcal{Y}\},\hfill \\ {\lambda }^{k+1}={\lambda }^{k+\frac{1}{2}}-\alpha \beta (A{x}^{k+1}+B{y}^{k+1}-b),\hfill \end{array}$$ (1.5)

where α ∈ (0, 1). Note that we follow the standard terminology in numerical linear algebra and call α ∈ (0, 1) an underdetermined relaxation factor; see also [18, 26]. As we shall show, the consideration of an additional relaxation factor in the PRSM scheme (1.5) ensures the sequence generated by (1.5) to be strictly contractive with respect to the solution set of (1.1). Thus we can establish some worst-case convergence rates for (1.5) without any further assumption on the model (1.1). Numerically, we can simply choose α close to 1.

The rest of this paper is organized as follows. In section 2, we summarize some useful preliminary results and prove some simple assertions for further analysis. Then, we prove some properties for the sequence generated by the strictly contractive PRSM (1.5) in section 3. In section 4, we establish a worst-case O(1/t) convergence rate in the ergodic sense for the PRSM scheme (1.4); and in section 5, we establish a worst-case O(1/t) convergence rate in a nonergodic sense for the strictly contractive PRSM scheme (1.5). Then, we show the numerical efficiency of the strictly contractive PRSM in section 6 by some applications to statistical learning and image processing. Some comparisons with existing efficient methods are also reported. Finally, some conclusions are drawn in section 7.

2. Preliminaries

In this section, we summarize some useful preliminaries known in the literature and prove some simple conclusions for further analysis.

2.1. Variational reformulation of (1.1)

First, as in the work [31, 32] for analyzing the convergence rate of ADMM, we need a variational inequality (VI) reformulation of the model (1.1) and a characterization of its solution set. More specifically, solving (1.1) is equivalent to finding w^* = (x^*, y^*, λ^*) ∈ ω := × × ℜ^m such that

$$\text{VI}(\mathrm{\Omega },F,\theta ):\theta (u)-\theta (u^{\ast })+{(w-w^{\ast })}^{T}F(w^{\ast })\ge 0\forall w\in \mathrm{\Omega },$$ (2.1a)

where

$$u=\left(\begin{array}{c}x\\ y\end{array}\right),w=\left(\begin{array}{c}x\\ y\\ \lambda \end{array}\right),F(w)=\left(\begin{array}{c}-A^T\lambda \\ -B^T\lambda \\ Ax+By-b\end{array}\right),\text{and}\theta (u)={\theta }_1(x)+{\theta }_2(y).$$ (2.1b)

Since the mapping F(w) defined in (2.1b) is affine with a skew-symmetric matrix, it is monotone. We denote by ω^* the solution set of VI(ω, F, θ), and it is not nonempty under our nonempty assumption onto S^*.

According to Theorem 2.3.5 in [19], a very useful characterization of ω^* can be summarized in the following theorem. Its proof can be found in [19, 31].

Theorem 2.1

The solution set of VI(ω, F, θ) is closed and convex, and it can be characterized as

$${\mathrm{\Omega }}^{\ast }=\underset{w\in \mathrm{\Omega }}{\cap }\{\stackrel{\sim }{w}\in \mathrm{\Omega }:\theta (u)-\theta (\stackrel{\sim }{u})+{(w-\stackrel{\sim }{w})}^{T}F(w)\ge 0\}.$$ (2.2)

Theorem 2.1 thus implies that w̃ ∈ ω is an approximate solution of VI(ω, F, θ) with an accuracy of O(1/t) if it satisfies

$$\underset{w\in \mathcal{D}}{sup}\{\theta (\stackrel{\sim }{u})-\theta (u)+{(\stackrel{\sim }{w}-w)}^{T}F(w)\}\le \epsilon ,$$ (2.3)

with ε = O(1/t) and ⊂ ω a compact set.³ In fact, this characterization makes it possible to analyze the convergence rate of ADMM and other splitting methods via the VI approach rather than the conventional approach based on the functional values in the literature. In the following, we shall show that either sequence (1.4) or sequence (1.5) enables us to find an approximate solution of (1.1) in the sense of (2.3) after t iterations.

2.2. Some notation

As mentioned in [6] for ADMM, the variable x is an intermediate variable during the PRSM iteration since it essentially requires only (y^k, λ^k) in (1.4) or (1.5) to generate the (k + 1)th iterate. For this reason, we define the notation v^k = (y^k, λ^k), = × ℜ^m,

$${\mathcal{V}}^{\ast }:=\{v^{\ast }=(y^{\ast },{\lambda }^{\ast })\mid w^{\ast }=(x^{\ast },y^{\ast },{\lambda }^{\ast })\in {\mathrm{\Omega }}^{\ast }\};$$

it suffices to analyze the convergence rate of the sequence {v^k} to the set in order to study the convergence rate of the sequence {w^k} generated by (1.4) or (1.5). Note that is also closed and convex.

Then, we define some matrices in order to present our analysis in a compact way. Let

$$M=\left(\begin{array}{cc}{I}_{n_2}& 0\\ -\alpha \beta B& 2\alpha I_m\end{array}\right)$$ (2.4)

and

$$Q_0=\left(\begin{array}{cc}0& 0\\ \beta B^{T}B& -\alpha B^T\\ -B& \frac{1}{\beta }{I}_m\end{array}\right).$$ (2.5)

In (2.5), “0” is a matrix with all zero entries in appropriate dimensionality. We further define

$$Q=\left(\begin{array}{cc}\beta B^{T}B& -\alpha B^T\\ -B& \frac{1}{\beta }{I}_m\end{array}\right)$$ (2.6)

as the submatrix of Q₀ excluding all the first zero rows. The matrices Q₀ and Q are associated with the analysis for the sequences {w^k} and {v^k}, respectively. Last, for α ∈ (0, 1] we define a symmetric matrix

$$H=\frac{1}{2}\left(\begin{array}{cc}(2-\alpha )\beta B^{T}B& -B^T\\ -B& \frac{1}{\alpha \beta }{I}_m\end{array}\right).$$ (2.7)

Below we prove some assertions regarding the matrices just defined. These assertions will be used in our theoretical analysis about the convergence rate of (1.4) and (1.5); their role is to make our proof presentable in compact notation.

Lemma 2.2

The matrix H defined in (2.7) is positive definite (if B is a full column rank matrix) for α ∈ (0, 1) and positive semidefinite for α = 1.

Proof

We have

$$H=\frac{1}{2}\left(\begin{array}{cc}\sqrt{\beta }{B}^T& 0\\ 0& \sqrt{\frac{1}{\beta }}{I}_m\end{array}\right)\left(\begin{array}{cc}(2-\alpha )I_m& -I_m\\ -I_m& \frac{1}{\alpha }{I}_m\end{array}\right)\left(\begin{array}{cc}\sqrt{\beta }B& 0\\ 0& \sqrt{\frac{1}{\beta }}{I}_m\end{array}\right).$$

Note that the matrix

$$\left(\begin{array}{cc}(2-\alpha )& -1\\ -1& \frac{1}{\alpha }\end{array}\right)$$

is positive definite if α ∈ (0, 1) and positive semidefinite if α = 1. The assertion of this lemma is thus proved.

Lemma 2.3

The matrices M, Q, and H defined, respectively, in (2.4), (2.6), and (2.7) have the following relationships:

$$HM=Q$$ (2.8)

and

$$Q^T+Q-M^{T}HM\succcurlyeq \frac{(1-\alpha )}{2(1+\alpha )}{M}^{T}HM.$$ (2.9)

Proof

Using the definitions of the matrices M, Q, and H, by a simple manipulation, we obtain

$$\begin{array}{l}HM=\frac{1}{2}\left(\begin{array}{cc}(2-\alpha )\beta B^{T}B& -B^T\\ -B& \frac{1}{\alpha \beta }{I}_m\end{array}\right)\left(\begin{array}{cc}{I}_{n_2}& 0\\ -\alpha \beta B& 2\alpha I_m\end{array}\right)\\ =\frac{1}{2}\left(\begin{array}{cc}2\beta B^{T}B& -2\alpha B^T\\ -2B& \frac{2}{\beta }{I}_m\end{array}\right)=Q.\end{array}$$

The first assertion is proved. Consequently, we get

$$\begin{array}{l}{M}^{T}HM=M^{T}Q=\left(\begin{array}{cc}{I}_{n_2}& -\alpha \beta B^T\\ 0& 2\alpha I_m\end{array}\right)\left(\begin{array}{cc}\beta B^{T}B& -\alpha B^T\\ -B& \frac{1}{\beta }{I}_m\end{array}\right)\\ =\left(\begin{array}{cc}(1+\alpha )\beta B^{T}B& -2\alpha B^T\\ -2\alpha B& \frac{2\alpha }{\beta }{I}_m\end{array}\right).\end{array}$$

Using (2.6) and the above equation, we have

$$(Q^T+Q)-M^{T}HM=(1-\alpha )\left(\begin{array}{cc}\beta B^{T}B& -B^T\\ -B& \frac{2}{\beta }{I}_m\end{array}\right).$$ (2.10)

Note that

$$2(1+\alpha )\left(\begin{array}{cc}\beta B^{T}B& -B^T\\ -B& \frac{2}{\beta }{I}_m\end{array}\right)-M^{T}HM=\left(\begin{array}{cc}(1+\alpha )\beta B^{T}B& -2B^T\\ -2B& \frac{4+2\alpha }{\beta }{I}_m\end{array}\right).$$ (2.11)

Because

$$\left(\begin{array}{cc}(1+\alpha )& -2\\ -2& 4+2\alpha \end{array}\right)\succcurlyeq 0\forall \alpha \ge 0,$$

the right-hand side of (2.11) is positive semidefinite. Thus, it follows that

$$\left(\begin{array}{cc}\beta B^{T}B& -B^T\\ -B& \frac{2}{\beta }{I}_m\end{array}\right)\succcurlyeq \frac{1}{2(1+\alpha )}{M}^{T}HM.$$ (2.12)

Substituting (2.12) into (2.10), we obtain (2.9), and the lemma is proved.

Remark 2.1

When α = 1, the matrices H defined in (2.7) and Q^T + Q − M^T HM are both positive semidefinite. However, in the following analysis we still use ||v − ṽ||_H and ||v − ṽ||_{(Q^T +Q − M^T HM)} to denote, respectively,

$${\Vert v-\stackrel{\sim }{v}\Vert }_H={({(v-\overline{v})}^{T}H(v-\stackrel{\sim }{v}))}^{1/2}$$

and

$${\Vert v-\stackrel{\sim }{v}\Vert }_{(Q^T+Q-M^{T}HM)}={({(v-\stackrel{\sim }{v})}^T(Q^T+Q-M^{T}HM)(v-\stackrel{\sim }{v}))}^{1/2}$$

for v, ṽ ∈ × ℜ^m This slight abuse of notation will simplify the notation in our analysis greatly.

3. Contraction analysis

In this section, we analyze the contraction property for the sequence {v^k} generated by the PRSM scheme (1.4) or the strictly contractive PRSM scheme (1.5) with respect to the set . The convergence rate analysis for (1.4) and (1.5) to be presented is based on this analysis of contraction property. Since (1.4) can be included by the strictly contractive PRSM scheme (1.5) if we extend the value of α = 1 and if the algebra of convergence analysis for these two schemes are of the same framework, below we only present the contraction analysis for (1.5); the analysis for (1.4) is readily obtained by taking α = 1 in our analysis.

First, to further simplify the notation in our analysis, we need to define an auxiliary sequence {w̃^k} as

$${\stackrel{\sim }{w}}^k=\left(\begin{array}{c}{\stackrel{\sim }{x}}^k\\ {\stackrel{\sim }{y}}^k\\ {\stackrel{\sim }{\lambda }}^k\end{array}\right)=\left(\begin{array}{c}{x}^{k+1}\\ y^{k+1}\\ {\lambda }^k-\beta (A{x}^{k+1}+B{y}^k-b)\end{array}\right),$$ (3.1)

where (x^k+1, y^k+1) is generated by (1.4) or (1.5). Note that with the notation of w̃^k, we immediately have

$$x^{k+1}={\stackrel{\sim }{x}}^k,y^{k+1}={\stackrel{\sim }{y}}^k,\text{and}{\lambda }^{k+\frac{1}{2}}={\lambda }^k-\alpha ({\lambda }^k-{\stackrel{\sim }{\lambda }}^k).$$ (3.2)

The strictly contractive PRSM (1.5) can be written as

$$\{\begin{array}{l}{\stackrel{\sim }{x}}^k=argmin\{{\theta }_1(x)-{({\lambda }^k)}^{T}Ax+\frac{\beta }{2}{\Vert Ax+B{y}^k-b\Vert }^2\mid x\in \mathcal{X}\},\hfill \\ {\stackrel{\sim }{y}}^k=argmin\{{\theta }_2(y)-{[{\lambda }^k-\alpha ({\lambda }^k-{\stackrel{\sim }{\lambda }}^k)]}^{T}By+\frac{\beta }{2}{\Vert A{\stackrel{\sim }{x}}^k+By-b\Vert }^2\mid y\in \mathcal{Y}\}.\hfill \end{array}$$ (3.3)

Then, based on (1.5) and (3.2), we immediately get

$$\begin{array}{l}{\lambda }^{k+1}={\lambda }^{k+\frac{1}{2}}-\alpha \beta (A{\stackrel{\sim }{x}}^k+B{\stackrel{\sim }{y}}^k-b)\\ ={\lambda }^k-\alpha ({\lambda }^k-{\stackrel{\sim }{\lambda }}^k)-\alpha [\beta (A{\stackrel{\sim }{x}}^k+B{y}^k-b)-\beta B(y^k-{\stackrel{\sim }{y}}^k)]\\ ={\lambda }^k-\alpha ({\lambda }^k-{\stackrel{\sim }{\lambda }}^k)-\alpha [({\lambda }^k-{\stackrel{\sim }{\lambda }}^k)-\beta B(y^k-{\stackrel{\sim }{y}}^k)]\\ ={\lambda }^k-[2\alpha ({\lambda }^k-{\stackrel{\sim }{\lambda }}^k)-\alpha \beta B(y^k-{\stackrel{\sim }{y}}^k)].\end{array}$$ (3.4)

Furthermore, together with y^k+1 = ỹ^k, we have the relationship

$$\left(\begin{array}{c}{y}^{k+1}\\ {\lambda }^{k+1}\end{array}\right)=\left(\begin{array}{c}{y}^k\\ {\lambda }^k\end{array}\right)-\left(\begin{array}{cc}{I}_{n_2}& 0\\ -\alpha \beta B& 2\alpha I_m\end{array}\right)\left(\begin{array}{c}{y}^k-{\stackrel{\sim }{y}}^k\\ {\lambda }^k-{\stackrel{\sim }{\lambda }}^k\end{array}\right),$$

which can be rewritten in a compact form by using the notation of v^k and ṽ^k:

$$v^{k+1}=v^k-M(v^k-{\stackrel{\sim }{v}}^k),$$ (3.5)

where M is as defined in (2.4).

Now, we start to prove some properties for the sequence {w̃^k} defined in (3.1). Recall that our primary purpose is to analyze the convergence rate for the sequences (1.4) and (1.5) based on the solution characterization (2.2), and the accuracy of an approximate solution w̃ ∈ Ω is measured by an upper bound of the quantity of θ(ũ) − θ(u) + (w̃ − w)^T F(w) for all w ∈ Ω (see (2.3)). Hence, we are interested in estimating how accurate the point w̃^k defined in (3.1) is to a solution point of VI(Ω, F, θ). The main result is proved in Theorem 3.4. But before that, we first show some lemmas. The first lemma presents an upper bound of θ(ũ) − θ(u)+(w̃ − w)^T F(w) for all w ∈ Ω in terms of a quadratic term involving the matrix Q.

Lemma 3.1

For given v^k ∈ × ℜ^m, let w^k+1 be generated by the strictly contractive PRSM scheme (1.5), and let w̃^k be as defined in (3.1). Then, we have w̃^k ∈ Ω and

$$\theta (u)-\theta ({\stackrel{\sim }{u}}^k)+{(w-{\stackrel{\sim }{w}}^k)}^{T}F({\stackrel{\sim }{w}}^k)\ge {(v-{\stackrel{\sim }{v}}^k)}^{T}Q(v^k-{\stackrel{\sim }{v}}^k)\forall w\in \mathrm{\Omega },$$ (3.6)

where the matrix Q is as defined in (2.6).

Proof

Since x^k+1 = x̃^k, by deriving the first-order optimality condition of the x-minimization problem in (3.3), we have

$${\theta }_1(x)-{\theta }_1({\stackrel{\sim }{x}}^k)+{(x-{\stackrel{\sim }{x}}^k)}^T\{A^T[\beta (A{\stackrel{\sim }{x}}^k+B{y}^k-b)-{\lambda }^k]\}\ge 0\forall x\in \mathcal{X}.$$ (3.7)

According to the definition (3.1), we have

$${\stackrel{\sim }{\lambda }}^k={\lambda }^k-\beta (A{\stackrel{\sim }{x}}^k+B{y}^k-b).$$ (3.8)

Using (3.8), the inequality (3.7) can be written as

$${\theta }_1(x)-{\theta }_1({\stackrel{\sim }{x}}^k)+{(x-{\stackrel{\sim }{x}}^k)}^T\{-A^T{\stackrel{\sim }{\lambda }}^k\}\ge 0\forall x\in \mathcal{X}.$$ (3.9)

Similarly, by deriving the first-order optimality condition of the y-minimization problem in (3.3), we get

$${\theta }_2(y)-{\theta }_2({\stackrel{\sim }{y}}^k)+{(y-{\stackrel{\sim }{y}}^k)}^T{B}^T\{\beta (A{\stackrel{\sim }{x}}^k+B{\stackrel{\sim }{y}}^k-b)-[{\lambda }^k-\alpha ({\lambda }^k-{\stackrel{\sim }{\lambda }}^k)]\}\ge 0\forall y\in \mathcal{Y}.$$ (3.10)

Again, using (3.8), we have

$$\begin{array}{l}\beta (A{\stackrel{\sim }{x}}^k+B{\stackrel{\sim }{y}}^k-b)-[{\lambda }^k-\alpha ({\lambda }^k-{\stackrel{\sim }{\lambda }}^k)]\\ =-({\stackrel{\sim }{\lambda }}^k-{\lambda }^k)+\beta B({\stackrel{\sim }{y}}^k-y^k)-[{\lambda }^k-\alpha ({\lambda }^k-{\stackrel{\sim }{\lambda }}^k)]\\ =-{\stackrel{\sim }{\lambda }}^k+\beta B({\stackrel{\sim }{y}}^k-y^k)-\alpha ({\stackrel{\sim }{\lambda }}^k-{\lambda }^k).\end{array}$$

Consequently, it follows from (3.10) that

$${\theta }_2(y)-{\theta }_2({\stackrel{\sim }{y}}^k)+{(y-{\stackrel{\sim }{y}}^k)}^T\{-B^T{\stackrel{\sim }{\lambda }}^k+\beta B^{T}B({\stackrel{\sim }{y}}^k-y^k)-\alpha B^T({\stackrel{\sim }{\lambda }}^k-{\lambda }^k)\}\ge 0\forall y\in \mathcal{Y}.$$ (3.11)

In addition, based on (3.1) we have

$$(A{\stackrel{\sim }{x}}^k+B{\stackrel{\sim }{y}}^k-b)-B({\stackrel{\sim }{y}}^k-y^k)+\frac{1}{\beta }({\stackrel{\sim }{\lambda }}^k-{\lambda }^k)=0.$$ (3.12)

Combining (3.9), (3.11), and (3.12), we get w̃^k = (x̃^k, ỹ^k, λ̃^k) ∈ Ω; and for any w = (x, y, λ) ∈ Ω, it holds that

$$\theta (u)-\theta ({\stackrel{\sim }{u}}^k)+{\left(\begin{array}{c}x-{\stackrel{\sim }{x}}^k\\ y-{\stackrel{\sim }{y}}^k\\ \lambda -{\stackrel{\sim }{\lambda }}^k\end{array}\right)}^T\left\{\left(\begin{array}{c}-A^T{\stackrel{\sim }{\lambda }}^k\\ -B^T{\stackrel{\sim }{\lambda }}^k\\ A{\stackrel{\sim }{x}}^k+B{\stackrel{\sim }{y}}^k-b\end{array}\right)+\left(\begin{array}{c}0\\ \beta B^{T}B({\stackrel{\sim }{y}}^k-y^k)-\alpha B^T({\stackrel{\sim }{\lambda }}^k-{\lambda }^k)\\ -B({\stackrel{\sim }{y}}^k-y^k)+\frac{1}{\beta }({\stackrel{\sim }{\lambda }}^k-{\lambda }^k)\end{array}\right)\right\}\ge 0.$$

The assertion (3.6) is only a compact form of the above inequality by using the notation of Q in (2.6), w and F in (2.1b), and v. The proof is complete.

Based on the optimality condition (2.1) and Lemma 3.1, we can prove the following lemma, which makes it possible to measure the accuracy of w̃^k to a solution point in by the quantity ${\Vert v^k-v^{k+1}\Vert }_H^2$. This is also an important assertion to establish a nonergodic convergence rate for the proposed strictly contractive PRSM in section 5.

Lemma 3.2

Let {w^k} be generated by the strictly contractive PRSM scheme (1.5), and let {w̃^k} be as defined in (3.1); let M, Q, and H be as defined in (2.4), (2.6), and (2.7), respectively. Then, w̃^k is a solution of VI(Ω, F, θ) if ${\Vert v^k-v^{k+1}\Vert }_H^2=0$.

Proof

By using Q = HM and M(v^k − ṽ^k) = v^k − v^k+1 (see (2.8) and (3.5)), it follows that

$${(v-{\stackrel{\sim }{v}}^k)}^{T}Q(v^k-{\stackrel{\sim }{v}}^k)={(v-{\stackrel{\sim }{v}}^k)}^{T}H(v^k-v^{k+1}).$$ (3.13)

Substituting this into (3.6), we get

$$\theta (u)-\theta ({\stackrel{\sim }{u}}^k)+{(w-{\stackrel{\sim }{w}}^k)}^{T}F({\stackrel{\sim }{w}}^k)\ge {(v-{\stackrel{\sim }{v}}^k)}^{T}H(v^k-v^{k+1})\forall w\in \mathrm{\Omega }.$$ (3.14)

Note that w̃^k ∈ Ω. Since H is positive semidefinite, in the case ${\Vert v^k-v^{k+1}\Vert }_H^2=0$, we have H(v^k − v^k+1) = 0, and thus

$${\stackrel{\sim }{w}}^k\in \mathrm{\Omega },\theta (u)-\theta ({\stackrel{\sim }{u}}^k)+{(w-{\stackrel{\sim }{w}}^k)}^{T}F({\stackrel{\sim }{w}}^k)\ge 0\forall w\in \mathrm{\Omega }.$$

According to (2.1), w̃^k is a solution of VI(Ω, F, θ).

In the next lemma, we aim at further bounding the term (v − ṽ^k)^TQ(v^k − ṽ^k) found in Lemma 3.1 by the difference of two quadratic terms involving two consecutive iterates of the sequence {v^k} and a quadratic term involving v^k and the auxiliary iterate ṽ^k. This refined bound is convenient for the manipulation over the whole sequence {v^k} recursively and thus for establishing the convergence rate of {v^k} in either the ergodic or a nonergodic sense.

Lemma 3.3

Let {w^k} be generated by the strictly contractive PRSM scheme (1.5), and let {w̃^k} be as defined in (3.1); let M, Q, and H be as defined in (2.4), (2.6), and (2.7), respectively. Then we have

$${(v-{\stackrel{\sim }{v}}^k)}^{T}Q(v^k-{\stackrel{\sim }{v}}^k)\ge \frac{1}{2}({\Vert v-v^{k+1}\Vert }_H^2-{\Vert v-v^k\Vert }_H^2)+\frac{(1-\alpha )}{4(1+\alpha )}{\Vert v^k-v^{k+1}\Vert }_H^2\forall v\in \mathcal{V}.$$ (3.15)

Proof

For the vectors a, b, c, d in the same space and a matrix H with appropriate dimensionality, we have the identity

$${(a-b)}^{T}H(c-d)=\frac{1}{2}\{{\Vert a-d\Vert }_H^2-{\Vert a-c\Vert }_H^2\}+\frac{1}{2}\{{\Vert c-b\Vert }_H^2-{\Vert d-b\Vert }_H^2\}.$$

In this identity, we take

$$a=v,b={\stackrel{\sim }{v}}^k,c=v^k,\text{and}d=v^{k+1}$$

and substitute it into the right-hand side of (3.13). The resulting equation is

$${(v-{\stackrel{\sim }{v}}^k)}^{T}Q(v^k-{\stackrel{\sim }{v}}^k)=\frac{1}{2}({\Vert v-v^{k+1}\Vert }_H^2-{\Vert v-v^k\Vert }_H^2)+\frac{1}{2}({\Vert v^k-{\stackrel{\sim }{v}}^k\Vert }_H^2-{\Vert v^{k+1}-{\stackrel{\sim }{v}}^k\Vert }_H^2).$$ (3.16)

Now, we deal with the last term of the right-hand side of (3.16). By using (3.5) and (2.8), we get

$$\begin{array}{l}{\Vert v^k-{\stackrel{\sim }{v}}^k\Vert }_H^2-{\Vert v^{k+1}-{\stackrel{\sim }{v}}^k\Vert }_H^2\\ ={\Vert v^k-{\stackrel{\sim }{v}}^k\Vert }_H^2-{\Vert (v^k-{\stackrel{\sim }{v}}^k)-(v^k-v^{k+1})\Vert }_H^2\\ \stackrel{(3.5)}{=}{\Vert v^k-{\stackrel{\sim }{v}}^k\Vert }_H^2-{\Vert (v^k-{\stackrel{\sim }{v}}^k)-M(v^k-{\stackrel{\sim }{v}}^k)\Vert }_H^2\\ =2{(v^k-{\stackrel{\sim }{v}}^k)}^{T}HM(v^k-{\stackrel{\sim }{v}}^k)-{(v^k-{\stackrel{\sim }{v}}^k)}^T{M}^{T}HM(v^k-{\stackrel{\sim }{v}}^k)\\ \stackrel{(2.8)}{=}{(v^k-{\stackrel{\sim }{v}}^k)}^T(Q^T+Q-M^{T}HM)(v^k-{\stackrel{\sim }{v}}^k)\\ \stackrel{(2.9)}{\ge }\frac{(1-\alpha )}{2(1+\alpha )}{(v^k-{\stackrel{\sim }{v}}^k)}^T{M}^{T}HM(v^k-{\stackrel{\sim }{v}}^k)\\ \stackrel{(3.5)}{=}\frac{(1-\alpha )}{2(1+\alpha )}{\Vert v^k-v^{k+1}\Vert }_H^2.\end{array}$$

Substituting into (3.16), we obtain the assertion (3.15). The proof is complete.

Now we are ready to present an inequality where an upper bound of θ(ũ^k) − θ(u)+ (w̃^k − w)^T F(w) is found for all w ∈ Ω. This inequality is also crucial for analyzing the contraction property and the convergence rate for the iterative sequence generated by either (1.4) or (1.5).

Theorem 3.4

For given v^k ∈ × ℜ^m, let w^k+1 be generated by the strictly contractive PRSM scheme (1.5), and let w̃^k be as defined in (3.1); let M and H be as defined in (2.4) and (2.7), respectively. Then, we have w̃^k ∈ Ω and

$$\begin{array}{l}\theta (u)-\theta ({\stackrel{\sim }{u}}^k)+{(w-{\stackrel{\sim }{w}}^k)}^{T}F(w)\\ \ge \frac{1}{2}({\Vert v-v^{k+1}\Vert }_H^2-{\Vert v-v^k\Vert }_H^2)+\frac{1-\alpha }{4(1+\alpha )}{\Vert v^k-v^{k+1}\Vert }_H^2\forall w\in \mathrm{\Omega }.\end{array}$$ (3.17)

Proof

First, because of the monotonicity of F(w), we have

$${(w-\stackrel{\sim }{w})}^T(F(w)-F(\stackrel{\sim }{w}))\ge 0\forall w,\stackrel{\sim }{w}\in \mathrm{\Omega }.$$

Then, using the above inequality and replacing the right-hand side term in (3.6) with the inequality (3.15), we obtain the assertion (3.17). The proof is complete.

The assertion (3.17) also enables us to study the contraction property of the sequence {v^k} generated by (1.4) or (1.5). In fact, setting w = w^* in (3.17) where w^* is an arbitrary solution point in Ω^*, we get

$${\Vert v^k-v^{\ast }\Vert }_H^2-{\Vert v^{k+1}-v^{\ast }\Vert }_H^2\ge \frac{1-\alpha }{2(1+\alpha )}{\Vert v^k-v^{k+1}\Vert }_H^2+2\{\theta ({\stackrel{\sim }{u}}^k)-\theta (u^{\ast })+{({\stackrel{\sim }{w}}^k-w^{\ast })}^{T}F(w^{\ast })\}.$$ (3.18)

Recall the optimality in (2.1). We thus have

$${\Vert v^{k+1}-v^{\ast }\Vert }_H^2\le {\Vert v^k-v^{\ast }\Vert }_H^2-\frac{1-\alpha }{2(1+\alpha )}{\Vert v^k-v^{k+1}\Vert }_H^2\forall v^{\ast }\in {\mathcal{V}}^{\ast }.$$ (3.19)

Therefore, when α = 1, i.e., for the PRSM scheme (1.4), we have

$${\Vert v^{k+1}-v^{\ast }\Vert }_H^2\le {\Vert v^k-v^{\ast }\Vert }_H^2,$$ (3.20)

which means the sequence {v^k} generated by (1.4) is contractive, but not strictly, to the set . In fact, it is possible that the sequence {v^k} stays away from the solution set with a constant distance (i.e., the equivalence in (3.20) holds for any k); hence no convergence of (1.4) is guaranteed under our assumption on (1.1). In [12] and [16], such an example was shown. On the other hand, when α ∈ (0, 1), the inequality (3.19) ensures a reduction of $\frac{1-\alpha }{2(1+\alpha )}{\Vert v^k-v^{k+1}\Vert }_H^2$ to the set at the (k + 1)th iteration; i.e., the strict contraction of {v^k} is guaranteed for the sequence generated by (1.5). Recall Lemma 3.2, which indicates that ${\Vert v^k-v^{k+1}\Vert }_H^2\ne 0$ whenever a solution is not yet found. Thus, the inequality (3.19) implies that the sequence {v^k} generated by the proposed strictly contractive PRSM (1.5) converges to with a guaranteed reduction of proximity to the solution set. As we have mentioned, the difference of contraction between (3.19) and (3.20) is also the reason we can establish a nonergodic convergence rate for the strictly contractive PRSM (1.5) in section 5 while only the ergodic convergence rate can be established for the original PRSM (1.4) in section 4.

4. Convergence rate of (1.4) in the ergodic sense

In this section, we show that although the original PRSM (1.4) might not be convergent to a solution point of the model (1.1), it is still possible to find an approximate solution of VI(Ω, F, θ) with an accuracy of O(1/t) based on the first t iterations of the PRSM scheme (1.4). This estimate helps us better understand the convergence property of the original PRSM (1.4).

Theorem 4.1

Let {w^k} be generated by PRSM (1.4) and {w̃^k} be defined by (3.1). Let w̃_t be defined as

$${\stackrel{\sim }{w}}_t=\frac{1}{t+1}\sum _{k=0}^t{\stackrel{\sim }{w}}^k.$$ (4.1)

Then, for any integer number t > 0, w̃_t ∈ Ω and

$$\theta ({\stackrel{\sim }{u}}_t)-\theta (u)+{({\stackrel{\sim }{w}}_t-w)}^{T}F(w)\le \frac{1}{2(t+1)}{\Vert v-v^0\Vert }_H^2\forall w\in \mathrm{\Omega },$$ (4.2)

where H is as defined in (2.7).

Proof

First, because of (3.1), it holds that w̃^k ∈ Ω for all k ≥ 0. Together with the convexity of and , (4.1) implies that w̃_t ∈ Ω. Second, by taking α = 1 in (3.17) we have

$$\theta (u)-\theta ({\stackrel{\sim }{u}}^k)+{(w-{\stackrel{\sim }{w}}^k)}^{T}F(w)+\frac{1}{2}{\Vert v-v^k\Vert }_H^2\ge \frac{1}{2}{\Vert v-v^{k+1}\Vert }_H^2\forall w\in \mathrm{\Omega }.$$ (4.3)

Summing the inequality (4.3) over k = 0, 1, …, t, we obtain

$$(t+1)\theta (u)-\sum _{k=0}^t\theta ({\stackrel{\sim }{u}}^k)+{\left((t+1)w-\sum _{k=0}^t{\stackrel{\sim }{w}}^k\right)}^{T}F(w)+\frac{1}{2}{\Vert v-v^0\Vert }_H^2\ge 0\forall w\in \mathrm{\Omega }.$$

Using the notation of w̃_t, it can be written as

$$\frac{1}{t+1}\sum _{k=0}^t\theta ({\stackrel{\sim }{u}}^k)-\theta (u)+{({\stackrel{\sim }{w}}_t-w)}^{T}F(w)\le \frac{1}{2(t+1)}{\Vert v-v^0\Vert }_H^2\forall w\in \mathrm{\Omega }.$$ (4.4)

Since θ(u) is convex and

$${\stackrel{\sim }{u}}_t=\frac{1}{t+1}\sum _{k=0}^t{\stackrel{\sim }{u}}^k,$$

we have that

$$\theta ({\stackrel{\sim }{u}}_t)\le \frac{1}{t+1}\sum _{k=0}^t\theta ({\stackrel{\sim }{u}}^k).$$

Substituting this into (4.4), the assertion of this theorem follows directly.

Let v⁰ = (y⁰, λ⁰) be the initial iterate. For a given compact set ⊂ × ℜ^m, let d = sup{||v − v⁰||_H | v ∈ }. Then, after t iterations of the PRSM (1.4), the point w̃_t ∈ Ω defined in (4.1) satisfies

$$\underset{w\in \mathcal{D}}{sup}\{\theta ({\stackrel{\sim }{u}}_t)-\theta (u)+{({\stackrel{\sim }{w}}_t-w)}^{T}F(w)\}\le \frac{d^2}{2(t+1)},$$

which means w̃_t is an approximate solution of VI(Ω, F, θ) with an accuracy of O(1/t) (recall (2.3)).

Remark 4.1

In the proof of Theorem 4.1, we take α = 1 in (4.3). Obviously, the proof is still valid if we take α ∈ (0, 1). Thus, a worst-case O(1/t) convergence rate in the ergodic sense can be established easily for the strictly contractive PRSM (1.5). As we shall show in section 5, this is less interesting because a nonergodic worst-case O(1/t) convergence rate can be established for (1.5). We thus omit the details.

5. Convergence rate of (1.5) in a nonergodic sense

In this section, we show that the sequence {v^k} generated by the strictly contractive PRSM scheme (1.5) is convergent to a point in , and its worst-case convergence rate is O(1/t) in a nonergodic sense. Our starting point for the analysis is the inequality (3.19), and a crucial property is the monotonicity of the sequence { ${\Vert v^k-v^{k+1}\Vert }_H^2$}. That is, we will prove that

$${\Vert v^{k+1}-v^{k+2}\Vert }_H^2\le {\Vert v^k-v^{k+1}\Vert }_H^2\forall k\ge 0.$$

We first take a closer look at the assertion (3.6) in Lemma 3.1.

Lemma 5.1

Let {w^k} be the sequence generated by the strictly contractive PRSM (1.5), let w̃^k be as defined in (3.1), and let the matrix Q be as defined in (2.6). Then, we have

$${({\stackrel{\sim }{v}}^k-{\stackrel{\sim }{v}}^{k+1})}^{T}Q\{(v^k-{\stackrel{\sim }{v}}^k)-(v^{k+1}-{\stackrel{\sim }{v}}^{k+1})\}\ge 0.$$ (5.1)

Proof

Setting w = w̃^k+1 in (3.6), we have

$$\theta ({\stackrel{\sim }{u}}^{k+1})-\theta ({\stackrel{\sim }{u}}^k)+{({\stackrel{\sim }{w}}^{k+1}-{\stackrel{\sim }{w}}^k)}^{T}F({\stackrel{\sim }{w}}^k)\ge {({\stackrel{\sim }{v}}^{k+1}-{\stackrel{\sim }{v}}^k)}^{T}Q(v^k-{\stackrel{\sim }{v}}^k).$$ (5.2)

Note that (3.6) is also true for k := k + 1, and thus

$$\theta (u)-\theta ({\stackrel{\sim }{u}}^{k+1})+{(w-{\stackrel{\sim }{w}}^{k+1})}^{T}F({\stackrel{\sim }{w}}^{k+1})\ge {(v-{\stackrel{\sim }{v}}^{k+1})}^{T}Q(v^{k+1}-{\stackrel{\sim }{v}}^{k+1})\forall w\in \mathrm{\Omega }.$$

Setting w = w̃^k in the above inequality, we obtain

$$\theta ({\stackrel{\sim }{u}}^k)-\theta ({\stackrel{\sim }{u}}^{k+1})+{({\stackrel{\sim }{w}}^k-{\stackrel{\sim }{w}}^{k+1})}^{T}F({\stackrel{\sim }{w}}^{k+1})\ge {({\stackrel{\sim }{v}}^k-{\stackrel{\sim }{v}}^{k+1})}^{T}Q(v^{k+1}-{\stackrel{\sim }{v}}^{k+1}).$$ (5.3)

Adding (5.2) and (5.3) and using the monotonicity of F, we get (5.1) immediately.

Lemma 5.2

Let {w^k} be the sequence generated by the strictly contractive PRSM (1.5); let the matrices M, Q, and H be as defined in (2.4), (2.6), and (2.7), respectively. Then, we have

$${(v^k-v^{k+1})}^{T}H\{(v^k-v^{k+1})-(v^{k+1}-v^{k+2})\}\ge \frac{3+\alpha }{4(1+\alpha )}{\Vert (v^k-v^{k+1})-(v^{k+1}-v^{k+2})\Vert }_H^2.$$ (5.4)

Proof

Adding the equation

$${\{(v^k-{\stackrel{\sim }{v}}^k)-(v^{k+1}-{\stackrel{\sim }{v}}^{k+1})\}}^{T}Q\{(v^k-{\stackrel{\sim }{v}}^k)-(v^{k+1}-{\stackrel{\sim }{v}}^{k+1})\}=\frac{1}{2}{\Vert (v^k-{\stackrel{\sim }{v}}^k)-(v^{k+1}-{\stackrel{\sim }{v}}^{k+1})\Vert }_{(Q^T+Q)}^2$$

to both sides of (5.1), we get

$${(v^k-v^{k+1})}^{T}Q\{(v^k-{\stackrel{\sim }{v}}^k)-(v^{k+1}-{\stackrel{\sim }{v}}^{k+1})\}\ge \frac{1}{2}{\Vert (v^k-{\stackrel{\sim }{v}}^k)-(v^{k+1}-{\stackrel{\sim }{v}}^{k+1})\Vert }_{(Q^T+Q)}^2.$$ (5.5)

By using Q = HM and M(v^k − ṽ^k) = v^k − v^k+1 (see (2.8) and (3.5)) in the left-hand side of (5.5), we obtain

$${(v^k-v^{k+1})}^{T}H\{(v^k-v^{k+1})-(v^{k+1}-v^{k+2})\}\ge \frac{1}{2}{\Vert (v^k-{\stackrel{\sim }{v}}^k)-(v^{k+1}-{\stackrel{\sim }{v}}^{k+1})\Vert }_{(Q^T+Q)}^2.$$ (5.6)

Due to (2.9) we have

$$Q^T+Q\succcurlyeq \frac{(3+\alpha )}{2(1+\alpha )}{M}^{T}HM.$$

Substituting this into the right-hand side of (5.6) and using M(v^k − ṽ^k) = v^k − v^k+1 again, we obtain (5.4), and the lemma is proved.

Now, we are ready to prove the monotonicity of the sequence { ${\Vert v^k-v^{k+1}\Vert }_H^2$}.

Theorem 5.3

Let {w^k} be the sequence generated by the strictly contractive PRSM (1.5), let w̃^k be as defined in (3.1), and let the matrix H be as defined in (2.7). Then, we have

$${\Vert v^{k+1}-v^{k+2}\Vert }_H^2\le {\Vert v^k-v^{k+1}\Vert }_H^2-\frac{1-\alpha }{2(1+\alpha )}{\Vert (v^k-v^{k+1})-(v^{k+1}-v^{k+2})\Vert }_H^2.$$ (5.7)

Proof

Setting a = (v^k − v^k+1) and b = (v^k+1 − v^k+2) in the identity

$${\Vert a\Vert }_H^2-{\Vert b\Vert }_H^2=2a^{T}H(a-b)-{\Vert a-b\Vert }_H^2,$$

we obtain

$$\begin{array}{l}{\Vert v^k-v^{k+1}\Vert }_H^2-{\Vert v^{k+1}-v^{k+2}\Vert }_H^2\\ =2{(v^k-v^{k+1})}^{T}H\{(v^k-v^{k+1})-(v^{k+1}-v^{k+2})\}-{\Vert (v^k-v^{k+1})-(v^{k+1}-v^{k+2})\Vert }_H^2.\end{array}$$

Inserting (5.4) into the first term of the right-hand side of the last equality, we obtain

$${\Vert v^k-v^{k+1}\Vert }_H^2-{\Vert v^{k+1}-v^{k+2}\Vert }_H^2\ge \frac{1-\alpha }{2(1+\alpha )}{\Vert (v^k-v^{k+1})-(v^{k+1}-v^{k+2})\Vert }_H^2.$$

The assertion (5.7) follows from the above inequality directly, and the proof is complete.

Now, we can establish a worst-case O(1/t) convergence rate in a nonergodic sense for the strictly contractive PRSM scheme (1.5).

Theorem 5.4

Let {w^t} be the sequence generated by the strictly contractive PRSM scheme (1.5). For any v^* ∈ , we have

$${\Vert v^t-v^{t+1}\Vert }_H^2\le \frac{2(1+\alpha )}{(t+1)(1-\alpha )}{\Vert v^0-v^{\ast }\Vert }_H^2.$$ (5.8)

Proof

First, it follows from (3.19) that

$$\frac{1-\alpha }{2(1+\alpha )}\sum _{k=0}^{\infty }{\Vert v^k-v^{k+1}\Vert }_H^2\le {\Vert v^0-v^{\ast }\Vert }_H^2\forall v^{\ast }\in {\mathcal{V}}^{\ast }.$$ (5.9)

According to Theorem 5.3, the sequence { ${\Vert v^k-v^{k+1}\Vert }_H^2$} is monotonically non-increasing. Therefore, we have

$$(t+1){\Vert v^t-v^{t+1}\Vert }_H^2\le \sum _{k=0}^t{\Vert v^k-v^{k+1}\Vert }_H^2.$$ (5.10)

The assertion (5.8) follows from (5.9) and (5.10) immediately. The proof is complete.

Notice that is convex and closed. Let v⁰ = (y⁰, λ⁰) be the initial iterate and d := inf{||v⁰ − v^*||_H | v^* ∈ }. Then, for any given ε > 0, Theorem 5.4 shows that the strictly contractive PRSM scheme (1.5) needs at most $k=\lfloor \frac{2(1+\alpha )d^2}{(1-\alpha )\epsilon }\rfloor$ iterations to ensure that ${\Vert v^k-v^{k+1}\Vert }_H^2\le \epsilon$. It follows from Lemma 3.2 that w^k+1 is a solution of VI(Ω, F, θ) if ${\Vert v^k-v^{k+1}\Vert }_H^2=0$. A worst-case O(1/t) convergence rate in a nonergodic sense for the strictly contractive PRSM scheme (1.5) is thus established in Theorem 5.4.

6. Numerical results

In this section, we verify the theoretical assertions analyzed in previous sections by some numerical experiments. We focus on some applications of the abstract model (1.1) in statistical learning and image processing, including the least absolute shrinkage and selection operator (LASSO) model, the group LASSO model, the sparse logistic regression model, the image deblurring problem, the image inpainting problem, and the magnetic resonance imaging (MRI) problem. We shall verify the following assertions.

1. The original PRSM (1.4) is indeed fast if it is convergent; the assertions in [20, 23] are thus further backed up.

2. For some scenarios of (1.1), the original PRSM (1.4) might fail to converge while the proposed strictly contractive PRSM (1.5) is convergent; the theoretical significance of the underdetermined relaxation factor α is thus verified.

3. For the proposed strictly contractive PRSM (1.5), the underdetermined relaxation factor α can be easily determined. In fact, empirically, α ∈ [0.8, 0.9] is preferred for all tested cases.

4. The proposed strictly contractive PRSM (1.5) converges quickly for a wide range of applications, and it numerically outperforms some efficient methods in the literature.

We shall use the previously mentioned statistical leaning models to illustrate the first three assertions, and the last assertion will be verified by the previously mentioned imaging models. Our code was written by MATLAB 2012a, and all the numerical experiments were conducted on a laptop computer with a 2.9GHz i7 processor and an 8GB memory.

6.1. Statistical learning problems

In this subsection, we test some popular sparse learning models in the area of statistical learning. As we have mentioned, via these models our purpose is to justify the advantages of the proposed strictly contractive PRSM (1.5) itself. We thus choose the ADMM (1.3) as the only benchmark for numerical comparison in this subsection. Note that the ADMM (1.3) has been shown to be a widely applicable efficient solver for some popular statistical learning problems; see, e.g., [6] for a review.

6.1.1. Models and iterative schemes

We first introduce the sparse learning models to be studied.

1. The LASSO model proposed in [51],

   $$min\left\{\frac{1}{2}{\Vert Dx-r\Vert }_2^2+\gamma {\Vert x\Vert }_1\mid x\in {\Re }^d\mid \right\},$$ (6.1)

   where r ∈ ℜⁿ is the response vector, D ∈ ℜ^n×d is the design matrix, n is the number of data points, d is the number of features, γ > 0 is a regularization parameter, and ${\Vert x\Vert }_1:={\sum }_{i=1}^n\mid x_i\mid$. The LASSO model provides a sparse estimation of x when there are more features than data points (i.e., d > n), and it has been very influential in several areas (e.g., bioinformatics [53, 40, 41], econometrics [7], and climate analysis [10]).

2. The group LASSO model proposed in [55],

   $$min\left\{\frac{1}{2}{\Vert Dx-r\Vert }_2^2+\gamma \sum _{i=1}^N{\Vert x_i\Vert }_2\mid x\in {\Re }^d,x_i\in {\Re }^{d_i}\right\},$$ (6.2)

   where ${\sum }_{i=1}^N{d}_i=d$, N represents the number of disjointed groups partitioned among the variable x, and all other settings are the same as in (6.1). The model (6.2) promotes selecting the grouped variables (factors), which are common in multifactor analysis-of-variance (ANOVA) problems or additive models with polynomial or nonparametric components. In these problems, important factors are groups of variables rather than the individual derived variables. Group LASSO reduces to LASSO when d_i = 1, which means each group contains only one variable.

3. The sparse logistic regression model in [45, 34, 35],

   $$min\left\{\frac{1}{n}\sum _{i=1}^{n}log\left(1+exp(-r_i(D_i^{T}x+x_0))\right)+\gamma {\Vert x\Vert }_1,\mid x\in {\Re }^d,x_0\in \Re \right\},$$ (6.3)
   where x ∈ ℜ^d is the coefficient vector, x₀ ∈ ℜ is the intercept scalar, D_i ∈ ℜ^d (i = 1, 2, …, n) are training data points, r_i ∈ {−1, 1} (i = 1, 2, …, n) are corresponding labels, n is the number of data points, d is the dimension of data, and γ > 0 is a regularization parameter. After obtaining the coefficient x and x₀ in (6.3), we can predict a binomial categorical label r_pred with the given input D_in ∈ ℜ^d by

   $$r_{\text{pred}}=\text{sign}\left(\frac{1}{1+e^{-(x_0+D_{\text{in}}^{T}x)}}\right).$$

Now we illustrate how to implement the ADMM (1.3), the PRSM (1.4), and the strictly contractive PRSM (1.5) for solving these statistical learning models. First, for the LASSO model (6.1), by introducing an auxiliary variable y ∈ ℜ^d, the model (6.1) can be reformulated as

$$min\left\{\frac{1}{2}{\Vert Dx-r\Vert }_2^2+\gamma {\Vert y\Vert }_1\mid x-y=0,x\in {\Re }^d,y\in {\Re }^d\right\},$$ (6.4)

which is a special case of (1.1) where $f(x)=\frac{1}{2}{\Vert Dx-r\Vert }_2^2$, g(y) = γ||y||₁, n₁ = n₂ = d, A = I_d, B = −I_d, b = 0, = ℜ^d, and = ℜ^d. Therefore, the iterative scheme of the ADMM (1.3) for (6.4) is

$$\{\begin{array}{l}{x}^{k+1}={(D^{T}D+\beta I)}^{-1}(D^{T}r+\beta y^k+{\lambda }^k),\hfill \\ y^{k+1}=S_{\gamma /\beta }(x^{k+1}-{\lambda }^k/\beta ),\hfill \\ {\lambda }^{k+1}={\lambda }^{k+\frac{1}{2}}-\beta (x^{k+1}-y^{k+1}).\hfill \end{array}$$ (6.5)

Moreover, the iterative schemes of the PRSM (1.4) and the strictly contractive PRSM (1.5) for (6.4) read, respectively, as

$$\{\begin{array}{l}{x}^{k+1}={(D^{T}D+\beta I)}^{-1}(D^{T}r+\beta y^k+{\lambda }^k),\hfill \\ {\lambda }^{k+\frac{1}{2}}={\lambda }^k-\beta (x^{k+1}-y^k),\hfill \\ y^{k+1}=S_{\gamma /\beta }(x^{k+1}-{\lambda }^{k+\frac{1}{2}}/\beta ),\hfill \\ {\lambda }^{k+1}={\lambda }^{k+\frac{1}{2}}-\beta (x^{k+1}-y^{k+1})\hfill \end{array}$$ (6.6)

and

$$\{\begin{array}{l}{x}^{k+1}={(D^{T}D+\beta I)}^{-1}(D^{T}r+\beta y^k+{\lambda }^k),\hfill \\ {\lambda }^{k+\frac{1}{2}}={\lambda }^k-\alpha \beta (x^{k+1}-y^k),\hfill \\ y^{k+1}=S_{\gamma /\beta }(x^{k+1}-{\lambda }^{k+\frac{1}{2}}/\beta ),\hfill \\ {\lambda }^{k+1}={\lambda }^{k+\frac{1}{2}}-\alpha \beta (x^{k+1}-y^{k+1}).\hfill \end{array}$$ (6.7)

In (6.5)–(6.7), S_κ(a) is the soft-thresholding operator defined as

$${(S_{\kappa }(a))}_i={(1-\kappa /\mid a_i\mid )}_{+}·a_i,i=1,\dots ,d,$$

for κ > 0 and a ∈ ℜ^d; see, e.g., [14].

Similarly, by introducing y_i = x_i for i = 1, 2, …, N, we can reformulate the group LASSO model (6.2) as

$$min\left\{\frac{1}{2}{\Vert Dx-r\Vert }_2^2+\gamma \sum _{i=1}^N{\Vert y_i\Vert }_2\mid x_i-y_i=0,i=1,2,\dots ,N;x_i\in {\Re }^{d_i},y_i\in {\Re }^{d_i}\right\},$$ (6.8)

for which the ADMM (1.3), the PRSM (1.4), and the strictly contractive PRSM (1.5) are implementable. The resulting iterative schemes are, respectively, as follows:

$$\{\begin{array}{l}{x}^{k+1}={(D^{T}D+\beta I)}^{-1}(D^{T}r+\beta y^k+{\lambda }^k),\hfill \\ y_i^{k+1}=S_{\gamma /\beta }(x_i^{k+1}-\beta {\lambda }_i^k),i=1,\dots ,N,\hfill \\ {\lambda }^{k+1}={\lambda }^{k+\frac{1}{2}}-\beta (x^{k+1}-y^{k+1}),\hfill \end{array}$$ (6.9)

$$\{\begin{array}{l}{x}^{k+1}={(D^{T}D+\beta I)}^{-1}(D^{T}r+\beta y^k+{\lambda }^k),\hfill \\ {\lambda }^{k+\frac{1}{2}}={\lambda }^k-\beta (x^{k+1}-y^k),\hfill \\ y_i^{k+1}=S_{\gamma /\beta }(x_i^{k+1}-\beta {\lambda }_i^{k+\frac{1}{2}}),i=1,\dots ,N,\hfill \\ {\lambda }^{k+1}={\lambda }^{k+\frac{1}{2}}-\beta (x^{k+1}-y^{k+1}),\hfill \end{array}$$ (6.10)

and

$$\{\begin{array}{l}{x}^{k+1}={(D^{T}D+\beta I)}^{-1}(D^{T}r+\beta y^k+{\lambda }^k),\hfill \\ {\lambda }^{k+\frac{1}{2}}={\lambda }^k-\alpha \beta (x^{k+1}-y^k),\hfill \\ y_i^{k+1}=S_{\gamma /\beta }(x_i^{k+1}-\beta {\lambda }_i^{k+\frac{1}{2}}),i=1,\dots ,N,\hfill \\ {\lambda }^{k+1}={\lambda }^{k+\frac{1}{2}}-\alpha \beta (x^{k+1}-y^{k+1}).\hfill \end{array}$$ (6.11)

Finally, the sparse logistic regression model (6.3) can be reformulated as

$$min\left\{\frac{1}{n}\sum _{i=1}^{n}log(1+exp(-r_i(D_i^{T}x+x_0))+\gamma {\Vert y\Vert }_1)\mid x-y=0,x_0\in \Re ,x\in {\Re }^d,y\in {\Re }^d\right\},$$ (6.12)

where y ∈ ℜ^d is an auxiliary variable. Therefore, the ADMM (1.3), the PRSM (1.4), and the strictly contractive PRSM (1.5) are all applicable to (6.12). For succinctness, we list only the iterative scheme of the strictly contractive PRSM (1.5) in detail:

$$\{\begin{array}{l}{x}^{k+1}=arg{min}_x\frac{1}{n}{\sum }_{i=1}^{n}log(1+exp(-r_i(D_i^{T}x+x_0)))+\frac{\beta }{2}{\Vert x-y^k-{\lambda }^k/\beta \Vert }_2^2,\hfill \\ {\lambda }^{k+\frac{1}{2}}={\lambda }^k-\alpha \beta (x^{k+1}-y^k),\hfill \\ y_i^{k+1}=S_{\gamma /\beta }(x_i^{k+1}-\beta {\lambda }_i^{k+\frac{1}{2}}),i=1,\dots ,N,\hfill \\ {\lambda }^{k+1}={\lambda }^{k+\frac{1}{2}}-\alpha \beta (x^{k+1}-y^{k+1}).\hfill \end{array}$$ (6.13)

Note that in (6.13) the x-subproblem has no closed-form solutions. We implement Newton’s method (coded by MATLAB) to solve this subproblem with a tolerance of 10⁻⁵ and a maximum iteration number of 10.

6.1.2. Implementation details

Now we specify the setting for the sparse learning models to be tested. For the LASSO model (6.1), we set n = 2000 and d = 4000; each entry of D is drawn from (0, 1), and then all columns of D are normalized; we generate a random sparse vector in ℜ⁴⁰⁰⁰ with 100 nonzero entries from (0, 1) as x; the noise vector ε ~ (0, 10⁻³I) and the vector r = Dx + ε; the regularization parameter is set as γ = 0.1||D^Tr||_∞. For the group LASSO model (6.2), we set n = 1500; D is also drawn from (0, 1) and normalized; the noise vector ε ~ (0, 10⁻³I) and the vector r = Dx + ε; we generate N = 200 blocks with size n_i uniformly distributed between 1 and 50, $d={\sum }_{i=1}^N{n}_i$. Among all the blocks, 5% have entries drawn from the standard Gaussian distribution, and the other blocks have entries all equal to zero; the regularization parameter γ is set as $\gamma =0.1max\{{\Vert D_1^{T}r\Vert }_2,\dots ,{\Vert D_N^{T}r\Vert }_2\}$ , where {D₁, D₂, …, D_N} with D_i ∈ ℜ^n×d_iis a disjointed partition of D’s columns in correspondence with the partition of {x₁, x₂, …, x_N}. For the sparse logistic regression model (6.3), we set n = 50 and d = 500; each vector D_i ∈ ℜ⁵⁰⁰ has 10 nonzero entries from (0, 1); the vectors r_i are generated by $r_i=\text{sign}(D_i^{T}x+x_0+{\epsilon }_i)$, where ε_i is the noise drawn from (0, 0.1); the vector x ∈ ℜ^d contains 100 nonzero entries drawn from (0, 1); the intercept x₀ is also from (0, 1), and γ is set according to [34]: γ = 0.1 γ_max, where γ_max is the maximum regularization parameter above which the solution x has all zero entries.

Then, we provide some details to implement the methods to be tested. Our code was constructed based on the code available at http://www.stanford.edu/~boyd/papers/admm/. Hence, the ADMM (1.3) can be implemented directly by this code package, while the implementation of the PRSM (1.4) and the strictly contractive PRSM (1.5) require only a slight modification on the Lagrange multiplier to this package. It is worth mentioning that in (6.5)–(6.11), we need to compute (D^T D + βI)⁻¹ and D^T r, which is quite time consuming if N and d are large. However, since these two terms are invariant in each iteration, we need only compute it once before all iterations. We define the stopping criterion as

$$max\left\{\beta {\Vert y^k-y^{k+1}\Vert }_2,\frac{1}{\beta }{\Vert {\lambda }^k-{\lambda }^{k+1}\Vert }_2\right\}\le \sqrt{d}\epsilon ,$$ (6.14)

where β||y^k − y^k+1||₂ and $\frac{1}{\beta }{\Vert {\lambda }^k={\lambda }^{k+1}\Vert }_2$ measure the primal and dual residuals, respectively, and ε > 0 is a tolerance; see, e.g., [6, 30, 56]. Note that because of Lemma 3.2 it is also reasonable to use this stopping criterion for the PRSM (1.4) and the strictly contractive PRSM (1.5). For LASSO (6.1) and group LASSO (6.2), we set ε = 10⁻⁴; and for the sparse logistic regression model (6.3), we set ε = 10⁻³ since its x-subproblem is solved approximately at each iteration. For the penalty parameter β, we set it as 1 for all methods; and we set α = 0.9 for the strictly contractive PRSM (1.4) (the reason will be explained later).

6.1.3. Results

In Table 1, we report the computing time in seconds (“time(s)”) and the number of iterations when the ADMM (1.3), the PRSM (1.4), and the strictly contractive PRSM (1.5) are applied to solve the above-mentioned statistical learning models. According to this table, we see that the original PRSM (1.4) is convergent only for the group LASSO model (6.2). For this case, the original PRSM (1.4) is really faster than the ADMM (1.3), and it is almost as efficient as the strictly contractive PRSM (1.4). The assertions in [20, 23] are thus verified again. For the other two models, however, the convergence of PRSM is not witnessed. But the proposed strictly contractive PRSM (1.4) still performs well—faster than ADMM.

Table 1.

Quantitative comparison among the strictly contractive PRSM, the original PRSM, and the ADMM for statistical learning models.

Algorithm	LASSO		Group LASSO		Sparse logistic regression
	time(s)	# iterations	time(s)	# iterations	time(s)	# iterations
Strictly contractive PRSM	1.27	53	0.53	21	2.14	41
ADMM	2.13	89	0.96	36	3.75	72
PRSM	-	-	0.53	23	-	-

“-” means that the stopping criterion $max\{\beta {\Vert y^k-y^{k+1}\Vert }_2,\frac{1}{\beta }{\Vert {\lambda }^k-{\lambda }^{k+1}\Vert }_2\}\le \sqrt{d}\epsilon$ is not satisfied after 10,000 iterations.

To further observe the convergence of the ADMM and the strictly contractive PRSM, in Figures 1–3 we visualize the evolution of convergence when these two methods are applied to solve these three sparse learning models. The evolution of the objective function value, the reduction of primal and dual residuals, and ||v^(k) − v^*||_H with respect to the iterations are plotted. Plots in Figure 1 show that the strictly contractive PRSM and ADMM reach the primal tolerance at almost the same time, but the former reaches the dual tolerance faster than the latter. Plots in Figure 2 indicate that the strictly contractive PRSM reaches both the primal and dual tolerances faster than ADMM. Last, plots in Figure 3 reveal that ADMM reaches the primal tolerance faster (but possibly jumps back to above the tolerance again), while the strictly contractive PRSM reaches the dual tolerance much faster than ADMM. The lower right plots in Figure 1–3 illustrate that the sequence ||v^(k) − v^*||_H decreases monotonically, which further back up our theoretical results in (3.19).

Fig. 1.

LASSO model (6.1): Evolution of the objective function value, primal residual, and dual residual, and ||v^(k) − v^*||_H for ADMM and the strictly contractive PRSM.

Fig. 3.

Sparse logistic regression model (6.3): Evolution of the objective function, primal residual, and dual residual, and ||v^(k) − v^*||_H for ADMM and the strictly contractive PRSM.

Fig. 2.

Group LASSO model (6.2): Evolution of the objective function, primal residual, and dual residual, and ||v^(k) − v^*||_H for ADMM and the strictly contractive PRSM.

6.1.4. Sensitivity to α

As we have analyzed, attaching an underdetermined relaxation factor α ∈ (0, 1) to the original PRSM (1.4) can make the resulting iterative sequence strictly contractive with respect to the solution set of (1.1). Thus, it becomes possible to ensure the convergence and establish a worst-case O(1/t) convergence rate in a nonergodic sense for the strictly contractive PRSM (1.5). Despite its significant theoretical role, we would emphasize that this underdetermined relaxation factor can be chosen easily to implement the strictly contractive PRSM (1.5). This is an important convenience for the implementation of the strictly contractive PRSM (1.5).

In this subsection, we take the LASSO model (6.1) to test the sensitivity of α for the strictly contractive PRSM (1.5). We fix β = 1 and choose different values of α in the interval [0.05, 0.99]. (More specifically, we choose α = {0.05, 0.10, 0.15, …, 0.85, 0.90, 0.91, 0.92, …, 0.98, 0.99}.) The computing time and number of iterations required by the strictly contractive PRSM (1.5) are recorded for each choice of α. Then, we plot them in Figure 4. For comparison purposes, we also plot for ADMM with β = 1. According to the curves in Figure 4, we see that the underdetermined relaxation factor α works for a wide range of values; thus it can be chosen easily in implementation. In particular, based on our experiments, some aggressive values close to 1 (e.g., [0.8, 0.9]) are preferred.

Fig. 4.

Sensitivity test on the underdetermined relaxation factor α when β = 1.

As is well known in the literature, the numerical performance of some augmented-Lagrangian-based methods including the ADMM (1.3) is highly dependent on the penalty parameter β. Theoretically, some strategies of adjusting this parameter automatically have been proposed; see, e.g., [29]. But for some concrete applications (especially some large-scale problems or models with matrix variables), realizing this kind of self-adaptive strategy might result in too much computation. Thus, a more popular way to choose this parameter is to tune manually and then fix it as a tuned value throughout. To the best of our knowledge, it is not clear so far how to determine an optimal value for β; it is highly possible that it is problem-dependent. This difficultly occurs also for the original PRSM (1.4). In this subsection, we test some fixed values of β and empirically verify that when implementing the strictly contractive PRSM (1.5), it is easy to choose α ∈ (0, 1) to accelerate the convergence.

We take the group LASSO model (6.2) to demonstrate the effectiveness of α for a fixed β. In our experiments, we test a set of value β = 0.25, 0.5, 1, 2, 4, 8, 16, 32. For each β, we choose different values of α = {0.05, 0.10, 0.15, …, 0.85, 0.90, 0.91, 0.92, …, 0.98, 0.99} and plot the computing time in seconds and number of iterations with respect to different choices of α in Figure 5. According to the plots in Figure 5, it seems that the original PRSM is very sensitive to the value of β. In fact, for some β such as β = 4, 8, 16, 32, the original PRSM (1.4) (i.e., α = 1) fails to satisfy the stopping criterion within 10, 000 iterations. This further emphasizes the importance of choosing β when implementing the original PRSM (1.4). At the same time, we see that for each of the tested β, α ∈ (0, 0.9) tends to accelerate the convergence of PRSM.

Fig. 5.

Acceleration of α for the group LASSO model (6.2).

6.2. Image reconstruction models

In this subsection, we test some digital image reconstruction models. Our aim is to further verify the efficiency of the proposed strictly contractive PRSM (1.5) by comparing it numerically with four well-known algorithms in the imaging literature: SALSA [1], TwIST [4], SpaRSA [52], FISTA [2], and YALL1/TVAL3 [36, 37, 38, 54, 57].

6.2.1. Models and iterative schemes

We first briefly review the background of digital image reconstruction problems; for more details we refer the reader to [28, 47]. A fundamental task in many areas such as medical and astronomical imaging, film restoration, image or video coding, and synthetic aperture radar (see, e.g., [49, 50, 33]), the image reconstruction problem is to reconstruct the original image p ∈ ℜⁿ from its degraded image p⁰ ∈ ℜⁿ. Note that we vectorize an N × M-pixel image P into an n-dimensional vector p in lexicographical order with n = NM. The relationship between p and p⁰ is given by

$$p^0=Dp+\epsilon ,$$ (6.15)

where ε ∈ ℜⁿ is a noise corrupting the original image p, and D ∈ ℜ^{n × n} is the matrix representation of a distortion operator such as a blurring (convolution), vignetting, inpainting, or zooming operator.

According to [13], we can classify image reconstruction models into two categories: the synthesis approach and the analysis approach. The synthesis approach defines x ∈ ℜ^d as the vector of wavelet coefficients of the original image p under a wavelet dictionary. Let W ∈ ℜ^{n × d} be the matrix of a wavelet dictionary, e.g., a group of orthogonal bases; we then have p = Wx. Since (6.15) is usually ill-posed, certain regularization techniques are required. Note that the image p processes a sparse representation under the wavelet dictionary W; that is, x is sparse with many zero entries. Therefore, it is natural to use ||x||₁, the l₁ norm of x, to regularize the data-fidelity term. We thus have

$$min\left\{\frac{1}{2}{\Vert \mathit{DWx}-p^0\Vert }_2^2+\gamma {\Vert x\Vert }_1\mid x\in {\Re }^d\right\}$$ (6.16)

as the synthesis approach of an image reconstruction model. Note that we consider only the case of additive nose. Thus the l₂ norm is used for the data-fidelity term in (6.16). Other cases such as the impulsive or uniform noise can also be considered. On the other hand, the analysis approach considers reconstructing the image directly and not under a wavelet domain. Let the image be represented by a vector x ∈ ℜⁿ. Under the consideration of additive noise in (6.15), the data-fidelity term is $\frac{1}{2}{\Vert Dx-p^0\Vert }_2^2$. For the regularization term, a very popular choice is the total variation (TV) regularization proposed in the seminal work [48], which is well known to be capable of preserving the edges of images. We thus have

$$min\left\{\frac{1}{2}{\Vert Dx-p^0\Vert }_2^2+\gamma \text{TV}(x)\mid x\in {\Re }^n\right\}$$ (6.17)

as the analysis approach of an image reconstruction model. In (6.17), TV(x) denotes the nonsmooth isotropic TV norm [48]:

$$\begin{array}{r}\text{TV}(x)=\sum _{i=1}^{N-1}\sum _{j=1}^{M-1}\sqrt{{(x_{i,j}-x_{i+1,j})}^2+{(x_{i,j}-x_{i,j+1})}^2}+\sum _{i=1}^{N-1}\mid x_{i,M}-x_{i+1,M}\mid \\ +\sum _{j=1}^{M-1}\mid X_{N,j}-x_{N,j+1}\mid ,\end{array}$$

where x ∈ ℜⁿ is the vectorized original N × M two-dimensional image in lexicographic order with n = NM and x_i,j denotes pixel value at the position (i, j).

Now we show how to implement the proposed strictly contractive PRSM (1.5) to solve the models (6.16) and (6.17). For (6.16), by introducing an auxiliary variable y ∈ ℜ^d, it can be reformulated as

$$min\left\{\frac{1}{2}{\Vert \mathit{DWx}-p^0\Vert }_2^2+\gamma {\Vert y\Vert }_1\mid x-y=0,x\in {\Re }^d,y\in {\Re }^d\right\}.$$ (6.18)

Similarly, the model (6.17) can be reformulated as

$$min\left\{\frac{1}{2}{\Vert Dx-p^0\Vert }_2^2+\gamma \text{TV}(y)\mid x-y=0,x\in {\Re }^n,y\in {\Re }^n\right\}.$$ (6.19)

Therefore, when the strictly contractive PRSM (1.5) is applied to solve the models (6.18) and (6.19), the iterative schemes read, respectively, as

$$\{\begin{array}{l}{x}^{k+1}={({(DW)}^T(DW)+\beta I)}^{-1}({(DW)}^T{p}^0+\beta y^k+{\lambda }^k),\hfill \\ {\lambda }^{k+\frac{1}{2}}={\lambda }^k-\alpha \beta (x^{k+1}-y^k),\hfill \\ y^{k+1}=S_{\gamma /\beta }(x^{k+1}-{\lambda }^{k+\frac{1}{2}}/\beta ),\hfill \\ {\lambda }^{k+1}={\lambda }^{k+\frac{1}{2}}-\alpha \beta (x^{k+1}-y^{k+1})\hfill \end{array}$$ (6.20)

and

$$\{\begin{array}{l}{x}^{k+1}={(D^{T}D+\beta I)}^{-1}(D^T{p}^0+\beta y^k+{\lambda }^k),\hfill \\ {\lambda }^{k+\frac{1}{2}}={\lambda }^k-\alpha \beta (x^{k+1}-y^k),\hfill \\ y^{k+1}=argmin\{\gamma \text{TV}(y)+\frac{\beta }{2}{\Vert x^{k+1}-{\lambda }^{k+\frac{1}{2}}/\beta +y\Vert }^2\mid y\in \mathcal{Y}\},\hfill \\ {\lambda }^{k+1}={\lambda }^{k+\frac{1}{2}}-\alpha \beta (x^{k+1}-y^{k+1}).\hfill \end{array}$$ (6.21)

Let us explain how to solve the subproblems in (6.20) and (6.21). For example, the x-subproblem in (6.20) might be computationally expensive due to the high dimensionality of x. For example, for the analysis approach, x ∈ ℜ²⁶²¹⁴⁴ for a 512 × 512-pixel image. According to [1], the matrices D and W are of special structures (e.g., W^TW = I); fast solvers such as the fast Fourier transform (FFT) are thus applicable. For the y-subproblem in (6.21) whose closed-form solution does not exist, we adopt the algorithm proposed in [8] to solve it.

We will test three scenarios for the models (6.16) and (6.17):

1. The synthesis-based (6.16) image deblurring model where D is the matrix representation of the blurring operator [27]. Here we use the 9 × 9 uniform convolution kernel with every element being 1/81 as the blurring operator.

2. The analysis-based (6.17) image inpainting model where D is the matrix representation of the missing pixel operator [27]. Specifically, D is a highly sparse matrix with only ones and zeros in the diagonal. The zeros in the diagonal correspond to the missing pixels.

3. The analysis-based (6.17) MRI image reconstruction model where D is matrix representation of the 22-radial-line mask in the frequency domain, which is visualized in Figure 11.

Fig. 11.

The original and masked images of the Shepp–Logan phantom.

6.2.2. Implementation details

Among the methods to be compared, SALSA in [1] and YALL1/TVAL3 in [38, 57] are ADMM-based algorithms (YALL1 is for l₁ norm regularized problems and TVAL3 is for TV norm regularized problems; thus we implement YALL1 to (6.16) and TVAL3 to (6.17)). The proposed strictly contractive PRSM (1.5) thus can be easily coded based on the source code of SALSA, which is publicly available. We followed the user guide [38, 57] to code YALL1/TVAL3 with tuned parameters. Codes for all other methods are downloaded from the respective authors’ web pages. We terminate the iteration of the strictly contractive PRSM when

$$\mid {\text{Objective}}_{k+1}-{\text{Objective}}_k\mid \le {10}^{-4},$$

where “Objective_k” represents the objective function value at the kth iterate for the model under consideration. To compare, for all other methods, their iterations are terminated when their objective functions are less than or equal to the objective function value obtained by the strictly contractive PRSM (1.5). In other words, we compare these methods subject to the criterion that they achieve the same objective function value.

We measure the noise of an image by the signal-to-noise ratio in units of dB,

$$\text{SNR}:=10{log}_{10}({\Vert p\Vert }_2^2/{\Vert p^0-p\Vert }_2^2),$$

where p and p⁰ are the clean image and the distorted image, respectively. The error of reconstruction is measured by the mean squared error (MSE)

$$\text{MSE}={\Vert p-\widehat{p}\Vert }_2^2/n,$$

where p̂ is the reconstructed image. We also define the improved signal-to-noise ratio as

$$\text{ISNR}:=10{log}_{10}({\Vert p^0-p\Vert }_2^2/{\Vert p-\widehat{p}\Vert }_2^2)$$

as a uniform measurement of the quality of reconstructed images. For different methods, we compare the speed in terms of computing time and number of iterations to achieve the same quality of reconstruction which is measured by MSE or ISNR.

For the synthesis image deblurring application, we set γ = 0.0075 and test the 256 × 256-pixel image of Lena. We choose the four-level redundant Haar wavelet frame as W [11]. To generate the convolution operator D, we choose a 9 × 9 uniform blur kernel in which every element equals 1/81 (with zero padding in the boundary). For the corrupted image, its SNR value is 40dB. The Gaussian noise vector ε is thus generated by (0, 0.449). We list the clean and blurred images in Figure 10. To implement the strictly contractive PRSM (1.5), we set β = 0.0075 and α = 0.8.

Fig. 10.

The original, blurred, and masked (with 40% missing pixels) images of Lena.

For the analysis image inpainting application, we set γ = 0.15 and test the 256 × 256-pixel image of Lena. The inpainting operator D contains 40% missing pixels which are chosen randomly. The masked image is shown in Figure 10. For the corrupted image, its SNR value is also 40dB, which means the noise is generated from (0, 0.529). To implement the strictly contractive PRSM (1.5), we set β = 0.05 and α = 0.9. To solve the y-subproblem in (6.21) by the method in [8], we allow a maximum of 20 for the inner iteration.

For the analysis MRI application, we set γ = 0.0001 and test the 128 × 128-pixel image of the Shepp–Logan phantom. The clean image is masked by 22 radial lines on its discrete Fourier transform, and only the frequency components covered by the radial lines are observed. We contaminate the frequency components using the circular complex Gaussian noise with ${\sigma }_{\epsilon }^2=0.5\times {10}^{-3}$; i.e., the real and imaginary parts of the noise are independent Gaussian with the standard deviation σ_ε. We list the clean and masked images of the Shepp–Logan phantom in Figure 11. The SNR value is 5.42. To implement the strictly contractive PRSM (1.5), we set β = 0.01 and α = 0.9. To solve the y-subproblem in (6.21) by the method in [8], we allow a maximum of 40 for the inner iteration.

6.2.3. Results

We first list the comparison of different methods for the imaging construction models in Tables 2–4.⁴ Then, we visualize the evolution of the objective function when these methods are applied to solve these imaging models in Figures 6–8. The evolution of the MSE is also plotted in Figure 9. In Figure 12, we display the clean, corrupted, and reconstructed images by the strictly contractive PRSM (1.5) for the tested scenarios. These tables and figures clearly show the efficiency of the proposed strictly contractive PRSM (1.5).

Table 2.

Quantitative comparison on image deblurring model.

Algorithm	Image deblurring
	time(s)	#iterations	MSE	ISNR(dB)
Strictly contractive PRSM	2.60	15	97.8	6.60
SALSA	6.86	41	97.7	6.60
TwIST	7.87	60	97.5	6.61
FISTA	8.18	97	103	6.63
SpaRSA	9.40	86	107	6.21
YALL1	9.38	86	107	6.21

Table 4.

Quantitative comparison on MRI model.

Algorithm	time(s)	#iterations	MSE	ISNR(dB)
Strictly contractive PRSM	19.81	207	1.03652e-06	42.2073
SALSA	30.46	311	1.03435e-06	42.2164
TwIST	53.69	451	1.10616e-06	41.9673
FISTA	25.23	460	1.04220e-06	42.1836
SpaRSA	97.74	1001	1.09543e-06	41.9250
TVAL3	45.36	601	1.03762e-06	42.2027

Fig. 6.

Visualization of the objective functions on image deblurring model.

Fig. 8.

Visualization of the objective functions on MRI image reconstruction.

Fig. 9.

Visualization of MSE on MRI image reconstruction.

Fig. 12.

The deblurred and reconstructed images of Lena, and the reconstructed Shepp–Logan phantom image by strictly contractive PRSM. The SNRs for the reconstructed images are 46.6dB, 57.5dB, and 47.6dB, respectively.

7. Conclusions

As a classical operator splitting method in the literature, the Peaceman–Rachford splitting method (PRSM) may fail to be convergent for solving a convex optimization problem with linear constraints and a separable objective function. This paper shows that this failure can be illustrated by showing that its iterative sequence is not strictly contractive with respect to the solution set of the model under consideration. This understanding from a contraction perspective inspires us to tackle the deficiency of PRSM by embedding an underdetermined factor into the iterative scheme of PRSM and to propose a strictly contractive PRSM. The strictly contractive PRSM is as easy to implement as that of the alternating direction method of multipliers (ADMM), and it is numerically faster. We verify these advantages by some applications in statistical learning and image processing. We also study the convergence rate of the proposed strictly contractive PRSM, establishing the worst-case O(1/t) convergence rate in both the ergodic and nonergodic senses.

Fig. 7.

Visualization of the objective functions on image inpainting model.

Table 3.

Quantitative comparison on image inpainting model.

Algorithm	Image inpainting
	time(s)	#iterations	MSE	ISNR(dB)
Strictly contractive PRSM	3.86	28	88.9	17.5
SALSA	6.82	46	93.2	17.3
TwIST	14.4	83	89.7	17.4
FISTA	6.76	91	90.4	17.4
TVAL3	6.04	117	90.2	17.4