A symmetric version of the generalized alternating direction method of multipliers for two-block separable convex programming

Abstract

This paper introduces a symmetric version of the generalized alternating direction method of multipliers for two-block separable convex programming with linear equality constraints, which inherits the superiorities of the classical alternating direction method of multipliers (ADMM), and which extends the feasible set of the relaxation factor α of the generalized ADMM to the infinite interval $[1,+\mathrm{\infty })$. Under the conditions that the objective function is convex and the solution set is nonempty, we establish the convergence results of the proposed method, including the global convergence, the worst-case $\mathcal{O}(1/k)$ convergence rate in both the ergodic and the non-ergodic senses, where k denotes the iteration counter. Numerical experiments to decode a sparse signal arising in compressed sensing are included to illustrate the efficiency of the new method.

Introduction

We consider the two-block separable convex programming with linear equality constraints, where the objective function is the sum of two individual functions with decoupled variables:

$$min\{{\theta }_1(x_1)+{\theta }_2(x_2)|A_1{x}_1+A_2{x}_2=b,x_1\in {\mathcal{X}}_1,x_2\in {\mathcal{X}}_2\},$$ 1

where ${\theta }_i:{\mathcal{R}}^{n_i}\to \mathcal{R}$ ($i=1,2$) are closed proper convex functions; $A_i\in {\mathcal{R}}^{l\times n_i}$ ($i=1,2$) and $b\in {\mathcal{R}}^l$, and ${\mathcal{X}}_i\subseteq {\mathcal{R}}^{n_i}$ ($i=1,2$) are given nonempty closed convex sets. The linear constrained convex problem (1) is a unified framework of many problems arising in real world, including compressed sensing, image restoration, and statistical learning, and so forth (see, for example, [1–3]). An important special case of (1) is the following linear inverse problem:

$$\underset{x\in {\mathcal{R}}^n}{min}\mu {\parallel x\parallel }_1+\frac{1}{2}{\parallel Ax-y\parallel }^2,$$ 2

where $A\in {\mathcal{R}}^{m\times n}$ and $y\in {\mathcal{R}}^m$ are given matrix and vector, $\mu >0$ is a regularization parameter and ${\parallel x\parallel }_1$ is the ${\ell }_1$-norm of a vector x defined as ${\parallel x\parallel }_1={\sum }_{i=1}^n|x_i|$. Then setting $x_1:=Ax-y$, $x_2:=x$, (2) can be converted into the following two-block separable convex programming:

$$\begin{array}{rl}& min\frac{1}{2}{\parallel x_1\parallel }^2+\mu {\parallel x_2\parallel }_1\\ & \text{s.t. }-x_1+A{x}_2=y,\\ & \phantom{\text{s.t.}}{x}_1\in {\mathcal{R}}^m,x_2\in {\mathcal{R}}^n,\end{array}$$ 3

which is a special case of problem (1) with the following specifications:

$${\theta }_1(x_1):=\frac{1}{2}{\parallel x_1\parallel }^2,{\theta }_2(x_2):=\mu {\parallel x_2\parallel }_1,A_1:=-I_m,A_2:=A,b:=y.$$

Existing algorithms

In their seminal work, Glowinski et al. [4] and Gabay et al. [5] independently developed the alternating direction method of multipliers (ADMM), which is an influential first-order method for solving problem (1). ADMM can be regarded as an application of the Douglas-Rachford splitting method (DRSM) [6] to the dual of (1), or a special case of the proximal point algorithm (PPA) [7, 8] in the cyclic sense. We refer to [9] for a more detailed relationship. With any initial vectors $x_2^0\in {\mathcal{X}}_2$, ${\lambda }^0\in {\mathcal{R}}^l$, the iterative scheme of ADMM reads

$$\{\begin{array}{c}{x}_1^{k+1}\in {argmin}_{x_1\in {\mathcal{X}}_1}\{{\theta }_1(x_1)-x_1^{\mathrm{\top }}{A}_1^{\mathrm{\top }}{\lambda }^k+\frac{\beta }{2}{\parallel A_1{x}_1+A_2{x}_2^k-b\parallel }^2\},\hfill \\ x_2^{k+1}\in {argmin}_{x_2\in {\mathcal{X}}_2}\{{\theta }_2(x_2)-x_2^{\mathrm{\top }}{A}_2^{\mathrm{\top }}{\lambda }^k+\frac{\beta }{2}{\parallel A_1{x}_1^{k+1}+A_2{x}_2-b\parallel }^2\},\hfill \\ {\lambda }^{k+1}={\lambda }^k-\beta (A_1{x}_1^{k+1}+A_2{x}_2^{k+1}-b),\hfill \end{array}$$ 4

where $\lambda \in {\mathcal{R}}^l$ is the Lagrangian multiplier and $\beta >0$ is a penalty parameter. The main characteristics of ADMM are that it in full exploits the separable structure of problem (1), and that it updates the variables $x_1$, $x_2$, λ in an alternating order by solving a series of low-dimensional sub-problems with only one unknown variable.

In the past few decades, ADMM has received a revived interest, and it has become a research focus in optimization community, especially in the (non)convex optimization. Many efficient ADMM-type methods have been developed, including the proximal ADMM [8, 10], the generalized ADMM [11], the symmetric ADMM [12], the inertial ADMM [13], and some proximal ADMM-type methods [14–18]. Specifically, the proximal ADMM attaches some proximal terms to the sub-problems of ADMM (4). The generalized ADMM updates the variables $x_2$ and λ by including a relaxation factor $\alpha \in (0,2)$, and $\alpha \in (1,2)$ is often advantageous to speed up its performance. The symmetric ADMM updates the Lagrangian multiplier λ twice at each iteration and includes two relaxation factors $\alpha \in (0,1)$, $\beta \in (0,1)$. Recent researches of the symmetric ADMM can be found in [12, 15, 18]. The inertial ADMM unifies the basic ideas of the inertial PPA and ADMM, which utilizes the latest two iterates to generate the new iterate, therefore it can be viewed as a multistep method. For the proximal ADMM, the objective functions of its sub-problems are often strongly convex, which are often easier to be solved than those of (4). However, a new challenge has arisen for the proximal ADMM-type methods. It is how to choose a proper proximal matrix. In fact, most proximal ADMM-type methods need to estimate the matrix norm $\parallel A_i^{\mathrm{\top }}{A}_i\parallel$ ($i=1,2$), which demands lots of calculations, especially for large $n_i$ ($i=1,2$). Quite recently, some customized Douglas-Rachford splitting algorithms [19–21], and the proximal ADMM-type methods with indefinite proximal regularization are developed [22, 23], which dissolve the above problem to some extent. All the above mentioned ADMM-type methods are generalizations of the classical ADMM, because they all reduce to the iterative scheme (4) by choosing some special parameters. For more new development of the ADMM-type methods, including the convergence rate, acceleration techniques, its generalization for solving multi-block separable convex programming and nonconvex, nonsmooth programming, we refer to [24–28].

Contributions and organization

We are going to further study the generalized ADMM. Note that the first sub-problem in the generalized ADMM is irrelevant to the relaxation factor α. That is, the updating formula for $x_1$ does not incorporate the relaxation factor α explicitly. Furthermore, $\alpha \in (1,2)$ is often advantageous for the generalized ADMM [14]. Therefore, in this paper, we are going to propose a new generalized ADMM, whose both sub-problems incorporate the relaxation factor α directly. The new method generalizes the method proposed in [29] by relaxing the feasible set of α from the interval $[1,2)$ to the infinite interval $[1,+\mathrm{\infty })$, and can be viewed as a symmetric version of the generalized ADMM.

The rest of the paper is organized as follows. In Section 2, we summarize some necessary preliminaries and characterize problem (1) by a mixed variational inequality problem. In Section 3, we describe the new symmetric version of the generalized ADMM and establish its convergence results in detail. In Section 4, some compressed sensing experiments are given to illustrate the efficiency of the proposed method. Some conclusions are drawn in Section 5.

Preliminaries

In this section, some necessary preliminaries which are useful for further discussions are presented, and to make our analysis more succinct, some positive definite or positive semi-definite block matrices are defined and their properties are investigated.

For two real matrices $A\in {\mathcal{R}}^{s\times m}$, $B\in {\mathcal{R}}^{n\times s}$, the Kronecker product of A and B is defined as $A\otimes B=(a_{ij}B)$. Let ${\parallel \cdot \parallel }_p$ ($p=1,2$) denote the standard definition of ${\ell }_p$-norm; in particular, $\parallel \cdot \parallel ={\parallel \cdot \parallel }_2$. For any two vectors $x,y\in {\mathcal{R}}^n$, $\langle x,y\rangle$ or $x^{\mathrm{\top }}y$ denote their inner product, and for any symmetric matrix $G\in {\mathcal{R}}^{n\times n}$, the symbol $G\succ 0$ (resp., $G⪰0$) denotes that G is positive definite (resp., semi-definite). For any $x\in {\mathcal{R}}^n$ and $G⪰0$, the G-norm ${\parallel x\parallel }_G$ of the vector x is defined as $\sqrt{x^{\mathrm{\top }}Gx}$. The effective domain of a closed proper function $f:\mathcal{X}\to (-\mathrm{\infty },+\mathrm{\infty }]$ is defined as $dom(f):=\{x\in \mathcal{X}|f(x)<+\mathrm{\infty }\}$, and the symbol $ri(\mathcal{C})$ denotes the set of all relative interior points of a given nonempty convex set $\mathcal{C}$. Furthermore, we use the following notations:

$$x=(x_1,x_2),w=(x,\lambda ).$$

Definition 2.1

[30]

A function $f:{\mathcal{R}}^n\to \mathcal{R}$ is convex if and only if

$$f(\alpha x+(1-\alpha )y)\le \alpha f(x)+(1-\alpha )f(y),\mathrm{\forall }x,y\in {\mathcal{R}}^n,\alpha \in [0,1].$$

Then, for a convex function $f:{\mathcal{R}}^n\to \mathcal{R}$, we have the following basic inequality:

$$f(x)\ge f(y)+\langle \xi ,x-y\rangle ,\mathrm{\forall }x,y\in {\mathcal{R}}^n,\xi \in \partial f(y),$$ 5

where $\partial f(y)=\{\xi \in {\mathcal{R}}^n:f(\overline{y})\ge f(y)+\langle \xi ,\overline{y}-y\rangle ,\text{for all }\overline{y}\in {\mathcal{R}}^n\}$ denotes the subdifferential of $f(\cdot )$ at the point y.

Throughout the paper, we make the following standard assumptions for problem (1).

Assumption 2.1

The functions ${\theta }_i(\cdot )$ ($i=1,2$) are convex.

Assumption 2.2

The matrices $A_i$ ($i=1,2$) are full-column rank.

Assumption 2.3

The generalized Slater condition holds, i.e., there is a point $({\hat{x}}_1,{\hat{x}}_2)\in ri(dom{\theta }_1\times dom{\theta }_2)\cap \{x=(x_1,x_2)\in {\mathcal{X}}_1\times {\mathcal{X}}_2|A_1{x}_1+A_2{x}_2=b\}$.

The mixed variational inequality problem

Under Assumption 2.3, it follows from Theorem 3.22 and Theorem 3.23 of [31] that $x^{\ast }=(x_1^{\ast },x_2^{\ast })\in {\mathcal{R}}^{n_1+n_2}$ is an optimal solution to problem (1) if and only if there exists a vector ${\lambda }^{\ast }\in {\mathcal{R}}^l$ such that $(x_1^{\ast },x_2^{\ast },{\lambda }^{\ast })$ is a solution of the following KKT system:

$$\{\begin{array}{c}0\in \partial {\theta }_i(x_i^{\ast })-A_i^{\mathrm{\top }}{\lambda }^{\ast }+N_{{\mathcal{X}}_i}(x_i^{\ast }),i=1,2,\hfill \\ A_1{x}_1^{\ast }+A_2{x}_2^{\ast }=b,\hfill \end{array}$$ 6

where $N_{{\mathcal{X}}_i}(x_i^{\ast })$ is the normal cone of the convex set ${\mathcal{X}}_i$ at the point $x_i^{\ast }$, which is defined as $N_{{\mathcal{X}}_i}(x_i^{\ast })=\{z\in {\mathcal{R}}^{n_i}|\langle z,x_i-x_i^{\ast }\rangle \le 0,\mathrm{\forall }{x}_i\in {\mathcal{X}}_i\}$. Then, for the nonempty convex set ${\mathcal{X}}_i$ and $\mathrm{\forall }{x}_i\in {\mathcal{X}}_i$, it follows from [32] (Example 2.123) that $N_{{\mathcal{X}}_i}(x_i)=\partial \delta (\cdot |{\mathcal{X}}_i)(x_i)$, where $\delta (\cdot |{\mathcal{X}}_i)$ is the indicator function of the set ${\mathcal{X}}_i$, and $\partial \delta (\cdot |{\mathcal{X}}_i)(x_i)$ is the subdifferential mappings of $\delta (\cdot |{\mathcal{X}}_i)$ at the point $x_i\in {\mathcal{X}}_i$.

Lemma 2.1

For any vector $x_i^{\ast }\in {\mathcal{R}}^{n_i}$, ${\lambda }^{\ast }\in {\mathcal{R}}^l$, the relationship $0\in \partial {\theta }_i(x_i^{\ast })-A_i^{\mathrm{\top }}{\lambda }^{\ast }+\partial \delta (\cdot |{\mathcal{X}}_i)(x_i^{\ast })$ is equivalent to $x_i^{\ast }\in {\mathcal{X}}_i$ and the inequality

$${\theta }_i(x_i)-{\theta }_i\left(x_i^{\ast }\right)+{(x_i-x_i^{\ast })}^{\mathrm{\top }}(-A_i^{\mathrm{\top }}{\lambda }^{\ast })\ge 0,\mathrm{\forall }{x}_i\in {\mathcal{X}}_i.$$

Proof

From $0\in \partial {\theta }_i(x_i^{\ast })-A_i^{\mathrm{\top }}{\lambda }^{\ast }+\partial \delta (\cdot |{\mathcal{X}}_i)(x_i^{\ast })$, we have $x_i^{\ast }\in {\mathcal{X}}_i$ and there exists ${\eta }_i\in \partial \delta (\cdot |{\mathcal{X}}_i)(x_i^{\ast })$ such that

$$A_i^{\mathrm{\top }}{\lambda }^{\ast }-{\eta }_i\in \partial {\theta }_i\left(x_i^{\ast }\right).$$

From the subgradient inequality (5), one has

$${\theta }_i(x_i)-{\theta }_i\left(x_i^{\ast }\right)\ge {(x_i-x_i^{\ast })}^{\mathrm{\top }}(A_i^{\mathrm{\top }}{\lambda }^{\ast }-{\eta }_i),\mathrm{\forall }{x}_i\in {\mathcal{R}}^{n_i}.$$

Thus,

$${\theta }_i(x_i)-{\theta }_i\left(x_i^{\ast }\right)-{(x_i-x_i^{\ast })}^{\mathrm{\top }}(-A_i^{\mathrm{\top }}{\lambda }^{\ast })\ge {(x_i-x_i^{\ast })}^{\mathrm{\top }}(-{\eta }_i)\ge 0,\mathrm{\forall }{x}_i\in {\mathcal{X}}_i,$$

where the second inequality comes from $x_i^{\ast }\in {\mathcal{X}}_i$ and ${\eta }_i\in \partial \delta (\cdot |{\mathcal{X}}_i)(x_i^{\ast })$.

Conversely, from ${\theta }_i(x_i)-{\theta }_i(x_i^{\ast })+{(x_i-x_i^{\ast })}^{\mathrm{\top }}(-A_i^{\mathrm{\top }}{\lambda }^{\ast })\ge 0$, $\mathrm{\forall }{x}_i\in {\mathcal{X}}_i$, we have

$${\theta }_i(x_i)+x_i^{\mathrm{\top }}(-A_i^{\mathrm{\top }}{\lambda }^{\ast })\ge {\theta }_i\left(x_i^{\ast }\right)+{\left(x_i^{\ast }\right)}^{\mathrm{\top }}(-A_i^{\mathrm{\top }}{\lambda }^{\ast }),\mathrm{\forall }{x}_i\in {\mathcal{X}}_i,$$

which together with $x_i^{\ast }\in {\mathcal{X}}_i$ implies that

$$x_i^{\ast }=\underset{x_i\in {\mathcal{X}}_i}{argmin}\{{\theta }_i(x_i)+x_i^{\mathrm{\top }}(-A_i^{\mathrm{\top }}{\lambda }^{\ast })\}.$$

From this and Theorem 3.22 of [31], we have $0\in \partial {\theta }_i(x_i^{\ast })-A_i^{\mathrm{\top }}{\lambda }^{\ast }+\partial \delta (\cdot |{\mathcal{X}}_i)(x_i^{\ast })$. This completes the proof. □

Remark 2.1

Based on (6) and Lemma 2.1, the vector $x^{\ast }=(x_1^{\ast },x_2^{\ast })\in {\mathcal{R}}^{n_1+n_2}$ is an optimal solution to problem (1) if and only if there exists a vector ${\lambda }^{\ast }\in {\mathcal{R}}^l$ such that

$$\{\begin{array}{c}(x_1^{\ast },x_2^{\ast })\in {\mathcal{X}}_1\times {\mathcal{X}}_2;\hfill \\ {\theta }_i(x_i)-{\theta }_i(x_i^{\ast })+{(x_i-x_i^{\ast })}^{\mathrm{\top }}(-A_i^{\mathrm{\top }}{\lambda }^{\ast })\ge 0,\mathrm{\forall }{x}_i\in {\mathcal{X}}_i,i=1,2;\hfill \\ A_1{x}_1^{\ast }+A_2{x}_2^{\ast }=b.\hfill \end{array}$$ 7

Moreover, any ${\lambda }^{\ast }\in {\mathcal{R}}^l$ satisfying (7) is an optimal solution to the dual of problem (1). Obviously, (7) can be written as the following mixed variational inequality problem, denoted by $VI(\mathcal{W},F,\theta )$: Find a vector $w^{\ast }\in \mathcal{W}$ such that

$$\theta (x)-\theta \left(x^{\ast }\right)+{(w-w^{\ast })}^{\mathrm{\top }}F\left(w^{\ast }\right)\ge 0,\mathrm{\forall }w\in \mathcal{W},$$ 8

where $\theta (x)={\theta }_1(x_1)+{\theta }_2(x_2)$, $\mathcal{W}={\mathcal{X}}_1\times {\mathcal{X}}_2\times {\mathcal{R}}^l$, and

$$F(w):=\left(\begin{array}{c}-A_1^{\mathrm{\top }}\lambda \\ -A_2^{\mathrm{\top }}\lambda \\ A_1{x}_1+A_2{x}_2-b\end{array}\right)=\left(\begin{array}{ccc}0& 0& -A_1^{\mathrm{\top }}\\ 0& 0& -A_2^{\mathrm{\top }}\\ A_1& A_2& 0\end{array}\right)\left(\begin{array}{c}{x}_1\\ x_2\\ \lambda \end{array}\right)-\left(\begin{array}{c}0\\ 0\\ b\end{array}\right).$$ 9

The solution set of $VI(\mathcal{W},F,\theta )$, denoted by ${\mathcal{W}}^{\ast }$, is nonempty by Assumption 2.3 and Remark 2.1. It is easy to verify that the linear function $F(\cdot )$ is not only monotone but also satisfies the following desired property:

$${(w^{\prime }-w)}^{\mathrm{\top }}(F\left(w^{\prime }\right)-F(w))=0,\mathrm{\forall }{w}^{\prime },w\in \mathcal{W}.$$

Three matrices and their properties

To present our analysis in a compact way, now let us define some matrices. For any $R_i\in {\mathcal{R}}^{n_i\times n_i}(i=1,2)⪰0$, set

$$M=\left(\begin{array}{ccc}{I}_{n_1}& 0& 0\\ 0& I_{n_2}& 0\\ 0& -\beta A_2& I_l\end{array}\right)$$ 10

and for $\alpha \in [1,+\mathrm{\infty })$, set

$$\begin{array}{rl}& Q=\left(\begin{array}{ccc}{R}_1& 0& 0\\ 0& R_2+(2\alpha -1)\beta A_2^{\mathrm{\top }}{A}_2& \frac{1-\alpha }{\alpha }{A}_2^{\mathrm{\top }}\\ 0& -A_2& \frac{1}{\alpha \beta }{I}_l\end{array}\right),\\ & H=\left(\begin{array}{ccc}{R}_1& 0& 0\\ 0& R_2+\frac{2{\alpha }^2-2\alpha +1}{\alpha }\beta A_2^{\mathrm{\top }}{A}_2& \frac{1-\alpha }{\alpha }{A}_2^{\mathrm{\top }}\\ 0& \frac{1-\alpha }{\alpha }{A}_2& \frac{1}{\alpha \beta }{I}_l\end{array}\right).\end{array}$$ 11

The above defined three matrices M, Q, H satisfy the following properties.

Lemma 2.2

If $\alpha \in \mathcal{R}$ and $R_i⪰0$ ($i=1,2$), then the matrix H defined in (11) is positive semi-definite.

Proof

Set $t=2{\alpha }^2-2\alpha +1$, which is positive for any $\alpha \in \mathcal{R}$. By (11), we have

$$H=\left(\begin{array}{ccc}{R}_1& 0& 0\\ 0& R_2& 0\\ 0& 0& 0\end{array}\right)+\left(\begin{array}{ccc}0& 0& 0\\ 0& \frac{t\beta }{\alpha }{A}_2^{\mathrm{\top }}{A}_2& \frac{1-\alpha }{\alpha }{A}_2^{\mathrm{\top }}\\ 0& \frac{1-\alpha }{\alpha }{A}_2& \frac{1}{\alpha \beta }{I}_l\end{array}\right).$$

Obviously, the first part is positive semi-definite, and we only need to prove the second part is also positive semi-definite. In fact, it can written as

$$\frac{1}{\alpha }\left(\begin{array}{ccc}0& 0& 0\\ 0& \sqrt{\beta }{A}_2^{\mathrm{\top }}& 0\\ 0& 0& \frac{1}{\sqrt{\beta }}{I}_l\end{array}\right)\left(\begin{array}{ccc}0& 0& 0\\ 0& t{I}_l& (1-\alpha )I_l\\ 0& (1-\alpha )I_l& I_l\end{array}\right)\left(\begin{array}{ccc}0& 0& 0\\ 0& \sqrt{\beta }{A}_2& 0\\ 0& 0& \frac{1}{\sqrt{\beta }}{I}_l\end{array}\right).$$

The middle matrix in the above expression can be further written as

$$\left(\begin{array}{ccc}0& 0& 0\\ 0& t& 1-\alpha \\ 0& 1-\alpha & 1\end{array}\right)\otimes I_l,$$

where ⊗ denotes the matrix Kronecker product. The matrix Kronecker product has a nice property: for any two matrices X and Y, the eigenvalue of $X\otimes Y$ equals the product of $\lambda (X)\lambda (Y)$, where $\lambda (X)$ and $\lambda (Y)$ are the eigenvalues of X and Y, respectively. Therefore, we only need to show the 2-by-2 matrix

$$\left(\begin{array}{cc}t& 1-\alpha \\ 1-\alpha & 1\end{array}\right)$$

is positive semi-definite. In fact,

$$t-{(1-\alpha )}^2={\alpha }^2\ge 0.$$

Therefore, the matrix H is positive semi-definite. The proof is then complete. □

Lemma 2.3

If $\alpha \in [1,+\mathrm{\infty })$ and $R_i⪰0$ ($i=1,2$), then the matrices M, Q, H defined, respectively, in (10), (11) satisfy the following relationships:

$$HM=Q$$ 12

and

$$Q^{\mathrm{\top }}+Q-M^{\mathrm{\top }}HM⪰\frac{\alpha -1}{2\alpha }{M}^{\mathrm{\top }}HM.$$ 13

Proof

From (10) and (11), we have

$$\begin{array}{rcl}HM& =& \left(\begin{array}{ccc}{R}_1& 0& 0\\ 0& R_2+\frac{2{\alpha }^2-2\alpha +1}{\alpha }\beta A_2^{\mathrm{\top }}{A}_2& \frac{1-\alpha }{\alpha }{A}_2^{\mathrm{\top }}\\ 0& \frac{1-\alpha }{\alpha }{A}_2& \frac{1}{\alpha \beta }{I}_l\end{array}\right)\left(\begin{array}{ccc}{I}_{n_1}& 0& 0\\ 0& I_{n_2}& 0\\ 0& -\beta A_2& I_l\end{array}\right)\\ & =& \left(\begin{array}{ccc}{R}_1& 0& 0\\ 0& R_2+(2\alpha -1)\beta A_2^{\mathrm{\top }}{A}_2& \frac{1-\alpha }{\alpha }{A}_2^{\mathrm{\top }}\\ 0& -A_2& \frac{1}{\alpha \beta }{I}_l\end{array}\right)=Q.\end{array}$$

Then the first assertion is proved. For (13), by some simple manipulations, we obtain

$$\begin{array}{rcl}{M}^{\mathrm{\top }}HM& =& M^{\mathrm{\top }}Q\\ & =& \left(\begin{array}{ccc}{I}_{n_1}& 0& 0\\ 0& I_{n_2}& -\beta A_2^{\mathrm{\top }}\\ 0& 0& I_l\end{array}\right)\left(\begin{array}{ccc}{R}_1& 0& 0\\ 0& R_2+(2\alpha -1)\beta A_2^{\mathrm{\top }}{A}_2& \frac{1-\alpha }{\alpha }{A}_2^{\mathrm{\top }}\\ 0& -A_2& \frac{1}{\alpha \beta }{I}_l\end{array}\right)\\ & =& \left(\begin{array}{ccc}{R}_1& 0& 0\\ 0& R_2+2\alpha \beta A_2^{\mathrm{\top }}{A}_2& -A_2^{\mathrm{\top }}\\ 0& -A_2& \frac{1}{\alpha \beta }{I}_l\end{array}\right).\end{array}$$

We now break up the proof into two cases. First, if $\alpha =1$, then

$$(Q^{\mathrm{\top }}+Q)-M^{\mathrm{\top }}HM=\left(\begin{array}{ccc}{R}_1& 0& 0\\ 0& R_2& 0\\ 0& 0& \frac{1}{\beta }{I}_l\end{array}\right)⪰0.$$

Therefore, (13) holds. Second, if $\alpha \in (1,+\mathrm{\infty })$, then

$$\begin{array}{c}(Q^{\mathrm{\top }}+Q)-M^{\mathrm{\top }}HM\hfill \\ =\left(\begin{array}{ccc}{R}_1& 0& 0\\ 0& R_2+(2\alpha -2)\beta A_2^{\mathrm{\top }}{A}_2& \frac{1-\alpha }{\alpha }{A}_2^{\mathrm{\top }}\\ 0& \frac{1-\alpha }{\alpha }{A}_2& \frac{1}{\alpha \beta }{I}_l\end{array}\right)\hfill \\ =\left(\begin{array}{ccc}{R}_1& 0& 0\\ 0& R_2& 0\\ 0& 0& 0\end{array}\right)+(2\alpha -2)\left(\begin{array}{ccc}0& 0& 0\\ 0& \beta A_2^{\mathrm{\top }}{A}_2& -\frac{1}{2\alpha }{A}_2^{\mathrm{\top }}\\ 0& -\frac{1}{2\alpha }{A}_2& \frac{1}{\alpha \beta (2\alpha -2)}{I}_l\end{array}\right).\hfill \end{array}$$ 14

Note that

$$\begin{array}{c}4\alpha \left(\begin{array}{cc}\beta A_2^{\mathrm{\top }}{A}_2& -\frac{1}{2\alpha }{A}_2^{\mathrm{\top }}\\ -\frac{1}{2\alpha }{A}_2& \frac{1}{\alpha \beta (2\alpha -2)}{I}_l\end{array}\right)-\left(\begin{array}{cc}2\alpha \beta A_2^{\mathrm{\top }}{A}_2& -A_2^{\mathrm{\top }}\\ -A_2& \frac{1}{\alpha \beta }{I}_l\end{array}\right)\hfill \\ =\left(\begin{array}{cc}2\alpha \beta A_2^{\mathrm{\top }}{A}_2& -A_2^{\mathrm{\top }}\\ -A_2& \frac{\alpha +1}{\alpha \beta (\alpha -1)}{I}_l\end{array}\right)\hfill \\ =\left(\begin{array}{cc}\sqrt{\beta }{A}_2^{\mathrm{\top }}& 0\\ 0& \frac{1}{\sqrt{\beta }}{I}_l\end{array}\right)\left(\begin{array}{cc}2\alpha I_l& -I_l\\ -I_l& \frac{\alpha +1}{\alpha (\alpha -1)}{I}_l\end{array}\right)\left(\begin{array}{cc}\sqrt{\beta }{A}_2& 0\\ 0& \frac{1}{\sqrt{\beta }}{I}_l\end{array}\right).\hfill \end{array}$$ 15

The middle matrix in the above expression can be further written as

$$\left(\begin{array}{cc}2\alpha & -1\\ -1& \frac{\alpha +1}{\alpha (\alpha -1)}\end{array}\right)\otimes I_l.$$

Since

$$\left(\begin{array}{cc}2\alpha & -1\\ -1& \frac{\alpha +1}{\alpha (\alpha -1)}\end{array}\right)⪰0,\mathrm{\forall }\alpha >1,$$

the right-hand side of (15) is also positive semi-definite. Thus, we have

$$\left(\begin{array}{cc}\beta A_2^{\mathrm{\top }}{A}_2& -\frac{1}{2\alpha }{A}_2^{\mathrm{\top }}\\ -\frac{1}{2\alpha }{A}_2& \frac{1}{\alpha \beta (2\alpha -2)}{I}_l\end{array}\right)⪰\frac{1}{4\alpha }\left(\begin{array}{cc}2\alpha \beta A_2^{\mathrm{\top }}{A}_2& -A_2^{\mathrm{\top }}\\ -A_2& \frac{1}{\alpha \beta }{I}_l\end{array}\right).$$ 16

Substituting (16) into (14) and by the expression of $M^{\mathrm{\top }}HM$, we obtain (13). The lemma is proved. □

Algorithm and convergence results

In this section, we first describe the symmetric version of the generalized alternating direction method of multipliers (SGADMM) for $VI(\mathcal{W},F,\theta )$ formally, and then we prove its global convergence in a contraction perspective and establish its worst-case $\mathcal{O}(1/k)$ convergence rate in both the ergodic and the non-ergodic senses step by step, where k denotes the iteration counter.

Algorithm

Algorithm 3.1

SGADMM

Step 0.

Choose the parameters $\alpha \in [1,+\mathrm{\infty })$, $\beta >0$, $R_i\in {\mathcal{R}}^{n_i\times n_i}⪰0$ ($i=1,2$), the tolerance $\epsilon >0$ and the initial iterate $(x_1^0,x_2^0,{\lambda }^0)\in {\mathcal{X}}_1\times {\mathcal{X}}_2\times {\mathcal{R}}^l$. Set $k:=0$.

Step 1.

Generate the new iterate $w^{k+1}=(x_1^{k+1},x_2^{k+1},{\lambda }^{k+1})$ by

$$\{\begin{array}{c}{x}_1^{k+1}\in {argmin}_{x_1\in {\mathcal{X}}_1}\{{\theta }_1(x_1)-x_1^{\mathrm{\top }}{A}_1^{\mathrm{\top }}{\lambda }^k+\frac{\alpha \beta }{2}{\parallel A_1{x}_1+A_2{x}_2^k-b\parallel }^2\hfill \\ \phantom{x_1^{k+1}\in }+\frac{1}{2}{\parallel x_1-x_1^k\parallel }_{R_1}^2\},\hfill \\ x_2^{k+1}\in {argmin}_{x_2\in {\mathcal{X}}_2}\{{\theta }_2(x_2)-x_2^{\mathrm{\top }}{A}_2^{\mathrm{\top }}{\lambda }^k+\frac{(2\alpha -1)\beta }{2}{\parallel A_1{x}_1^{k+1}+A_2{x}_2-b\parallel }^2\hfill \\ \phantom{x_2^{k+1}\in }+\frac{1}{2}{\parallel x_2-x_2^k\parallel }_{R_2}^2\},\hfill \\ {\lambda }^{k+1}={\lambda }^k-\beta [\alpha A_1{x}_1^{k+1}-(1-\alpha )(A_2{x}_2^k-b)+A_2{x}_2^{k+1}-b].\hfill \end{array}$$ 17

Step 2.

If

$$max\{\parallel R_1{x}_1^k-R_1{x}_1^{k+1}\parallel ,\parallel R_2{x}_2^k-R_2{x}_2^{k+1}\parallel ,\parallel A_2{x}_2^k-A_2{x}_2^{k+1}\parallel ,\parallel {\lambda }^k-{\lambda }^{k+1}\parallel \}<\epsilon ,$$ 18

then stop and return an approximate solution $(x_1^{k+1},x_2^{k+1},{\lambda }^{k+1})$ of $VI(\mathcal{W},F,\theta )$; else set $k:=k+1$, and goto Step 1.

Remark 3.1

Obviously, the iterative scheme (17) reduces to the generalized ADMM when $\alpha =1$, and further reduces to (4) when $R_i=0$ ($i=1,2$). That is to say, if the parameters $\alpha =1$ and $R_i=0$ ($i=1,2$), then the classical ADMM is recovered. Since the convergence results of the (proximal) ADMM have been established in the literature [23, 33, 34], in the following, we only consider $\alpha \in (1,+\mathrm{\infty })$.

Global convergence

For further analysis, we need to define an auxiliary sequence $\{{\hat{w}}^k\}$ as follows:

$${\hat{w}}^k=\left(\begin{array}{c}{\hat{x}}_1^k\\ {\hat{x}}_2^k\\ {\hat{\lambda }}^k\end{array}\right)=\left(\begin{array}{c}{x}_1^{k+1}\\ x_2^{k+1}\\ {\lambda }^k-\alpha \beta (A_1{x}_1^{k+1}+A_2{x}_2^k-b)\end{array}\right).$$ 19

Lemma 3.1

Let $\{{\lambda }^{k+1}\}$ and $\{{\hat{\lambda }}^k\}$ be the two sequences generated by SGADMM. Then

$${\lambda }^{k+1}={\hat{\lambda }}^k-\beta (A_2{\hat{x}}_2^k-A_2{x}_2^k)$$ 20

and

$${\hat{\lambda }}^k-(\frac{1}{\alpha }-1)({\hat{\lambda }}^k-{\lambda }^k)={\lambda }^k-(2\alpha -1)\beta (A_1{\hat{x}}_1^k+A_2{x}_2^k-b).$$ 21

Proof

From the definition of ${\lambda }^{k+1}$, we get

$$\begin{array}{rcl}{\lambda }^{k+1}& =& {\lambda }^k-\beta [\alpha A_1{\hat{x}}_1^k-(1-\alpha )(A_2{x}_2^k-b)+A_2{\hat{x}}_2^k-b]\\ & =& {\lambda }^k-\beta [\alpha (A_1{\hat{x}}_1^k+A_2{x}_2^k-b)+(A_2{\hat{x}}_2^k-A_2{x}_2^k)]\\ & =& {\hat{\lambda }}^k-\beta (A_2{\hat{x}}_2^k-A_2{x}_2^k).\end{array}$$

Then (20) is proved. For (21), we have

$$\begin{array}{c}{\hat{\lambda }}^k-(\frac{1}{\alpha }-1)({\hat{\lambda }}^k-{\lambda }^k)\hfill \\ ={\lambda }^k-\alpha \beta (A_1{\hat{x}}_1^k+A_2{x}_2^k-b)+(\frac{1}{\alpha }-1)\alpha \beta (A_1{\hat{x}}_1^k+A_2{x}_2^k-b)\hfill \\ ={\lambda }^k-(2\alpha -1)\beta (A_1{\hat{x}}_1^k+A_2{x}_2^k-b).\hfill \end{array}$$

Therefore (21) is also right. This completes the proof. □

Thus, based on (19) and (20), the two sequences $\{w^k\}$ and $\{{\hat{w}}^k\}$ satisfies the following relationship:

$$w^{k+1}=w^k-M(w^k-{\hat{w}}^k),$$ 22

where M is defined in (10).

The following lemma shows that the stopping criterion (18) of SGADMM is reasonable.

Lemma 3.2

If $R_i{x}_i^k=R_i{x}_i^{k+1}$ ($i=1,2$), $A_2{x}_2^k=A_2{x}_2^{k+1}$ and ${\lambda }^k={\lambda }^{k+1}$, then the iterate ${\hat{w}}^k=({\hat{x}}_1^k,{\hat{x}}_2^k,{\hat{\lambda }}^k)$ produced by SGADMM is a solution of $VI(\mathcal{W},F,\theta )$.

Proof

By invoking the optimality condition of the three sub-problems in (4), we have the following mixed variational inequality problems: for any $w=(x_1,x_2,\lambda )\in \mathcal{W}$,

$$\{\begin{array}{c}{\theta }_1(x_1)-{\theta }_1({\hat{x}}_1^k)+{(x_1-{\hat{x}}_1^k)}^{\mathrm{\top }}\{-A_1^{\mathrm{\top }}[{\lambda }^k-\alpha \beta (A_1{\hat{x}}_1^k+A_2{x}_2^k-b)]+R_1({\hat{x}}_1^k-x_1^k)\}\ge 0,\hfill \\ {\theta }_2(x_2)-{\theta }_2({\hat{x}}_2^k)+{(x_2-{\hat{x}}_2^k)}^{\mathrm{\top }}\{-A_2^{\mathrm{\top }}[{\lambda }^k-(2\alpha -1)\beta (A_1{\hat{x}}_1^k+A_2{\hat{x}}_2^k-b)]\}\hfill \\ +R_2({\hat{x}}_2^k-x_2^k)\ge 0,\hfill \\ {(\lambda -{\hat{\lambda }}^k)}^{\mathrm{\top }}[\alpha A_1{\hat{x}}_1^k-(1-\alpha )(A_2{x}_2^k-b)+A_2{\hat{x}}_2^k-b-({\lambda }^k-{\lambda }^{k+1})/\beta ]\ge 0.\hfill \end{array}$$

Then, adding the above three inequalities and by (20), (21), we get

$$\begin{array}{c}\theta (x)-\theta \left({\hat{x}}^k\right)+{(w-{\hat{w}}^k)}^{\mathrm{\top }}\{\left(\begin{array}{c}-A_1^{\mathrm{\top }}{\hat{\lambda }}^k\\ -A_2^{\mathrm{\top }}{\hat{\lambda }}^k\\ A_1{\hat{x}}_1^k+A_2{\hat{x}}_2^k-b\end{array}\right)\hfill \\ +\left(\begin{array}{c}{R}_1({\hat{x}}_1^k-x_1^k)\\ (2\alpha -1)\beta A_2^{\mathrm{\top }}(A_2{\hat{x}}_2^k-A_2{x}_2^k)+(1-\alpha )A_2^{\mathrm{\top }}({\hat{\lambda }}^k-{\lambda }^k)/\alpha +R_2({\hat{x}}_2^k-x_2^k)\\ (1-\alpha )(A_2{\tilde{x}}_2^k-A_2{x}_2^k)/\alpha +({\lambda }^{k+1}-{\lambda }^k)/(\alpha \beta )\end{array}\right)\}\ge 0.\hfill \end{array}$$

Then by (19), we obtain

$$\begin{array}{c}\theta (x)-\theta \left({\hat{x}}^k\right)+{(w-{\hat{w}}^k)}^{\mathrm{\top }}\{F\left({\hat{w}}^k\right)\phantom{\left(\begin{array}{c}{R}_1({\hat{x}}_1^k-x_1^k)\\ (2\alpha -1)\beta A_2^{\mathrm{\top }}(A_2{\tilde{x}}_2^k-A_2{x}_2^k)+(1-\alpha )A_2^{\mathrm{\top }}({\hat{\lambda }}^k-{\lambda }^k)/\alpha +R_2({\hat{x}}_2^k-x_2^k)\\ -(A_2{\hat{x}}_2^k-A_2{x}_2^k)+({\hat{\lambda }}^k-{\lambda }^k)/(\alpha \beta )\end{array}\right)}\hfill \\ +\left(\begin{array}{c}{R}_1({\hat{x}}_1^k-x_1^k)\\ (2\alpha -1)\beta A_2^{\mathrm{\top }}(A_2{\tilde{x}}_2^k-A_2{x}_2^k)+(1-\alpha )A_2^{\mathrm{\top }}({\hat{\lambda }}^k-{\lambda }^k)/\alpha +R_2({\hat{x}}_2^k-x_2^k)\\ -(A_2{\hat{x}}_2^k-A_2{x}_2^k)+({\hat{\lambda }}^k-{\lambda }^k)/(\alpha \beta )\end{array}\right)\}\ge 0.\hfill \end{array}$$

Then, by (11) (the definition of Q), the above inequality can be rewritten as

$$\theta (x)-\theta \left({\hat{x}}^k\right)+{(w-{\hat{w}}^k)}^{\mathrm{\top }}F\left({\hat{w}}^k\right)\ge {(w-{\hat{w}}^k)}^{\mathrm{\top }}Q(w^k-{\hat{w}}^k),$$ 23

for any $w\in \mathcal{W}$. Therefore, if $R_i{x}_i^k=R_i{x}_i^{k+1}$ ($i=1,2$), $A_2{x}_2^k=A_2{x}_2^{k+1}$ and ${\lambda }^k={\lambda }^{k+1}$, then by (20), we have ${\lambda }^{k+1}={\hat{\lambda }}^k$. Then ${\hat{\lambda }}^k={\lambda }^k$. Thus, we have

$$Q(w^k-{\hat{w}}^k)=0,$$

which together with (23) implies that

$$\theta (x)-\theta \left({\hat{x}}^k\right)+{(w-{\hat{w}}^k)}^{\mathrm{\top }}F\left({\hat{w}}^k\right)\ge 0,\mathrm{\forall }w\in \mathcal{W}.$$

This indicates that the vector ${\hat{w}}^k$ is a solution of $VI(\mathcal{W},F,\theta )$. This completes the proof. □

Lemma 3.3

Let $\{w^k\}$ and $\{{\hat{w}}^k\}$ be two sequences generated by SGADMM. Then, for any $w\in \mathcal{W}$, we have

$${(w-{\hat{w}}^k)}^{\mathrm{\top }}Q(w^k-{\hat{w}}^k)\ge \frac{1}{2}({\parallel w-w^{k+1}\parallel }_H^2-{\parallel w-w^k\parallel }_H^2)+\frac{\alpha -1}{2\alpha }{\parallel w^k-w^{k+1}\parallel }_H^2.$$ 24

Proof

Applying the identity

$${(a-b)}^{\mathrm{\top }}H(c-d)=\frac{1}{2}({\parallel a-d\parallel }_H^2-{\parallel a-c\parallel }_H^2)+\frac{1}{2}({\parallel c-b\parallel }_H^2-{\parallel d-b\parallel }_H^2),$$

with

$$a=w,b={\hat{w}}^k,c=w^k,d=w^{k+1},$$

we obtain

$$\begin{array}{rcl}{(w-{\hat{w}}^k)}^{\mathrm{\top }}H(w^k-w^{k+1})& =& \frac{1}{2}({\parallel w-w^{k+1}\parallel }_H^2-{\parallel w-w^k\parallel }_H^2)\\ & & +\frac{1}{2}({\parallel w^k-{\hat{w}}^k\parallel }_H^2-{\parallel w^{k+1}-{\hat{w}}^k\parallel }_H^2).\end{array}$$

This together with (12) and (22) implies that

$$\begin{array}{rcl}{(w-{\hat{w}}^k)}^{\mathrm{\top }}Q(w^k-{\hat{w}}^k)& =& \frac{1}{2}({\parallel w-w^{k+1}\parallel }_H^2-{\parallel w-w^k\parallel }_H^2)\\ & & +\frac{1}{2}({\parallel w^k-{\hat{w}}^k\parallel }_H^2-{\parallel w^{k+1}-{\hat{w}}^k\parallel }_H^2).\end{array}$$ 25

Now let us deal with the last term in (25), which can be written as

$$\begin{array}{c}{\parallel w^k-{\hat{w}}^k\parallel }_H^2-{\parallel w^{k+1}-{\hat{w}}^k\parallel }_H^2\hfill \\ ={\parallel w^k-{\hat{w}}^k\parallel }_H^2-{\parallel (w^k-{\hat{w}}^k)-(w^k-w^{k+1})\parallel }_H^2\hfill \\ ={\parallel w^k-{\hat{w}}^k\parallel }_H^2-{\parallel (w^k-{\hat{w}}^k)-M(w^k-{\hat{w}}^k)\parallel }_H^2(\text{using (22)})\hfill \\ =2{(w^k-{\hat{w}}^k)}^{\mathrm{\top }}HM(w^k-{\hat{w}}^k)-{(w^k-{\hat{w}}^k)}^{\mathrm{\top }}{M}^{\mathrm{\top }}HM(w^k-{\hat{w}}^k)\hfill \\ =(w^k-{\hat{w}}^k)(Q^{\mathrm{\top }}+Q-M^{\mathrm{\top }}HM)(w^k-{\hat{w}}^k)\hfill \\ \ge \frac{\alpha -1}{2\alpha }{(w^k-{\hat{w}}^k)}^{\mathrm{\top }}{M}^{\mathrm{\top }}HM(w^k-{\hat{w}}^k)(\text{using (13)})\hfill \\ =\frac{\alpha -1}{2\alpha }{\parallel w^k-w^{k+1}\parallel }_H^2(\text{using (22)}).\hfill \end{array}$$

Substituting the above inequality into (25), the assertion of this lemma is proved. □

Theorem 3.1

Let $\{w^k\}$ and $\{{\hat{w}}^k\}$ be two sequences generated by SGADMM. Then, for any $w\in \mathcal{W}$, we have

$$\begin{array}{c}\theta (x)-\theta \left({\hat{x}}^k\right)+{(w-{\hat{w}}^k)}^{\mathrm{\top }}F(w)\hfill \\ \ge \frac{1}{2}({\parallel w-w^{k+1}\parallel }_H^2-{\parallel w-w^k\parallel }_H^2)+\frac{\alpha -1}{2\alpha }{\parallel w^k-w^{k+1}\parallel }_H^2.\hfill \end{array}$$ 26

Proof

First, combining (23) and (24), we get

$$\begin{array}{c}\theta (x)-\theta \left({\hat{x}}^k\right)+{(w-{\hat{w}}^k)}^{\mathrm{\top }}F\left({\hat{w}}^k\right)\hfill \\ \ge \frac{1}{2}({\parallel w-w^{k+1}\parallel }_H^2-{\parallel w-w^k\parallel }_H^2)+\frac{\alpha -1}{2\alpha }{\parallel w^k-w^{k+1}\parallel }_H^2.\hfill \end{array}$$

From the monotonicity of $F(\cdot )$, we have

$${(w-{\hat{w}}^k)}^{\mathrm{\top }}(F(w)-F\left({\hat{w}}^k\right))\ge 0.$$

Adding the above two inequalities, we obtain the assertion (26). The proof is completed. □

With the above theorem in hand, we are ready to establish the global convergence of SGADMM for solving $VI(\mathcal{W},F,\theta )$.

Theorem 3.2

Let $\{w^k\}$ be the sequence generated by SGADMM. If $\alpha >1$, $R_i+\beta A_i^{\mathrm{\top }}{A}_i\succ 0$ ($i=1,2$), then the corresponding sequence $\{w^k\}$ converges to some $w^{\mathrm{\infty }}$, which belongs to ${\mathcal{W}}^{\ast }$.

Proof

Setting $w=w^{\ast }$ in (26), we have

$$\begin{array}{c}{\parallel w^k-w^{\ast }\parallel }_H^2-\frac{\alpha -1}{\alpha }{\parallel w^k-w^{k+1}\parallel }_H^2\hfill \\ \ge 2\{\theta \left({\hat{x}}^k\right)-\theta \left(x^{\ast }\right)+{({\hat{w}}^k-w^{\ast })}^{\mathrm{\top }}F\left(w^{\ast }\right)\}+{\parallel w^{k+1}-w^{\ast }\parallel }_H^2\hfill \\ \ge {\parallel w^{k+1}-w^{\ast }\parallel }_H^2,\hfill \end{array}$$

where the second inequality follows from $w^{\ast }\in {\mathcal{W}}^{\ast }$. Thus, we have

$${\parallel w^{k+1}-w^{\ast }\parallel }_H^2\le {\parallel w^k-w^{\ast }\parallel }_H^2-\frac{\alpha -1}{\alpha }{\parallel w^k-w^{k+1}\parallel }_H^2.$$ 27

Summing over $k=0,1,\dots ,\mathrm{\infty }$, it yields

$$\sum _{k=0}^{\mathrm{\infty }}{\parallel w^k-w^{k+1}\parallel }_H^2\le \frac{\alpha }{\alpha -1}{\parallel w^0-w^{\ast }\parallel }_H^2.$$

By $\alpha >1$ and the positive semi-definite of H, the above inequality implies that

$$\underset{k\to \mathrm{\infty }}{lim}{\parallel w^k-w^{k+1}\parallel }_H^2=0.$$

Thus, by the definition of H, we have

$$\underset{k\to \mathrm{\infty }}{lim}{\parallel x_1^k-x_1^{k+1}\parallel }_{R_1}^2=\underset{k\to \mathrm{\infty }}{lim}{\parallel v^k-v^{k+1}\parallel }_{H_1}^2=0,$$ 28

where

$$H_1=\left(\begin{array}{cc}{R}_2+\frac{2{\alpha }^2-2\alpha +1}{\alpha }\beta A_2^{\mathrm{\top }}{A}_2& \frac{1-\alpha }{\alpha }{A}_2^{\mathrm{\top }}\\ \frac{1-\alpha }{\alpha }{A}_2& \frac{1}{\alpha \beta }{I}_l\end{array}\right),$$

is positive definite by $R_2+\beta A_2^{\mathrm{\top }}{A}_2\succ 0$. From (27) again, we have

$${\parallel w^{k+1}-w^{\ast }\parallel }_H^2\le {\parallel w^0-w^{\ast }\parallel }_H^2,$$

which indicates that the sequence $\{H{w}^k\}$ is bounded. Thus, ${\{R_1{x}_1^k\}}_{k=0}^{\mathrm{\infty }}$ and ${\{H_1{v}^k\}}_{k=0}^{\mathrm{\infty }}$ are both bounded. Then ${\{v^k\}}_{k=0}^{\mathrm{\infty }}$ is bounded. If $R_1\succ 0$, ${\{x_1^k\}}_{k=0}^{\mathrm{\infty }}$ is bounded; otherwise, $A_1^{\mathrm{\top }}{A}_1\succ 0$, that is, $A_1$ is full-column rank, which together with $A_1{x}_1=({\lambda }^k-{\lambda }^{k+1})/(\alpha \beta )+(1-\alpha )(A_2{x}_2^k-b)/\alpha -(A_2{x}_2^{k+1}-b)/\alpha$ implies that ${\{x_1^k\}}_{k=0}^{\mathrm{\infty }}$ is bounded. In conclusion, ${\{w^k\}}_{k=0}^{\mathrm{\infty }}$ is bounded.

Then, from (28) and $H_1\succ 0$, the sequence $\{v^k\}$ is convergent. Suppose it converges to $v^{\mathrm{\infty }}$. Let $w^{\mathrm{\infty }}=(x_1^{\mathrm{\infty }},v^{\mathrm{\infty }})$ be a cluster point of $\{w^k\}$ and $\{w^{k_j}\}$ be the corresponding subsequence. On the other hand, by (20) and (28), we have

$$\underset{k\to \mathrm{\infty }}{lim}{R}_1(x_1^k-{\hat{x}}_1^k)=0,\underset{k\to \mathrm{\infty }}{lim}(x_2^k-{\hat{x}}_2^k)=0$$

and

$$\underset{k\to \mathrm{\infty }}{lim}({\lambda }^k-{\hat{\lambda }}^k)=\underset{k\to \mathrm{\infty }}{lim}({\lambda }^k-{\lambda }^{k+1}+\beta (A_2{\hat{x}}_2^k-A_2{x}_2^k))=0.$$

Thus,

$$\underset{k\to \mathrm{\infty }}{lim}Q(w^k-{\hat{w}}^k)=0.$$ 29

Then, taking the limit along the subsequence $\{w^{k_j}\}$ in (23) and using (29), for any $w\in \mathcal{W}$, we obtain

$$\theta (x)-\theta \left(x^{\mathrm{\infty }}\right)+{(w-w^{\mathrm{\infty }})}^{\mathrm{\top }}F\left(w^{\mathrm{\infty }}\right)\ge 0,$$

which indicates that $w^{\mathrm{\infty }}$ is a solution of $VI(\mathcal{W},F,\theta )$. Then, since $w^{\ast }$ in (27) is arbitrary, we can set $w^{\ast }=w^{\mathrm{\infty }}$ and conclude that the whole generated sequence $\{w^k\}$ converges by $R_i+\beta A_i^{\mathrm{\top }}{A}_i\succ 0$ ($i=1,2$). This completes the proof. □

Convergence rate

Now, we are going to prove the worst-case $\mathcal{O}(1/t)$ convergence rate of SGADMM in both the ergodic and the non-ergodic senses.

Theorem 3.3

Let $\{w^k\}$ and $\{{\hat{w}}^k\}$ be the sequences generated by SGADMM, and set

$${\overline{w}}^t=\frac{1}{t+1}\sum _{k=0}^t{\hat{w}}^k.$$

Then, for any integer $t\ge 0$, we have ${\overline{w}}^t\in \mathcal{W}$, and

$$\theta ({\overline{x}}_t)-\theta (x)+{({\overline{w}}_t-w)}^{\mathrm{\top }}F(w)\le \frac{1}{2(t+1)}{\parallel w-w^0\parallel }_H^2,\mathrm{\forall }w\in \mathcal{W}.$$ 30

Proof

From (17) and the convexity of the set $\mathcal{W}$, we have ${\overline{w}}^k\in \mathcal{W}$. From (26), we have

$$\theta (x)-\theta \left({\hat{x}}^k\right)+{(w-{\hat{w}}^k)}^{\mathrm{\top }}F(w)+\frac{1}{2}{\parallel w-w^k\parallel }_H^2\ge \frac{1}{2}{\parallel w-w^{k+1}\parallel }_H^2,\mathrm{\forall }w\in \mathcal{W}.$$

Summing the above inequality over $k=0,1,\dots ,t$, we get

$$(t+1)\theta (x)-\sum _{k=0}^t\theta \left({\hat{x}}^k\right)+{((t+1)w-\sum _{k=0}^t{\hat{w}}^k)}^{\mathrm{\top }}F(w)+\frac{1}{2}{\parallel w-w^0\parallel }_H^2\ge 0,\mathrm{\forall }w\in \mathcal{W}.$$

By the definition of ${\overline{w}}^t$ and the convexity of $\theta (\cdot )$, the assertion (30) follows immediately from the above inequality. This completes the proof. □

The proof of the next two lemmas is referred to those of Lemmas 5.1 and 5.2 in [24]. For completeness, we give the detail proof.

Lemma 3.4

Let $\{w^k\}$ be the sequence generated by SGADMM. Then we have

$$\begin{array}{c}{(w^k-w^{k+1})}^{\mathrm{\top }}H\{(w^k-w^{k+1})-(w^{k+1}-w^{k+2})\}\hfill \\ \ge \frac{3\alpha -1}{4\alpha }{\parallel (w^k-w^{k+1})-(w^{k+1}-w^{k+2})\parallel }_H^2.\hfill \end{array}$$ 31

Proof

Setting $w={\hat{w}}^{k+1}$ in (23), we have

$$\theta \left({\hat{x}}^{k+1}\right)-\theta \left({\hat{x}}^k\right)+{({\hat{w}}^{k+1}-{\hat{w}}^k)}^{\mathrm{\top }}F\left({\hat{w}}^k\right)\ge {({\hat{w}}^{k+1}-{\hat{w}}^k)}^{\mathrm{\top }}Q(w^k-{\hat{w}}^k).$$

Similarly setting $w={\hat{w}}^k$ in (23) for $k:=k+1$, we get

$$\theta \left({\hat{x}}^k\right)-\theta \left({\hat{x}}^{k+1}\right)+{({\hat{w}}^k-{\hat{w}}^{k+1})}^{\mathrm{\top }}F\left({\hat{w}}^{k+1}\right)\ge {({\hat{w}}^k-{\hat{w}}^{k+1})}^{\mathrm{\top }}Q(w^{k+1}-{\hat{w}}^{k+1}).$$

Then, adding the above two inequalities and using the monotonicity of the mapping $F(\cdot )$, we get

$${({\hat{w}}^k-{\hat{w}}^{k+1})}^{\mathrm{\top }}Q\{(w^k-{\hat{w}}^k)-(w^{k+1}-{\hat{w}}^{k+1})\}\ge 0.$$ 32

By (32), we have

$$\begin{array}{c}{(w^k-w^{k+1})}^{\mathrm{\top }}Q\{(w^k-{\hat{w}}^k)-(w^{k+1}-{\hat{w}}^{k+1})\}\hfill \\ ={\{(w^k-{\hat{w}}^k)-(w^{k+1}-{\hat{w}}^{k+1})+({\hat{w}}^k-{\hat{w}}^{k+1})\}}^{\mathrm{\top }}Q\{(w^k-{\hat{w}}^k)-(w^{k+1}-{\hat{w}}^{k+1})\}\hfill \\ ={\parallel (w^k-{\hat{w}}^k)-(w^{k+1}-{\hat{w}}^{k+1})\parallel }_Q^2+{({\hat{w}}^k-{\hat{w}}^{k+1})}^{\mathrm{\top }}Q\{(w^k-{\hat{w}}^k)-(w^{k+1}-{\hat{w}}^{k+1})\}\hfill \\ \ge {\parallel (w^k-{\hat{w}}^k)-(w^{k+1}-{\hat{w}}^{k+1})\parallel }_Q^2.\hfill \end{array}$$

Using (13), (22) and $Q=HM$ on both sides of the above inequality, we get

$$\begin{array}{c}{(w^k-w^{k+1})}^{\mathrm{\top }}H\{(w^k-w^{k+1})-(w^{k+1}-w^{k+2})\}\hfill \\ ={(w^k-w^{k+1})}^{\mathrm{\top }}Q{M}^{-1}\{(w^k-w^{k+1})-(w^{k+1}-w^{k+2})\}\hfill \\ ={(w^k-w^{k+1})}^{\mathrm{\top }}Q\{(w^k-{\hat{w}}^k)-(w^{k+1}-{\hat{w}}^{k+1})\}\hfill \\ \ge {\parallel (w^k-{\hat{w}}^k)-(w^{k+1}-{\hat{w}}^{k+1})\parallel }_Q^2\hfill \\ ={[(w^k-{\hat{w}}^k)-(w^{k+1}-{\hat{w}}^{k+1})]}^{\mathrm{\top }}Q[(w^k-{\hat{w}}^k)-(w^{k+1}-{\hat{w}}^{k+1})]\hfill \\ ={[(w^k-w^{k+1})-(w^{k+1}-w^{k+2})]}^{\mathrm{\top }}{M}^{-1}Q{M}^{-1}[(w^k-w^{k+1})-(w^{k+1}-w^{k+2})]\hfill \\ \ge \frac{3\alpha -1}{4\alpha }{[(w^k-w^{k+1})-(w^{k+1}-w^{k+2})]}^{\mathrm{\top }}\hfill \\ \times M^{-1}MHM{M}^{-1}[(w^k-w^{k+1})-(w^{k+1}-w^{k+2})]\hfill \\ =\frac{3\alpha -1}{4\alpha }{\parallel (w^k-w^{k+1})-(w^{k+1}-w^{k+2})\parallel }_H^2.\hfill \end{array}$$

Then we get the assertion (31). The proof is completed. □

Lemma 3.5

Let $\{w^k\}$ be the sequence generated by SGADMM. Then we have

$${\parallel w^{k+1}-w^{k+2}\parallel }_H^2\le {\parallel w^k-w^{k+1}\parallel }_H^2-\frac{\alpha -1}{2\alpha }{\parallel (w^k-w^{k+1})-(w^{k+1}-w^{k+2})\parallel }_H^2.$$ 33

Proof

Setting $a:=(w^k-w^{k+1})$ and $b:=(w^{k+1}-w^{k+2})$ in the identity

$${\parallel a\parallel }_H^2-{\parallel b\parallel }_H^2=2a^{\mathrm{\top }}H(a-b)-{\parallel a-b\parallel }_H^2,$$

we can derive

$$\begin{array}{c}{\parallel w^k-w^{k+1}\parallel }_H^2-{\parallel w^{k+1}-w^{k+2}\parallel }_H^2\hfill \\ =2{(w^k-w^{k+1})}^{\mathrm{\top }}H\{(w^k-w^{k+1})-(w^{k+1}-w^{k+2})\}-{\parallel (w^k-w^{k+1})-(w^{k+1}-w^{k+2})\parallel }_H^2\hfill \\ \ge \frac{3\alpha -1}{2\alpha }{\parallel (w^k-w^{k+1})-(w^{k+1}-w^{k+2})\parallel }_H^2-{\parallel (w^k-w^{k+1})-(w^{k+1}-w^{k+2})\parallel }_H^2\hfill \\ =\frac{\alpha -1}{2\alpha }{\parallel (w^k-w^{k+1})-(w^{k+1}-w^{k+2})\parallel }_H^2,\hfill \end{array}$$

which completes the proof of the lemma. □

Based on Lemma 3.5, now we establish the worst-case $\mathcal{O}(1/t)$ convergence rate of SGADMM in a non-ergodic sense.

Theorem 3.4

Let $\{w^k\}$ be the sequence generated by SGADMM. Then, for any $w^{\ast }\in {\mathcal{W}}^{\ast }$ and integer $t\ge 0$, we have

$${\parallel w^t-w^{t+1}\parallel }_H^2\le \frac{\alpha }{(t+1)(\alpha -1)}{\parallel w^0-w^{\ast }\parallel }_H^2.$$ 34

Proof

By (27), we get

$$\frac{\alpha -1}{\alpha }\sum _{k=0}^t{\parallel w^k-w^{k+1}\parallel }_H^2\le {\parallel w^0-w^{\ast }\parallel }_H^2.$$

This and (33) imply that

$$\frac{(t+1)(\alpha -1)}{\alpha }{\parallel w^t-w^{t+1}\parallel }_H^2\le {\parallel w^0-w^{\ast }\parallel }_H^2.$$

Therefore, the assertion of this theorem comes from the above inequality immediately. The proof is completed. □

Remark 3.2

From (34), we see that the larger α is, the smaller $\frac{\alpha }{\alpha -1}$, which controls the upper bounds of ${\parallel w^t-w^{t+1}\parallel }_H^2$. Therefore, it seems that larger values of α are more beneficial for speeding up the convergence of SGADMM.

Numerical experiments

In this section, we present some numerical experiments to verify the efficiency of SGADMM for solving compressed sensing. Those numerical experiments are performed in Matlab R2010a on a ThinkPad computer equipped with Windows XP, 997 MHz and 2 GB of memory.

Compressed sensing (CS) is to recover a sparse signal $\overline{x}\in {\mathcal{R}}^n$ from an undetermined linear system $b=A\overline{x}$, where $A\in {\mathcal{R}}^{m\times n}$ ($m\ll n$), can be depicted as problem (2).

Obviously, Problem (2) is equivalent to the following two models:

1. Model 1: Problem (3).

2. Model 2:

   $$\begin{array}{rl}& min\mu {\parallel x_1\parallel }_1+\frac{1}{2}{\parallel A{x}_2-y\parallel }^2\\ & \text{s.t. }-x_1-x_2=0,\\ & \phantom{\text{s.t.}}{x}_1\in {\mathcal{R}}^n,x_2\in {\mathcal{R}}^n.\end{array}$$ 35

The iterative schemes for (3) and (35)

Since (3) and (35) are both some concrete models of (1), SGADMM are applicable to them. Below, we elaborate on how to derive the closed-form solutions for the sub-problems resulting by SGADMM.

For problem (3), its first two sub-problems resulting by SGADMM are as follows.

• With the given $x_2^k$ and ${\lambda }^k$, the $x_1$-sub-problem in (17) is (here $R_1=0$)

$$x_1^{k+1}=\underset{x_1\in {\mathcal{R}}^n}{argmin}\{\frac{1}{2}{\parallel x_1\parallel }_2^2+x_1^{\mathrm{\top }}\lambda +\frac{\alpha \beta }{2}{\parallel x_1-A{x}_2^k+y\parallel }^2\},$$

which has the following closed-form solution:

$$x_1^{k+1}=\frac{1}{1+\alpha \beta }(\alpha \beta (A{x}_2^k-y)-{\lambda }^k).$$

• With the updated $x_1^{k+1}$, the $x_2$-sub-problem in (17) is (here $R_2=\tau I_n-(2\alpha -1)\beta A^{\mathrm{\top }}A$ with $\tau \ge (2\alpha -1)\beta \parallel A^{\mathrm{\top }}A\parallel$)

$$x_2^{k+1}=\underset{x_2\in {\mathcal{R}}^n}{argmin}\{\mu {\parallel x_2\parallel }_1-x_2^{\mathrm{\top }}{A}^{\mathrm{\top }}{\lambda }^k+\frac{(2\alpha -1)\beta }{2}{\parallel -x_1^{k+1}+A{x}_2-y\parallel }^2+\frac{1}{2}{\parallel x_2-x_2^k\parallel }_{R_2}^2\},$$

and its closed-form solution is given by

$$x_2^{k+1}={shrink}_{\frac{\mu }{\tau }}((2\alpha -1)\beta A^{\mathrm{\top }}(x_1^{k+1}+y)/\tau +(\tau I_n-(2\alpha -1)\beta A^{\mathrm{\top }}A)x_2^k/\tau +A^{\mathrm{\top }}{\lambda }^k/\tau ),$$

where, for any $c>0$, ${shrink}_c(\cdot )$ is defined as

$${shrink}_c(g):=g-min\{c,|g|\}\frac{g}{|g|},\mathrm{\forall }g\in {\mathcal{R}}^n,$$

and ${(g/|g|)}_i$ should be taken 0 if ${|g|}_i=0$.

Similarly, for problem (35), its first two sub-problems resulting by SGADMM are as follows.

• With the given $x_2^k$ and ${\lambda }^k$, the $x_1$-sub-problem in (17) is (here $R_1=0$)

$$x_1^{k+1}=\underset{x_1\in {\mathcal{R}}^n}{argmin}\{\mu {\parallel x_1\parallel }_1+\frac{\alpha \beta }{2}{\parallel x_1-(x_2^k+\frac{1}{\alpha \beta }{\lambda }^k)\parallel }^2\},$$

and its closed-form solution is given by

$$x_1^{k+1}={shrink}_{\frac{\mu }{\alpha \beta }}(x_2^k+\frac{1}{\alpha \beta }{\lambda }^k).$$

• With the updated $x_1^{k+1}$, the $x_2$-sub-problem in (17) is (here $R_2=\tau I_n-A^{\mathrm{\top }}A$ with $\tau \ge \parallel A^{\mathrm{\top }}A\parallel$)

$$x_2^{k+1}=\underset{x_2\in {\mathcal{R}}^n}{argmin}\{\frac{1}{2}{\parallel A{x}_2-y\parallel }^2+x_2^{\mathrm{\top }}{\lambda }^k+\frac{(2\alpha -1)\beta }{2}{\parallel x_2-x_1^{k+1}\parallel }^2+\frac{1}{2}{\parallel x_2-x_2^k\parallel }_{R_2}^2\},$$

and its closed-form solution is given by

$$x_2^{k+1}=\frac{1}{\tau +(2\alpha -1)\beta }(A^{\mathrm{\top }}y-{\lambda }^k+(2\alpha -1)\beta x_1^{k+1}+A^{\mathrm{\top }}A{x}_2^k).$$

Obviously, the above two iterative schemes both need to compute $A^{\mathrm{\top }}A$ and $A^{\mathrm{\top }}y$, which is quite time consuming if n is large. However, noting that these two terms are invariant during the iteration process, therefore we need only compute them once before all iterations.

Regarding the penalty parameter β and the constant α in SGADMM, any $\beta >0$ and $\alpha \ge 1$ can ensure the convergence of SGADMM in theory. There are two traditional methods to determine them in practice. One is the tentative method, which is easy to execute. The other is the self-adaptive adjustment method, which needs much computation. In this experiment, for β and α, we use the tentative method to determine their suitable values. For β, Xiao et al. [35] set $\beta =\mathtt{mean}(\mathtt{abs}(y))$ for ADMM. Motivated by this choice, we set $\beta =\mathtt{mean}(\mathtt{abs}(y))/(2\alpha -1)$ in our algorithm. As for the parameter α, we have pointed out in Remark 3.2 that larger values of α may be beneficial for our algorithm. Here, we use (3) to do a little experiment to test this. We choose different values of α in the interval $[1,2]$. Specifically, we choose $\alpha \in \{1.0,1.1,\dots ,2\}$. Other data about this experiment are as follows: the proximal parameter τ is set as $\tau =1.01(2\alpha -1)\beta \parallel A^{\mathrm{\top }}A\parallel$; the observed signal y is set as $y=Ax+0.01\times \mathtt{randn}(m,1)$ in Matlab; the sensing matrix A and the original signal x are generated by

$$\overline{A}=\mathtt{randn}(m,n),[Q,R]=\mathtt{qr}({\overline{A}}^{\prime },0),A=Q^{\prime },$$

and

$$x=\mathtt{zeros}(n,1);p=\mathtt{randperm}(n);x(p(1:k))=\mathtt{randn}(k,1).$$

Then the observed signal y is further set as ${(R^{\mathrm{\top }})}^{-1}y$. The initial points are set as $x_2^0=A^{\mathrm{\top }}y$, ${\lambda }^0=A{x}_2^0$. In addition, we set the regularization parameter $\mu =0.01$, and the dimensions of the problem are set as $n=1\text{,}000$, $m=300$, $k=60$, where k denotes the number of the non-zeros in the original signal x. To evaluate the quality of the recovered signal, let us define the quantity ‘RelErr’ as follows:

$$RelErr=\frac{\parallel \tilde{x}-x\parallel }{\parallel x\parallel },$$

where x̃ denotes the recovered signal. The stopping criterion is

$$\frac{\parallel f_k-f_{k-1}\parallel }{\parallel f_{k-1}\parallel }<{10}^{-5},$$

where $f_k$ denotes the function value of (2) at the iterate $x_k$.

Numerical results

The numerical results are graphicly shown in Figure 1. Clearly, the numerical results in Figure 1 indicate that Remark 3.2 is reasonable. Both CPU time and number of iterations are descent with respect to α. Then, in the following, we set $\alpha =1.4$, which is a moderate choice for α.

Figure 1.

Sensitivity test on the parameter α .

Now, let us graphically show the recovered results of SGADMM for (3) and (35). The proximal parameter τ is set as $\tau =1.01(2\alpha -1)\beta \parallel A^{\mathrm{\top }}A\parallel$ for (3), and $\tau =1.01\parallel A^{\mathrm{\top }}A\parallel$ for (35). The initial points are set as $x_2^0=A^{\mathrm{\top }}y$, ${\lambda }^0=A{x}_2^0$ for (3), and $x_2^0=A^{\mathrm{\top }}y$, ${\lambda }^0=x_2^0$ for (35). Other parameters are set the same as above. Figure 2 reports the numerical results of SGADMM for (3) and (35).

Figure 2.

Numerical results of SGADMM for ( 3 ) and ( 35 ). The top: the original signal; the second: the noisy measurement; the bottom two: recovered signal.

The bottom two subplots in Figure 2 indicate that our new method SGADMM can be used to solve (3) and (35).

In the following, we do some numerical comparisons to illustrate the advantage of our new method and to analyze which one is more suitable to compressed sensing (2) between the two models (3) and (35). SGADMM for (3) is denoted by SGADMM1, SGADMM for (35) is denoted by SGADMM2. We also compare SGADMM with the classical ADMM. The numerical results are listed in Table 1, where ‘Time’ denotes the CPU time (in seconds), and ‘Iter’ denotes the number of iterations required for the whole recovering process, $m=\mathtt{floor}(\gamma n)$, $k=\mathtt{floor}(\sigma m)$. The numerical results are the average of the numerical results of ten runs with different combinations of γ and σ.

Table 1.

Comparison of SGADMM1, SGADMM2 and ADMM

n	γ	σ	SGADMM1			SGADMM2			ADMM
			Time	Iter	RelErr	Time	Iter	RelErr	Time	Iter	RelErr
1,000	0.3	0.2	0.6911	92.4	0.0387	0.9578	266.0	0.0394	0.9812	264.0	0.0393
	0.2	0.2	0.6661	118.6	0.0825	1.3915	421.8	0.0915	1.3603	419.6	0.0915
	0.2	0.1	0.5008	85.3	0.0609	0.4758	139.0	0.0579	0.5008	138.0	0.0539
2,000	0.3	0.2	2.1965	90.0	0.0437	3.6535	267.7	0.0447	3.5412	265.6	0.0447
	0.2	0.2	2.2339	109.6	0.0785	5.2182	431.4	0.0874	5.1543	429.0	0.0874
	0.2	0.1	1.5428	79.9	0.0534	1.7893	142.8	0.0467	1.7613	140.8	0.0513

Discussion

The numerical results in Table 1 indicate that: (1) by the criterion ‘RelErr’, all methods successfully solved all the cases; (2) by the criteria ‘Time’ and ‘Iter’, SGADMM1 performs better than the other two methods. Especially the number of iterations of SGADMM1 is about at most two-thirds of the other two methods. This experiment also indicate that the model (3) is also an effective model for compressed sensing, and sometimes it is more efficient than the model (35), though they are equivalent in theory. In conclusion, by choosing some relaxation factor $\alpha \in [1,+\mathrm{\infty })$, SGADMM may be more efficient than the classical ADMM.

Conclusions

In this paper, we have proposed a symmetric version of the generalized ADMM (SGADMM), which generalizes the feasible set of the relaxation factor α from $(0,2)$ to $[1,+\mathrm{\infty })$. Under the same conditions, we have proved the convergence results of the new method. Some numerical results illustrate that it may perform better than the classical ADMM. In the future, we shall study SGADMM with $\alpha \in (0,1)$ to perfect the theoretical system.