Feature-splitting Algorithms for Ultrahigh Dimensional Quantile Regression

Abstract

This paper is concerned with computational issues related to penalized quantile regression (PQR) with ultrahigh dimensional predictors. Various algorithms have been developed for PQR, but they become ineffective and/or infeasible in the presence of ultrahigh dimensional predictors due to the storage and scalability limitations. The variable updating schema of the feature-splitting algorithm that directly applies the ordinary alternating direction method of multiplier (ADMM) to ultrahigh dimensional PQR may make the algorithm fail to converge. To tackle this hurdle, we propose an efficient and parallelizable algorithm for ultrahigh dimensional PQR based on the three-block ADMM. The compatibility of the proposed algorithm with parallel computing alleviates the storage and scalability limitations of a single machine in the large-scale data processing. We establish the rate of convergence of the newly proposed algorithm. In addition, Monte Carlo simulations are conducted to compare the finite sample performance of the proposed algorithm with that of other existing algorithms. The numerical comparison implies that the proposed algorithm significantly outperforms the existing ones. We further illustrate the proposed algorithm via an empirical analysis of a real-world data set.

1. Introduction

Quantile regression (QR) is well acknowledged as a powerful tool for analyzing data with heterogeneous effect. Since the seminal work of Koenker and Bassett (1978), QR has been extensively applied in many research fields, in particular, in econometrics. For a complete review of QR, refer to Koenker (2017) and Koenker et al. (2017). Many recent advances and achievements of QR can be found in literature. Wang and He (2022) provided a unified theory for high-dimensional quantile regression with both convex and nonconvex penalties. Gimenes and Guerre (2022) proposed a QR inference framework for first-price auctions, and Cai et al. (2022) reexamined the heterogeneous predictability of US stock returns at different quantile levels. Other recent studies of QR include, but not limited to, D’Haultfœuille et al. (2018), Altunbaş and Thornton (2019), Giessing and He (2019), Gu and Volgushev (2019), Firpo et al. (2022), He et al. (2022) , and Narisetty and Koenker (2022).

For variable selection in QR, penalized quantile regression (PQR) has been developed with fixed and finite dimensional predictors in Li and Zhu (2008) and Wu and Liu (2009). Furthermore, PQR with high-dimensional predictors has also been studied in the statistical literature, since as the advent of data science, high-dimensional data analysis has become one of the most important research topics in the last decade. Belloni and Chernozhukov (2011) derived a nice error bound for PQR with the Lasso penalty (ℓ₁-QR for short). Wang et al. (2012) studied the PQR with folded concave penalty such as the smoothed clip absolute deviation (SCAD) penalty (Fan and Li, 2001) and minimax concave penalty (MCP) (Zhang, 2010), and further established the oracle property for PQR in the ultrahigh dimension setting under mild conditions. In summary, estimation and theory of PQR are well studied and understood in the literature.

The numerical minimization problem for searching solutions to PQR, however, is challenging due to the nonsmoothness objective function with the possible nonconvexity of folded concave penalty. Sherwood and Maidman (2017) developed an R-package rqPen for ℓ₁-QR, and the algorithm is similar to the ℓ₁-QR introduced in Koenker and Mizera (2014). Peng and Wang (2015) developed an iterative coordinate descent algorithm (QICD) for solving PQR with nonconvex penalty. Gu et al. (2018) introduced fast alternating direction method of multiplier (ADMM) (Boyd et al., 2011) for PQR in high dimension.

As the advent of big data, it is of crucial importance to study numerical algorithms for PQR in ultrahigh dimension and/or a large data size. The ADMM (Boyd et al., 2011) has been introduced to cope with PQR with a large data size. Yu et al. (2017) and Fan et al. (2021) developed parallel algorithms for PQR based on sample-splitting ADMM. By sample-splitting, it means by its name that the algorithm partitions the data across samples. Ultrahigh dimensionality adds another challenge in minimizing the objective function of ultrahigh dimensional PQR. This work aims to tackle simultaneous challenges of nonsmoothness, nonconvexity and ultrahigh dimensionality by developing feature-splitting algorithms for PQR.

In this paper, we propose an efficient and parallelizable algorithm for PQR in ultrahigh dimension based on three-block ADMM. It is noteworthy that Yu and Lin (2017) briefly mentioned one direct extension of the feature-splitting ADMM for PQR without theoretical justifications and numerical studies. The variable update schema in Yu and Lin (2017) makes the convergence of the algorithm uncertain. Chen et al. (2016) showed that Gauss-Seidel multi-block ADMM is not necessarily convergent. For more detailed discussion on this, see Section 2. The uncertain convergence motivates us to avoid the direct extension of the feature-splitting ADMM, and instead to develop a three-block ADMM algorithm for ultrahigh dimensional PQR. Using related techniques in Sun et al. (2015), we establish the rate of convergence of the proposed algorithm and the theoretical convergence guarantee, and address the convergence uncertainty. The compatibility of the proposed three-block ADMM algorithm with parallel computing alleviates the storage and scalability limitations of a single machine in the large-scale data processing. The proposed three-block ADMM algorithms also enjoy numerical efficiency over the directly extended two-block ADMM. It is worthy to note that the newly proposed algorithms can be directly applied to PQR with various penalties including the ${\ell }_1$, the SCAD and the MCP penalties by local linear approximation to the penalties (Zou and Li, 2008). Based on theories developed in Wang et al. (2013) and Fan et al. (2014), the proposed algorithms are able to obtain an PQR estimate with strong oracle property in ultrahigh dimension.

The rest of this article is organized as follows. In Section 2, we present the computational framework based on the three-block ADMM for PQR and establish the linear rate of convergence of the algorithm. In Section 3, we demonstrate the numerical and statistical efficiency of the proposed framework in high and ultra-high dimensional settings through Monte Carlo simulation, and illustrate the proposed algorithm via an empirical analysis of a Chinese supermarket data set. Technical proofs are given in the Appendix.

Throughout the paper, we adopt the following notations. For a matrix $\mathit{M}={\left(m_{ij}\right)}_{s\times t}$, denote ${\parallel \mathit{M}\parallel }_{\max }={\max }_{(i,j)}\left|m_{ij}\right|,{\parallel \mathit{M}\parallel }_{\min }={\min }_{(i,j)}\left|m_{ij}\right|$, and ${\lambda }_{\min }(\mathit{M})$ and ${\lambda }_{\max }(\mathit{M})$ as the smallest and largest eigenvalues of M, respectively. X_A denotes the sub-matrix of X with the columns indexed by A. $\mathit{M}\succ 0$ indicates that M is positive definite. For a positive semidefinite operator or matrix $\mathit{M},{\parallel \mathit{x}\parallel }_M^2={\mathit{x}}^T\mathit{M}\mathit{x}$.

2. Feature-splitting Algorithms for PQR

Suppose that $\left\{{\mathit{x}}_i,y_i\right\},i=1,\cdots ,n$, is a random sample from linear regression model

$$y_i={\mathit{x}}_i^T\mathit{\beta }+{\epsilon }_i,$$

where $\mathit{\beta }$ is p-dimensional vector of regression coefficients, and ${\epsilon }_i$ is a random error with $E\left({\epsilon }_i\mid {\mathit{x}}_i\right)=0$. In this paper, we are interested in solving QR in ultrahigh dimensional regime, in which $p\gg n$. Define $\mathit{y}={\left(y_1,\cdots ,y_n\right)}^T$ as the response vector, and $\mathit{X}={\left({\mathit{x}}_1,\cdots ,{\mathit{x}}_n\right)}^T$ as the corresponding design matrix. For a given $\tau \in (0,1)$, the quantile of interest, define the loss function ${\rho }_{\tau }(z)=z[\tau -I(z<0)]=\tau {(z)}_{+}+(1-\tau ){(z)}_{-}$ where I(⋅) is the indicator function, ${(z)}_{+}=\max \{0,z\}$, and ${(z)}_{-}={(-z)}_{+}$. The QR is to minimize its objective function

$$L(\mathit{y}-\mathit{X}\mathit{\beta })=\frac{1}{n}\sum _{i=1}^n{\rho }_{\tau }(y_i-x_i^T\mathit{\beta })$$ (1)

with respect to $\mathit{\beta }$, and this leads to the QR estimate of $\mathit{\beta }$. The minimization problem in QR can be reformulated as a linear programming problem. The Frisch-Newton algorithm can be applied to solve the minimization problem with computational complexity growing as a cubic function of p when p < n.

2.1. Penalized quantile regression

In the presence of ultrahigh dimensional predictors, it is common to impose sparsity assumption on $\mathit{\beta }$. That is, only a small portion of elements in $\mathit{\beta }$ are nonzero. This implies that only a small portion of predictors are significant in the model. Thus, it is critical to identify the significant predictors in QR in ultrahigh dimension. The variable selection in QR is similar to that in linear regression for which penalized least squares methods have been proposed. It is then natural to extend the PQR method to the variable selection for QR. With PQR, it is to minimize the penalized quantile loss function

$$Q(\mathit{\beta })=L(\mathit{y}-\mathit{X}\mathit{\beta })+\sum _{j=1}^p{p}_{{\lambda }_j}\left(\left|{\beta }_j\right|\right),$$ (2)

where $p_{{\lambda }_j}(\cdot )$ is a penalty function with a regularization parameter ${\lambda }_j$ that controls model complexity. The algorithms to be developed in this paper allow that different regression coefficients have different penalties, although it is common to take all $p_{{\lambda }_j}(\cdot )$ to be the same and denoted by $p_{\lambda }(\cdot )$. This paper concentrates on the two most commonly-used penalties: the Lasso (i.e., ${\ell }_1$) penalty $p_{\lambda }(|\beta |)=\lambda |\beta |$ and the SCAD penalty whose first derivative is defined as

$$p_{\lambda }^{\prime }\left(\left|\beta \right|\right)=\lambda \left\{I\left(\left|\beta \right|\le \lambda \right)+\frac{{\left(a\lambda -\left|\beta \right|\right)}_{+}}{\left(a-1\right)\lambda }I\left(\left|\beta \right|>\lambda \right)\right\}$$ (3)

with $p_{\lambda }^{\prime }(0):=p_{\lambda }^{\prime }(0+)=\lambda$ and a = 3.7 as suggested in Fan and Li (2001). The proposed algorithms are directly applicable for other folded concave penalties (Fan et al., 2020).

It is challenging in minimizing the objective function of PQR in (2) since both the loss function and penalty function are nonsmoothing. When folded concave penalty such as the SCAD penalty is used in ultrahigh dimensional PQR, the minimization problem becomes even more challenging due to its nonconvexity and ultrahigh dimensionality. It is noteworthy that it is a convex minimization problem for the PQR with the ${\ell }_1$ penalty, and when $p\le n$, it has unique minimizer. For the PQR with a folded concave penalty, minimizing the objective function in (2) may be achieved by iteratively minimizing PQR with reweighted ${\ell }_1$ penalty with the aid of the local linear approximation (LLA) to the penalty function. Specifically, given ${\mathit{\beta }}^k={\left({\beta }_1^k,\cdots ,{\beta }_p^k\right)}^T$ updated from the k-th step in the course of iterations, we first approximate

$$p_{\lambda }(|{\beta }_j|)\approx q_{\lambda }(|{\beta }_j|;|{\beta }_j^k|)=p_{\lambda }(|{\beta }_j^k|)+p_{\lambda }^{\prime }(|{\beta }_j^k|)(|{\beta }_j|-|{\beta }_j^k|),$$ (4)

which is referred to as the LLA. Then at the $(k+1)$-th step we minimize

$$Q^{k+1}(\mathit{\beta })=L(\mathit{y}-\mathit{X}\mathit{\beta })+\lambda \sum _{j=1}^p{\lambda }^{-1}{p}_{{\lambda }_j}^{\prime }(|{\beta }_j^k|)|{\beta }_j|=L(\mathit{y}-\mathit{X}\mathit{\beta })+\lambda \sum _{j=1}^p{\alpha }_j|{\beta }_j|,$$ (5)

where ${\alpha }_j={\lambda }^{-1}{p}_{{\lambda }_j}^{\prime }(|{\beta }_j^k|)\ge 0$. The function in (5) is the objective function of PQR with reweighted ${\ell }_1$ penalty and weights ${\alpha }_j$’s that are updated at every step.

The LLA was first proposed in Zou and Li (2008) for penalized likelihood with finite dimensional predictors, and further adopted in Wang et al. (2013) and Fan et al. (2014) for penalized least squares for ultrahigh dimensional linear regression models. Note that if we set initial value ${\mathit{\beta }}^0=\mathbf{0}$, the ${\mathit{\beta }}^1$ is the PQR-Lasso estimator defined as the PQR with ${\ell }_1$ penalty. Then ${\mathit{\beta }}^2$ can be regarded as the one-step sparse estimator with initial value being the PQR-Lasso estimator. With properly chosen tuning parameters, Wang et al. (2013) and Fan et al. (2014) showed that under some regularity conditions on high dimensional linear models, the corresponding penalized least squares estimator ${\mathit{\beta }}^2$ enjoys the strong oracle property with probability tending to one. This motivates us to focus on developing feature-splitting algorithm for PQR with weighted ${\ell }_1$ penalty.

2.2. Three-block ADMM

Define the PQR estimator with weighted ${\ell }_1$-penalty to be

$$\widehat{\mathit{\beta }}=\underset{\mathit{\beta }}{\text{argmin}}L(\mathit{y}-\mathit{X}\mathit{\beta })+\lambda {\parallel \mathit{\alpha }\circ \mathit{\beta }\parallel }_1,$$ (6)

where ${\parallel \mathit{\alpha }\circ \mathit{\beta }\parallel }_1={\sum }_{j=1}^p|{\alpha }_j{\beta }_j|$ with $\mathit{\alpha }$ being the weight vector.

The non-smoothness of the objective function in (6) hinders an efficient application of gradient-based methods. To decouple the non-smooth parts in computation, we decentralize problem (6) into the following constrained optimization problem,

$$\underset{\mathit{\beta },\mathit{z}}{\min }L(\mathit{z})+\lambda {\parallel \mathit{\alpha }\circ \mathit{\beta }\parallel }_1,\text{s.t.}\mathit{z}+\mathit{X}\mathit{\beta }=\mathit{y}.$$ (7)

Problem (7) is a natural candidate of classical two-block ADMM algorithm. Define augmented Lagrangian function as

$${ℒ}_{\varphi }(\mathit{\beta },\mathit{z};\mathit{\gamma })=L(\mathit{z})+\lambda {\parallel \mathit{\alpha }\circ \mathit{\beta }\parallel }_1+\langle \mathit{\gamma },\mathit{z}+\mathit{X}\mathit{\beta }-\mathit{y}\rangle +\frac{\varphi }{2}{\parallel \mathit{z}+\mathit{X}\mathit{\beta }-\mathit{y}\parallel }_2^2,$$ (8)

where $\mathit{\gamma }\in {ℛ}^n$ is the Lagrangian multiplier, and ϕ > 0 is the parameter associated with the quadratic term. The classic iterative scheme at the iteration k for two-block ADMM is

$${\mathit{\beta }}^{k+1}=\underset{\mathit{\beta }}{\text{argmin}}{ℒ}_{\varphi }(\mathit{\beta },{\mathit{z}}^k;{\mathit{\gamma }}^k),$$

$${\mathit{z}}^{k+1}=\underset{\mathit{z}}{\text{argmin}}{ℒ}_{\varphi }({\mathit{\beta }}^{k+1},\mathit{z};{\mathit{\gamma }}^k),$$

$${\mathit{\gamma }}^{k+1}={\mathit{\gamma }}^k+\theta \varphi ({\mathit{z}}^{k+1}+\mathit{X}{\mathit{\beta }}^{k+1}-\mathit{y}),$$

where $\theta$ is a tuning parameter controlling the step size. The effect of tuning parameter $\theta$ on the algorithm convergence has been discussed in literature (Fortin and Glowinski, 2000; Fazel et al., 2013), where the convergences are established when $\theta$ is constrained in $(0,(1+\sqrt{5})/2)$. In our numerical experiments, we set $\theta =1.618$ that is slightly less than $(1+\sqrt{5})/2$ for faster convergence. Gu et al. (2018) proposed an efficient algorithm (qradmm) to solve PQR based on the two block ADMM algorithm. While qradmm performs very well for moderate dimensions, we found it can still be out of memory with larger p in our numerical study. This motivates us to split the high dimensional variable to smaller blocks and speed up updates through parallelization.

We next propose a new three-block semi-proximal ADMM framework that capacitates a parallel update of $\mathit{\beta }$ to cope with the ultrahigh dimensionality. Major computational cost of two-block ADMM for solving (7) comes from the $\mathit{\beta }$ update, which takes up to O(np) operations and may impede an efficient execution of the algorithm with ultrahigh dimension p. This calls for feature-splitting algorithm for PQR in ultrahigh dimension. For a pre-specified G, let us partition X and $\mathit{\beta }$ as follows,

$$\mathit{X}=({\mathit{X}}_1,\cdots ,{\mathit{X}}_G),\mathit{\beta }={({\mathit{\beta }}_1^T,{\mathit{\beta }}_2^T,\cdots ,{\mathit{\beta }}_G^T)}^T,\mathit{X}\mathit{\beta }=\sum _{g=1}^G{\mathit{X}}_g{\mathit{\beta }}_g.$$

Then problem (7) can be rewritten as a three-block optimization problem

$$\begin{gathered} \underset{\mathit{\beta },\mathit{z},\mathit{\omega }}{\min }L(\mathit{z})+\sum _{g=1}^G\lambda {\Vert {\alpha }_g\circ {\beta }_g\Vert }_1, \\ \text{s.t.}{\mathit{X}}_1{\mathit{\beta }}_1+\mathit{z}+{\mathit{\omega }}_2+\cdots +{\mathit{\omega }}_G=\mathit{y},{\mathit{X}}_g{\mathit{\beta }}_g={\mathit{\omega }}_g,g=2,\cdots ,G. \end{gathered}$$ (9)

Intuitively, slack variables ${\mathit{\omega }}_g,g=2,\cdots ,G$ store information of each local update ${\mathit{\beta }}_g$. Each ${\mathit{\beta }}_g$ is updated independently and we view $\mathit{\beta }={({\mathit{\beta }}_1^T,{\mathit{\beta }}_2^T,\cdots ,{\mathit{\beta }}_G^T)}^T$ as a single variable block in the algorithm. Likewise, all ${\mathit{\omega }}_g$ together make up the third variable block. There may exist multiple ways to transform a problem into a form that ADMM can handle. For example, in formulation (9), the role of ${\mathit{X}}_1{\mathit{\beta }}_1$ is not special and ${\mathit{X}}_g{\mathit{\beta }}_g,g=1,\cdots ,G$ are exchangeable. In this paper, we use formulation (9) to illustrate the computational framework.

The augmented Lagrangian function for (9) is given by

$$\begin{gathered} {ℒ}_{\varphi }(\mathit{\beta },\mathit{z},\mathit{\omega };\mathit{\gamma })=\frac{1}{n}[\tau {\mathbf{1}}^T{(\mathit{z})}_{+}+(1-\tau ){\mathbf{1}}^T{(\mathit{z})}_{-}]+\lambda \sum _{g=1}^G{\Vert {\mathit{\alpha }}_g\circ {\mathit{\beta }}_g\Vert }_1 \\ +{\mathit{\gamma }}_1^T({\mathit{X}}_1{\mathit{\beta }}_1+\mathit{z}+{\mathit{\omega }}_2+\cdots +{\mathit{\omega }}_G-\mathit{y}) \\ +\frac{\varphi }{2}{\Vert {\mathit{X}}_1{\mathit{\beta }}_1+\mathit{z}+{\mathit{\omega }}_2+\cdots +{\mathit{\omega }}_G-\mathit{y}\Vert }_2^2 \\ +\sum _{g=2}^G{\mathit{\gamma }}_g^T({\mathit{X}}_g{\mathit{\beta }}_g-{\mathit{\omega }}_g)+\frac{\varphi }{2}\sum _{g=2}^G{\Vert {\mathit{X}}_g{\mathit{\beta }}_g-{\mathit{\omega }}_g\Vert }_2^2. \end{gathered}$$ (10)

As seen from (10), each ${\mathit{\beta }}_g$ is decoupled in the quadratic term, which allows a natural parallelization for $\mathit{\beta }$ updates. Two-block ADMM can be directly extended to solve (9), and the corresponding algorithm is referred to as Gauss-Seidel multi-block ADMM. At the kth iteration, it updates each variable with

$$\{\begin{array}{l}{\mathit{\beta }}^{k+1}=\text{argmin}{ℒ}_{\varphi }(\mathit{\beta },{\mathit{z}}^k,{\mathit{\omega }}^k;{\mathit{\gamma }}^k)\hfill \\ {\mathit{z}}^{k+1}=\text{argmin}{ℒ}_{\varphi }({\mathit{\beta }}^{k+1},\mathit{z},{\mathit{\omega }}^k;{\mathit{\gamma }}^k)\hfill \\ {\mathit{\omega }}^{k+1}=\text{argmin}{ℒ}_{\varphi }({\mathit{\beta }}^{k+1},{\mathit{z}}^{k+1},\mathit{\omega };{\mathit{\gamma }}^k)\hfill \\ {\mathit{\gamma }}_1^{k+1}={\mathit{\gamma }}_1^k+\theta \varphi ({\mathit{X}}_1{\mathit{\beta }}_1^{k+1}+{\mathit{z}}^{k+1}+\sum _{g=2}^G{\mathit{\omega }}_g^{k+1}-\mathit{y})\hfill \\ {\mathit{\gamma }}_g^{k+1}={\mathit{\gamma }}_g^k+\theta \varphi ({\mathit{X}}_g{\mathit{\beta }}_g^{k+1}-{\mathit{\omega }}_g^{k+1}),g=2,\cdots ,G.\hfill \end{array}$$ (11)

Procedure (11) may perform well in practice. However, its theoretical convergence has remained unclear until the work by Chen et al. (2016), in which the authors showed that Gauss-Seidel multi-block ADMM is not necessarily convergent. To address the convergence uncertainty, Sun et al. (2015) proposed a symmetric Gauss-Seidel based semi-proximal ADMM (sGS-sPADMM) for convex programming problems, which enjoys both theoretical convergence guarantee and numerical efficiency over the directly extended multi-block ADMM. This convergent semi-proximal ADMM has three separable blocks in the objective function with the third part being linear and updates $\mathit{\omega }$ twice to improve convergence, but the extra step may incur additional computational cost.

Inspired by Sun et al. (2015), we now propose the three-block ADMM algorithm for solving PQR with weighted ℓ₁ penalty using the following special iterative cycle $(\mathit{\beta }\to \mathit{\omega }\to \mathit{z}\to \mathit{\omega })$

$$\{\begin{array}{l}{\mathit{\beta }}^{k+1}=\text{argmin}{ℒ}_{\varphi }(\mathit{\beta },{\mathit{z}}^k,{\mathit{\omega }}^k;{\mathit{\gamma }}^k)+\frac{\varphi }{2}{\Vert \mathit{\beta }-{\mathit{\beta }}^k\Vert }_{{𝒯}_f}^2\\ {\mathit{\omega }}^{k+\frac{1}{2}}=\text{argmin}{ℒ}_{\varphi }({\mathit{\beta }}^{k+1},{\mathit{z}}^k,\mathit{\omega };{\mathit{\gamma }}^k)\\ {\mathit{z}}^{k+1}=\text{argmin}{ℒ}_{\varphi }({\mathit{\beta }}^{k+1},\mathit{z},{\mathit{\omega }}^{k+\frac{1}{2}};{\mathit{\gamma }}^k)+\frac{\varphi }{2}{\Vert \mathit{z}-{\mathit{z}}^k\Vert }_{{𝒯}_g}^2\\ {\mathit{\omega }}^{k+1}=\text{argmin}{ℒ}_{\varphi }({\mathit{\beta }}^{k+1},{\mathit{z}}^{k+1},\mathit{\omega };{\mathit{\gamma }}^k)\\ {\mathit{\gamma }}_1^{k+1}={\mathit{\gamma }}_1^k+\theta \varphi ({\mathit{X}}_1{\mathit{\beta }}_1^{k+1}+{\mathit{z}}^{k+1}+{\sum }_{i=2}^K{\mathit{\omega }}_i^{k+1}-\mathit{y})\\ {\mathit{\gamma }}_i^{k+1}={\mathit{\gamma }}_i^k+\theta \varphi ({\mathit{X}}_i{\mathit{\beta }}_i^{k+1}-{\mathit{\omega }}_i^{k+1}),i=2,\cdots ,K,\end{array}$$

where ${𝒯}_f$ and ${𝒯}_g$ are some positive semidefinite matrices.

Given the augmented Lagrangian function defined in (10),

$${\mathit{\beta }}^{k+1}=\text{argmin}{ℒ}_{\varphi }(\mathit{\beta },{\mathit{z}}^k,{\mathit{\omega }}^k;{\mathit{\gamma }}^k)$$

becomes

$$\begin{gathered} {\mathit{\beta }}_1^{k+1}=\underset{{\mathit{\beta }}_1\in {ℛ}^{p_1}}{\text{argmin}}\lambda {\Vert {\mathit{\alpha }}_1\circ {\mathit{\beta }}_1\Vert }_1+\frac{\varphi }{2}{\Vert {\mathit{X}}_1{\mathit{\beta }}_1+\sum _{g=2}^G{\mathit{\omega }}_g^k+{\mathit{z}}^k-\mathit{y}+\frac{{\mathit{\gamma }}_1^k}{\varphi }\Vert }_2^2, \\ {\mathit{\beta }}_g^{k+1}=\underset{{\mathit{\beta }}_g\in {ℛ}^{p_g}}{\text{argmin}}\lambda {\Vert {\mathit{\alpha }}_g\circ {\mathit{\beta }}_g\Vert }_1+\frac{\varphi }{2}{\Vert {\mathit{X}}_g{\mathit{\beta }}_g-{\mathit{\omega }}_g^k+\frac{{\mathit{\gamma }}_g^k}{\varphi }\Vert }_2^2,g=2,\cdots ,G. \end{gathered}$$ (12)

It can be seen that $\mathit{\beta }$ subproblems are a series of weighted ${\ell }_1$-penalized least squares problems. If p_g is too large, ${\mathit{X}}_g\in {ℛ}^{n\times p_g}$ may not be full column rank, and thus, the generated sequences may not be well-defined. This concern can be addressed with an additional general position condition (Koenker, 2017), which indicated the existence of the unique solution of QR in a rather general condition. Standard quadratic solvers can be applied to solve (12) efficiently. In our numerical studies, we use R solver ‘glmnet’ to compute $\mathit{\beta }$ through the coordinate descent (CD) algorithm.

${\mathit{\omega }}_g,g=2,\cdots ,G$, and z are updated in the following cycle:

$$\begin{gathered} {\mathit{\omega }}_g^{k+\frac{1}{2}}=\frac{1}{G}(\mathit{y}-{\mathit{z}}^k+G{\mathit{X}}_g{\mathit{\beta }}_g^{k+1}-\sum _{j=1}^G{\mathit{X}}_j{\mathit{\beta }}_j^{k+1}), \\ {\mathit{z}}^{k+1}={\left(\mathit{y}-{\mathit{X}}_1{\mathit{\beta }}_1^{k+1}-\sum _{g=2}^G{\mathit{\omega }}_g^{k+\frac{1}{2}}-\frac{{\mathit{\gamma }}_1^k}{\varphi }-\frac{\tau }{n\varphi }\right)}_{+} \\ -{\left(-\mathit{y}+{\mathit{X}}_1{\mathit{\beta }}_1^{k+1}+\sum _{g=2}^G{\mathit{\omega }}_g^{k+\frac{1}{2}}+\frac{{\mathit{\gamma }}_1^k}{\varphi }+\frac{\tau -1}{n\varphi }\right)}_{+}, \\ {\mathit{\omega }}_g^{k+1}=\frac{1}{G}(\mathit{y}-{\mathit{z}}^{k+1}+G{\mathit{X}}_g{\mathit{\beta }}_g^{k+1}-\sum _{j=1}^G{\mathit{X}}_j{\mathit{\beta }}_j^{k+1}), \end{gathered}$$ (13)

in which we perform an extra intermediate step to compute ${\mathit{\omega }}^{k+\frac{1}{2}}$ before computing ${\mathit{z}}^{k+1}$. As seen from (13), the extra cost for update ω is negligible. The derivations of updates are given in Appendix A.1.

Finally, we update ${\mathit{\gamma }}_1$ and ${\mathit{\gamma }}_g$ via gradient ascent,

$$\begin{gathered} {\mathit{\gamma }}_1^{k+1}={\mathit{\gamma }}_1^k+\theta \varphi ({\mathit{X}}_1{\mathit{\beta }}_1^{k+1}+{\mathit{z}}^{k+1}+\sum _{g=2}^G{\mathit{\omega }}_g^{k+1}-\mathit{y}), \\ {\mathit{\gamma }}_g^{k+1}={\mathit{\gamma }}_g^k+\theta \varphi ({\mathit{X}}_g{\mathit{\beta }}_g^{k+1}-{\mathit{\omega }}_g^{k+1}),g=2,\cdots ,G. \end{gathered}$$ (14)

We call this algorithm FS-QRADMM-CD, and summarize it in Algorithm 1. From our numerical studies, we observe that FS-QRADMM-CD has favorable practical performances.

Algorithm 1.

FS-QRADMM-CD for weighted ${\ell }_1$-penalized QR

Initialization: ${\mathit{\beta }}^0,{\mathit{\omega }}^0,{\mathit{z}}^0,{\gamma }^0$, and $\varphi >0,\theta >0$ are given.

while the stopping criterion is not satisfied, do

Compute ${\mathit{\beta }}^{k+1}$ by (12) using CD algorithm.

Compute ${\mathit{\omega }}^{k+\frac{1}{2}},{\mathit{z}}^{k+1}$ and ${\mathit{\omega }}^{k+1}$ by (13).

Update ${\mathit{\gamma }}^{k+1}$ by (14).

end while

Besides using coordinate descent algorithm to update $\mathit{\beta }$, we have another solution for $\mathit{\beta }$ update. To ensure that solutions from (12) are well-defined, we add G self-adjoint positive semidefinite matrices, denoted as ${𝒯}_g,g=1,\cdots ,G$, to (12). A general principle is that ${𝒯}_g$ should be as small as possible, while the optimization problems are still easy to compute. Here we add proximal terms $\frac{1}{2}{\Vert {\mathit{\beta }}_g-{\mathit{\beta }}_g^k\Vert }_{{𝒯}_g}^2,g=1,\cdots ,G$, to each of the $\mathit{\beta }$-subproblems, where the proximal operators ${𝒯}_g$ is positive definite. The positive definiteness of ${𝒯}_g$ makes $\left\{{\mathit{\beta }}^k\right\}$ automatically well-defined. In this paper, we take ${𝒯}_g={\eta }_g{\mathit{I}}_{p_g}-\varphi {\mathit{X}}_g^T{\mathit{X}}_g$ with ${\eta }_g>\varphi {\lambda }_{\max }({\mathit{X}}_g^T{\mathit{X}}_g)$. This essentially is a linearization step of the $\mathit{\beta }$ update, as it uses ${\eta }_g{\mathit{I}}_{p_g}$ to approximate the Hessian matrix ${\mathit{X}}_g^T{\mathit{X}}_g$. The modified minimization problem admits a closed-form solution, which can be carried out componentwisely,

$$\begin{gathered} {\mathit{\beta }}_1^{k+1}=\text{S}\left({\mathit{\beta }}_1^k-\frac{\varphi }{{\eta }_1}{\mathit{X}}_1^T({\mathit{X}}_1{\mathit{\beta }}_1^k+\sum _{g=2}^G{\mathit{\omega }}_g^k+{\mathit{z}}^k-\mathit{y}+\frac{{\gamma }_1^k}{\varphi }),\frac{{\mathit{\alpha }}_1\lambda }{{\eta }_1}\right), \\ {\mathit{\beta }}_g^{k+1}=\text{S}\left({\mathit{\beta }}_g^k-\frac{\varphi }{{\eta }_g}{\mathit{X}}_g^T({\mathit{X}}_g{\mathit{\beta }}_g^k-{\mathit{\omega }}_g^k+\frac{{\gamma }_g^k}{\varphi }),\frac{{\mathit{\alpha }}_g\lambda }{{\eta }_g}\right),g=2,\cdots ,G, \end{gathered}$$ (15)

where $\text{S}(x,t)=\text{sign}(x)(|x|-t)I(|x|>t)$ is the soft thresholding function.

Updates in (15) manifest one advantage of splitting feature space into lower dimensions. The $\mathit{\beta }$ update can be regarded as a one-step iteration of the proximal gradient. After feature-splitting, ${\eta }_g$’s are relatively small compared to the “un-splitted” $\eta$, as $\eta$ needs to be larger than $\varphi {\lambda }_{\max }({\mathit{X}}^T\mathit{X})$. Since η increases significantly with p for high dimensional data, the step size for the update (i.e., $\frac{1}{\eta }$) can be rather small and slow down the convergence of the algorithm. The update for $\mathit{\omega },\mathit{z}$ and $\mathit{\gamma }$ in this algorithm is exactly same as those in Algorithm 1. We use FS-QRADMM-prox to denote this algorithm and summarize it in Algorithm 2. Note that ${𝒯}_g\succ 0$ is also required in the proof of $\{{\mathit{\beta }}^k\}$ convergence.

Thus, we compute ${\mathit{\beta }}_2,\cdots ,{\mathit{\beta }}_G$ on separate processors/cores in the manner of parallel computing, and then aggregate the updated information to compute other variables.

We establish the linear rate of convergence for Algorithm 2 in Theorem 1 in which the proximal term is necessary for establishing the theory and whose proof is given in the Appendix A.

Theorem 1. For $\theta \in (0,(1+\sqrt{5})/2)$, the sequence $\left({\mathit{\beta }}^k,{\mathit{z}}^k,{\mathit{\omega }}^k,{\mathit{\gamma }}^k\right)$ generated by Algorithm 2 converges to a limit point $(\overline{\mathit{\beta }},\overline{\mathit{z}},\overline{\mathit{\omega }},\overline{\mathit{\gamma }})$, where $(\overline{\mathit{\beta }},\overline{\mathit{z}},\overline{\mathit{\omega }})$ is the primal optimal and $\overline{\mathit{\gamma }}$ is the dual optimal. Furthermore, there exists a constant μ ∈ (0, 1) such that ${\mathit{Dist}}^{k+1}\le \mu {\mathit{Dist}}^k$, where ${\mathit{Dist}}^k$ at the k-th iteration is defined as,

$$\begin{gathered} {\mathit{Dist}}^k={\Vert {\mathit{z}}^k-\overline{\mathit{z}}\Vert }_2^2+\frac{G-1}{G}{\Vert {\mathit{z}}^k-{\mathit{z}}^{k-1}\Vert }_2^2+\sum _{g=1}^G{\Vert {\mathit{X}}_g({\mathit{\beta }}_g^k-{\overline{\mathit{\beta }}}_g)-\frac{1}{G}(\mathit{X}{\mathit{\beta }}^k-\mathit{X}\overline{\mathit{\beta }})\Vert }_2^2 \\ +\frac{m_1}{G}{\Vert \sum _{g=1}^G{\mathit{X}}_g({\mathit{\beta }}_g^k-{\overline{\mathit{\beta }}}_g)+({\mathit{z}}^k-\overline{\mathit{z}})\Vert }_2^2+\sum _{g=1}^G{\Vert {\mathit{\beta }}_g^k-{\overline{\mathit{\beta }}}_g\Vert }_{{𝒯}_g}^2, \end{gathered}$$ (16)

where $m_1=1+d_1-d_1\theta -\left(1-d_1\right)\min \left\{\theta ,\frac{1}{\theta }\right\}$ and $d_1\in \left(0,\frac{1}{2}\right)$.

Algorithm 2.

FS-QRADMM-prox for weighted ${\ell }_1$-penalized QR

Initialization: ${\mathit{\beta }}^0,{\mathit{\omega }}^0,{\mathit{z}}^0,{\mathit{\gamma }}^0$, and $\varphi >0,\theta >0$ are given. ${𝒯}_g={\eta }_g{\mathit{I}}_{p_g}-\varphi {\mathit{X}}_g^T{\mathit{X}}_g$

with ${\eta }_g>\varphi {\lambda }_{\max }({\mathit{X}}_g^T{\mathit{X}}_g),g=1,\cdots ,G.$

while the stopping criterion is not satisfied, do

Compute ${\mathit{\beta }}^{k+1}$ by (15).

Compute ${\mathit{\omega }}^{k+\frac{1}{2}},{\mathit{z}}^{k+1}$ and ${\mathit{\omega }}^{k+1}$ by (13).

Update ${\mathit{\gamma }}^{k+1}$ by (14).

end while

Remark. The minimization problem for searching the solution of the penalized quantile regression can be written as a linear programming problem. The primal and dual problems are feasible. The minimizer of the dual problem equals the solution of the linear programming problem (primal problem) by strong duality. Thus both the optimal values of the primal and dual problems equal the optimal value of the penalized quantile regression problem.

The effect of G on the convergence is twofold. On the one hand, increasing G reduces the dimension of subproblems and the value of $\eta$, and thus it accelerates the computation of each sub-problem. On the other hand, increasing G leads to an increased number of sub-problems and may raise the value of $\mu$. This slows down the convergence both practically and theoretically. In our numerical experiments, it seems that choosing G from 5 to 10 works well for p ranging from thousands to tens of thousands.

2.3. PQR-Lasso and PQR-SCAD

In this paper, PQR-Lasso refers to the PQR in (2) with the ${\ell }_1$ penalty, $p_{\lambda }(|\beta |)=\lambda |\beta |$. Thus, the PQR-Lasso can be solved by Algorithms 1 and 2 directly with all weights ${\alpha }_j=1,j=1,\cdots ,p$ in (6). The resulting solutions from Algorithms 1 and 2 for the PQR-Lasso are denoted by FS-QRADMM-CD(Lasso) and FS-QRADMM-prox(Lasso) in Section 3, respectively.

Parallel to the PQR-Lasso, PQR-SCAD refers to the PQR in (2) with the SCAD penalty whose first-order derivative is defined in (3). Since the SCAD penalty is folded concave, the objective function of PQR-SCAD may have multiple local minimizers. To avoid this issue, we recommend (a) using the proposed algorithm to obtain the PQR-Lasso estimate ${\widehat{\mathit{\beta }}}_L={({\widehat{\beta }}_{L,1},\cdots ,{\widehat{\beta }}_{L,p})}^T$, and then (b) further to obtain the PQR with weighted ${\ell }_1$ penalty, in which the weight ${\alpha }_j$ is ${\lambda }^{-1}{p}_{\lambda }^{\prime }(|{\widehat{\beta }}_{L,j}|)$ with $p_{\lambda }^{\prime }(|\beta |)$ being the first-order derivative of the SCAD penalty. We refer to the resulting estimate as the two-step PQR-SCAD estimate. Note that both ${\ell }_1$ penalty and the SCAD-based weighted ${\ell }_1$ penalty are convex. The two-step SCAD estimate is well defined when L(y–Xβ) is strictly convex with respect to $\mathit{\beta }$. Denote FS-QRADMM-CD(TS-SCAD) and FS-QRADMM-prox(TS-SCAD) to be the resulting solutions of Algorithms 1 and 2 for the two-step PQR-SCAD. The corresponding algorithms of FS-QRADMM-CD and FS-QRADMM-prox for the two-step PQR-SCAD are presented in Algorithms 3 and 4 in Section A.3 in the Appendix.

The two-step PQR-SCAD shares the same spirit of one-step sparse maximum likelihood estimation proposed in Zou and Li (2008) for folded concave penalization problems. The second step in the two-step PQR-SCAD is to correct the bias inherited in ${\ell }_1$ penalty which is known to over-penalize large coefficients and introduce bias to the resulting model. As shown in Corollary 8 in Fan et al. (2014), the two-step PQR-SCAD can find the oracle estimator among multiple local minimums with overwhelming probability, under certain regularity conditions. This provides theoretical justification for the two-step SCAD. In other words, the resulting solutions of Algorithms 3 and 4 for the two-step PQR-SCAD enjoy strong oracle property in the terminology of Fan et al. (2014).

The two-step PQR-SCAD procedure can be extended to two-step PQR with a general folded concave penalty characterized by the following conditions: (a) p_λ(t) is nondecreasing and concave for $t\in [0,\infty )$ with p_λ (0) = 0; (b) p_λ(t) is differentiable in (0, ∞); (c) for some positive constants a₁ and $a_2,p_{\lambda }^{\prime }(t)\ge a_1\lambda$ for $t\in \left(0,a_2\lambda \right]$, and (d) $p_{\lambda }^{\prime }(t)=0$ for $t\in [a\lambda ,\infty )$ with a > 1. As shown in Fan et al. (2014), the two-step PQR with a general folded concave penalty also enjoys the strong oracle property under certain regularity conditions.

It is desirable to have a data-driven method to select the regularization parameters in PQR-Lasso and PQR-SCAD. In our numerical study, we set the same penalty and tuning parameter for all coefficients, and λ is chosen by HBIC criterion proposed in Lee et al. (2014),

$$\text{HBIC}(\lambda )=\log \{\sum _{i=1}^n{\rho }_{\tau }(y_i-{\mathit{x}}_i^T\mathit{\beta })\}+|𝒜|\frac{\log (\log n)\log (p)}{n},$$ (17)

where $|𝒜|$ is the cardinality of active set. We select the λ that minimizes HBIC.

Wang et al. (2013) recommends using different $\lambda$’s in the first step and the second step in the penalized least squares setting to ensure the resulting Lasso estimate satisfying a certain rate of convergence. Denote ${\lambda }_1$ and $\lambda$ to be regularization parameters used in the first and second step, respectively. Following the recommendation in Wang et al. (2013), we choose ${\lambda }_1=\upsilon \lambda$, where $\upsilon >0$ and tends to 0 as $n\to \infty$. We set υ = λ suggested by Wang et al. (2013) in our numerical studies in Section 3.

3. Numerical Studies

In this section, we assess the performance of the proposed algorithms via simulation studies and illustrate the application of the newly proposed procedure via an empirical analysis. For all ADMM-based methods, we implement the warm-start technique introduced in Friedman et al. (2007) and Friedman et al. (2010), which uses the solution from the previous $\lambda$ to initialize computation at the current $\lambda$. The way of splitting the features has no influence on the convergence property of the algorithm. We equally distribute the features into K groups without adjusting the order in our numerical studies. The stopping criterion of ADMM-based algorithms is provided in the Appendix.

3.1. Simulation study

In this simulation, we compare the performance of Algorithms 1 and 2 with R packages rqPen (Sherwood and Maidman, 2017), qradmm (Gu et al., 2018), hqReg (Yi and Huang, 2017) and Conquer (Tan et al., 2022). Since qradmm package is boosted by FORTRAN, we re-implement its core algorithm, i.e., a two-block proximal ADMM, using R code for a relatively fair comparison. The R package rqPen implements an iterative coordinate descent algorithm (QICD) proposed in Peng and Wang (2015) to solve sparse quantile regression. QICD applies a convex majorization function on the concave penalty term, and solves the majorized objective function by coordinate descent. The R package qradmm implements a two-block proximal ADMM for PQR with ${\ell }_1$ penalty proposed in Gu et al. (2018). We use the R packages hqreg and conquer to implement the methods proposed by Yi and Huang (2017) and Tan et al. (2022), respectively. The regularization parameter λ in all algorithms to be compared is selected by the HBIC criterion defined in (17).

We take the simulation setting similar to that of Peng and Wang (2015). We generate $\tilde{\mathit{z}}={\left(Z_1,Z_2,\cdots ,Z_p\right)}^T$ from ${\mathit{N}}_p(\mathbf{0},\mathbf{\text{Σ}})$, where $\mathbf{\text{Σ}}=\left({\sigma }_{ij}\right)$ with ${\sigma }_{ij}={0.5}^{|i-j|}$. Then set $X_1=\mathbf{\text{Φ}}\left(Z_1\right)$ and $X_j=Z_j$ for $j=2,\cdots ,p$, where $\mathbf{\text{Φ}}(\cdot )$ is the cumulative distribution function of N(0, 1). The response variable Y is generated from the following heteroscedastic regression model,

$$Y=X_6+X_{100}+X_{500}+X_{1000}+0.7X_1\epsilon ,$$ (18)

where $\epsilon \sim N(0,1)$. We consider three different quantiles $\tau =0.3,0.5$ and 0.7. Note that X₁ does not affect the center of the conditional distribution Y given x, but affects the conditional distribution when $\tau =0.3$ or 0.7. In our simulation, we set n = 400 and p = 1000 and 50000. For each case, we conduct 500 replications.

The following criteria are used to compare the performance of different algorithms.

1. Average absolute error: the average and standard deviation of ${\parallel \widehat{\mathit{\beta }}-\mathit{\beta }\parallel }_1={\sum }_{j=1}^p\left|{\widehat{\beta }}_j-{\beta }_j\right|$ over 500 replications.

2. Size: the average number of nonzero ${\widehat{\beta }}_j$’s over 500 replications.

3. P₁: the proportion of models that select all active features except for X₁ over 500 replications

4. P₂: the proportion of models that select X₁ over 500 replications.

The proportion P₂ is expected to be close to 0 when $\tau =0.5$, and be close to 1 when $\tau =0.3$ and 0.7.

The simulation results over 500 replications are summarized in Tables 1 and 2. Compared to the PQR-Lasso, two-step PQR-SCAD produces models with significantly smaller absolute error and better selection accuracy in general. FS-QRADMM-CD(TS-SCAD) and FS-QRADMM-prox (TS-SCAD) have the best performances with respect to estimation and variable selection accuracy. When p = 1000, three ADMM-based methods perform comparably well and outperform rqPen, hqReg and Conquer by a significant margin. rqPen, hqReg and Conquer obtain relatively larger estimation errors and is more likely to miss X₁ when $\tau =0.3$, and 0.7. The current version of rqPen runs out of memory when solving two-step PQR-SCAD, as noted in the table. Nonetheless, when p = 50000, both Qradmm and rqPen fail due to their demanding memory usage. In fact, we notice that the efficiency of Qradmm deteriorates sharply when p increases. hqReg and Conquer are able to finish the job when p = 50000, but the proposed methods still outperform hqReg and Conquer.

Table 1:

Comparison of algorithms for PQR when p = 1000 and n = 400.

n = 400, p = 1000	$\tau$	${\Vert \widehat{\mathit{\beta }}-\mathit{\beta }\Vert }_1$	P1	P2	Size
FS-QRADMM-CD (Lasso)	0.3	0.295 (0.003)	100%	100%	5.56 (0.03)
	0.5	0.210 (0.003)	100%	5.4%	4.36 (0.03)
	0.7	0.281 (0.003)	100%	100%	5.56 (0.03)
FS-QRADMM-prox (Lasso)	0.3	0.295 (0.003)	100%	100%	5.62 (0.03)
	0.5	0.198 (0.003)	100%	4.6%	4.34 (0.02)
	0.7	0.301 (0.003)	100%	100%	5.56 (0.03)
qradmm(Lasso)	0.3	0.310 (0.003)	100%	100%	5.68 (0.03)
	0.5	0.230 (0.003)	100%	9%	5.32 (0.06)
	0.7	0.327 (0.005)	100%	100%	6.73 (0.08)
rqPen(Lasso)	0.3	0.598 (0.004)	100%	61.2%	5.10 (0.04)
	0.5	0.267 (0.003)	100%	0%	4.23 (0.02)
	0.7	0.601 (0.004)	100%	56.6%	5.04 (0.04)
hqReg(Lasso)	0.3	0.593 (0.006)	100%	50%	4.95 (0.04)
	0.5	0.235 (0.003)	100%	0%	4.31 (0.03)
	0.7	0.589 (0.006)	100%	51.6%	4.97 (0.04)
Conquer(Lasso)	0.3	0.590 (0.005)	100%	45%	4.73 (0.03)
	0.5	0.231 (0.002)	100%	0%	4.27 (0.02)
	0.7	0.586 (0.005)	100%	45%	4.72 (0.03)
FS-QRADMM-CD (TS-SCAD)	0.3	0.119 (0.002)	100%	100%	5.00 (0.00)
	0.5	0.035 (0.001)	100%	0.2%	4.00 (0.00)
	0.7	0.125 (0.002)	100%	100%	5.00 (0.00)
FS-QRADMM-prox (TS-SCAD)	0.3	0.115 (0.002)	100%	100%	5.00 (0.00)
	0.5	0.040 (0.001)	100%	0.2%	4.00 (0.00)
	0.7	0.123 (0.001)	100%	100%	5.00 (0.00)
qradmm (TS-SCAD)	0.3	0.122 (0.002)	100%	100%	5.00 (0.00)
	0.5	0.038 (0.001)	100%	0.4%	4.00 (0.00)
	0.7	0.129 (0.002)	100%	100%	5.00 (0.00)
rqPen (TS-SCAD)	The algorithm runs out of memory for τ = 0.3, 0.5, 0.7
Conquer(SCAD)	0.3	0.339 (0.004)	100%	63.4%	4.67 (0.02)
	0.5	0.049 (0.001)	100%	0%	4.07 (0.01)
	0.7	0.350 (0.004)	100%	57%	4.60 (0.02)

Table 2:

Comparison of algorithms for PQR when p = 50000 and n = 400.

n = 400, p = 50000	$\tau$	${\Vert \widehat{\mathit{\beta }}-\mathit{\beta }\Vert }_1$	P1	P2	Size
FS-QRADMM-CD (Lasso)	0.3	0.320 (0.003)	100%	98.2%	5.34 (0.02)
	0.5	0.250 (0.003)	100%	2%	4.25 (0.03)
	0.7	0.349 (0.003)	100%	100%	5.15 (0.03)
FS-QRADMM-prox (Lasso)	0.3	0.326 (0.003)	100%	92.4%	4.93 (0.01)
	0.5	0.121 (0.001)	100%	0%	4.01 (0.00)
	0.7	0.394 (0.002)	100%	95.6%	5.01 (0.11)
qradmm(Lasso)	The algorithm runs out of memory for τ = 0.3, 0.5, 0.7
rqPen(Lasso)	The algorithm runs out of memory for τ = 0.3, 0.5, 0.7
hqReg(Lasso)	0.3	0.812 (0.004)	100%	2.2%	4.23 (0.02)
	0.5	0.365 (0.003)	100%	0%	4.20 (0.02)
	0.7	0.808 (0.004)	100%	4.4%	4.26 (0.02)
Conquer(Lasso)	0.3	0.717 (0.003)	100%	18%	5.88 (0.07)
	0.5	0.303 (0.002)	100%	0%	8.63 (0.11)
	0.7	0.705 (0.003)	100%	26.8%	6.13 (0.07)
FS-QRADMM-CD (TS-SCAD)	0.3	0.180 (0.003)	100%	98.8%	4.99 (0.00)
	0.5	0.047 (0.001)	100%	0%	4.00 (0.00)
	0.7	0.172 (0.003)	100%	99.6%	5.00 (0.03)
FS-QRADMM-prox (TS-SCAD)	0.3	0.158 (0.002)	100%	100%	5.00 (0.00)
	0.5	0.069 (0.005)	100%	2.2%	7.31 (0.47)
	0.7	0.244 (0.007)	100%	99.2%	6.64 (0.14)
qradmm (TS-SCAD)	The algorithm runs out of memory for τ = 0.3, 0.5, 0.7
rqPen (TS-SCAD)	The algorithm runs out of memory for τ = 0.3, 0.5, 0.7
Conquer(SCAD)	0.3	0.396 (0.002)	100%	48.4%	5.04 (0.04)
	0.5	0.058 (0.001)	100%	0%	5.78 (0.07)
	0.7	0.390 (0.003)	100%	49.6%	5.12 (0.05)

Figure 1 plots the curves of ${\Vert \widehat{\mathit{\beta }}-{\mathit{\beta }}^{\ast }\Vert }_1$ with respect to the iteration steps averaged over 500 replications when n = 400, p = 1000, and $\tau =\left(0.3,0.5,0.7\right)$. We can see that Algorithms 1 and 2 converge to true $\mathit{\beta }$ within approximately 20 iterations.

Figure 1:

Convergence curves of ${\Vert \widehat{\mathit{\beta }}-{\mathit{\beta }}^{\ast }\Vert }_1$ of FS-QRADMM-CD(Lasso) (left panel) and FS-QRADMM-prox(Lasso) (right panel) over 500 replications.

We next examine the performance of the proposed algorithms when the sample size is large. To this end, we conduct a simulation with n = 30000 and p = 1000. The simulation results are summarized in Table 3, from which it can be seen that the proposed two algorithm and the ADMM for quantile regression have the same performance and perform better than the conquer algorithm.

Table 3:

Performance of proposed algorithms for PQR when p = 1000 and n = 30000.

n = 30000, p = 1000	$\tau$	${\Vert \widehat{\mathit{\beta }}-\mathit{\beta }\Vert }_1$	P1	P2	Size
FS-QRADMM-CD (Lasso)	0.3	0.031 (0.0004)	100%	100%	5.06 (0.003)
	0.5	0.020 (0.0004)	100%	0.4%	4.04 (0.003)
	0.7	0.029 (0.0004)	100%	100%	5.06 (0.003)
FS-QRADMM-prox (Lasso)	0.3	0.029 (0.0003)	100%	100%	5.05 (0.003)
	0.5	0.020 (0.0003)	100%	0.5%	4.03 (0.002)
	0.7	0.029 (0.0003)	100%	100%	5.05 (0.003)
qradmm(Lasso)	0.3	0.030 (0.0004)	100%	100%	5.08 (0.003)
	0.5	0.023 (0.0003)	100%	0.6%	4.05 (0.004)
	0.7	0.029 (0.0004)	100%	100%	5.04 (0.004)
rqPen(Lasso)	The algorithm runs out of memory for τ = 0.3, 0.5, 0.7
hqReg(Lasso)	0.3	0.040 (0.0008)	100%	100%	6.06 (0.09)
	0.5	0.020 (0.0002)	100%	1%	4.60 (0.07)
	0.7	0.040 (0.0008)	100%	51.6%	5.90 (0.09)
Conquer(Lasso)	0.3	0.066 (0.0009)	100%	100%	5.23 (0.04)
	0.5	0.020 (0.0003)	100%	0%	4.11 (0.04)
	0.7	0.065 (0.0009)	100%	100%	5.25 (0.05)
FS-QRADMM-CD (TS-SCAD)	0.3	0.012 (0.0005)	100%	100%	5.00 (0.00)
	0.5	0.004 (0.0002)	100%	0.2%	4.00 (0.00)
	0.7	0.013 (0.0004)	100%	100%	5.00 (0.00)
FS-QRADMM-prox (TS-SCAD)	0.3	0.011 (0.0003)	100%	100%	5.00 (0.00)
	0.5	0.004 (0.0002)	100%	0%	4.00 (0.00)
	0.7	0.012 (0.0003)	100%	100%	5.00 (0.00)
qradmm (TS-SCAD) (TS-SCAD)	0.3	0.012 (0.0004)	100%	100%	5.00 (0.00)
	0.5	0.004 (0.0003)	100%	0%	4.00 (0.00)
	0.7	0.011 (0.0005)	100%	100%	5.00 (0.00)
rqPen (TS-SCAD)	The algorithm runs out of memory for τ = 0.3, 0.5, 0.7
Conquer(SCAD)	0.3	0.028 (0.0008)	100%	100%	5.15 (0.039)
	0.5	0.005 (0.0002)	100%	1%	4.45 (0.061)
	0.7	0.029 (0.0008)	100%	100%	5.14 (0.037)

3.2. A real data example

QR model is widely adopted in the analysis of consumer markets due to its robustness against outliers. In this section, we apply the proposed algorithm for an empirical analysis of a supermarket data set studied in Wang (2009) and compare it with other existing algorithms. This data set contains the daily number of customers and the daily sale volumes of 6398 products from a supermarket in China over 464 days. Following Wang (2009), we set the response to be the daily number of customers, and predictors to be the daily sale volumes of products. Since the sample size n = 464 is much less than the dimension p = 6398, it is reasonable to assume that only a small proportion of predictors have significant effects on the response. The distribution of the number of customers is highly skewed. This motivates us to consider PQR with the proposed algorithm in this example. We standardize the response and the predictors for our analysis.

We randomly split observations into training and testing datasets of sizes 300 and 164, respectively, and fit PQR-Lasso and two-step QR-SCAD on the training data with $\tau =0.3,0.5$ and 0.7. The regularization parameter $\lambda$ is chosen by HBIC criterion. We report the averaged predictive error and its standard deviation on the testing data over 100 replications in Table 4. The predictive error is measured by the loss function $\frac{1}{n}{\sum }_{i=1}^n{\rho }_{\tau }\left(y_i-{\widehat{y}}_i\right)$. We also report the average model sizes and its corresponding standard deviation to evaluate the interpretability of models selected from different methods. For PQR-Lasso, we observe that ADMM-based algorithms have similar performance with that of rqPen and hqReg in terms of prediction error. The average values and standard deviations of the loss function are very close among those methods. In general, all methods perform the best when $\tau =0.5$. The proposed method selects fewer products than qradmm, rqPen, hqReg do in most scenarios, which indicates a better model interpretability.

Table 4:

Performances of ADMM and lpSolve of sparse quantile regression on the Chinese Supermarket Data.

	$\tau$	$\frac{1}{n}{\sum }_{i=1}^n{\rho }_{\tau }\left(y_i-{\widehat{y}}_i\right)$	Size
FS-QRADMM-CD (Lasso)	0.3	0.118 (0.001)	97.35 (0.62)
	0.5	0.116 (0.001)	100.56 (0.53)
	0.7	0.127 (0.001)	97.37 (0.71)
FS-QRADMM-prox (Lasso)	0.3	0.117 (0.001)	103.23 (1.13)
	0.5	0.113 (0.001)	118.61 (0.93)
	0.7	0.127 (0.001)	96.02 (1.06)
qradmm(Lasso)	0.3	0.115 (0.000)	119.01 (0.49)
	0.5	0.116 (0.001)	121.26 (0.37)
	0.7	0.130 (0.000)	127.35 (0.58)
rqPen(Lasso)	0.3	0.117 (0.001)	113.62 (0.73)
	0.5	0.115 (0.001)	117.17 (0.56)
	0.7	0.128 (0.001)	120.11 (0.62)
hqReg(Lasso)	0.3	0.117 (0.001)	49.31 (0.46)
	0.5	0.116 (0.001)	90.85 (0.65)
	0.7	0.127 (0.001)	43.42 (0.42)
Conquer(Lasso)	0.3	0.118 (0.001)	80.8 (1.47)
	0.5	0.114 (0.001)	42.9 (0.58)
	0.7	0.125 (0.001)	39.6 (0.49)
FS-QRADMM-CD (TS-SCAD)	0.3	0.112 (0.000)	63.86 (0.39)
	0.5	0.111 (0.001)	69.77 (0.47)
	0.7	0.116 (0.001)	72.71 (0.61)
FS-QRADMM-prox (TS-SCAD)	0.3	0.116 (0.000)	97.03 (0.96)
	0.5	0.110 (0.000)	100.33 (1.02)
	0.7	0.113 (0.000)	95.66 (0.79)
qradmm (TS-SCAD)	0.3	0.113 (0.001)	469.88 (2.56)
	0.5	0.114 (0.001)	477.72 (2.33)
	0.7	0.120 (0.001)	521.33 (1.99)
rqPen(TS-SCAD)	The algorithm runs out of memory for the three τs
Conquer(SCAD)	0.3	0.115 (0.001)	65.7 (1.95)
	0.5	0.113 (0.001)	65.7(0.82)
	0.7	0.120 (0.001)	36.0 (0.48)

Similar results are also observed with two-step the PQR-SCAD. However, the rqPen for the two-step SCAD fails in this example due to the limitation of computing memory. The proposed algorithms have similar prediction error to that of qradmm, but the model sizes are much smaller. When $\tau =0.7$, the proposed methods outperform qradmm, with fewer products included in the QR model. Conquer with SCAD penalty has similar performance to the proposed method under this scenario. We can also notice that PQR-SCAD gives better loss than PQR-Lasso does, and the two-step PQR-SCAD procedures select fewer products when the proposed algorithms are implemented.

4. Conclusion

QR model is a powerful data analytic tool in econometrics. To promote the application of QR in the high/ultrahigh dimension, in this paper, we propose efficient and parallelizable algorithms for PQR based on a three-block ADMM algorithm with feature-splitting, and further establish the convergence of the proposed algorithms. Due to the nature of feature-splitting algorithm, the proposed algorithms can be used to minimize the objective function of PQR in ultrahigh dimension. Our numerical study implies that the proposed algorithms outperform existing ones for PQR. To illustrate the performance of proposed methods, we conduct a comprehensive simulation study. The numerical experiments suggest that the proposed method is stable when the dimension of the data is huge while existing algorithms run out of memory and fail to accomplish the tasks. The proposed algorithms may be extended to other statistical models such as supporting vector machine which has similar loss function to the loss function of QR. This is an interesting topic for future research.

Appendix: Technical Details and Proofs

In this appendix, we first provide details of how to update each variable in Algorithm 2, and then provide technical proofs of Theorem 1.

A.1. Sub-problems in Algorithm 2

In this subsection, we derive the updates for $\mathit{\beta },\mathit{z}$ and $\mathit{\omega }$ in Algorithm 2. For ease of notation, define a set of functions f, g, h.

$$f(\mathit{\beta })=n\lambda \sum _{g=1}^G{\Vert {\mathit{\alpha }}_g\circ {\mathit{\beta }}_g\Vert }_1,h(\mathit{\omega })=0,g(\mathit{z})=\tau {\mathbf{1}}^T{(\mathit{z})}_{+}+(1-\tau )1^T{(\mathit{z})}_{-},$$ (A.1)

Thus, f, g, h are closed proper convex functions. Further define matrices F, G, H

$$\begin{gathered} \mathit{F}=Diag\left({\mathit{X}}_1,{\mathit{X}}_2,\cdots ,{\mathit{X}}_G\right),\mathit{G}={\left(I_n,0,\cdots ,0\right)}^T, \\ \mathit{H}=\left(\begin{array}{cccc}{I}_n& I_n& \cdots & I_n\\ -I_n& 0& \cdots & 0\\ ⋮& \ddots & & ⋮\\ 0& \cdots & -I_n& 0\\ 0& 0& \cdots & -I_n\end{array}\right). \end{gathered}$$ (A.2)

Then Problem (9) can be expressed as a general three-block constrained optimization problem,

$$\underset{\mathit{\beta },\mathit{z},\mathit{\omega }}{\min }\{f(\mathit{\beta })+g(\mathit{z})+h(\mathit{\omega })\mid \mathit{F}\mathit{\beta }+\mathit{G}\mathit{z}+\mathit{H}\mathit{\omega }=\mathit{c}\},$$ (A.3)

where by definitions of F, G and H,

$$\mathit{F}\mathit{\beta }=\left(\begin{array}{c}{\mathit{X}}_1{\mathit{\beta }}_1\\ {\mathit{X}}_2{\mathit{\beta }}_2\\ ⋮\\ {\mathit{X}}_G{\mathit{\beta }}_G\end{array}\right),\mathit{G}\mathit{z}=\left(\begin{array}{c}\mathit{z}\\ 0\\ ⋮\\ 0\end{array}\right),\mathit{H}\mathit{\omega }=\left(\begin{array}{c}{\mathit{\omega }}_2+\cdots +{\mathit{\omega }}_G\\ -{\mathit{\omega }}_2\\ ⋮\\ -{\mathit{\omega }}_G\end{array}\right).$$ (A.4)

As in sGS-sPADMM proposed by Sun et al. (2015), we update the three-block variables using a special cycle,

$$\{\begin{array}{l}{\mathit{\beta }}^{k+1}=\text{argmin}{ℒ}_{\varphi }\left(\mathit{\beta },{\mathit{z}}^k,{\mathit{\omega }}^k;{\mathit{\gamma }}^k\right)+\left(\frac{\varphi }{2}{\Vert \mathit{\beta }-{\mathit{\beta }}^k\Vert }_{{𝒯}_f}^2\right),\\ {\mathit{\omega }}^{k+\frac{1}{2}}={\left({\mathit{H}}^T\mathit{H}\right)}^{-1}\mathit{H}\left(\mathit{c}-\mathit{F}{\mathit{\beta }}^{k+1}-\mathit{G}{\mathit{z}}^k\right),\\ {\mathit{z}}^{k+1}=\text{argmin}{ℒ}_{\varphi }\left({\mathit{\beta }}^{k+1},\mathit{z},{\mathit{\omega }}^{k+\frac{1}{2}};{\mathit{\gamma }}^k\right)+\left(\frac{\varphi }{2}{\Vert \mathit{z}-{\mathit{z}}^k\Vert }_{{𝒯}_h}^2\right),\\ {\mathit{\omega }}^{k+1}={\left({\mathit{H}}^T\mathit{H}\right)}^{-1}\mathit{H}\left(\mathit{c}-\mathit{F}{\mathit{\beta }}^{k+1}-\mathit{G}{\mathit{z}}^{k+1}\right),\\ {\mathit{\gamma }}^{k+1}={\mathit{\gamma }}^k+\theta \varphi \left(\mathit{F}{\mathit{\beta }}^{k+1}+\mathit{G}{\mathit{z}}^{k+1}+\mathit{H}{\mathit{\omega }}^{k+1}-\mathit{c}\right),\end{array}$$ (A.5)

where ${𝒯}_f$ and ${𝒯}_h$ are optionally added self-adjoint positive semidefinite operators. To update $\mathit{\omega }$, we need to compute ${({\mathit{H}}^T\mathit{H})}^{-1}$. Since

$${\mathit{H}}^T\mathit{H}=\left(\begin{array}{ccc}{I}_n& & \\ & \ddots & \\ & & I_n\end{array}\right)+\left(\begin{array}{c}{I}_n\\ ⋮\\ I_n\end{array}\right){\left(\begin{array}{c}{I}_n\\ ⋮\\ I_n\end{array}\right)}^T,$$

we apply the Sherman-Morrison-Woodebury formula to compute ${({\mathit{H}}^T\mathit{H})}^{-1}$ and it follows that

$$\begin{gathered} {\mathit{\omega }}_i^{k+\frac{1}{2}}={({\mathit{H}}^T\mathit{H})}^{-1}{\mathit{H}}^T(\mathit{c}-\mathit{F}{\mathit{\beta }}^{k+1}-\mathit{G}{\mathit{z}}^k) \\ =\frac{1}{G}(\mathit{y}-{\mathit{z}}^k+G{\mathit{X}}_i{\mathit{\beta }}_i^{k+1}-\sum _{j=1}^G{\mathit{X}}_j{\mathit{\beta }}_j^{k+1}),i=2,\cdots ,G, \end{gathered}$$ (A.6)

In the z-subproblem, we set ${𝒯}_h=0$ and then we have

$$\begin{gathered} {\mathit{z}}^{k+1}=\underset{\mathit{z}}{\text{argmin}}{ℒ}_{\varphi }({\mathit{\beta }}^{k+1},\mathit{z},{\mathit{\omega }}^{k+\frac{1}{2}};{\mathit{\gamma }}^k) \\ =\underset{\mathit{z}}{\text{argmin}}\{\frac{1}{n}\sum _{i=1}^n{\rho }_{\tau }({\mathit{z}}_i)+{\mathit{\gamma }}_1^T({\mathit{X}}_1{\mathit{\beta }}_1^{k+1}+\mathit{z}+\sum _{i=2}^G{\mathit{\omega }}_i^{k+\frac{1}{2}}-\mathit{y}) \\ +\frac{\varphi }{2}{\Vert {\mathit{X}}_1{\mathit{\beta }}_1^{k+1}+\mathit{z}+\sum _{i=2}^G{\mathit{\omega }}_i^{k+\frac{1}{2}}-\mathit{y}\Vert }_2^2\} \\ =\underset{\mathit{z}}{\text{argmin}}\frac{1}{n}\sum _{i=1}^n{\rho }_{\tau }({\mathit{z}}_i)+\frac{\varphi }{2}{\Vert {\mathit{X}}_1{\mathit{\beta }}_1^{k+1}+\mathit{z}+\sum _{i=2}^G{\mathit{\omega }}_i^{k+\frac{1}{2}}-\mathit{y}+\frac{{\mathit{\gamma }}_1^k}{\varphi }\Vert }_2^2. \end{gathered}$$ (A.7)

The closed-form solution of the z-subproblem can be easily derived as

$$\begin{gathered} {\mathit{z}}^{k+1}=\max \left(\mathit{y}-{\mathit{X}}_1{\mathit{\beta }}_1^{k+1}-\sum _{i=2}^G{\mathit{\omega }}_i^{k+\frac{1}{2}}-\frac{{\mathit{\gamma }}_1^k}{\varphi }-\frac{\tau }{n\varphi },0\right) \\ -\max \left(-(\mathit{y}-{\mathit{X}}_1{\mathit{\beta }}_1^{k+1}-\sum _{i=2}^G{\mathit{\omega }}_i^{k+\frac{1}{2}}-\frac{{\mathit{\gamma }}_1^k}{\varphi }-\frac{\tau }{n\varphi }+\frac{1}{n\varphi }),0\right). \end{gathered}$$ (A.8)

A.2. Proof of Theorem 1

We first show Lemmas A.1, A.2 and A.3, which are used in the proof of Theorem 1. From (A.2), we have Fact 1 below.

Fact 1. ${\mathit{H}}^T\mathit{H}$ is positive definite.

Assumptions 1 below is imposed to obtain theoretical guarantee on feasibility and convergence of the sequence $({\mathit{\beta }}^k,{\mathit{z}}^k,{\mathit{\omega }}^k,{\mathit{\gamma }}^k)$.

Assumption 1. There exists $(\widehat{\mathit{\beta }},\widehat{\mathit{z}},\widehat{\mathit{\omega }})\in {ℛ}^{p_1}\times {ℛ}^{p_2}\times {ℛ}^{p_3}$, such that $\mathit{F}\widehat{\mathit{\beta }}+\mathit{G}\widehat{\mathit{z}}+\mathit{H}\widehat{\mathit{\omega }}=\mathit{c}$.

For algorithm (A.5), the projection matrix $𝒫=\mathit{H}{({\mathit{H}}^T\mathit{H})}^{-1}{\mathit{H}}^T$ plays an important role in the convergence analysis. Let $𝒬=I-𝒫$. Since $\mathit{\omega }$ can be expressed as $\mathit{\omega }(\mathit{\beta },\mathit{z})={({\mathit{H}}^T\mathit{H})}^{-1}{\mathit{H}}^T(\mathit{c}-\mathit{F}\mathit{\beta }-\mathit{G}\mathit{z})$, it follows that $\mathit{H}\mathit{\omega }=𝒫(\mathit{c}-\mathit{F}\mathit{\beta }-\mathit{G}\mathit{z})$. Given that $h(\mathit{\omega })=\mathbf{0}$ in our case, we can now rewrite (A.3) as

$$\underset{\mathit{\beta },\mathit{z}}{\min }\{f(\mathit{\beta })+g(\mathit{z})\mid 𝒬(\mathit{F}\mathit{\beta }+\mathit{G}\mathit{z}-\mathit{c})=0\}.$$ (A.9)

Stopping Criterion. In the implementation of Algorithm 2, We use the same stopping criterion as the one introduced in Boyd et al. (2011). The primal and dual residuals are often used in characterizing the convergence stage. Define ${\mathit{r}}^{k+1}={\left({\Vert {\mathit{X}}_1{\mathit{\beta }}_1^{k+1}+{\mathit{z}}^{k+1}+{\mathit{\omega }}_2^{k+1}+\cdots +{\mathit{\omega }}_G^{k+1}-\mathit{y}\Vert }_2^2+{\Vert {\sum }_{g=2}^G\left({\mathit{X}}_g{\mathit{\beta }}_g^{k+1}-{\mathit{\omega }}_g^{k+1}\right)\Vert }_2^2\right)}^{0.5}$ as the primal residual and ${\mathit{s}}^{k+1}={\Vert \varphi /G{\left({\mathit{X}}_1^T,\cdots ,{\mathit{X}}_G^T\right)}^T\left({\mathit{z}}^{k+1}-{\mathit{z}}^k\right)\Vert }_2$ as the dual residual at the (k + 1)th iteration. The termination criterion is

$${\Vert {\mathit{r}}^k\Vert }_2\le {ϵ}^{\text{pri}}\text{and}{\Vert {\mathit{s}}^k\Vert }_2\le {ϵ}^{\text{dual}},$$ (A.10)

where ${ϵ}^{\text{pri}}>0$ and ${ϵ}^{\text{dual}}>0$ are feasibility tolerances chosen as ${ϵ}^{\text{pri}}=\sqrt{n}{ϵ}^{\text{abs}}+\frac{{ϵ}^{\text{rel}}}{\sqrt{G}}\max ({\Vert \mathit{X}{\mathit{\beta }}^k\Vert }_2,{\Vert {\mathit{z}}^k\Vert }_2,{\Vert \mathit{c}\Vert }_2)$, and ${ϵ}^{\text{dual}}=\sqrt{p}{ϵ}^{\text{abs}}+\frac{{ϵ}^{\text{rel}}}{G}{\Vert F^T𝒬{\mathit{\gamma }}^k\Vert }_2$. A common choice could be ${ϵ}^{\text{abs}}=0.001$ and ${ϵ}^{\text{rel}}=0.001$.

The augmented Lagrangian function for (A.9) is given by

$${ℒ}_{\varphi }(\mathit{z},\mathit{\beta };\mathit{\gamma })=f(\mathit{\beta })+g(\mathit{z})+\langle \gamma ,𝒬(\mathit{F}\mathit{\beta }+\mathit{G}\mathit{z}-\mathit{c})\rangle +\frac{\varphi }{2}{\Vert 𝒬(\mathit{F}\mathit{\beta }+\mathit{G}\mathit{z}-\mathit{c})\Vert }_2^2.$$

Using similar arguments in Sun et al. (2015), it follows that applying the updates in A.5 to problem (A.3) is equivalent to applying the following 2-block semi-proximal ADMM to (A.9),

$$\{\begin{array}{l}{\mathit{\beta }}^{k+1}=\text{argmin}{ℒ}_{\varphi }(\mathit{\beta },{\mathit{z}}^k;{\mathit{\gamma }}^k)+\frac{\varphi }{2}{\Vert \mathit{\beta }-{\mathit{\beta }}^k\Vert }_{{\mathit{F}}^T𝒫\mathit{F}+{𝒯}_f}^2,\hfill \\ {\mathit{z}}^{k+1}=\text{argmin}{ℒ}_{\varphi }({\mathit{\beta }}^{k+1},\mathit{z};{\mathit{\gamma }}^k)+\frac{\varphi }{2}{\Vert \mathit{z}-{\mathit{z}}^k\Vert }_{{\mathit{G}}^T𝒫\mathit{G}+{𝒯}_h}^2,\hfill \\ {\mathit{\gamma }}^{k+1}={\mathit{\gamma }}^k+\theta \varphi 𝒬(\mathit{F}{\mathit{\beta }}^{k+1}+\mathit{G}{\mathit{z}}^{k+1}-\mathit{c}).\hfill \end{array}$$ (A.11)

The Karush–Kuhn–Tucker (KKT) optimality condition of (A.9) is

$$0\in {(𝒬\mathit{F})}^T\mathit{\gamma }+\partial f(\mathit{\beta }),0\in {(𝒬\mathit{G})}^T\mathit{\gamma }+\partial g(\mathit{z}),𝒬(\mathit{c}-\mathit{F}\mathit{\beta }-\mathit{G}\mathit{z})=0.$$ (A.12)

Denote the solution set to (A.12) as $\overline{\text{Ω}}$, then we can replace Assumption 1 by assuming that $\overline{\text{Ω}}$ is non-empty. Let $\overline{\mathit{u}}=(\overline{\mathit{\beta }},\overline{\mathit{z}},\overline{\mathit{\gamma }})$ be an optimal solution to (A.9). We have the following lemma on the convergence of the proposed algorithm by utilizing its equivalence to the updates in (A.11).

Lemma A.1. Suppose Assumption 1 holds. ${𝒯}_f$ and ${𝒯}_h$ are chosen such that ${𝒯}_f+{\mathit{F}}^T\mathit{F}$ and ${𝒯}_h+{\mathit{G}}^T\mathit{G}$ are positive definite. Then under the condition $\theta \in (0,(1+\sqrt{5})/2)$, the sequence $\left({\mathit{\beta }}^k,{\mathit{z}}^k,{\mathit{\omega }}^k,{\mathit{\gamma }}^k\right)$ generated by (A.5) converges to a limit point $(\overline{\mathit{\beta }},\overline{\mathit{z}},\overline{\mathit{\omega }},\overline{\mathit{\gamma }})$ with $(\overline{\mathit{\beta }},\overline{\mathit{z}},\overline{\mathit{\omega }})$ solving (A.3) and $\overline{\mathit{\gamma }}$ is the dual optimal.

Lemma A.1 follows by a direct application of Theorem 3.2 in Han et al. (2018). Based on (A.2), we have the following fact.

Fact 2. Suppose ${\mathit{u}}^k$ converges to $\overline{\mathit{u}}\in \overline{\text{Ω}}$. There exists a positive constant q such that

$${\Vert {\mathit{u}}^k-\overline{\mathit{u}}\Vert }_2^2\le q^2({\Vert {\mathit{\beta }}^k-pro{x}_f({\mathit{\beta }}^k-{(𝒬\mathit{F})}^T{\mathit{\gamma }}^k)\Vert }_2^2+{\Vert {\mathit{z}}^k-pro{x}_h({\mathit{z}}^k-{(𝒬\mathit{G})}^T{\mathit{\gamma }}^k)\Vert }_2^2+{\Vert 𝒬(\mathit{c}-\mathit{F}{\mathit{\beta }}^k-\mathit{G}{\mathit{z}}^k)\Vert }_2^2),$$ (A.13)

for a sufficiently large k.

For any convex function P, prox_P(·) denotes the proximal mapping associated with P. That is,

$${\text{prox}}_P(x)=\underset{y}{\text{argmin}}\left\{\frac{1}{2}\parallel x-y{\parallel }_2^2+P(y)\right\}.$$ (A.14)

Denote $\mathscr{H}=C\times \text{Diag}\left({\mathit{F}}^T𝒫\mathit{F}+{𝒯}_f,{\mathit{G}}^T\mathit{G}+{𝒯}_h,\left({\theta }^{-2}{\varphi }^{-1}\right)I\right)$, where $C=\max \left\{3{\varphi }^2\Vert {\mathit{F}}^T𝒫\mathit{F}+{𝒯}_f\Vert {}_2,3{\varphi }^2{\lambda }_{\max }\left(\mathit{F}{\mathit{F}}^T\right),2{\varphi }^2{\Vert {\mathit{G}}^T𝒫\mathit{G}+{𝒯}_h\Vert }_2,3{(1-\theta )}^2\varphi {\lambda }_{\max }\left(𝒬\mathit{F}{\mathit{F}}^T𝒬\right))+2{(1-\theta )}^2\varphi {\lambda }_{\max }\left(𝒬\mathit{G}{\mathit{G}}^T𝒬\right)+\frac{1}{\varphi }\right\}$, then we have the following relationship.

Lemma A.2. Suppose the sequence ${\mathit{u}}^k=({\mathit{\beta }}^k,{\mathit{z}}^k,{\mathit{\gamma }}^k)$ is generated by algorithm (A.5), then for any $k\ge 0$,

$${\Vert {\mathit{u}}^{k+1}-\overline{\mathit{u}}\Vert }_2^2\le q^2{\Vert {\mathit{u}}^{k+1}-{\mathit{u}}^k\Vert }_{\mathscr{H}}^2.$$ (A.15)

Proof. Consider the optimality conditions of subproblems in (A.11), we have

$$\begin{gathered} 0\in \partial f({\mathit{\beta }}^{k+1})+{(𝒬\mathit{F})}^T{\mathit{\gamma }}^k+\varphi {(𝒬\mathit{F})}^T𝒬(\mathit{F}{\mathit{\beta }}^{k+1}+\mathit{G}{\mathit{z}}^k-\mathit{c}) \\ +\varphi ({\mathit{F}}^T𝒫\mathit{F}+{𝒯}_f)({\mathit{\beta }}^{k+1}-{\mathit{\beta }}^k), \\ 0\in \partial g({\mathit{z}}^{k+1})+{(𝒬\mathit{G})}^T{\mathit{\gamma }}^k+\varphi {(𝒬\mathit{G})}^T𝒬(\mathit{F}{\mathit{\beta }}^{k+1}+\mathit{G}{\mathit{z}}^{k+1}-\mathit{c}) \\ +\varphi ({\mathit{G}}^T𝒫\mathit{G}+{𝒯}_h)({\mathit{z}}^{k+1}-{\mathit{z}}^k), \\ 0={(\theta \varphi )}^{-1}({\mathit{\gamma }}^{k+1}-{\mathit{\gamma }}^k)-𝒬(\mathit{F}{\mathit{\beta }}^{k+1}+\mathit{G}{\mathit{z}}^{k+1}-\mathit{c}). \end{gathered}$$ (A.16)

Then we have $𝒬(\mathit{F}{\mathit{\beta }}^{k+1}+\mathit{G}{\mathit{z}}^k-\mathit{c})={(\theta \varphi )}^{-1}({\mathit{\gamma }}^{k+1}-{\mathit{\gamma }}^k)-𝒬\mathit{G}({\mathit{z}}^{k+1}-{\mathit{z}}^k)$ and it follows that

$$\begin{gathered} {\mathit{\beta }}^{k+1}={\text{prox}}_f({\mathit{\beta }}^{k+1}-{(𝒬\mathit{F})}^T\left({\mathit{\gamma }}^k+{\theta }^{-1}({\mathit{\gamma }}^{k+1}-{\mathit{\gamma }}^k)-\varphi 𝒬\mathit{G}({\mathit{z}}^{k+1}-{\mathit{z}}^k)\right) \\ +\varphi ({\mathit{F}}^T𝒫\mathit{F}+{𝒯}_f)({\mathit{\beta }}^{k+1}-{\mathit{\beta }}^k)), \\ {\mathit{z}}^{k+1}={\text{prox}}_g\left({\mathit{z}}^{k+1}-{(𝒬\mathit{G})}^T({\mathit{\gamma }}^k+{\theta }^{-1}({\mathit{\gamma }}^{k+1}-{\mathit{\gamma }}^k))+\varphi ({\mathit{G}}^T𝒫\mathit{G}+{𝒯}_h)({\mathit{z}}^{k+1}-{\mathit{z}}^k)\right), \\ {\mathit{\gamma }}^{k+1}={\mathit{\gamma }}^k+\theta \varphi 𝒬(\mathit{F}{\mathit{\beta }}^{k+1}+\mathit{G}{\mathit{z}}^{k+1}-\mathit{c}). \end{gathered}$$

and we have

$$\begin{gathered} {\Vert {\mathit{\beta }}^{k+1}-{\text{prox}}_f({\mathit{\beta }}^{k+1}-{(𝒬\mathit{F})}^T{\mathit{\gamma }}^{k+1})\Vert }_2^2+{\Vert {\mathit{z}}^{k+1}-{\text{prox}}_g({\mathit{z}}^{k+1}-{(𝒬\mathit{G})}^T{\mathit{\gamma }}^{k+1})\Vert }_2^2 \\ +{\Vert 𝒬(\mathit{c}-\mathit{F}{\mathit{\beta }}^{k+1}-\mathit{G}{\mathit{z}}^{k+1})\Vert }_2^2 \\ \le {\Vert {\mathit{u}}^{k+1}-{\mathit{u}}^k\Vert }_{\mathscr{H}}^2. \end{gathered}$$ (A.17)

We first bound the term ${\Vert {\mathit{\beta }}^{k+1}-{\text{prox}}_f({\mathit{\beta }}^{k+1}-{(𝒬\mathit{F})}^T{\mathit{\gamma }}^{k+1})\Vert }_2^2$. By the fact that the proximal mapping is Lipschitz continuous with constant 1, i.e., ${\parallel {\text{prox}}_h(x)-{\text{prox}}_h(y)\parallel }_2\le {\parallel x-y\parallel }_2$ for any mapping h,

$$\begin{gathered} {\Vert {\mathit{\beta }}^{k+1}-{\text{prox}}_f({\mathit{\beta }}^{k+1}-{(𝒬\mathit{F})}^T{\mathit{\gamma }}^{k+1})\Vert }_2^2 \\ \le \Vert {\mathit{\beta }}^{k+1}-{(𝒬\mathit{F})}^T\left({\mathit{\gamma }}^k+{\theta }^{-1}({\mathit{\gamma }}^{k+1}-{\mathit{\gamma }}^k)-\varphi 𝒬\mathit{G}({\mathit{z}}^{k+1}-{\mathit{z}}^k)\right) \\ +{\varphi ({\mathit{F}}^T𝒫\mathit{F}+{𝒯}_f)({\mathit{\beta }}^{k+1}-{\mathit{\beta }}^k)-{\mathit{\beta }}^{k+1}+{(Q\mathit{F})}^T{\mathit{\gamma }}^{k+1}\Vert }_2^2 \\ =\Vert \varphi ({\mathit{F}}^T𝒫\mathit{F}+{𝒯}_f)({\mathit{\beta }}^{k+1}-{\mathit{\beta }}^k)+\varphi {\mathit{F}}^T𝒬\mathit{G}({\mathit{z}}^{k+1}-{\mathit{z}}^k) \\ +{(1-\frac{1}{\theta }){(Q\mathit{F})}^T({\mathit{\gamma }}^{k+1}-{\mathit{\gamma }}^k)\Vert }_2^2 \\ ={\Vert \varphi ({\mathit{F}}^T𝒫\mathit{F}+{𝒯}_f)({\mathit{\beta }}^{k+1}-{\mathit{\beta }}^k)\Vert }_2^2+{\Vert \varphi {\mathit{F}}^T𝒬\mathit{G}({\mathit{z}}^{k+1}-{\mathit{z}}^k)\Vert }_2^2 \\ +{(1-\frac{1}{\theta })}^2{\Vert {(Q\mathit{F})}^T({\mathit{\gamma }}^{k+1}-{\mathit{\gamma }}^k)\Vert }_2^2 \\ +2{\varphi }^2({\mathit{\beta }}^{k+1}-{\mathit{\beta }}^k){({\mathit{F}}^T𝒫\mathit{F}+{𝒯}_f)}^T{\mathit{F}}^T𝒬\mathit{G}({\mathit{z}}^{k+1}-{\mathit{z}}^k) \\ +2(1-\frac{1}{\theta })\varphi ({\mathit{\beta }}^{k+1}-{\mathit{\beta }}^k){({\mathit{F}}^T𝒫\mathit{F}+{𝒯}_f)}^T{(Q\mathit{F})}^T({\mathit{\gamma }}^{k+1}-{\mathit{\gamma }}^k) \\ +2(1-\frac{1}{\theta })\varphi {({\mathit{z}}^{k+1}-{\mathit{z}}^k)}^T{\mathit{G}}^T𝒬\mathit{F}{(Q\mathit{F})}^T({\mathit{\gamma }}^{k+1}-{\mathit{\gamma }}^k) \end{gathered}$$ (A.18)

By taking into account the fact that

$$\begin{gathered} 2{\varphi }^2({\mathit{\beta }}^{k+1}-{\mathit{\beta }}^k){({\mathit{F}}^T𝒫\mathit{F}+{𝒯}_f)}^T{\mathit{F}}^T𝒬\mathit{G}({\mathit{z}}^{k+1}-{\mathit{z}}^k) \\ \le {\Vert \varphi ({\mathit{F}}^T𝒫\mathit{F}+{𝒯}_f)({\mathit{\beta }}^{k+1}-{\mathit{\beta }}^k)\Vert }_2^2+{\Vert \varphi {\mathit{F}}^T𝒬\mathit{G}({\mathit{z}}^{k+1}-{\mathit{z}}^k)\Vert }_2^2 \\ 2(1-\frac{1}{\theta })\varphi ({\mathit{\beta }}^{k+1}-{\mathit{\beta }}^k){({\mathit{F}}^T𝒫\mathit{F}+{𝒯}_f)}^T{(Q\mathit{F})}^T({\mathit{\gamma }}^{k+1}-{\mathit{\gamma }}^k) \\ \le {\Vert \varphi ({\mathit{F}}^T𝒫\mathit{F}+{𝒯}_f)({\mathit{\beta }}^{k+1}-{\mathit{\beta }}^k)\Vert }_2^2+{(1-\frac{1}{\theta })}^2{\Vert {(Q\mathit{F})}^T({\mathit{\gamma }}^{k+1}-{\mathit{\gamma }}^k)\Vert }_2^2, \end{gathered}$$

and

$$\begin{gathered} 2(1-\frac{1}{\theta })\varphi {({\mathit{z}}^{k+1}-{\mathit{z}}^k)}^T{\mathit{G}}^T𝒬\mathit{F}{(Q\mathit{F})}^T({\mathit{\gamma }}^{k+1}-{\mathit{\gamma }}^k) \\ \le {(1-\frac{1}{\theta })}^2{\Vert {(Q\mathit{F})}^T({\mathit{\gamma }}^{k+1}-{\mathit{\gamma }}^k)\Vert }_2^2+{\Vert \varphi {\mathit{F}}^T𝒬\mathit{G}({\mathit{z}}^{k+1}-{\mathit{z}}^k)\Vert }_2^2, \end{gathered}$$

and the inequality that

$$\begin{gathered} {\Vert {\mathit{F}}^T𝒬\mathit{G}({\mathit{z}}^{k+1}-{\mathit{z}}^k)\Vert }_2^2={(𝒬\mathit{G}({\mathit{z}}^{k+1}-{\mathit{z}}^k))}^T\mathit{F}{\mathit{F}}^T{(𝒬\mathit{G}({\mathit{z}}^{k+1}-{\mathit{z}}^k))}^T \\ \le {\lambda }_{\max }(\mathit{F}{\mathit{F}}^T){\Vert 𝒬\mathit{G}({\mathit{z}}^{k+1}-{\mathit{z}}^k)\Vert }_2^2, \end{gathered}$$

where ${\lambda }_{\max }(\mathit{F}{\mathit{F}}^T)$ is the largest eigenvalue of FF^T, (A.18) can be reduced to

$$\begin{gathered} {\Vert {\mathit{\beta }}^{k+1}-{\text{prox}}_f({\mathit{\beta }}^{k+1}-{(𝒬\mathit{F})}^T{\mathit{\gamma }}^{k+1})\Vert }_2^2 \\ \le 3{\varphi }^2{\Vert {\mathit{F}}^T𝒫\mathit{F}+{𝒯}_f\Vert }_2{\Vert {\mathit{\beta }}^{k+1}-{\mathit{\beta }}^k\Vert }_{{\mathit{F}}^T𝒫\mathit{F}+{𝒯}_f}^2+3{\varphi }^2{\lambda }_{\max }(\mathit{F}{\mathit{F}}^T){\Vert {\mathit{z}}^{k+1}-{\mathit{z}}^k\Vert }_{G^T𝒬G}^2 \\ +3{(1-\frac{1}{\theta })}^2{\Vert {(Q\mathit{F})}^T({\mathit{\gamma }}^{k+1}-{\mathit{\gamma }}^k)\Vert }_2^2. \end{gathered}$$ (A.19)

Similarly we can bound the term ${\Vert {\mathit{z}}^{k+1}-{\text{prox}}_g({\mathit{z}}^{k+1}-{(𝒬\mathit{G})}^T{\mathit{\gamma }}^{k+1})\Vert }_2^2$,

$$\begin{gathered} {\Vert {\mathit{z}}^{k+1}-{\text{prox}}_h({\mathit{z}}^{k+1}-{(𝒬\mathit{G})}^T{\mathit{\gamma }}^{k+1})\Vert }_2^2 \\ \le 2{\varphi }^2{\Vert {\mathit{G}}^T𝒫\mathit{G}+𝒯\Vert }_h{}_2{\Vert {\mathit{z}}^{k+1}-{\mathit{z}}^k\Vert }_{{\mathit{G}}^T\mathit{G}+{𝒯}_h}^2+2{(1-\frac{1}{\theta })}^2{\Vert {(𝒬\mathit{G})}^T({\mathit{\gamma }}^{k+1}-{\mathit{\gamma }}^k)\Vert }_2^2. \end{gathered}$$ (A.20)

From the update of $\mathit{\gamma }$, we have

$${\Vert 𝒬(\mathit{c}-\mathit{F}{\mathit{\beta }}^{k+1}-\mathit{G}{\mathit{z}}^{k+1})\Vert }_2^2={(\theta \varphi )}^{-2}{\Vert {\mathit{\gamma }}^{k+1}-{\mathit{\gamma }}^k\Vert }_2^2.$$ (A.21)

Combining (A.19), (A.20) and (A.21), we can obtain that

$$\begin{gathered} {\Vert {\mathit{\beta }}^{k+1}-{\text{prox}}_f({\mathit{\beta }}^{k+1}-{(𝒬\mathit{F})}^T{\mathit{\gamma }}^{k+1})\Vert }_2^2+{\Vert {\mathit{z}}^{k+1}-{\text{prox}}_h({\mathit{z}}^{k+1}-\mathit{G}{\mathit{\gamma }}^{k+1})\Vert }_2^2 \\ +{\Vert 𝒬(\mathit{c}-\mathit{F}{\mathit{\beta }}^{k+1}-\mathit{G}{\mathit{z}}^{k+1})\Vert }_2^2 \\ \le 3{\varphi }^2{\Vert {\mathit{F}}^T𝒫\mathit{F}+{𝒯}_f\Vert }_2{\Vert {\mathit{\beta }}^{k+1}-{\mathit{\beta }}^k\Vert }_{{\mathit{F}}^T𝒫\mathit{F}+{𝒯}_f}+3{\varphi }^2{\lambda }_{\max }(\mathit{F}{\mathit{F}}^T){\Vert {\mathit{z}}^{k+1}-{\mathit{z}}^k\Vert }_{{\mathit{G}}^T𝒬\mathit{G}}^2 \\ +{(\theta \varphi )}^{-2}{\Vert {\mathit{\gamma }}^{k+1}-{\mathit{\gamma }}^k\Vert }_2^2 \\ +3{(1-\frac{1}{\theta })}^2{\Vert {(𝒬\mathit{F})}^T({\mathit{\gamma }}^{k+1}-{\mathit{\gamma }}^k)\Vert }_2^2+2{\varphi }^2{\Vert {\mathit{G}}^T𝒫\mathit{G}+{𝒯}_h\Vert }_2{\Vert {\mathit{z}}^{k+1}-{\mathit{z}}^k\Vert }_{{\mathit{G}}^T𝒫\mathit{G}+{𝒯}_h}^2 \\ +2{(1-\frac{1}{\theta })}^2{\Vert {(𝒬\mathit{G})}^T({\mathit{\gamma }}^{k+1}-{\mathit{\gamma }}^k)\Vert }_2^2 \\ \le C\times \left({\Vert {\mathit{\beta }}^{k+1}-{\mathit{\beta }}^k\Vert }_{{\mathit{F}}^T𝒫+{𝒯}_f}^2+{\Vert {\mathit{z}}^{k+1}-{\mathit{z}}^k\Vert }_{{\mathit{G}}^T\mathit{G}+{𝒯}_h}^2+{\theta }^{-2}{\varphi }^{-1}{\Vert {\mathit{\gamma }}^{k+1}-{\mathit{\gamma }}^k\Vert }_2^2\right) \end{gathered}$$ (A.22)

□

Lemma A.3. Suppose that Assumptions 1 holds, and assume that both ${\mathit{F}}^T\mathit{F}+{𝒯}_f$ and ${\mathit{G}}^T\mathit{G}+{𝒯}_h$ are positive definite. Then for all k sufficiently large and $\theta \in (0,\frac{1+\sqrt{5}}{2})$, there exists $\mu \in (0,1)$ such that

$${\Vert {\mathit{u}}^{k+1}-\overline{\mathit{u}}\Vert }_{{\mathscr{H}}_1}+{\Vert {\mathit{z}}^{k+1}-{\mathit{z}}^k\Vert }_{{\mathit{G}}^T𝒫\mathit{G}+{𝒯}_h}^2\le \mu \left({\Vert {\mathit{u}}^k-\overline{\mathit{u}}\Vert }_{{\mathscr{H}}_1}+{\Vert {\mathit{z}}^k-{\mathit{z}}^{k-1}\Vert }_{\mathit{G}T𝒫\mathit{G}+{𝒯}_h}^2\right),$$ (A.23)

where

$${\mathscr{H}}_1=\left(\begin{array}{cccc}{\mathit{F}}^T\left(m_1𝒬+𝒫\right)\mathit{F}+{𝒯}_f& m_1{\mathit{F}}^T𝒬\mathit{G}& \cdots & \mathbf{0}\\ m_1{\mathit{G}}^T𝒬\mathit{F}& {\mathit{G}}^T\left(𝒫+\left(m_1+1\right)𝒬\right)\mathit{G}& \cdots & \mathbf{0}\\ \mathbf{0}& \mathbf{0}& \cdots & {\theta }^{-1}{\varphi }^{-2}I\end{array}\right)$$ (A.24)

with $m_1\in (0,1)$.

Proof. From Theorem 1 in Han et al. (2018), we can derive the following results.

$$\begin{gathered} \{{\Vert {\mathit{\beta }}^k-\overline{\mathit{\beta }}\Vert }_{{\mathit{F}}^T𝒫\mathit{F}+{𝒯}_f}^2+{\Vert {\mathit{z}}^k-\overline{\mathit{z}}\Vert }_{{\mathit{G}}^T\mathit{G}+{𝒯}_h}^2+{\Vert {\mathit{z}}^k-{\mathit{z}}^{k-1}\Vert }_{{\mathit{G}}^T𝒫\mathit{G}+{𝒯}_h}^2 \\ +(1-\min \{\theta ,\frac{1}{\theta }\}){\Vert 𝒬(\mathit{F}{\mathit{\beta }}^k+\mathit{G}{\mathit{z}}^k-\mathit{c})\Vert }_2^2+{\theta }^{-1}{\varphi }^{-2}{\Vert {\mathit{\gamma }}^k-\overline{\mathit{\gamma }}\Vert }_2^2\} \\ -\{{\Vert {\mathit{\beta }}^{k+1}-\overline{\mathit{\beta }}\Vert }_{{\mathit{F}}^T{𝒫}_{\mathit{F}}+{𝒯}_f}^2+{\Vert {\mathit{z}}^{k+1}-\overline{\mathit{z}}\Vert }_{{\mathit{G}}^T\mathit{G}+{𝒯}_h}^2+{\Vert {\mathit{z}}^{k+1}-{\mathit{z}}^k\Vert }_{{\mathit{G}}^T𝒫\mathit{G}+{𝒯}_h}^2 \\ +(1-\min \{\theta ,\frac{1}{\theta }\}){\Vert 𝒬(\mathit{F}{\mathit{\beta }}^{k+1}+\mathit{G}{\mathit{z}}^{k+1}-\mathit{c})\Vert }_2^2+{\theta }^{-1}{\varphi }^{-2}{\Vert {\mathit{\gamma }}^{k+1}-\overline{\mathit{\gamma }}\Vert }_2^2\} \\ \ge {\Vert {\mathit{z}}^{k+1}-{\mathit{z}}^k\Vert }_{{\mathit{G}}^T𝒫\mathit{G}+{𝒯}_h+(\theta -{\theta }^2+\min ({\theta }^2,1))\mathit{G}{𝒬}^T\mathit{G}}^2+{\Vert {\mathit{\beta }}^{k+1}-{\mathit{\beta }}^k\Vert }_{{\mathit{F}}^T𝒫\mathit{F}+{𝒯}_f}^2 \\ +(1-\theta +\min \left\{\theta ,{\theta }^{-1}\right\}){\Vert 𝒬(\mathit{F}{\mathit{\beta }}^{k+1}+\mathit{G}{\mathit{z}}^{k+1}-\mathit{c})\Vert }_2^2 \end{gathered}$$ (A.25)

When $\theta \in (0,\frac{1+\sqrt{5}}{2})$, it is ensured that $\left(1-\theta +\varphi \min \left\{\theta ,{\theta }^{-1}\right\}\right)>0$. Let $d_1\in (0,\frac{1}{2})$, then we have

$$\begin{gathered} \{{\Vert {\mathit{\beta }}^k-\overline{\mathit{\beta }}\Vert }_{{\mathit{F}}^T𝒫\mathit{F}+{𝒯}_f}^2+{\Vert {\mathit{z}}^k-\overline{\mathit{z}}\Vert }_{{\mathit{G}}^T\mathit{G}+{𝒯}_h}^2+{\Vert {\mathit{z}}^k-{\mathit{z}}^{k-1}\Vert }_{{\mathit{G}}^T𝒫\mathit{G}+{𝒯}_h}^2 \\ +(1+d_1-d_1\theta -(1-d_1)\min \{\theta ,\frac{1}{\theta }\}){\Vert 𝒬(\mathit{F}{\mathit{\beta }}^k+\mathit{G}{\mathit{z}}^k-\mathit{c})\Vert }_2^2 \\ +{\theta }^{-1}{\varphi }^{-2}{\Vert {\mathit{\gamma }}^k-\overline{\mathit{\gamma }}\Vert }_2^2\} \\ -\{{\Vert {\mathit{\beta }}^{k+1}-\overline{\mathit{\beta }}\Vert }_{{\mathit{F}}^T𝒫\mathit{F}+{𝒯}_f}^2+{\Vert {\mathit{z}}^{k+1}-\overline{\mathit{z}}\Vert }_{{\mathit{G}}^T\mathit{G}+{𝒯}_h}^2+{\Vert {\mathit{z}}^{k+1}-{\mathit{z}}^k\Vert }_{{\mathit{G}}^T𝒫\mathit{G}+{𝒯}_h}^2 \\ +(1+d_1-d_1\theta -(1-d_1)\min \{\theta ,\frac{1}{\theta }\}){\Vert 𝒬(\mathit{F}{\mathit{\beta }}^{k+1}+\mathit{G}{\mathit{z}}^{k+1}-\mathit{c})\Vert }_2^2 \\ +{\theta }^{-1}{\varphi }^{-2}{\Vert {\mathit{\gamma }}^{k+1}-\overline{\mathit{\gamma }}\Vert }_2^2\} \end{gathered}$$ (A.26)

$$\begin{gathered} \ge {\Vert {\mathit{z}}^{k+1}-{\mathit{z}}^k\Vert }_{{\mathit{G}}^T𝒫\mathit{G}+{𝒯}_h+(\theta -{\theta }^2+\min ({\theta }^2,1)){\mathit{G}}^T𝒬\mathit{G}}^2+{\Vert {\mathit{\beta }}^{k+1}-{\mathit{\beta }}^k\Vert }_{{\mathit{F}}^T𝒫\mathit{F}+{𝒯}_f}^2 \\ +(1-d_1)(1-\theta +\min \left\{\theta ,{\theta }^{-1}\right\}){\Vert 𝒬(\mathit{F}{\mathit{\beta }}^{k+1}+\mathit{G}{\mathit{z}}^{k+1}-\mathit{c})\Vert }_2^2 \\ +d_1(1-\theta +\min \left\{\theta ,{\theta }^{-1}\right\}){\Vert 𝒬(\mathit{F}{\mathit{\beta }}^k+\mathit{G}{\mathit{z}}^k-\mathit{c})\Vert }_2^2 \\ ={\Vert {\mathit{z}}^{k+1}-{\mathit{z}}^k\Vert }_{{\mathit{G}}^T𝒫\mathit{G}+{𝒯}_h+(\theta -{\theta }^2+\min ({\theta }^2,1)){\mathit{G}}^T𝒬\mathit{G}}^2+{\Vert {\mathit{\beta }}^{k+1}-{\mathit{\beta }}^k\Vert }_{{\mathit{F}}^T𝒫\mathit{F}+{𝒯}_f}^2 \\ +(1-2d_1))(1-\theta +\min \left\{\theta ,{\theta }^{-1}\right\}){\theta }^{-2}{\varphi }^{-2}{\Vert {\mathit{\gamma }}^{k+1}-{\mathit{\gamma }}^k\Vert }_2^2 \\ +d_1(1-\theta +\min \left\{\theta ,{\theta }^{-1}\right\})({\Vert 𝒬(\mathit{F}{\mathit{\beta }}^k+\mathit{G}{\mathit{z}}^k-\mathit{c})\Vert }_2^2 \\ +{\Vert 𝒬(\mathit{F}{\mathit{\beta }}^{k+1}+\mathit{G}{\mathit{z}}^{k+1}-\mathit{c})\Vert }_2^2) \\ \ge {\Vert {\mathit{z}}^{k+1}-{\mathit{z}}^k\Vert }_{{\mathit{G}}^T𝒫\mathit{G}+{𝒯}_h+(\theta -{\theta }^2+\min ({\theta }^2,1)){\mathit{G}}^T𝒬\mathit{G}}^2+{\Vert {\mathit{\beta }}^{k+1}-{\mathit{\beta }}^k\Vert }_{{\mathit{F}}^T𝒫\mathit{F}+{𝒯}_f}^2 \\ +(1-2d_1)(1-\theta +\min \left\{\theta ,{\theta }^{-1}\right\}){\theta }^{-2}{\varphi }^{-2}{\Vert {\mathit{\gamma }}^{k+1}-{\mathit{\gamma }}^k\Vert }_2^2 \\ +\frac{1}{2}{d}_1(1-\theta +\min \left\{\theta ,{\theta }^{-1}\right\}){\Vert 𝒬\mathit{F}({\mathit{\beta }}^{k+1}-{\mathit{\beta }}^k)+𝒬\mathit{G}({\mathit{z}}^{k+1}-{\mathit{z}}^k)\Vert }_2^2 \end{gathered}$$ (A.27)

Note that $𝒬(\mathit{F}{\mathit{\beta }}^{k+1}+\mathit{G}{\mathit{z}}^{k+1}-\mathit{c})=𝒬\mathit{F}({\mathit{\beta }}^{k+1}-\overline{\mathit{\beta }})+𝒬\mathit{G}({\mathit{z}}^{k+1}-\overline{\mathit{z}})$, and we have

$$\begin{gathered} \{{\Vert {\mathit{\beta }}^k-\overline{\mathit{\beta }}\Vert }_{{\mathit{F}}^T𝒫\mathit{F}+{𝒯}_f}^2+{\Vert {\mathit{z}}^k-\overline{\mathit{z}}\Vert }_{{\mathit{G}}^T\mathit{G}+{𝒯}_h}^2+{\Vert {\mathit{z}}^k-{\mathit{z}}^{k-1}\Vert }_{{\mathit{G}}^T𝒫\mathit{G}+{𝒯}_h}^2 \\ +(1+d_1-d_1\theta -(1-d_1)\min \{\theta ,\frac{1}{\theta }\})𝒬\mathit{F}({\mathit{\beta }}^k-\overline{\mathit{\beta }})+𝒬\mathit{G}{({\mathit{z}}^k-\overline{\mathit{z}})}_2^2 \\ +{\theta }^{-1}{\varphi }^{-2}{\Vert {\mathit{\gamma }}^k-\overline{\mathit{\gamma }}\Vert }_2^2\} \\ -\{{\Vert {\mathit{\beta }}^{k+1}-\overline{\mathit{\beta }}\Vert }_{{\mathit{F}}^T𝒫\mathit{F}+{𝒯}_f}^2+{\Vert {\mathit{z}}^{k+1}-\overline{\mathit{z}}\Vert }_{{\mathit{G}}^T\mathit{G}+{𝒯}_h}^2+{\Vert {\mathit{z}}^{k+1}-{\mathit{z}}^k\Vert }_{{\mathit{G}}^T𝒫\mathit{G}+{𝒯}_h}^2 \\ +(1+d_1-d_1\theta -(1-d_1)\min \{\theta ,\frac{1}{\theta }\})\parallel 𝒬\mathit{F}({\mathit{\beta }}^{k+1}-\overline{\mathit{\beta }})+𝒬\mathit{G}({\mathit{z}}^{k+1}-\overline{\mathit{z}})){\parallel }_2^2 \\ +{\theta }^{-1}{\varphi }^{-2}{\Vert {\mathit{\gamma }}^{k+1}-\overline{\mathit{\gamma }}\Vert }_2^2\} \\ \ge {\Vert {\mathit{z}}^{k+1}-{\mathit{z}}^k\Vert }_{{\mathit{G}}^T𝒫\mathit{G}+{𝒯}_h+(\theta -{\theta }^2+\min ({\theta }^2,1)){\mathit{G}}^T𝒬\mathit{G}}^2+{\Vert {\mathit{\beta }}^{k+1}-{\mathit{\beta }}^k\Vert }_{{\mathit{F}}^T𝒫\mathit{F}+{𝒯}_f}^2 \\ +(1-2d_1)(1-\theta +\min \left\{\theta ,{\theta }^{-1}\right\}){\theta }^{-2}{\varphi }^{-2}{\Vert {\mathit{\gamma }}^{k+1}-{\mathit{\gamma }}^k\Vert }_2^2 \\ +\frac{1}{2}{d}_1(1-\theta +\min \left\{\theta ,{\theta }^{-1}\right\})𝒬\mathit{F}({\mathit{\beta }}^{k+1}-{\mathit{\beta }}^k)+𝒬\mathit{G}{({\mathit{z}}^{k+1}-{\mathit{z}}^k)}_2^2 \\ \ge (1-\theta +\min \left\{\theta ,{\theta }^{-1}\right\})\min \{\frac{1}{2}{d}_1,1-2d_1,\theta \}({\Vert {\mathit{\beta }}^{k+1}-{\mathit{\beta }}^k\Vert }_{{\mathit{F}}^T𝒫\mathit{F}+{𝒯}_f}^2 \\ +{\Vert {\mathit{z}}^{k+1}-{\mathit{z}}^k\Vert }_{{\mathit{G}}^T\mathit{G}+{𝒯}_h}^2+{\theta }^{-2}{\varphi }^{-2}{\Vert {\mathit{\gamma }}^{k+1}-{\mathit{\gamma }}^k\Vert }_2^2) \end{gathered}$$ (A.28)

Let $m_1=1+d_1-d_1\theta -\left(1-d_1\right)\min \{\theta ,\frac{1}{\theta }\}$ in ${\mathscr{H}}_1$ defined in (A.24), and $m_2=\left(1-\theta +\min \left\{\theta ,{\theta }^{-1}\right\}\right)\min \{\frac{1}{2}{d}_1,1-2d_1,\theta \}$. Note that when $\theta \in (0,\frac{1+\sqrt{5}}{2})$, the following relationship holds.

$${\mathit{F}}^T\mathit{F}+{𝒯}_f\succ 0\text{and}{\mathit{G}}^T\mathit{G}+{𝒯}_h\succ 0\iff {\mathscr{H}}_1\succ 0.$$

Combining with Lemma A.2, we have

$$\begin{gathered} {\Vert {\mathit{u}}^k-\overline{\mathit{u}}\Vert }_{{\mathscr{H}}_1}^2+{\Vert {\mathit{z}}^k-{\mathit{z}}^{k-1}\Vert }_{{\mathit{G}}^T\mathit{G}+{𝒯}_h}^2-({\Vert {\mathit{u}}^{k+1}-\overline{\mathit{u}}\Vert }_{{\mathscr{H}}_1}+{\Vert {\mathit{z}}^{k+1}-{\mathit{z}}^k\Vert }_{{\mathit{G}}^T\mathit{G}+{𝒯}_h}^2) \\ \ge \frac{m_2}{C}(C\times ({\Vert {\mathit{\beta }}^{k+1}-{\mathit{\beta }}^k\Vert }_{{\mathit{F}}^T𝒫\mathit{F}+{𝒯}_f}^2+{\Vert {\mathit{z}}^{k+1}-{\mathit{z}}^k\Vert }_{{\mathit{G}}^T\mathit{G}+{𝒯}_h}^2 \\ +{\theta }^{-2}{\varphi }^{-2}{\Vert {\mathit{\gamma }}^{k+1}-{\mathit{\gamma }}^k\Vert }_2^2)) \\ =\frac{m_2}{C}{\Vert {\mathit{u}}^{k+1}-{\mathit{u}}^k\Vert }_{\mathscr{H}}^2\ge \frac{m_2{d}_2}{C{q}^2}{\Vert {\mathit{u}}^{k+1}-\overline{\mathit{u}}\Vert }_2^2+\frac{m_2\left(1-d_2\right)}{C{q}^2}{\Vert {\mathit{z}}^{k+1}-{\mathit{z}}^k\Vert }_{{\mathit{G}}^T\mathit{G}+{𝒯}_h}^2 \\ \ge \frac{m_2{d}_2}{C{q}^2{\lambda }_{\max }\left({\mathscr{H}}_1\right)}{\Vert {\mathit{u}}^{k+1}-\overline{\mathit{u}}\Vert }_{{\mathscr{H}}_1}^2+\frac{m_2\left(1-d_2\right)}{C{q}^2}{\Vert {\mathit{z}}^{k+1}-{\mathit{z}}^k\Vert }_{{\mathit{G}}^T\mathit{G}+{𝒯}_h}^2. \end{gathered}$$ (A.29)

Take $d_2=\frac{{\lambda }_{\max }\left({\mathscr{H}}_1\right)}{1+{\lambda }_{\max }\left({\mathscr{H}}_1\right)}$, then we can obtain (A.23) with $\mu ={\left[1+\frac{m_2}{C{q}^2\left(1+{\lambda }_{\max }\left({\mathscr{H}}_1\right)\right)}\right]}^{-1}$. □

Proof of Theorem 1. Since $f={\parallel \cdot \parallel }_1$ and $g=\frac{1}{n}{1}_n^T{(\cdot )}_{+}$ are piecewise linear-quadratic functions, thus both prox_f(·) and prox_g(·) are piecewise polyhedral (Poliquin and Rockafellar, 1993) which implies Fact 2 (Han et al., 2018). Since we take ${𝒯}_g={\eta }_i{I}_{p_g}-{\mathit{X}}_g^T{\mathit{X}}_g,g=1,\cdots ,G$, then ${𝒯}_g+{\mathit{F}}^T\mathit{F}=\text{Diag}\left({\eta }_1{I}_{p_1},\cdots ,{\eta }_K{I}_{p_K}\right)$ is positive definite, and this together with the fact that ${\mathit{G}}^T\mathit{G}=I_n\succ 0$ imply that the sequence $\left({\mathit{\beta }}^k,{\mathit{z}}^k,{\mathit{\omega }}^k,{\mathit{\gamma }}^k\right)$ is automatically well defined. By Lemma A.1, under the condition $\theta \in (0,(1+\sqrt{5})/2)$, the sequence $\left({\mathit{\beta }}^k,{\mathit{z}}^k,{\mathit{\omega }}^k,{\mathit{\gamma }}^k\right)$ generated by algorithm (A.5) converges to a limit point $(\overline{\mathit{\beta }},\overline{\mathit{z}},\overline{\mathit{\omega }},\overline{\mathit{\gamma }})$ with $(\overline{\mathit{\beta }},\overline{\mathit{z}},\overline{\mathit{\omega }})$ solving (9) and $\overline{\mathit{\gamma }}$ is the dual optimal.

To derive the rate of convergence, we first compute ${\mathscr{H}}_1$. By definition,

$$𝒫=\mathit{H}{({\mathit{H}}^T\mathit{H})}^{-1}{\mathit{H}}^T=\frac{1}{G}\left(\begin{array}{cccc}(G-1)I& -I& \cdots & -I\\ -I& (G-1)I& \cdots & -I\\ ⋮& \ddots & & ⋮\\ ⋮& & \ddots & ⋮\\ -I& -I& \cdots & (G-1)I\end{array}\right)$$

It follows that

$$\begin{gathered} {\Vert {\mathit{\beta }}^{k+1}-\overline{\mathit{\beta }}\Vert }_{{\mathit{F}}^T𝒫\mathit{F}}^2=\sum _{i=1}^G{\Vert {\mathit{X}}_i({\mathit{\beta }}_i^{k+1}-{\overline{\mathit{\beta }}}_i)\Vert }_2^2-\frac{1}{G}{\Vert \sum _{i=1}^G{\mathit{X}}_i^T({\mathit{\beta }}_i^{k+1}-{\overline{\mathit{\beta }}}_i)\Vert }_2^2, \\ {m}_1{\Vert 𝒬\mathit{F}({\mathit{\beta }}^{k+1}-\overline{\mathit{\beta }})+𝒬\mathit{G}({\mathit{z}}^{k+1}-\overline{\mathit{z}})\Vert }_2^2=\frac{m_1}{G}{\Vert \sum _{i=1}^G{\mathit{X}}_i({\mathit{\beta }}_i^{k+1}-{\overline{\mathit{\beta }}}_i)+({\mathit{z}}^{k+1}-\overline{\mathit{z}})\Vert }_2^2, \\ {\Vert {\mathit{z}}^{k+1}-\overline{\mathit{z}}\Vert }_{{\mathit{G}}^T\mathit{G}+{𝒯}_h}^2={\Vert {\mathit{z}}^{k+1}-\overline{\mathit{z}}\Vert }_2^2, \\ {\Vert {\mathit{z}}^{k+1}-{\mathit{z}}^k\Vert }_{{\mathit{G}}^T𝒫\mathit{G}+{𝒯}_h}^2=\frac{G-1}{G}{\Vert {\mathit{z}}^{k+1}-{\mathit{z}}^k\Vert }_2^2. \end{gathered}$$ (A.30)

Plugging equations (A.30) back into (A.29), we derive the results in Theorem 1 easily.

A.3. Algorithms for Two-Step PQR-SCAD

This section presents two three-block ADMM algorithms for PQR-SCAD proposed in Section 2.3.

Algorithm 3.

FS-QRADMM-CD for Two-Step PQR-SCAD

Initialization: ${\tilde{\mathit{\beta }}}^0,\lambda ,v,{\tilde{\mathit{z}}}^0,{\tilde{\mathit{\gamma }}}^0,{\tilde{\mathit{\omega }}}_i^0$, and $\varphi >0,\theta =1.618,k=0$.

while the stopping criterion is not satisfied, do

Update ${\tilde{\mathit{\beta }}}^{k+1}$ by

${\tilde{\mathit{\beta }}}_1^{k+1}=\underset{{\mathit{\beta }}_1\in {ℛ}^{p_1}}{\text{argmin}}nv\lambda {\Vert {\mathit{\beta }}_1\Vert }_1+\frac{\varphi }{2}{\Vert {\mathit{X}}_1{\mathit{\beta }}_1+\sum _{g=2}^G{\tilde{\mathit{\omega }}}_g^k+{\tilde{\mathit{z}}}^k-\mathit{y}+\frac{{\tilde{\mathit{\gamma }}}_1^k}{\varphi }\Vert }_2^2,$

${\tilde{\mathit{\beta }}}_g^{k+1}=\underset{{\mathit{\beta }}_g\in {ℛ}^{p_g}}{\text{argmin}}nv\lambda {\Vert {\mathit{\beta }}_g\Vert }_1+\frac{\varphi }{2}{\Vert {\mathit{X}}_g{\mathit{\beta }}_g-{\tilde{\mathit{\omega }}}_g^k+\frac{{\tilde{\mathit{\gamma }}}_g^k}{\varphi }\Vert }_2^2,g=2,\cdots ,G$

Compute ${\tilde{\mathit{\omega }}}^{k+\frac{1}{2}},{\tilde{\mathit{z}}}^{k+1}$ and ${\tilde{\mathit{\omega }}}^{k+1}$ by (13).

Update ${\tilde{\mathit{\gamma }}}^{k+1}$ by (14).

end while The solution is denoted as ${\widehat{\mathit{\beta }}}^{{\ell }_1},{\widehat{\mathit{z}}}^{{\ell }_1},{\widehat{\mathit{\omega }}}^{{\ell }_1}$.

Initialization: ${\widehat{\mathit{\beta }}}^0={\widehat{\mathit{\beta }}}^{{\ell }_1},{\widehat{\mathit{z}}}^0={\widehat{\mathit{z}}}^{{\ell }_1},{\widehat{\mathit{\omega }}}^0={\widehat{\mathit{\omega }}}^{{\ell }_1}$ and $\varphi >0,\theta =1.618,k=0$. Compute

${\alpha }_j={\lambda }^{-1}{p}_{\lambda }(|{\widehat{\beta }}_j^0|)$ for $j=1,\cdots ,p$.

while the stopping criterion is not satisfied, do

Update ${\widehat{\mathit{\beta }}}^{k+1}$ by (12).

Compute ${\widehat{\mathit{\omega }}}^{k+\frac{1}{2}},{\widehat{\mathit{z}}}^{k+1}$ and ${\widehat{\mathit{\omega }}}^{k+1}$ by (13).

Update ${\widehat{\mathit{\gamma }}}^{k+1}$ by (14).

end while

Algorithm 4.

FS-QRADMM-prox for Two-Step PQR-SCAD

Initialization: ${\tilde{\mathit{\beta }}}^0,\lambda ,v,{\tilde{\mathit{z}}}^0,{\tilde{\mathit{\gamma }}}^0,{\tilde{\mathit{\omega }}}_i^0$, and $\varphi >0,\theta =1.618,k=0$.

while the stopping criterion is not satisfied, do

Update ${\tilde{\mathit{\beta }}}^{k+1}$ by

${\tilde{\mathit{\beta }}}_1^{k+1}=\text{Shrink}{\left({\tilde{\mathit{\beta }}}_{1j}^k-\frac{\varphi }{{\eta }_1}{\mathit{X}}_{1j}^T({\mathit{X}}_1{\tilde{\mathit{\beta }}}_1^k+\sum _{g=2}^G{\tilde{\mathit{\omega }}}_g^k+{\tilde{\mathit{z}}}^k-\mathit{y}+\frac{{\tilde{\mathit{\gamma }}}_1^k}{\varphi }),\frac{nv\lambda }{{\eta }_1}\right)}_{j=1,\cdots ,p_1}$

${\tilde{\mathit{\beta }}}_g^{k+1}=\text{Shrink}{\left({\tilde{\mathit{\beta }}}_{gj}^k-\frac{\varphi }{{\eta }_g}{\mathit{X}}_{gj}^T({\mathit{X}}_g{\tilde{\mathit{\beta }}}_g^k-{\tilde{\mathit{\omega }}}_g^k+\frac{{\tilde{\mathit{\gamma }}}_g^k}{\varphi }),\frac{nv\lambda }{{\eta }_g}\right)}_{j=1,\cdots ,p_g},g=2,\cdots ,G.$

Compute ${\tilde{\mathit{\omega }}}^{k+\frac{1}{2}},{\tilde{\mathit{z}}}^{k+1}$ and ${\tilde{\mathit{\omega }}}^{k+1}$by (13).

Update ${\tilde{\mathit{\gamma }}}^{k+1}$ by (14).

end while denote the solution as ${\widehat{\mathit{\beta }}}^{{\ell }_1},{\widehat{\mathit{z}}}^{{\ell }_1},{\widehat{\mathit{\omega }}}^{{\ell }_1}$

Initialization: ${\widehat{\mathit{\beta }}}^0={\widehat{\mathit{\beta }}}^{{\ell }_1},{\widehat{\mathit{z}}}^0={\widehat{\mathit{z}}}^{{\ell }_1},{\widehat{\mathit{\omega }}}^0={\widehat{\mathit{\omega }}}^{{\ell }_1}$ and $\varphi >0,\theta =1.618,k=0$. Compute

${\alpha }_j={\lambda }^{-1}{p}_{\lambda }(|{\widehat{\beta }}_j^0|)$ for $j=1,\cdots ,p$.

while the stopping criterion is not satisfied, do

Update ${\widehat{\mathit{\beta }}}^{k+1}$ by (15).

Compute ${\widehat{\mathit{\omega }}}^{k+\frac{1}{2}},{\widehat{\mathit{z}}}^{k+1}$ and ${\widehat{\mathit{\omega }}}^{k+1}$ by (13).

Update ${\widehat{\mathit{\gamma }}}^{k+1}$ by (14).

end while