The linearized alternating direction method of multipliers for low-rank and fused LASSO matrix regression model

Abstract

Datasets with matrix and vector form are increasingly popular in modern scientific fields. Based on structures of datasets, matrix and vector coefficients need to be estimated. At present, the matrix regression models were proposed, and they mainly focused on the matrix without vector variables. In order to fully explore complex structures of datasets, we propose a novel matrix regression model which combines fused LASSO and nuclear norm penalty, which can deal with the data containing matrix and vector variables meanwhile. Our main work is to design an efficient algorithm to solve the proposed low-rank and fused LASSO matrix regression model. Following the existing idea, we design the linearized alternating direction method of multipliers and establish its global convergence. Finally, we carry out numerical experiments to demonstrate the efficiency of our method. Especially, we apply our model to two real datasets, i.e. the signal shapes and the trip time prediction from partial trajectories.

1. Introduction

In the era of big data, modern scientific applications are more complex, and sampling units combine matrices with vectors instead of containing one form. A well-known example is the study of electroencephalography (EEG) data set of alcoholism, as in [23]. The study consists of 122 subjects with two groups, an alcoholic group and a normal control group, with each subject being exposed to a stimulus. Voltage values are measured from 64 channels of electrodes placed on the subject's scalp for 256-time points, so each sampling unit is a $256\times 64$ matrix. We would face intricate challenges if turning this matrix to vector. On the one hand, the dimension is $p=256\times 64=16384$, but the sample size is $n=122\ll 16384$. On the other hand, vectorization destroys the structural information in matrix data. It is crucial to propose a novel matrix regression model to deal with this sampling data. Zhou and Li [23] proposed the matrix regression

$$y_i=\mathrm{trace}(X_i^{\mathrm{T}}B)+Z_i^{\mathrm{T}}\gamma +{\epsilon }_i,$$ (1)

where $y_i\in \mathbb{R}$ is the response, $B\in {\mathbb{R}}^{m\times q},\gamma \in {\mathbb{R}}^p$ are the objective regression coefficients, $X_i\in {\mathbb{R}}^{m\times q}$ is the matrix variate and $Z_i\in {\mathbb{R}}^p$ is the vector variate, ${\epsilon }_i\in \mathbb{R}$ is the noise which follows a normal distribution with mean 0 and standard deviation σ. However, Zhou and Li [23] only analyse the properties and algorithm when γ is 0. At present, few researchers considered matrix regression model (1), let alone this model with penalized regularization. It is necessary to study this model because of the data's form containing matrix and vector at the same time.

In this paper, we focus on the low-rank and fused LASSO (LRFL) matrix regression

$$\underset{\genfrac{}{}{0ex}{}{B\in {\mathbb{R}}^{m\times q}}{\gamma \in {\mathbb{R}}^p}}{min}\frac{1}{2}\sum _{i=1}^N{(y_i-\mathrm{trace}(X_i^{\mathrm{T}}B)-Z_i^{\mathrm{T}}\gamma )}^2+{\lambda }_1\Vert B{\Vert }_{\ast }+{\lambda }_2\Vert \gamma {\Vert }_1+{\lambda }_3\sum _{j=2}^p|{\gamma }_j-{\gamma }_{j-1}|,$$ (2)

where $0<{\lambda }_i(i=1,\dots ,3)$ are tuning parameters. Unlike some typical models only containing vectors or matrices, the first and second $l_1$-norm in this model induce the sparsity of both the coefficients γ and their successive differences, and the nuclear norm $\Vert B{\Vert }_{\ast }$ induces the low rank of unknown regression matrix B. The sparsity of γ can help us to choose the most important variables. And the low rank of B is used to pick up the pivotal information in matrix variable. In model (2), if we get rid of B, the model degenerates into the fused LASSO (FLASSO) introduced by Tibshirani et al. [18]. If we take away γ from model (2), it comes into the nuclear norm regularized matrix regression studied in Zhou and Li [23]. In this paper, we will mainly focus on designing an efficient method for solving LRFL matrix regression model (2).

In recent years, researchers proposed many regularized regression models with different penalties, such as Power family [7], elastic net [24], log-penalty [1,3], SCAD [6], and MC+ [22]. Meanwhile, Zhou and Li [23] proposed matrix regression model and considered the low rank of B based on spectral regularization. There is also some work related to matrix data, such as [4,13,14,17,20,21]. However, none of them considered matrix and vector variables together. Thus, the model combining matrix and vector variables is essential to be studied. The basic work is studying the statistical property and designing the algorithm for computing solution of LRFL matrix regression model (2). Zhou and Li [23] used the Nesterov method to solve spectral regularized matrix regression. Although this method is analytically simple, it is not suitable for model (2) because of two variables. What's more, Li et al. proposed the linearized alternating direction method of multipliers (LADMM) algorithm in [12] for solving FLASSO. Owing to objective function in our model (2) is convex with respect to B and γ, we can consider to have a natural extension to our model. Fortunately, following the procedure of Li et al. [12], we develop a LADMM algorithm for solving model (2).

The rest of the paper is organized as follows. In Section 2, some preliminaries which are useful for further discussion will be introduced. Especially, we present two important optimization problems appeared in our algorithm. In Section 3, we give the LADMM algorithm for the LRFL matrix regression model (2). In Section 4, we demonstrate the convergence of the obtained algorithm. We conduct extensive numerical experiments to evaluate the performance of our algorithm for LRFL matrix regression model (2) in Section 5. We conclude this paper in the final section.

2. Preliminary

In this section, we introduce some preliminaries which are useful for further discussion. Firstly, we give some symbols about derivative. Then, we briefly introduce the solution of two important optimization problems.

2.1. Matrix calculation

• If$f(B)$ is a real-valued function of $B\in {\mathbb{R}}^{m\times q}$, the derivative of y with respect to B is defined that

  $${\mathrm{\nabla }}_{B}f=\left(\begin{array}{cccc}{\displaystyle \frac{\mathrm{\partial }f}{\mathrm{\partial }{B}_{11}}}& {\displaystyle \frac{\mathrm{\partial }f}{\mathrm{\partial }{B}_{12}}}& \cdots & {\displaystyle \frac{\mathrm{\partial }f}{\mathrm{\partial }{B}_{1q}}}\\ {\displaystyle \frac{\mathrm{\partial }f}{\mathrm{\partial }{B}_{21}}}& {\displaystyle \frac{\mathrm{\partial }f}{\mathrm{\partial }{B}_{22}}}& \cdots & {\displaystyle \frac{\mathrm{\partial }f}{\mathrm{\partial }{B}_{2q}}}\\ ⋮& ⋮& ⋮& ⋮\\ {\displaystyle \frac{\mathrm{\partial }f}{\mathrm{\partial }{B}_{m1}}}& {\displaystyle \frac{\mathrm{\partial }f}{\mathrm{\partial }{B}_{m2}}}& \cdots & {\displaystyle \frac{\mathrm{\partial }f}{\mathrm{\partial }{B}_{mq}}}\end{array}\right).$$
• If $B\in {\mathbb{R}}^{m\times q},Y\in {\mathbb{R}}^{n\times p}$ and $Y_{ij}$ is real-valued function of B, the derivative of Y with respect to B is defined that

  $${\mathrm{\nabla }}_{B}Y=\left(\begin{array}{cccc}{\displaystyle \frac{\mathrm{\partial }{Y}_{11}}{\mathrm{\partial }{B}_{11}}}& {\displaystyle \frac{\mathrm{\partial }{Y}_{11}}{\mathrm{\partial }{B}_{12}}}& \cdots & {\displaystyle \frac{\mathrm{\partial }{Y}_{11}}{\mathrm{\partial }{B}_{mq}}}\\ {\displaystyle \frac{\mathrm{\partial }{Y}_{12}}{\mathrm{\partial }{B}_{11}}}& {\displaystyle \frac{\mathrm{\partial }{Y}_{12}}{\mathrm{\partial }{B}_{12}}}& \cdots & {\displaystyle \frac{\mathrm{\partial }{Y}_{12}}{\mathrm{\partial }{B}_{mq}}}\\ ⋮& ⋮& ⋮& ⋮\\ {\displaystyle \frac{\mathrm{\partial }{Y}_{np}}{\mathrm{\partial }{B}_{11}}}& {\displaystyle \frac{\mathrm{\partial }{Y}_{np}}{\mathrm{\partial }{B}_{12}}}& \cdots & {\displaystyle \frac{\mathrm{\partial }{Y}_{np}}{\mathrm{\partial }{B}_{mq}}}\end{array}\right).$$
• For any matrix $A\in {\mathbb{R}}^{n\times m},B\in {\mathbb{R}}^{m\times q},C\in {\mathbb{R}}^{q\times n},$ the derivative of function $\mathrm{tr}(\cdot )$ is defined as

  $$\frac{\mathrm{\partial }\mathrm{tr}(ABC)}{\mathrm{\partial }B}=A^{\mathrm{T}}{C}^{\mathrm{T}}.$$

2.2. Two important optimization problems

Here, we present two results which play an important role in the algorithm.

2.2.1. Soft-thresholding

The minimization problem

$$\underset{x\in {\mathbb{R}}^p}{min}\{\Vert x{\Vert }_1+\frac{\beta }{2}\Vert x-r{\Vert }^2\},$$

with $\beta >0$, and $r\in {\mathbb{R}}^p$ has a closed-form solution, which is given by the soft-thresholding operator

$$x^{\ast }=\mathrm{shrink}(r,1/\beta ):=\mathrm{sign}(r)\ast max\{0,|r|-1/\beta \},$$ (3)

where $\mathrm{sign}(\cdot )$ is the sign function.

2.2.2. Matrix soft-thresholding

The minimization problem

$$\underset{B\in {\mathbb{R}}^{m\times q}}{min}\{\Vert B{\Vert }_{\ast }+\frac{\beta }{2}\Vert \mathrm{vec}(B)-r{\Vert }^2\},$$

with $\beta >0$, and $r\in {\mathbb{R}}^{mq}$ has a closed-form solution, which is defined as

$$B=U\cdot \mathrm{shrink}(\lambda ,1/\beta )\cdot V^{\mathrm{T}},$$ (4)

where $U,V,\lambda$ satisfy the singular value decomposition (SVD) of R, i.e. $R=U\mathrm{\Lambda }{V}^{\mathrm{T}}$ and λ is the diagonal of Λ. We arrange r to an $m\times q$ column-wise matrix, and denote it by R. The proof can be found in [2] (Theorem 2.1) or [19] (Theorem 3).

3. Linearized alternating direction method of multipliers

In this section, we will propose the LADMM algorithm to solve LRFL matrix regression model (2). For completeness, we firstly explain the ADMM algorithm in the following. Considering a convex minimization problem

$$\begin{array}{ll}min& F(\beta )+G(\gamma )\\ \mathrm{s}.\mathrm{t}.& X\beta +Y\gamma =b,\\ & \beta \in \mathrm{\Pi },\gamma \in \mathrm{\Gamma }.\end{array}$$ (5)

The objective function (5) is separable and the constraints are linear. The $\mathrm{\Pi }\subseteq {\mathbb{R}}^n$ and $\mathrm{\Gamma }\subseteq {\mathbb{R}}^m$ are given non-empty, closed and convex sets, and F, G are closed convex functions. The $X\in {\mathbb{R}}^{l\times n}$ and $Y\in {\mathbb{R}}^{l\times m}$ are given matrices, and $b\in \mathbb{R}$ is a given vector. Throughout, we assume that the solution set of (5) denoted by ${\mathrm{\Omega }}^{\ast }$ is non-empty. We get the augmented Lagrangian function of (5),

$$L_{\mu }(\beta ,\gamma ,\alpha ):=F(\beta )+G(\gamma )-{\alpha }^{\mathrm{T}}(X\beta +Y\gamma -b)+\frac{\mu }{2}\Vert X\beta +Y\gamma -b{\Vert }_2^2,$$

where $\mu >0$ is a penalty parameter, $\alpha \in \mathbb{R}$ is Lagrange multiplier. The augmented Lagrangian method (ALM) in Hestenes [10] and Powell [16] can be applied to solve (5). With given initial point $({\beta }^0,{\gamma }^0,{\alpha }^0)$, the iterative scheme of ALM for (5) is

$$\begin{array}{rl}({\beta }^{k+1},{\gamma }^{k+1})& =\mathrm{argmin}\{L_{\mu }(\beta ,\gamma ,{\alpha }^k)|\beta \in \mathrm{\Pi },\gamma \in \mathrm{\Gamma }\},\\ {\alpha }^{k+1}& ={\alpha }^k-\mu (X{\beta }^{k+1}+Y{\gamma }^{k+1}-b).\end{array}$$ (6)

The direct application of the ALM to (5) is the scheme (6). At each iteration, it requires to minimize β and γ simultaneously. The ADMM algorithm in Gabay and Mercier [8] and Glowinski and Marrocco [9] decomposes the minimization problem (6) into two separable problems and minimizes them iteratively

$$\begin{array}{rl}& {\beta }^{k+1}=\mathrm{argmin}\{L_{\mu }(\beta ,{\gamma }^k,{\alpha }^k)|\beta \in \mathrm{\Pi }\},\\ & {\gamma }^{k+1}=\mathrm{argmin}\{L_{\mu }({\beta }^{k+1},\gamma ,{\alpha }^k)|\gamma \in \mathrm{\Gamma }\},\\ & {\alpha }^{k+1}={\alpha }^k-\mu (X{\beta }^{k+1}+Y{\gamma }^{k+1}-b).\end{array}$$ (7)

Because the constraint optimization problem (5) is transformed into an unconstrained optimization problem (7), the subproblems in (7) are easier than the original problem (5). Moreover, the resulting subproblems in (7) have closed-formed solutions for many applications including the LASSO and GLASSO. This fact particularly makes the application of ADMM efficient for a wide class of problems. So we consider applying the ADMM algorithm to the LRFL matrix regression model (2).

Now we analyse how to solve the LRFL matrix regression model (2) by applying the ADMM. In order to reformulate the model (2), we define a matrix $A\in {\mathbb{R}}^{(p-1)\times p}$ as

$$A=\left(\begin{array}{ccccc}-1& 1& 0& \cdots & 0\\ 0& -1& 1& \cdots & 0\\ ⋮& ⋮& \ddots & \ddots & ⋮\\ 0& 0& \cdots & -1& 1\end{array}\right).$$

In fact, we denote $y=(y_1,y_2,\dots ,y_n{)}^{\mathrm{T}},X=(\mathrm{vec}(X_1),\mathrm{vec}(X_2),\dots ,\mathrm{vec}(X_n){)}^{\mathrm{T}},Z=(Z_1,Z_2,\dots ,Z_n{)}^{\mathrm{T}}$, and the model (2) can be written as

$$\underset{\genfrac{}{}{0ex}{}{B\in {\mathbb{R}}^{m\times q}}{\gamma \in {\mathbb{R}}^p}}{min}\{\frac{1}{2}\Vert y-Z\gamma -X\mathrm{vec}(B){\Vert }_2^2+{\lambda }_1\Vert B{\Vert }_{\ast }+{\lambda }_2\Vert \gamma {\Vert }_1+{\lambda }_3\Vert A\gamma {\Vert }_1\}.$$ (8)

Letting $\xi =A\gamma \in {\mathbb{R}}^{p-1}$, (8) can be rewritten as

$$\underset{\genfrac{}{}{0ex}{}{B\in {\mathbb{R}}^{m\times q}}{\genfrac{}{}{0ex}{}{\gamma \in {\mathbb{R}}^p}{\xi \in {\mathbb{R}}^{p-1}}}}{min}\{\frac{1}{2}{\Vert y-Z\gamma -X\mathrm{vec}(B)\Vert }_2^2+{\lambda }_1\Vert B{\Vert }_{\ast }+{\lambda }_2\Vert \gamma {\Vert }_1+{\lambda }_3\Vert \xi {\Vert }_1\}.\mathrm{s}.\mathrm{t}.A\gamma =\xi .$$ (9)

The augmented Lagrangian function of (9) is

$$\begin{array}{rl}{L}_{\mu }(B,\gamma ,\xi ,\alpha )& :=\frac{1}{2}{\Vert y-Z\gamma -X\mathrm{vec}(B)\Vert }_2^2+{\lambda }_1\Vert B{\Vert }_{\ast }+{\lambda }_2\Vert \gamma {\Vert }_1+{\lambda }_3\Vert \xi {\Vert }_1-{\alpha }^{\mathrm{T}}(A\gamma -\xi )\\ & +\frac{\mu }{2}\Vert A\gamma -\xi {\Vert }_2^2,\end{array}$$ (10)

where $\alpha \in {\mathbb{R}}^{p-1}$ is the Lagrange multiplier, and μ is a given penalty parameter. The iterative scheme of ADMM for (10) is

$$\begin{array}{rl}& B^{k+1}=\underset{B\in {\mathbb{R}}^{m\times q}}{\mathrm{argmin}}{L}_{\mu }(B,{\gamma }^k,{\xi }^k,{\alpha }^k),\\ & {\gamma }^{k+1}=\underset{\gamma \in {\mathbb{R}}^p}{\mathrm{argmin}}{L}_{\mu }(B^{k+1},\gamma ,{\xi }^k,{\alpha }^k),\\ & {\xi }^{k+1}=\underset{\xi \in {\mathbb{R}}^{p-1}}{\mathrm{argmin}}{L}_{\mu }(B^{k+1},{\gamma }^{k+1},\xi ,{\alpha }^k),\\ & {\alpha }^{k+1}={\alpha }^k-\mu (A{\gamma }^{k+1}-{\xi }^{k+1}).\end{array}$$ (11)

Now let us look at the resulting subproblems in (11). First, for the B-subproblem in (11), after trivial manipulation, it can be written as

$$\begin{array}{rl}{B}^{k+1}& =\underset{B\in {\mathbb{R}}^{m\times q}}{\mathrm{argmin}}\{\frac{1}{2}\Vert y-Z{\gamma }^k-X\mathrm{vec}(B){\Vert }_2^2+{\lambda }_1\Vert B{\Vert }_{\ast }\}\\ & =\underset{B\in {\mathbb{R}}^{m\times q}}{\mathrm{argmin}}\{{\lambda }_1\Vert B{\Vert }_{\ast }+\frac{1}{2}\Vert X\mathrm{vec}(B)-{\hat{y}}_k{\Vert }^2\},\end{array}$$ (12)

where ${\hat{y}}_k=y-Z{\gamma }^k\in {\mathbb{R}}^n$. The subproblem does not have a closed-form solution because of the non-identity matrix X. As Wang Yuan [19], we can linearize the quadratic term $\frac{1}{2}\Vert X\mathrm{vec}(B)-{\hat{y}}_k{\Vert }^2$ in (12), which is replaced by

$${\left(X^{\mathrm{T}}(X\mathrm{vec}(B^k)-{\hat{y}}_k)\right)}^{\mathrm{T}}(\mathrm{vec}(B)-\mathrm{vec}(B^k))+\frac{v}{2}\Vert \mathrm{vec}(B)-\mathrm{vec}(B^k){\Vert }_2^2,$$

where the parameter v>0 controls the proximity to $B^k$. Overall, we solve the following subproblem:

$$\begin{array}{rl}{B}^{k+1}& =\underset{B\in {\mathbb{R}}^{m\times q}}{\mathrm{argmin}}\{{\lambda }_1\Vert B{\Vert }_{\ast }+{\left(X^{\mathrm{T}}(X\mathrm{vec}(B^k)-{\hat{y}}_k)\right)}^{\mathrm{T}}(\mathrm{vec}(B)-\mathrm{vec}(B^k))\\ & \phantom{{\left(X^{\mathrm{T}}(X\mathrm{vec}(B^k)-{\hat{y}}_k)\right)}^{\mathrm{T}}}+\frac{v}{2}\Vert \mathrm{vec}(B)-\mathrm{vec}(B^k){\Vert }_2^2\}\\ & =\underset{B\in {\mathbb{R}}^{m\times q}}{\mathrm{argmin}}\{{\lambda }_1\Vert B{\Vert }_{\ast }+\frac{v}{2}\Vert \mathrm{vec}(B)-\mathrm{vec}(B^k)+X^{\mathrm{T}}(X\mathrm{vec}(B^k)-{\hat{y}}_k)/v{\Vert }_2^2\}\\ & =\underset{B\in {\mathbb{R}}^{m\times q}}{\mathrm{argmin}}\{{\lambda }_1\Vert B{\Vert }_{\ast }+\frac{v}{2}⟨B-C_1,B-C_1⟩\}\\ & =\underset{B\in {\mathbb{R}}^{m\times q}}{\mathrm{argmin}}\{{\lambda }_1\Vert B{\Vert }_{\ast }+\frac{v}{2}\Vert B-C_1{\Vert }_{\mathrm{F}}^2\},\end{array}$$ (13)

where $C_1$ is a matrix with $\mathrm{vec}(C_1)=C$, and $C=\mathrm{vec}(B^k)-X^{\mathrm{T}}(X\mathrm{vec}(B^k)-{\hat{y}}_k)/v$. Then, following (4), the closed-form solution of (13) is

$$B^{k+1}=U\cdot \mathrm{shrink}(\lambda ,{\lambda }_1/v)\cdot V^{\mathrm{T}},$$ (14)

where $U,V,\lambda$ satisfy the singular value decomposition (SVD) of $C_1$, i.e. $C_1=U\mathrm{\Lambda }{V}^{\mathrm{T}}$ and λ is the diagonal of Λ.

Second, for the γ-subproblem in (11), we will briefly display the solution of subproblem γ as follows:

$$\begin{array}{rl}{\gamma }^{k+1}& =\underset{\gamma \in {\mathbb{R}}^p}{\mathrm{argmin}}\{\frac{1}{2}\Vert y-Z\gamma -X\mathrm{vec}(B^{k+1}){\Vert }_2^2+{\lambda }_2\Vert \gamma {\Vert }_1+\frac{\mu }{2}\Vert A\gamma -{\xi }^k-{\alpha }^k/2{\Vert }^2\}\\ & =\underset{\gamma \in {\mathbb{R}}^p}{\mathrm{argmin}}\{{\lambda }_2\Vert \gamma {\Vert }_1+\frac{1}{2}\Vert \tilde{X}\gamma -{\tilde{y}}_k{\Vert }^2\},\end{array}$$

where $\tilde{X}=(Z^{\mathrm{T}},\sqrt{\mu }{A}^{\mathrm{T}}{)}^{\mathrm{T}},{\tilde{y}}_k=((y-X\mathrm{vec}(B^{k+1}){)}^{\mathrm{T}},\sqrt{\mu }(I_{p-1}{\xi }^k+{\alpha }^k/\mu {)}^{\mathrm{T}}{)}^{\mathrm{T}}$. The subproblem does not have a closed-form solution because of the non-identity matrix $\tilde{X}$. Similarly, we can linearize $\frac{1}{2}\Vert \tilde{X}\gamma -{\tilde{y}}_k{\Vert }^2$ the quadratic term, which is replaced by

$${\tilde{X}}^{\mathrm{T}}(\tilde{X}{\gamma }^k-{\tilde{y}}_k{)}^{\mathrm{T}}(\gamma -{\gamma }^k)+\frac{v}{2}\Vert \gamma -{\gamma }^k{\Vert }_2^2,$$

where the parameter v>0 controls the proximity to ${\gamma }^k$. Overall, we solve the following subproblem:

$$\begin{array}{rl}{\gamma }^{k+1}& =\underset{\gamma \in {\mathbb{R}}^p}{\mathrm{argmin}}\{{\lambda }_2\Vert \gamma {\Vert }_1+({\tilde{X}}^{\mathrm{T}}(\tilde{X}{\gamma }^k-{\tilde{y}}_k{)}^{\mathrm{T}}(\gamma -{\gamma }^k)+\frac{v}{2}\Vert \gamma -{\gamma }^k{\Vert }_2^2\}\\ & =\underset{\gamma \in {\mathbb{R}}^p}{\mathrm{argmin}}\{{\lambda }_2\Vert \gamma {\Vert }_1+\frac{v}{2}\Vert \gamma -{\gamma }^k+{\tilde{X}}^{\mathrm{T}}(\tilde{X}{\gamma }^k-{\tilde{y}}_k)/v{\Vert }_2^2\}.\end{array}$$

Then, according to (3), the closed-form solution of (11) is

$${\gamma }^{k+1}=\mathrm{shrink}({\gamma }^k-{\tilde{X}}^{\mathrm{T}}(\tilde{X}{\gamma }^k-{\tilde{y}}_k)/v,{\lambda }_2/v).$$ (15)

Third, for the ξ-subproblem in (11), its closed-form solution can be obtained directly by determining the derivative of the augmented Lagrangian function, which is

$$\begin{array}{rl}{\xi }^{k+1}& =\underset{\xi \in {\mathbb{R}}^{p-1}}{\mathrm{argmin}}\{{\lambda }_3\Vert \xi {\Vert }_1-{{\alpha }^k}^{\mathrm{T}}(A{\gamma }^{k+1}-\xi )+\frac{\mu }{2}\Vert A{\gamma }^{k+1}-\xi {\Vert }_2^2\}\\ & =\underset{\xi \in {\mathbb{R}}^{p-1}}{\mathrm{argmin}}\{\frac{\mu }{2}\Vert \xi -A{\gamma }^{k+1}+{\alpha }^k/\mu {\Vert }_2^2+{\lambda }_3\Vert \xi {\Vert }_1\}.\end{array}$$

We obtain the solution by (3),

$${\xi }^{k+1}=\mathrm{shrink}(A{\gamma }^{k+1}-{\alpha }^k/\mu ,{\lambda }_3/\mu ).$$ (16)

In summary, the iterative scheme of LADMM algorithm for LRFL matrix regression model can be described as follows.

Remark 3.1

When solving B-subproblem, the rank of $B^{k+1}$ is determined by ${\lambda }_1/v$. If ${\lambda }_1/v$ increases, the rank of $B^{k+1}$ will decrease; if ${\lambda }_1/v$ gets smaller, the rank of $B^{k+1}$ will get larger. Thus, if we want to obtain a low-rank estimator, we only need to choose a large ${\lambda }_1/v$. On the other hand, for a given ${\lambda }_1/v$, we do not need to compute all the singular values of $C_1$. From the solution in (14), the singular values which are greater than ${\lambda }_1/v$ make sense. Thus in the implement of Algorithm 1, we use the truncation technique which can be found in [15].

4. Convergence analysis

In this section, we will focus on the convergence analysis of Algorithm 1. In fact, this procedure is similar to some existing work, such as [12]. In order to better understand, we give a succinct proof here.

4.1. Convergence of Algorithm 1

Note that, the Lagrange function of (9) is

$$\frac{1}{2}\Vert y-Z\gamma -X\mathrm{vec}(B){\Vert }_2^2+{\lambda }_1\Vert B{\Vert }_{\ast }+{\lambda }_2\Vert \gamma {\Vert }_1+{\lambda }_3\Vert \xi {\Vert }_1-{\alpha }^{\mathrm{T}}(A\gamma -\xi ),$$ (17)

where $\alpha \in {\mathbb{R}}^{p-1}$ is the Lagrange multiplier. By the first-order optimality condition of (17), it is easy to see that solving (9) is equivalent to find $(B^{\ast },{\gamma }^{\ast },{\xi }^{\ast },{\alpha }^{\ast })\in \mathrm{\Omega }:={\mathbb{R}}^{m\times q}\times {\mathbb{R}}^p\times {\mathbb{R}}^{p-1}\times {\mathbb{R}}^{p-1},f(B^{\ast })\in \frac{\mathrm{\partial }\Vert B^{\ast }{\Vert }_{\ast }}{\mathrm{\partial }\mathrm{vec}(B^{\ast })},g({\gamma }^{\ast })\in \mathrm{\partial }(\Vert {\gamma }^{\ast }{\Vert }_1)$, and $h({\xi }^{\ast })\in \mathrm{\partial }(\Vert {\xi }^{\ast }{\Vert }_1)$ such that

$$\begin{array}{rl}0& ={\lambda }_{1}f(B^{\ast })-X^{\mathrm{T}}(y-Z{\gamma }^{\ast }-X\mathrm{vec}(B^{\ast })),\\ 0& ={\lambda }_{2}g({\gamma }^{\ast })-Z^{\mathrm{T}}(y-Z{\gamma }^{\ast }-X\mathrm{vec}(B^{\ast }))-A^{\mathrm{T}}{\alpha }^{\ast },\\ 0& ={\lambda }_{3}h({\xi }^{\ast })+{\alpha }^{\ast },\\ 0& =A{\gamma }^{\ast }-{\xi }^{\ast }.\end{array}$$ (18)

Note that $\mathrm{\partial }(\cdot )$ denotes the subdifferential operator of a non-smooth convex function. We denote all the elements in Ω that satisfy (18) by ${\mathrm{\Omega }}^{\ast }$. Then, using the notation ${\omega }^{\ast }=((\mathrm{vec}(B^{\ast }){)}^{\mathrm{T}},{{\gamma }^{\ast }}^{\mathrm{T}},{{\xi }^{\ast }}^{\mathrm{T}},{{\alpha }^{\ast }}^{\mathrm{T}}{)}^{\mathrm{T}},f(B)\in \frac{\mathrm{\partial }\Vert B{\Vert }_{\ast }}{\mathrm{\partial }\mathrm{vec}(B)},g(\gamma )\in \mathrm{\partial }(\Vert \gamma {\Vert }_1)$, and $h(\xi )\in \mathrm{\partial }(\Vert \xi {\Vert }_1)$, (18) can be written as a variational inequality (VI) problem: finding a ${\omega }^{\ast }\in {\mathrm{\Omega }}^{\ast },f(B^{\ast })\in \frac{\mathrm{\partial }\Vert B^{\ast }{\Vert }_{\ast }}{\mathrm{\partial }\mathrm{vec}(B^{\ast })},g({\gamma }^{\ast })\in \mathrm{\partial }(\Vert {\gamma }^{\ast }{\Vert }_1)$, and $h({\xi }^{\ast })\in \mathrm{\partial }(\Vert {\xi }^{\ast }{\Vert }_1)$ such that

$${(\omega -{\omega }^{\ast })}^{\mathrm{T}}F({\omega }^{\ast })\ge 0\mathrm{\forall }\omega \in \mathrm{\Omega },$$ (19)

where $\omega =(\mathrm{vec}(B{)}^{\mathrm{T}},{\gamma }^{\mathrm{T}},{\xi }^{\mathrm{T}},{\alpha }^{\mathrm{T}}{)}^{\mathrm{T}}$ and

$$F(\omega )=\left(\begin{array}{c}{\lambda }_{1}f(B)-X^{\mathrm{T}}(y-Z\gamma -X\mathrm{vec}(B))\\ {\lambda }_{2}g(\gamma )-Z^{\mathrm{T}}(y-Z\gamma -X\mathrm{vec}(B))-A^{\mathrm{T}}\alpha \\ {\lambda }_{3}h(\xi )+\alpha \\ A\gamma -\xi \end{array}\right).$$

For purpose of expressing concisely, we need to use the following matrix:

$$G=\left(\begin{array}{cccc}v{I}_{mq}-X^{\mathrm{T}}X& 0& 0& 0\\ 0& v{I}_p-{\tilde{X}}^{\mathrm{T}}\tilde{X}& 0& 0\\ 0& 0& \mu I_{p-1}& 0\\ 0& 0& 0& {\displaystyle \frac{1}{\mu }}{I}_{p-1}\end{array}\right).$$

From the procedure of proof, the matrix G must be positive definite. Considering $\tilde{X}=(Z^{\mathrm{T}},\sqrt{\mu }{A}^{\mathrm{T}}{)}^{\mathrm{T}}$, the positive definiteness of G can be achieved by the condition $v>\rho (X^{\mathrm{T}}X)$ and $v>\rho (Z^{\mathrm{T}}Z+\mu A^{\mathrm{T}}A)$, where $\rho (\cdot )$ denotes the spectral radius of matrix. In order to establish the convergence of LADMM algorithm, in the following lemma we characterize the $(k+1)$th iteration of Algorithm 1 as a VI problem.

Lemma 4.1

Let $\{{\omega }^k\}$ be a sequence generated by Algorithm 1. Then we have

$${({\omega }^{\prime }-{\omega }^{k+1})}^{\mathrm{T}}(F({\omega }^{k+1})+M({\xi }^k-{\xi }^{k+1})-G({\omega }^k-{\omega }^{k+1}))\ge 0\mathrm{\forall }{\omega }^{\prime }\in \mathrm{\Omega },$$

where

$$M=\left(\begin{array}{c}{O}_{mq\times (p-1)}\\ -\mu A^{\mathrm{T}}\\ \mu I_{p-1}\\ O_{p-1}\end{array}\right).$$

Proof.

First, we have

$$\begin{array}{rl}{X}^{\mathrm{T}}(X\mathrm{vec}(B^{k+1})-{\hat{y}}_k)& =X^{\mathrm{T}}(X\mathrm{vec}(B^{k+1})-(y-Z{\gamma }^k))\\ & =X^{\mathrm{T}}(X\mathrm{vec}(B^{k+1})-y)+X^{\mathrm{T}}Z{\gamma }^k\end{array}$$

and

$$\begin{array}{rl}{\tilde{X}}^{\mathrm{T}}(\tilde{X}{\gamma }^{k+1}-{\tilde{y}}_k)& =Z^{\mathrm{T}}Z{\gamma }^{k+1}-Z^{\mathrm{T}}y+Z^{\mathrm{T}}X\mathrm{vec}(B^{k+1})\\ & +\mu A^{\mathrm{T}}A{\gamma }^{k+1}-\mu A^{\mathrm{T}}{\xi }^k-A^{\mathrm{T}}{\alpha }^k\\ & =\left(Z^{\mathrm{T}}(Z{\gamma }^{k+1}-(y-X\mathrm{vec}(B^{k+1})))\right)\\ & +\mu A^{\mathrm{T}}A{\gamma }^{k+1}-\mu A^{\mathrm{T}}{\xi }^k-A^{\mathrm{T}}{\alpha }^k\\ & =\left(Z^{\mathrm{T}}(Z{\gamma }^{k+1}-(y-X\mathrm{vec}(B^{k+1})))\right)\\ & -A^{\mathrm{T}}{\alpha }^{k+1}-\mu A^{\mathrm{T}}({\xi }^k-{\xi }^{k+1}).\end{array}$$

It follows from (11) that

$${\alpha }^k={\alpha }^{k+1}+\mu (A{\gamma }^{k+1}-{\xi }^{k+1}).$$ (20)

Deriving the first-order optimality condition of the minimization problems (14), (15) and (16), we see that the iterative scheme (11) is equivalent to find ${\omega }^{k+1}=({\mathrm{vec}(B^{k+1})}^{\mathrm{T}},{{\gamma }^{k+1}}^{\mathrm{T}},{{\xi }^{k+1}}^{\mathrm{T}},{{\alpha }^{k+1}}^{\mathrm{T}}{)}^{\mathrm{T}}\in \mathrm{\Omega },f(B)\in \frac{\mathrm{\partial }\Vert B{\Vert }_{\ast }}{\mathrm{\partial }\mathrm{vec}(B)},g({\gamma }^{k+1})\in \mathrm{\partial }(\Vert \gamma {\Vert }_1)$, and $h({\xi }^{k+1})\in \mathrm{\partial }(\Vert \xi {\Vert }_1)$ such that

$$\begin{array}{rl}0& ={\lambda }_{1}f(B^{k+1})+X^{\mathrm{T}}(X\mathrm{vec}(B^k)-{\hat{y}}_k)+v(\mathrm{vec}(B^{k+1})-\mathrm{vec}(B^k)),\\ 0& ={\lambda }_{2}g({\gamma }^{k+1})+{\tilde{X}}^{\mathrm{T}}(\tilde{X}{\gamma }^k-{\tilde{y}}_k)+v({\gamma }^{k+1}-{\gamma }^k),\\ 0& ={\lambda }_{3}h({\xi }^{k+1})+{\alpha }^k-\mu (A{\gamma }^{k+1}-{\xi }^{k+1}),\\ 0& =A{\gamma }^{k+1}-{\xi }^{k+1}-({\alpha }^k-{\alpha }^{k+1})/\mu .\end{array}$$ (21)

According to the definition of G, inserting (20) into (21), the (21) can be written as

$$\begin{array}{rl}0& =({\lambda }_{1}f(B^{k+1})-X^{\mathrm{T}}(y-Z{\gamma }^k-X\mathrm{vec}(B^{k+1})))\\ & +(v{I}_{mq}-X^{\mathrm{T}}X)(\mathrm{vec}(B^{k+1})-\mathrm{vec}(B^k)),\\ 0& =({\lambda }_{2}g({\gamma }^{k+1})-Z^{\mathrm{T}}(y-Z{\gamma }^{k+1}-X\mathrm{vec}(B^{k+1}))-A^{\mathrm{T}}{\alpha }^{k+1})\\ & +(v{I}_p-{\tilde{X}}^{\mathrm{T}}\tilde{X})({\gamma }^{k+1}-{\gamma }^k)-\mu A^{\mathrm{T}}({\xi }^k-{\xi }^{k+1}),\\ 0& ={\lambda }_{3}h({\xi }^{k+1})+{\alpha }^k-\mu (A{\gamma }^{k+1}-{\xi }^{k+1}),\\ 0& =(A{\gamma }^{k+1}-{\xi }^{k+1})-({\alpha }^k-{\alpha }^{k+1})/\mu .\end{array}$$

Then we obtain the conclusion.

The following lemma can be easily derived by Lemma 4.1. For completeness, we show the proof in detail.

Lemma 4.2

Let $\{{\omega }^k\}$ be a sequence generated by Algorithm 1. Then $\mathrm{\forall }{\omega }^{\ast }\in {\mathrm{\Omega }}^{\ast },$ we have

$$\begin{array}{rl}{({\omega }^k-{\omega }^{\ast })}^{\mathrm{T}}G({\omega }^k-{\omega }^{k+1})& \ge {({\omega }^k-{\omega }^{k+1})}^{\mathrm{T}}G({\omega }^k-{\omega }^{k+1})\\ & -{({\alpha }^k-{\alpha }^{k+1})}^{\mathrm{T}}({\xi }^k-{\xi }^{k+1}).\end{array}$$

Proof.

From Lemma 4.1, for any ${\omega }^{\prime }\in \mathrm{\Omega }$, by setting ${\omega }^{\prime }={\omega }^{\ast }$, we obtain

$${({\omega }^{\ast }-{\omega }^{k+1})}^{\mathrm{T}}(F({\omega }^{k+1})+M({\xi }^k-{\xi }^{k+1})-G({\omega }^k-{\omega }^{k+1}))\ge 0,$$ (22)

where ${\omega }^{\ast }$ is an arbitrary solution point in ${\mathrm{\Omega }}^{\ast }$. Note that we have $A{\gamma }^{\ast }={\xi }^{\ast }$ for ${\omega }^{\ast }\in {\mathrm{\Omega }}^{\ast }$, thus (22) leads to

$$\begin{array}{rl}{({\omega }^{k+1}-{\omega }^{\ast })}^{\mathrm{T}}G({\omega }^k-{\omega }^{k+1})& \ge {({\omega }^{k+1}-{\omega }^{\ast })}^{\mathrm{T}}F({\omega }^{k+1})\\ & -\mu {(A{\gamma }^{k+1}-{\xi }^{k+1})}^{\mathrm{T}}({\xi }^k-{\xi }^{k+1}).\end{array}$$

Since $({\alpha }^k-{\alpha }^{k+1})=\mu (A{\gamma }^{k+1}-{\xi }^{k+1})$, the above inequality becomes

$$\begin{array}{rl}{({\omega }^{k+1}-{\omega }^{\ast })}^{\mathrm{T}}G({\omega }^k-{\omega }^{k+1})& \ge {({\omega }^{k+1}-{\omega }^{\ast })}^{\mathrm{T}}F({\omega }^{k+1})\\ & -{({\alpha }^k-{\alpha }^{k+1})}^{\mathrm{T}}({\xi }^k-{\xi }^{k+1}).\end{array}$$

On the other hand, since $\Vert \cdot {\Vert }_1$ and $\Vert \cdot {\Vert }_2$ are both convex, it is obvious that the mapping $F(\omega )$ is monotone. We thus have

$${({\omega }^{k+1}-{\omega }^{\ast })}^{\mathrm{T}}(F({\omega }^{k+1})-F({\omega }^{\ast }))\ge 0$$ (23)

and

$${({\omega }^{k+1}-{\omega }^{\ast })}^{\mathrm{T}}F({\omega }^{k+1})\ge {({\omega }^{k+1}-{\omega }^{\ast })}^{\mathrm{T}}F({\omega }^{\ast })\ge 0.$$ (24)

Then, replacing ${\omega }^{k+1}-{\omega }^{\ast }$ by $({\omega }^{k+1}-{\omega }^k)+({\omega }^k-{\omega }^{\ast })$ in (23) and using (24), we get the desired conclusion.

Using Lemma 4.2, we can obtain an upper bound of the difference between the sequence $\{{\omega }^k\}$ generated by Algorithm 1 and true solution.

Lemma 4.3

Let $\{{\omega }^k\}$ be the sequence generated by Algorithm 1. Then we have

$$\Vert {\omega }^{k+1}-{\omega }^{\ast }{\Vert }_G^2\le \Vert {\omega }^k-{\omega }^{\ast }{\Vert }_G^2-\Vert {\omega }^k-{\omega }^{k+1}{\Vert }_G^2\mathrm{\forall }{\omega }^{\ast }\in {\mathrm{\Omega }}^{\ast }.$$ (25)

Proof.

For $\mathrm{\forall }{\omega }^{\ast }\in {\mathrm{\Omega }}^{\ast }$, it follows from Lemma 4.2 that

$$\begin{array}{rl}\Vert {\omega }^{k+1}-{\omega }^{\ast }{\Vert }_G^2& =\Vert {\omega }^k-{\omega }^{\ast }{\Vert }_G^2+\Vert {\omega }^k-{\omega }^{k+1}{\Vert }_G^2-2{({\omega }^k-{\omega }^{\ast })}^{\mathrm{T}}G({\omega }^k-{\omega }^{k+1})\\ & \le \Vert {\omega }^k-{\omega }^{\ast }{\Vert }_G^2-\Vert {\omega }^k-{\omega }^{k+1}{\Vert }_G^2+2{({\alpha }^k-{\alpha }^{k+1})}^{\mathrm{T}}({\xi }^k-{\xi }^{k+1}).\end{array}$$ (26)

As shown before, we have ${\lambda }_{3}h({\xi }^k)+{\alpha }^k=0$ for any k. Thus, we have

$$\begin{array}{rl}& {({\xi }^{k+1}-{\xi }^k)}^{\mathrm{T}}({\lambda }_{3}h({\xi }^k)+{\alpha }^k)\ge 0,\\ & {({\xi }^k-{\xi }^{k+1})}^{\mathrm{T}}({\lambda }_{3}h({\xi }^{k+1})+{\alpha }^{k+1})\ge 0.\end{array}$$

We obtain that

$${({\alpha }^k-{\alpha }^{k+1})}^{\mathrm{T}}({\xi }^k-{\xi }^{k+1})\le {\lambda }_3{({\xi }^k-{\xi }^{k+1})}^{\mathrm{T}}(h({\xi }^{k+1})-h({\xi }^k))\le 0.$$ (27)

Inserting (27) into (26), we prove the assertion (25).

Lemma 4.3 implies that the sequence generated by Algorithm 1 is contractive with respect to the solution set ${\mathrm{\Omega }}^{\ast }$. The following corollary is trivial based on the inequality (25). Thus, we omit the proof.

Corollary 4.4

Let $\{{\omega }^k\}$ be the sequence generated by Algorithm 1. Then we have

1. $\underset{k\to \mathrm{\infty }}{lim}\Vert {\omega }^k-{\omega }^{k+1}{\Vert }_G=0$.

2. The sequence $\{{\omega }^k\}$ is bounded.

3. For any ${\omega }^{\ast }\in {\mathrm{\Omega }}^{\ast },$ the sequence $\Vert {\omega }^k-{\omega }^{\ast }{\Vert }_G$ is monotonically non-increasing.

Now we can obtain the convergence of the linearized alternating direction method of multipliers for the LRFL matrix regression model.

Theorem 4.5

For any $\mu >0,v>\rho (X^{\mathrm{T}}X)$ and $v>\rho (Z^{\mathrm{T}}Z+\mu A^{\mathrm{T}}A),$ the sequence $\{{\omega }^k=(\mathrm{vec}(B^k),{\gamma }^k,{\xi }^k,{\alpha }^k)\}$ is generated by Algorithm 1. Then $\{\mathrm{vec}(B^k),{\gamma }^k,{\xi }^k,{\alpha }^k\}$ converges to a point ${\omega }^{\mathrm{\infty }}=(\mathrm{vec}(B^{\mathrm{\infty }}),{\gamma }^{\mathrm{\infty }},{\xi }^{\mathrm{\infty }},{\alpha }^{\mathrm{\infty }}),$ where $(\mathrm{vec}(B^{\mathrm{\infty }}),{\gamma }^{\mathrm{\infty }},{\xi }^{\mathrm{\infty }})$ is an optimal solution of the LRFL matrix regression model (9).

Proof.

The property (a) in Corollary 4.4 means that

$$\begin{array}{rl}& \underset{k\to \mathrm{\infty }}{lim}\Vert \mathrm{vec}(B^k)-\mathrm{vec}(B^{k+1})\Vert =0,\underset{k\to \mathrm{\infty }}{lim}\Vert {\gamma }^k-{\gamma }^{k+1}\Vert =0,\\ & \underset{k\to \mathrm{\infty }}{lim}\Vert {\xi }^k-{\xi }^{k+1}\Vert =0,\underset{k\to \mathrm{\infty }}{lim}\Vert {\alpha }^k-{\alpha }^{k+1}\Vert =0.\end{array}$$

In addition, the property (b) in Corollary 4.4 implies that the sequence $\{{\omega }^k\}$ has at least one cluster point. We denote it by ${\omega }^{\mathrm{\infty }}=(\mathrm{vec}(B^{\mathrm{\infty }}),{\gamma }^{\mathrm{\infty }},{\xi }^{\mathrm{\infty }},{\alpha }^{\mathrm{\infty }})$ and let $\{{\omega }^{k_j}\}$ be a subsequence converging to ${\omega }^{\mathrm{\infty }}$. Thus, we have

$$\mathrm{vec}(B^{k_j})\to \mathrm{vec}(B^{\mathrm{\infty }}),{\gamma }^{k_j}\to {\gamma }^{\mathrm{\infty }},{\xi }^{k_j}\to {\xi }^{\mathrm{\infty }},{\alpha }^{k_j}\to {\alpha }^{\mathrm{\infty }}$$ (28)

and

$$\begin{array}{rl}& \underset{k\to \mathrm{\infty }}{lim}\Vert \mathrm{vec}(B^{k_j})-\mathrm{vec}(B^{k_j+1})\Vert =0,\underset{k\to \mathrm{\infty }}{lim}\Vert {\gamma }^{k_j}-{\gamma }^{k_j+1}\Vert =0,\\ & \underset{k\to \mathrm{\infty }}{lim}\Vert {\xi }^{k_j}-{\xi }^{k_j+1}\Vert =0,\underset{k\to \mathrm{\infty }}{lim}\Vert {\alpha }^{k_j}-{\alpha }^{k_j+1}\Vert =0.\end{array}$$ (29)

Next we show that the cluster point ${\omega }^{\mathrm{\infty }}$ satisfies the optimality condition (18). Due to the variational inequality (19) and (29), we have

$$\underset{j\to \mathrm{\infty }}{lim}({\omega }^{\prime }-{\omega }^{k_j})F({\omega }^{k_j})\ge 0\mathrm{\forall }{\omega }^{\prime }\in \mathrm{\Omega }.$$

Then, according to (28), we obtain that

$$({\omega }^{\prime }-{\omega }^{\mathrm{\infty }})F({\omega }^{\mathrm{\infty }})\ge 0\mathrm{\forall }{\omega }^{\prime }\in \mathrm{\Omega }.$$

Thus, the limiting point ${\omega }^{\mathrm{\infty }}$ satisfies (19), i.e. ${\omega }^{\mathrm{\infty }}\in {\mathrm{\Omega }}^{\ast }$. Considering the property (c) in Corollary 4.4, we have

$${\Vert {\omega }^{k+1}-{\omega }^{\mathrm{\infty }}\Vert }_G\le {\Vert {\omega }^k-{\omega }^{\mathrm{\infty }}\Vert }_G\mathrm{\forall }k\ge 0.$$

Therefore, the sequence $\{{\omega }^k\}$ has the unique cluster point ${\omega }^{\mathrm{\infty }}$. Thus, $(\mathrm{vec}(B^{\mathrm{\infty }}),{\gamma }^{\mathrm{\infty }},{\xi }^{\mathrm{\infty }})$ is an optimal solution of the LRFL matrix regression model (9).

4.2. Convergence rate

Following the work in [24], we can easily establish a worst-case $O(1/k)$ convergence rate measured by the iteration complexity in the ergodic sense for the proposed Algorithm 1. That is, after k iterations, the average of all these k iterates generated by Algorithm 1 is an approximate solution of LRFL matrix regression model with an accuracy of $O(1/k)$. For succinctness, we omit the proof.

5. Numerical experiments

5.1. Simulation

We consider a class of matrix models with different ranks and sparsity levels used in [23]. Specifically, we generate the matrix covariates X of size $64\times 64$ and the vector covariates Z in ${\mathbb{R}}^{500}$, both of which consist of independent standard normal entries. We set the sample size at n = 500, where the number of parameters is $64\times 64+500=4596$. We set $\gamma =(1,\dots ,1{)}^{\mathrm{T}}\in {\mathbb{R}}^{500}$ and generate the true array signal as $B=B_1{B}_2^{\mathrm{T}}$, where $B_1\in {\mathbb{R}}^{m\times R},B_2\in {\mathbb{R}}^{q\times R}$, and R controls the rank of the signal. Moreover, each entry of B is 0 or 1, and the percentage of non-zero entries is controlled by a sparsity level constant s, i.e. each entry of $B_1,B_2$ is a Bernoulli distribution with probability of 1 equal to $\sqrt{(1-(1-s{)}^{1/R})}$. We vary the rank R = 1, 5 and the level of (non-)sparsity s = 0.01, 0.05, 0.1 ( s = 0.05 means that about $5\mathrm{\%}$ of entries of B are 1s and the rest are 0s). We generate response y with the systematic part as $y=⟨B^{\ast },X⟩+⟨{\gamma }^{\ast },Z⟩+\epsilon$ with ϵ satisfying a standard normal distribution. In addition, all computations are performed on an Intel Core(TM)i7-2640M CPU (2.80 GHz) and 8 GB RAM. The code for Algorithm 1 is written in MATLAB, and the initial point for this method is set to be $B^0=O_{m\times q},{\gamma }^0=O_p,{\xi }^0=O_{p-1},{\alpha }^0={\mathbf{1}}_{p-1}$. The maximum iteration number is set as 500. For the tuning parameters ${\lambda }_1,{\lambda }_2,{\lambda }_3$, we take a large grid of values. For each model, we choose the parameters which give the best performance of test root-mean-square error (RMSE), where the test set is generated as the above besides n = 2500. The other parameters are chosen as $\mu =(1+\sqrt{5})/2$ and $v=10max\{\rho (\Vert X^{\mathrm{T}}X{\Vert }_2),\rho (\Vert Z^{\mathrm{T}}Z+\mu A^{\mathrm{T}}A{\Vert }_2)\}.$

In the experiment, we simulate the model 100 times. Then we evaluate the performance of our method from two aspects: parameter estimation and prediction. For the former, we employ

$$max\{\frac{\Vert B^k-B^{k-1}{\Vert }_{\mathrm{F}}}{max\{\Vert B^k{\Vert }_{\mathrm{F}},1\}},\frac{\Vert {\gamma }^k-{\gamma }^{k-1}{\Vert }_2}{max\{\Vert {\gamma }^k{\Vert }_2,1\}}\}<{10}^{-4}$$

as the evaluation criterion. For the latter, we use independent validation data to evaluate the prediction error measured by the RMSE of the response. For simplicity, we use LRFL to stand for LRFL matrix regression model. We report the performance in Tables 1–5. Specifically, RMSE-B is the root-mean-squared error of B for training data, RMSE-γ is the root-mean-squared error of γ for training data, RMSE-PRE is the root-mean-squared error of prediction for test data, and the numerical values in parentheses are the standard deviations of the corresponding terms. The average CPU time (in seconds) is also included.

Table 1. Performance of Algorithm 1 when the true rank of coefficient matrix R = 1.

Sparsity $s\mathrm{\%}$	RMSE-B	RMSE-γ	RMSE-PRE	CPU
0.01	0.23(0.007)	0.05(0.025)	0.23(0.006)	1.48
0.05	0.30(0.006)	0.08(0.019)	0.31(0.016)	1.67
0.1	0.40(0.004)	0.07(0.030)	0.40(0.008)	1.58

Notes: All the measurements are the mean of the results after repeated 100 times. The numbers in parentheses are the corresponding standard errors.

Table 2. Performance of Algorithm 1 when the true rank of coefficient matrix R = 5.

Sparsity $s\mathrm{\%}$	RMSE-B	RMSE-γ	RMSE-PRE	CPU
0.01	0.20(0.008)	0.05(0.023)	0.20(0.014)	1.86
0.05	0.27(0.008)	0.09(0.018)	0.29(0.014)	1.54
0.1	0.40(0.005)	0.09(0.028)	0.40(0.013)	1.56

Notes: All the measurements are the mean of the results after repeated 100 times. The numbers in parentheses are the corresponding standard errors.

Table 3. Results of comparing LRFL with matrix LASSO when the true rank of coefficient matrix R = 2.

	R = 2
Sparsity $s\mathrm{\%}$	Method	RMSE-B	RMSE-γ	RMSE-PRE	Rank
0.01	LRFL	0.38(0.055)	0.54(0.023)	0.66(0.017)	1
	matrix LASSO	0.33(0.042)	0.71(0.018)	0.78(0.037)	24.5
0.05	LRFL	0.46(0.027)	0.54(0.015)	0.70(0.018)	1
	matrix LASSO	0.38(0.019)	0.71(0.011)	0.81(0.018)	24
0.1	LRFL	0.45(0.042)	0.55(0.024)	0.72(0.031)	1
	matrix LASSO	0.38(0.029)	0.71(0.018)	0.81(0.030)	22
0.2	LRFL	0.59(0.036)	0.55(0.017)	0.80(0.037)	1
	matrix LASSO	0.54(0.017)	0.72(0.086)	0.90(0.014)	25
0.5	LRFL	0.62(0.056)	0.57(0.021)	0.84(0.037)	1
	matrix LASSO	0.95(0.020)	0.74(0.019)	1.22(0.039)	27

Notes: All the measurements are the mean of the results after repeated 100 times. The numbers in parentheses are the corresponding standard errors.

Table 4. Results of comparing LRFL with matrix LASSO when the true rank of coefficient matrix R = 5.

	R = 5
Sparsity $s\mathrm{\%}$	Method	RMSE-B	RMSE-γ	RMSE-PRE	Rank
0.01	LRFL	0.29(0.025)	0.67(0.012)	0.75(0.032)	6.5
	matrix LASSO	0.31(0.069)	0.79(0.023)	0.85(0.028)	10.5
0.05	LRFL	0.48(0.046)	0.55(0.022)	0.72(0.028)	6
	matrix LASSO	0.43(0.028)	0.72(0.010)	0.84(0.027)	25
0.1	LRFL	0.55(0.025)	0.53(0.013)	0.74(0.023)	6
	matrix LASSO	0.44(0.031)	0.71(0.012)	0.84(0.023)	23.5
0.2	LRFL	0.66(0.018)	0.55(0.023)	0.84(0.039)	7
	matrix LASSO	0.61(0.024)	0.72(0.017)	0.94(0.023)	26
0.5	LRFL	0.80(0.060)	0.56(0.023)	0.98(0.053)	6
	matrix LASSO	0.90(0.028)	0.73(0.015)	1.15(0.046)	27.5

Notes: All the measurements are the mean of the results after repeated 100 times. The numbers in parentheses are the corresponding standard errors.

Table 5. Results of comparing LRFL with matrix LASSO when the true rank of coefficient matrix R = 10.

	R = 10
Sparsity $s\mathrm{\%}$	Method	RMSE-B	RMSE-γ	RMSE-PRE	Rank
0.01	LRFL	0.43(0.043)	0.55(0.043)	0.70(0.039)	11
	matrix LASSO	0.31(0.057)	0.71(0.011)	0.78(0.038)	24
0.05	LRFL	0.48(0.026)	0.53(0.019)	0.72(0.023)	15
	matrix LASSO	0.34(0.032)	0.71(0.012)	0.78(0.017)	23
0.1	LRFL	0.54(0.015)	0.53(0.012)	0.76(0.027)	15.5
	matrix LASSO	0.46(0.028)	0.72(0.012)	0.86(0.035)	24
0.2	LRFL	0.72(0.014)	0.55(0.018)	0.91(0.020)	16.5
	matrix LASSO	0.68(0.015)	0.73(0.016)	1.01(0.026)	26.5
0.5	LRFL	0.88(0.072)	0.59(0.042)	1.07(0.068)	15
	matrix LASSO	1.05(0.024)	0.74(0.015)	1.29(0.033)	28

Notes: All the measurements are the mean of the results after repeated 100 times. The numbers in parentheses are the corresponding standard errors.

From Tables 1 and 2 and Figure 1, we can find that the accuracy of $B,\gamma$ and test prediction are becoming worse with the increase of the sparsity. The standard deviation, however, is almost not changed. This result suggests that our method is convergent. Compared the accuracy of B with $\gamma ,\gamma$ is better than B.

Figure 1.

The boxplot of RMSE-B, RMSE-γ, RMSE-PRE, where $(R,s),R\in \{1,5\}$ is the rank of $B^{\ast },s\in \{0.01,0.05,0.1\}$ is the sparsity level of $B^{\ast }$.

In Tables 3–5, we compared LRFL with matrix LASSO in the accuracy and estimation rank of B. The ranks listed in Tables 3–5 are the median of the 100 simulation results. For the matrix LASSO method, we stack up the elements of the predictor X by column. Then, the matrix model is reformed to vector model. Thus, we can use matrix LASSO to solve it. From Tables 3–5, for the accuracy of B, LASSO gives a better performance. However, LRFL shows a better rank estimation. There is no doubt that vectorization destroys the structure of X. This result in the coefficient matrix B is not low-rank.

5.2. Real data

In this subsection, we will use our algorithm to deal with two real datasets.

5.2.1. Signal shapes

Recently, signal shapes have attracted wide attention of researchers [11,23]. In LRFL matrix regression model (2), X is a $64\times 64$ matrix with entries generated as independent standard normal distribution, and Z is a five-dimensional vector. We set that B is binary with the true signal shapes, and γ satisfies a standard normal distribution. We easily know that the response is $y=⟨X,B⟩+⟨Z,\gamma ⟩+\epsilon ,$ where ϵ satisfies a normal distribution. In our numerical experiments, we change sample sizes at n = 500, 750 and 1000, and results are displayed in Tables 6–8. Tables 6–8 show the root-mean-squared errors (RMSEs) for vector coefficient γ, matrix coefficient B, response y and their standard deviations. Note that RMSEs of $B,\gamma$ and y have significantly reduced with increasing of dimension of sample.

Table 6. Performance for signal shapes of Algorithm 1 when n = 500.

Shapes	RMSE-B	RMSE-γ	RMSE-y
Cross	0.29(0.004)	1.64(0.007)	0.20(0.002)
Square	0.32(0.004)	1.89(0.005)	0.25(0.007)
Tshape	0.32(0.005)	1.12(0.005)	0.27(0.008)
Triangle	0.31(0.001)	1.99(0.002)	0.24(0.001)

Notes: All the measurements are the mean of the results after repeated 100 times. The numbers in parentheses are the corresponding standard errors.

Table 7. Performance for signal shapes of Algorithm 1 when n = 750.

Shapes	RMSE-B	RMSE-γ	RMSE-y
Cross	0.21(0.001)	1.44(0.008)	0.18(0.007)
Square	0.28(0.002)	0.95(0.005)	0.23(0.004)
Tshape	0.26(0.002)	1.01(0.003)	0.22(0.005)
Triangle	0.22(0.009)	1.64(0.007)	0.19(0.004)

Notes: All the measurements are the mean of the results after repeated 100 times. The numbers in parentheses are the corresponding standard errors.

Table 8. Performance for signal shapes of Algorithm 1 when n = 1000.

Shapes	RMSE-B	RMSE-γ	RMSE-y
Cross	0.20(0.008)	1.29(0.001)	0.17(0.004)
Square	0.26(0.001)	0.88(0.004)	0.22(0.007)
Tshape	0.24(0.002)	0.98(0.001)	0.22(0.002)
Triangle	0.20(0.003)	0.73(0.012)	0.18(0.003)

Notes: All the measurements are the mean of the results after repeated 100 times. The numbers in parentheses are the corresponding standard errors.

5.2.2. Trip time prediction from partial trajectories

A classical example is the ECML Discovery challenge 2015 competition to estimate of taxi's travel time for complete trips [5,11]. It contains the 7733 trajectories of taxis in Porto for a period of 1 year, which every trajectory includes multiple features. We record the latitude and longitude coordinates every 15 s when running. It also contains seven regular variates, such as trip id, call type, origin stand, day type and so on. The trip id contains a unique identifier for every trip. Call type identifies some way to demand this service, such as, this trip was dispatched from the central, demanded directly to a taxi driver at a specific stand and otherwise. Origin stand contains a unique identifier for the taxi stand. Day type contains holiday or any other special day, a day before holiday and otherwise. In LRFL matrix regression model (2), X is a matrix in ${\mathbb{R}}^{922\times 2}$ and Z is a vector in ${\mathbb{R}}^7$. We remove 32 trajectories due to the missing observation of coordinates. We use our matrix regression model to predict taxis travel time for complete journeys.

Table 9 shows that the root-mean-squared error of prediction of test data under 5-fold or 10-fold Cross-Validation. Furthermore, we choose 1000 trips as a test dataset, and change the dimension of the training dataset. Results are displayed in Table 10.

Table 9. Results of trip time prediction under 5-fold or 10-fold cross-validation.

5-fold rate			10-fold rate
Training dataset	Test dataset	RMSE-PRE	Training dataset	Test dataset	RMSE-PRE
6161	1540	0.86(0.010)	6931	770	0.82(0.013)

Notes: All the measurements are the mean of the results after repeated 100 times. The numbers in parentheses are the corresponding standard errors.

Table 10. All the measurements are the mean of the results after repeated 100 times.

Training dataset	Test dataset	RMSE-PRE
200	1000	0.86(0.034)
500	1000	0.86(0.041)
1000	1000	0.81(0.089)
1500	1000	0.82(0.068)
2000	1000	0.77(0.046)

Note: The numbers in parentheses are the corresponding standard errors.

6. Summary

In this paper, we propose the LRFL matrix regression model which combines nuclear norm and fused LASSO penalty. The inspiration for this model comes from the fact which sampling units containing matrix and vector at the same time in kinds of fields. In order to solve the LRFL matrix regression model, we explore a linearized ADMM algorithm and establish the global convergence. Finally, we demonstrate the efficiency of our method through numerical experiments on simulation and real datasets. We compare LRFL matrix regression model with matrix LASSO. As we can see, our model can give more accurate and lower rank estimation. We mainly focus on the algorithm for solving the LRFL matrix regression model. Therefore, the statistical properties are also worthy of study.

Funding Statement

The work was supported in part by the National Natural Science Foundation of China (11671029), the Fundamental Research Funds for the Central Universities (2019YJS200) and Colleges and Universities in Hebei Province Science and Technology Research Project (Z2019032).

Disclosure statement

No potential conflict of interest was reported by the author(s).