Optimal Exact Least Squares Rank Minimization

Abstract

In multivariate analysis, rank minimization emerges when a low-rank structure of matrices is desired as well as a small estimation error. Rank minimization is nonconvex and generally NP-hard, imposing one major challenge. In this paper, we consider a nonconvex least squares formulation, which seeks to minimize the least squares loss function with the rank constraint. Computationally, we develop efficient algorithms to compute a global solution as well as an entire regularization solution path. Theoretically, we show that our method reconstructs the oracle estimator exactly from noisy data. As a result, it recovers the true rank optimally against any method and leads to sharper parameter estimation over its counterpart. Finally, the utility of the proposed method is demonstrated by simulations and image reconstruction from noisy background.

1. INTRODUCTION

In multivariate analysis, estimation of lower-dimensional structures has received attention in statistics, signal processing and machine learning. One type of such structures is the low-rank of matrices [5, 22], where the rank measures the dimension of a multivariate response. Rank minimization approximates multivariate data with the smallest possible rank of matrices. It has many applications in, for instance, multi-task learning [6, 11], multi-class classification [2], matrix completion [8, 17], collaborative filtering [33, 1], clustering [20, 29], computer vision [35, 21, 18], among others. The central topic this article addresses is least squares rank minimization.

Consider multi-response linear regression in which a k-dimensional response vector z_i follows

$${\mathit{z}}_i={\mathbf{\Theta }}^T{\mathit{a}}_i+{\mathit{\epsilon }}_i;E{\mathit{\epsilon }}_i=\mathbf{0},\mathit{\text{Cov}}({\mathit{\epsilon }}_i)={\sigma }^2{\mathit{I}}_{k\times k};i=1,\cdots ,n,$$ (1)

where a_i is a p-dimensional design vector, Θ is a p × k regression parameter matrix, and components of ε_i are independent. Model (1) reduces to the widely-used linear model in compressed sensing when k = 1 and becomes a multivariate autoregressive model with a_i = z_i−1. Denote the rank of Θ as r(Θ) and rewrite (1) in the matrix form as follows:

$$\mathit{Z}=\mathit{A}\mathbf{\Theta }+\mathit{e},$$ (2)

where Z = (z₁, ⋯, z_n)^T ∈ ℝ^n×k, A = (a₁, ⋯, a_n)^T ∈ ℝ^{n × p} and e = (ε₁, ⋯, ε_n)^T ∈ ℝ^n×k are the data, design and error matrices. In (1), we estimate Θ based on n pairs of observation vectors ${({\mathit{a}}_i,{\mathit{z}}_i)}_{i=1}^n$, with a priori knowledge that r(Θ) is relatively small in comparison to min(n, k, p), where the number of unknown parameters k, p can greatly exceed the sample size n.

Least squares rank minimization, as described, solves

$$\underset{\mathbf{\Theta }}{\text{min}}{\Vert \mathit{A}\mathbf{\Theta }-\mathit{Z}\Vert }_F^2\mathrm{s}.\mathrm{t}.r(\mathbf{\Theta })\le s,$$ (3)

where ‖·‖_F is the Frobenius-norm and s is an integer-valued tuning parameter taking values from [1, min(n, k, p)]. The general rank minimization is nonconvex and NP-hard [23], which is like the L₀-minimization in univariate analysis. Therefore an exact global solution to (3) is not known as well as its statistical properties, due primarily to discreteness and non-convexity of the rank function.

Estimation under the restriction that r(Θ) = r has been studied when n → ∞ with k and p held fixed, see [3, 4, 15, 28, 26]. Two major computational approaches have been proposed for approximating the optimal solution of (3). The first one involves regularization using surrogate function, such as the nuclear-norm, which is a convex envelope of the rank function [13] and can be solved by efficient algorithms [8, 19, 34, 25, 16]. In some cases, the solution of this convex problem coincides with a global minimizer of (3) under certain isometry assumptions [27]. However, these assumptions can be strong and difficult to check. Recently, [7] obtained a global minimizer of a regularized version of (3).

The second approach attacks (3) by approximating the rank function iteratively through calculating the largest singular vector using greedy search [30], or by singular value projection (SVP) through a local gradient method [17]. Under weaker isometry assumptions [27, 10, 9] than that of the nuclear-norm approach, these methods guarantee an exact solution of (3) but suffer from the same difficulties as the regularization method [30], although they have achieved promising results on both simulated and real-world data.

Theoretically, some loss error bounds of the first regularization approach are obtained in [24] under Frobenius-norm, and rank selection consistency is established in [7]. Unfortunately, to our best knowledge, whether similar conclusions hold for our formulation (3) remains largely unknown.

In this paper, we have advanced on two fronts. Computationally, we derive a general closed-form for a global minimizer of (3) in Theorem 1, and give a condition under which (3) and its nonconvex regularized counterpart are equivalent with regard to global minimizers, although these two methods are not generally equivalent. Moreover, we develop an efficient algorithm for computing the entire regularization solution path at the cost of computing only one solution for a single regularization parameter. Theoretically, we establish optimality for a global minimizer of (3). More specifically, the proposed method is optimal against any other ones in that it reconstructs the oracle estimator exactly, thus the true rank, under (1). It is important to note that this exact recovery result is a much stronger property than consistency, which is attributed to the discrete nature of the rank function as well as tuning parameter s. Such a result may not be shared by its regularized counterpart with a continuum tuning parameter. In addition, the method enjoys a higher degree of accuracy for parameter estimation than nuclear-norm rank estimation.

After the first draft of this paper was completed, we were aware that [14] and [32] gave an expression of Theorem 1. However, neither paper considered computational and statistical aspects of the solution. Inevitably, some partial overlaps exist between our Theorem 1 and theirs.

The rest of the paper is organized as follows. Section 2 presents a closed-form solution to (3). Section 3 gives an efficient path algorithm for a regularized version of (3). Section 4 is devoted to theoretical investigation, followed by Section 5 discussing methods for tuning. Section 6 presents proofs for all the theorems we develop and Section 7 presents results of empirical evaluations, where several rank minimization methods are compared. Section 8 concludes the paper.

Notation: Table 1 summarizes the notations used in the rest of the paper.

Table 1.

Notation used throughout the paper

Notation	Description
A	The design matrix
e	Error matrix
Θ0	The ground truth model
Θ̂s	Estimator obtained by optimizing (3)
${\mathrm{\Theta ̂}}_{\lambda }^{*}$	Estimator obtained by optimizing (7)
r(A)	The rank of matrix A
℘s(Z)	The best s-rank approximation of matrix Z in terms of the Frobenius-norm

2. PROPOSED METHOD: CLOSED-FORM SOLUTION

This section derives a closed-form solution to (3). The strategy is to simplify (3) through the singular value decomposition (SVD) of matrix A and utilizing the properties of the rank function. Before proceeding, we present a motivating lemma, known as the Eckart-Young theorem [12].

Lemma 1. The best s-rank approximation, in terms of the Frobenius-norm, for a t-rank matrix Z with t ≥ s, i.e., a global minimizer Θ* of

$$\underset{\mathbf{\Theta }}{\text{min}}{\Vert \mathbf{\Theta }-\mathit{Z}\Vert }_F^{2}s.t.r(\mathbf{\Theta })\le s$$ (4)

is given by ${\mathbf{\Theta }}^{*}={\wp }_s(\mathit{Z})={\mathit{U}}_z{\mathit{D}}_s{\mathit{V}}_z^T$, where D_s consists of the largest s singular values of Z given the SVD of $\mathit{Z}={\mathit{U}}_z\mathit{D}{\mathit{V}}_z^T$.

Intuitively ℘_s(Z) may be viewed as a projection of Z onto a set of matrices whose ranks are no more than s. Note that (4) is a special case of (3) with matrix A being the identity matrix. This motivates us to solve (3) through the simpler problem (4).

When A is nonsingular, clearly (3) has a global minimizer A⁻¹ ℘_s(Z) by rank preserveness of any nonsingular matrix in matrix multiplication. When A is singular, we first assume that r(A) ≥ s. However, this assumption is by no means mandatory, which will be discussed later. Given the SVD of A = UDV^T, with orthogonal matrices U ∈ ℝ^n×n and V ∈ ℝ^p×p and diagonal matrix D ∈ ℝ^n×p, we have

$${\Vert \mathit{A}\mathbf{\Theta }-\mathit{Z}\Vert }_F={\Vert {\mathit{U}}^T(\mathit{A}\mathbf{\Theta }-\mathit{Z})\Vert }_F={\Vert \mathit{D}{\mathit{V}}^T\mathbf{\Theta }-{\mathit{U}}^T\mathit{Z}\Vert }_F.$$

This follows from the fact that the Frobenius-norm is invariant under any orthogonal transformation. Let Y = V^T Θ and W = U^TZ, clearly we have r(Y) = r(Θ). Now solving (3) amounts to solving an equivalent form:

$$\underset{\mathit{Y}}{\text{min}}{\Vert \mathit{D}\mathit{Y}-\mathit{W}\Vert }_F^2\mathrm{s}.\mathrm{t}.r(\mathit{Y})\le s.$$ (5)

Consequently, a global minimizer of (3) becomes VY*, where Y* is a global minimizer of (5) and is given by the following theorem.

Theorem 1. Let D, Y, Z and s be as defined above. If s ≤ r(A), then a global minimizer of (5) is given by

$${\mathit{Y}}^{*}=\left[\begin{array}{c}\hfill {\mathit{D}}_{r(\mathit{A})}^{-1}{\wp }_s({\mathit{W}}_{r(\mathit{A})})\hfill \\ \hfill \mathit{a}\hfill \end{array}\right],$$ (6)

where D_r(A) is a diagonal matrix consisting of all the nonzero singular value of A, a can be set as the zero matrix, and W_r(A) consists of the first r(A) rows of W.

Here are some remarks regarding the above theorem:

Remark 1. The solution to problem (5) is generally not unique. Specifically, the matrix a in (6) need not be fixed at zero, as long as it does not change the rank of Y*. However, if A is of full column rank, i.e., when r(A) = p, then a vanishes and Y* can be uniquely determined. In this case, the optimal solution of (3) is also unique.

Remark 2. The optimal Y* can also be computed for the general matrix A with an arbitrary rank, i.e., when r(A) < s. See the proof of Theorem 1 in the Section 6.

It is important to note that the value of a is irrelevant for prediction, but matters for parameter estimation. In other words, when r(A) < p, a global minimizer is not unique, hence that parameter estimation is not identifiable: see Section 4 for a discussion. For simplicity, we set a = 0 for Y* subsequently.

In what follows, our estimator is defined as Θ̂^s, as well as an estimated rank r̂. Algorithm 1 below summarizes the main steps for computing Θ̂^s with regard to s ≤ min(n, k, p), where the LSRM stands for Least Squares Rank Minimization.

Algorithm 1.

Exact solution of (3)

Input: A, Z, s ≤ r(A)
Output: A global minimizer Θ of (3)
Function: LSRM(A, Z, s)
1:	if A is nonsingular then
2:	Θ = A−1℘s(Z)
3:	else
4:	Perform SVD on A: A = UDVT
5:	Extract the first r rows of UTZ and denote it as Wr(A)
6	$\mathbf{\Theta }=\mathit{V}\left[\begin{array}{c}\hfill {\mathit{D}}_{r(\mathit{A})}^{-1}{\wp }_s({\mathit{W}}_{r(\mathit{A})})\hfill \\ \hfill \mathbf{0}\hfill \end{array}\right]$
7:	end if
8:	return Θ

The complexity of Algorithm 1 is determined mainly by the most expensive operations–matrix inversion and SVD, specifically, at most one matrix inversion and two SVDs. Such operations roughly require a cubic time complexity¹.

3. REGULARIZATION AND SOLUTION PATH

This section studies a regularized counterpart of (3):

$$\underset{\mathbf{\Theta }}{\text{min}}{\Vert \mathit{A}\mathbf{\Theta }-\mathit{Z}\Vert }_F^2+\lambda r(\mathbf{\Theta }),$$ (7)

where λ > 0 is a continuous regularization parameter corresponding to s in (3), and ${\mathbf{\Theta }}_{\lambda }^{*}$ is a global minimizer of (7). The next theorem establishes an equivalence between (7) and (3) when ${\mathbf{\Theta }}_{\lambda }^{*}$ is unique, occurring when r(A) = p. Such a result is not generally anticipated for a nonconvex problem.

Theorem 2 (Equivalence). When p = r(A), (7) has a unique global minimizer. Moreover, (7) and (3) are equivalent with respect to their solutions. For any ${\mathbf{\Theta }}_{\lambda }^{*}$ with λ ≥ 0, there exists $1\le s^{*}=r({\mathbf{\Theta }}_{\lambda }^{*})$ such that ${\mathbf{\Theta }}_{\lambda }^{*}={\mathbf{\Theta ̂}}^s$, and vice verse.

Next we develop an algorithm for computing an entire solution path for all values of λ with complexity comparable to that of solving (7) at a single λ-value. For motivation, first consider a special case of the identity A in (7):

$$g(\lambda )=\underset{\mathbf{\Theta }}{\text{min}}{\Vert \mathbf{\Theta }-\mathit{Z}\Vert }_F^2+\lambda r(\mathbf{\Theta }).$$ (8)

3.1 Monotone property

In (8), r(Θ) decreases as λ increases from 0, where r(Θ) goes through all integer values from r(Z) down to 0 when λ becomes sufficiently large. In addition, the value of g(λ) is nondecreasing as λ increases. The next theorem summarizes these results.

Theorem 3 (Monotone property). Let r(Z) be r. Then the following properties hold:

1. There exists a solution path vector 𝒮 of length r + 2 satisfying the following:

   $${𝒮}_0=0,{𝒮}_{r+1}=+\infty ,{𝒮}_{k+1}>{𝒮}_k,k=0,1,\cdots ,r{\mathbf{\Theta }}_{\lambda }^{*}={\wp }_{r-k}(\mathit{Z}),\text{if}{𝒮}_k\le \lambda <{𝒮}_{k+1},$$

2. Function g(λ) is nondecreasing and piecewise linear.

The monotone property leads to an efficient algorithm for calculating the pathwise solution of (8). Figure 1 displays the solution path by illustrating the function value g(λ) and $r({\mathbf{\Theta }}_{\lambda }^{*})$ as a function of λ.

Figure 1.

Piecewise linearity of g(·) and the rank of the optimal solution with respect to λ.

3.2 Pathwise algorithm

Through the monotone property, we compute the optimal solution of (8) at a particular λ by locating λ in the correct interval in the solution path vector 𝒮, which can be achieved efficiently via a simple binary search. Algorithm 2 describes the main steps.

Algorithm 2.

Pathwise solution of (8)

Input: Θ, Z
Output: Solution path vector 𝒮, pathwise solution Θ
Function: pathwise(Θ, Z)
1:	Initialize: 𝒮0 = 0, Θ0 = Z, r = r(Z)
2:	Perform SVD on Z: Z = UDVT
3	for i = r down to 1 do
4:	${𝒮}_{r-i+1}{\sigma }_i^2$
5:	${\mathbf{\Theta }}_{r-i+1}={\mathbf{\Theta }}_{r-i}-{\sigma }_i{u}_i{\upsilon }_i^T$
6:	end for
7:	return 𝒮, Θ

Algorithm 2 requires only one SVD operation, therefore its complexity is of the same order as that of Algorithm 1 at a single s-value. When Z is a low-rank matrix, existing software for SVD computation such as PROPACK is applicable to further improve computational efficiency.

3.3 Extension to general A

For a general design matrix A, note that

$${\Vert A\mathbf{\Theta }-\mathit{Z}\Vert }_F^2={\Vert \mathit{D}\mathit{Y}-\mathit{W}\Vert }_F^2={\Vert \mathit{W}\prime \Vert }_F^2+{\Vert {\mathit{D}}_r{\mathit{Y}}_r-{\mathit{W}}_r\Vert }_F^2,$$

where $\mathit{W}=\left[\begin{array}{c}\hfill {\mathit{W}}_r\hfill \\ \hfill \mathit{W}\prime \hfill \end{array}\right]$. After ignoring the constant term ${\Vert \mathit{W}\prime \Vert }_F^2$, we solve

$$\underset{{\mathit{Y}}_r}{\text{min}}{\Vert {\mathit{D}}_r{\mathit{Y}}_r-{\mathit{W}}_r\Vert }_F^2+\lambda r({\mathit{Y}}_r).$$

Note that D_r is nonsingular, then the problem reduces to the simple case

$$\underset{\mathit{Ŷ}}{\text{min}}{\Vert \mathit{Ŷ}-{\mathit{W}}_r\Vert }_F^2+\lambda r(\mathit{Ŷ}),$$

where Ŷ = D_rY_r. The solution path can be obtained directly from Algorithm 2.

4. STATISTICAL PROPERTIES

This section is devoted to the theoretical investigation of least squares rank minimization, which remains largely unexplored, although nuclear-norm regularization has been studied. In particular, we will reveal what is the best performance for prediction as well as the optimal risk for parameter estimation. Moreover, we will establish the optimality of the proposed method. In fact, the proposed method reconstructs the oracle estimator, the optimal one as if the true rank were known in advance. Here the oracle estimator Θ̂⁰ is defined as a global minimizer of ${\Vert \mathit{A}\mathbf{\Theta }-\mathit{Z}\Vert }_F^2$ subject to r(Θ) = r₀, where Θ⁰ and r₀ = r(Θ⁰) ≥ 1 denote the true parameter matrix and the true rank respectively. This leads to the exact rank recovery, in addition to the reconstruction of the optimal performance of the oracle estimator. In other words, the proposed method is optimal against any other methods such as the nuclear-norm rank regularization.

Given the design matrix A, we study the accuracy of rank recovery as well as prediction and parameter estimation. Let ℙ and 𝔼 be the true probability and expectation under Θ⁰ given A. For rank recovery, we use the metric ℙ(r̂ ≠ r₀). For prediction and parameter estimation, we employ the risk 𝔼K(Θ̂^s, Θ⁰) and $𝔼{\Vert {\mathbf{\Theta ̂}}^s-{\mathbf{\Theta }}^0\Vert }_F^2$ respectively, where

$$K({\mathbf{\Theta ̂}}^s,{\mathbf{\Theta }}^0)={(2{\sigma }^{2}n)}^{-1}\sum _{i=1}^n{\Vert {\mathit{a}}_i^T({\mathbf{\Theta ̂}}^s-{\mathbf{\Theta }}^0)\Vert }_2^2={(2{\sigma }^{2}n)}^{-1}{\Vert \mathit{A}({\mathbf{\Theta ̂}}^s-{\mathbf{\Theta }}^0)\Vert }_F^2$$

is the Kullback-Leibler loss and ‖·‖₂ is the L₂-norm for a vector. Note that the predictive risk equals to 2σ²𝔼K(Θ̂^s, Θ⁰) and parameter estimation is considered when it is identifiable, i.e., r(A) = p.

Now we present the risk bounds under (1) without a Gaussian error assumption.

Theorem 4. Under (1), the oracle estimator is exactly reconstructed by our method in that Θ̂^r₀ = Θ⁰ under ℙ, when r₀ ≤ min(r(A), p). As a result, exact reconstruction of the optimal performance is achieved by our estimator Θ̂^r₀. In particular,

$$𝔼K({\mathbf{\Theta ̂}}^{r_0},{\mathbf{\Theta }}^0)\{\begin{array}{ccc}\hfill =\hfill & \hfill \frac{r_{0}k}{2n}\hfill & \hfill \mathit{\text{if}}{r}_0=r(\mathit{A})\hfill \\ \hfill \le \hfill & \hfill \frac{2𝔼({\sum }_{j=1}^{r_0}{\sigma }_j^2)}{n}\hfill & \hfill \mathit{\text{if}}{r}_0<r(\mathit{A})\hfill \end{array}$$

and

$$𝔼{\Vert {\mathbf{\Theta ̂}}^{r_0}-{\mathbf{\Theta }}^0\Vert }_F^2\{\begin{array}{ccc}\hfill =\hfill & \hfill \frac{r_{0}k}{{\sigma }_{\text{min}}^2(\mathit{A})n}\hfill & \hfill \mathit{\text{if}}{r}_0=r(\mathit{A})=p\hfill \\ \hfill \le \hfill & \hfill \frac{𝔼({\sum }_{j=1}^{r_0}{\sigma }_j^2)}{{\sigma }_{\text{min}}^{2}n}\hfill & \hfill \mathit{\text{if}}{r}_0<r(\mathit{A})=p\hfill \end{array},$$

where σ_j and σ_min > 0 are the jth largest and the smallest nonzero singular value of ${\mathit{e}}^{\star }={\mathit{U}}_{r(A)}^T\mathit{e}$ and n^−1/2 A, respectively, and U_r(A) denotes the first r(A) rows of U.

Remark: In general, $𝔼{\sum }_{j=1}^{r_0}{\sigma }_j^2\le r_0𝔼{\sigma }_1^2$.

Theorem 4 says that the optimal oracle estimator is exactly reconstructed by our method. Interestingly, the true rank is exactly recovered from noisy data, which is attributed to discreteness of the rank and is analogous to the maximum likelihood estimation over a discrete parameter space. Concerning prediction and parameter estimation, the optimal Kullback-Leibler risk is $\frac{r_{0}k}{n}$ but the risk under the Frobenius-norm is $\frac{r_{0}k}{{\sigma }_{\text{min}}^{2}n}$. For prediction, only the effective degrees of freedom ${\sum }_{j=1}^{r_0}{\sigma }_j^2$ matters as opposed to p, which is in contrast to a rate $\frac{r_{0}kp}{n}$ without a rank restriction. This permits p to be much larger than n, or k, p ≫ n. For estimation, however, p enters the risk through ${\sigma }_{\text{min}}^2$, where p can not be larger than n, or max(k, p) ≤ n.

5. TUNING

As shown in Section 4, theoretically, exact rank reconstruction can be accomplished through tuning. Practically, we employ a predictive measure for rank selection.

The predictive performance of Θ̂^s is measured by

$$\mathrm{M}\mathrm{S}\mathrm{E}({\mathbf{\Theta ̂}}^s)=\frac{1}{n}𝔼{\Vert \mathit{A}{\mathbf{\Theta ̂}}^s-\mathit{Z}\Vert }_F^2,$$

which is proportional to the risk R(Θ̂^s), where the expectation is taken with respect to (Z, A).

To estimate s over integer values in [0, ⋯, min(n, p, k)], one may cross-validate through a tuning data set, which estimates the MSE. Alternatively, one may use the generalized degrees of freedom [31] through data perturbation without a tuning set.

$$\widehat{\text{GDF}}({\mathbf{\Theta ̂}}^s)=n^{-1}{\Vert \mathit{Z}-\mathit{A}{\mathbf{\Theta ̂}}^s\Vert }^2+2n^{-1}\sum _{i=1}^n\sum _{l=1}^k\widehat{\text{Cov}}({\mathit{z}}_{il},{({\mathit{a}}_i^T{\mathbf{\Theta ̂}}^s)}_l),$$ (9)

where $\widehat{\text{Cov}}({\mathit{z}}_{il},{({\mathit{a}}_i^T{\mathbf{\Theta ̂}}^s)}_l)$ is the estimated covariance between the lth component of z_i and the lth component of ${\mathit{a}}_i^T{\mathbf{\Theta ̂}}^s$. In the case that e_i in (1) follows N(0, σ² I_k×k), the method of data perturbation of [31] is applicable. Specifically, sample ${\mathit{e}}_i^{*}$ from N(0, σ²I_k×k) and let

$${\mathit{Z}}^{*}=(1-\tau )\mathit{Z}+\tau \mathit{Z̃}\widehat{\text{Cov}}({\mathit{z}}_{il},{({\mathit{a}}_i^T{\mathbf{\Theta ̂}}^s)}_l)={\tau }^{-1}{\text{Cov}}^{*}({\mathit{z}}_{il}^{*},{({\mathit{a}}_i^T{\mathbf{\Theta ̂}}^{*s})}_l)$$

where ${\text{Cov}}^{*}({\mathit{z}}_{il}^{*},{({\mathit{a}}_i{\mathbf{\Theta ̂}}^{*s})}_l)$ is the sample covariance with the Monte Carlo size T. For the types of problems we consider, we fixed T to be n.

6. PROOF OF THEOREMS

In this section we present detailed proofs to the theorems developed in the previous sections. We first present a technical lemma to be used in the proof of Theorem 4.

Lemma 2. Suppose A and B are two n₁ × n₂ matrices. Then,

$$\langle \mathit{A},\mathit{B}\rangle \le {\Vert \mathit{A}\Vert }_F{\Vert {\wp }_{r(\mathit{A})}(\mathit{B})\Vert }_F,$$ (10)

where 〈A,B〉 = Tr(A^TB) = Tr(B^TA), Tr denotes the trace, and r(A) is the rank of A.

Proof. Let the singular value decomposition of A and B be $\mathit{A}={\mathit{U}}_1{\mathbf{\Sigma }}_1{\mathit{V}}_1^T$ and $\mathit{B}={\mathit{U}}_2{\mathbf{\Sigma }}_2{\mathit{V}}_2^T$ where U_i and V_i, i = 1, 2, are orthogonal matrices, and Σ₁ and Σ₂ are diagonal matrices with their diagonal elements being the singular values of A and B, respectively. Then

$$\langle \mathit{A},\mathit{B}\rangle =Tr({\mathit{V}}_1{\mathbf{\Sigma }}_1^T{\mathit{U}}_1^T{\mathit{U}}_2{\mathbf{\Sigma }}_2{\mathit{V}}_2^T)=Tr({\mathbf{\Sigma }}_1^T{\mathit{U}}_1^T{\mathit{U}}_2{\mathbf{\Sigma }}_2{\mathit{V}}_2^T{\mathit{V}}_1)\equiv Tr({\mathbf{\Sigma }}_1^T\mathit{U}{\mathbf{\Sigma }}_2{\mathit{V}}^T),$$

where $\mathit{U}={\mathit{U}}_1^T{\mathit{U}}_2$ and $\mathit{V}={\mathit{V}}_1^T{\mathit{V}}_2$ continue to be orthogonal. Let the ordered singular values of A be σ₁ ≥ ⋯ ≥ σ_r(A) and B̃ = (b̃_ij) = UΣ₂V^T. By Cauchy-Schwarz’s inequality,

$$Tr({\mathbf{\Sigma }}_1^T\mathit{U}{\mathbf{\Sigma }}_2{\mathit{V}}^T)=Tr({\mathbf{\Sigma }}_1^T\mathit{B̃})=\sum _{i=1}^{r(\mathit{A})}{\sigma }_i{\mathrm{b̃}}_{ii}\le \sqrt{\sum _{i=1}^{r(\mathit{A})}{\sigma }_i^2}\sqrt{\sum _{i=1}^{r(\mathit{A})}{\mathrm{b̃}}_{ii}^2}={\Vert \mathit{A}\Vert }_F\sqrt{\sum _{i=1}^{r(\mathit{A})}{\mathrm{b̃}}_{ii}^2}.$$ (11)

Similarly, let the ordered singular values of B be η₁ ≥ ⋯ ≥ η_r(B). Then it suffices to show that ${\sum }_{i=1}^{r(\mathit{A})}{\mathrm{b̃}}_{ii}^2\le {\sum }_{i=1}^{r(\mathit{A})}{\eta }_i^2$. Assume, without loss of generality, that η_i = 0 if i > r(B). Let n = min(n₁, n₂). By Cauchy-Schwarz’s inequality,

$$\sum _{i=1}^{r(\mathit{A})}{\mathrm{b̃}}_{ii}^2=\sum _{i=1}^{r(\mathit{A})}{(\sum _{k=1}^n{u}_{ik}{\eta }_k{\upsilon }_{ik})}^2\le \sum _{i=1}^{r(\mathit{A})}(\sum _{k=1}^n{u}_{ik}^2{\eta }_k^2)(\sum _{k=1}^n{\upsilon }_{ik}^2)\le \sum _{i=1}^{r(\mathit{A})}(\sum _{k=1}^n{u}_{ik}^2{\eta }_k^2)=\sum _{k=1}^n{\eta }_k^2(\sum _{i=1}^{r(\mathit{A})}{u}_{ik}^2)\le \sum _{k=1}^{r(\mathit{A})}{\eta }_k^2,$$

where the last step uses the fact that ${\sum }_{i=1}^{r(\mathit{A})}{u}_{ik}^2\le 1$ and ${\sum }_{k=1}^n{\sum }_{i=1}^{r(\mathit{A})}{u}_{ik}^2={\sum }_{i=1}^{r(\mathit{A})}{\sum }_{k=1}^n{u}_{ik}^2\le r(\mathit{A})$. A combination of the above bounds lead to the desired results. This completes the proof.

6.1 Proof of Theorem 1

First partition D and W as follows:

$$\mathit{D}=\left[\begin{array}{cc}\hfill {\mathit{D}}_{r(\mathit{A})}\hfill & \hfill \mathbf{0}\hfill \\ \hfill \mathbf{0}\hfill & \hfill \mathbf{0}\hfill \end{array}\right],\mathit{W}=\left[\begin{array}{c}\hfill {\mathit{W}}_{r(\mathit{A})}\hfill \\ \hfill \mathit{W}\prime \hfill \end{array}\right],$$

then

$$\mathit{D}\mathit{Y}-\mathit{W}=\left[\begin{array}{c}\hfill {\mathit{D}}_{r(\mathit{A})}{\mathit{Y}}_{r(\mathit{A})}\hfill \\ \hfill \mathbf{0}\hfill \end{array}\right]-\left[\begin{array}{c}\hfill {\mathit{W}}_{r(\mathit{A})}\hfill \\ \hfill \mathit{W}\prime \hfill \end{array}\right]=\left[\begin{array}{c}\hfill {\mathit{D}}_{r(\mathit{A})}{\mathit{Y}}_{r(\mathit{A})}-{\mathit{W}}_{r(\mathit{A})}\hfill \\ \hfill -\mathit{W}\prime \hfill \end{array}\right].$$

Evidently, only the first r(A) rows of Y are involved in minimizing ${\Vert \mathit{D}\mathit{Y}-\mathit{W}\Vert }_2^2$, which amounts to computing the global minimizer ${\mathit{Y}}_{r(\mathit{A})}^{*}$ of

$$\text{arg}\underset{{\mathit{Y}}_r(\mathit{A})}{\text{min}}{\Vert {\mathit{D}}_{r(\mathit{A})}{\mathit{Y}}_{r(\mathit{A})}-{\mathit{W}}_{r(\mathit{A})}\Vert }_F^2.$$

Then ${\mathit{Y}}_{r(\mathit{A})}^{*}={\mathit{D}}_{r(\mathit{A})}^{-1}{\wp }_s({\mathit{W}}_{r(\mathit{A})})$ by non-singularity of D_r(A) and Lemma 1 with s ≤ r(A). For s > r(A), recall that, only the upper part of Y* is relevant in minimizing (5). The result then follows. This completes the proof.

6.2 Proof of Theorem 2

For any ${\mathbf{\Theta }}_{\lambda }^{*}$ with λ > 0, let $s^{*}=r({\mathbf{\Theta }}_{\lambda }^{*})$. Next we prove by contradiction that ${\mathbf{\Theta }}_{\lambda }^{*}={\mathbf{\Theta ̂}}^{s^{*}}$. Suppose ${\mathbf{\Theta }}_{\lambda }^{*}\ne {\mathbf{\Theta ̂}}^{s^{*}}$. By uniqueness of Θ̂^s* given in Theorem 1 and the definition of minimization, ${\Vert \mathit{A}{\mathbf{\Theta ̂}}^{s^{*}}-\mathit{Z}\Vert }_F^2<{\Vert \mathit{A}{\mathbf{\Theta }}_{\lambda }^{*}-\mathit{Z}\Vert }_F^2$. This, together with $r({\mathbf{\Theta ̂}}^{s^{*}})=r({\mathbf{\Theta }}_{\lambda }^{*})$, implies that

$${\Vert \mathit{A}{\mathbf{\Theta ̂}}^{s^{*}}-\mathit{Z}\Vert }_F^2+\lambda r({\mathbf{\Theta ̂}}^{s^{*}})<{\Vert \mathit{A}{\mathbf{\Theta }}_{\lambda }^{*}-\mathit{Z}\Vert }_F^2+\lambda r({\mathbf{\Theta }}_{\lambda }^{*}).$$

This contradicts to the fact that ${\mathbf{\Theta }}_{\lambda }^{*}$ is a minimizer of (7). The converse is revealed in the proof of Theorem 3.

6.3 Proof of Theorem 3

We prove the first conclusion by constructing such a solution path vector 𝒮. Let 𝒮₀ = 0, 𝒮_r+1 = + ∞. Define 𝒮_k for 1 ≤ k ≤ r as the solution of equation:

$${\Vert {\wp }_{r-k+1}(\mathit{Z})-\mathit{Z}\Vert }_F^2+{𝒮}_k(r-k+1)={\Vert {\wp }_{r-k}(\mathit{Z})-\mathit{Z}\Vert }_F^2+{𝒮}_k(r-k).$$

It follows that

$${𝒮}_k={\Vert {\wp }_{r-k}(\mathit{Z})-\mathit{Z}\Vert }_F^2-{\Vert {\wp }_{r-k+1}(\mathit{Z})-\mathit{Z}\Vert }_F^2=\sum _{j=r-k+1}^r{\sigma }_j^2-\sum _{j=r-k+2}^r{\sigma }_j^2={\sigma }_{r-k+1}^2,$$ (12)

where σ_j is the jth largest non-zero singular value of Z. By (12), 𝒮_k is increasing. In addition, by definition of S_k and S_k+1, whenever λ falls into the interval [𝒮_k, 𝒮_k+1), the rank of a global minimizer Θ* of (8) would be no more than r − k and larger than r − k − 1. In other words, ${\mathbf{\Theta }}_{\lambda }^{*}$ is always of rank r − k and is given by ℘_r−k(Z). Therefore, the constructed solution path vector 𝒮 satisfies all the requirements in the theorem.

Moreover, when 𝒮_k ≤ λ < 𝒮_k+1,

$$g(\lambda )={\Vert {\wp }_{r-k}(\mathit{Z})-\mathit{Z}\Vert }_F^2+\lambda r({\wp }_{r-k}(\mathit{Z}))={\Vert {\wp }_{r-k}(\mathit{Z})-\mathit{Z}\Vert }_F^2+(r-k)\lambda .$$ (13)

Since ℘_r−k(Z) is independent of λ, g(λ) is a nondecreasing linear function of λ in each interval [𝒮_k, 𝒮_k+1). Combined with the definition of the solution path vector 𝒮, we conclude that g(λ) is nondecreasing and piecewise linear with each element of 𝒮 as a kink point, as shown in Figure 1. The proof is completed.

6.4 Proof of Theorem 4

The proof uses direct calculations. First we bound the Kullback-Leibler loss. By Theorem 1,

$$\mathit{A}{\mathbf{\Theta ̂}}^{r_0}={\mathit{U}}_{r(\mathit{A})}{\wp }_{r_0}({\mathit{W}}_{r(\mathit{A})}),$$

where

$${\mathit{W}}_{r(\mathit{A})}={\mathit{D}}_{r(\mathit{A})}{({\mathit{V}}^T{\mathbf{\Theta }}^0)}_{r(\mathit{A})}+{({\mathit{U}}^T\mathit{e})}_{r(\mathit{A})}.$$

Denote B = D_r(A) (V^T Θ⁰)_r(A) with rank r(B) ≤ r₀. Since the Frobenius-norm is invariant under orthogonal transformation, it follows that

$${\Vert \mathit{A}{\mathbf{\Theta ̂}}^{r_0}-\mathit{A}{\mathbf{\Theta }}^0\Vert }_F^2={\Vert {\mathit{U}}_{r(\mathit{A})}{\wp }_{r_0}({\mathit{W}}_{r(\mathit{A})})-\mathit{U}\mathit{D}{\mathit{V}}^T{\mathbf{\Theta }}^0\Vert }_F^2={\Vert {\mathit{U}}_{r(\mathit{A})}{\wp }_{r_0}({\mathit{W}}_{r(\mathit{A})})-{\mathit{U}}_{r(\mathit{A})}{\mathit{D}}_{r(\mathit{A})}{({\mathit{V}}^T{\mathbf{\Theta }}^0)}_{r(\mathit{A})}\Vert }_F^2={\Vert {\wp }_{r_0}({\mathit{W}}_{r(\mathit{A})})-{\mathit{D}}_{r(\mathit{A})}{({\mathit{V}}^T{\mathbf{\Theta }}^0)}_{r(\mathit{A})}\Vert }_F^2={\Vert {\wp }_{r_0}(\mathit{B}+{({\mathit{U}}^T\mathit{e})}_{r(\mathit{A})})-\mathit{B}\Vert }_F^2,={\Vert {\wp }_{r_0}(\mathit{B}+{\mathit{e}}^{\star }-\mathit{B})\Vert }_F^2,$$

where e^⋆ = (U^Te)_r(A). From the definition of ℘_r0 (B + e^⋆) we can conclude that:

$${\Vert {\wp }_{r_0}(\mathit{B}+{\mathit{e}}^{\star })-\mathit{B}-{\mathit{e}}^{\star }\Vert }_F^2\le {\Vert \mathit{B}-\mathit{B}-{\mathit{e}}^{\star }\Vert }_F^2={\Vert {\mathit{e}}^{\star }\Vert }_F^2,$$

which implies that,

$${\Vert {\wp }_{r_0}(\mathit{B}+{\mathit{e}}^{\star })-\mathit{B}\Vert }_F^2\le 2\langle {\wp }_{r_0}(\mathit{B}+{\mathit{e}}^{\star })-\mathit{B},{\mathit{e}}^{\star }\rangle \le 2{\Vert {\wp }_{r_0}(\mathit{B}+{\mathit{e}}^{\star })-\mathit{B}\Vert }_F{\Vert {\wp }_{r_0}({\mathit{e}}^{\star })\Vert }_F,$$

where the last inequality follows from Lemma 2. Thus,

$${\Vert {\wp }_{r_0}(\mathit{B}+{\mathit{e}}^{\star })-\mathit{B}\Vert }_F^2\le 4{\Vert {\wp }_{r_0}({\mathit{e}}^{\star })\Vert }_F^2=4\sum _{j=1}^{r_0}{\sigma }_j^2.$$ (14)

The risk bounds then follow.

Second we bound ${\Vert {\mathbf{\Theta ̂}}^{r_0}-{\mathbf{\Theta }}^0\Vert }_F^2$, which is equal to

$${\Vert \mathit{V}\left[\begin{array}{c}\hfill {\mathit{D}}_{r(\mathit{A})}^{-1}{\wp }_{r_0}({\mathit{W}}_{r(\mathit{A})})\hfill \\ \hfill \mathbf{0}\hfill \end{array}\right]-{\mathbf{\Theta }}^0\Vert }_F^2={\Vert \left[\begin{array}{c}\hfill {\mathit{D}}_{r(\mathit{A})}^{-1}{\wp }_{r_0}({\mathit{W}}_{r(\mathit{A})})\hfill \\ \hfill \mathbf{0}\hfill \end{array}\right]-{\mathit{V}}^T{\mathbf{\Theta }}^0\Vert }_F^2={\Vert {\mathit{D}}_{r(\mathit{A})}^{-1}{\wp }_{r_0}({\mathit{W}}_{r(\mathit{A})})-{({\mathit{V}}^T{\mathbf{\Theta }}^0)}_{r(\mathit{A})}\Vert }_F^2+{\Vert {({\mathit{V}}^T{\mathbf{\Theta }}^0)}_{r{(\mathit{A})}^c}\Vert }_F^2\le \frac{1}{{\sigma }_{\text{min}}^{2}n}{\Vert {\wp }_{r_0}({\mathit{W}}_{r(\mathit{A})})-{\mathit{D}}_{r(\mathit{A})}{({\mathit{V}}^T{\mathbf{\Theta }}^0)}_{r(\mathit{A})}\Vert }_F^2+{\Vert {({\mathit{V}}^T{\mathbf{\Theta }}^0)}_{r{(\mathit{A})}^c}\Vert }_F^2,$$

where σ_r(A)(n^−1/2 A) = n^1/2 σ_min. If p = r(A), then the last term vanishes. Thus

$${\Vert {\mathbf{\Theta ̂}}^{r_0}-{\mathbf{\Theta }}^0\Vert }_F^2\le \frac{1}{{\sigma }_{r{(\mathit{A})}^n}^2}{\Vert {\wp }_{r_0}({\mathit{W}}_{r(\mathit{A})})-{\mathit{D}}_{r(\mathit{A})}{({\mathit{V}}^T{\mathbf{\Theta }}^0)}_{r(\mathit{A})}\Vert }_F^2\le \frac{4{\sum }_{j=1}^{r_0}{\sigma }_j^2}{{\sigma }_{\text{min}}^{2}n}.$$

Finally, if r(A) ≥ r₀, $𝔼{\Vert {\mathbf{\Theta ̂}}^{r_0}-{\mathbf{\Theta }}^0\Vert }_F^2\le \frac{4}{{\sigma }_{r(\mathit{A})}^2}𝔼({\sum }_{j=1}^{r_0}{\sigma }_j^2)$ and $𝔼{\Vert \mathit{A}{\mathbf{\Theta ̂}}^{r_0}-\mathit{A}{\mathbf{\Theta }}^0\Vert }_F^2\le 4𝔼({\sum }_{j=1}^{r_0}{\sigma }_j^2)$. In particular, if r(A) = r₀, then

$$𝔼{\Vert {\mathbf{\Theta ̂}}^{r_0}-{\mathbf{\Theta }}^0\Vert }_F^2=\frac{1}{{\sigma }_{\text{min}}^2}\frac{r_{0}k}{n}𝔼{\Vert \mathit{A}{\mathbf{\Theta ̂}}^{r_0}-\mathit{A}{\mathbf{\Theta }}^0\Vert }_F^2=\frac{r_{0}k}{n}.$$

7. EMPIRICAL EVALUATIONS

This section examines effectiveness of the proposed method and compares it with the nuclear-norm regularization as well as the SVP method [17]. One benefit of our method is that it can evaluate the approximation quality of the inexact ones. For the SVP, we choose a default initial value 0 for this local method since no other choices are guaranteed to deliver better performance. Our numerical experiment, which is not reported in here, suggests that the SVP method is indeed sensitive to the choice of the initial value. For nuclear-norm regularization, we select a suitable regularization parameter value giving the solution satisfying the rank constraint in (3).

7.1 Synthetic Data

Simulations are performed under (2). First, the n × p design matrix A is sampled, with each entry being iid N(10, 1). Second, the p×k truth Θ⁰ is generated by multiplying a p×r matrix and an r × k matrix, each entry of which has a normal distribution over N(10, 1). The data matrix Z is then sampled according to (2) with e following iid N(0, σ²) with σ = 0.5.

For predictive performance and rank recovery, we compute the MSE ${\Vert \mathit{A}({\mathbf{\Theta }}^0-{\mathbf{\Theta ̂}}^{\mathrm{r̂}})\Vert }_F^2$, the absolute difference |r̂ − r₀| and record the training time for each method, averaged over 100 simulation replications on a test set of size 10n, where r̂ is tuned over integers in [0, min(n, p)] by an independent tuning set of size 2n. We consider three possible situations, i.e., when k = r₀ < p, k > p > r₀ and p > k > r₀. To illustrate our theoretical conclusion in Theorem 4 that the prediction error bound does not depend on the value of p, we add one more case of p > k > r₀ with a larger value of p. The simulation results are summarized in Tables 2–4.

Table 2.

Prediction results for the synthetic data: averaged MSEs as well as their standard deviations, for three competing methods based on the selected tuning parameters over 100 simulation replications. Our, SVP, and Nuclear-Norm refer to our method, the SVP method and nuclear-norm regularization method.

Set-up	Our	SVP	Nuclear-Norm
k = r0 < p, 5 = 5 < 99	0.0110 (0.0065)	0.4591 (0.0386)	0.4130 (0.0360)
k > p > r0, 50 > 40 > 5	12.0010 (0.6125)	10166.5052 (1236.3237)	7709.0779 (960.6982)
p > k > r0, 50 > 40 > 5	13.1945 (0.5682)	10185.7699 (1132.3515)	8116.2653 (943.6058)
p > k > r0, 99 > 40 > 5	121.4019 (70.3054)	14597.0728 (1185.7200)	13153.3217 (1056.5155)

Table 4.

Run time for the synthetic data: average training time in seconds for three competing methods based on the selected tuning parameters over 100 simulation replications. Our, SVP, and Nuclear-Norm refer to our method, the SVP method and nuclear-norm regularization.

Set-up	Our	SVP	Nuclear-Norm
k = r0, < p, 5 = 5 < 99	0.0224 (0.0045)	14.1523 (0.3933)	0.9766 (0.0074)
k > p > r0, 50 > 40 > 5	0.0349 (0.0031)	41.2122 (1.8511)	5.8937 (0.2945)
p > k > r0, 50 > 40 > 5	0.0368 (0.0025)	29.5856 (1.1786)	5.7447 (0.1173)
p > k > r0, 99 > 40 > 5	0.0542 (0.0039)	79.9850 (3.6724)	7.6046 (0.1011)

As suggested in Tables 2 and 3, our exact method is much more precise in prediction in all cases and rank recovery except one case of k = r₀ = 5 < p = 99, than the other two methods. This is in agreement with the theoretical results in Theorem 4 that exact reconstruction of the oracle estimator is achieved through tuning. Note that the best MSE value does not necessarily yield best rank recovery, as in the case of k = r₀ = 5 < p = 99, which is due to the bias/variance trade-off phenomenon. As indicated in Table 4, our method is, on average, 10–20 times faster than the other two.

Table 3.

Rank recovery for the synthetic data: averaged values of |r̂ − r₀| as well as their standard deviations, for three competing methods based on the selected tuning parameters over 100 simulation replications. Our, SVP, and Nuclear-Norm refer to our method, the SVP method and nuclear-norm regularization method.

Set-up	Our	SVP	Nuclear-Norm
k = r0 < p, 5 = 5 < 99	0.8400 (0.8005)	0.0000 (0.0000)	0.0000 (0.0000)
k > p > r0, 50 > 40 > 5	0.0000 (0.0000)	3.5300 (1.6358)	1.6400 (0.6439)
p > k > r0, 50 >40 > 5	0.0000 (0.0000)	4.0800 (1.3830)	1.6500 (0.5925)
p > k > r0, 99 > 40 > 5	0.0000 (0.0000)	2.9800 (1.9121)	0.2200 (0.4399)

7.2 MIT logo Recovery

Next we examine the three competing methods for reconstructing the MIT logo image, which was studied in [27, 17]. The original logo is displayed in Figure 2, where we use the gray image of size 44 × 85 and the image matrix has rank 7.

Figure 2.

Original MIT logo image.

Our objective is to reconstruct this image from its noisy version and examine the quality of reconstruction. Towards this end, we sample the design matrix A with each entry being iid N(0, 1), where the sample size n ranges from 20 to 80. To generate a noisy version, we add random error sampled from N(0, 0.5²) to each element of the sample data. The reconstruction results are displayed in Figure 3 for the three methods with the default initial value 0 for the SVP.

Figure 3.

Reconstruction of a noisy version of the MIT logo with varying sampling size n. From left to right (for each case of n): SVP; nuclear-norm regularization; our method.

Visually, our method delivers highly competitive performance as compared to the other two methods, as displayed in Figure 3, and yields nearly perfect reconstruction when the size of the design matrix A becomes larger, say n = 60 and n = 80. For all the methods, better reconstruction can be reached as n increases, and comparable results are achieved when n is small, say n = 20 and n = 40. This conclusion is consistent with our theoretical analysis.

In practice, the exact rank of the matrix to be estimated is unknown but may reasonably be assumed to be small. In this sense, s needs to be tuned or estimated. Next we investigate the effect of estimation of s with regard to the reconstruction quality, measured by the MSE and the recovery rate. The latter is defined as the ratio ‖Θ̂^s − Θ⁰‖_F/‖Θ⁰‖_F, commonly used to measure the quality of parameter estimation. In particular, we display both of them as a function of s in Figures 4 and 5, where s is defined in (3).

Figure 4.

Relative recovery error as a function of the sampling size for the MIT logo image under different rank constraints.

Figure 5.

MSE as a function of the sampling size for the MIT logo image under different rank constraints. Note that the MSE of our method in (b) is not zero but indistinguishable from zero, which is unlike (c) and (d) in which the MSEs are identical to zero.

For the relative recovery error, a clear transition of our solution occurs around s = 44, after which a perfect recovery is achieved, whereas no improvement occurs for the other two methods when s increases. In the case of an underdetermined A, i.e., A is not of full column rank, all three methods produce similar recovery results. For the MSEs, our method yields the perfect result zero when s ≥ 7 and a reasonably small value when s = 3, 5, whereas the other two methods lead to elevated MSEs as s increases. This is in accordance with our theory which suggests that our method constructs the oracle estimator giving perfect image reconstruction when s ≤ 7.

8. CONCLUSION

This paper considers a nonconvex least squares formulation based on the rank constraint/regularization. We establish the optimality of the global solution, against any other ones. Experimental results on synthetic and real data demonstrate the efficiency and effectiveness of the proposed algorithm. In future work, we plan to expand the present work to a general loss function.