Reduced rank regression via adaptive nuclear norm penalization

Summary

We propose an adaptive nuclear norm penalization approach for low-rank matrix approximation, and use it to develop a new reduced rank estimation method for high-dimensional multivariate regression. The adaptive nuclear norm is defined as the weighted sum of the singular values of the matrix, and it is generally non-convex under the natural restriction that the weight decreases with the singular value. However, we show that the proposed non-convex penalized regression method has a global optimal solution obtained from an adaptively soft-thresholded singular value decomposition. The method is computationally efficient, and the resulting solution path is continuous. The rank consistency of and prediction/estimation performance bounds for the estimator are established for a high-dimensional asymptotic regime. Simulation studies and an application in genetics demonstrate its efficacy.

1. Introduction

Given n observations of the response y_i ∈ ℜ^q and predictor x_i ∈ ℜ^p, we consider the multi-variate linear regression model

$$Y={XC}_0+E,$$ (1)

where Y = (y₁, …, y_n)^T, X = (x₁, …, x_n)^T, C₀ is an p × q coefficient matrix, and E = (e₁, …, e_n)^T is an n × q matrix of independently and identically distributed random errors with mean zero and variance σ². Throughout, we write p ∧ q = min(p, q), n ∧ q = min(n, q), r^* = r(C₀) and r_x = r(X), where r(·) denotes the rank of a matrix.

We consider the scenario in which both the predictor dimension p and response dimension q may depend on and even exceed the sample size n. Such high-dimensional regression problems are increasingly encountered. Ordinary least squares estimation is equivalent to separately regressing each response on the set of predictors, but this ignores the dependence structure of the multivariate response and may be infeasible in high-dimensional settings. The curse of dimensionality can be mitigated by assuming that the true coefficient matrix C₀ has some low-dimensional structure and employing regularization/penalization approaches for model estimation. For Gaussian data, it is appropriate to estimate C₀ by minimizing the penalized least squares criterion

$$\frac{1}{2}\mathcal{J}(C)+{\mathcal{P}}_{\lambda }(C)$$ (2)

with respect to C ∈ ℜ^p×q, where $\mathcal{J}(C)={\Vert Y-XC\Vert }_F^2$ is the sum of squared errors, with ||·||_F denoting the Frobenius norm, (·) is some penalty function measuring the complexity of the enclosed matrix, and λ is a non-negative tuning parameter controlling the penalty.

Within this general framework, an important model is reduced rank regression (Anderson, 1951, 1999, 2002; Izenman, 1975; Reinsel & Velu, 1998), in which dimension reduction is achieved by constraining the coefficient matrix to have low rank. The classical small-p case and maximum likelihood inference for the rank-constrained approach have been extensively investigated. Recently, Bunea et al. (2011) proposed a rank selection criterion that is valid for high dimensional settings, revealing that rank-constrained estimation can be viewed as a penalized regression method (2) with a penalty proportional to the rank of C. The penalty can also be cast as an l₀ penalty in terms of the number of non-zero singular values of C, i.e., ${\mathcal{P}}_{\lambda }(C)=\lambda r(C)=\lambda {\sum }_{i=1}^{p\wedge q}I\{d_i(C)\ne 0\}$, where I(·) is the indicator function, and d_i(·) represents the ith largest singular value of a matrix. This results in an estimator obtained by hard-thresholded singular value decomposition; see Section 2. Yuan et al. (2007) proposed a nuclear norm penalized least squares criterion, in which the penalty is defined as ${\mathcal{P}}_{\lambda }(C)=\lambda {\Vert C\Vert }_{\ast }=\lambda {\sum }_{i=1}^{p\wedge q}{d}_i(C)$, where ||·||_* denotes the nuclear norm. This l₁ penalty encourages sparsity among the singular values and achieves simultaneous rank reduction and shrinkage estimation (Negahban & Wainwright, 2011; Bunea et al., 2011; Lu et al., 2012). Rohde & Tsybakov (2011) investigated the theoretical properties of the Schatten-b quasi-norm penalty, which is defined as ${\mathcal{P}}_{\lambda }(C)=\lambda {\sum }_{i=1}^{p\wedge q}{d}_i^b(C)$for 0 < b ≤ 1, and obtained non-asymptotic bounds for prediction risk. Several other extensions and theoretical developments related to reduced rank estimation exist; see, e.g., Aldrin (2000), Negahban & Wainwright (2011), Mukherjee & Zhu (2011), and Chen et al. (2012). Reduced rank methodology has connections with many popular tools including principal component analysis and canonical correlation analysis, and has been extensively studied in matrix completion problems (Candès & Recht, 2009; Candès et al., 2011; Koltchinskii et al., 2011).

The aforementioned reduced rank approaches are closely related to the singular value decomposition (Eckart & Young, 1936; Reinsel & Velu, 1998). It is intriguing that the rank and nuclear norm penalization approaches can be viewed as l₀ and l₁ singular value penalization methods, respectively. Moreover, the squared l₂ singular value penalty is in fact a ridge penalty, because ${\sum }_{i=1}^{p\wedge q}{d}_i^2(C)={\Vert C\Vert }_F^2$. Motivated by these connections and with a desire to close the gap between the l₀ and l₁ penalization schemes, we propose the adaptive nuclear norm regularization method. The adaptive nuclear norm of a matrix C ∈ ℜ^p×q is defined as a weighted sum of its singular values:

$${\Vert C\Vert }_{\ast w}=\sum _{i=1}^{p\wedge q}{w}_i{d}_i(C),$$ (3)

where w_is are the non-negative weights; a similar idea can be found in an unpublished 2009 University of Illinois manuscript by Jiaming Xu. We show that the adaptive nuclear norm is non-convex when the weight of the singular value decreases with the singular value, a condition needed for a meaningful regularization; see Section 2. Despite the non-convexity, we show below that the adaptive nuclear norm penalized estimator has a closed-form solution in matrix approximation problems.

Based on the proposed adaptive nuclear norm, we develop a new method for conducting simultaneous dimension reduction and coefficient estimation in high-dimensional multivariate regression. Our proposal combines two main ideas. Firstly, the proposed method builds a bridge between the l₀ and l₁ singular value penalization methods, and it can be viewed as analogous to the adaptive lasso (Tibshirani, 1996; Zou, 2006; Huang et al., 2008) developed for univariate regression. Secondly, we penalize XC rather than C, which allows the reduced rank estimation problem to be solved explicitly and efficiently. This setup was used by Klopp (2011) and Koltchinskii et al. (2011) in trace regression problems. Compared to the computationally intensive l₁ method (Yuan et al., 2007) which tends to overestimate the rank, the proposed method with the aid of some well-chosen adaptive weights may improve rank determination. Compared to the discontinuous l₀ method (Bunea et al., 2011), the proposed method results in a continuous solution path and allows a more flexible bias-variance tradeoff in model fitting.

2. Adaptive nuclear norm penalty

The adaptive nuclear norm ||C||_*w forms a rich class of penalty functions indexed by the weights. Clearly, it includes the nuclear norm as a special case with unit weights. Since the nuclear norm is convex and is a matrix norm, an immediate question arises as to whether or not its weighted extension (3) preserves the convexity, which is the case for the entrywise lasso and adaptive lasso penalties (Zou, 2006). However, the following theorem shows that the convexity of (3) depends on the ordering of the non-negative weights.

Theorem 1

For any matrix C ∈ ℜ^p×q, let f(C) = ||C||_*w be defined in (3). Then f(·) is convex in C if and only if w₁ ≥ ··· ≥ w_p∧q ≥ 0.

Hence, for the adaptive nuclear norm (3) to be a convex function, the weights must be nondecreasing with the singular value. However, for penalized estimation, the opposite is desirable, i.e., we would and shall henceforth impose the order constraint

$$0\le w_1\le \cdots \le w_{p\wedge q},$$ (4)

which ensures that a larger singular value receives a lighter penalty to help reduce the bias and a smaller singular value receives a heavier penalty to help promote sparsity. The non-convexity of f(·) arises from the constraint (4), so f(·) is no longer a matrix norm. Here is an example showing that (3) is neither convex nor concave under constraint (4). Consider p = q = 2, and

$$C_1=\left(\begin{array}{cc}2& 0\\ 0& 1\end{array}\right),C_2=\left(\begin{array}{cc}1& 0\\ 0& 2\end{array}\right).$$

Let w₁ = 1 and w₂ = 2. It can be verified that f(C₁) = f(C₂) = f (−C₂) = 4, while f{(C₁ + C₂)/2} = 4·5 > {f(C₁) + f(C₂)}/2; also, f{(C₁ − C₂)/2} = 1·5 < {f(C₁) + f(−C₂)}/2.

Consider the low-rank matrix approximation problem, Y = C₀ + E, a special case of model (1) where X is an identity matrix and C₀ is n × q. This model provides a framework for denoising the data matrix Y. Certain low-rank estimators of C₀ can be derived from the singular value decomposition of Y ∈ ℜ^n×q,

$$Y={\mathit{UDV}}^{\mathrm{T}},D=\text{diag}\{d(Y)\},$$ (5)

where U and V are respectively n × (n ∧ q) and q × (n ∧ q) orthonormal matrices, d(Y) = {d₁(Y), …, d_n∧q(Y)}^T consists of the singular values of Y in descending order, and diag(·) denotes a diagonal matrix with the enclosed vector on its diagonal. Consider the following two kinds of estimators of C₀,

$${\mathcal{H}}_{\lambda }(Y)=U{\mathcal{H}}_{\lambda }(D)V^{\mathrm{T}},{\mathcal{H}}_{\lambda }(D)=\text{diag}[d_i(Y)I\{d_i(Y)>\lambda \},i=1,\dots ,n\wedge q],$$ (6)

$${\mathcal{S}}_{\lambda }(Y)=U{\mathcal{S}}_{\lambda }(D)V^{\mathrm{T}},{\mathcal{S}}_{\lambda }(D)=\text{diag}[{\{d_i(Y)-\lambda \}}_{+},i=1,\dots ,n\wedge q],$$ (7)

where λ ≥ 0 and x₊ = max(0, x). The following proposition shows that the hard-thresholding estimator in (6) yields the l₀ rank penalized estimator (Eckart & Young, 1936), and the soft-thresholding estimator in (7) yields the l₁ nuclear norm penalized estimator (Cai et al., 2010).

Proposition 1

For any λ ≥ 0 and Y ∈ ℜ^n×q, (Y) defined by (6) can be characterized as ${\mathcal{H}}_{\lambda }(Y)=arg{min}_C\{{\Vert Y-C\Vert }_F^2+{\lambda }^{2}r(C)\}$, and (Y) in (7) can be characterized as ${\mathcal{S}}_{\lambda }(Y)=arg{min}_C\{{\Vert Y-C\Vert }_F^2/2+\lambda {\Vert C\Vert }_{\ast }\}$.

The estimator (Y) eliminates any singular values below a threshold λ, while (Y) shrinks all the singular values by λ towards zero. These operations are natural extensions of the hard/soft-thresholding rules for scalars and vectors (Donoho & Johnstone, 1995; Cai et al., 2010). In general, a soft-thresholding estimator has smaller variance but larger bias than its hard-thresholding counterpart. Soft-thresholding may, however, be preferable when data are noisy and highly correlated (Donoho & Johnstone, 1995).

The preceding discussion on the connections between different thresholding rules and penalty terms motivates us to consider the use of the adaptive nuclear norm in bridging the gap between the l₀ and l₁ singular value penalties and fine-tuning the bias-variance tradeoff. In the context of low-rank matrix approximation, we are able to obtain, with complete characterization, an explicit global minimizer of the least squares criterion penalized by a non-convex adaptive nuclear norm.

Theorem 2

For any λ ≥ 0, 0 ≤ w₁ ≤ ··· ≤ w_n∧q, and Y ∈ ℜ^n×q with a singular value decomposition Y = UDV^T, a global optimal solution to the optimization problem

$$\underset{C}{min}\{\frac{1}{2}{\Vert Y-C\Vert }_F^2+\lambda {\Vert C\Vert }_{\ast w}\}$$ (8)

is (Y), where

$${\mathcal{S}}_{\lambda w}(Y)=U{\mathcal{S}}_{\lambda w}(D)V^{\mathrm{T}},{\mathcal{S}}_{\lambda w}(D)=\mathit{diag}[{\{d_i(Y)-\lambda w_i\}}_{+},i=1,\dots ,n\wedge q].$$ (9)

Further, if all the nonzero singular values of Y are distinct, then (Y) is the unique optimal solution.

The fact that a closed-form global minimizer can be found for the non-convex problem (8) is rather surprising. The result stems from the von Neumann’s trace inequality (Mirsky, 1975); see the Appendix. Following Zou (2006), the weights can be set to be $w_i=d_i^{-\gamma }(Y)$, for i = 1, …, n ∧ q, where γ ≥ 0 is a pre-specified constant. In this way, the order constraint (4) is automatically satisfied. We discuss a general way to construct the weights in Section 7.

3. Adaptive nuclear norm penalization in multivariate regression

3·1. Rank and nuclear norm penalized regression methods

We now consider estimating the coefficient matrix C₀, which is possibly of low rank, in the multivariate regression model (1) with an arbitrary design matrix X. Below, let P = X(X^TX)⁻X^T be the projection matrix onto the column space of X and Ĉ_L = (X^T X)⁻X^T Y the least squares estimator of C₀, where (·)⁻ denotes a Moore–Penrose inverse.

The results in Section 2 can be readily applied to derive certain low-rank estimator of C₀ in the general regression setting. First, consider the rank selection criterion (Bunea et al., 2011),

$${\Vert Y-XC\Vert }_F^2+{\lambda }^{2}r(C).$$ (10)

Minimizing (10) is the same as minimizing $L_1(C)={\Vert X{\widehat{C}}_{\mathrm{L}}-XC\Vert }_F^2+{\lambda }^{2}r(C)$, owing to Pythagoras’ theorem. We now demonstrate that this is also equivalent to minimizing $L_2(C)={\Vert X{\widehat{C}}_{\mathrm{L}}-XC\Vert }_F^2+{\lambda }^{2}r(XC)$, which can be cast as a constrained matrix approximation problem,

$$\underset{M}{min}\{{\Vert X{\widehat{C}}_{\mathrm{L}}-M\Vert }_F^2+{\lambda }^{2}r(M)\}\text{subject}\text{to}M=XC\text{for}\text{some}C.$$ (11)

However, the constrained minimizer of (11) turns out to be the same as its unconstrained counterpart. To see this, let V̂D̂²V̂^T be the eigenvalue decomposition of Y^TPY = (XĈ_L)^TXĈ_L. The singular value decomposition of XĈ_L is then given by ÛD̂V̂^T, where Û = PYV̂D̂⁻ = XĈ_LV̂D̂⁻. It follows from Proposition 1 that the unconstrained minimizer equals

$${\mathcal{H}}_{\lambda }(X{\widehat{C}}_{\mathrm{L}})=\widehat{U}{\mathcal{H}}_{\lambda }(\widehat{D}){\widehat{V}}^{\mathrm{T}}=X{\widehat{C}}_{\mathrm{L}}\widehat{V}{\widehat{D}}^{-}{\mathcal{H}}_{\lambda }(\widehat{D}){\widehat{V}}^T=X{\widehat{C}}_{\mathrm{H}}^{(\lambda )},$$ (12)

where ${\widehat{C}}_{\mathrm{H}}^{(\lambda )}={\widehat{C}}_{\mathrm{L}}\widehat{V}{\widehat{D}}^{-}{\mathcal{H}}_{\lambda }(D){\widehat{V}}^{\mathrm{T}}$. Therefore, $X{\widehat{C}}_{\mathrm{H}}^{(\lambda )}$ is the desired constrained minimizer of (11). Moreover, $r\{{\mathcal{H}}_{\lambda }(\widehat{D})\}=r(X{\widehat{C}}_{\mathrm{H}}^{(\lambda )})\le r({\widehat{C}}_{\mathrm{H}}^{(\lambda )})\le r\{{\mathcal{H}}_{\lambda }(\widehat{D})\}$, i.e., $r(X{\widehat{C}}_{\mathrm{H}}^{(\lambda )})=r({\widehat{C}}_{\mathrm{H}}^{(\lambda )})$. Consequently, $L_1({\widehat{C}}_{\mathrm{H}}^{(\lambda )})=L_2({\widehat{C}}_{\mathrm{H}}^{(\lambda )})\le L_2(C)\le L_1(C)$, for any C ∈ ℜ^p×q, i.e., ${\widehat{C}}_{\mathrm{H}}^{(\lambda )}$ minimizes (10). It can also be shown that the set of rank-constrained estimators, obtained by minimizing ${\Vert Y-XC\Vert }_F^2$ subject to r(C) ≤ r, for r = 1, …, p ∧ q, spans the solution path of (10) (Reinsel & Velu, 1998).

The nuclear norm penalized least squares criterion (Yuan et al., 2007) is

$$\frac{1}{2}{\Vert Y-XC\Vert }_F^2+\lambda {\Vert C\Vert }_{\ast }.$$ (13)

We denote the minimizer of (13) by ${\widehat{C}}_{\mathrm{N}}^{(\lambda )}$, which generally does not have an explicit form. Extensive research has been devoted to this minimization problem (Cai et al., 2010; Toh & Yun, 2010). One popular algorithm is to alternate between a majorization step of the objective function and a minimization step by singular value soft-thresholding operation until convergence, but it is computationally intensive for large-scale data (Cai et al., 2010).

3·2. Adaptive nuclear norm penalized regression method

Predictive accuracy and computational efficiency are both important in high dimensional regression problems. Motivated by criteria (10) and (13), we propose to estimate C₀ by minimizing

$$\frac{1}{2}{\Vert Y-XC\Vert }_F^2+\lambda {\Vert XC\Vert }_{\ast w},$$ (14)

where the weights {w_i} are required to be non-negative and non-decreasing. In practice, a fore-most task of using (14) is setting proper adaptive weights. Following Zou (2006), this can be based on the least squares solution,

$$w_i=d_i^{-\gamma }(PY)=d_i^{-\gamma }(X{\widehat{C}}_{\mathrm{L}}),$$ (15)

where PY = XĈ_L is the projection of Y onto the column space of X and γ is a non-negative constant.

The proposed method (14) is built on two main ideas. Firstly, the criterion focuses on the fitted values XC, and encourages sparsity among the singular values of XC rather than those of C. This may yield a low-rank estimator for XC₀ and hence for C₀ (Koltchinskii et al., 2011). A prominent advantage of this setup is that the problem can then be solved explicitly and efficiently. Secondly, the adaptive penalization of the singular values allows a flexible bias-variance tradeoff: a large singular value receives a small penalty to control possible bias, and a small singular value receives a large penalty to induce sparsity and hence reduce the rank. The following corollary shows that this criterion admits an explicit minimizer.

Corollary 1

A minimizer of (14), denoted by ${\widehat{C}}_S^{(\lambda w)}$, is given by

$$X{\widehat{C}}_S^{(\lambda w)}={\mathcal{S}}_{\lambda w}(X{\widehat{C}}_L)=\widehat{U}{\mathcal{S}}_{\lambda w}(\widehat{D}){\widehat{V}}^{\mathrm{T}},{\widehat{C}}_S^{(\lambda w)}={\widehat{C}}_L\widehat{V}{\widehat{D}}^{-}{\mathcal{S}}_{\lambda w}(\widehat{D}){\widehat{V}}^{\mathrm{T}},$$ (16)

where Ĉ_L is the least squares estimator of C₀, ÛD̂V̂^T is the singular value decomposition of XĈ_L, and (·) is defined in (9).

By Pythagoras’ theorem, minimizing criterion (14) is equivalent to minimizing ${\Vert X{\widehat{C}}_{\mathrm{L}}-XC\Vert }_F^2/2+\lambda {\Vert XC\Vert }_{\ast w}$ with respect to C. The above result then follows directly from Theorem 2. The proposed method first projects Y onto the column space of X, i.e., PY = XĈ_L; the estimator is then obtained as a low-rank approximation of PY by adaptively soft-thresholding the singular values. The thresholding level is data-driven: the smaller an initially estimated singular value, the larger its thresholding level. Therefore, the estimated rank corresponds to the smallest singular value of PY that exceeds its thresholding level, i.e., r̂ = max{r : d_r(PY) > λw_r}. For the choice of the weights in (15), the estimated rank is

$$\widehat{r}=max\{r:d_r(PY)>{\lambda }^{1/(\gamma +1)}\},$$ (17)

for $0\le \lambda <d_1^{\gamma +1}(PY)$ and r̂ = 0 for $\lambda \ge d_1^{\gamma +1}(PY)$. Therefore, the plausible range of the tuning parameter is $\lambda \in [0,d_1^{\gamma +1}(PY)]$, with λ = 0 corresponding to the least squares solution and $\lambda =d_1^{\gamma +1}(PY)$ to the null solution.

The proposed estimator ${\widehat{C}}_{\mathrm{S}}^{(\lambda w)}$ and the l₀ estimator ${\widehat{C}}_{\mathrm{H}}^{(\lambda )}$ in (12) differ only in their estimated singular values for XC₀, but the difference can be consequential. While the solution path of the l₀ method is discontinuous and the number of possible solutions equals the maximum rank, the proposed criterion offers more flexibility in that the resulting solution path is continuous and guided by the data-driven weights. The two methods can both be efficiently computed, in contrast to the computationally intensive l₁ method in (13).

For any fixed λ ≥ 0, ${\widehat{C}}_{\mathrm{S}}^{(\lambda w)}$ can be computed by (16). To choose an optimal λ and hence an optimal solution, K-fold cross validation method can be used, based on predictive performance of the models (Stone, 1974). In our numerical studies, we first compute the solutions over a grid of 100 λ values equally spaced on the log scale and select the best λ value; subsequently we refine the selection process around the chosen λ value with a finer grid of 100 λ values.

4. Rank consistency and error bounds

We study the rank estimation and prediction properties of the proposed adaptive nuclear norm penalized regression method. Our theoretical analysis is built on the framework developed by Bunea et al. (2011). We mainly focus on the random weights constructed in (15), in line with the adaptive lasso method (Zou, 2006) developed for univariate regression. Similar results are obtained for any pre-specified sequence of non-random weights satisfying certain order restriction and boundedness requirements. All the proofs are given in the Appendix.

The rank of the coefficient matrix C₀, denoted as r^*, can be viewed as the number of effective linear combinations of the predictors linked to the responses. Rank determination is always a foremost task of reduced rank estimation. The quality of the rank estimator, given in (17), clearly depends on the signal to noise ratio. Following Bunea et al. (2011), we shall use the r^*th largest singular value of XC₀, i.e., d_r^*(XC₀), to measure the signal strength, and use the largest singular value of the projected noise matrix PE, i.e., d₁(PE), to measure the noise level. Intuitively, if d₁(PE) is much larger than the size of the signal, some of the signal could be masked by the noise and lost during the thresholding procedure; as such, r̂ may be much smaller than the true rank. The lemma below characterizes the limit or the true target of r̂ and its relationship with the signal and noise levels, as well as the adaptive weights.

Lemma 1

Suppose that there exists an index s ≤ r^* such that d_s(XC₀) > (1 + δ)λ^1/(γ+1) and d_s+1(XC₀) ≤ (1 − δ)λ^1/(γ+1) for some δ ∈ (0, 1]. Then pr(r̂ = s) ≥ 1 − pr{d₁(PE) ≥ δλ^1/(γ+1)}, where r^* is the rank of C₀, r̂ the estimated rank given in (17), P the projection matrix onto the column space of X, E the error matrix in model (1), and γ the power parameter in the adaptive weights (15).

To achieve consistent rank estimation, we consider the following assumptions:

• Assumption 1. The error matrix E has independent N(0, σ²) entries.

• Assumption 2. For any θ > 0, λ = {(1 + θ)σ(√r_x + √q)/δ}^γ+1 with δ defined in Lemma 1, and d_r^*(XC₀) > 2λ^1/(γ+1).

Assumption 1 ensures that the noise level d₁(PE) is of order √r_x + √q; see Lemma 2 in the Appendix (Bunea et al., 2011). Assumption 2 concerns the signal strength relative to the noise level and the appropriate rate of the tuning parameter.

Theorem 3

Suppose Assumptions 1–2 hold. Let r̂ be the estimated rank given in (17), and r_x = r(X) the rank of X. Then pr(r̂ = r^*) → 1 as r_x + q → ∞.

Theorem 3 shows that the proposed estimator is able to identify the correct rank with probability tending to 1 as r_x + q goes to infinity. As in Bunea et al. (2011), the consistency results can be extended to the case of sub-Gaussian errors and can also be easily adapted to the case when r_x + q is bounded and the sample size n goes to infinity. The rank consistency of the proposed estimator is thus valid for both classical and high-dimensional asymptotic regimes.

Our main results about the prediction performance of the proposed estimator are presented in Theorem 4. For simplicity, we write Ĉ_S for ${\widehat{C}}_{\mathrm{S}}^{(\lambda w)}$.

Theorem 4

Suppose Assumptions 1–2 hold. Let c = d₁(XC₀)/d_r^*(XC₀) ≥ 1. Then

$${\Vert X{\widehat{C}}_S-{XC}_0\Vert }_F^2\le \frac{1+a}{1-a}{\Vert XB-{XC}_0\Vert }_F^2+\frac{1}{a(1-a)}{\{\delta \surd 2+2{(2-\delta )}^{-\gamma }-{(2c+\delta )}^{-\gamma }\}}^2{\lambda }^{2/(\gamma +1)}{r}^{\ast },$$

with probability greater than 1 − exp{−θ²(r_x + q)/2}, for any 0 < a < 1 and any p × q matrix B with r(B) ≤ r^*. Moreover, taking B = C₀ and a = 1/2 yields

$$\begin{array}{l}{\Vert X{\widehat{C}}_S-{XC}_0\Vert }_F^2\le 4{\{\delta \surd 2+2{(2-\delta )}^{-\gamma }-{(2c+\delta )}^{-\gamma }\}}^2{\lambda }^{2/(\gamma +1)}{r}^{\ast }\\ =4{\{\surd 2+2{(2-\delta )}^{-\gamma }/\delta -{(2c+\delta )}^{-\gamma }/\delta \}}^2{(1+\theta )}^2{\sigma }^2{(\surd r_x+\surd q)}^2{r}^{\ast }\end{array}$$ (18)

with probability greater than 1 − exp{−θ²(r_x + q)/2}.

The established bound in (18) shows that the prediction error is bounded by $d_1^2(PE)r^{\ast }$ up to some multiplicative constant, with probability 1 − exp{−θ²(r_x + q)/2}, i.e., the smaller the noise level or the true rank, the smaller the prediction error. The bound is valid for any X and C₀. The estimation error bound of Ĉ_S can also be readily derived from Theorem 4, e.g., if d_rx(X) ≥ ρ > 0 for some constant ρ, then under Assumptions 1–2, ${\Vert {\widehat{C}}_{\mathrm{S}}-C_0\Vert }_F^2\le 4{\rho }^{-2}{\{\delta \surd 2+2{(2-\delta )}^{-\gamma }-{(2c+\delta )}^{-\gamma }\}}^2{\lambda }^{2/(\gamma +1)}{r}^{\ast }$.

So far, we have considered random weights based on the least squares solution, as given in (15). We now briefly outline the results for any pre-specified sequence of non-negative and nondecreasing, non-random weights {w_i; i = 1, …, n ∧ q}.

Corollary 2

Suppose Assumption 1 holds, and that (i) 0 ≤ w₁ ≤ ··· ≤ w_n∧q, and there exists 0 < m < ∞ such that w_r^* ≤ m ≤ w_r^*+1, (ii) the tuning parameter λ = (1 + θ)σ(√r_x + √q)/m and (iii) d_r^*(XC₀) > 2λm. Then

1. pr(r̂ = r^*) → 1 as r_x + q → ∞; and

2. ${\Vert X{\widehat{C}}_S-{XC}_0\Vert }_F^2\le 4{(\surd 2+2-w_1/m)}^2{(1+\theta )}^2{(\surd r_x+\surd q)}^2{r}^{\ast }$ with probability greater than 1 − exp{−θ²(r_x + q)/2}.

The proof is similar to that of Theorems 3 and 4 and hence is omitted.

The error bounds of the proposed estimator established in Theorem 4 and Corollary 2 are comparable to those of the l₀ rank penalized estimator and the l₁ nuclear norm penalized estimator (Bunea et al., 2011; Rohde & Tsybakov, 2011). The rate of convergence is (r_x + q)r^* because (√r_x + √q)² ≤ 2(r_x + q), which is the optimal minimax rate for rank sparsity under suitable regularity conditions (Rohde & Tsybakov, 2011; Bunea et al., 2012). However, the bounds for the nuclear norm penalized estimator were obtained with extra restrictions on the design matrix, and a tuning sequence for achieving the smallest mean squared error usually does not lead to correct rank recovery (Bunea et al., 2011). While both the rank selection criterion and the proposed method are able to achieve correct rank recovery and minimal mean squared error simultaneously, the latter possesses a continuous solution path produced by data-driven adaptive penalization, which may lead to improved empirical performance.

5. Robustification of reduced rank estimation

As suggested by a referee and motivated by Mukherjee & Zhu (2011), we discuss the robustification of the reduced rank methods by adding a ridge penalty. Mukherjee & Zhu (2011) proposed a reduced rank ridge regression method; see also Bunea et al. (2011) and She (2012). The shrinkage estimation induced by a ridge penalty makes the reduced rank estimation robust and hence is especially suitable when the predictors are highly correlated. The method can be viewed as minimizing the following criterion

$${\Vert Y-XC\Vert }_F^2+{\lambda }_1^{2}r(C)+{\lambda }_2{\Vert C\Vert }_F^2$$ (19)

where ${\Vert C\Vert }_F^2=\text{tr}(C^{\mathrm{T}}C)={\sum }_{i=1}^{p\wedge q}{d}_i^2(C)$, and λ₁ and λ₂ are tuning parameters. We denote the resulting robustified estimator by ${\widehat{C}}_{\text{HR}}^{({\lambda }_1,{\lambda }_2)}$, which can be obtained by data augmentation. Specifically, letting

$$Y^{\ast }=\left(\begin{array}{c}Y\\ 0_{p\times q}\end{array}\right),X^{\ast }=\left(\begin{array}{c}X\\ \surd {\lambda }_2{I}_{p\times p}\end{array}\right),$$

(19) can be written as a rank selection criterion ${\Vert Y^{\ast }-X^{\ast }C\Vert }_F^2+{\lambda }_1^{2}r(C)$, whose solution is given in (12); see Mukherjee & Zhu (2011).

The adaptive nuclear norm penalization method can also be robustified by incorporating a ridge penalty term. Similar to (14), for efficient computation, we impose a ridge penalty on XC rather than C,

$$\frac{1}{2}{\Vert Y-XC\Vert }_F^2+{\lambda }_1{\Vert XC\Vert }_{\ast w}+\frac{{\lambda }_2}{2}{\Vert XC\Vert }_F^2.$$ (20)

Interestingly, criterion (20) is analogous to the adaptive elastic net criterion (Zou & Zhang, 2009) in univariate regression. We denote the minimizer of (20) by ${\widehat{C}}_{\text{SR}}^{({\lambda }_{1}w,{\lambda }_2)}$. It can be verified that

$${\widehat{C}}_{\text{SR}}^{({\lambda }_{1}w,{\lambda }_2)}=\frac{1}{1+{\lambda }_2}{\widehat{C}}_{\mathrm{S}}^{({\lambda }_{1}w)},$$ (21)

where ${\widehat{C}}_{\mathrm{S}}^{({\lambda }_{1}w)}$, defined in (16), denotes the proposed estimator in the absence of the ridge penalty.

For each fixed λ₂, solving (19) requires inverting an p × p matrix (X^TX + λ₂I) and performing a singular value decomposition of an q × q matrix. When p is much greater than n, the Woodbury matrix identity is useful in speeding up computation (Hager, 1989), i.e., ${(X^{\mathrm{T}}X+{\lambda }_{2}I)}^{-1}={\lambda }_2^{-1}I-{\lambda }_2^{-2}{X}^{\mathrm{T}}{(I+{\lambda }_2^{-1}X{X}^{\mathrm{T}})}^{-1}X$. Following Mukherjee & Zhu (2011), in practice we use K-fold cross validation to determine the optimal rank and select the optimal λ₂ from a sequence of 100 values. On the other hand, obtaining the whole solution path of (20) only requires a one-time matrix inversion and singular value decomposition. We perform a 100 × 100 grid search of (λ₁, λ₂) to obtain the final estimator.

6. Empirical studies

6·1. Simulation

We compare the prediction, estimation and rank determination of various reduced rank estimators including the nuclear norm penalized estimator ${\widehat{C}}_{\mathrm{N}}^{(\lambda )}$ (Yuan et al., 2007), the rank penalized estimator ${\widehat{C}}_{\mathrm{H}}^{(\lambda )}$ (Bunea et al., 2011), and our proposed adaptive nuclear norm penalized estimator ${\widehat{C}}_{\mathrm{S}}^{(\lambda w)}$ with several choices of the weight parameter γ. The robustified versions, namely, ${\widehat{C}}_{\text{HR}}^{({\lambda }_1,{\lambda }_2)}$ and ${\widehat{C}}_{\text{SR}}^{({\lambda }_{1}w,{\lambda }_2)}$, are also considered. For simplicity, we suppress the superscripts from the notations of the various estimators. We used the accelerated proximal gradient algorithm implemented in Matlab by Toh & Yun (2010) for computing Ĉ_N. R code for computing Ĉ_HR was provided by the original authors (Mukherjee & Zhu, 2011), and we modified their code to make use of the Woodbury matrix identity. We implemented all the other methods in R (R Development Core Team, 2013). All computation was done on computers with 3.4 GHz CPU, 8 GB RAM and the Linux operating system.

We consider the same simulation models as in Bunea et al. (2011). The coefficient matrix C₀ is constructed as $C_0={bC}_1{C}_2^{\mathrm{T}}$, where b > 0, C₁ ∈ ℜ^p×r*, C₂ ∈ ℜ^q×r* and all entries in C₁ and C₂ are random samples from N(0, 1). Two scenarios of model dimensions are considered, with p, q < n and p, q > n. In Model I, we set n = 100, p = q = 25 and r^* = 10. The matrix X is constructed by generating its n rows as random samples from N(0, Γ), where Γ = (Γ_ij)_p×p and Γ_ij = ρ^|i−j| with some 0 < ρ < 1. In Model II, we set n = 20, p = q = 25, r^* = 5 and r_x = 10. The matrix X is generated as X = X₀Γ^1/2, where Γ is defined as above, X₀ = X₁X₂, X₁ ∈ ℜ^n×r_x, X₂ ∈ ℜ^r_x×p, and all entries of X₁, X₂ are N(0, 1) random samples.

The data matrix Y is then generated by Y = XC₀ + E, where the elements of E are N(0, 1) random samples. Each simulated model is characterized by the sample size n, the number of predictors p, the number of responses q, the true model rank r^*, the rank of the design matrix r_x, the correlation ρ ∈ {0·1, 0·5, 0·9}, and the signal strength b ∈ {0·05, 0·1, 0·3}. The experiment was replicated 500 times for each parametric setting.

One way to alleviate inaccuracy in the empirical tuning parameter selection and to reveal the true potential of each method for fair comparison is to tune each method based on its predictive accuracy evaluated with a very large independently generated validation data set; this yields optimally tuned estimators. We have also tried ten-fold cross validation for selecting the tuning parameters but the results are omitted for brevity, as they are similar to or slightly worse than those of the optimal tuning procedure; see the Supplementary Material. For each method, the model accuracy is measured by the average of the scaled mean squared errors from all 500 runs, i.e., $\text{Est}(\widehat{C})=100{\Vert C_0-\widehat{C}\Vert }_F^2/(pq)$ for estimation, and $\text{Pred}(\widehat{C})=100{\Vert {XC}_0-X\widehat{C}\Vert }_F^2/(nq)$ for prediction. To evaluate the rank determination performance, we report the average of the estimated ranks from all runs and the percentage of correct rank identification. Tables 1 and 2 summarize the simulation results and list the average computation time per simulation run for Models I and II.

Table 1.

Comparison of optimally tuned reduced rank estimators using Model I. The estimation and prediction errors are reported, with their standard errors in parentheses.

b		ĈN	ĈH	ĈS(2)	ĈS(0)	ĈHR	ĈSR(2)
		ρ = 0·9
0·05	Est	1·6 (0·2)	3·2 (0·6)	2·6 (0·4)	2·5 (0·3)	1·6 (0·2)	2·5 (0·4)
	Pred	7·8 (0·8)	12·2 (1·3)	9·9 (1·0)	10·5 (1·0)	8·1 (0·8)	9·8 (1·0)
	Rank	7·7, 3%	3·3, 0%	5·5, 0%	10·9, 26%	7·6, 4%	6·0, 0%
0·1	Est	3·5 (0·4)	5·6 (0·7)	4·5 (0·5)	4·3 (0·5)	3·6 (0·4)	4·4 (0·5)
	Pred	12·1 (1·0)	16·1 (1·4)	13·5 (1·2)	14·4 (1·1)	12·5 (1·0)	13·4 (1·1)
	Rank	11·2, 19%	6·2, 0%	8·1, 5%	14·1, 0%	7·9, 5%	8·4, 9%
0·3	Est	6·4 (0·7)	6·8 (0·8)	6·2 (0·7)	6·8 (0·7)	6·1 (0·7)	6·2 (0·7)
	Pred	16·4 (1·2)	16·9 (1·2)	16·0 (1·2)	18·5 (1·2)	15·9 (1·2)	15·9 (1·2)
	Rank	10·7, 30%	9·8, 81%	10·3, 64%	17·0, 0%	9·9, 86%	10·4, 59%
		ρ = 0·5
0·05	Est	0·8 (0·1)	1·2 (0·1)	0·9 (0·1)	0·9 (0.1)	0·8 (0·1)	0·9 (0·1)
	Pred	12·4 (1·0)	16·8 (1·5)	13·6 (1·2)	13·3 (1·1)	13·0 (1·1)	13·0 (1·1)
	Rank	12·8, 1%	6·0, 0%	8·0, 5%	13·3, 0%	7·9, 6%	9·1, 27%
0·1	Est	1·2 (0·1)	1·4 (0·1)	1·2 (0·1)	1·2 (0·1)	1·2 (0·1)	1·1 (0·1)
	Pred	16·2 (1·2)	17·4 (1·2)	15·7 (1·1)	16·8 (1·1)	15·7 (1·1)	15·4 (1·1)
	Rank	15·2, 0%	9·2, 32%	10·0, 58%	15·8, 0%	9·4, 47%	10·4, 52%
0·3	Est	1·3 (0·1)	1·3 (0·1)	1·2 (0·1)	1·5 (0·1)	1·2 (0·1)	1·2 (0·1)
	Pred	17·1 (1·3)	16·2 (1·2)	16·0 (1·2)	19·5 (1·3)	16·0 (1·2)	15·9 (1·2)
	Rank	10·9, 14%	10·0, 100%	10·2, 80%	17·8, 0%	10·0, 100%	10·3, 74%
		ρ = 0·1
0·05	Est	0·6 (0·1)	0·9 (0·1)	0·7 (0·1)	0·7 (0·1)	0·7 (0·1)	0·7 (0·1)
	Pred	13·3 (1·0)	17·4 (1·4)	14·1 (1·2)	13·7 (1·1)	14·0 (1·2)	13·5 (1·1)
	Rank	14·3, 0%	6·6, 1%	8·5, 12%	13·6, 0%	8·1, 10%	9·6, 35%
0·1	Est	0·9 (0·1)	0·9 (0·1)	0·8 (0·1)	0·9 (0·1)	0·8 (0·1)	0·8 (0·1)
	Pred	16·8 (1·2)	17·3 (1·3)	15·9 (1·2)	17·2 (1·2)	16·1 (1·2)	15·5 (1·2)
	Rank	16·5, 0%	9·5, 52%	10·2, 64%	16·1, 0%	9·6, 63%	10·6, 44%
0·3	Est	0·9 (0·1)	0·8 (0·1)	0·8 (0·1)	1·0 (0·1)	0·8 (0·1)	0·8 (0·1)
	Pred	17·7 (1·5)	16·2 (1·2)	16·0 (1·2)	19·6 (1·3)	16·0 (1·2)	16·0 (1·2)
	Rank	11·3, 3%	10·0, 100%	10·2, 83%	17·9, 0%	10·0, 100%	10·3, 76%
Time		17·5	0·0	0·2	0·2	3·9	2·1

Ĉ_N, nuclear norm penalized estimator (Yuan et al., 2007); Ĉ_H, rank penalized estimator (Bunea et al., 2011); Ĉ_S(γ): adaptive nuclear norm penalized estimator with weight parameter γ; Ĉ_HR, robustified rank penalized estimator; Ĉ_SR(γ), robustified adaptive nuclear norm penalized estimator with weight parameter γ; Est, estimation error; Pred, prediction error; Rank, average of estimated rank and percentage of correct rank identification; Time, average computation time in seconds per simulation run.

Table 2.

Comparison of optimally tuned reduced rank estimators using Model II. The layout of the table is the same as in Table 1.

b		ĈN	ĈH	ĈS(2)	ĈS(0)	ĈHR	ĈSR(2)
		ρ = 0·9
0·05	Est	1·1 (0·2)	1·2 (0·2)	1·2 (0·2)	1·2 (0·2)	1·2 (0·2)	1·2 (0·2)
	Pred	34·1 (4·1)	31·5 (4)	29·1 (3·5)	35·3 (4·2)	29·8 (3·5)	28·9 (3·5)
	Rank	7·4, 2%	4·7, 73%	5·2, 71%	8·3, 0%	4·8, 83%	5·3, 66%
0·1	Est	4·5 (0·7)	4·5 (0·7)	4·5 (0·7)	4·5 (0·7)	4·5 (0·7)	4·5 (0·7)
	Pred	37·0 (4·4)	30·8 (3·6)	30·0 (3·6)	38·9 (4·8)	30·3 (3·6)	29·9 (3·6)
	Rank	7·6, 3%	5·0, 99%	5·2, 79%	8·8, 0%	5·0, 99%	5·3, 75%
0·3	Est	40·4 (5·6)	40·6 (5·9)	40·6 (5·9)	40·6 (5·9)	40·6 (5·9)	40·6 (5·9)
	Pred	32·3 (5·5)	30·2 (3·5)	30·1 (3·5)	41·3 (5·0)	30·0 (3·5)	30·0 (3·5)
	Rank	5·5, 75%	5·0, 100%	5·2, 82%	9·1, 0%	5·0, 100%	5·3, 75%
		ρ = 0·5
0·05	Est	1·1 (0·2)	1·1 (0·2)	1·1 (0·2)	1·1 (0·2)	1·1 (0·2)	1·1 (0·2)
	Pred	35·0 (4·2)	31·0 (3·9)	29·1 (3·6)	35·9 (4·7)	29·7 (3·6)	29·0 (3·6)
	Rank	8·0, 0%	4·9, 90%	5·2, 77%	8·4, 0%	4·9, 93%	5·3, 68%
0·1	Est	4·5 (0·6)	4·5 (0·6)	4·5 (0·6)	4·5 (0·6)	4·5 (0·6)	4·5 (0·6)
	Pred	37·8 (4·6)	30·2 (3·5)	29·6 (3·3)	39·0 (4·2)	29·8 (3·4)	29·5 (3·4)
	Rank	7·8, 7%	5·0, 99%	5·2, 81%	8·9, 0%	5·0, 99%	5·3, 74%
0·3	Est	40·4 (6·1)	40·1 (6·0)	40·1 (6·0)	40·1 (6·0)	40·1 (6·0)	40·1 (6·0)
	Pred	31·6 (4·9)	30·3 (3·1)	30·2 (3·1)	41·3 (4·0)	30·2 (3·1)	30·2 (3·1)
	Rank	5·3, 86%	5·0, 100%	5·2, 78%	9·2, 0%	5·0, 100%	5·3, 70%
		ρ = 0·1
0·05	Est	1·1 (0·2)	1·1 (0·2)	1·1 (0·2)	1·1 (0·2)	1·1 (0·2)	1·1 (0·2)
	Pred	35·2 (4·2)	31·0 (3·8)	29·3 (3·5)	36·0 (4·3)	29·9 (3·5)	29·1 (3·6)
	Rank	8·1, 0%	4·9, 88%	5·2, 76%	8·4, 0%	4·9, 91%	5·3, 65%
0·1	Est	4·6 (0·7)	4·5 (0·7)	4·5 (0·7)	4·5 (0·7)	4·5 (0·7)	4·5 (0·7)
	Pred	37·7 (4·6)	30·2 (3·5)	29·7 (3·5)	38·9 (4·4)	29·8 (3·5)	29·6 (3·4)
	Rank	7·9, 5%	5·0, 100%	5·2, 79%	8·8, 0%	5·0, 100%	5·3, 73%
0·3	Est	40·5 (6·2)	40·5 (5·6)	40·5 (5·6)	40·5 (5·6)	40·5 (5·6)	40·5 (5·6)
	Pred	31·3 (5·1)	30·1 (3·4)	30·0 (3·4)	41·2 (4·6)	29·9 (3·4)	29·9 (3·4)
	Rank	5·3, 86%	5·0, 100%	5·2, 76%	9·1, 0%	5·0, 100%	5·3, 68%
Time		20·2	0·0	0·2	0·2	4·0	2·5

We first examine the effects of the adaptive weights on the proposed estimator Ĉ_S. For the case of equal weights, i.e., γ = 0, Ĉ_S tends to overestimate the rank and does not have good predictive performance in most cases. The performance of Ĉ_S is substantially better when γ = 2, which implements data-driven weights, than when γ = 0. We have also experimented with other γ values, and our results show that γ = 2 is generally a good choice; see the Supplementary Material. Henceforth we refer to the case of γ = 2 in the following comparisons.

A sharper comparison between the various estimators can be obtained by contrasting their performance on each simulated dataset. For instance, for each experimental setting, we compare Ĉ_H with Ĉ_S by computing the percentage reduction in the mean squared prediction error of Ĉ_S relative to Ĉ_H for each of the 500 simulated datasets:

$$100\times \frac{\text{Pred}({\widehat{C}}_{\mathrm{H}})-\text{Pred}({\widehat{C}}_{\mathrm{S}})}{\text{Pred}({\widehat{C}}_{\mathrm{H}})}\%,$$

where Pred(·) denotes the scaled mean squared prediction error of a method. Figure 1 displays the notched boxplots of the percentage reduction in the mean squared prediction error of Ĉ_S relative to Ĉ_H across all experimental settings, whereas Figure 2 displays those of Ĉ_S relative to Ĉ_N. The notches in each boxplot extend 1·58/√500 times its inter-quartile range from the median (McGill et al., 1978).

Fig. 1.

Notched boxplots of the percentage reduction in the mean squared prediction error of Ĉ_S relative to Ĉ_H. The boxplots are shaded in dark grey for ρ = 0.9, in light grey for ρ = 0.5, and in white for ρ = 0.1.

Fig. 2.

Notched boxplots of the percentage reduction in the mean squared prediction error of Ĉ_S relative to Ĉ_N. All other settings are same as in Figure 1.

The proposed estimator Ĉ_S generally outperforms the rank penalized estimator Ĉ_H, especially in Model I. Figure 1 shows that the improvement in prediction can be substantial, when the signal is weak or moderate and the correlation among the predictors is high. For rank determination, both estimators perform well when the signal is moderate to strong and the correlation among the predictors is weak to moderate. The proposed estimator Ĉ_S, however, tends to slightly overestimate the rank.

The proposed method often outperforms nuclear norm penalized regression (Yuan et al., 2007), and is more parsimonious than the latter in both rank reduction and computation. Table 1 and Figure 2 show that in Model I, when the signal is weak and/or the correlation among the predictors is high, the nuclear norm penalized estimator Ĉ_N performs better than Ĉ_S in estimation and prediction. However, this gain has a price, for Ĉ_N often overestimates the rank and is much harder to compute. In the high dimensional setting of Model II, Ĉ_S generally enjoys similar or better predictive performance than Ĉ_N. Our findings agree with those of Bunea et al. (2011).

Table 1 shows that shrinkage estimation due to the additional ridge penalty generally enhances an estimator, especially in the presence of highly correlated predictors. However, Ĉ_S benefits much less from the additional ridge penalty than does Ĉ_H, because, unlike the latter, Ĉ_S is already an adaptive shrinkage estimator, owing to the soft-thresholding operation. In general, it is worthwhile to incorporate ridge penalization in order to further improve prediction, if the increased computational cost is affordable.

6·2. Application

We consider a breast cancer data set (Witten et al., 2009), consisting of gene expression measurements and comparative genomic hybridization measurements for n = 89 subjects. The data were used to demonstrate the effectiveness of the rank selection criterion in a preprint of Bunea et al. (2011) posted at the website http://arxiv.org. The data set is available in the R package PMA (Witten et al., 2009), and a detailed description can be found in Chin et al. (2006).

Prior studies have demonstrated that certain types of cancer are characterized by abnormal DNA copy-number changes (Pollack et al., 2002; Peng et al., 2010). It is thus of interest to examine the relationship between DNA copy-number variations and gene expression profiles, for which multivariate regression methods can be useful. Biologically, it makes sense to regress gene expression profiles on copy-number variations because the amplification or deletion of the portion of DNA corresponding to a given gene may result in a corresponding increase or decrease in expression of that gene. The reverse approach is also meaningful, in that the resulting predictive model may identify functionally relevant copy-number variations. This approach has been shown to be promising in enhancing the limited comparative genomic hybridization data analysis with the wealth of gene expression data (Geng et al., 2011; Zhou et al., 2012). We have tried both approaches, i.e., setting 1: designating the copy-number variations of a chromosome as predictors and the gene expression profiles of the same chromosome as responses, and setting 2: reversing the roles of the predictors and the responses. We find that in setting 1, none of the methods provides an adequate fit to the data, and the rank selection criterion may even fail to pick up any signals. The reduced rank models give much better results under setting 2. We thus report only the results for setting 2.

We focus the analysis on chromosome 21, for which p = 227 and q = 44. Both the responses and predictors are standardized. We compare the various reduced rank methods by the following cross-validation procedure. The data were randomly split into a training set of size n_train = 79 and a test set of size n_test = 10. All model estimation was carried out using the training data, with the tuning parameters selected by ten-fold cross validation. We used the test data to calibrate the predictive performance of each estimator Ĉ, specifically, by its mean squared prediction error ${\Vert Y_{\text{test}}-X_{\text{test}}\widehat{C}\Vert }_F^2/({qn}_{\text{test}})$, where (Y_test, X_test) denotes the test set. The random-splitting process was repeated 100 times to yield the average mean squared prediction error and the average rank estimate for each method; see the upper panel of Table 3.

Table 3.

Comparison of the model fits to the chromosome 21 data for various reduced rank methods. The mean squared prediction errors and the estimated ranks are reported, with their standard errors in parentheses.

	ĈN	ĈH	ĈHR	ĈS(2)	ĈSR(2)
	Full data
MSPE	0·71 (0·2)	0·.69 (0·1)	0·68 (0·1)	0·68 (0·1)	0·68 (0.·1)
Rank	6·2 (0·8)	1·0 (0·0)	1·0 (0·0)	1·0 (0·2)	1·8 (0·4)
	Selected predictors
MSPE	0·85 (0·2)	0·90 (0·2)	0·82 (0·1)	0·85 (0·1)	0·84 (0·1)
Rank	4·6 (0·7)	1·3 (0·5)	1·9 (0·3)	2·1 (0·2)	2·4 (0·5)

MSPE, average of mean squared prediction errors based on 100 replications of 79/10 splits of the data; Rank: corresponding average of the rank estimates. The notations of the estimators are the same as in Table 1.

As the number of predictors is much greater than the sample size, it is reasonable to assume that only a subset of predictors is important. Therefore, a perhaps better modeling strategy is subset multivariate regression with a selected subset of predictors. Recently, several variable selection methods have been proposed in the context of reduced rank regression (Chen et al., 2012; Chen & Huang, 2012; Bunea et al., 2012). We modified the preceding cross-validation procedure for comparing the reduced rank subset regression methods. The only modification was that for each random split, we first applied the method of Chen et al. (2012) using the training set to select a set of predictors, with which the reduced rank methods were subsequently carried out using the training set and calibrated using the test set. Since our main goal is to compare the various reduced rank methods, we omit the description of the predictor selection procedure but refer the interested reader to Chen et al. (2012) for details. The results are summarized in the lower panel of Table 3.

Table 3 shows that the proposed estimator Ĉ_S enjoys slightly better predictive performance than both Ĉ_H and Ĉ_N. The numbers of selected predictors in the 100 splits range from 71 to 102, hence incorporating variable selection greatly reduces the number of predictors and may potentially improve model interpretation. However, in this example, reduced rank estimation using a subset of predictors results in higher mean squared prediction error than using all predictors, uniformly for all methods, but more so for Ĉ_H than for other methods. The nuclear norm penalized estimator Ĉ_N generally yields a higher rank estimate than the other methods. Incorporating ridge penalization improves the predictive performance of the reduced rank methods. Particularly, Ĉ_HR may substantially outperform its non-robust counterpart Ĉ_H. We find that both Ĉ_N and Ĉ_HR can be computationally intensive for large datasets, while other methods are much faster to compute. These results are consistent with the simulation findings in Section 6·1.

7. Discussion

Adaptive nuclear norm penalization can serve as a building block to study a family of singular value penalties. This is based on the connection between an adaptive l₁ penalty and many concave penalty functions (Knight & Fu, 2000; Fan & Li, 2001; Huang et al., 2008). Consider the regression problem (2) with a general singular value penalty ${\mathcal{P}}_{\lambda }(C)={\sum }_{i=1}^{p\wedge q}{p}_{\lambda }\{d_i(C)\}$, where p_λ(·) is a penalty function, e.g., $p_{\lambda }\{d_i(C)\}=\lambda d_i^b(C)$ for some 0 < b ≤ 1 (Huang et al., 2008; Rohde & Tsybakov, 2011). In this setup the optimization of (2) can be challenging. A promising approach is to adopt a local linear approximation (Zou & Li, 2008), $p_{\lambda }\{d_i(C)\}\approx p_{\lambda }({\stackrel{\sim }{d}}_i)+p_{\lambda }^{\prime }({\stackrel{\sim }{d}}_i)\{d_i(C)-{\stackrel{\sim }{d}}_i\}$, for d_i(C) ≈ d̃_i, where d̃_i is some initial estimator of d_i(C). It can be seen that for fixed d̃_i, up to a constant, the first-order approximation yields exactly an adaptive nuclear norm penalty. This suggests that these problems may be solved by an iteratively reweighted adaptive nuclear norm penalization approach.

Incorporating an extra ridge penalty can improve reduced rank estimation (Mukherjee & Zhu, 2011; She, 2012). When combined with the adaptive nuclear norm penalty, such a criterion bears resemblance to the adaptive elastic-net criterion (Zou & Hastie, 2005; Zou & Zhang, 2009) in univariate regression. It would be interesting to investigate the theoretical properties of this approach and compare it with the nonlinear fusion of nuclear norm and ridge penalties in Owen (2007) and She (2012). Another pressing problem is to extend regularized reduced rank regression methods to generalized linear and nonparametric regression models (Yee & Hastie, 2003; Li & Chan, 2007; She, 2012). On the optimization aspect, it is interesting to study the use of adaptive nuclear norm in some classical sparse optimization areas, such as matrix completion (Candès et al., 2011).

Appendix. Technical details

Proof of Theorem 1

First we show by a counter-example that if we have an index k such that w_k < w_k+1, then f (·) is non-convex. Let C and Z be diagonal p × p matrices such that c_ii = i, for i = 1, …, p, while Z equals C but with entries switched at positions p − k + 1 and p − k for some 1 ≤ k ≤ p − 1 on the diagonal. It is then easy to verify that

$$\begin{array}{l}f(C)=f(Z)=\sum _{i=1}^p{w}_i(p-i+1),\\ f(\frac{C+Z}{2})-\frac{f(C)}{2}-\frac{f(Z)}{2}=(p-k+\frac{1}{2})(w_k+w_{k+1})-(p-k+1)w_k-(p-k)w_{k+1}\\ =\frac{1}{2}(w_k+w_{k+1})-w_k>0,\end{array}$$

, where f(·) is defined in (3). Therefore f(·) is non-convex.

Next we prove that f(C) = ||C||_w* is a convex function of C ∈ ℜ^p×q for w₁ ≥ ··· ≥ w_p∧q ≥ 0. Without loss of generality, assume p ≤ q so that we can simply write p = p ∧ q. First consider the case that w_p > 0, and define the following function on ℜ^p:

$$w(x)=\sum _{i=1}^p{w}_i{\mid x\mid }_{\delta (i)},$$ (A1)

where δ is a permutation of {1,…, p} determined by x such that |x|_δ(1) ≥ ··· ≥ |x|_δ(p). We claim that w(·) in (A1) is a symmetric gauge function (Horn & Johnson, 1985, Definition 7.4.23), i.e., it satisfies the following six conditions: (a) w(x) ≥ 0, for any x ∈ ℜ; (b) w(x) = 0 if and only if x = 0; (c) w(αx) = |α|w(x), for any α ∈ ℜ; (d) w(x + y) ≤ w(x) + w(y); (e) w(x) = w(|x|); (f) w(x) = w{τ (x)} for any τ that is a permutation of indices {1,…, p}.

All conditions except (d) are trivial to verify. To prove (d), let δ, σ and τ be permutations such that |x + y|_δ(i), |x|_σ(i) and |y|_τ(i) are placed in non-increasing order respectively.

$$\begin{array}{l}w(x+y)=\sum _{i=1}^p{w}_i{\mid x+y\mid }_{\delta (i)}\le \sum _{i=1}^p\left(w_i{\mid x\mid }_{\delta (i)}+w_i{\mid y\mid }_{\delta (i)}\right)\\ \le \sum _{i=1}^p\left(w_i{\mid x\mid }_{\sigma (i)}+w_i{\mid y\mid }_{\tau (i)}\right)=w(x)+w(y).\end{array}$$

where the second inequality is due to the Hardy–Littlewood–Pólya inequality (Hardy et al., 1967). By a straightforward application of Horn & Johnson (1985, Theorem 7.4.24), since ||C||_w* = w{d(C)} where d(C) = {d₁(C),…, d_p(C)}^T, the function f(·) = ||·||_w* defines a matrix norm and hence is convex.

For the case that w_p = 0, let s be the largest index such that w_s > 0. For 0 < ε < w_s, consider the perturbed w̃ that w̃_i = w_i, for i = 1,…, s, and w̃_i = ε, for i = s + 1,…, p. Then for any C,Z ∈ ℜ^p×q, ||C + Z||_w̃*/2 ≤ ||C||_w̃*/2 + ||Z||_w̃*/2. By taking ε → 0, ||C + Z||_w*/2 ≤ ||C||_w*/2 + ||Z||_w*/2. Therefore ||·||_w* is convex.

Proof of Theorem 2

We first prove that (Y) is indeed a global optimal solution to (8). Below, we write h for n ∧ q. Let $g={\{g_i\}}_{i=1}^h=d(C)$, which implies the entries of g are in non-increasing order. Since the penalty term only depends on the singular values of C, (8) can be equivalently written as:

$$\underset{g:g_1\ge \cdots \ge g_h\ge 0}{min}\left\{\underset{C\in {\mathfrak{R}}^{n\times q},d(C)=g}{min}\left(\frac{1}{2}{\Vert Y-C\Vert }_F^2+\lambda \sum _{i=1}^h{w}_i{g}_i\right)\right\}.$$

For the inner minimization, we have the inequality

$$\begin{array}{l}{\Vert Y-C\Vert }_F^2=\text{tr}({YY}^{\mathrm{T}})-2\text{tr}({YC}^{\mathrm{T}})+\text{tr}({CC}^{\mathrm{T}})\\ =\sum _{i=1}^h{d}_i^2(Y)-2\text{tr}({YC}^{\mathrm{T}})+\sum _{i=1}^h{g}_i^2\\ \ge \sum _{i=1}^h{d}_i^2(Y)-2d{(Y)}^{\mathrm{T}}g+\sum _{i=1}^h{g}_i^2.\end{array}$$

The last inequality is due to von Neumann’s trace inequality (von Neumann, 1937; Mirsky, 1975). Equality holds when C admits the singular value decomposition C = Udiag(g)V^T, where U and V are defined in (5) as the left and right singular matrices of Y. Then the optimization reduces to

$$\underset{g:g_1\ge \cdots \ge g_h\ge 0}{min}\left(\sum _{i=1}^h\left[\frac{1}{2}{g}_i^2-\{d_i(Y)-\lambda w_i\}{g}_i+\frac{1}{2}{d}_i^2(Y)\right]\right).$$ (A2)

The objective function is completely separable and is minimized only when g_i = {d_i(Y) − λw_i}₊. This is a feasible solution because {d_i(Y)} is in non-increasing order, while {w_i} is in non-decreasing order. Therefore (Y) = Udiag[{d(Y) − λw}₊]V^T is a global optimal solution to (8). The uniqueness follows by the equality condition for the von Neumann’s trace inequality when Y has distinct nonzero singular values, and the uniqueness of the strictly convex optimization (A2). This concludes the proof.

Proof of Lemma 1

By (17), r̂ > s holds if and only if d_s+1(PY) > λ^1/(γ+1) and r̂ < s holds if and only if d_s(PY) ≤ λ^1/(γ+1). Then

$$\text{pr}(\widehat{r}\ne s)=\text{pr}\left\{d_{s+1}(PY)>{\lambda }^{1/(\gamma +1)}\text{or}{d}_s(PY)\le {\lambda }^{1/(\gamma +1)}\right\}.$$

Based on Weyl’s inequalities on singular values (Franklin, 2000) and observing that PY = XC₀ + PE, we have d₁(PE) ≥ d_s+1(PY) − d_s+1(XC₀) and d₁(PE) ≥ d_s(XC₀) − d_s(PY). Hence d_s+1(PY) > λ^1/(γ+1) implies d₁(PE) ≥ λ^1/(γ+1) − d_s+1(XC₀), and d_s(PY) ≤ λ^1/(γ+1) implies d₁(PE) ≥ d_s(XC₀) − λ^1/(γ+1). It then follows that

$$\text{pr}(\widehat{r}\ne s)\le \text{pr}\left[d_1(PE)\ge min\{{\lambda }^{1/(\gamma +1)}-d_{s+1}({XC}_0),d_s({XC}_0)-{\lambda }^{1/(\gamma +1)}\}\right].$$

Note that min{λ^1/(γ+1) − d_s+1(XC₀), d_s(XC₀) − λ^1/(γ+1)} ≥ δλ^1/(γ+1). This completes the proof.

Lemma 2 (Bunea et al., 2011)

Let r_x denote the rank of X and suppose Assumption 1 holds. Then for any t > 0, E{d₁(PE)} ≤ σ(√r_x + √q), and pr[d₁(PE) ≥ E{d₁(PE)} + σt] ≤ exp(−t²/2).

Proof of Theorem 3

When d_r^* (XC₀) > 2λ^1/(γ+1), we have d_r^* (XC₀) > 2λ^1/(γ+1) ≥ (1 + δ)λ^1/(γ+1) and d_r^*+1(XC₀) = 0 ≤ (1 − δ)λ^1/(γ+1), for some 0 < δ ≤ 1. The effective rank s defined in Lemma 1 equals the true rank, i.e., s = r^*, and min{λ^1/(γ+1) − d_r^*+1(XC₀), d_r^* (XC₀) − λ^1/(γ+1) } ≥ δλ^1/(γ+1). It then follows from Lemma 2 that

$$\begin{array}{l}\text{pr}(\widehat{r}=r^{\ast })\ge 1-\text{pr}\{d_1(PE)\ge \delta {\lambda }^{1/(\gamma +1)}\}\\ =1-\text{pr}\{d_1(PE)\ge (1+\theta )\sigma (\surd r_x+\surd q)\}\\ \ge 1-exp\{{\theta }^2(r_x+q)/2\}\\ \to 1\end{array}$$

as r_x + q → ∞. This completes the proof.

Proof of Theorem 4

We write h for n ∧ q. By the definition of Ĉ_S in (16),

$${\Vert Y-X{\widehat{C}}_{\mathrm{S}}\Vert }_F^2+2\lambda \sum _{i=1}^h{w}_i{d}_i(X{\widehat{C}}_{\mathrm{S}})\le {\Vert Y-XB\Vert }_F^2+2\lambda \sum _{i=1}^h{w}_i{d}_i(XB),$$

for any p × q matrix B. Note that

$$\begin{array}{l}{\Vert Y-X{\widehat{C}}_{\mathrm{S}}\Vert }_F^2={\Vert Y-{XC}_0\Vert }_F^2+{\Vert X{\widehat{C}}_{\mathrm{S}}-{XC}_0\Vert }_F^2+2<E,{XC}_0-X{\widehat{C}}_{\mathrm{S}}{>}_F,\\ {\Vert Y-XB\Vert }_F^2={\Vert Y-{XC}_0\Vert }_F^2+{\Vert XB-{XC}_0\Vert }_F^2+2<E,{XC}_0-XB{>}_F.\end{array}$$

Then we have

$$\begin{array}{l}{\Vert X{\widehat{C}}_{\mathrm{S}}-{XC}_0\Vert }_F^2\le {\Vert XB-{XC}_0\Vert }_F^2+2<E,X{\widehat{C}}_{\mathrm{S}}-XB{>}_F+2\lambda \{\sum _{i=1}^h{w}_i{d}_i(XB)-\sum _{i=1}^h{w}_i{d}_i(X{\widehat{C}}_{\mathrm{S}})\}\\ \le {\Vert XB-{XC}_0\Vert }_F^2+2<PE,X{\widehat{C}}_{\mathrm{S}}-XB{>}_F+2\lambda \{\sum _{i=1}^h{w}_i{d}_i(XB)-\sum _{i=1}^h{w}_i{d}_i(X{\widehat{C}}_{\mathrm{S}})\}\\ \le {\Vert XB-{XC}_0\Vert }_F^2+2d_1(PE){\Vert X{\widehat{C}}_{\mathrm{S}}-XB\Vert }_{\ast }+2\lambda \{\sum _{i=1}^h{w}_i{d}_i(XB)-\sum _{i=1}^h{w}_i{d}_i(X{\widehat{C}}_{\mathrm{S}})\}\\ \le {\Vert XB-{XC}_0\Vert }_F^2+2d_1{(PE)}_r{(X{\widehat{C}}_{\mathrm{S}}-XB)}^{1/2}{\Vert X{\widehat{C}}_{\mathrm{S}}-XB\Vert }_F+2\lambda \{\sum _{i=1}^h{w}_i{d}_i(XB)-\sum _{i=1}^h{w}_i{d}_i(X{\widehat{C}}_{\mathrm{S}})\}.\end{array}$$ (A3)

Now consider any B with r(B) ≤ r̂,

$$\sum _{i=1}^h{w}_i{d}_i(XB)-\sum _{i=1}^h{w}_i{d}_i(X{\widehat{C}}_{\mathrm{S}})=w_{\widehat{r}}\sum _{i=1}^{\widehat{r}}{d}_i(XB)-w_{\widehat{r}}\sum _{i=1}^{\widehat{r}}{d}_i(X{\widehat{C}}_{\mathrm{S}})+\sum _{i=1}^{\widehat{r}}(w_{\widehat{r}}-w_i)d_i(X{\widehat{C}}_{\mathrm{S}})-\sum _{i=1}^{\widehat{r}}(w_{\widehat{r}}-w_i)d_i(XB).$$

Recall that $w_i=d_i^{-\gamma }(PY)$, so w_r̂ − w₁ ≥ ··· ≥ w_r̂ − w_r̂−1 ≥ 0. Therefore, both $p_1(·)={\sum }_{i=1}^{\widehat{r}}{d}_i(·)$ and $p_2(·)={\sum }_{i=1}^{\widehat{r}}(w_{\widehat{r}}-w_i)d_i(·)$ satisfy the triangle inequality; see the proof of Theorem 1. Moreover, Weyl’s inequalities (Franklin, 2000) and the equality PY = XC₀ + PE imply that d_r̂ (PY) ≥ d_r̂(XC₀) − d₁(PE) and d₁(PY) ≤ d₁(XC₀) + d₁(PE). Hence,

$$\begin{array}{l}\sum _{i=1}^h{w}_i{d}_i(XB)-\sum _{i=1}^h{w}_i{d}_i(X{\widehat{C}}_{\mathrm{S}})\le w_{\widehat{r}}\sum _{i=1}^{\widehat{r}}{d}_i(X{\widehat{C}}_{\mathrm{S}}-XB)+\sum _{i=1}^{\widehat{r}}(w_{\widehat{r}}-w_i)d_i(X{\widehat{C}}_{\mathrm{S}}-XB)\\ \le \{2d_{\widehat{r}}^{-\gamma }(PY)-d_1^{-\gamma }(PY)\}\sum _{i=1}^{\widehat{r}}{d}_i(X{\widehat{C}}_{\mathrm{S}}-XB)\\ \le [2{\{d_{\widehat{r}}({XC}_0)-d_1(PE)\}}^{-\gamma }-{\{d_1({XC}_0)+d_1(PE)\}}^{-\gamma }]\sum _{i=1}^{\widehat{r}}{d}_i(X{\widehat{C}}_{\mathrm{S}}-XB)\\ \le [2{\{d_{\widehat{r}}({XC}_0)-d_1(PE)\}}^{-\gamma }-{\{d_1({XC}_0)+d_1(PE)\}}^{-\gamma }]{\Vert X{\widehat{C}}_{\mathrm{S}}-XB\Vert }_F\surd \widehat{r}.\end{array}$$

The last inequality is due to the Cauchy–Schwarz inequality. Using (A3), r(XĈ_S − XB) ≤ r(Ĉ_S − B) ≤ 2r̂ and the inequality 2xy ≤ x²/a + ay², we have

$$\begin{array}{l}{\Vert X{\widehat{C}}_{\mathrm{S}}-{XC}_0\Vert }_F^2\le {\Vert XB-{XC}_0\Vert }_F^2+a{\Vert X{\widehat{C}}_{\mathrm{S}}-XB\Vert }_F^2\\ +\frac{1}{a}{[d_1(PE)\surd (2\widehat{r})+2\lambda {\{d_{\widehat{r}}({XC}_0)-d_1(PE)\}}^{-\gamma }\surd \widehat{r}-\lambda {\{d_1({XC}_0)+d_1(PE)\}}^{-\gamma }\surd \widehat{r}]}^2.\end{array}$$

Since ${\Vert X{\widehat{C}}_{\mathrm{S}}-XB\Vert }_F^2\le {\Vert X{\widehat{C}}_{\mathrm{S}}-{XC}_0\Vert }_F^2+{\Vert XB-{XC}_0\Vert }_F^2$, consequently, for any 0 < a < 1,

$$\begin{array}{l}{\Vert X{\widehat{C}}_{\mathrm{S}}-{XC}_0\Vert }_F^2\le \frac{1+a}{1-a}{\Vert XB-{XC}_0\Vert }_F^2\\ +\frac{1}{a(1-a)}{[d_1(PE)\surd 2+2\lambda {\{d_{\widehat{r}}({XC}_0)-d_1(PE)\}}^{-\gamma }-\lambda {\{d_1({XC}_0)+d_1(PE)\}}^{-\gamma }]}^2\widehat{r}.\end{array}$$

As shown in Theorem 3, on the event {d₁(PE) < δλ^1/(γ+1)}, the estimated rank r̂ equals the true rank r^*, i.e., r̂ = r^*, and pr{d₁(PE) ≥ δλ^1/(γ+1)}≤ exp{−θ²(r_x + q)/2}. Also, d_r^* (XC₀) > 2λ^1/(γ+1) and c = d₁(XC₀)/d_r^* (XC₀) ≥ 1. Therefore, with probability at least 1 − exp{−θ²(r_x + q)/2},

$$\begin{array}{l}{\Vert X{\widehat{C}}_{\mathrm{S}}-{XC}_0\Vert }_F^2\le \frac{1+a}{1-a}{\Vert XB-{XC}_0\Vert }_F^2+\frac{1}{a(1-a)}{\left\{\delta {\lambda }^{1/(\gamma +1)}\surd 2+2\lambda {(2-\delta )}^{-\gamma }{\lambda }^{-\gamma /(\gamma +1)}-\lambda {(2c+\delta )}^{-\gamma }{\lambda }^{-\gamma /(\gamma +1)}\right\}}^2{r}^{\ast }\\ \le \frac{1+a}{1-a}{\Vert XB-{XC}_0\Vert }_F^2+\frac{1}{a(1-a)}{\{\delta \surd 2+2{(2-\delta )}^{-\gamma }-{(2c+\delta )}^{-\gamma }\}}^2{\lambda }^{2/(\gamma +1)}{r}^{\ast }.\end{array}$$

Since B is an arbitrary matrix with r(B) ≤ r^*, the second part of the theorem is obtained by taking B = C₀ and a = 1/2. This completes the proof.