L2RM: Low-rank Linear Regression Models for High-dimensional Matrix Responses

Abstract

The aim of this paper is to develop a low-rank linear regression model (L2RM) to correlate a high-dimensional response matrix with a high dimensional vector of covariates when coefficient matrices have low-rank structures. We propose a fast and efficient screening procedure based on the spectral norm of each coefficient matrix in order to deal with the case when the number of covariates is extremely large. We develop an efficient estimation procedure based on the trace norm regularization, which explicitly imposes the low rank structure of coefficient matrices. When both the dimension of response matrix and that of covariate vector diverge at the exponential order of the sample size, we investigate the sure independence screening property under some mild conditions. We also systematically investigate some theoretical properties of our estimation procedure including estimation consistency, rank consistency and non-asymptotic error bound under some mild conditions. We further establish a theoretical guarantee for the overall solution of our two-step screening and estimation procedure. We examine the finite-sample performance of our screening and estimation methods using simulations and a large-scale imaging genetic dataset collected by the Philadelphia Neurodevelopmental Cohort (PNC) study.

1. Introduction

Multivariate regression modeling with a matrix response $\mathbf{Y}\in {\mathbb{R}}^{p\times q}$ and a multivariate covariate $\mathrm{x}\in {\mathbb{R}}^s$ is an important statistical tool in modern high-dimensional inference, with wide applications in various large-scale applications, such as imaging genetic studies. Specifically, in imaging genetics, matrix responses (Y) as phenotypic variables often represent the weighted (or binary) adjacency matrix of a finite graph for characterizing structural (or functional) connectivity pattern, whereas covariates (x) include genetic markers (e.g., single nucleotide polymorphisms (SNPs)), age, and gender, among others. The joint analysis of imaging and genetic data may ultimately lead to discoveries of genes for many neuropsychiatric and neurological disorders, such as schizophrenia (Scharinger et al., 2010; Peper et al., 2007; Chiang et al., 2011; Thompson et al., 2013; Medlan et al., 2014). This motivates us to systematically investigate a statistical model with a multivariate response Y and a multivariate covariate x.

Let {(x_i, Y_i) : 1 ≤ i ≤ n} denote independent and identically distributed (i.i.d.) observations, where x_i = (x_i,1, …, x_is)^T is a s × 1 vector of scalar covariates (e.g., clinical variables and genetic variants) and Y_i is a p × q response matrix. Without loss of generality, we assume that x_il has mean 0 and variance 1 for every 1 ≤ l ≤ s, and Y_i has mean 0. Throughout the paper, we consider a L2RM, which is given by

$${\mathbf{Y}}_i=\sum _{l=1}^s{x}_{il}\ast {\mathbf{B}}_l+{\mathbf{E}}_i,$$ (1)

where B_l is a p × q coefficient matrix characterizing the effect of the lth covariate on Y_i and E_i is a p × q matrix of random errors with mean 0. The symbol “ * “ denotes the scalar multiplication. Model (1) differs significantly from the existing matrix regression, which was developed for matrix covariates and univariate responses (Leng and Tang, 2012; Zhao and Leng, 2014; Zhou and Li, 2014). Our goal is to discover a small set of important covariates from x that strongly influence Y.

We focus on the most challenging setting that both the dimension of Y (or pq) and that of x (or s) can diverge with the sample size. Such a setting is general enough to cover high-dimensional univariate and multivariate linear regression models in the literature (Negahban et al., 2012; Fan and Lv, 2010; Buhlmann and van de Geer, 2011; Tibshirani, 1997; Yuan et al., 2007; Candes and Tao, 2007; Breiman and Friedman, 1997; Cook et al., 2013; Park et al., 2017). In the literature, there are two major categories of statistical methods for jointly analyzing high-dimensional matrix Y and high-dimensional vector x.

The first category is a set of mass univariate methods. Specifically, it fits a marginal linear regression to correlate each element of Y_i with each element of x_i, leading to a total of pqs massive univariate analyses and an expanded search space with pqs elements. It is also called voxel-wise genome-wide association analysis (VGAWS) in the imaging genetics literature (Hibar et al., 2011; Shen et al., 2010; Huang et al., 2015; Zhang et al., 2014; Medland et al., 2014; Zhang et al., 2014; Thompson et al., 2014; Liu and Calhoun, 2014). For instance, Stein et al. (2010) used 300 high performance CPU nodes to run approximately 27 hours to carry out a VGWAS analysis on an imaging genetic dataset with only 448,293 SNPs and 31,622 imaging measures for 740 subjects. Such computational challenges are becoming more severe as the field is rapidly advancing to the most challenging setting with large pq and s. More seriously, for model (1), the massive univariate method can miss some important components of x that strongly influence Y due to the interaction among x.

The second category is to fit a model accommodating all (or part of) covariates and responses (Vounou et al., 2010, 2012; Zhu et al., 2014; Wang et al., 2012a,b; Peng et al.,2010). These methods use regularization methods, such as Lasso or group Lasso, to select a set of covariate-response pairs. However, when the product pqs is extremely large, it is very difficult to allocate computer memory for such an array of size pqs in order to accommodate all coefficient matrices B_ls, rendering all these regularization methods being intractable. Therefore, almost all existing methods in this category have to use some dimension reduction techniques (e.g., screening methods) to reduce both the number of responses and that of covariates. Subsequently, these methods fit a multivariate linear regression model with the selected elements of Y as new responses and those of x as new covariates. However, this approach can be unsatisfactory, since it does not incorporate the matrix structural information.

The aim of this paper is to develop a low-rank linear regression model (L2RM) as a novel extension of both VGAWS and regularization methods. Specifically, instead of repeatedly fitting a univariate model to each covariate-response pair, we consider all elements in Y_i as a high-dimensional matrix response and focus on the coefficient matrix of each covariate, which is approximately low-rank (Candès and Recht, 2009). There is a literature on the development of matrix variate regression (Ding and Cook, 2014; Fosdick and Hoff, 2015; Zhou and Li, 2014), but these papers focus on the case when covariates have a matrix structure. In contrast, there is a large literature on the development of various function-on-scalar regression models that emphasize the inherent functional structure of responses. See Chapter 13 of Ramsay and Silverman (2005) for a comprehensive review on this topic. Variable selection methods have been developed for some function-on-scalar regression models (Wang et al., 2007; Chen et al., 2016), but these methods focus on one-dimensional functional response rather than two-dimensional matrix response. Recently, there has been some literature considering matrix or tensor responses regression (Ding and Cook, 2018; Li and Zhang, 2017; Raskutti and Yuan, 2018; Rabusseau and Kadri, 2016), but they only consider the case when the dimension of the covariates is fixed or slowly diverging with the sample size.

In this paper, we aim at efficiently correlating matrix responses with a high dimensional vector of covariates. Four major methodological contributions of this paper are as follows.

• We introduce a low-rank linear regression model to fit high-dimensional matrix responses with a high dimensional vector of covariates, while explicitly accounting for the low-rank structure of coefficient matrices.

• We introduce a novel rank-one screening procedure based on the spectral norm of the estimated coefficient matrix to eliminate most “noisy” scalar covariates and show that our screening procedure enjoys the sure independence screening property (Fan and Lv, 2008) with vanishing false selection rate. The use of such spectral norm is critical for dealing with a large number of noisy covariates.

• When the number of covariates is relatively small, we propose a low rank estimation procedure based on trace norm regularization, which explicitly characterizes the low-rank structure of coefficient matrices. An efficient algorithm for solving the optimization problem is developed. We systematically investigate some theoretical properties of our estimation procedure, including estimation and rank consistency when both p and q are fixed and an non-asymptotic error bound when both p and q are allowed to diverge.

• We investigate how incorrectly screening results can affect the low-rank regression model estimation both numerically and theoretically. We establish a theoretical guarantee for the overall solution, while accounting for the randomness of the first-step screening procedure.

The rest of this paper is organized as follows. In Section 2, we introduce a rank-one screening procedure to deal with a high dimensional vector of covariates and describe our estimation procedure when the number of covariates is relatively small. Section 3 investigates the theoretical properties of our method. Simulations are conducted in Section 4 to evaluate the finite-sample performance of the proposed two-step screening and estimation procedure. Section 5 illustrates an application of L2RM in the joint analysis of imaging and genetic data from the Philadelphia Neurodevelopmental Cohort (PNC) study discussed above. We finally conclude with some discussions in Section 6.

2. Methodology

Throughout the paper, we focus on addressing three fundamental issues for L2RM as follows:

• (I) The first one is to eliminate most ‘noisy’ covariates x_il when the number of candidate covariates and that of response matrix are much larger than n, that is min(s, pq) >> n.

• (II) The second one is to estimate the coefficient matrix B_l when B_l does have a low-rank structure.

• (III) The third one is to investigate some theoretical properties of the screening and estimation methods.

2.1. Rank-one Screening Method

We consider the case that both pq and s diverge at an exponential order of n, and we also denote s by s_n. To address (I), it is common to assume that most scalar covariates have no effects on the matrix responses, that is, B_l0 = 0 for most 1 ≤ l ≤ s_n, where B_l0 is the true value for B_l. In this case, we define the true model and its size as

$$\mathcal{M}=\{1\le l\le s_n:{\mathbf{B}}_{l0}\ne \mathbf{0}\}\text{and}{s}_0=\mid \mathcal{M}\mid <n.$$ (2)

Our aim is to estimate the set $\mathcal{M}$ and coefficient matrices B_l. Simultaneously estimating $\mathcal{M}$ and B_l is difficult since it is computationally infeasible to fit a model when both s_n and pq are quite high. For example, in the PNC data, we have pq = 69² = 4, 761 and s_n ≈ 5 × 10⁶. Therefore, it may be imperative to employ a screening technique to reduce the model size. However, developing a screening technique for model (1) can be more challenging than many existing screening methods, which focus on univariate responses (Fan and Lv, 2008; Fan and Song, 2010).

Similar to Fan and Lv (2008) and Fan and Song (2010), it is assumed that all covariates have been standardized so that

$$E(x_{il})=0\text{and}E(x_{il}^2)=1\text{for}l=1,\dots ,s_n.$$

We also assume that every element of Y_i = (Y_i,jk) has been standardized, that is,

$$E(Y_{i,jk})=0\text{and}E(Y_{i,jk}^2)=1\text{for}j=1,\dots ,p\text{and}k=1,\dots ,q.$$

We propose to screen covariates based on the estimated marginal ordinary least squares (OLS) coefficient matrix ${\widehat{\mathbf{B}}}_l^M=n^{-1}{\sum }_{i=1}^n{x}_{il}\ast {\mathbf{Y}}_i$ for l = 1, …, s_n. Although the interpretations and implications of the marginal models are biased from the joint model, the nonsparse information about the joint model can be passed along to the marginal model under a mild condition. Hence it is suitable for the purpose of variable screening (Fan and Song, 2010). Specifically, we calculate the spectral norm (operator norm or largest singular value) of ${\widehat{\mathbf{B}}}_l^M$, denoted as $\Vert {\widehat{\mathbf{B}}}_l^M{\Vert }_{op}$, and define a submodel as

$${\widehat{\mathcal{M}}}_{{\gamma }_n}=\{1\le l\le s_n:\Vert {\widehat{\mathbf{B}}}_l^M{\Vert }_{op}\ge {\gamma }_n\},$$ (3)

where γ_n is a prefixed threshold.

The key advantage of using $\Vert {\widehat{\mathbf{B}}}_l^M{\Vert }_{op}$ is that it explicitly accounts for the low-rank structure of B_l0s for most noisy covariates, while being robust to noise and more sensitive to various signal patterns (e.g., sparsely strong signals and low rank weak signals) in coefficient matrices. In our screening step, we use the marginal OLS estimates of the coefficient matrices, which can be regarded as the true coefficient matrices corrupted with some noise. One may directly use some other summary statistics of ${\widehat{\mathbf{B}}}_l^M$ based on the component-wise information of ${\widehat{\mathbf{B}}}_l^M$, such as $\Vert {\widehat{\mathbf{B}}}_l^M{\Vert }_1$ (sum of the absolute value of all the elements), $\Vert {\widehat{\mathbf{B}}}_l^M{\Vert }_F$, or the global Wald-type statistic used in Huang et al. (2015). It is well known that those summary statistics are sensitive to noise and suffer from the curse of dimensionality. This is further confirmed in our simulation studies that our rank-one screening based on $\Vert {\widehat{\mathbf{B}}}_l^M{\Vert }_{op}$ is more robust to noise and sensitive to small signal regions. Moreover, the other advantage of using $\Vert {\widehat{\mathbf{B}}}_l^M{\Vert }_{op}$ is that it is computationally efficient. In contrast, we may calculate some other regularized estimates (e.g., Lasso or fused Lasso) for screening, but it is computationally infeasible for L2RM when s_n is much larger than the sample size.

A difficult issue in (3) is how to properly select γ_n. As shown in Section 3.1, when γ_n is chosen properly, our screening procedure enjoys the sure independence property (Fan and Lv, 2008). However, it is difficult to precisely determine γ_n in practice since it involves in two unknown positive constant terms C₁ and α as shown in Theorem 1, which cannot be easily determined for finite sample. We propose to use random decoupling to select γ_n, which is similar to that used in Barut et al. (2016). Let {${\mathrm{x}}_i^{\ast }$, i = 1, …, n} be a random permutation of the original data (x_i = 1, …, n}. We apply our screening procedure on the random decoupling data $\{{\mathrm{x}}_i^{\ast },{\mathbf{Y}}_i{\}}_{i=1}^n$. As the original association between x_i and Y_i is destroyed by random decoupling, when we perform screening using $\{{\mathrm{x}}_i^{\ast },{\mathbf{Y}}_i{\}}_{i=1}^n$, it mimics the null model, e. the model when there is no association. We obtain the estimated marginal coefficient matrix $({\widehat{\mathbf{B}}}_l^M{)}^{\ast }$, which is a statistical estimate of zero matrix, and the corresponding operator norm $\Vert ({\widehat{\mathbf{B}}}_l^M{)}^{\ast }{\Vert }_{op}$ for all 1 ≤ l ≤ s_n. Define ν_n = max_1≤l≤sn $\Vert ({\widehat{\mathbf{B}}}_l^M{)}^{\ast }{\Vert }_{op}$, which is the the minimum thresholding parameter that makes no false positives. Since ν_n depends on the realization of the permutation, we set the threshold value γ_n as the median of these threshold values {${\nu }_n^{(k)}$, 1 ≤ k ≤ K} from K different random permutations, where ${\nu }_n^{(k)}$ is the threshold value for the kth permutation. We set K = 10 in this paper.

2.2. Estimation Method

To address (II), we consider the estimation of B when the true coefficient matrices B_l0s truly have a low-rank structure. The following refined estimation step can be applied after the screening step when the number of covariates is relatively small. For simplicity, we denote the set selected by the screening step ${\widehat{\mathcal{M}}}_{{\gamma }_n}$ by $\widehat{\mathcal{M}}$. Suppose $\widehat{\mathcal{M}}=\{l_1,\dots ,l_{\mid \widehat{\mathcal{M}}\mid }\}$, where $1\le l_1<\dots <l_{\mid \widehat{\mathcal{M}}\mid }\le s_n$. Define $\mathbf{B}=[{\mathbf{B}}_l,l\in \widehat{\mathcal{M}}]=[{\mathbf{B}}_{l_1},\dots ,{\mathbf{B}}_{l_{\mid \widehat{\mathcal{M}}\mid }}]\in {\mathbb{R}}^{p\times q\mid \widehat{\mathcal{M}}\mid }$.

Recently, the trace norm regularization ∥B_l∥_* = ∑_k σ_k(B_l) has been widely used to recover the low-rank structure of B_l due to its computational efficiency, where α_k (B_l) is the kth singular value of B_l. For instance, the trace norm has been used for matrix completion (Candès and Recht, 2009), for matrix regression models with matrix covariates and univariate responses (Zhou and Li, 2014), and for multivariate linear regression with vector responses and scalar covariates (Yuan et al., 2007). Similarly, we propose to calculate the regularized least squares estimator of B by minimizing

$$Q(\mathbf{B})=\frac{1}{2n}\sum _{i=1}^n\Vert {\mathbf{Y}}_i-\sum _{l\in \widehat{\mathcal{M}}}{x}_{il}\ast {\mathbf{B}}_l{\Vert }_F^2+\mathrm{\lambda }\sum _{l\in \widehat{\mathcal{M}}}\Vert {\mathbf{B}}_l{\Vert }_{\ast },$$ (4)

where ∥ · ∥_F is the Frobenius norm of a matrix and λ is a tuning parameter. The low rank structure can be regarded as a special spatial structure, since it is very similar to functional principal component analysis. We use the five-fold cross validation to select the tuning parameter λ. Ideally, we may choose different tuning parameters for different B_l, but it can dramatically increase computational complexity.

We apply the Nesterov gradient method to solve problem (4) even though Q(B) is non-smooth (Nesterov, 2004; Beck and Teboulle, 2009). The Nesterov gradient method utilizes the first-order gradient of the objective function to obtain the next iterate based on the current search point. Unlike the standard gradient descent algorithm, the Nesterov gradient algorithm uses two previous iterates to generate the next search point by extrapolating, which can dramatically improve the convergence rate. Before we introduce the Nesterov gradiant algorithm, we first state a singular value thresholding formula for the trace norm (Cai et al., 2010).

Proposition 1. For a matrix A with {a_k}_1≤k≤r being its singular values, the solution to

$$\underset{\mathrm{B}}{\text{min}}\{\frac{1}{2}\Vert \mathbf{B}-\mathbf{A}{\Vert }_F^2+\mathrm{\lambda }\Vert \mathbf{B}{\Vert }_{\ast }\}$$ (5)

shares the same singular vectors as A and its singular values are b_k = (a_k – λ)₊ for k = 1, …, r.

We present the Nesterov gradient algorithm for problem (4) as follows. Denote $R(\mathbf{B})=(2n{)}^{-1}{\sum }_{i=1}^n\Vert {\mathbf{Y}}_i-{\sum }_{l\in \widehat{\mathcal{M}}}{x}_{il}\ast {\mathbf{B}}_l{\Vert }_F^2$ and $J(\mathbf{B})=\lambda {\sum }_{l\in \widehat{\mathcal{M}}}\Vert {\mathbf{B}}_l{\Vert }_{\ast }$. We also define

$$\begin{array}{cc}\hfill g(\mathbf{B}\mid {\mathbf{S}}^{(t)},\delta )& =R({\mathbf{S}}^{(t)})+<\nabla R({\mathbf{S}}^{(t)}),\mathbf{B}-{\mathbf{S}}^{(t)}>+(2\delta {)}^{-1}\Vert \mathbf{B}-{\mathbf{S}}^{(t)}{\Vert }_F^2+J(\mathbf{B})\hfill \\ \hfill & =(2\delta {)}^{-1}\Vert \mathbf{B}-[{\mathbf{S}}^{(t)}-\delta \nabla R({\mathbf{S}}^{(t)})]{\Vert }_F^2+J(\mathbf{B})+c^{(t)},\hfill \end{array}$$

where ∇R(S^(t)) denotes the first-order gradient of R(S^(t)) with respect to S^(t), S^(t) is an interpolation between B^(t) and B^(t–1) and will be defined below, c^(t) denotes all terms that are irrelevant to B, and δ > 0 is a suitable step size. Given a previous search point S^(t), the next search point S^(t+1) would be the minimizer of g(B∣S^(t), δ). For the search point S^(t), it can be generated by linearly extrapolating two previous algorithmic iterates. A key advantage of using the Nestrov gradient method is that it has an explicit solution at each iteration. Specifically, let B_ld, ${\mathbf{S}}_{l_d}^{(t)}$, and ∇R(S^(t))_ld be the (dq – q + 1)th to the dqth columns of the corresponding $p\times q\mid \widehat{\mathcal{M}}\mid$ matrices B, S^(t), and ∇R(S^(t)), respectively. Minimizing $(2\delta {)}^{-1}\Vert \mathbf{B}-[{\mathbf{S}}^{(t)}-\delta \nabla R({\mathbf{S}}^{(t)})]{\Vert }_F^2+\lambda {\sum }_{l\in \widehat{\mathcal{M}}}\Vert {\mathbf{B}}_l{\Vert }_{\ast }$ is equivalent to solving $\mid \widehat{\mathcal{M}}\mid$ sub-problems, each of which minimizes $(2\delta {)}^{-1}\Vert {\mathbf{B}}_{l_d}-[{\mathbf{S}}_{l_d}^{(t)}-\delta \nabla R({\mathbf{S}}^{(t)}{)}_{l_d}]{\Vert }_F^2+\lambda \Vert {\mathbf{B}}_{l_d}{\Vert }_{\ast }$ for d = 1, …, $\mid \widehat{\mathcal{M}}\mid$, while each sub-problem can be exactly solved by using the singular value thresholding formula given in Proposition 1.

Define ${\mathbf{X}}_{\widehat{\mathcal{M}}}=(x_{il}{)}_{1\le i\le n,l\in \widehat{\mathcal{M}}}$ is an $n\times \mid \widehat{\mathcal{M}}\mid$ matrix and λ_max(·) denotes the largest eigenvalue of a matrix. Our algorithm can be stated as follows:

1. Initialize B⁽⁰⁾ = B⁽¹⁾, α⁽⁰⁾ = 0 and α⁽¹⁾ = 1, t = 1, and $\delta =n∕{\lambda }_{\max }\{({\mathbf{X}}_{\widehat{\mathcal{M}}}{)}^{\mathrm{T}}{\mathbf{X}}_{\widehat{\mathcal{M}}}\}$.

2. Repeat

   $$\begin{array}{c}{\mathbf{S}}^{(t)}={\mathbf{B}}^{(t)}+(\frac{{\alpha }^{(t)}-1}{{\alpha }^{(t)}})({\mathbf{B}}^{(t)}-{\mathbf{B}}^{(t-1)});\hfill \\ \text{for}d=1:\mid \widehat{\mathcal{M}}\mid ,\hfill \\ \text{i}.({\mathbf{A}}_{temp}{)}_{l_d}={\mathbf{S}}_{l_d}^{(t)}-\delta \nabla R({\mathbf{S}}^{(t)}{)}_{l_d};\hfill \\ \text{ii}.\text{compute singular value decomposition}(\text{SVD})({\mathbf{A}}_{temp}{)}_{l_d}={\mathbf{U}}_{l_d}\text{diag}({\mathrm{a}}_{l_d}){\mathbf{V}}_{l_d}^{\mathrm{T}};\hfill \\ \text{iii}.{\mathbf{b}}_{l_d}={\mathrm{a}}_{l_d}-\lambda \delta \ast 1;\hfill \\ \text{iv}.({\mathbf{B}}_{temp}{)}_{l_d}={\mathbf{U}}_{l_d}\text{diag}({\mathbf{b}}_{l_d}){\mathbf{V}}_{l_d}^{\mathrm{T}};\hfill \\ \text{end}\hfill \\ \text{Combine}\{({\mathbf{B}}_{temp}{)}_{l_d},1\le d\le \mid \widehat{\mathcal{M}}\mid \}\text{sub}-\text{matrices and get the entire matrix}{\mathbf{B}}_{temp};\hfill \\ {\mathbf{B}}^{(t+1)}={\mathbf{B}}_{temp};\hfill \\ {\alpha }^{(t+1)}=\{1+\sqrt{1+(2{\alpha }^{(t)}{)}^2}\}∕2;\hfill \\ t=t+1;\hfill \end{array}$$

3. until objective function Q(B^(t)) converges.

For the above $p\times q\mid \widehat{\mathcal{M}}\mid$ matrices A_temp and B_temp, (A_temp)_ld and (B_temp)_ld denote the (dq – q + 1)th to the (dq)th columns of the corresponding matrices, respectively.

A sufficient condition for the convergence of {B^(t)}_t≥1 is that the step size δ should be smaller than or equal to 1/L(R), where L(R) is the smallest Lipschitz constant of the function R(B) (Beck and Teboulle, 2009; Facchinei and Pang, 2003). In our case, L(R) is equal to $n^{-1}{\lambda }_{\max }\{({\mathbf{X}}_{\widehat{\mathcal{M}}}{)}^{\mathrm{T}}{\mathbf{X}}_{\widehat{\mathcal{M}}}\}$.

Remarks: For model (1), it is assumed that x_il has mean 0 and variance 1 for every 1 ≤ l ≤ s, and Y_i has mean 0 throughout the paper. If these assumptions are not valid in practice, a simple solution is to carry out a standardization step including standardizing covariates and centering responses. We use this approach in simulations and real data analysis. An alternative approach is to introduce an intercept matrix term B₀ in model (1). Our screening procedure is invariant to such standardization step if we calculate $B_{l,jk}^M$, the (j, k)–th element of B_l, as the sample correlation between x_il and Y_i,jk. In the Supplementary Material, we present a modified algorithm of our estimation procedure and evaluate the effects of the standardization step on estimating B_l by using simulations. According to our experience, scaling covariates is necessary in order to ensure that all covariates are at the same scale, whereas centering covariates and responses is not critical.

3. Theoretical Properties

To address (III), we systematically investigate several key theoretical properties of the screening procedure and the regularized estimation procedure as well as a theoretical guarantee of our two-step estimator. First, we investigate the sure independence screening property of the rank-one screening procedure when s (also denoted by s_n) diverges at an exponential rate of the sample size. Second, we investigate the estimation and rank consistency of our regularized estimator when both p and q are fixed. Third, we derive the non-asymptotic error bound for our estimator when both p and q are diverging. Finally, we establish an overall theoretical guarantee for our two-step estimator. We state the following theorems, whose detailed proofs can be found in the Appendix B.

3.1. Sure Screening Property

The following assumptions are used to facilitate the technical details, even though they may not be the weakest conditions but help to simplify the proof.

(A0) The covariates X_i are i.i.d from a distribution with mean 0 and covariance matrix ∑_x. Define ${\sigma }_l^2=({\Sigma }_x{)}_{ll}$. The vectorized error matrices vec(E_i) are i.i.d from a distribution with 0 and covariance matrix ∑_e, where vec(·) denotes the vectorization of a matrix. Moreover, x_i and E_i = (E_i,jk) are independent.

(A1) There exist some constants C₁ > 0, b > 0, and 0 < κ < 1/2 such that

$$\underset{l\in \mathcal{M}}{\text{min}}\Vert \text{cov}(\sum _{l^{\prime }\in \mathcal{M}}{x}_{i{l}^{\prime }}\ast {\mathbf{B}}_{l^{\prime }0},x_{il}){\Vert }_{op}\ge C_1(pq{)}^{1∕2}{n}^{-\kappa }\text{and}\underset{l\in \mathcal{M}}{\text{max}}\Vert {\mathbf{B}}_{l0}{\Vert }_{\infty }<b,$$

where $\text{cov}({\sum }_{l^{\prime }\in \mathcal{M}}{x}_{i{l}^{\prime }}\ast {\mathbf{B}}_{l^{\prime }0},x_{il})$ is a p × q matrix with the (j, k)th element being $\text{cov}({\sum }_{l^{\prime }\in \mathcal{M}}{x}_{i{l}^{\prime }}\ast {\mathbf{B}}_{l^{\prime }0,jk},x_{il})$, and $\Vert {\mathbf{B}}_{l0}{\Vert }_{\infty }={\max }_{1\le j\le p,1\le k\le q}\mid B_{l0,jk}\mid$.

(A2) There exist positive constants C₂ and C₃ such that

$$\text{max}(\mathrm{E}\{\text{exp}(C_2{x}_{il}^2)\},\mathrm{E}\{\text{exp}(C_2{E}_{i,jk}^2)\})\le C_3$$

for every 1 ≤ l ≤ s_n, 1 ≤ j ≤ p and 1 ≤ k ≤ q.

(A3) There exists a constant C₄ > 0 such that log(s_n) = C₄n^ξ for ξ ϵ (0, 1 – 2κ).

(A4) There exist constants C₅ > 0 and τ > 0 such that λ_max(∑_x) ≤ C₅n^τ.

(A5) We assume log(pq) = o(n^1–2κ).

Remarks: Assumptions (A0)-(A5) are used to establish the theory of our screening procedure when s_n diverges to infinity. Assumption (A1) is analogous to Condition 3 in Fan and Lv (2008) and equation (4) in Fan and Song (2010), in which κ controls the rate of probability error in recovering the true sparse model. Assumption (A2) is analogous to Condition (D) in Fan and Song (2010) and Condition (E) in Fan et al. (2011). Assumption (A2) requires that x_il and E_i,jk are sub-gaussian, which ensures the tail probability to be exponentially light. Assumption (A3) allows the dimension s_n to diverge at an exponential rate of the sample size n, which is analogous to Condition 1 in Fan and Lv (2008). Assumption (A4) is analogous to Condition 4 in Fan and Lv (2008), which rules out the case of strong collinearity. Assumption (A5) allows the product of the row and column dimensions of the matrix pq to diverge at an exponential rate of the sample size n.

The following theorems show the sure screening property of the screening procedure. We allow p and q to be either fixed or diverging with sample size n.

Theorem 1. Under Assumptions (A0)-(A3) and (A5), let γ_n = αC₁(pq)^1/2n^−κ with 0 < α < 1, then we have $P(\mathcal{M}\subseteq {\widehat{\mathcal{M}}}_{{\gamma }_n})\to 1$ as n → ∞.

Theorem 1 shows that if γ_n is chosen properly, then our rank-one screening procedure will not miss any significant variables with an overwhelming probability. Since the screening procedure automatically includes all the significant covariates for small values of γ_n, it is necessary to consider the size of ${\widehat{\mathcal{M}}}_{{\gamma }_n}$ when γ_n = αC₁(pq)^1/2n^−κ holds.

Theorem 2. Under Assumptions (A0)-(A5), we have $P(\mid {\widehat{\mathcal{M}}}_{{\gamma }_n}\mid =O(n^{2\kappa +\tau }))\to 1$ for γ_n = αC₁(pq)^1/2n^−κ with 0 < α < 1 as n → ∞.

Theorem 2 indicates that the selected model size with the sure screening property is only at a polynomial order of n, even though the original model size is at an exponential order of n. Therefore, the false selection rate of our screening procedure vanishes as n → ∞.

3.2. Theory for Estimation Procedure

From this subsection, we will denote ${\widehat{\mathcal{M}}}_{{\gamma }_n}$ by $\widehat{\mathcal{M}}$ for notation simplicity. We first provide some theoretical results for our estimation procedure. We assume that we can exactly select all the important variables in $\mathcal{M}$, i.e. $\widehat{\mathcal{M}}=\mathcal{M}$, and $s_0=\mid \mathcal{M}\mid$ is fixed. The results are also applicable if our original s is fixed, in which we only need to apply our estimation procedure.

We need more notations before we introduce more assumptions. Suppose the rank of B_l0 is r_l. For every l = 1, …, s_n, we denote ${\mathbf{U}}_{l0}{ϴ}_{l0}{\mathbf{V}}_{l0}^{\mathrm{T}}$ as the singular value decomposition of B_l0 and use ${\mathbf{U}}_{l0}^{\perp }$ and ${\mathbf{V}}_{l0}^{\perp }$ to denote the orthogonal complements of U_l0 and V_l0, respectively. Define ${\Sigma }_{\mathcal{M}}$ as the covariance matrix for ${\mathrm{x}}_{i,\mathcal{M}}$, where ${\mathrm{x}}_{i,\mathcal{M}}=(x_{il}{)}_{l\in \mathcal{M}}\in {\mathbb{R}}^{\mid \mathcal{M}\mid }$. We further define $\mathbf{A}={\Sigma }_{\mathcal{M}}\otimes {\mathbf{I}}_{pq\times pq}$, ${\mathbf{K}}_l={\mathbf{V}}_{l0}^{\perp }\otimes {\mathbf{U}}_{l0}^{\perp }$ and ${\mathrm{d}}_l=-\text{vec}({\mathbf{U}}_{l0}{\mathbf{V}}_{l0}^{\mathrm{T}})$ for $l\in \mathcal{M}$, where ⊗ denotes the Kronecker product. Let $\mathrm{d}=({\mathrm{d}}_{l_1}^{\mathrm{T}},\dots ,{\mathrm{d}}_{l_{\mid \mathcal{M}\mid }}^{\mathrm{T}}{)}^{\mathrm{T}}$ and $\mathbf{K}=\text{diag}\{{\mathbf{K}}_{l_1},\dots ,{\mathbf{K}}_{l_{\mid \mathcal{M}\mid }}\}$. We define ${\Lambda }_l\in {\mathbb{R}}^{(p-r_l)\times (q-r_l)}$ for $l\in \mathcal{M}$ such that $\text{vec}(\Lambda )=(\text{vec}({\Lambda }_{l_1}{)}^{\mathrm{T}},\dots ,\text{vec}({\Lambda }_{l_{\mid \mathcal{M}\mid }}{)}^{\mathrm{T}}{)}^{\mathrm{T}}=({\mathbf{K}}^{\mathrm{T}}{\mathbf{A}}^{-1}\mathbf{K}{)}^{-1}{\mathbf{K}}^{\mathrm{T}}{\mathbf{A}}^{-1}\mathrm{d}$. The Λ_l has some interesting interpretation. For instance, it can be shown that it is the Lagrange multiplier of an optimization problem. We include more interpretation of Λ_l in the Appendix C.

We then state additional assumptions that are needed to establish the theory of our estimation procedure when both p and q are assumed to be fixed.

The following assumptions (A6)-(A8) are needed.

(A6) The ${\Sigma }_{\mathcal{M}}$ is nonsingular.

(A7) The ${\max }_l\{\text{rank}({\mathbf{B}}_{l0}):l\in \mathcal{M}\}<\text{min}(p,q)$ holds.

(A8) For every $l\in \mathcal{M}$, we assume ∥Λ_l∥_op < 1.

Remarks: Assumption (A6) is a regularity condition in the low dimensional context, which rules out the scenario when one covariate is exactly a linear combination of other covariates. Assumption (A7) is used for rank consistency. Assumption (A8) can be regarded as the irrepresentable condition of Zhao and Yu (2006) in the rank consistency context. A similar condition can be found in Bach (2008).

Define ${\widehat{\mathbf{B}}}_l$ the regularized low rank estimator of B_l for $l\in \mathcal{M}$. We have the following consistent results when the tuning parameter converges in different rates when both p and q are fixed.

Theorem 3. (Estimation Consistency) Under Assumptions (A0) and (A6), we have

• (i)if n^1/2 λ → ∞ and λ → 0, then ${\lambda }^{-1}({\widehat{\mathbf{B}}}_l-{\mathbf{B}}_{l0})=O_p(1)$ for all $l\in \mathcal{M}$;
• (ii)if n^1/2 λ → ρ ϵ [0, ∞) and n → ∞, then $n^{1∕2}({\widehat{\mathbf{B}}}_l-{\mathbf{B}}_{l0})=O_p(1)$ for all $l\in \mathcal{M}$.

Theorem 3 reveals an interesting phase-transition phenomenon. When λ is relatively small or moderate, the convergence rate of ${\widehat{\mathbf{B}}}_l-{\mathbf{B}}_{l0}$ is of order n^−1/2, whereas as γ gets large, the convergence rate of ${\widehat{\mathbf{B}}}_l-{\mathbf{B}}_{l0}$ can be approximated as the order of λ. Although we have established the consistency of ${\widehat{\mathbf{B}}}_l$ as λ → 0, the next question is whether the rank of ${\widehat{\mathbf{B}}}_l$ is consistent under the same set of conditions. It turns out that such rank consistency only holds for relatively large λ, whose convergence rate is slower than n^−1/2.

Theorem 4. (Rank Consistency) Under Assumptions (A0) and (A6)-(A8), if λ → 0 and n^1/2λ → ∞ hold, we have that $P(rank({\widehat{\mathbf{B}}}_l)=rank({\mathbf{B}}_l))\to 1$ for all $l\in \mathcal{M}$.

Theorem 4 establishes the rank consistency of our regularized estimates. Theorems 3 and 4 reveal that both of the element consistency and the rank consistency hold only for λ → 0 and n^1/2λ → ∞. This phenomenon is similar to that for the Lasso estimator. Specifically, although the Lasso estimator can achieve model selection consistency, the convergence rate of the Lasso estimator cannot achieve the rate of n^−1/2 when selection consistency is satisfied (Zou, 2006).

We then consider the case when p and q are assumed to be diverging. The following assumptions (A9)-(A12) are needed.

(A9) There exist positive constants C_L and C_M such that $0<C_L\le {\lambda }_{\text{min}}({\Sigma }_{\mathcal{M}})\le {\lambda }_{\max }({\Sigma }_{\mathcal{M}})\le C_M<\infty$.

(A10) We assume that ${\mathrm{x}}_{i,\mathcal{M}}$ are i.i.d multivariate normal with mean 0 and covariance matrix ${\Sigma }_{\mathcal{M}}$.

(A11) The vectorized error matrices vec(E_i) are i.i.d N(0, ∑_e), where ${\lambda }_{max}({\Sigma }_e)\le C_U^2<\infty$.

(A12) We assume max(p, q) → ∞ and max(p, q) = o(n) as n → ∞.

Remarks: Assumptions (A9)-(A12) are needed for our estimation procedure when both p and q are diverging with the sample size n. Assumption (A9) assumes the largest eigenvalue of ${\Sigma }_{\mathcal{M}}$ is bounded and the smallest eigenvalue of ${\Sigma }_{\mathcal{M}}$ is greater than 0. Assumption (A10) assumes that the covariates x_ils are gaussian. Assumption (A11) assumes that the largest eigenvalue of ∑_e is bounded. Assumption (A12) allows p and q to diverge slower than n, but it does allow that pq > n.

We then show the following non-asymptotic bound for our estimation procedure when both p and q are diverging.

Theorem 5. (Nonasymptotic bound when both p and q diverge) Under Assumptions (A9)-(A12), when $\lambda \ge 4C_U{C}_M^{1∕2}{n}^{-1∕2}(p^{1∕2}+q^{1∕2})$, there exist some positive constants c₁, c₂ and c₃ such that with probability at least 1 – c₁ exp{−c₂(p + q)} – c₃ exp(−n), we have

$$\Vert \widehat{\mathbf{B}}-{\mathbf{B}}_0{\Vert }_F^2\le C(\sum _{l\in \mathcal{M}}{r}_l){\mathrm{\lambda }}^2{C}_L^{-2}$$

for some constant C > 0.

Theorem 5 implies that when $\{r_l,l\in \mathcal{M}\}$ and $\mid \mathcal{M}\mid$ are fixed and λ ≍ n^−1/2(p^1/2 + q^1/2), the estimator $\widehat{\mathbf{B}}$ would be consistent with probability going to 1. The convergence rate of the estimator in Theorem 5 coincides with that in Corollary 5 of Negahban et al. (2009), where they studied the low-rank matrix learning problem using the trace norm regularization. Although considering different models, they also require the dimension of the matrix max(p, q) = o(n). It differs significantly from the L1 regularized problem, where the dimension of the matrix may diverge at the exponential order of the sample size. The result in this theorem can also be regarded as a special case of the result in Raskutti and Yuan (2018), where they derived non-asymptotic error bound in a class of tensor regression model with sparse or low-rank penalties.

3.3. Theory for Two-step Estimator

In this section, we give a unified theory for our two-step estimator. In particular, we derive the non-asymptotic bound for our final estimate. To begin with, we first introduce some notations. For simplicity, we will use $\widehat{\mathcal{M}}$ to denote ${\widehat{\mathcal{M}}}_{{\gamma }_n}$, which is the set selected from the first step. Define ${\mathbf{B}}^{\widehat{\mathcal{M}}}=[{\mathbf{B}}_l,l\in \widehat{\mathcal{M}}]\in {\mathbb{R}}^{p\times q\mid \widehat{\mathcal{M}}\mid }$ and the true value of ${\mathbf{B}}^{\widehat{M}}$ as ${\mathbf{B}}_0^{\widehat{M}}=[{\mathbf{B}}_{l0},l\in \widehat{\mathcal{M}}]\in {\mathbb{R}}^{p\times q\mid \widehat{\mathcal{M}}\mid }$. Define ${\widehat{\mathbf{B}}}^{\widehat{\mathcal{M}}}=[{\widehat{\mathbf{B}}}_l,l\in \widehat{\mathcal{M}}]\in {\mathbb{R}}^{p\times q\mid \widehat{\mathcal{M}}\mid }$ as the solution of the regularized trace norm penalization problem given by

$$\underset{{\mathbf{B}}^{\widehat{\mathcal{M}}}}{\text{min}}Q({\mathbf{B}}^{\widehat{\mathcal{M}}})=\underset{{\mathbf{B}}^{\widehat{\mathcal{M}}}}{\text{min}}\{\frac{1}{2n}\sum _{i=1}^n\Vert {\mathbf{Y}}_i-\sum _{l\in \widehat{\mathcal{M}}}{x}_{il}\ast {\mathbf{B}}_l{\Vert }_F^2+\mathrm{\lambda }\sum _{l\in \widehat{\mathcal{M}}}\Vert {\mathbf{B}}_l{\Vert }_{\ast }\}.$$ (6)

We need the following assumptions.

(A13) Assume 2κ + τ < 1. Define ${ı}_L≔{\text{min}}_{\Vert \delta {\Vert }_0\le m,\delta \ne 0}{\delta }^{\mathrm{T}}(n^{-1}{\sum }_{i=1}^n{\mathrm{x}}_i{\mathrm{x}}_i^{\mathrm{T}})\delta ∕\Vert \delta {\Vert }_2^2$ for any m = O(n^2κ+τ) and $\delta \in {\mathbb{R}}^s$. We further assume ι_L > 0.

(A14) Assume max(p, q)/ log(n) → ∞ and max(p, q) = o(n^1–2τ) as n → ∞ with τ < 1/2.

Theorem 6. (Nonasymptotic bound for two-step estimator) Under Assumptions (A0)-(A5), (A10), (A11), (A13), and (A14), when λ ≥ 4C₅n^τ–1/2(p^1/2 + q^1/2), there exist some positive constants c₁, c₂, c₃, c₄, c₅ such that with probability at least 1–c₁n^2κ+τ exp{−c₂(p+q)} – c₃n^2κ+τ exp(−n) – c₄ exp(−c₅n^1−2κ), we have

$$\Vert {\widehat{\mathbf{B}}}^{\widehat{\mathcal{M}}}-{\mathbf{B}}_0^{\widehat{\mathcal{M}}}{\Vert }_F^2\le C(\sum _{l\in \mathcal{M}}{r}_l){\mathrm{\lambda }}^2{ı}_L^{-2}$$

for some constant C > 0.

Theorem 6 implies that when $\{r_l:l\in \mathcal{M}\}$ and $\mid \mathcal{M}\mid$ are fixed and ι_L is fixed, the estimator ${\widehat{\mathbf{B}}}^{\widehat{\mathcal{M}}}$ is consistent with probability going to 1 when λ ≍ n^τ–1/2(p^1/2+q^1/2). Theorem 6 gives an overall theoretical guarantee for our two-step estimator by considering the random selection procedure in the first step. A key fact that we use in the proof of Theorem 6 is that our first-step screening procedure enjoys the sure independence property. In this case, we only need to derive the non-asymptotic bound for the case when we exactly select or over-select the important variables as it holds with overwhelming probability.

4. Simulations

We conduct simulations to examine the finite sample performance of the proposed estimation and screening procedures. For the sake of space, we include additional simulation results in the Supplementary Material.

4.1. Regularized Low-rank Estimate

In the first simulation, we simulate 64 × 64 matrix responses according to model (1) with s = 4 covariates. We set the four true coefficient matrices to be a cross shape (B₁₀), a square shape (B₂₀), a triangle shape (B₃₀), and a butterfly shape (B₄₀). The images of B_l0 are shown in Figure 1, and each of them consists of a yellow region of interest (ROI) containing ones and a blue ROI containing zeros.

Figure 1:

Simulation I setting: the four 64×64 true coefficient matrices for the first simulation setting: the cross shape for B₁₀ in panel (a), the square shape for B₂₀ in panel (b), the triangle shape of B₃₀ in panel (c), and the butterfly shape for B₄₀ in panel (d). The regression coefficient at each pixel is either 0 (blue) or 1 (yellow).

We independently generate all scalar covariates x_i from N(0, ∑_x), where ∑_x = (σ_x,ll′) is a covariance matrix with an autoregressive structure such that ${\sigma }_{x,l{l}^{\prime }}={\rho }_1^{\mid l-l^{\prime }\mid }$ holds for 1 ≤ l,l′ ≤ s with ρ₁ = 0.5. We independently generate vec(E_i) from N(0, ∑_e). Specifically, we set the variances of all elements in E_i to be ${\sigma }_e^2$ and the correlation between E_i,jk and E_i,j′k′ to be ${\rho }_2^{\mid j-j^{\prime }\mid +\mid k-k^{\prime }\mid }$ for 1 ≤ j, k, j′, k′ ≤ 64 with ρ₂ = 0.5. We consider three different sample sizes including n = 100, 200, and 500, and set ${\sigma }_e^2$ to be 1 and 25.

We use 100 replications to evaluate the finite sample performance of our regularized low-rank (RLR) estimates ${\widehat{\mathbf{B}}}_l$ defined as $\Vert {\widehat{\mathbf{B}}}_l-{\mathbf{B}}_{l0}{\Vert }_F^2$. To evaluate the estimation accuracy, we compute the mean squared errors of ${\widehat{\mathbf{B}}}_l$, denoted by MSE(${\widehat{\mathbf{B}}}_l$), for all 1 ≤ l ≤ 4. We also calculate the prediction errors (PE) by generating n^test = 500 independent test observations.

We compare our method with OLS, Lasso, fused Lasso and tensor envelope method (Li and Zhang, 2017). For fair comparison, we also use five-fold cross validation to select regularization parameters of Lasso and fused Lasso and the envelope dimension of the tensor envelope method. The results are shown in Table 1. We also plot the RLR, OLS, Lasso, fused Lasso and tensor envelope estimates of (${\widehat{\mathbf{B}}}_l$, 1 ≤ l ≤ 4} obtained from a randomly selected data set with n = 500 and ${\sigma }_e^2=25$ in Figure 2.

Table 1:

Simulation I results: the means of PEs and MSEs for regularized low-rank (RLR), OLS, Lasso, fused Lasso (Fused) and tensor envelope (Envelope) estimates and their associated standard errors in the parentheses. For each case, 100 simulated data sets are used.

(n, ${\sigma }_e^2$)	Method	MSE(B1)	MSE(B2)	MSE(B3)	MSE(B4)	PE
(100, 1)	RLR	11.67(0.21)	9.96(0.22)	43.21(0.43)	44.88(0.52)	1.03(0.0002)
	OLS	58.08(0.83)	72.38(1.08)	71.66(1.01)	58.00(0.92)	1.05(0.0004)
	Lasso	42.96(0.79)	53.79(1.08)	53.21(0.98)	44.87(0.75)	1.04(0.0004)
	Fused	11.85(0.20)	11.25(0.22)	13.61(0.23)	17.87(0.26)	1.02(0.0002)
	Envelope	21.20(0.34)	24.95(0.36)	51.14(0.49)	55.62(0.67)	1.04(0.0002)
(200, 1)	RLR	7.27(0.09)	6.73(0.10)	23.77(0.20)	23.08(0.23)	1.02(0.0001)
	OLS	28.61(0.30)	34.88(0.36)	34.85(0.38)	28.11(0.32)	1.03(0.0001)
	Lasso	19.29(0.38)	23.86(0.43)	23.73(0.41)	20.30(0.29)	1.02(0.0002)
	Fused	5.93(0.09)	5.62(0.08)	6.59(0.10)	8.63(0.10)	1.01(0.0001)
	Envelope	11.21(0.16)	13.13(0.14)	38.40(0.30)	43.33(0.35)	1.03(0.0001)
(500, 1)	RLR	3.46(0.03)	3.53(0.03)	10.54(0.06)	9.75(0.06)	1.006(0.00003)
	OLS	11.01(0.08)	13.87(0.10)	13.88(0.09)	11.04(0.07)	1.009(0.00003)
	Lasso	5.93(0.17)	7.89(0.17)	7.78(0.18)	6.70(0.13)	1.007(0.00009)
	Fused	2.36(0.03)	2.37(0.02)	2.73(0.03)	3.29(0.03)	1.004(0.00002)
	Envelope	5.44(0.07)	6.60(0.07)	31.72(0.21)	38.29(0.30)	1.02(0.0001)
(100, 25)	RLR	121.61(1.69)	119.58(2.37)	227.58(2.01)	263.90(2.77)	25.37(0.0027)
	OLS	1451.95(20.64)	1809.40(27.01)	1791.53(25.36)	1450.10(23.10)	26.27(0.0099)
	Lasso	1360.23(19.03)	1683.95(24.74)	1669.88(23.97)	1367.71(21.34)	26.22(0.0093)
	Fused	238.87(3.04)	254.78(4.36)	290.50(4.47)	283.07(3.89)	25.42(0.0031)
	Envelope	175.09(1.57)	139.95(2.71)	259.48(2.18)	286.98(1.81)	25.39(0.0023)
(200, 25)	RLR	79.44(1.01)	71.27(1.25)	171.12(1.21)	201.43(1.63)	25.26(0.0013)
	OLS	715.28(7.49)	872.02(8.93)	871.33(9.41)	702.70(8.00)	25.66(0.0037)
	Lasso	657.54(7.19)	798.43(8.58)	798.01(9.04)	652.91(7.14)	25.62(0.0035)
	Fused	156.75(2.00)	162.27(1.95)	174.79(2.34)	175.61(2.20)	25.27(0.0016)
	Envelope	151.68(1.55)	105.18(1.61)	202.96(2.78)	230.24(2.63)	25.29(0.0016)
(500, 25)	RLR	42.17(0.50)	39.70(0.59)	110.16(0.79)	125.08(0.75)	25.10(0.0005)
	OLS	275.31(2.05)	346.69(2.54)	346.93(2.36)	276.05(1.83)	25.24(0.0008)
	Lasso	238.43(2.33)	299.22(2.99)	298.81(2.90)	243.44(1.95)	25.22(0.0011)
	Fused	80.31(0.84)	89.14(0.79)	93.22(0.90)	89.8(0.83)	25.10(0.0006)
	Envelope	95.49(0.99)	75.41(0.99)	142.24(1.51)	171.34(1.31)	25.14(0.0008)

Figure 2:

Simulation I results: the RLR (panels (a)-(d)), OLS (panels (e)-(h)), Lasso (panels (i)-(l)), Fused Lasso (panels (m)-(p)) and Envelope (panels (q)-(t)) estimates of coefficient matrices from a randomly selected training dataset with n = 500, ρ₁ = 0.5, ρ₂ = 0.5 and σ² = 25: ${\widehat{\mathbf{B}}}_1$ (the first column); ${\widehat{\mathbf{B}}}_2$ (the second column); ${\widehat{\mathbf{B}}}_3$ (the third column); and ${\widehat{\mathbf{B}}}_4$ (the fourth column).

Inspecting Figure 2 and Table 1 reveals the following findings. First, our method always outperforms OLS and envelope method. Second, when the images are of low rank (cross and square), our estimation method truly outperforms Lasso. Third, our method outperforms fused Lasso when either the sample size is small or the noise variance is large, whereas fused Lasso outperforms our method in other cases. Fourth, when the images are not of low rank (triangle and butterfly), fused Lasso performs best in most cases, whereas our method outperforms Lasso when either noise level is high or sample size is small.

These findings are not surprising. First, in all settings, since all the true coefficient matrices are piecewise sparse, the fused Lasso method is expected to perform well. Second, Lasso works reasonably well since it still imposes sparse structure. Third, since our method is designed for low rank cases, it performs well for the low rank cross and square cases, whereas it performs relatively worse for the triangle and butterfly cases.

We then conduct the second simulation study when the images only have low rank structure, but no sparse structure. Specifically, we simulate 64 × 64 matrix responses according to model (1) with s = 2 covariates. We set the two true coefficient matrices as ${\mathbf{B}}_{10}={\sum }_{j=1}^{10}{\lambda }_{1,j}{\mathbf{u}}_{1,j}{\mathbf{v}}_{1,j}^{\mathrm{T}}$ and ${\mathbf{B}}_{20}={\sum }_{j=1}^5{\lambda }_2{\mathbf{u}}_{2,j}{\mathbf{v}}_{2,j}^{\mathrm{T}}$, where λ₁ = (λ_1,1, …, λ_1,10^T) = (2, 1.8, 1.6, 1.4, 1.2, 1, 0.8, 0.6, 0.4, 0.2)^T, λ₂ = (λ_2,1, λ_2,2, λ_2,3, λ_2,4, λ_2,5)^T = (2, 1.6, 1.2, 0.8, 0.4)^T, and u_1,j, u_2,j, v_1,j, v_1,j, are column vectors of dimension 64. For U₁ = (u_1,1, …, u_1,10) and V₁ = (v_1,1, …, v_1,10), each of them is generated by orthogonalizing a 64 × 10 matrix with all elements being i.i.d standard normal. For U₂ = (u_2,1, …, u_2,5) and V₂ = (v_2,1, …, v_2,5), each of them is generated by orthogonalizing a 64 × 5 matrix with all elements being i.i.d standard normal. For all other settings, they are the same as those in Section 4.1. Table 2 summarizes the obtained results. Our method outperforms all the comparison methods when the true coefficient matrices are of low rank structure, but of no sparse structure.

Table 2:

Simulation II results: the means of PEs and MSEs for regularized low-rank (RLR) OLS, Lasso, fused Lasso (Fused) and tensor envelope (Envelope) estimates and their associated standard errors in the parentheses. For each case, 100 simulated data sets are used.

(n, ${\sigma }_e^2$	Method	MSE(B1)	MSE(B2)	PE
(100, 1)	RLR	21.86(0.33)	13.91(0.20)	1.02(0.0001)
	OLS	41.85(0.16)	56.82(0.82)	1.03(0.0003)
	Lasso	55.78(0.88)	54.40(0.73)	1.03(0.0002)
	Fused	57.39(0.94)	56.61(0.77)	1.03(0.0002)
	Envelope	41.46(0.15)	33.23(0.54)	1.02(0.0002)
(200, 1)	RLR	10.84(0.12)	6.77(0.07)	1.011(0.00005)
	OLS	20.75(0.07)	27.80(0.30)	1.018(0.0001)
	Lasso	27.90(0.31)	27.03(0.29)	1.018(0.0001)
	Fused	27.69(0.30)	27.29(0.29)	1.018(0.0001)
	Envelope	20.68(0.07)	18.33(0.22)	1.014(0.00008)
(500, 1)	RLR	4.19(0.05)	2.73(0.02)	1.004(0.00002)
	OLS	8.22(0.03)	10.94(0.08)	1.006(0.00003)
	Lasso	12.49(0.18)	12.18(0.17)	1.007(0.00006)
	Fused	10.99(0.08)	10.91(0.09)	1.006(0.00003)
	Envelope	8.26(0.03)	9.37(0.09)	1.006(0.00003)
(100, 25)	RLR	391.95(5.50)	254.67(3.54)	25.37(0.0024)
	OLS	1044(4.10)	1447(23.94)	25.77(0.0060)
	Lasso	1378.32(22.99)	1360.95(18.52)	25.75(0.0059)
	Fused	1232.31(17.74)	1042.72(12.13)	25.68(0.0055)
	Envelope	1033.69(3.66)	626.57(7.71)	25.46(0.0027)
(200, 25)	RLR	219.13(2.14)	136.41(1.33)	25.26(0.0011)
	OLS	518.63(1.83)	694.98(7.47)	25.45(0.0025)
	Lasso	657.39(7.19)	644.78(7.20)	25.44(0.0025)
	Fused	637.01(6.45)	589.9(5.68)	25.43(0.0022)
	Envelope	516.52(1.81)	395.64(3.67)	25.33(0.0015)
(500, 25)	RLR	101.8(0.91)	64.19(0.57)	25.09(0.0005)
	OLS	206.57(0.73)	275.3(2.05)	25.16(0.0008)
	Lasso	259.2(1.95)	254.26(2.17)	25.16(0.0008)
	Fused	265.44(1.89)	255.88(1.99)	25.16(0.0008)
	Envelope	206.41(0.73)	226.94(1.54)	25.14(0.0006)

4.2. Rank-one Screening using SNP Covariates

We generate 64 × 64 matrix responses according to model (1). We use the same method as Section 4.1 to generate E_i with ρ₂ = 0.5 and ${\sigma }_e^2=1$ or 25. We generate genetic covariates by mimicking the SNP data used in Section 5. Specifically, we use Linkage Disequilibrium (LD) blocks defined by the default method (Gabriel et al., 2002) of Haploview (Barrett et al., 2005) and PLINK (Purcell et al., 2007) to form SNP-sets. To calculate LD blocks, n subjects are simulated by randomly combining haplotypes of HapMap CEU subjects. We use PLINK to determine the LD blocks based on these subjects. We randomly select s_n/10 blocks, and combine haplotypes of HapMap CEU subjects in each block to form genotype variables for these subjects. We randomly select 10 SNPs in each block, and thus we have s_n SNPs for each subject. We set s_n = 2, 000 and 5, 000 and choose the first 20 SNPs as the significant SNPs. That is, we set the first 20 true coefficient matrices as nonzero matrices B_1,0 = … = B_20,0 = B_true, and the remaining coefficient matrices as zero. We consider three types of coefficient matrices B_true with different significant regions, i.e. (p_s, q_s) = (4,4), (8, 8), and (16, 16), where p_s and q_s denote the true size of the significant regions of interest. Figure presents the true images B_true and each of them contains a yellow ROI containing ones and a blue ROI containing zeros.

In this subsection, we evaluate the effect of using different γ_n on the finite sample performance of the screening procedure. We will investigate the proposed random decoupling in the next subsection. Specifically, by sorting the magnitude of $\Vert {\widehat{\mathbf{B}}}_l^M{\Vert }_{op}$ in descending order, we define ${\widehat{\mathcal{M}}}^k$ as

$${\widehat{\mathcal{M}}}^k=\{1\le l\le s_n:\Vert {\widehat{\mathbf{B}}}_l^M{\Vert }_{op}\text{is among the}\text{first}\mathit{k}\text{largest of all covariates}\}.$$ (7)

We apply our screening procedure to each simulated data set and then report the average true nonzero coverage proportion as k varies from 1 to 200. In this case, $\mathcal{M}=\{1,2,\dots ,20\}$ is the set of all true nonzero indices, and ${\widehat{\mathcal{M}}}^k$ is the selected index set by using our screening method. The true nonzero coverage proportion is defined as $\mid {\widehat{\mathcal{M}}}^k\cap \mathcal{M}\mid ∕\mid \mathcal{M}\mid$. We consider three different sample sizes including n = 100, 200, and 500. We run 100 Monte Carlo replications for each scenario.

We consider four screening methods including the rank-one screening method, the L1 entrywise norm screening, the Frobenius norm screening, and the global Wald test screening proposed in Huang et al. (2015). The curves of percentage of the average true nonzero coverage proportion for different threshold values are presented for the case (${\sigma }_e^2$, s_n) = (1, 2000) in Figure 4 and for the case (${\sigma }_e^2$, s_n) = (25, 2000) in Figure 5. Inspecting Figures 4 and 5 reveals that the rank-one screening significantly outperforms all other three methods, followed by the Frobenius norm screening. As expected, increasing the sample size n and/or k increases the true nonzero coverage proportion of all four methods. We also include additional simulation results for the cases (${\sigma }_e^2$, s_n) = (1, 5000) in Figure S1 and (${\sigma }_e^2$, s_n) = (25, 5000) in Figure S2 in the Supplementary Material. The findings are similar. Overall, the rank-one screening method is more robust to noise and signal region size.

Figure 4:

Screening simulation results for the case (${\sigma }_e^2$, s_n) = (1, 2000): the curves of percentage of the average true nonzero coverage proportion. The black solid, blue dashed, red dotted, and purple dashed dotted lines correspond to the rank-one screening, the L1 entrywise norm screening, the Frobenius norm screening, and the global Wald test screening, respectively. Panels (a)-(i) correspond to (n, p_s, q_s) = (100, 4, 4), (200, 4, 4), (500, 4, 4), (100, 8, 8), (200, 8, 8), (500, 8, 8), (100, 16, 16), (200, 16, 16), and (500, 16, 16), respectively.

Figure 5:

Screening simulation results for the case (${\sigma }_e^2$, s_n) = (25, 2000): the curves of percentage of the average true nonzero coverage proportion. The black solid, blue dashed, red dotted, and purple dashed dotted lines correspond to the rank-one screening, the L1 entrywise norm screening, the Frobenius norm screening, and the global Wald test screening, respectively. Panels (a)-(i) correspond to (n, p_s, q_s) = (100, 4, 4), (200, 4, 4), (500, 4, 4), (100, 8, 8), (200, 8, 8), (500, 8, 8), (100, 16, 16), (200, 16, 16), and (500, 16, 16), respectively.

4.3. Simulation study for two-step procedure

In this subsection, we perform a simulation study to evaluate our two-step screening and estimation procedure. We simulate 64 × 64 matrix responses according to model (1) with s_n covariates. We set the first four true coefficient matrices to be a cross shape (B₁₀), a square shape (B₂₀), a triangle shape (B₃₀), and a butterfly shape (B₄₀) shown in Figure 1. For the remaining coefficient matrices {B_l0, 5 ≤ l ≤ s_n}, we set them as zero matrices. We consider s_n = 2000 and 5000.

We independently generate all scalar covariates x_i from N(0, ∑_x), where ∑_x = (σ_x,ll′) is a covariance matrix with an autoregressive structure such that ${\sigma }_{x,l{l}^{\prime }}={\rho }_1^{\mid l-l^{\prime }\mid }$ holds for 1 ≤ l,l′ ≤ s with ρ₁ = 0.5. We independently generate vec(E_i) from N(0, ∑_e). Specifically, we set the variances of all elements in E_i to be ${\sigma }_e^2$ and the correlation between E_i,jk and E_i,j′k′ to be ${\rho }_2^{\mid j-j^{\prime }\mid +\mid k-k^{\prime }\mid }$ for 1 ≤ j, k, j′, k′ ≤ 64 with ρ₂ = 0.5. We consider three different sample sizes including n = 100, 200, and 500, and set ${\sigma }_e^2$ to be 1 and 25.

First, we evaluate the finite sample performance of the random decoupling. We perform our screening procedure based on the random decoupling and then apply our regularized low rank estimation procedure. We report the MSEs of ${\widehat{\mathbf{B}}}_l$ (l = 1, 2, 3, 4), model size, and prediction error based on 100 replications in Table 3. We report the proportion of times that we exactly select the true model $\mathcal{M}=\{1,2,3,4\}$, the proportion that we over-select some variables, but include all the true ones, and the proportion that we miss some of the important covariates in Table 4. The proposed random decoupling works pretty well in choosing γ_n, since the selected covariate set based on γ_n includes the true covariates with high probabilities in all scenarios.

Table 3:

The means of predictor errors (PEs) and MSEs for our two-step procedure, and the average selected model size for our screening procedure. Their associated standard errors are in the parentheses. For each case, 100 simulated data sets are used.

(n, sn, ${\sigma }_e^2$)	MSE(B1)	MSE(B2)	MSE(B3)	MSE(B4)	PE	Model Size
(100, 2000, 1)	15.87(3.22)	10.73(0.67)	43.52(0.46)	47.07(0.58)	1.05(0.001)	5.24(0.11)
(200, 2000, 1)	5.92(0.10)	5.10(0.11)	27.61(0.27)	28.23(0.31)	1.03(0.0003)	5.87(0.11)
(100, 5000, 1)	32.28(6.57)	12.60(1.08)	44.28(0.52)	47.14(0.65)	1.07(0.002)	5.03(0.10)
(200, 5000, 1)	5.92(0.09)	4.94(0.09)	28.03(0.29)	28.35(0.29)	1.03(0.0005)	5.83(0.13)
(100, 2000, 25)	126.17(1.84)	119.15(2.56)	227.86(1.96)	279.22(3.17)	25.99(0.027)	5.15(0.11)
(200, 2000, 25)	84.42(1.25)	73.69(1.41)	177.90(1.66)	214.67(2.15)	25.54(0.012)	5.82(0.11)
(100, 5000, 25)	136.27(3.45)	118.73(2.63)	228.65(2.19)	278.43(3.42)	26.04(0.024)	4.96(0.11)
(200, 5000, 25)	82.69(1.12)	73.53(1.37)	177.44(1.57)	211.25(2.12)	25.56(0.012)	5.75(0.13)

Table 4:

The means of PEs and MSEs for our two-step procedure in three scenarios: exact selection (“Exact”), over selection (“Over”) and missing variables (“Miss”). The proportion of times among 100 simulated data sets for each scenario is also reported. The “NA” denotes the values that are not applicable.

(n, sn, ${\sigma }_e^2$)	Scenario	MSE(B1)	MSE(B2)	MSE(B3)	MSE(B4)	PE	Proportion
(100, 2000, 1)	Exact	11.61(0.18)	9.18(0.16)	43.22(0.38)	45.55(0.47)	1.06(0.001)	0.27
	Over	11.17(0.17)	10.11(0.20)	43.6(0.46)	47.31(0.56)	1.05(0.001)	0.71
	Miss	240(0)	53.49(1.68)	44.58(1.61)	59.11(1.18)	1.1(0.001)	0.02
(200, 2000, 1)	Exact	7.09(0.08)	6.6(0.12)	23.97(0.16)	23.11(0.09)	1.03(0.0005)	0.04
	Over	5.87(0.09)	5.03(0.11)	27.76(0.26)	28.44(0.30)	1.03(0.0003)	0.96
	Miss	NA	NA	NA	NA	NA	0
(100, 5000, 1)	Exact	11.69(0.17)	9.74(0.19)	44.14(0.65)	45.79(0.41)	1.07(0.001)	0.27
	Over	11.76(0.20)	9.75(0.19)	44.24(0.43)	46.60(0.54)	1.06(0.001)	0.64
	Miss	240(0)	41.48(1.93)	44.96(0.65)	55.10(1.23)	1.12(0.001)	0.09
(200, 5000, 1)	Exact	7.52(0.09)	6.69(0.11)	24.91(0.22)	24.24(0.08)	1.04(0.001)	0.05
	Over	5.84(0.08)	4.84(0.08)	28.19(0.28)	28.57(0.28)	1.03(0.0005)	0.95
	Miss	NA	NA	NA	NA	NA	0
(100, 2000, 25)	Exact	121.75(1.27)	114.12(2.19)	229.79(1.95)	270.16(3.47)	26.18(0.024)	0.29
	Over	126.37(1.49)	121.47(2.69)	227.21(1.98)	283.29(3.00)	25.91(0.023)	0.70
	Miss	240(NA)	102.16(NA)	217.49(NA)	256.52(NA)	26.45(NA)	0.01
(200, 2000, 25)	Exact	79.26(0.80)	64.22(0.61)	177.62(0.86)	207.41(1.18)	25.54(0.017)	0.07
	Over	84.81(1.27)	74.4(1.43)	177.92(1.71)	215.21(2.20)	25.54(0.012)	0.93
	Miss	NA	NA	NA	NA	NA	0
(100, 5000, 25)	Exact	122.88(1.74)	114.5(2.49)	217.93(1.87)	260.99(2.64)	26.20(0.019)	0.34
	Over	129.81(1.53)	122.59(2.63)	233.46(2.17)	286.15(3.34)	25.92(0.020)	0.58
	Miss	240(0)	108.61(3.01)	239.31(2.06)	296.51(4.25)	26.23(0.022)	0.08
(200, 5000, 25)	Exact	85.2(0.87)	72.92(1.48)	174.83(1.20)	206.34(2.32)	25.66(0.014)	0.06
	Over	82.53(1.14)	73.57(1.37)	177.61(1.60)	211.56(2.12)	25.55(0.012)	0.94
	Miss	NA	NA	NA	NA	NA	0

Second, we consider over-selecting and/or missing some covariates. For each of the three above cases, we report the MSEs of ${\widehat{\mathbf{B}}}_l$ (l = 1, 2, 3, 4) and the prediction error in Table 4. When the screening procedure over-selects more irrelevant variables, the MSEs of the true non-zero coefficient matrices and prediction error of the fitted model are similar to those obtained from the model with the correct set of covariates. In contrast, if the screening procedure misses several important variables, then the estimates corresponding to these missed variables completely fail since the corresponding coefficient matrices are estimated zero. However, according to the simulation results, the MSEs corresponding to those important variables that have been selected, are similar to those obtained from the model with the correct set of covariates. The prediction error increases due to missing some important variables.

5. The Philadelphia Neurodevelopmental Cohort

5.1. Data Description and Preprocessing Pipeline

To motivate the proposed methodology, we consider a large database with imaging, genetic, and clinical data collected by the Philadelphia Neurodevelopmental Cohort (PNC) study. This study was a collaboration between the Center for Applied Genomics (CAG) at Children’s Hospital of Philadelphia (CHOP) and the Brain Behavior Laboratory at the University of Pennsylvania (Penn). The PNC cohort consists of youths aged 8-21 years in the CHOP network and volunteered to participate in genomic studies of complex pediatric disorders. All participants underwent clinical assessment and a neuroscience based computerized neurocognitive battery (CNB) and a subsample underwent neuroimaging. We consider 814 subjects with 429 females and 385 males. The age range of the 814 participants is 8-21 (years) with mean value 14.36 (years) and standard deviation 3.48 (years). Specifically, each subject has a resting state functional magnetic resonance imaging (rs-fMRI) connectivity matrix, which is represented as a 69×69 matrix, and a large genetic data set with around 5, 400, 000 genotyped and imputed single-nucleotide polymorphisms (SNPs) on all of the 22 chromosomes. Other clinical variables of interest include age and gender, among others. Our primary question of interest is to identify novel genetic effects on the local rs-fMRI connectivity changes.

We preprocess the resting state fMRI data using C-PAC pipeline. First, we register the fMRI data to the standard MNI 2mm resolution level and did segmentation using the C-PAC default setting. Next, we do motion correction using the Friston 24-parameter method. We also perform nuisance signal correction by regressing out the following variables: top 5 principle components in the noise regions of interest (ROIs), Cerebrospinal fluid (CSF), motion parameters, and the linear trends in time series. Finally, we extract the ROI time series by taking the average of voxel-wise time series in each ROI. The atlases that we use are HarvardOxford Cortical Atlas (48 regions) and HarvardOxford Subcortical Atlas (21 regions), which could be found in FSL. In total, we extract time series for each of the 69 regions and each time series has 120 observations after deleting the first and last 3 scans.

5.2. Analysis and Results

We first fit model (1) with the rs-fMRI connectivity matrices from 814 subjects as 69 × 69 matrix responses and age and gender as clinical covariates. We also include the first 5 principal component scores based on the SNP data as covariates to correct for population stratification. We first calculate the ordinary least squares estimates of coefficient matrices and then compute the corresponding residual matrices for the brain connectivity response matrix after adjusting the effects of the clinical covariates and the SNP principal component scores.

Second, we apply the rank-one screening procedure by using the residual matrices as responses to select important SNPs from the whole set of 5, 354, 265 SNPs that are highly associated with the residual matrices. We use the random decoupling method described in Section 2.1 to choose the thresholding value γ_n and select all those indices whose $\Vert {\widehat{\mathbf{B}}}_l{\Vert }_{op}$ is the larger than γ_n. Finally, seven covariates are selected, where the names are shown in Table 5. Among these seven SNPS, the first three ones on Chromosome 5 have exactly the same genotypes for all the subjects and the next four ones on Chromosome 10 have exactly the same genotypes for all the subjects.

Table 5:

PNC data analysis results: the top 7 SNPs selected by our screening procedure.

Ranking	Chromosome	SNP
1	5	rs72775042
2	5	rs6881067
3	5	rs72775059
4	10	rs200328746
5	10	rs75860012
6	10	rs200248696
7	10	rs78309702

Finally, we examine the effects of these selected SNPs on our matrix response. We first fit the OLS to these 7 SNPs. Since the first three ones have exactly the same genotypes and the next four ones have exactly the same genotypes, we regress our matrix response on the first selected SNP and the fourth selected SNP, yielding two coefficient matrix estimates ${\widehat{\mathbf{B}}}_{(1)}^{ols}$ and ${\widehat{\mathbf{B}}}_{(2)}^{ols}$. The OLS estimates for the 7 SNPs are defined as ${\widehat{\mathbf{B}}}_{(1)}^{ols}={\widehat{\mathbf{B}}}_{(2)}^{ols}={\widehat{\mathbf{B}}}_{(3)}^{ols}={\widehat{\mathbf{B}}}_{(1)}^{ols}∕3$ and ${\widehat{\mathbf{B}}}_{(4)}^{ols}={\widehat{\mathbf{B}}}_{(5)}^{ols}={\widehat{\mathbf{B}}}_{(6)}^{ols}={\widehat{\mathbf{B}}}_{(7)}^{ols}={\widehat{\mathbf{B}}}_{(2)}^{ols}∕4$. We then calculate the singular values of these 7 OLS estimates and plot these singular values in decreasing order in Figure 6. Inspecting Figure 6 reveals that these estimated coefficient matrices have a clear low rank pattern since the first few singular values dominate the remaining ones. This motivates us to apply our RLR estimation procedure to estimate the coefficient matrices corresponding to these 7 SNP covariates. Figure 7(a)-(g) presents the coefficient matrix estimates associated with these SNPs. The coefficient matrices corresponding to the first three selected SNPs are the same and the coefficient matrices corresponding to next four selected SNPs are the same. The estimated ranks of these seven coefficient matrices are given by 11, 11, 11, 8, 8, 8, and 8, respectively.

Figure 6:

PNC data: Panel (a)-(g) are the plots for the singular values of the OLS estimates corresponding to the 7 SNPs selected by our screening step, with singular values sorted from largest to smallest.

Figure 7:

PNC data: Panel (a)-(g) are the plots for our RLR estimates corresponding to the 7 SNPs selected by our screening step.

6. Discussion

Motivated from the analysis of imaging genetic data, we have proposed a low-rank linear regression model to correlate high-dimensional matrix responses with a high dimensional vector of covariates when coefficient matrices are approximately low-rank. We have developed a fast and efficient rank-one screening procedure, which enjoys the sure independence screening property as well as vanishing false selection rate, to reduce the covariate space. We have developed a regularized estimate of coefficient matrices based on the trace norm regularization, which explicitly incorporates the low-rank structure of coefficient matrices, and established its estimation consistency. We have further established a theoretical guarantee for the overall solution obtained from our two-step screening and estimation procedure. We have demonstrated the efficiency of our methods by using simulations and the analysis of PNC dataset.

Figure 3:

Screening setting: Panel (a)-(c) are the true coefficient images B_true with regions of interest with different sizes: effective regions of interest (yellow ROI) and non-effective regions of interest (blue ROI).

A. Auxiliary Lemmas

In this section, we include the auxiliary lemmas needed for the theorems and their proofs.

Lemma 1. (Bernstein’s inequality) Let Z₁, …, Z_n be independent random variables with zero mean such that E∣Z_i∣^m ≤ m!M^m–2v_i/2 for every m ≥ 2 (and all i) and some positive constants M and v_i. Then P(∣Z₁ + … + Z_n∣ > x) ≤ 2 exp[−x² / {2(v + Mx)}] for v ≥ v₁ + … + v_n.

This lemma is Lemma 2.2.11 of van der Vaart and Wellner (2000), and we omit the proof.

Lemma 2. Under Assumptions A0-A2, for arbitrary t > 0 and every l, l′, j, k, we have that

$$P(\mid \sum _{i=1}^n\{x_{il}{x}_{i{l}^{\prime }}-E(x_{il}{x}_{i{l}^{\prime }})\}\mid \ge t)\le 2\text{exp}\{-\frac{t^2}{2(2n{C}_2^2{e}^{C_2}{C}_3+t∕C_2)}\},$$

and

$$P(\mid \sum _{i=1}^n(x_{il}{E}_{i,jk})\mid \ge t)\le 2\text{exp}\{-\frac{t^2}{2(2n{C}_2^2{e}^{C_2}{C}_3+t∕C_2)}\}.$$

Proofs of Lemma 2: By Assumptions A1 and A2, we have

$$\begin{array}{cc}\hfill & E\left[\text{exp}\{C_2\mid x_{il}{x}_{i{l}^{\prime }}-E(x_{il}{x}_{i{l}^{\prime }})\mid \}\right]\le e^{C_2\mid E(x_{il}{x}_{i{l}^{\prime }})\mid }E\{e^{C_2\mid x_{il}{x}_{i{l}^{\prime }}\mid }\}\hfill \\ \hfill \le & e^{C_2}E\{e^{C_2{x}_{il}^2∕2}{e}^{C_2{x}_{i{l}^{\prime }}^2∕2}\}\le e^{C_2}{\left[E\{e^{C_2{x}_{il}^2}\}E\{e^{C_2{x}_{i{l}^{\prime }}^2}\}\right]}^{1∕2}\le e^{C_2}{C}_3.\hfill \end{array}$$

For every m ≥ 2, one has

$$E\{\mid x_{il}{x}_{i{l}^{\prime }}-E(x_{il}{x}_{i{l}^{\prime }}){\mid }^m\}\le m!C_2^{-m}E\{\text{exp}(C_2\mid x_{il}{x}_{i{l}^{\prime }}-E(x_{il}{x}_{i{l}^{\prime }})\mid )\}\le m!C_2^{-m}{e}^{C_2}{C}_3.$$

It follows from Lemma 1 that we have

$$P(\mid \sum _{i=1}^n\{x_{il}{x}_{i{l}^{\prime }}-E(x_{il}{x}_{i{l}^{\prime }})\}\mid \ge t)\le 2\text{exp}\{-\frac{t^2}{2(2n{C}_2^2{e}^{C_2}{C}_3+t∕C_2)}\}.$$

Similarly, we obtain

$$\begin{array}{cc}\hfill E\{\text{exp}(C_2\mid x_{il}{E}_{i,jk}\mid )\}& \le e^{C_2}E(e^{C_2{x}_{il}^2∕2}{e}^{C_2{E}_{i,jk}^2∕2})\hfill \\ \hfill & \le e^{C_2}{\left\{E(e^{C_2{x}_{il}^2})E(e^{C_2{E}_{i,jk}^2})\right\}}^{1∕2}\le e^{C_2}{C}_3.\hfill \end{array}$$

For every m ≥ 2, we have $E\mid x_{il}{E}_{i,jk}{\mid }^m\le m!C_2^{-m}E\{\text{exp}(C_2\mid x_{il}{E}_{i,jk}\mid )\}\le m!C_2^{-m}{e}^{C_2}{C}_3$. Therefore, it follows from Lemma 1 that we have

$$P(\mid \sum _{i=1}^n(x_{il}{E}_{i,jk})\mid \ge t)\le 2\text{exp}\{-\frac{t^2}{2(2n{C}_2^2{e}^{C_2}{C}_3+t∕C_2)}\}.$$

This completes the proof of Lemma 2.

The next lemma is about the subdifferential and directional derivatives of the trace norm. For more details about this lemma and its proof, please refer to Recht et al. (2010) and Borwein and Lewis (2010).

Lemma 3. For an arbitrary matrix W, its singular value decomposition is denoted by W = UDV^T, where $\mathbf{U}\in {\mathbb{R}}^{p\times m}$ and $\mathbf{V}\in {\mathbb{R}}^{q\times m}$ have orthonormal columns, D = diag(d₁, …, d_m), and d₁ ≥ … ≥ d_m > 0 are the singular values of W. Then the trace norm of W is $\Vert \mathbf{W}{\Vert }_{\ast }={\sum }_{i=1}^m{d}_i$ and its subdifferential is equal to

$$\partial \Vert \mathbf{W}{\Vert }_{\ast }=\{{\mathbf{\text{UDV}}}^{\mathrm{T}}+\mathbf{N},\mathrm{such}\mathrm{that}\Vert \mathbf{N}{\Vert }_{op}\le 1,{\mathbf{U}}^{\mathrm{T}}\mathbf{N}=0,\mathbf{\text{NV}}=0\}.$$

The directional derivative at W is

$$\underset{ϵ\to 0^{+}}{\text{lim}}\frac{\Vert \mathbf{W}+ϵ\mathbf{{\rm Y}}{\Vert }_{\ast }-\Vert \mathbf{W}{\Vert }_{\ast }}{ϵ}=tr({\mathbf{U}}^{\mathrm{T}}\mathbf{{\rm Y}}\mathbf{V})+\Vert ({\mathbf{U}}^{\perp }{)}^{\mathrm{T}}\mathbf{{\rm Y}}{\mathbf{V}}^{\perp }{\Vert }_{\ast },$$

where U^⊥, V^⊥ are the orthonormal complements of U and V.

The following lemma is a standard result called Gaussian comparison inequality (Anderson, 1955).

Lemma 4. Let X and Y be zero-mean vector Gaussian random vectors with covariance matrix ∑_X and ∑_Y respectively. If ∑_X – ∑_Y is positive semi-definite, then for any convex symmetric set $\mathcal{C}$, $P(X\in \mathcal{C})\le P(Y\in \mathcal{C})$.

B. Proof of Theorems

Proof of Theorem 1: Recall that ${\mathbf{B}}_{l0}^M=\text{cov}({\sum }_{l^{\prime }\in \mathcal{M}}{x}_{i{l}^{\prime }}\ast {\mathbf{B}}_{l^{\prime }0},x_{il})$. For every 1 ≤ j ≤ p, 1 ≤ k ≤ q and 1 ≤ l ≤ s_n, we have

$${\widehat{B}}_{l,jk}^M-B_{l0,jk}^M=n^{-1}\sum _{i=1}^n\{x_{il}{Y}_{i,jk}-E(x_{il}{Y}_{i,jk})\}.$$

It follows from Assumptions (A0) (A1) (A2) and Lemma 2 that for any t > 0, we have

$$\begin{array}{cc}\hfill & P(\mid {\widehat{B}}_{l,jk}^M-B_{l0,jk}^M\mid \ge t)=P(\mid \sum _{i=1}^n\{x_{il}{Y}_{i,jk}-E(x_{il}{Y}_{i,jk})\}\mid \ge nt)\hfill \\ \hfill =& P(\mid \sum _{l^{\prime }\in \mathcal{M}}\sum _{i=1}^n\{x_{il}{x}_{i{l}^{\prime }}-E(x_{il}{x}_{i{l}^{\prime }})\}{B}_{l^{\prime }0,jk}+\sum _{i=1}^n{x}_{il}{E}_{i,jk}\mid \ge nt)\hfill \\ \hfill \le & \sum _{l^{\prime }\in \mathcal{M}}P(\mid \sum _{i=1}^n\{x_{il}{x}_{i{l}^{\prime }}-E(x_{il}{x}_{i{l}^{\prime }})\}\mid \ge \frac{nt}{b(s_0+1)})+P(\sum _{i=1}^n\mid x_{il}{E}_{i,jk}\mid \ge \frac{nt}{(s_0+1)})\hfill \\ \hfill \le & 2s_0\text{exp}\{-\frac{n{t}^2{b}^{-2}(s_0+1{)}^{-2}}{2(2C_2^2{e}^{C_2}{C}_3+C_2^{-1}{b}^{-1}(s_0+1{)}^{-1}t)}\}+2\text{exp}\{-\frac{n{t}^2(s_0+1{)}^{-2}}{2(2C_2^2{e}^{C_2}{C}_3+C_2^{-1}(s_0+1{)}^{-1}t)}\}.\hfill \end{array}$$

For every $l\in \mathcal{M}$, we have

$$\begin{array}{cc}\hfill & P(\Vert {\widehat{\mathbf{B}}}_l^M{\Vert }_{op}\le {\gamma }_n)\le P(\Vert {\widehat{\mathbf{B}}}_l^M-{\mathbf{B}}_{l0}^M{\Vert }_{op}\ge (pq{)}^{1∕2}(1-\alpha )C_1{n}^{-\kappa })\hfill \\ \hfill \le & P(\Vert {\widehat{\mathbf{B}}}_l^M-{\mathbf{B}}_{l0}^M{\Vert }_F\ge (pq{)}^{1∕2}(1-\alpha )C_1{n}^{-\kappa })=P(\sum _{j,k}\mid {\widehat{B}}_{l,jk}^M-B_{l0,jk}^M{\mid }^2\ge pq\{(1-\alpha )C_1{n}^{-\kappa }{\}}^2)\hfill \\ \hfill \le & \sum _{j,k}P(\mid {\widehat{B}}_{l,jk}^M-B_{l0,jk}^M\mid \ge (1-\alpha )C_1{n}^{-\kappa })\hfill \\ \hfill \le & 2pq(s_0\text{exp}\{-\frac{n^{1-2\kappa }[(1-\alpha )C_1{b}^{-1}(s_0+1{)}^{-1}{]}^2}{2\{2C_2^2{e}^{C_2}{C}_3+C_2^{-1}{b}^{-1}(s_0+1{)}^{-1}(1-\alpha )C_1{n}^{-\kappa }\}}\}\phantom{)}\hfill \\ \hfill & +\phantom{(}\text{exp}\{-\frac{n^{1-2\kappa }[(1-\alpha )C_1(s_0+1{)}^{-1}{]}^2}{2\{2C_2^2{e}^{C_2}{C}_3+C_2^{-1}(s_0+1{)}^{-1}(1-\alpha )C_1{n}^{-\kappa }\}}\})\hfill \\ \hfill \le & 2pq(s_0\text{exp}\{-\frac{n^{1-2\kappa }[(1-\alpha )C_1{b}^{-1}(s_0+1{)}^{-1}{]}^2}{2\{2C_2^2{e}^{C_2}{C}_3+C_2^{-1}{b}^{-1}(s_0+1{)}^{-1}(1-\alpha )C_1\}}\}\phantom{)}\hfill \\ \hfill & +\phantom{(}\text{exp}\{-\frac{n^{1-2\kappa }[(1-\alpha )C_1(s_0+1{)}^{-1}{]}^2}{2\{2C_2^2{e}^{C_2}{C}_3+C_2^{-1}(s_0+1{)}^{-1}(1-\alpha )C_1\}}\}).\hfill \end{array}$$

Let c₁ = 2pq(s₀ + 1),

$$\begin{array}{cc}\hfill & c_2=\frac{[(1-\alpha )C_1{b}^{-1}(s_0+1{)}^{-1}{]}^2}{2\{2C_2^2{e}^{C_2}{C}_3+C_2^{-1}{b}^{-1}(s_0+1{)}^{-1}(1-\alpha )C_1\}},\text{and}\hfill \\ \hfill & c_3=\frac{[(1-\alpha )C_1(s_0+1{)}^{-1}{]}^2}{2\{2C_2^2{e}^{C_2}{C}_3+C_2^{-1}(s_0+1{)}^{-1}(1-\alpha )C_1\}}.\hfill \end{array}$$

We have $P(\mid {\widehat{\mathbf{B}}}_l^M\mid \le {\gamma }_n)\le 2pq(s_0+1)\text{exp}(-c_0{n}^{1-2\kappa })$, where c₀ = min{c₂, c₃}. By Assumption (A5), one has

$$\begin{array}{cc}\hfill & P(\mathcal{M}\subseteq {\widehat{\mathcal{M}}}_{{\gamma }_n})=P(\bigcap _{l\in \mathcal{M}}\{\mid {\widehat{\mathbf{B}}}_l^M\mid >{\gamma }_n\})\hfill \\ \hfill =& 1-P(\bigcup _{l\in \mathcal{M}}\{\mid {\widehat{\mathbf{B}}}_l^M\mid \le {\gamma }_n\})\ge 1-\sum _{l\in \mathcal{M}}P(\mid {\widehat{\mathbf{B}}}_l^M\mid \le {\gamma }_n)\hfill \\ \hfill \ge & 1-s_0{c}_1\text{exp}(-c_0{n}^{1-2k})=1-2pq(s_0+1)s_0\text{exp}(-c_0{n}^{1-2\kappa })\to 1.\hfill \end{array}$$

This completes the proof of Theorem 1.

Proof of Theorem 2: The proof consists of two steps. In Step 1, we will show that $P({\widehat{\mathcal{M}}}_{{\gamma }_n}\subseteq {\mathcal{M}}^o)\to 1$, where ${\mathcal{M}}^o=\{1\le l\le s_n:{\Vert {\mathbf{B}}_{l0}^M\Vert }_{op}\ge {\gamma }_n∕2\}$. It follows from the definition of ${\widehat{\mathcal{M}}}_{{\gamma }_n}$ that we have

$$P({\widehat{\mathcal{M}}}_{{\gamma }_n}\subseteq {\mathcal{M}}^o)\ge P(\bigcap _{1\le l\le s_n}\{\Vert {\widehat{\mathbf{B}}}_l^M-{\mathbf{B}}_{l0}^M{\Vert }_{op}\le {\gamma }_n∕2\}),$$

Moreover, we have

$$\begin{array}{cc}\hfill & P(\bigcap _{1\le l\le s_n}\{\Vert {\widehat{\mathbf{B}}}_l^M-{\mathbf{B}}_{l0}^M{\Vert }_{op}\le {\gamma }_n∕2\})=1-P(\bigcup _{1\le l\le s_n}\{\Vert {\widehat{\mathbf{B}}}_l^M-{\mathbf{B}}_{l0}^M{\Vert }_{op}\ge {\gamma }_n∕2\})\hfill \\ \hfill \ge & 1-\sum _{1\le l\le s_n}P(\Vert {\widehat{\mathbf{B}}}_l^M-{\mathbf{B}}_{l0}^M{\Vert }_{op}\ge {\gamma }_n∕2)\ge 1-\sum _{1\le l\le s_n}P(\Vert {\widehat{\mathbf{B}}}_l^M-{\mathbf{B}}_{l0}^M{\Vert }_F\ge {\gamma }_n∕2)\hfill \\ \hfill \ge & 1-\sum _{1\le l\le s_n}\sum _{j,k}P(\mid {\widehat{B}}_{l,jk}^M-B_{l0,jk}^M\mid \ge C_1{n}^{-\kappa }∕2)\hfill \\ \hfill \ge & 1-2s_{n}pq[s_0\text{exp}\{-\frac{{\alpha }^2{C}_1^2{b}^{-2}(s_0+1{)}^{-2}{2}^{-2}{n}^{1-2\kappa }}{2(2C_2^2{e}^{C_2}{C}_3+C_2^{-1}{b}^{-1}(s_0+1{)}^{-1}\alpha C_1{2}^{-1}{n}^{-\kappa })}\}\phantom{]}\hfill \\ \hfill & +\phantom{[}\text{exp}\{-\frac{{\alpha }^2{C}_1^2(s_0+1{)}^{-2}{2}^{-2}{n}^{1-2\kappa }}{2(2C_2^2{e}^{C_2}{C}_3+C_2^{-1}(s_0+1{)}^{-1}\alpha C_1{2}^{-1}{n}^{-\kappa })}\}]\hfill \\ \hfill \ge & 1-2s_{n}pq[s_0\text{exp}\{-\frac{{\alpha }^2{C}_1^2{b}^{-2}(s_0+1{)}^{-2}{2}^{-2}{n}^{1-2\kappa }}{2(2C_2^2{e}^{C_2}{C}_3+C_2^{-1}{b}^{-1}(s_0+1{)}^{-1}\alpha C_1{2}^{-1})}\}\phantom{]}\hfill \\ \hfill & +\phantom{[}\text{exp}\{-\frac{{\alpha }^2{C}_1^2(s_0+1{)}^{-2}{2}^{-2}{n}^{1-2\kappa }}{2(2C_2^2{e}^{C_2}{C}_3+C_2^{-1}(s_0+1{)}^{-1}\alpha C_1{2}^{-1})}\}]\hfill \\ \hfill =& 1-2pq\text{exp}(C_4{n}^{\xi })[s_0\text{exp}\{-\frac{{\alpha }^2{C}_1^2{b}^{-2}(s_0+1{)}^{-2}{2}^{-2}{n}^{1-2\kappa }}{2(2C_2^2{e}^{C_2}{C}_3+C_2^{-1}{b}^{-1}(s_0+1{)}^{-1}\alpha C_1{2}^{-1})}\}\phantom{]}\hfill \\ \hfill & +\phantom{[}\text{exp}\{-\frac{{\alpha }^2{C}_1^2(s_0+1{)}^{-2}{2}^{-2}{n}^{1-2\kappa }}{2(2C_2^2{e}^{C_2}{C}_3+C_2^{-1}(s_0+1{)}^{-1}\alpha C_1{2}^{-1})}\}].\hfill \end{array}$$

By Assumptions (A3) and (A5), one has $P({\bigcap }_{1\le l\le s_n}\{\Vert {\widehat{\mathbf{B}}}_l^M-{\mathbf{B}}_{l0}^M{\Vert }_{op}\le {\gamma }_n∕2\})\ge 1-c_4\text{exp}(-c_5{n}^{1-2\kappa })$ for some constants c₄ > 0 and c₅ > 0. Therefore, we have $P({\widehat{\mathcal{M}}}_{{\gamma }_n}\subseteq {\mathcal{M}}^o)\to 1$ by Assumption (A1).

In Step 2, we will show that $\mid {\mathcal{M}}^o\mid =O(n^{2\kappa +\tau })$. Define ${\mathcal{M}}^1=\{1\le l\le s_n:{\Vert {\mathbf{B}}_{l0}^M\Vert }_F^2\ge {\gamma }_n^2∕4\}$. As ${\Vert {\mathbf{B}}_{l0}^M\Vert }_{op}\le {\Vert {\mathbf{B}}_{l0}^M\Vert }_F$, we have ${\mathcal{M}}^o\subseteq {\mathcal{M}}^1$. By the definition of ${\mathcal{M}}^1$, we have

$$\begin{array}{cc}\hfill \mid {\mathcal{M}}^1\mid {\gamma }_n^2∕4& \le \sum _{l=1}^{s_n}\Vert {\mathbf{B}}_{l0}^M{\Vert }_F^2\hfill \\ \hfill & =\sum _{j,k}\sum _{l=1}^{s_n}(B_{l0,jk}^M{)}^2=\sum _{j,k}\sum _{l=1}^{s_n}\{E(x_{il}{Y}_{i,jk}){\}}^2=\sum _{j,k}\Vert E({\mathrm{x}}_i\ast Y_{i,jk}){\Vert }^2.\hfill \end{array}$$

Define B_0,jk = (B_10,jk, –, B_s0,jk)^T, we can write $Y_{i,jk}={\mathbf{x}}_i^{\mathrm{T}}{\mathbf{B}}_{0,jk}+E_{i,jk}$. Multiplying X_i on both sides and taking expectations yield ∑_xB_0,jk = E(x_i * Y_i,jk). Therefore, we have

$$\begin{array}{cc}\hfill & \mid {\mathcal{M}}^1\mid {\gamma }_n^2∕4\le \sum _{j,k}\Vert {\Sigma }_x{\mathbf{B}}_{0,jk}{\Vert }^2\le {\mathrm{\lambda }}_{\max }({\Sigma }_x)\sum _{j,k}{\mathbf{B}}_{0,jk}^{\mathrm{T}}{\mathbf{B}}_{0,jk}\hfill \\ \hfill =& {\mathrm{\lambda }}_{\max }({\Sigma }_x)\sum _{j,k}\{\text{var}(Y_{i,jk})-\text{var}(Y_{i,jk}\mid {\mathrm{x}}_i)\}\le pq{\mathrm{\lambda }}_{\max }({\Sigma }_x).\hfill \end{array}$$

By Assumption A4, we have $\mid {\mathcal{M}}^1\mid \le 4pq{\lambda }_{\max }({\Sigma }_x){\gamma }_n^{-2}=O(n^{2\kappa +\tau })$, which implies that $\mid {\mathcal{M}}^0\mid \le \mid {\mathcal{M}}^1\mid =O(n^{2\kappa +\tau })$.

Combining the results of above two steps leads to

$$P(\mid {\widehat{\mathcal{M}}}_{{\gamma }_n}\mid =O(n^{2\kappa +\tau }))\ge P({\widehat{\mathcal{M}}}_{{\gamma }_n}\subseteq {\mathcal{M}}^0)\to 1.$$

This completes the proof of Theorem 2.

Theorems 3, 4 and 5 are theoretical results for our estimation procedure, and we assume $\widehat{\mathcal{M}}=\mathcal{M}$ and $\widehat{\mathcal{M}}$ is fixed.

Proof of Theorem 3: Without loss of generality, for the proof of Theorem 3, we assume $\widehat{\mathcal{M}}=\mathcal{M}=\{1,\dots ,s\}$ with s fixed for notation simplicity. We first prove Theorem 3 (i). We define

$$\begin{array}{cc}\hfill & L({\mathbf{\Delta }}_1,\dots ,{\mathbf{\Delta }}_s)={\mathrm{\lambda }}^{-2}\{Q(\mathrm{\lambda }{\mathbf{\Delta }}_1+{\mathbf{B}}_{10},\dots ,\mathrm{\lambda }{\mathbf{\Delta }}_s+{\mathbf{B}}_{s0})-Q({\mathbf{B}}_{10},\dots ,{\mathbf{B}}_{s0})\}\hfill \\ \hfill =& 2^{-1}\sum _{l=1}^s\sum _{l^{\prime }=1}^s{n}^{-1}(\sum _{i=1}^n{x}_{il}{x}_{i{l}^{\prime }})tr({\mathbf{\Delta }}_l^{\mathrm{T}}{\mathbf{\Delta }}_{l^{\prime }})-{\mathrm{\lambda }}^{-1}\sum _{l}tr({\mathbf{\Delta }}_l^{\mathrm{T}}{n}^{-1}\sum _{i=1}^n{x}_{il}{\mathrm{E}}_i)\hfill \\ \hfill & +{\mathrm{\lambda }}^{-1}\sum _l\{\Vert {\mathbf{B}}_{l0}+\mathrm{\lambda }{\mathbf{\Delta }}_l{\Vert }_{\ast }-\Vert {\mathbf{B}}_{l0}{\Vert }_{\ast }\},\hfill \end{array}$$

where Δ_l = λ⁻¹(B_l – B_l0) for l = 1, …, s. Therefore, we have

$$({\widehat{\mathbf{\Delta }}}_1,\cdots ,{\widehat{\mathbf{\Delta }}}_s)=\text{arg min}\{L({\mathbf{\Delta }}_1,\cdots ,{\mathbf{\Delta }}_s)\},$$

where ${\widehat{\mathbf{\Delta }}}_l={\lambda }^{-1}({\widehat{\mathbf{B}}}_l-{\mathbf{B}}_{l0})$ for l = 1, …, s.

When λ → 0, n^1/2λ → ∞, we have

$$n^{-1}\sum _{i=1}^n{x}_{il}{x}_{i{l}^{\prime }}{\to }_p{\Sigma }_{\mathcal{M},l{l}^{\prime }},\text{for}\text{every}1\le l,l^{\prime }\le s,$$

where ${\Sigma }_{\mathcal{M},l{l}^{\prime }}$ is the (l, l′)–th element of ${\Sigma }_{\mathcal{M}}$ for 1 ≤ l, l’ ≤ s. By the Central Limit Theorem, $n^{-1∕2}{\sum }_{i=1}^n{x}_{il}{\mathbf{E}}_i$ converges in distribution to a normally distributed matrix D_l with mean 0 and var(vec(D_l)) = m_ll∑_e for every 1 ≤ l ≤ s. Hence

$${\mathrm{\lambda }}^{-1}{n}^{-1}\sum _{i=1}^n{x}_{il}{\mathbf{E}}_i={\mathrm{\lambda }}^{-1}{n}^{-1∕2}{O}_p(1){\to }_{p}0,\text{for}\text{every}1\le l\le s.$$

For every l = 1, …, s, recall that the singular value decomposition of B_l0 is ${\mathbf{U}}_{l0}{ϴ}_{l0}{\mathbf{V}}_{l0}^{\mathrm{T}}$, and ${\mathbf{U}}_{l0}^{\perp }$, and ${\mathbf{V}}_{l0}^{\perp }$ denote orthogonal complements of U_l0 and V_l0, respectively. By Lemma 3, we have

$${\mathrm{\lambda }}^{-1}\sum _j\{\Vert {\mathbf{B}}_{l0}+\mathrm{\lambda }{\mathbf{\Delta }}_l{\Vert }_{\ast }-\Vert {\mathbf{B}}_{l0}{\Vert }_{\ast }\}\to \sum _{l=1}^{s}tr({\mathbf{U}}_{l0}^T{\mathbf{\Delta }}_l{\mathbf{V}}_{l0})+\sum _{l=1}^s\Vert ({\mathbf{U}}_{l0}^{\perp }{)}^{\mathrm{T}}{\mathbf{\Delta }}_l{\mathbf{V}}_{l0}^{\perp }{\Vert }_{\ast }.$$

Consequently, L(Δ₁, …, Δ_s) →_p L⁰(Δ₁, …, Δ_s) for each Δ₁ ϵ G_l,l = 1, …, s with G_l’s compact sets in ${\mathbb{R}}^{p\times q}$, where

$$\begin{array}{cc}\hfill & L^0({\mathbf{\Delta }}_1,\cdots ,{\mathbf{\Delta }}_s)\hfill \\ \hfill =& 2^{-1}\sum _{l=1}^s\sum _{l^{\prime }=1}^s{\Sigma }_{\mathcal{M},l{l}^{\prime }}tr({\mathbf{\Delta }}_l^{\mathrm{T}}{\mathbf{\Delta }}_{l^{\prime }})+\sum _{l=1}^{s}tr({\mathbf{U}}_{l0}^T{\mathbf{\Delta }}_l{\mathbf{V}}_{l0})+\sum _{l=1}^s\Vert ({\mathbf{U}}_{l0}^{\perp }{)}^{\mathrm{T}}{\mathbf{\Delta }}_l{\mathbf{V}}_{l0}^{\perp }{\Vert }_{\ast }\hfill \end{array}$$

One can see that L⁰ (Δ₁, …, Δ_s) is convex, hence it has unique minimum value point (Δ₁₀, …, Δ_s0). As L(Δ₁, …, Δ_s) is also convex, by (Knight and Fu, 2000) we have ${\widehat{\mathbf{\Delta }}}_l{\to }_p{\mathbf{\Delta }}_{l0}$. This implies that ${\lambda }^{-1}({\widehat{\mathbf{B}}}_l-l0)=O_p(1)$, l = 1, ... , s.

We second prove Theorem 3 (ii). We define

$$\begin{array}{cc}\hfill & f({\Psi }_1,\dots ,{\Psi }_s)=n(Q(n^{-1∕2}{\Psi }_l+{\mathbf{B}}_{l0})-Q({\mathbf{B}}_{l0}))\hfill \\ \hfill =& 2^{-1}\sum _{l=1}^s\sum _{l^{\prime }=1}^s{n}^{-1}(\sum _{i=1}^n{x}_{il}{x}_{i{l}^{\prime }})tr({\Psi }_l^{\mathrm{T}}{\Psi }_{l^{\prime }})-\sum _{l}tr({\Psi }_l^{\mathrm{T}}{n}^{-1∕2}\sum _{i=1}^n{x}_{il}\ast {\mathrm{E}}_i)\hfill \\ \hfill & +\mathrm{\lambda }n\sum _l\{\Vert {\mathbf{B}}_{l0}+n^{-1∕2}{\Psi }_l{\Vert }_{\ast }-\Vert {\mathbf{B}}_{l0}{\Vert }_{\ast }\},\hfill \end{array}$$

where Ψ_l = n^1/2(B_l – B_l0) for l = 1, …, s. Let $({\widehat{\Psi }}_1,\dots ,{\widehat{\Psi }}_s)=\arg \min \{f({\Psi }_1,\dots ,{\Psi }_s)\}$, then we have that ${\widehat{\Psi }}_l=n^{1∕2}({\widehat{\mathbf{B}}}_l-{\mathbf{B}}_{l0})$, l = 1, …, s. Under the Assumption (A6), and n^1/2λ → ρ, we have f(Ψ₁ …, Ψ_s) → f⁰(Ψ₁ …, Ψ_s) and

$$\begin{array}{cc}\hfill & f^0({\mathbf{\Psi }}_1,\cdots ,{\mathbf{\Psi }}_s)\hfill \\ \hfill =& 2^{-1}\sum _{l=1}^s\sum _{l^{\prime }=1}^s{\Sigma }_{\mathcal{M},l{l}^{\prime }}tr({\mathbf{\Psi }}_l^{\mathrm{T}}{\mathbf{\Psi }}_{l^{\prime }})-\sum _{l=1}^{s}tr({\mathbf{\Psi }}_l^{\mathrm{T}}{\mathbf{D}}_l)+\rho \{\sum _{l=1}^{s}tr({\mathbf{U}}_{l0}^T{\mathbf{\Delta }}_l{\mathbf{V}}_{l0})+\sum _{l=1}^s\Vert ({\mathbf{U}}_{l0}^{\perp }{)}^{\mathrm{T}}{\mathbf{\Delta }}_l{\mathbf{V}}_{l0}^{\perp }{\Vert }_{\ast }\},\hfill \end{array}$$

where D_l is a random matrix, and vec(D₁) is normally distributed. One can see that f⁰(Ψ₁, …, Ψ_s) is convex, hence it has unique minimum value point (Ψ₁₀, …, Ψ_s0) with Ψ_l0 = O_p(1) for l = 1, …, s. Consequently, by (Knight and Fu, 2000), we have ${\widehat{\Psi }}_l{\to }_d{\Psi }_{l0}$ for l = 1, …, s, which indicates that $n^{1∕2}({\widehat{\mathbf{B}}}_l-{\mathbf{B}}_{l0})=O_p(1)$ for l = 1, …, s. This completes the proof of Theorem 3.

Proof of Theorem 4: Without loss of generality, for the proof of Theorem 4, we assume $\widehat{\mathcal{M}}=\mathcal{M}=\{1,\dots ,s\}$ with s fixed for notation simplicity. It follows from Theorem 3(i) that ${\lambda }^{-1}({\widehat{\mathbf{B}}}_l-{\mathbf{B}}_{l0})=O_p(1)$ holds for every 1 ≤ l ≤ s. Since the rank function is lower semi-continuous, $P(\mathit{rank}({\widehat{\mathbf{B}}}_l)\ge \mathit{rank}({\mathbf{B}}_{l0}))\to 1$. We will then prove $\mathit{rank}(\widehat{\mathbf{B}}=\mathit{rank}({\mathbf{B}}_{l0})$ for every 1 ≤ l ≤ s with probability tending to one.

Denote the singular value decomposition of ${\widehat{\mathbf{B}}}_l$ as ${\widehat{\mathbf{B}}}_l={\widehat{\mathbf{U}}}_l{\widehat{ϴ}}_l{\widehat{\mathbf{V}}}_l^{\mathrm{T}}$, where ${\widehat{\mathbf{U}}}_l\in {\mathbb{R}}^{p\times p}$ and ${\widehat{\mathbf{V}}}_l\in {\mathbb{R}}^{q\times q}$. Let ${\widehat{\mathbf{U}}}_l^{\perp }$ be the submatrix of ${\widehat{\mathbf{U}}}_l$ without the first r_l columns, and ${\widehat{\mathbf{V}}}_l^{\perp }$ is the submatrix of ${\widehat{\mathbf{V}}}_l$ without the first r_l columns, where r_l is the rank of B_l0. Denote the rank of ${\widehat{\mathbf{B}}}_l$ by ${\widehat{r}}_l$. We prove the theorem by two steps.

Step 1. In this step, we will show if

$$\Vert ({\widehat{\mathbf{U}}}_l^{\perp }{)}^{\mathrm{T}}\{n^{-1}\sum _{i=1}^n{x}_{il}[\sum _{l^{\prime }=1}^s{x}_{i{l}^{\prime }}\ast ({\widehat{\mathbf{B}}}_{l^{\prime }}-{\mathbf{B}}_{l^{\prime }0})-{\mathbf{E}}_i]\}{\widehat{\mathbf{V}}}_l^{\perp }{\Vert }_{op}<\mathrm{\lambda },$$

then ${\widehat{r}}_l=r_l$. We will prove the statement by contradiction.

Let ${\widehat{\mathbf{U}}}_{l1}$ be the submatrix of ${\widehat{\mathbf{U}}}_l$ corresponding to the first ${\widehat{r}}_l$ columns, and ${\widehat{\mathbf{V}}}_{l1}$ be the submatrix of ${\widehat{\mathbf{V}}}_l$ corresponding to the first ${\widehat{r}}_l$ columns. If ${\widehat{r}}_l\ge r_l$, we can write ${\widehat{\mathbf{U}}}_l^{\perp }$, ${\widehat{\mathbf{V}}}_l^{\perp }$ as $({\widehat{\mathbf{U}}}_{l1}^{\perp },{\widehat{\mathbf{U}}}_{l2}^{\perp })$, and $({\widehat{\mathbf{V}}}_{l1}^{\perp },{\widehat{\mathbf{V}}}_{l2}^{\perp })$ respectively, where ${\widehat{\mathbf{U}}}_{l1}^{\perp }\in {\mathbb{R}}^{p\times ({\widehat{r}}_l-r_l)}$, ${\widehat{\mathbf{U}}}_{l2}^{\perp }\in {\mathbb{R}}^{p\times (p-{\widehat{r}}_l)}$, ${\widehat{\mathbf{V}}}_{l1}^{\perp }\in {\mathbb{R}}^{q\times ({\widehat{r}}_l-r_l)}$, and ${\widehat{\mathbf{V}}}_{l2}^{\perp }\in {\mathbb{R}}^{p\times (q-{\widehat{r}}_l)}$. By the definition of ${\widehat{\mathbf{B}}}_l$, we have

$${\widehat{\mathbf{B}}}_l=\text{arg}\underset{{\mathbf{B}}_l}{\text{min}}\frac{1}{2n}\sum _{i=1}^n\Vert Y_i-\sum _{l^{\prime }\ne l}{x}_{i{l}^{\prime }}{\widehat{\mathbf{B}}}_l-x_{il}{\mathbf{B}}_l{\Vert }_F^2+\mathrm{\lambda }\Vert {\mathbf{B}}_l{\Vert }_{\ast }.$$

Hence, by Lemma 3, we have

$$\{n^{-1}\sum _{i=1}^n{x}_{il}[\sum _{l^{\prime }=1}^s{x}_{i{l}^{\prime }}\ast ({\widehat{\mathbf{B}}}_{l^{\prime }}-{\mathbf{B}}_{l^{\prime }0})-{\mathbf{E}}_i]\}+\mathrm{\lambda }({\widehat{\mathbf{U}}}_{l1}{\widehat{\mathbf{V}}}_{l1}^{\mathrm{T}}+{\mathbf{N}}_l)=0,$$

with ${\widehat{\mathbf{U}}}_{l1}^{\mathrm{T}}{\mathbf{N}}_l=0$, ${\widehat{\mathbf{N}}}_l{\widehat{\mathbf{V}}}_{l1}=0$ and ∥N_l∥_op ≤ 1. Furthermore, we have

$$\begin{array}{cc}\hfill & ({\widehat{\mathbf{U}}}_l^{\perp }{)}^{\mathrm{T}}\{n^{-1}\sum _{i=1}^n{x}_{il}[\sum _{l^{\prime }=1}^s{x}_{i{l}^{\prime }}\ast ({\widehat{\mathbf{B}}}_{l^{\prime }}-{\mathbf{B}}_{l^{\prime }0})-{\mathbf{E}}_i]\}{\widehat{\mathbf{V}}}_l^{\perp }\hfill \\ \hfill =& -\mathrm{\lambda }({\widehat{\mathbf{U}}}_l^{\perp }{)}^{\mathrm{T}}({\widehat{\mathbf{U}}}_{l1}{\widehat{\mathbf{V}}}_{l1}^{\mathrm{T}}+{\mathbf{N}}_l){\widehat{\mathbf{V}}}_l^{\perp }\hfill \\ \hfill =& -\mathrm{\lambda }({\widehat{\mathbf{U}}}_{l1}^{\perp },{\widehat{\mathbf{U}}}_{l2}^{\perp }{)}^{\mathrm{T}}\{{\widehat{\mathbf{U}}}_{l1}^{\perp }({\widehat{\mathbf{V}}}_{l1}^{\perp }{)}^{\mathrm{T}}+{\mathbf{N}}_l\}({\widehat{\mathbf{V}}}_{l1}^{\perp },{\widehat{\mathbf{V}}}_{l2}^{\perp })\hfill \\ \hfill =& -\mathrm{\lambda }\left(\begin{array}{cc}\hfill {\mathbf{I}}_{({\widehat{r}}_l-r_l)\times ({\widehat{r}}_l-r_l)}\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill ({\widehat{\mathbf{U}}}_{l2}^{\perp }{)}^{\mathrm{T}}{\mathbf{N}}_l{\widehat{\mathbf{V}}}_{l2}^{\perp }\hfill \end{array}\right).\hfill \end{array}$$

From the above formula, it follows that we have ${\Vert {({\widehat{\mathbf{U}}}_l^{\perp })}^T\{n^{-1}{\sum }_{i=1}^n{x}_{il}[{\sum }_{i^{\prime }=1}^n{x}_{i{l}^{\prime }}\ast ({\widehat{\mathbf{B}}}_{l^{\prime }}-{\mathbf{B}}_{l^{\prime }0})-{\mathbf{E}}_i]\}{\widehat{\mathbf{V}}}_l^{\perp }\Vert }_{op}=\lambda$ as long as ${\widehat{r}}_l>r_l$. Consequently,

if ${\Vert {({\widehat{\mathbf{U}}}_l^{\perp })}^T\{n^{-1}{\sum }_{i=1}^n{x}_{il}[{\sum }_{i^{\prime }=1}^s{x}_{i{l}^{\prime }}\ast ({\widehat{\mathbf{B}}}_{l^{\prime }}-{\mathbf{B}}_{l^{\prime }0})-{\mathbf{E}}_i]{\widehat{\mathbf{V}}}_{l1}^{\perp }\}\Vert }_{op}<\lambda$, we have ${\widehat{r}}_l=r_l$.

Step 2. In this step, we will prove that with probability tending to 1, one has

$$\Vert ({\widehat{\mathbf{U}}}_l^{\perp }{)}^{\mathrm{T}}\{n^{-1}\sum _{i=1}^n{x}_{il}[\sum _{l^{\prime }=1}^s{x}_{i{l}^{\prime }}\ast ({\widehat{\mathbf{B}}}_{l^{\prime }}-{\mathbf{B}}_{l^{\prime }0})-{\mathbf{E}}_i]\}{\widehat{\mathbf{V}}}_l^{\perp }{\Vert }_{op}<\mathrm{\lambda }.$$

We have

$$\begin{array}{cc}\hfill & ({\widehat{\mathbf{U}}}_l^{\perp }{)}^{\mathrm{T}}\{n^{-1}\sum _{i=1}^n{x}_{il}[\sum _{l^{\prime }=1}^s{x}_{i{l}^{\prime }}\ast ({\widehat{\mathbf{B}}}_{l^{\prime }}-{\mathbf{B}}_{l^{\prime }0})-{\mathbf{E}}_i]\}{\widehat{\mathbf{V}}}_l^{\perp }\hfill \\ \hfill =& ({\widehat{\mathbf{U}}}_l^{\perp }{)}^{\mathrm{T}}\{\mathrm{\lambda }\sum _{l^{\prime }=1}^s({\Sigma }_{\mathcal{M},l{l}^{\prime }}+o(1)){\widehat{\mathbf{\Delta }}}_l-O_p(n^{-1∕2})\}{\widehat{\mathbf{V}}}_l^{\perp }=\mathrm{\lambda }({\widehat{\mathbf{U}}}_l^{\perp }{)}^{\mathrm{T}}\sum _{l^{\prime }=1}^s{\Sigma }_{\mathcal{M},l{l}^{\prime }}{\widehat{\mathbf{\Delta }}}_l{\widehat{\mathbf{V}}}_l^{\perp }+o_p(\mathrm{\lambda }).\hfill \end{array}$$

Since ${\widehat{\mathbf{B}}}_l$ is a consistent estimator of B_l0, we have ${\widehat{\mathbf{U}}}_l^{\perp }{({\widehat{\mathbf{U}}}_l^{\perp })}^{\mathrm{T}}={\widehat{\mathbf{U}}}_{l0}^{\perp }{({\widehat{\mathbf{U}}}_{l0}^{\perp })}^{\mathrm{T}}+o_p(1)$ and ${\widehat{\mathbf{V}}}_l^{\perp }{({\widehat{\mathbf{V}}}_l^{\perp })}^{\mathrm{T}}={\widehat{\mathbf{V}}}_{l0}^{\perp }{({\widehat{\mathbf{V}}}_{l0}^{\perp })}^{\mathrm{T}}+o_p(1)$. Consequently, we have

$$\begin{array}{cc}\hfill & \Vert ({\widehat{\mathbf{U}}}_l^{\perp }{)}^{\mathrm{T}}\{n^{-1}\sum _{i=1}^n{x}_{il}[\sum _{l^{\prime }=1}^s{x}_{i{l}^{\prime }}\ast ({\widehat{\mathbf{B}}}_{l^{\prime }}-{\mathbf{B}}_{l^{\prime }0})-{\mathbf{E}}_i]\}{\widehat{\mathbf{V}}}_l^{\perp }{\Vert }_{op}\hfill \\ \hfill =& \Vert {\widehat{\mathbf{U}}}_l^{\perp }({\widehat{\mathbf{U}}}_l^{\perp }{)}^{\mathrm{T}}\{n^{-1}\sum _{i=1}^n{x}_{il}[\sum _{l^{\prime }=1}^s{x}_{i{l}^{\prime }}\ast ({\widehat{\mathbf{B}}}_{l^{\prime }}-{\mathbf{B}}_{l^{\prime }0})-{\mathbf{E}}_i]\}{\widehat{\mathbf{V}}}_l^{\perp }({\widehat{\mathbf{V}}}_l^{\perp }{)}^{\mathrm{T}}{\Vert }_{op}\hfill \\ \hfill =& \lambda \Vert {\widehat{\mathbf{U}}}_{l0}^{\perp }({\widehat{\mathbf{U}}}_{l0}^{\perp }{)}^{\mathrm{T}}(\sum _{l^{\prime }=1}^s{\Sigma }_{\mathcal{M},l{l}^{\prime }}{\widehat{\mathbf{\Delta }}}_{l^{\prime }}){\widehat{\mathbf{V}}}_{l0}^{\perp }({\widehat{\mathbf{V}}}_{l0}^{\perp }{)}^{\mathrm{T}}{\Vert }_{op}+o_p(\lambda )\hfill \\ \hfill =& \lambda \Vert {\widehat{\mathbf{U}}}_{l0}^{\perp }({\widehat{\mathbf{U}}}_{l0}^{\perp }{)}^{\mathrm{T}}(\sum _{l^{\prime }=1}^s{\Sigma }_{\mathcal{M},l{l}^{\prime }}{\mathbf{\Delta }}_{l^{\prime }0}){\widehat{\mathbf{V}}}_{l0}^{\perp }({\widehat{\mathbf{V}}}_{l0}^{\perp }{)}^{\mathrm{T}}(1+o_p(1)){\Vert }_{op}+o_p(\lambda )\hfill \\ \hfill =& \lambda \Vert {\mathbf{U}}_{l0}^{\perp }{\mathbf{\Lambda }}_L({\mathbf{V}}_{l0}^{\perp }{)}^{\mathrm{T}}{\Vert }_{op}+o_p(\lambda )=\lambda \{\Vert {\mathbf{\Lambda }}_l{\Vert }_{op}+o_p(1)\}.\hfill \end{array}$$

As ∥Λ_l∥_op < 1, we have ${\Vert {({\widehat{\mathbf{U}}}_l^{\perp })}^T\{n^{-1}{\sum }_{i=1}^n{x}_{il}[{\sum }_{i^{\prime }=1}^s{x}_{i{l}^{\prime }}\ast ({\widehat{\mathbf{B}}}_{l^{\prime }}-{\mathbf{B}}_{l^{\prime }0})-{\mathbf{E}}_i]{\widehat{\mathbf{V}}}_l^{\perp }\}\Vert }_{op}<\lambda$. with probability 1. This completes the proof of Theorem 4.

Proof of Theorem 5. Without loss of generality, for the proof of Theorem 5, we assume $\widehat{\mathcal{M}}=\mathcal{M}=\{1,\dots ,s\}$ with s fixed for notation simplicity. To prove Theorem 5, we first introduce some notations and definitions used in Negahban et al. (2012). Given a pair of subspaces $M\subseteq \overline{M}$, a norm based regularizer J is decomposable with respect to ($(M,{\overline{M}}^{\perp })$) if

$$J(\theta +\gamma )=J(\theta )+J(\gamma )\text{for}\text{all}\theta \in M\text{and}\gamma \in {\overline{M}}^{\perp },$$

where ${\overline{M}}^{\perp }$ is the orthogonal complement of the space $\overline{M}$ defined as ${\overline{M}}^{\perp }=\{v\mid \langle u,v\rangle =0\text{for all}u\in \overline{M}\}$.

We define the projection operator

$${\Pi }_M(u)={\text{argmin}}_{v\in M}\Vert u-v\Vert .$$

Similarly, we can define the projections Π_M^⊥, ${\Pi }_{\overline{M}}$ and ${\Pi }_{{\overline{M}}^{\perp }}$.

We then introduce the definition of the subspace compatibility constant. For the subspace M, the subspace compatibility constant with respect to the pair (J, ∥ · ∥) is given by

$$\psi (M)≔\underset{u\in M\setminus \{0\}}{\text{sup}}\frac{J(u)}{\Vert u\Vert }$$

We introduce the definition of restricted strong convexity. For a loss function L(θ), define δL(Δ, θ) = L(θ + Δ) – L(θ) – ⟨∇L(θ),Δ), where $\nabla L(\theta )=\frac{dL(\theta )}{d\theta }$. The loss function satisfies a restricted strong convexity condition with curvature κ_L > 0 and tolerance function τ_L if

$$\delta L(\mathrm{\Delta },\theta )\ge {\kappa }_L\Vert \mathrm{\Delta }{\Vert }^2-{\tau }_L^2(\theta )\text{for}\text{all}\mathrm{\Delta }\in \mathcal{C}(M,{\overline{M}}^{\perp },\theta ),$$

where $\mathcal{C}(M,{\overline{M}}^{\perp },\theta )=\{\mathrm{\Delta }\mid J({\mathrm{\Delta }}_{{\overline{M}}^{\perp }})\le 3J({\mathrm{\Delta }}_{\overline{M}})+4J({\theta }_{M^{\perp }})\}$.

Now we begin to prove Theorem 5. We need to use the result in Theorem 1 of Negahban et al. (2012). We first check the conditions of the theorem under our context.

Recall that $\mathbf{B}=[{\mathbf{B}}_1,\dots ,{\mathbf{B}}_s]\in {\mathbb{R}}^{p\times qs}$, and r_l = rank(B_l0). Let us consider the class of matrices ${ϴ}_l\in {\mathbb{R}}^{p\times q}$ that have rank r_l ≤ min{p, q} and we define $ϴ=[{ϴ}_1,\dots ,{ϴ}_s]\in {\mathbb{R}}^{p\times qs}$.

Let $\mathit{row}({ϴ}_l)\subseteq {\mathbb{R}}^p$ and $\mathit{col}({ϴ}_l)\subseteq {\mathbb{R}}^q$ denote its row space and column space, respectively. Let U_l and V_l be a given pair of r_l-dimensional subspaces $U_l\subseteq {\mathbb{R}}^p$ and $V_l\subseteq {\mathbb{R}}^q$. Define U = [U₁, …, U_l] and V = [V₁, …, V_l]. For a given pair (U, V), we can define the subspaces M(U, V), $\overline{M}(U,V)$ and ${\overline{M}}^{\perp }(U,V)$ of ${\mathbb{R}}^{p\times qs}$ given by

$$\begin{array}{c}\hfill M(U,V)=\{\mathrm{\Theta }\in {\mathbb{R}}^{p\times qs}\mid row({\mathrm{\Theta }}_l)\subseteq V_l\text{and}\mathit{col}({\mathrm{\Theta }}_l)\subseteq U_l\text{for}1\le l\le s\},\hfill \\ \hfill \overline{M}(U,V)=\{\mathrm{\Theta }\in {\mathbb{R}}^{p\times qs}\mid row({\mathrm{\Theta }}_l)\subseteq V_l\text{or}\mathit{col}({\mathrm{\Theta }}_l)\subseteq U_l\text{for}1\le l\le s\},\hfill \end{array}$$

and

$${\overline{M}}^{\perp }(U,V)=\{\mathrm{\Theta }\in {\mathbb{R}}^{p\times qs}\mid row({\mathrm{\Theta }}_l)\subseteq V_l^{\perp }\text{and}\mathit{col}({\mathrm{\Theta }}_l)\subseteq U_l^{\perp }\text{for}1\le l\le s\},$$

where ${\overline{M}}^{\perp }(U,V)$ is the orthogonal complement of the space $\overline{M}(U,V)$. For simplicity, we will use M, $\overline{M}$ and ${\overline{M}}^{\perp }$ to denote M(U,V), $\overline{M}(U,V)$ and ${\overline{M}}^{\perp }(U,V)$ respectively in the following proof.

Define $J(\mathbf{B})={\sum }_{l=1}^s{\Vert {\mathbf{B}}_l\Vert }_{\ast }$, and we can easily see J(B) is a norm. It is easy to see that the norm J is decomposable with respect to the subspace pair (M, $(M,{\overline{M}}^{\perp })$), where $M\subseteq \overline{M}$. Therefore, the regularizer J satisfies Condition (G1) in Negahban et al. (2012).

Under condition (A9), it is easy to see the loss function R is convex and differentiable, and satisfies the restricted strong convexity with curvature κ_L = C_L and tolerance τ_L = 0, and therefore the Condition (G2) in Negahban et al. (2012) holds.

After we check the conditions, we need to calculate $\psi (\overline{M})$ and $R({\{{\mathbf{B}}_0\}}_{{\overline{M}}^{\perp }})$. It is easy to see $R({\{{\mathbf{B}}_0\}}_{{\overline{M}}^{\perp }})$. For $\psi (\overline{M})$, one has

$$\begin{array}{cc}\hfill \psi (\overline{M})& =\underset{u\in \overline{M}\setminus \{0\}}{\text{sup}}\frac{J(u)}{\Vert u\Vert }=\underset{{\mathbf{B}}_l\in \overline{M}\setminus \{0\}}{\text{sup}}\frac{{\sum }_{l=1}^s\Vert {\mathbf{B}}_l{\Vert }_{\ast }}{\Vert \mathbf{B}{\Vert }_F}\le \frac{{\sum }_{l=1}^s\sqrt{2r_l}\Vert {\mathbf{B}}_l{\Vert }_F}{\Vert \mathbf{B}{\Vert }_F}\hfill \\ \hfill & \le \frac{\sqrt{{\sum }_{l=1}^s(\sqrt{2r_l}{)}^2}\sqrt{{\sum }_{l=1}^s\Vert {\mathbf{B}}_l{\Vert }_F^2}}{\Vert \mathbf{B}{\Vert }_F}\le \sqrt{2\sum _{l=1}^s{r}_l.}\hfill \end{array}$$

Therefore, by Theorem 1 in Negahban et al. (2012), when λ ≥ 2J*(∇R(B₀)), one has ${\Vert \widehat{\mathbf{B}}-{\mathbf{B}}_0\Vert }_F^2\le C({\sum }_{l=1}^s{r}_l){\lambda }^2{C}_L^{-2}$ for some constant C > 0.

The term J*(∇R(B₀)) is actually a random quantity, and our next step is to derive the order of this term.

Define J*(·) as the dual norm of J(·). For any matrix $\mathbf{A}=[{\mathbf{A}}_1,\dots ,{\mathbf{A}}_s]\in {\mathbb{R}}^{p\times qs}$, we will first prove the following result

$$J^{\ast }(\mathbf{A})=\underset{J(\mathbf{B})\le 1}{\text{sup}}\langle \mathbf{A},\mathbf{B}\rangle =\underset{1\le l\le s}{\max }\Vert {\mathbf{A}}_l{\Vert }_{op}.$$ (8)

To prove (8), we first show that J*(A) ≥ max_1≤l≤s ∥A_l∥_op. Let ${\mathbf{B}}^{(l)}=[{\mathbf{B}}_1^{(l)},\dots ,{\mathbf{B}}_s^{(l)}]$ with ${\mathbf{B}}_k^{(l)}=0$ for any k ≠ l and $\Vert {\mathbf{B}}_l^{(l)}\Vert \le 1$. One has

$$J^{\ast }(\mathbf{A})\ge \underset{\Vert {\mathbf{B}}^{(l)}{\Vert }_{\ast }\le 1}{\text{sup}}\langle \mathbf{A},{\mathbf{B}}^{(l)}\rangle =\underset{\Vert {\mathbf{B}}_l^{(l)}{\Vert }_{\ast }\le 1}{\sup }\langle {\mathbf{A}}_l,{\mathbf{B}}_l^{(l)}\rangle =\Vert {\mathbf{A}}_l{\Vert }_{op}.$$

It is easy to see J*(A) ≥ ∥A_l∥_op holds for any 1 ≥ l ≥ s. Consequently, one has J*(A) ≥ max_1≤l≤s ∥A_l∥_op.

Our next step is to show that J*(A) ≤ max_1≤l≤s ∥A_l∥_op. Define the singular value decomposition of ${\mathbf{B}}_l={\mathbf{U}}_l{ϴ}_l{\mathbf{V}}_l^{\mathrm{T}}$. One has

$$\begin{array}{cc}\hfill J^{\ast }(\mathbf{A})& =\underset{J(\mathrm{B})\le 1}{\text{sup}}\{\sum _{l=1}^s\langle {\mathbf{U}}_l{\mathrm{\Theta }}_l{\mathbf{V}}_l^{\mathrm{T}},{\mathbf{A}}_l\rangle \}\hfill \\ \hfill & =\underset{J(\mathbf{B})\le 1}{\text{sup}}\{\sum _{l=1}^s\text{Tr}({\mathbf{V}}_l{\mathrm{\Theta }}_l{\mathbf{U}}_l^{\mathrm{T}}{\mathbf{A}}_l{\mathbf{V}}_l)\}=\underset{J(\mathrm{B})\le 1}{\text{sup}}\{\sum _{l=1}^s\text{Tr}({\mathrm{\Theta }}_l{\mathrm{U}}_l^{\mathrm{T}}{\mathbf{A}}_l{\mathbf{V}}_l)\}\hfill \\ \hfill & =\underset{J(\mathrm{B})\le 1}{\text{sup}}\{\sum _{l=1}^s\langle {\mathbf{U}}_l^{\mathrm{T}}{\mathbf{A}}_l{\mathbf{V}}_l{\mathrm{\Theta }}_l\rangle \}=\underset{J(\mathrm{B})\le 1}{\text{sup}}\{\sum _{l=1}^s\sum _{k=1}^{\text{min}\{p,q\}}{\theta }_{lk}({\mathbf{U}}_l^{\mathrm{T}}{\mathbf{A}}_l{\mathbf{V}}_l{)}_{kk}\}\hfill \\ \hfill & =\underset{J(\mathrm{B})\le 1}{\text{sup}}\{\sum _{l=1}^s\sum _{k=1}^{\text{min}\{p,q\}}{\theta }_{lk}(({\mathbf{U}}_l{)}_{(k)}{)}^{\mathrm{T}}{\mathbf{A}}_l({\mathbf{V}}_l{)}_{(k)}\}\hfill \\ \hfill & \le \underset{J(\mathrm{B})\le 1}{\text{sup}}\{\sum _{l=1}^s\sum _{k=1}^{\text{min}\{p,q\}}{\theta }_{lk}\Vert {\mathbf{A}}_l{\Vert }_{op}\}\le \underset{J(\mathbf{B})\le 1}{\text{sup}}\{\sum _{l=1}^s\sum _{k=1}^{\text{min}\{p,q\}}{\theta }_{lk}\underset{1\le l\le s}{\max }\Vert {\mathbf{A}}_l{\Vert }_{op}\}\hfill \\ \hfill & \le \underset{1\le l\le s}{\max }\Vert {\mathbf{A}}_l{\Vert }_{op},\hfill \end{array}$$

where θ_lk is the kth diagonal element of the diagonal matrix ϴ_l, ${({\mathbf{U}}_l^{\mathrm{T}}{\mathbf{A}}_l{\mathbf{V}}_l)}_{kk}$ is the kkth element of the matrix ${\mathbf{U}}_l^{\mathrm{T}}{\mathbf{A}}_l{\mathbf{V}}_l$, (U_l)(_k) and (V_l)(_k) are the kth column of the matrices U_l and V_l respectively.

Combining the two inequalities, we show that J*(A) = max_1≤l≤s ∥A_l∥_op.

Next we need to calculate J*(∇R(B₀)), where $\nabla R({\mathbf{B}}_0)=[{\mathbf{D}}_1,\dots ,{\mathbf{D}}_s]\in {\mathbb{R}}^{p\times qs}$ with ${\mathbf{D}}_l=-2n^{-1}{\sum }_{i=1}^n{x}_{il}\ast {\mathbf{E}}_i$. We first need to calculate ∥D_l∥_op. We know that operator norm is the dual norm of the trace norm.

From the definition of J*(·), one has

$$\Vert {\mathbf{D}}_l{\Vert }_{op}=2\underset{\Vert \mathbf{A}{\Vert }_{\ast }\le 1}{\text{sup}}\langle \mathbf{A},n^{-1}\sum _{i=1}^n{x}_{il}\ast {\mathbf{E}}_i\rangle$$

To obtain a bound for ∥D_l∥_op, we use similar technique as the one used in Raskutti and Yuan (2018). Let W_i be a p × q random matrix with each entry i.i.d. standard normal. Assuming condition (A11) and by Lemma 4, conditioning on x_il, we get

$$P\{\underset{\Vert \mathbf{A}{\Vert }_{\ast }\le 1}{\text{sup}}\langle \mathbf{A},n^{-1}\sum _{i=1}^n{x}_{il}\ast {\mathbf{E}}_i\rangle >t\}\le P\{\underset{\Vert \mathbf{A}{\Vert }_{\ast }\le 1}{\text{sup}}\langle \mathrm{A},n^{-1}\sum _{i=1}^n{x}_{il}\ast {\mathbf{W}}_i\rangle >\frac{t}{C_U}\},$$

since ${\Sigma }_e\preccurlyeq C_U^2{I}_{pq\times pq}$.

As ${\sup }_{{\Vert \mathbf{A}\Vert }_{\ast }\le 1}\langle \mathbf{A},n^{-1}{\sum }_{i=1}^n{x}_{il}\ast {\mathbf{W}}_i\rangle ={\Vert n^{-1}{\sum }_{i=1}^n{x}_{il}\ast {\mathbf{W}}_i\Vert }_{op}$, conditioning on W_i, each entry of the matrix $n^{-1}{\sum }_{i=1}^n{x}_{il}\ast {\mathbf{W}}_i$ is i.i.d $N(0,\frac{{\Vert {\mathbf{X}}_l\Vert }_{op}^2}{n^2})$, where X_l = (x_1l, …, x_nl)^T. Since $\frac{{\Vert {\mathbf{X}}_l\Vert }_{op}^2}{{\sigma }_l^2})$ is a χ² random variable with n degree of freedom, where ${\sigma }_l^2=({\Sigma }_{\mathcal{M}}{)}_{ll}$, one has

$$P\{\frac{\Vert {\mathbf{X}}_l{\Vert }_{op}^2}{n{\sigma }_l^2}\ge 4\}\le \text{exp}(-n)$$

using the tail bounds of χ². Then combining with the standard random matrix theory, we know that ${\Vert n^{-1}{\sum }_{i=1}^n{x}_{il}\ast {\mathbf{W}}_i\Vert }_{op}\le 2n^{-1∕2}{\sigma }_l(p^{1∕2}+q^{1∕2})$ with probability at least $1-c_1^{\ast }\exp \{-c_2^{\ast }(p+q)\}-\exp (-n)$ where $c_1^{\ast }$ and $c_2^{\ast }$ are some positive constants. Therefore, under conditions (A10) and (A12), there exist some positive constants c₁, c₂ and c₃ such that max_1≤l≤s ∥D_l∥_op ≤ 4C_Un^−1/2(max_1≤l≤s σ_l)(p^1/2 + q^1/2) holds with probability at least 1 – c₁ exp{−c₂(p + q)} – c₃exp(−n). Thus, when $\lambda \ge 4C_U{C}_M^{1∕2}{n}^{-1∕2}(p^{1∕2}+q^{1∕2})$, λ ≥ J*(∇R(B₀)) with probability at least 1 – c₁ exp{−c₂(p + q)} – c₃ exp(−n).

Therefore, with probability 1 – c₁ exp{—c₂(p + q)} – c₃ exp(−n), one has ${\Vert \widehat{\mathbf{B}}-{\mathbf{B}}_0\Vert }_F^2\le C({\sum }_{l\in \mathcal{M}}{r}_l){\lambda }^2{C}_L^{-2}$ for some positive constant C. This completes the proof of Theorem 5.

Proof of Theorem 6:

To prove the theorem, we consider the event $\{\mathcal{M}\subseteq \widehat{\mathcal{M}}\}$ as it holds with probability goes to 1. We will derive the non-asymptotic error bound under the event $\{\mathcal{M}\subseteq \widehat{\mathcal{M}}\}$. Recall that r_l = rank(B_l0), one has r_l = 0 for $l\in \mathcal{M}$. Let us consider the class of matrices ${ϴ}_l\in {\mathbb{R}}^{p\times q}$ that have rank r_l ≤ min{p, q} and we define $ϴ=[{ϴ}_l,l\in \widehat{\mathcal{M}}]\in {\mathbb{R}}^{p\times q\mid \widehat{\mathcal{M}}\mid }$. Let $\mathit{row}({ϴ}_l)\subseteq {\mathbb{R}}^p$ and $\mathit{col}({ϴ}_l)\subseteq {\mathbb{R}}^q$ denote its row space and column space, respectively. Let U_l and V_l be a given pair of r_l-dimensional subspaces $U_l\subseteq {\mathbb{R}}^p$ and $V_l\subseteq {\mathbb{R}}^q$, respectively. Define $U=[U_l,l\in \widehat{\mathcal{M}}]\in {\mathbb{R}}^{p\times q\mid \widehat{\mathcal{M}}\mid }$ and $V=[V_l,l\in \widehat{\mathcal{M}}]\in {\mathbb{R}}^{p\times q\mid \widehat{\mathcal{M}}\mid }$. For a given pair (U, V), we can define the subspaces $\widehat{M}(U,V)$, $\overline{\widehat{M}}(U,V)$ and ${\overline{\widehat{M}}}^{\perp }(U,V)$ of ${\mathbb{R}}^{p\times q\mid \widehat{\mathcal{M}}\mid }$ as follows:

$$\begin{array}{c}\hfill \widehat{M}(U,V)=\{\mathrm{\Theta }\in {\mathbb{R}}^{p\times q\mid \widehat{\mathcal{M}}\mid }\mid row({\mathrm{\Theta }}_l)\subseteq V_l\text{and}\mathit{col}({\mathrm{\Theta }}_l)\subseteq U_l\text{for}l\in \widehat{\mathcal{M}}\},\hfill \\ \hfill \overline{\widehat{M}}(U,V)=\{\mathrm{\Theta }\in {\mathbb{R}}^{p\times q\mid \widehat{\mathcal{M}}\mid }\mid row({\mathrm{\Theta }}_l)\subseteq V_l\text{or}\mathit{col}({\mathrm{\Theta }}_l)\subseteq U_l\text{for}l\in \widehat{\mathcal{M}}\},\hfill \\ \hfill {\overline{\widehat{M}}}^{\perp }(U,V)=\{\mathrm{\Theta }\in {\mathbb{R}}^{p\times q\mid \widehat{\mathcal{M}}\mid }\mid row({\mathrm{\Theta }}_l)\subseteq V_l^{\perp }\text{and}\mathit{col}({\mathrm{\Theta }}_l)\subseteq U_l^{\perp }\text{for}l\in \widehat{\mathcal{M}}\},\hfill \end{array}$$

where ${\overline{\widehat{M}}}^{\perp }(U,V)$ is the orthogonal complement of the space $\overline{\widehat{M}}(U,V)$. For simplicity, we will use $\widehat{M}$, $\overline{\widehat{M}}$ and ${\overline{\widehat{M}}}^{\perp }$ to denote $\widehat{M}(U,V)$, $\overline{\widehat{M}}(U,V)$ and ${\overline{\widehat{M}}}^{\perp }(U,V)$, respectively.

For the norm $J({\mathbf{B}}^{\widehat{\mathcal{M}}})={\sum }_{l\in \widehat{\mathcal{M}}}{\Vert {\mathbf{B}}_l\Vert }_{\ast }$, it is easy to see that the norm J is decomposable with respect to the subspace pair $(\widehat{M},{\overline{\widehat{M}}}^{\perp })$, where $\widehat{M}\subseteq \overline{\widehat{M}}$. Therefore, the regularizer J satisfies Condition (G1) in Negahban et al. (2012).

We need to calculate $\psi \overline{\widehat{M}}$ and $R({\{{\mathbf{B}}_0^{\widehat{\mathcal{M}}}\}}_{{\overline{\widehat{M}}}^{\perp }})$. It is easy to see $R({\{{\mathbf{B}}_0^{\widehat{\mathcal{M}}}\}}_{{\overline{\widehat{M}}}^{\perp }})=0$. For $\psi \overline{\widehat{M}}$, since r_l = 0 holds for $l\in \mathcal{M}$, one has

$$\begin{array}{cc}\hfill \psi (\overline{\widehat{M}})& =\underset{u\in \overline{\widehat{M}}\setminus \{0\}}{\text{sup}}\frac{J(u)}{\Vert u\Vert }=\underset{{\mathbf{B}}_l\in \overline{\widehat{M}}\setminus \{0\}}{\text{sup}}\frac{{\sum }_{l\in \widehat{\mathcal{M}}}\Vert {\mathbf{B}}_l{\Vert }_{\ast }}{\Vert {\mathbf{B}}^{\widehat{\mathcal{M}}}{\Vert }_F}\le \frac{{\sum }_{l\in \mathcal{M}}\sqrt{2r_l}\Vert {\mathbf{B}}_l{\Vert }_F}{\sqrt{{\sum }_{l\in \widehat{\mathcal{M}}}\Vert {\mathbf{B}}_l{\Vert }_F^2}}\hfill \\ \hfill & \le \frac{\sqrt{{\sum }_{l\in \mathcal{M}}(\sqrt{2r_l}{)}^2}\sqrt{{\sum }_{l\in \mathcal{M}}\Vert {\mathbf{B}}_l{\Vert }_F^2}}{\sqrt{{\sum }_{l\in \mathcal{M}}\Vert {\mathbf{B}}_l{\Vert }_F^2}}\le \sqrt{2\sum _{l\in \mathcal{M}}{r}_l.}\hfill \end{array}$$

For any $\mathbf{\Delta }\in {\mathbb{R}}^{p\times q\mid \widehat{\mathcal{M}}\mid }$, we define $F:{\mathbb{R}}^{p\times q\mid \widehat{\mathcal{M}}\mid }\to \mathbb{R}$ as

$$F(\mathbf{\Delta })≔R({\mathbf{B}}_0^{\widehat{\mathcal{M}}}+\mathbf{\Delta })-R({\mathbf{B}}_0^{\widehat{\mathcal{M}}})+\lambda \{J({\mathbf{B}}_0^{\widehat{\mathcal{M}}}+\mathbf{\Delta })-J({\mathbf{B}}_0^{\widehat{\mathcal{M}}})\}.$$

We will derive a lower bound on F(Δ). In particular, we have

$$\begin{array}{cc}\hfill F(\mathrm{\Delta })& =R({\mathbf{B}}_0^{\widehat{\mathcal{M}}}+\mathbf{\Delta })-R({\mathbf{B}}_0^{\widehat{\mathcal{M}}})+\lambda \{J({\mathbf{B}}_0^{\widehat{\mathcal{M}}}+\mathbf{\Delta })-J({\mathbf{B}}_0^{\widehat{\mathcal{M}}})\}\hfill \\ \hfill & \ge \langle \nabla (R({\mathbf{B}}_0^{\widehat{\mathcal{M}}}),\mathbf{\Delta }\rangle +{ı}_L\Vert \mathbf{\Delta }{\Vert }^2+\lambda \{J({\mathbf{B}}_0^{\widehat{\mathcal{M}}}+\mathbf{\Delta })-J({\mathbf{B}}_0^{\widehat{\mathcal{M}}})\}\hfill \\ \hfill & \ge \langle \nabla (R({\mathbf{B}}_0^{\widehat{\mathcal{M}}}),\mathbf{\Delta }\rangle +{ı}_L\Vert \mathbf{\Delta }{\Vert }^2+\lambda \{J({\mathbf{\Delta }}_{{\overline{\widehat{M}}}^{\perp }})-J({\mathbf{\Delta }}_{\overline{\widehat{M}}})-2J(({\mathbf{B}}_0^{\widehat{\mathcal{M}}}{)}_{{\widehat{M}}^{\perp }})\},\hfill \end{array}$$

where the first inequality follows from condition (A13) and the second inequality follows from Lemma 3 in Negahban et al. (2012) by applying to the pair $(\overline{\widehat{M}},{\overline{\widehat{M}}}^{\perp })$.

By the Cauchy-Schwarz inequality applied to the regularizer J and its dual J*, we have $\mid \langle \nabla R({\mathbf{B}}_0^{\widehat{\mathcal{M}}}),\mathrm{\Delta }\rangle \mid \le J^{\ast }(\nabla R({\mathbf{B}}_0^{\widehat{\mathcal{M}}}))J(\mathbf{\Delta })$. Since $\lambda \ge 2J^{\ast }(\nabla R({\mathbf{B}}_0^{\widehat{\mathcal{M}}}))$ holds by assumption, one has $\mid \langle \nabla R({\mathbf{B}}_0^{\widehat{\mathcal{M}}}),\mathrm{\Delta }\rangle \mid \le 0.5\lambda J(\mathbf{\Delta })\le 0.5\lambda (J({\mathbf{\Delta }}_{{\overline{\widehat{M}}}^{\perp }})+J({\mathbf{\Delta }}_{\overline{\widehat{M}}}))$, where the second inequality holds due to the triangle inequality. Therefore, we have

$$\begin{array}{cc}\hfill F(\mathbf{\Delta })& \ge -\frac{\lambda }{2}\{J({\mathbf{\Delta }}_{{\overline{\widehat{M}}}^{\perp }})+J({\mathbf{\Delta }}_{\overline{\widehat{M}}})\}+{ı}_L\Vert \mathbf{\Delta }{\Vert }^2+\lambda \{J({\mathbf{\Delta }}_{{\overline{\widehat{M}}}^{\perp }})-J({\mathbf{\Delta }}_{\overline{\widehat{M}}})-2J(({\mathbf{B}}_0^{\widehat{\mathcal{M}}}{)}_{{\widehat{M}}^{\perp }})\}\hfill \\ \hfill & ={ı}_L\Vert \mathbf{\Delta }{\Vert }^2+\lambda \{\frac{1}{2}J({\mathbf{\Delta }}_{{\overline{\widehat{M}}}^{\perp }})-\frac{3}{2}J({\mathbf{\Delta }}_{\overline{\widehat{M}}})-2J(({\mathbf{B}}_0^{\widehat{\mathcal{M}}}{)}_{{\widehat{M}}^{\perp }})\}\hfill \\ \hfill & \ge {ı}_L\Vert \mathbf{\Delta }{\Vert }^2-\frac{1}{2}\lambda \{3J({\mathbf{\Delta }}_{\overline{\widehat{M}}})+4J(({\mathbf{B}}_0^{\widehat{\mathcal{M}}}{)}_{{\widehat{M}}^{\perp }})\}.\hfill \end{array}$$

By the subspace compatibility, we have $J({\mathbf{\Delta }}_{\overline{\widehat{M}}})\le \psi (\overline{\widehat{M}})\Vert {\mathbf{\Delta }}_{\overline{\widehat{M}}}\Vert$. As the projection is non-expansive and $0\in \overline{\widehat{M}}$, one has $\Vert {\mathbf{\Delta }}_{\overline{\widehat{M}}}\Vert \le \Vert \mathbf{\Delta }\Vert$, and thus $J({\mathbf{\Delta }}_{\overline{\widehat{M}}})\le \psi (\overline{\widehat{M}})\Vert \mathbf{\Delta }\Vert$. Substituting it into the previous inequality, and noticing that $J({({\mathbf{B}}_0^{\widehat{\mathcal{M}}})}_{{\widehat{M}}^{\perp }})=0$, we obtain $F(\mathbf{\Delta })\ge {\iota }_L{\Vert \mathbf{\Delta }\Vert }^2-\frac{3}{2}\lambda \psi (\overline{\widehat{M}})\Vert \mathbf{\Delta }\Vert$. The righthand side is a quadratic form of Δ, as long as ${\Vert \mathbf{\Delta }\Vert }^2>\frac{9{\lambda }^2}{4{\iota }_L^2}\psi (\overline{\widehat{M}})$, one has F(Δ) > 0. By Lemma 4 in Negahban et al. (2012), we have ${\Vert {\widehat{\mathbf{B}}}^{\widehat{\mathcal{M}}}-{\mathbf{B}}_0^{\widehat{\mathcal{M}}}\Vert }_F^2\le C({\sum }_{l\in \mathcal{M}}{r}_l){\lambda }^2{\iota }_L^{-2}$ for some positive constant C.

Next we need to calculate $J^{\ast }(\nabla R({\mathbf{B}}_0^{\widehat{\mathcal{M}}}))$, where $\nabla R({\mathbf{B}}_0^{\widehat{\mathcal{M}}})=[{\mathbf{D}}_l,l\in \widehat{\mathcal{M}}]\in {\mathbb{R}}^{p\times q\mid \widehat{\mathcal{M}}\mid }$ with ${\mathbf{D}}_l=-2n^{-1}{\sum }_{i=1}^n{x}_{il}\ast {\mathbf{E}}_i$. By similar argument as the one in the proof of Theorem 5, one has $J^{\ast }(\nabla R({\mathbf{B}}_0^{\widehat{\mathcal{M}}}))={\max }_{l\in \widehat{\mathcal{M}}}{\Vert {\mathbf{D}}_l\Vert }_{op}$. To calculate ∥D_l∥_op, by the same argument as the one in proof of Theorem 5, one has ${\Vert n^{-1}{\sum }_{i=1}^n{x}_{il}\ast {\mathbf{W}}_i\Vert }_{op}\le 2n^{-1∕2}{\sigma }_l(p^{1∕2}+q^{1∕2})$ with probability at least $1-c_1^{\ast }\exp \{-c_2^{\ast }(p+q)\}-\exp (-n)$, where $c_1^{\ast }$ and $c_2^{\ast }$ are some positive constants. Therefore, one has $J^{\ast }(\nabla R({\mathbf{B}}_0^{\widehat{\mathcal{M}}}))={\max }_{l\in \widehat{\mathcal{M}}}{\Vert {\mathbf{D}}_l\Vert }_{op}\le 4n^{-1∕2}({\max }_{l\in \widehat{\mathcal{M}}}{\sigma }_l)(p^{1∕2}+q^{1∕2})$ with probability at least $1-\mid \widehat{\mathcal{M}}\mid c_1^{\ast }\exp \{-c_2^{\ast }(p+q)\}-\mid \widehat{\mathcal{M}}\mid \exp (-n)$. By condition (A4), one has ${\max }_{l\in \widehat{\mathcal{M}}}{\sigma }_l\le {\lambda }_{\max }({\Sigma }_x)\le C_5{n}^{\tau }$. By the proof of Theorem 2, one has $\mid \widehat{\mathcal{M}}\mid =O(n^{2\kappa +\tau })$ with probability at least $1-c_4^{\ast }\exp (-c_5^{\ast }{n}^{1-2\kappa })$ for some positive constants $c_4^{\ast }$ and $c_5^{\ast }$. Thus, when λ ≥ 4C₅n^τ–1/2(p^1/2 + q^1/2), one has $\lambda \ge J^{\ast }(\nabla R({\mathbf{B}}_0^{\widehat{\mathcal{M}}}))$ with probability at least $1-c_1{n}^{2\kappa +\tau }\exp \{-c_2(p+q)\}-c_3{n}^{2\kappa +\tau }\exp (-n)-c_4^{\ast }\exp (-c_5^{\ast }{n}^{1-2\kappa })$ for some positive constants c₁, c₂, c₃, $c_4^{\ast }$ and $c_5^{\ast }$.

By the proof of Theorem 1, the event $\{\mathcal{M}\subseteq \widehat{\mathcal{M}}\}$ holds with probability goes to 1. In particular, $P(\{\mathcal{M}\subseteq \widehat{\mathcal{M}}\})\ge 1-c_4^{\ast \ast }\exp (-c_5^{\ast \ast }{n}^{1-2k})$ for some positive constants $c_4^{\ast \ast }$ and $c_5^{\ast \ast }$. Therefore, there exists some positive constants c₁, c₂, c₃, c₄ and c₅ such that with probability 1 – c₁n^2κ+τ exp{−c₂(p + q)} – c₃n^2κ+τ exp(−n) – c₄ exp(−c₅n^1–2κ), one has ${\Vert {\widehat{\mathbf{B}}}^{\widehat{\mathcal{M}}}-{\mathbf{B}}_0^{\widehat{\mathcal{M}}}\Vert }_F^2\le C({\sum }_{l\in \mathcal{M}}{r}_l){\lambda }^2{\iota }_L^{-2}$ for some positive constant C. When Assumptions (A5) and (A14) hold, with probability goes to 1, one has ${\Vert {\widehat{\mathbf{B}}}^{\widehat{\mathcal{M}}}-{\mathbf{B}}_0^{\widehat{\mathcal{M}}}\Vert }_F^2\le C({\sum }_{l\in \mathcal{M}}{r}_l){\lambda }^2{\iota }_L^{-2}$. This completes the proof of Theorem 6.

C. Interpretations of Λ_l

In this section, we include some detailed interpretations of the definition Λ_l. Without loss of generality, we assume $\widehat{\mathcal{M}}=\mathcal{M}=\{1,\dots ,s\}$ with s fixed for notation simplicity. We first give a necessary condition for rank consistency presented in Theorem 4. By proposition 18 of (Bach, 2008), for any 1 ≤ l ≤ s, we have ${({\mathbf{U}}_{l0}^{\perp })}^{\mathrm{T}}{\widehat{\mathbf{\Delta }}}_l{\mathbf{V}}_{l0}^{\perp }=o_p(1)$ if $\mathit{rank}({\widehat{\mathbf{B}}}_l)=\mathit{rank}({\mathbf{B}}_{l0})=r_l$. Since ${\widehat{\mathbf{\Delta }}}_l{\to }_p{\mathbf{\Delta }}_{l0}$ and Δ_l0 is a nonrandom quantity, we have ${({\mathbf{U}}_{l0}^{\perp })}^{\mathrm{T}}{\mathbf{\Delta }}_{l0}{\mathbf{V}}_{l0}^{\perp }=0$. Recall that {Δ_l0 : 1 ≤ l ≤ s} is the minimizer of l⁰(Δ₁, …, Δ_s), and thus {Δ_l0 : 1 ≤ l ≤ s} is the solution of the optimal problem

$$\text{min}{l}^0(\mathbf{\Delta })\text{subject}\text{to}({\mathbf{U}}_{l0}^{\perp }{)}^{\mathrm{T}}{\mathbf{\Delta }}_l{\mathbf{V}}_{l0}^{\perp }=0\text{for}\text{every}1\le l\le s.$$ (9)

Using Lagrange multiplier method, consider the minimizer of

$$L(\mathbf{\Delta },{\mathbf{\Lambda }}_1,\dots ,{\mathbf{\Lambda }}_s)=2^{-1}\text{vec}(\mathbf{\Delta }{)}^{\mathrm{T}}{\Sigma }_{\mathcal{M}}\text{vec}(\mathbf{\Delta })+\sum _{l=1}^{s}tr({\mathbf{U}}_{l0}^{\mathrm{T}}{\mathbf{\Delta }}_l{\mathbf{V}}_{l0})+\sum _{l=1}^{s}tr({\mathbf{\Lambda }}_l^{\mathrm{T}}({\mathbf{U}}_{l0}^{\perp }{)}^{\mathrm{T}}{\mathbf{\Delta }}_l{\mathbf{V}}_{l0}^{\perp })$$

where {Λ_l, l = 1, …, s} are Lagrange multipliers. Thus, for l = 1, …, s, {Δ_l0 : 1 ≤ l ≤ s} satisfies

$$\begin{array}{c}\hfill \frac{\partial L}{\partial {\mathbf{\Delta }}_l}=\sum _{l^{\prime }=1}^s{\Sigma }_{\mathcal{M},l{l}^{\prime }}{\mathbf{\Delta }}_{l0}+{\mathbf{U}}_{l0}{\mathbf{V}}_{l0}^{\mathrm{T}}+{\mathbf{U}}_{l0}^{\perp }{\mathbf{\Lambda }}_l({\mathbf{V}}_{l0}^{\perp }{)}^{\mathrm{T}}=0,\hfill \\ \hfill \frac{\partial L}{\partial {\mathbf{\Lambda }}_l}=({\mathbf{U}}_{l0}^{\perp }{)}^{\mathrm{T}}{\mathbf{\Delta }}_{l0}{\mathbf{V}}_{l0}^{\perp }=0.\hfill \end{array}$$

Recall that $\mathbf{A}={\Sigma }_{\mathcal{M}}\otimes {\mathbf{I}}_{pq\times pq}$, ${\mathbf{K}}_l={\mathbf{V}}_{l0}^{\perp }\otimes {\mathbf{U}}_{l0}^{\perp }$ , and ${\mathbf{d}}_l=-\mathrm{vec}({\mathbf{U}}_{l0}{\mathbf{V}}_{l0}^{\mathbf{T}})$ for l = 1, . . . s, where ⊗ denotes the Kronecker product. Let $\mathbf{d}=({\mathbf{d}}_1^{\mathrm{T}},\dots ,{\mathbf{d}}_s^{\mathrm{T}})\mathrm{T}$ , K = diag{K₁, …, K_s}, ${\Lambda }_l\in {\mathbb{R}}^{(p-r_l)\times (q-r_l)}$ for l = 1, …, s such that

$$\text{vec}(\mathbf{\Lambda })=(\text{vec}({\mathbf{\Lambda }}_1{)}^{\mathrm{T}},\dots ,\text{vec}({\Lambda }_s{)}^{\mathrm{T}}{)}^{\mathrm{T}}=({\mathbf{K}}^{\mathrm{T}}{\mathbf{A}}^{-1}\mathbf{K}{)}^{-1}{\mathbf{K}}^{\mathrm{T}}{\mathbf{A}}^{-1}\mathbf{d}.$$

Then the Lagrange equation can be written as

$$\left(\begin{array}{cc}\hfill \mathbf{A}\hfill & \hfill \mathbf{K}\hfill \\ \hfill {\mathbf{K}}^{\mathrm{T}}\hfill & \hfill 0\hfill \end{array}\right)\left(\begin{array}{c}\hfill \text{vec}(\mathbf{\Delta })\hfill \\ \hfill \text{vec}(\mathbf{\Lambda })\hfill \end{array}\right)=\left(\begin{array}{c}\hfill \mathbf{d}\hfill \\ \hfill \mathbf{0}\hfill \end{array}\right).$$

It is easy to show that

$$\text{vec}(\mathbf{\Delta })={\mathbf{A}}^{-1}(\mathbf{d}-\mathbf{K}\text{vec}(\mathbf{\Lambda }))\text{and}\text{vec}(\Lambda )=({\mathbf{K}}^{\mathrm{T}}{\mathbf{A}}^{-1}\mathbf{K}{)}^{-1}{\mathbf{K}}^{\mathrm{T}}{\mathbf{A}}^{-1}\mathbf{d}.$$

From the above calculation, we can see that vec(Λ) = (vec(Λ₁)^T, …, vec(Λ_s)^T)^T is actually the Lagrange multiplier for the optimization problem (9).