Multivariate Regression with Calibration

Abstract

We propose a new method named calibrated multivariate regression (CMR) for fitting high dimensional multivariate regression models. Compared to existing methods, CMR calibrates the regularization for each regression task with respect to its noise level so that it is simultaneously tuning insensitive and achieves an improved finite-sample performance. Computationally, we develop an efficient smoothed proximal gradient algorithm which has a worst-case iteration complexity O(1/ε), where ε is a pre-specified numerical accuracy. Theoretically, we prove that CMR achieves the optimal rate of convergence in parameter estimation. We illustrate the usefulness of CMR by thorough numerical simulations and show that CMR consistently outperforms other high dimensional multivariate regression methods. We also apply CMR on a brain activity prediction problem and find that CMR is as competitive as the handcrafted model created by human experts.

1 Introduction

Given a design matrix X ∈ ℝ^n×d and a response matrix Y ∈ ℝ^n×m, we consider a multivariate linear model Y = XB⁰ + Z, where B⁰ ∈ ℝ^d×m is an unknown regression coefficient matrix and Z ∈ ℝ^n×m is a noise matrix [1]. For a matrix A = [A_jk] ∈ ℝ^d×m, we denote A_j* = (A_j1, …, A_jm) ∈ ℝ^m and A_*k = (A_1k, …, A_dk)^T ∈ ℝ^d to be its j^th row and k^th column respectively. We assume that all Z_i*’s are independently sampled from an m-dimensional Gaussian distribution with mean 0 and covariance matrix Σ ∈ ℝ^m×m.

We can represent the multivariate linear model as an ensemble of univariate linear regression models: ${\mathbf{Y}}_{\ast k}={\mathbf{XB}}_{\ast k}^0+{\mathbf{Z}}_{\ast k}$, k = 1, …, m. Then we get a multi-task learning problem [3, 2, 26]. Multi-task learning exploits shared common structure across tasks to obtain improved estimation performance. In the past decade, significant progress has been made towards designing a variety of modeling assumptions for multivariate regression.

A popular assumption is that all the regression tasks share a common sparsity pattern, i.e., many ${\mathbf{B}}_{j\ast }^0$’s are zero vectors. Such a joint sparsity assumption is a natural extension of that for univariate linear regressions. Similar to the L₁-regularization used in Lasso [23], we can adopt group regularization to obtain a good estimator of B⁰ [25, 24, 19, 13]. Besides the aforementioned approaches, there are other methods that aim to exploit the covariance structure of the noise matrix Z [7, 22]. For instance, [22] assume that all Z_i*’s follow a multivariate Gaussian distribution with a sparse inverse covariance matrix Ω = Σ⁻¹. They propose an iterative algorithm to estimate sparse B⁰ and Ω by maximizing the penalized Gaussian log-likelihood. Such an iterative procedure is effective in many applications, but the theoretical analysis is difficult due to its nonconvex formulation.

In this paper, we assume an uncorrelated structure for the noise matrix Z, i.e., $\mathbf{\sum }=\text{diag}({\sigma }_1^2,{\sigma }_2^2,\dots ,{\sigma }_{m-1}^2,{\sigma }_m^2)$. Under this setting, we can efficiently solve the resulting estimation problem with a convex program as follows

$$\widehat{\mathbf{B}}=\underset{\mathbf{B}}{\text{argmin}}\frac{1}{\sqrt{n}}{\Vert \mathbf{Y}-\mathbf{XB}\Vert }_{\mathrm{F}}^2+\lambda {\Vert \mathbf{B}\Vert }_{1,p},$$ (1.1)

where λ > 0 is a tuning parameter, and ${\Vert \mathbf{A}\Vert }_{\mathrm{F}}=\sqrt{{\sum }_{j,k}{\mathbf{A}}_{jk}^2}$ is the Frobenius norm of a matrix A. Popular choices of p include p = 2 and $p=\infty :{\Vert \mathbf{B}\Vert }_{1,2}={\sum }_{j=1}^d\sqrt{{\sum }_{k=1}^m{\mathbf{B}}_{jk}^2}$ and ${\Vert \mathbf{B}\Vert }_{1,\infty }={\sum }_{j=1}^d{max}_{1\le k\le m}\mid {\mathbf{B}}_{jk}\mid$. Computationally, the optimization problem in (1.1) can be efficiently solved by some first order algorithms [11, 12, 4].

The problem with the uncorrelated noise structure is amenable to statistical analysis. Under suitable conditions on the noise and design matrices, let σ_max = max_k σ_k, if we choose $\lambda =2c·{\sigma }_{max}\left(\sqrt{logd}+m^{1-1/p}\right)$, for some c > 1, then the estimator B̂ in (1.1) achieves the optimal rates of convergence¹ [13], i.e., there exists some universal constant C such that with high probability, we have

$$\frac{1}{\sqrt{m}}{\Vert \widehat{\mathbf{B}}-{\mathbf{B}}^0\Vert }_{\mathrm{F}}\le C·{\sigma }_{max}\left(\sqrt{\frac{\mathit{s}logd}{nm}}+\sqrt{\frac{{sm}^{1-2/p}}{n}}\right),$$

where s is the number of rows with non-zero entries in B⁰. However, the estimator in (1.1) has two drawbacks: (1) All the tasks are regularized by the same tuning parameter λ, even though different tasks may have different σ_k’s. Thus more estimation bias is introduced to the tasks with smaller σ_k’s to compensate the tasks with larger σ_k’s. In another word, these tasks are not calibrated. (2) The tuning parameter selection involves the unknown parameter σ_max. This requires tuning the regularization parameter over a wide range of potential values to get a good finite-sample performance.

To overcome the above two drawbacks, we formulate a new convex program named calibrated multivariate regression (CMR). The CMR estimator is defined to be the solution of the following convex program:

$$\widehat{\mathbf{B}}=\underset{\mathbf{B}}{\text{argmin}}{\Vert \mathbf{Y}-\mathbf{XB}\Vert }_{2,1}+\lambda {\Vert \mathbf{B}\Vert }_{1,p},$$ (1.2)

where ${\Vert \mathbf{A}\Vert }_{2,1}={\sum }_k\sqrt{{\sum }_j{\mathbf{A}}_{jk}^2}$ is the nonsmooth L_2,1 norm of a matrix A = [A_jk] ∈ ℝ^d×m. This is a multivariate extension of the square-root Lasso [5]. Similar to the square-root Lasso, the tuning parameter selection of CMR does not involve σ_max. Moreover, the L_2,1 loss function can be viewed as a special example of the weighted least square loss, which calibrates each regression task (See more details in §2). Thus CMR adapts to different σ_k’s and achieves better finite-sample performance than the ordinary multivariate regression estimator (OMR) defined in (1.1).

Since both the loss and penalty functions in (1.2) are nonsmooth, CMR is computationally more challenging than OMR. To efficiently solve CMR, we propose a smoothed proximal gradient (SPG) algorithm with an iteration complexity O(1/ε), where ε is the pre-specified accuracy of the objective value [18, 4]. Theoretically, we provide sufficient conditions under which CMR achieves the optimal rates of convergence in parameter estimation. Numerical experiments on both synthetic and real data show that CMR universally outperforms existing multivariate regression methods. For a brain activity prediction task, prediction based on the features selected by CMR significantly outperforms that based on the features selected by OMR, and is even competitive with that based on the handcrafted features selected by human experts.

Notations

Given a vector v = (v₁, …, v_d)^T ∈ ℝ^d, for 1 ≤ p ≤ ∞, we define the L_p-vector norm of v as ${\Vert \mathit{v}\Vert }_p={\left({\sum }_{j=1}^d{\mid v_j\mid }^p\right)}^{1/p}$ if 1 ≤ p < ∞ and ||v||_p = max_1≤j≤d |v_j| if p = ∞. Given two matrices A = [A_jk] and C = [C_jk] ∈ ℝ^d×m, we define the inner product of A and C as $\langle \mathbf{A},\mathbf{C}\rangle ={\sum }_{j=1}^d{\sum }_{k=1}^m{\mathbf{A}}_{jk}{\mathbf{C}}_{jk}=\text{tr}({\mathbf{A}}^T\mathbf{C})$, where tr(A) is the trace of a matrix A. We use A_*k = (A_1k, …, A_dk)^T and A_j* = (A_j1, …, A_jm) to denote the k^th column and j^th row of A. Let be some subspace of ℝ^d×m, we use to denote the projection of A onto : ${\mathbf{A}}_{\mathcal{S}}={\text{argmin}}_{\mathbf{C}\in \mathcal{S}}{\Vert \mathbf{C}-\mathbf{A}\Vert }_{\mathrm{F}}^2$. Moreover, we define the Frobenius and spectral norms of A as ${\Vert \mathbf{A}\Vert }_{\mathrm{F}}=\sqrt{\langle \mathbf{A},\mathbf{A}\rangle }$ and ||A||₂ = ψ₁(A), ψ₁(A) is the largest singular value of A. In addition, we define the matrix block norms as ${\Vert \mathbf{A}\Vert }_{2,1}={\sum }_{k=1}^m{\Vert {\mathbf{A}}_{\ast k}\Vert }_2$, ||A||_2,∞ = max_1≤k≤m ||A_*k||₂, ${\Vert \mathbf{A}\Vert }_{1,p}={\sum }_{j=1}^d{\Vert {\mathbf{A}}_{j\ast }\Vert }_p$, and ||A||_∞,q = max_1≤j≤d ||A_j*||_q, where 1 ≤ p ≤ ∞ and 1 ≤ q ≤ ∞. It is easy to verify that ||A||_2,1 is the dual norm of ||A||_2,∞. Let 1/∞ = 0, then if 1/p + 1/q = 1, ||A||_∞,q and ||A||_1,p are also dual norms of each other.

2 Method

We solve the multivariate regression problem by the following convex program,

$$\widehat{\mathbf{B}}=\underset{\mathbf{B}}{\text{argmin}}{\Vert \mathbf{Y}-\mathbf{XB}\Vert }_{2,1}+\lambda {\Vert \mathbf{B}\Vert }_{1,p}.$$ (2.1)

The only difference between (2.1) and (1.1) is that we replace the L₂-loss function by the nonsmooth L_2,1-loss function. The L_2,1-loss function can be viewed as a special example of the weighted square loss function. More specifically, we consider the following optimization problem,

$$\widehat{\mathbf{B}}=\underset{\mathbf{B}}{\text{argmin}}\sum _{k=1}^m\frac{1}{{\sigma }_k\sqrt{n}}{\Vert {\mathbf{Y}}_{\ast k}-{\mathbf{XB}}_{\ast k}\Vert }_2^2+\lambda {\Vert \mathbf{B}\Vert }_{1,p},$$ (2.2)

where $\frac{1}{{\sigma }_k\sqrt{n}}$ is a weight assigned to calibrate the k^th regression task. Without prior knowledge on σ_k’s, we use the following replacement of σ_k’s,

$${\stackrel{\sim }{\sigma }}_k=\frac{1}{\sqrt{n}}{\Vert {\mathbf{Y}}_{\ast k}-{\mathbf{XB}}_{\ast k}\Vert }_2,k=1,\dots ,m.$$ (2.3)

By plugging (2.3) into the objective function in (2.2), we get (2.1). In another word, CMR calibrates different tasks by solving a penalized weighted least square program with weights defined in (2.3).

The optimization problem in (2.1) can be solved by the alternating direction method of multipliers (ADMM) with a global convergence guarantee [20]. However, ADMM does not take full advantage of the problem structure in (2.1). For example, even though the L_2,1 norm is nonsmooth, it is nondifferentiable only when a task achieves exact zero residual, which is unlikely in applications. In this paper, we apply the dual smoothing technique proposed by [18] to obtain a smooth surrogate function so that we can avoid directly evaluating the subgradient of the L_2,1 loss function. Thus we gain computational efficiency like other smooth loss functions.

We consider the Fenchel’s dual representation of the L_2,1 loss:

$${\Vert \mathbf{Y}-\mathbf{XB}\Vert }_{2,1}=\underset{{\Vert \mathbf{U}\Vert }_{2,\infty }\le 1}{max}\langle \mathbf{U},\mathbf{Y}-\mathbf{XB}\rangle .$$ (2.4)

Let μ > 0 be a smoothing parameter. The smooth approximation of the L_2,1 loss can be obtained by solving the following optimization problem

$${\Vert \mathbf{Y}-\mathbf{XB}\Vert }_{\mu }=\underset{{\Vert \mathbf{U}\Vert }_{2,\infty }\le 1}{max}\langle \mathbf{U},\mathbf{Y}-\mathbf{XB}\rangle -\frac{\mu }{2}{\Vert \mathbf{U}\Vert }_{\mathrm{F}}^2,$$ (2.5)

where ${\Vert \mathbf{U}\Vert }_{\mathrm{F}}^2$ is the proximity function. Due to the fact that ${\Vert \mathbf{U}\Vert }_{\mathrm{F}}^2\le m{\Vert \mathbf{U}\Vert }_{2,\infty }^2$, we obtain the following uniform bound by combing (2.4) and (2.5),

$${\Vert \mathbf{Y}-\mathbf{XB}\Vert }_{2,1}-\frac{m\mu }{2}\le {\Vert \mathbf{Y}-\mathbf{XB}\Vert }_{\mu }\le {\Vert \mathbf{Y}-\mathbf{XB}\Vert }_{2,1}.$$ (2.6)

From (2.6), we see that the approximation error introduced by the smoothing procedure can be controlled by a suitable μ. Figure 2.1 shows several two-dimensional examples of the L₂ norm smoothed by different μ’s. The optimization problem in (2.5) has a closed form solution Û^B with ${\widehat{\mathbf{U}}}_{\ast k}^{\mathbf{B}}=({\mathbf{Y}}_{\ast k}-{\mathbf{XB}}_{\ast k})/max\{{\Vert {\mathbf{Y}}_{\ast k}-{\mathbf{XB}}_{\ast k}\Vert }_2,\mu \}$.

Figure 2.1.

The L₂ norm (μ = 0) and its smooth surrogates with μ = 0.1, 0.25, 0.5. A larger μ makes the approximation more smooth, but introduces a larger approximation error.

The next lemma shows that ||Y − XB||_μ is smooth in B with a simple form of gradient.

Lemma 2.1

For any μ > 0, ||Y − XB||_μ is a convex and continuously differentiable function in B. In addition, G^μ(B)—the gradient of ||Y − XB||_μ w.r.t. B—has the form

$${\mathbf{G}}^{\mu }(\mathbf{B})=\frac{\partial \left(\langle {\widehat{\mathbf{U}}}^{\mathbf{B}},\mathbf{Y}-\mathbf{XB}\rangle +\mu {\Vert {\widehat{\mathbf{U}}}^{\mathbf{B}}\Vert }_{\mathrm{F}}^2/2\right)}{\partial \mathbf{B}}=-{\mathbf{X}}^T{\widehat{\mathbf{U}}}^{\mathbf{B}}.$$ (2.7)

Moreover, let $\gamma ={\Vert \mathbf{X}\Vert }_2^2$, then we have that G^μ(B) is Lipschitz continuous in B with the Lipschitz constant γ/μ, i.e., for any B′, B″ ∈ ℝ^d×m,

$${\Vert {\mathbf{G}}^{\mu }({\mathbf{B}}^{\prime })-{\mathbf{G}}^{\mu }({\mathbf{B}}^{″})\Vert }_{\mathrm{F}}={\Vert \langle \mathbf{X},{\widehat{\mathbf{U}}}^{{\mathbf{B}}^{\prime }}-{\widehat{\mathbf{U}}}^{{\mathbf{B}}^{″}}\rangle \Vert }_{\mathrm{F}}\le \frac{1}{\mu }{\Vert {\mathbf{X}}^T\mathbf{X}({\mathbf{B}}^{\prime }-{\mathbf{B}}^{″})\Vert }_{\mathrm{F}}\le \frac{\gamma }{\mu }{\Vert {\mathbf{B}}^{\prime }-{\mathbf{B}}^{″}\Vert }_{\mathrm{F}}.$$

Lemma 2.1 is a direct result of Theorem 1 in [18] and implies that ||Y − XB||_μ has good computational structure. Therefore we apply the smooth proximal gradient algorithm to solve the smoothed version of the optimization problem as follows,

$$\stackrel{\sim }{\mathbf{B}}=\underset{\mathbf{B}}{\text{argmin}}{\Vert \mathbf{Y}-\mathbf{XB}\Vert }_{\mu }+\lambda {\Vert \mathbf{B}\Vert }_{1,p}.$$ (2.8)

We then adopt the fast proximal gradient algorithm to solve (2.8) [4]. To derive the algorithm, we first define three sequences of auxiliary variables {A^(t)}, {V^(t)}, and {H^(t)} with A⁽⁰⁾ = H⁽⁰⁾ = V⁽⁰⁾ = B⁽⁰⁾, a sequence of weights {θ_t = 2/(t + 1)}, and a nonincreasing sequence of step-sizes {η_t > 0}. For simplicity, we can set η_t = μ/γ. In practice, we use the backtracking line search to dynamically adjust η_t to boost the performance. At the t^th iteration, we first take V^(t) = (1 − θ_t)B^(t−1) + θ_tA^(t−1). We then consider a quadratic approximation of ||Y − XH||_μ as

$$Q\left(\mathbf{H},{\mathbf{V}}^{(t)},{\eta }_t\right)={\Vert \mathbf{Y}-\mathbf{X}{\mathbf{V}}^{(t)}\Vert }_{\mu }+\langle {\mathbf{G}}^{\mu }({\mathbf{V}}^{(t)}),\mathbf{H}-{\mathbf{V}}^{(t)}\rangle +\frac{1}{2{\eta }_t}{\Vert \mathbf{H}-{\mathbf{V}}^{(t)}\Vert }_{\mathrm{F}}^2.$$

Consequently, let H̃^(t) = V^(t) − η_tG^μ(V^(t)), we take

$${\mathbf{H}}^{(t)}=\underset{\mathbf{H}}{\text{argmin}}Q\left(\mathbf{H},{\mathbf{V}}^{(t)},{\eta }_t\right)+\lambda {\Vert \mathbf{H}\Vert }_{1,p}=\underset{\mathbf{H}}{\text{argmin}}\frac{1}{2{\eta }_t}{\Vert \mathbf{H}-{\stackrel{\sim }{\mathbf{H}}}^{(t)}\Vert }_{\mathrm{F}}^2+\lambda {\Vert \mathbf{H}\Vert }_{1,p}.$$ (2.9)

When p = 2, (2.9) has a closed form solution ${\mathbf{H}}_{j\ast }^{(t)}={\stackrel{\sim }{\mathbf{H}}}_{j\ast }·max\left\{1-{\eta }_t\lambda /{\Vert {\stackrel{\sim }{\mathbf{H}}}_{j\ast }\Vert }_2,0\right\}$. More details about other choices of p in the L_1,p norm can be found in [11] and [12]. To ensure that the objective value is nonincreasing, we choose

$${\mathbf{B}}^{(t)}=\underset{\mathbf{B}\in \{{\mathbf{H}}^{(t)},{\mathbf{B}}^{(t-1)}\}}{\text{argmin}}{\Vert \mathbf{Y}-\mathbf{XB}\Vert }_{\mu }+\lambda {\Vert \mathbf{B}\Vert }_{1,p}.$$ (2.10)

At last, we take ${\mathbf{A}}^{(t)}={\mathbf{B}}^{(t-1)}+\frac{1}{{\theta }_t}({\mathbf{H}}^{(t)}-{\mathbf{B}}^{(t-1)})$. The algorithm stops when ||H^(t)−V^(t)||_F ≤ ε, where ε is the stopping precision.

The numerical rate of convergence of the proposed algorithm with respect to the original optimization problem (2.1) is presented in the following theorem.

Theorem 2.2

Given a pre-specified accuracy ε and let μ = ε/m, after $t=2\sqrt{m\gamma }{\Vert {\mathbf{B}}^{(0)}-\widehat{\mathbf{B}}\Vert }_{\mathrm{F}}/\epsilon -1=O(1/\epsilon )$ iterations, we have ||Y − XB^(t)||_2,1 + λ||B^(t)||_1,p ≤ ||Y − XB̂||_2,1 + λ||B̂||_1,p + ε.

The proof of Theorem 2.2 is provided in Appendix A.1. This result achieves the minimax optimal rate of convergence over all first order algorithms [18].

3 Statistical Properties

For notational simplicity, we define a re-scaled noise matrix W = [W_ik] ∈ ℝ^n×m with W_ik = Z_ik/σ_k, where $\mathbb{E}{\mathbf{Z}}_{ik}^2={\sigma }_k^2$. Thus W is a random matrix with all entries having mean 0 and variance 1. We define G⁰ to be the gradient of ||Y − XB||_2,1 at B = B⁰. It is easy to see that

$${\mathbf{G}}_{\ast k}^0=\frac{{\mathbf{X}}^T{\mathbf{Z}}_{\ast k}}{{\Vert {\mathbf{Z}}_{\ast k}\Vert }_2}=\frac{{\mathbf{X}}^T{\mathbf{W}}_{\ast k}{\sigma }_k}{{\Vert {\mathbf{W}}_{\ast k}{\sigma }_k\Vert }_2}=\frac{{\mathbf{X}}^T{\mathbf{W}}_{\ast k}}{{\Vert {\mathbf{W}}_{\ast k}\Vert }_2}$$

does not depend on the unknown quantities σ_k for all k = 1, …, m. ${\mathbf{G}}_{\ast k}^0$ works as an important pivotal in our analysis. Moreover, our analysis exploits the decomposability of the L_1,p norm [17]. More specifically, we assume that B⁰ has s rows with all zero entries and define

$$\mathcal{S}=\{\mathbf{C}\in {ℝ}^{d\times m}\mid {\mathbf{C}}_{j\ast }=\mathbf{0}\text{for}\text{all}j\text{such}\text{that}{\mathbf{B}}_{j\ast }^0=\mathbf{0}\},$$ (3.1)

$$\mathcal{N}=\{\mathbf{C}\in {ℝ}^{d\times m}\mid {\mathbf{C}}_{j\ast }=\mathbf{0}\text{for}\text{all}j\text{such}\text{that}{\mathbf{B}}_{j\ast }^0\ne \mathbf{0}\}.$$ (3.2)

Note that we have B⁰ ∈ and the L_1,p norm is decomposable with respect to the pair ( , ), i.e.,

$${\Vert \mathbf{A}\Vert }_{1,p}={\Vert {\mathbf{A}}_{\mathcal{S}}\Vert }_{1,p}+{\Vert {\mathbf{A}}_{\mathcal{N}}\Vert }_{1,p}.$$

The next lemma shows that when λ is suitably chosen, the solution to the optimization problem in (2.1) lies in a restricted set.

Lemma 3.1

Let B⁰ ∈ and B̂ be the optimum to (2.1), and 1/p + 1/q = 1. We denote the estimation error as Δ̂ = B̂ − B⁰. If λ ≥ c||G⁰||_∞,q for some c > 1, we have

$$\widehat{\mathbf{\Delta }}\in {\mathcal{M}}_c:=\left\{\mathbf{\Delta }\in {ℝ}^{d\times m}\mid {\Vert {\mathbf{\Delta }}_{\mathcal{N}}\Vert }_{1,p}\le \frac{c+1}{c-1}{\Vert {\mathbf{\Delta }}_{\mathcal{S}}\Vert }_{1,p}\right\}.$$ (3.3)

The proof of Lemma 3.1 is provided in Appendix B.1. To prove the main result, we also need to assume that the design matrix X satisfies the following condition.

Assumption 3.1

Let B⁰ ∈ , then there exist positive constants κ and c > 1 such that

$$\kappa \le \underset{\mathbf{\Delta }\in {\mathcal{M}}_c\backslash \{\mathbf{0}\}}{min}\frac{{\Vert \mathbf{X}\mathbf{\Delta }\Vert }_{\mathrm{F}}}{\sqrt{n}{\Vert \mathbf{\Delta }\Vert }_{\mathrm{F}}}.$$

Assumption 3.1 is the generalization of the restricted eigenvalue conditions for analyzing univariate sparse linear models [17, 15, 6], Many common examples of random design satisfy this assumption [13, 21].

Note that Lemma 3.1 is a deterministic result of the CMR estimator for a fixed λ. Since G is essentially a random matrix, we need to show that λ ≥ cR^*(G⁰) holds with high probability to deliver a concrete rate of convergence for the CMR estimator in the next theorem.

Theorem 3.2

We assume that each column of X is normalized as $m^{1/2-1/p}{\Vert {\mathbf{X}}_{\ast j}\Vert }_2=\sqrt{n}$ for all j = 1, …, d. Then for some universal constant c₀ and large enough n, taking

$$\lambda =\frac{2c(m^{1-1/p}+\sqrt{logd})}{\sqrt{1-c_0}},$$ (3.4)

with probability at least $1-2exp(-2logd)-2exp(-{nc}_0^2/8+logm)$, we have

$$\frac{1}{\sqrt{m}}{\Vert \widehat{\mathbf{B}}-{\mathbf{B}}^0\Vert }_{\mathrm{F}}\le \frac{16c{\sigma }_{max}}{{\kappa }^2(c-1)}\sqrt{\frac{1+c_0}{1-c_0}}\left(\sqrt{\frac{{sm}^{1-2/p}}{n}}+\sqrt{\frac{\mathit{s}logd}{nm}}\right).$$

The proof of Theorem 3.2 is provided in Appendix B.2. Note that when we choose p = 2, the column normalization condition is reduced to ${\Vert {\mathbf{X}}_{\ast j}\Vert }_2=\sqrt{n}$. Meanwhile, the corresponding error bound is further reduced to

$$\frac{1}{\sqrt{m}}{\Vert \widehat{\mathbf{B}}-{\mathbf{B}}^0\Vert }_{\mathrm{F}}=O_P\left(\sqrt{\frac{s}{n}}+\sqrt{\frac{\mathit{s}logd}{nm}}\right),$$

which achieves the minimax optimal rate of convergence presented in [13]. See Theorem 6.1 in [13] for more technical details. From Theorem 3.2, we see that CMR achieves the same rates of convergence as the noncalibrated counterpart, but the tuning parameter λ in (3.4) does not involve σ_k’s. Therefore CMR not only calibrates all the regression tasks, but also makes the tuning parameter selection insensitive to σ_max.

4 Numerical Simulations

To compare the finite-sample performance between the calibrated multivariate regression (CMR) and ordinary multivariate regression (OMR), we generate a training dataset of 200 samples. More specifically, we use the following data generation scheme: (1) Generate each row of the design matrix X_i*, i = 1, …, 200, independently from a 800-dimensional normal distribution N(0, Σ) where Σ_jj = 1 and Σ_jℓ = 0.5 for all ℓ ≠ j.(2) Let k = 1, …, 13, set the regression coefficient matrix B⁰ ∈ ℝ^800×13 as ${\mathbf{B}}_{1k}^0=3,{\mathbf{B}}_{2k}^0=2,{\mathbf{B}}_{4k}^0=1.5$, and ${\mathbf{B}}_{jk}^0=0$ for all j ≠ 1, 2, 4. (3) Generate the random noise matrix Z = WD, where W ∈ ℝ^200×13 with all entries of W are independently generated from N(0, 1), and D is either of the following matrices

$$\begin{array}{r}{\mathbf{D}}_I={\sigma }_{max}·\text{diag}\left(2^{0/4},2^{-1/4},\cdots ,2^{-11/4},2^{-12/4}\right)\in {ℝ}^{13\times 13}\\ {\mathbf{D}}_H={\sigma }_{max}·\mathbf{I}\in {ℝ}^{13\times 13}.\end{array}$$

We generate a validation set of 200 samples for the regularization parameter selection and a testing set of 10,000 samples to evaluate the prediction accuracy.

In numerical experiments, we set σ_max = 1, 2, and 4 to illustrate the tuning insensitivity of CMR. The regularization parameter λ of both CMR and OMR is chosen over a grid Λ = {2^40/4 λ₀, 2^39/4 λ₀, ···, 2^−17/4 λ₀, 2^−18/4 λ₀}, where ${\lambda }_0=\sqrt{logd}+\sqrt{m}$. The optimal regularization parameter λ̂ is determined by the prediction error as $\widehat{\lambda }={\text{argmin}}_{\lambda \in \mathbf{\Lambda }}{\Vert \stackrel{\sim }{\mathbf{Y}}-\stackrel{\sim }{\mathbf{X}}{\widehat{\mathbf{B}}}^{\lambda }\Vert }_{\mathrm{F}}^2$, where B̂^λ denotes the obtained estimate using the regularization parameter λ, and X̃ and Ỹ denote the design and response matrices of the validation set.

Since the noise level σ_k’s are different in regression tasks, we adopt the following three criteria to evaluate the empirical performance: $\text{Pre}.\text{Err}.=\frac{1}{10000}{\Vert \overline{\mathbf{Y}}-\overline{\mathbf{X}}\widehat{\mathbf{B}}\Vert }_{\mathrm{F}},\text{Adj}.\text{Pre}.\text{Err}.=\frac{1}{10000m}{\Vert (\overline{\mathbf{Y}}-\overline{\mathbf{X}}\widehat{\mathbf{B}}){\mathbf{D}}^{-1}\Vert }_{\mathrm{F}}^2$, and $\text{Est}.\text{Err}.=\frac{1}{m}{\Vert \widehat{\mathbf{B}}-{\mathbf{B}}^0\Vert }_{\mathrm{F}}^2$, where X̄ and Ȳ denotes the design and response matrices of the testing set.

All simulations are implemented by MATLAB using a PC with Intel Core i5 3.3GHz CPU and 16GB memory. CMR is solved by the proposed smoothing proximal gradient algorithm, where we set the stopping precision ε = 10⁻⁴, the smoothing parameter μ = 10⁻⁴. OMR is solved by the monotone fast proximal gradient algorithm, where we set the stopping precision ε = 10⁻⁴. We set p = 2, but the extension to arbitrary p > 2 is straightforward.

We first compare the smoothed proximal gradient (SPG) algorithm with the ADMM algorithm (the detailed derivation of ADMM can be found in Appendix A.2). We adopt the backtracking line search to accelerate both algorithms with a shrinkage parameter α = 0.8. We set σ_max = 2 for the adopted multivariate linear models. We conduct 200 simulations. The results are presented in Table 4.1. The SPG and ADMM algorithms attain similar objective values, but SPG is up to 4 times faster than ADMM. Both algorithms also achieve similar estimation errors.

Table 4.1.

Quantitive comparison of the computational performance between SPG and ADMM with the noise matrices generated using D_I. The results are averaged over 200 replicates with standard errors in parentheses. SPG and ADMM attain similar objective values, but SPG is up to about 4 times faster than ADMM.

λ	Algorithm	Timing (second)	Obj. Val.	Num. Ite.	Est. Err.
2λ0	SPG	2.8789(0.3141)	508.21(3.8498)	493.26(52.268)	0.1213(0.0286)
	ADMM	8.4731(0.8387)	508.22(3.7059)	437.7(37.4532)	0.1215(0.0291)
λ0	SPG	3.2633(0.3200)	370.53(3.6144)	565.80(54.919)	0.0819(0.0205)
	ADMM	11.976(1.460)	370.53(3.4231)	600.94(74.629)	0.0822(0.0233)
0.5λ0	SPG	3.7868(0.4551)	297.24(3.6125)	652.53(78.140)	0.1399(0.0284)
	ADMM	18.360(1.9678)	297.25(3.3863)	1134.0(136.08)	0.1409(0.0317)

We then compare the statistical performance between CMR and OMR. Tables 4.2 and 4.3 summarize the results averaged over 200 replicates. In addition, we also present the results of the oracle estimator, which is obtained by solving (2.2), since we know the true values of σ_k’s. Note that the oracle estimator is only for comparison purpose, and it is not a practical estimator. Since CMR calibrates the regularization for each task with respect to σ_k, CMR universally outperforms OMR, and achieves almost the same performance as the oracle estimator when we adopt the scale matrix D_I to generate the random noise. Meanwhile, when we adopt the scale matrix D_H, where all σ_k’s are the same, CMR and OMR achieve similar performance. This further implies that CMR can be a safe replacement of OMR for multivariate regressions.

Table 4.2.

Quantitive comparison of the statistical performance between CMR and OMR with the noise matrices generated using D_I. The results are averaged over 200 simulations with the standard errors in parentheses. CMR universally outperforms OMR, and achieves almost the same performance as the oracle estimator.

σmax	Method	Pre. Err.	Adj. Pre.Err	Est. Err.
1	Oracle	5.8759(0.0834)	1.0454(0.0149)	0.0245(0.0086)
	CMR	5.8761(0.0673)	1.0459(0.0123)	0.0249(0.0071)
	OMR	5.9012(0.0701)	1.0581(0.0162)	0.0290(0.0091)
2	Oracle	23.464(0.3237)	1.0441(0.0148)	0.0926(0.0342)
	CMR	23.465(0.2598)	1.0446(0.0121)	0.0928(0.0279)
	OMR	23.580(0.2832)	1.0573(0.0170)	0.1115(0.0365)
4	Oracle	93.532(0.8843)	1.0418(0.0962)	0.3342(0.1255)
	CMR	93.542(0.9794)	1.0421(0.0118)	0.3346(0.1063)
	OMR	94.094(1.0978)	1.0550(0.0166)	0.4125(0.1417)

Table 4.3.

Quantitive comparison of the statistical performance between CMR and OMR with the noise matrices generated using D_H. The results are averaged over 200 simulations with the standard errors in parentheses. CMR and OMR achieve similar performance.

σmax	Method	Pre. Err.	Adj. Pre.Err	Est. Err.
1	CMR	13.565(0.1408)	1.0435(0.0108)	0.0599(0.0164)
	OMR	13.697(0.1554)	1.0486(0.0142)	0.0607(0.0128)
2	CMR	54.171(0.5771)	1.0418(0.0110)	0.2252(0.0649)
	OMR	54.221(0.6173)	1.0427(0.0118)	0.2359(0.0821)
4	CMR	215.98(2.104)	1.0384(0.0101)	0.80821(0.25078)
	OMR	216.19(2.391)	1.0394(0.0114)	0.81957(0.31806)

In addition, we also examine the optimal regularization parameters for CMR and OMR over all replicates. We visualize the distribution of all 200 selected λ̂’s using the kernel density estimator. In particular, we adopt the Gaussian kernel, and the kernel bandwidth is selected based on the 10-fold cross validation. Figure 4.1 illustrates the estimated density functions. The horizontal axis corresponds to the rescaled regularization parameter as $log\left(\frac{\widehat{\lambda }}{\sqrt{logd}+\sqrt{m}}\right)$. We see that the optimal regularization parameters of OMR significantly vary with different σ_max. In contrast, the optimal regularization parameters of CMR are more concentrated. This is inconsistent with our claimed tuning insensitivity.

Figure 4.1.

The distributions of the selected regularization parameters using the kernel density estimator. The numbers in the parentheses are σ_max’s. The optimal regularization parameters of OMR are spreader with different σ_max than those of CMR and the oracle estimator.

5 Real Data Experiment

We apply CMR on a brain activity prediction problem which aims to build a parsimonious model to predict a person’s neural activity when seeing a stimulus word. As is illustrated in Figure 5.1, for a given stimulus word, we first encode it into an intermediate semantic feature vector using some corpus statistics. We then model the brain’s neural activity pattern using CMR. Creating such a predictive model not only enables us to explore new analytical tools for the fMRI data, but also helps us to gain deeper understanding on how human brain represents knowledge [16].

Figure 5.1.

An illustration of the fMRI brain activity prediction problem [16]. (a) To collect the data, a human participant sees a sequence of English words and their images. The corresponding fMRI images are recorded to represent the brain activity patterns; (b) To build a predictive model, each stimulus word is encoded into intermediate semantic features (e.g. the co-occurrence statistics of this stimulus word in a large text corpus). These intermediate features can then be used to predict the brain activity pattern.

Our experiments involves 9 participants, and Table 5.1 summarizes the prediction performance of different methods on these participants. We see that the prediction based on the features selected by CMR significantly outperforms that based on the features selected by OMR, and is as competitive as that based on the handcrafted features selected by human experts. But due to the space limit, we present the details of the real data experiment in the technical report version.

Table 5.1.

Prediction accuracies of different methods (higher is better). CMR outperforms OMR for 8 out of 9 participants, and outperforms the handcrafted basis words for 6 out of 9 participants

Method	P.1	P.2	P.3	P.4	P.5	P.6	P.7	P.8	P.9
CMR	0.840	0.794	0.861	0.651	0.823	0.722	0.738	0.720	0.780
OMR	0.803	0.789	0.801	0.602	0.766	0.623	0.726	0.749	0.765
Handcraft	0.822	0.776	0.773	0.727	0.782	0.865	0.734	0.685	0.819

6 Discussions

A related method is the square-root sparse multivariate regression [8]. They solve the convex program with the Frobenius loss function and L_1,p regularization function

$$\widehat{\mathbf{B}}=\underset{\mathbf{B}}{\text{argmin}}{\Vert \mathbf{Y}-\mathbf{XB}\Vert }_{\mathrm{F}}+\lambda {\Vert \mathbf{B}\Vert }_{1,p}.$$ (6.1)

The Frobenius loss function in (6.1) makes the regularization parameter selection independent of σ_max, but it does not calibrate different regression tasks. Note that we can rewrite (6.1) as

$$(\widehat{\mathbf{B}},\widehat{\sigma })=\underset{\mathbf{B},\sigma }{\text{argmin}}\frac{1}{\sqrt{nm}\sigma }{\Vert \mathbf{Y}-\mathbf{XB}\Vert }_{\mathrm{F}}^2+\lambda {\Vert \mathbf{B}\Vert }_{1,p}\mathrm{s}.\mathrm{t}.\sigma =\frac{1}{\sqrt{nm}}{\Vert \mathbf{Y}-\mathbf{XB}\Vert }_{\mathrm{F}}.$$ (6.2)

Since σ in (6.2) is not specific to any individual task, it cannot calibrate the regularization. Thus it is fundamentally different from CMR.