Bayesian sparse reduced rank multivariate regression

Abstract

Many modern statistical problems can be cast in the framework of multivariate regression, where the main task is to make statistical inference for a possibly sparse and low-rank coefficient matrix. The low-rank structure in the coefficient matrix is of intrinsic multivariate nature, which, when combined with sparsity, can further lift dimension reduction, conduct variable selection, and facilitate model interpretation. Using a Bayesian approach, we develop a unified sparse and low-rank multivariate regression method to both estimate the coefficient matrix and obtain its credible region for making inference. The newly developed sparse and low-rank prior for the coefficient matrix enables rank reduction, predictor selection and response selection simultaneously. We utilize the marginal likelihood to determine the regularization hyperparameter, so our method maximizes its posterior probability given the data. For theoretical aspect, the posterior consistency is established to discuss an asymptotic behavior of the proposed method. The efficacy of the proposed approach is demonstrated via simulation studies and a real application on yeast cell cycle data.

1. Introduction

In various fields of scientific research such as genomics, economics, image processing, astronomy, etc., massive amount of data are routinely collected, and many associated statistical problems can be cast in the framework of multivariate regression, where both the number of response variables and the number of predictors are possibly of high dimensionality For example, in genomics study, it is critical to explore the relationship between genetic markers and gene expression profiles in order to understand the gene regulatory network; in a study of human lung disease mechanism, the detailed CT-scanned lung imaging data enable us to examine the systematic variations in airway tree measurements across various lung disease status and pulmonary function test results. To formulate, suppose we have n independent observations of the response vector y_i ∈ ℝ^q and the predictor vector x_i ∈ ℝ^p, i = 1, …, n. Consider the multivariate linear regression model

$$\mathbf{Y}=\mathbf{X}\mathbf{C}+\mathbf{E},$$ (1)

where Y = (y₁, …, y_n)^⊤ ∈ ℝ^n×q is the response matrix, X = (x₁, …, x_n)^⊤ ∈ ℝ^n×p is the predictor matrix, C ∈ ℝ^n×q is the unknown regression coefficient matrix, and E = (e₁, …,e_n)^⊤ ∈ ℝ^n×q is the error matrix with e_i's being independently and identically distributed (i.i.d.) with mean zero and covariance matrix Σ_e, a q × q positive definite matrix. Following Bunea et al. [8, 9], Chen et al. [11] and Mukherjee et al. [28], we assume Σ_e = σ²I_q. We further assume the response variables and the predictors are all centered, and there is no intercept term. In what follows, we use ${\mathbf{a}}_j^{\top }$ to denote the j^th row of a generic matrix A and ã_ℓ the ℓ^th column of A, e.g., C = (c₁, …, c_p)^⊤ = (c̃₁, …, c̃_q). A fundamental goal of multivariate regression is thus to estimate and make inference about the coefficient matrix C so that meaningful dependence structure between the responses and predictors can be revealed.

When the predictor dimension p and the response dimension q are large relative to the sample size n, classical estimation methods such as ordinary least squares (OLS) may fail miserably. The curse of dimensionality can be mitigated by assuming that C admits certain low-dimensional structures, and regularization/penalization approaches are then commonly deployed to conduct dimension reduction and model estimation. The celebrated reduced rank regression (RRR) [2, 24, 32] achieved dimension reduction through constraining the coefficient matrix C to be rank deficient, building upon the belief that the response variables are related to the predictors through only a few latent directions, i.e., some linear combinations of the original predictors. As such, low-rank structure induces and models dependency among responses, which is the essence of conducting multivariate analysis. Bunea et al. [8] generalized the classical RRR to high dimensional settings, casting reduced-rank estimation as a penalized least squares problem with the penalty being proportional to the rank of C. Yuan et al. [37] utilized the nuclear norm penalty, defined as the ℓ₁ norm of the singular values. See also, Chen et al. [11], Mukherjee and Zhu [29], Negahban and Wainwright [30], and Rohde and Tsybakov [33].

It is worth noting that low-rankness in C is of intrinsic multivariate nature; when combined with row and/or column-wise sparsity, it can further lift dimension reduction and facilitate model interpretation. For example, in the aforementioned genomics study, it is plausible that the gene expression profiles (responses) and the genetic markers (predictors) are associated through only a few latent pathways (linear combinations of possibly highly-correlated genetic markers), and moreover, very likely such linear associations only involve a small subset of genetic markers and/or gene profiles. Therefore, recovering a low-rank and also sparse coefficient matrix C in model (1) hold the key to reveal such interesting connections between the responses and predictors. Chen et al. [10] proposed a regularized sparse singular value decomposition (SVD) approach with known rank, in which each latent variable is constructed from only a subset of the predictors and is associated with only a subset of the responses. Chen and Huang [12] proposed a rank-constrained adaptive group Lasso approach to recover a low-rank coefficient matrix C with sparse rows; for each zero row in C, the corresponding predictor is then completely eliminated from the model. Bunea et al. [9] also proposed a joint sparse and low-rank estimation approach and derived its nonasymptotic oracle error bounds. Both methods required to solve the nonconvex rank-constrained problem by fitting models of various ranks. Recently, Ma et al. [27] proposed a subspace assisted regression with row sparsity method which was shown to achieve near optimal nonasymptotic minimax rates in estimation.

While all the aforementioned regularized regression techniques produce attractive point estimators of the coefficient matrix C, it remains a difficult problem to assess the uncertainty of the obtained estimators. To overcome this limitation, there has already been a rich literature on Bayesian approaches of the reduced rank regression. From a Bayesian perspective, the unknown parameter is considered as a random variable, and thus the statistical inference can be made by the posterior distribution. The first attempt to develop the Bayesian reduced rank regression was made by Geweke [20]. The coefficient matrix is assumed to be C = AB^⊤ with A ∈ ℝ^p×r and B ∈ ℝ^q×r, where r < min(p, q) is assumed to be known. Then, by assigning Gaussian prior for each elements of A and B, the induced posterior achieves the low-rank structure of the prespecified rank. As an alternative, Lim and Teh [26] proposed to start from the largest possible rank r = min(p, q), assign a column-wise shrinkage Gaussian prior on each columns of A and B. The posterior for redundant columns of A and B is forced to be concentrated around zero, so the (approximate) rank reduction can be accomplished. The main challenge of this Bayesian approach is the choice of the hyperparameters of the Gaussian priors in order to control the amount of shrinkage. There have been several attempts to overcome this challenge by assigning priors on the hyperparameters, so that they can be determined in the estimation procedure. For instance, Salakhutdinov and Mnih [34] proposed to utilize the Wishart distribution as the hyperprior. Similar hierarchical Bayesian methods were also proposed in the context of matrix completion, matrix completion deals with missing values, but we do not [4, 40]. However, none of the aforementioned studies dealt with the sparsity of the coefficient matrix C. Recently, Zhu et al. [41] introduced a Bayesian low-rank regression model with high-dimensional responses and covariates. To enable sparse estimation under low rank constraint with a prefixed rank, they utilized a sparse singular value decomposition (SVD) structure [10] with Gaussian-mixtures of gamma priors on all the elements of the decomposed matrices. Then, the sparsity of C was achieved using Bayesian thresholding method. For a survey on Bayesian reduce rank models, see Alquier [1] and the references therein.

We develop in this article a novel Bayesian simultaneous dimension reduction and variable selection approach. Our method aims to tackle several challenges regarding both the estimation and inference in the sparse and low-rank regression problems. First, the proposed method enables us to simultaneously estimate the unknown rank and remove irrelevant predictors, in contrast to several existing methods in which rank selection has to be resolved by comparing fitted models of various ranks or by some ad hoc approach such as scree plot. In addition, we also seek potential column sparsity of the coefficient matrix, so that it is applicable to problems with high-dimensional responses where response selection is highly desirable (to be elaborated below). Second, by careful construction of the prior distribution, our method alleviates the many difficulties brought by the use of nonsmooth and non-convex penalty functions and by the tuning parameter selection procedure in penalized regression analysis. From a Bayesian perspective, the penalty function can be viewed as a negative logarithm of the prior density function [25, 31, 36]. We develop a general prior for C mimicking the rank penalty and the group ℓ₀ row/column penalty, to achieve simultaneous rank reduction and variable selection through the induced posterior distribution, yet the computation is kept tractable and efficient, where the group ℓ₀ penalty directly restricts the number of nonzero rows and columns. Since the tuning parameters are considered as random variables in our Bayesian formulation, the optimal ones are selected to achieve the highest posterior probability given the data. Furthermore, using our Bayesian approach, the credibility intervals for the regression coefficients and their functions can be easily constructed using the Markov Chain Monte Carlo (MCMC) technique. In contrast, there has been little work on quantifying the estimation uncertainty in regularized regression approaches.

We now formally state our assumptions or prior beliefs about the coefficient matrix C in model (1).

• A1. (Reduced rank) r* ≤ r, where r* = rank(C) indicates the rank of C and r = min(p, q).

• A2. (Row-wise sparsity) p* ≤ p, where $p^{\ast }=\text{card}(\{j:{\mathbf{c}}_j^{\top }{\mathbf{c}}_j\ne 0\})$ and ${\mathbf{c}}_j^{\top }$ denotes the j^th row of C, where card(·) denotes the cardinality of a set.

• A3. (Column-wise sparsity) q* ≤ q, where $q^{\ast }=\text{card}(\{\ell :{\stackrel{\sim }{c}}_l^{\top }{\stackrel{\sim }{c}}_{\ell }\ne 0\})$ and c̃_ℓ denotes the ℓ^th column of C.

A1 states that C is possibly of low rank. In A2, excluding the j^th predictor from model (1) is equivalent to setting all entries of the j^th row of C as zero. Therefore, the first two assumptions concern rank reduction and predictor selection. The third assumption is about “response selection”, i.e., if the ℓ^th column of C is zero, the ℓ^th response is modeled as a noise variable. While such structural assumption can be treated as optional depending on the specific application, we stress that there are many circumstances where response selection is highly desirable [10]. For example, in many applications the dimension of the responses can be very high, and there may exist noise variables that are not related to any predictors in the model. In addition, eliminating irrelevant predictors dramatically reduces the number of free parameters of the model and thus it improves the accuracy of parameter estimation [5]. In addition, allowing possible response selection provides more flexibility and generality in the multivariate linear regression framework, since the case of selecting all responses can be viewed as a special case.

The remainder of the paper is organized as follows. In Section 2, we briefly introduce a general penalized regression approach for conducting sparse and low-rank estimation. In Section 3, we develop our new Bayesian approach, and explore the connections between the penalized least squares and our Bayes estimators. The full conditionals are obtained in Section 4, and we describe the posterior optimization algorithm and posterior sampling technique. In Section 5, we study the posterior consistency of the proposed method. Simulation studies and a real application on yeast cycle data are presented in Section 6 and Section 7. Some concluding remarks are given in Section 8.

2. Penalized regression approach

In the regularized estimation framework, the unknown coefficient matrix C in model (1) can be estimated by the following penalized least squares (PLS) method,

$${\widehat{\mathbf{C}}}_{\text{pls}}=arg\underset{\mathbf{C}}{min}\{\frac{1}{2}{\Vert \mathbf{Y}-\mathbf{X}\mathbf{C}\Vert }_F^2+{\mathcal{P}}_{\lambda }(\mathbf{C})\},$$ (2)

where ${\Vert \mathbf{C}\Vert }_F=\sqrt{\text{trace}({\mathbf{C}}^{\top }\mathbf{C})}=\sqrt{\text{trace}({\mathbf{C}\mathbf{C}}^{\top })}$ denotes the Frobenius norm and 𝒫_λ(C) is a penalty function with non-negative tuning parameter λ controlling the amount of regularization. It is natural to construct a penalty function of an additive form,

$${\mathcal{P}}_{\lambda }(\mathbf{C})={\mathcal{P}}_{{\lambda }_1}^{\text{RR}}(\mathbf{C})+{\mathcal{P}}_{{\lambda }_2}^{\text{RS}}(\mathbf{C})+{\mathcal{P}}_{{\lambda }_3}^{\text{cs}}(\mathbf{C}),$$

where ${\mathcal{P}}_{{\lambda }_1}^{\text{RR}}(\mathbf{C})$, ${\mathcal{P}}_{{\lambda }_2}^{\text{RS}}(\mathbf{C})$ and ${\mathcal{P}}_{{\lambda }_3}^{\text{CS}}(\mathbf{C})$ induce the low-rankness, row-wise sparsity and column-wise sparsity in C, with tuning parameters λ₁, λ₂ and λ₃, respectively. There are numerous choices of the penalty functions. Note that the rank of matrix C is same as the number of non-zero singular values, i.e., rank(C) = card ({k: s_k(C) > 0}) = r*, where s_k(C) denotes the k^th singular value of C. Hence, rank reduction can be achieved by penalizing the singular values of C, i.e.,

$${\mathcal{P}}_{{\lambda }_1}^{\text{RR}}(\mathbf{C})={\lambda }_1\sum _{k=1}^r{\rho }_1\{s_k(\mathbf{C})\},$$ (3)

where ρ₁ is a sparsity-inducing penalty function. In particular, choosing ρ₁(|a|) = 1 (|a| ≠ 0) corresponds to directly penalizing/restraining the rank of C, and that ρ₁(|a|) = |a|^β₁ gives the Schatten-β quasi-norm penalty when 0 < β₁ < 1 and the convex nuclear norm penalty λ₁‖C‖_* when β₁ = 1, where ${\Vert \mathbf{C}\Vert }_{\ast }={\sum }_{k=1}^r{s}_k(\mathbf{C})$ denotes the nuclear norm. For promoting rowwise/column-wise sparsity, selecting or eliminating parameters by groups is needed, which can be achieved by penalizing the row/column ℓ₂ norms of C,

$${\mathcal{P}}_{{\lambda }_2}^{\text{RS}}(\mathbf{C})=\frac{1}{2}{\lambda }_2\sum _{j=1}^p{\rho }_2({\Vert {\mathbf{c}}_j\Vert }_2),$$ (4)

$${\mathcal{P}}_{{\lambda }_3}^{\text{CS}}(\mathbf{C})=\frac{1}{2}{\lambda }_3\sum _{\ell =1}^q{\rho }_3{\Vert {\stackrel{\sim }{\mathbf{c}}}_{\ell }\Vert }_2),$$ (5)

where ${\Vert \mathbf{c}\Vert }_2=\sqrt{{\mathbf{c}}^{\top }\mathbf{c}}$ denotes the ℓ₂ norm. Choosing ρ₂(|a|) = 1 (|a| ≠ 0) corresponds to directly counting and penalizing the number of nonzero rows, and ρ₂(|a|) = |a| corresponds to the convex group Lasso penalty [38]. Other methods include group SCAD [16] and group MCP [7, 39]; see Fan and Lv [18] and Huang et al. [22] for comprehensive reviews. In principal, rank reduction and variable selection can be accomplished by solving the PLS problem (2) with any sparsity-inducing penalties ρ₁, ρ₂ and ρ₃.

The pros and cons of using convex penalties in model selection are well understood. In low-rank estimation, Bunea et al. [8] showed that while the convex nuclear norm penalized estimator has similar estimation properties to those of the nonconvex rank penalized estimator, the former requires stronger conditions and is in general not as parsimonious as the latter in rank selection. For sparse group selection, it is known that the convex group Lasso criterion often leads to over-selection and substantial estimation bias, and adopting nonconvex penalties may lead to superior properties in both model estimation and variable selection under milder conditions [23, 27]. Unfortunately, the nonconvexity of a penalized regression criterion also imposes great challenges in both understanding its theoretical properties and solving the optimization problem in computation. Therefore, trading off computation efficiency and statistical properties is critical in formulating penalized estimation criterion, and it is particularly relevant when dealing with large data applications. The problem of tuning parameter selection can also be troublesome, especially so for the problem of interest here as it requires multiple tuning parameters. Furthermore, it is still a largely unsolved problem on how to make statistical inference and attach error measures to any penalized estimator. All these concerns motivate us to tackle the sparse and low-rank estimation problem in a Bayesian fashion, to achieve a computationally efficient implementation and be able to make valid inference about the composite low-dimensional structure of C.

3. Bayesian sparse and low-rank regression

From a Bayesian perspective, the PLS estimator in (2) can be viewed as the maximum a posteriori (MAP) estimator from the following posterior density function,

$$\pi (\mathbf{C}|\mathbf{Y},\lambda ,)\propto f(\mathbf{Y}|\mathbf{C})\pi (\mathbf{C}|\lambda )\propto exp(-\frac{1}{2{\sigma }^2}{\Vert \mathbf{Y}-\mathbf{X}\mathbf{C}\Vert }_F^2)exp\{-\frac{1}{{\sigma }^2}{\mathcal{P}}_{\lambda }(\mathbf{C})\},$$

where f(Y | C) denotes the likelihood function and π(C | λ) denotes the prior density function of C given the tuning parameter λ. Since the MAP estimator of C is free of σ², without loss of generality, throughout this paper we assume that σ² = 1. Motivated by the connections between PLS and MAP and by the penalty function defined in (3)-(5), it is natural to consider the following prior,

$$\begin{array}{c}\pi (\mathbf{C}|\lambda )\propto exp\{-\frac{1}{2}{\lambda }_1\sum _{k=1}^r\mathbf{1}(|s_k(\mathbf{C})|>\epsilon )\}\\ \times exp\{-\frac{1}{2}{\lambda }_2\sum _{j=1}^p\mathbf{1}({\Vert {\mathbf{c}}_j\Vert }_2>\epsilon )\}\\ \times exp\{-\frac{1}{2}{\lambda }_3\sum _{\ell =1}^q\mathbf{1}({\Vert {\stackrel{\sim }{c}}_{\ell }\Vert }_2>\epsilon )\},\end{array}$$ (6)

where ε > 0 is a suitably small value and λ = (λ₁, λ₂, λ₃). Note that as ε → 0, this prior converges to the following penalty function under the MAP estimation:

$${\mathcal{P}}_{\mathit{\lambda }}(\mathbf{c})=\frac{1}{2}{\lambda }_1\sum _{k=1}^r{\ell }_0(|s_k(\mathbf{C})|)+\frac{1}{2}{\lambda }_2\sum _{j=1}^{\mathcal{p}}{\ell }_0({\Vert {\mathbf{c}}_j\Vert }_2)+\frac{1}{2}{\lambda }_3\sum _{\ell =1}^q{\ell }_0({\Vert {\stackrel{\sim }{\mathbf{c}}}_{\ell }\Vert }_2).$$

In penalized regression, this £₀ penalty corresponds to directly penalize the rank, the number of nonzero rows, and the number of nonzero columns of C, which leads to an intractable combinatory problem. Similarly, in Bayesian framework, even though this setup directly targets on the desired structure of C, there are several difficulties in using such a prior distribution. Since the prior density function in (6) involves the singular values of C, it induces an improper posterior distribution, as it is generally difficult to be considered as a probability density function of C. Moreover, the non-differentiability of indicator function induces a discontinuous posterior density function.

To overcome the first difficulty, i.e., avoiding direct use of the singular values, we propose an indirect modeling method through decomposing the matrix C. We write C = AB^⊤, where A is a p × r matrix, Bisaq×r matrix, and r is an upper bound of the true rank r* of C, e.g., a trivial one is r = min(p, q). Apparently such a decomposition is not unique, as with any nonsingular r×r matrix Q, C = AB^⊤ = AQQ⁻¹B^⊤ = ÃB̃^⊤ where à = AQ and B̃ = B (Q⁻¹)^⊤. Interestingly, the following lemma reveals that the low-rankness and the row/column sparsity of C can all be represented as certain row/column sparsity of A and B, and more importantly, the representations are invariant to any nonsingular transformation.

Lemma 1. Let C ∈ ℝ^p×q, and suppose C = AB^⊤ for some A ∈ ℝ^p×r, B ∈ ℝ^q×r with r = min(p, q). Let ã_k and b̃_k denote the k^th column of A and B, respectively. Let ${\mathbf{a}}_j^{\top }$ and ${\mathbf{b}}_{\ell }^{\top }$ denote the j^th row of A and the ℓ^th row of B, respectively. Then

1. $$\text{rank}(\mathbf{C})\le min(p,q)-{\sum }_{k=1}^r\mathbf{1}({\Vert {\stackrel{\sim }{\mathbf{a}}}_k\Vert }_2+{\Vert {\stackrel{\sim }{\mathbf{b}}}_k\Vert }_2=0).$$

2. {j: ‖c_j‖₂≠0} ⊂ {j: ‖a_j‖₂≠0}.

3. {ℓ: ‖c̃_ℓ‖₂ ≠ 0} ⊂ {ℓ: ‖b_ℓ‖₂ ≠ 0}.

The first statement in the above lemma suggests the following rank-reducing prior,

$${\pi }^{\text{RR}}(\mathbf{A},\mathbf{B}|{\lambda }_1)\propto exp\{-\frac{1}{2}{\lambda }_1\sum _{k=1}^r\mathbf{1}({\Vert {\stackrel{\sim }{\mathbf{a}}}_k\Vert }_2+{\Vert {\stackrel{\sim }{\mathbf{b}}}_k\Vert }_2>\epsilon )\},$$ (7)

where ε > 0 and λ₁ > 0. For a sufficiently small ε, this prior induces sparsity on columns of A and B simultaneously and thus reduces the rank of C. Similarly, Lemma 1 suggests the following row-wise and column-wise sparsity-inducing priors,

$${\pi }^{\text{RS}}(\mathbf{A}|{\lambda }_2)\propto exp\{-\frac{1}{2}{\lambda }_2\sum _{j=1}^p\mathbf{1}({\Vert {\mathbf{a}}_j\Vert }_2>\epsilon )\},$$ (8)

$${\pi }^{\text{cs}}(\mathbf{B}|{\lambda }_3)\propto exp\{-\frac{1}{2}{\lambda }_3\sum _{\ell =1}^q\mathbf{1}({\Vert {\mathbf{b}}_{\ell }\Vert }_2>\epsilon )\},$$ (9)

where ε > 0, λ₂ > 0 and λ₃ > 0. Combining (7), (8) and (9) leads us to the following prior,

$$\pi (\mathbf{A},\mathbf{B}|\mathbf{\lambda })\propto {\pi }^{\text{RR}}(\mathbf{A},\mathbf{B}|{\lambda }_1){\pi }^{\text{RS}}(\mathbf{A}|{\lambda }_2){\pi }^{\text{CS}}(\mathbf{B}|{\lambda }_3),$$ (10)

where λ = (λ₁, λ₂, λ₃).

The discontinuity problem still presents in (10) due to the indicator functions. We address this problem by approximating ℓ₀-norm by a well-behaved smooth function. Let D be a m × n matrix. Define

$${\mathcal{P}}_{\lambda ,\omega }(\mathbf{D})=\frac{1}{2}\lambda \sum _{i=1}^m\frac{{\mathbf{d}}_i^{\top }{\mathbf{d}}_i}{{(\omega +{\mathbf{d}}_i^{\top }{\mathbf{d}}_i)}^{1-\frac{\beta }{2}}},$$ (11)

where D = (d₁, …, d_m)^⊤, 0 ≤ β ≤ 1 and ω > 0. We have

$$\frac{{\mathbf{d}}_i^{\top }{\mathbf{d}}_i}{{(\omega +{\mathbf{d}}_i^{\top }{\mathbf{d}}_i)}^{1-\frac{\beta }{2}}}\to {({\Vert {\mathbf{d}}_i\Vert }_2)}^{\beta },$$

as ω → 0, where we define (0)⁰ = 0. This implies that the proposed penalty approximates the group ℓ_β-norm penalty when ω is sufficiently small. In particular, when β = 0 and ω is chosen to be a small positive constant ω₀, (11) gives an approximate group ℓ₀-norm penalty and produces approximately sparse solutions, while it is continuous as well as differentiable with respect to D. In all our numerical studies, we set ω₀ = 10⁻¹⁰ and utilize tolerance level 10⁻⁵ to determine zero estimates. Fig. 1 shows the plots of f(x) = x²/(x² + ω₀) for varying ω₀ = 10^−k, k = 4, 6, 8,10. Indeed, when w₀ = 10⁻¹⁰, the function closely mimics the ℓ₀ penalty.

Figure 1.

Plot of f(x) = x²/(x² + ω₀) for ω₀ = 10^−k (k = 4, 6, 8, 10).

We now propose the following prior distribution for our Bayesian Sparse and Reduced-rank Regression (BSRR) method,

$$\begin{array}{c}{\pi }^{\text{BSRR}}(\mathbf{A},\mathbf{B}|\mathbf{\lambda })\propto exp(-\frac{1}{2}{\lambda }_1\sum _{k=1}^r\frac{{\stackrel{\sim }{\mathbf{a}}}_k^{\top }{\stackrel{\sim }{\mathbf{a}}}_k+{\stackrel{\sim }{\mathbf{b}}}_k^{\top }{\stackrel{\sim }{\mathbf{b}}}_k}{{\omega }_0+{\stackrel{\sim }{\mathbf{a}}}_k^{\top }{\stackrel{\sim }{\mathbf{a}}}_k+{\stackrel{\sim }{\mathbf{b}}}_k^{\top }{\stackrel{\sim }{\mathbf{b}}}_k})\\ \times exp(-\frac{1}{2}{\lambda }_2\sum _{j=1}^p\frac{{\mathbf{a}}_j^{\top }{\mathbf{a}}_j}{{\omega }_0+{\mathbf{a}}_j^{\top }{\mathbf{a}}_j})\\ \times exp(-\frac{1}{2}{\lambda }_3\sum _{\ell =1}^q\frac{{\mathbf{b}}_{\ell }^{\top }{\mathbf{b}}_{\ell }}{{\omega }_0+{\mathbf{b}}_{\ell }^{\top }{\mathbf{b}}_{\ell }}),\end{array}$$ (12)

where λ = (λ₁, λ₂, λ₃) and λ_i, ≥ 0 (i = 1, 2, 3). The BSRR posterior is then given as

$${\pi }^{\text{BSRR}}(\mathbf{A},\mathbf{B}|\mathbf{Y},\mathit{\lambda })\propto f(\mathbf{Y}|\mathbf{C}=\mathbf{A}{\mathbf{B}}^{\top }){\pi }^{\text{BSRR}}(\mathbf{A},\mathbf{B}|\mathit{\lambda }),$$ (13)

and the MAP estimate (Â_map, B̂_map) is defined as

$$({\widehat{\mathbf{A}}}_{\text{map}},{\widehat{\mathbf{B}}}_{\text{map}})=arg\underset{\mathbf{A},\mathbf{B}}{\text{max}}\{\pi (\mathbf{A},\mathbf{B}|\mathbf{Y},\mathit{\lambda })\},$$

Since C = AB^⊤, the MAP estimator for C, named BSRR estimator, is given by Ĉ_BSRR = Â_map(B̂_map)^⊤.

The following lemma tells us the BSRR model can be expressed as a hierarchical Bayesian model by introducing auxiliary variables.

Lemma 2. Define d = (d_1,1, …, d_1,r, d_2,1,…, d_2,p, d_3,1,…, d_3,q). Let π (A, B, d | λ) be a density function of (A, B, d) such that

$$\begin{array}{c}\pi (\mathbf{A},\mathbf{B},\mathbf{d}|\mathit{\lambda })\propto \pi (\mathbf{A},\mathbf{B}|\mathbf{d})\pi (\mathbf{d}|\mathit{\lambda })\\ \propto exp[-\frac{1}{2}\{\sum _{k=1}^r{d}_{1,k}({\stackrel{\sim }{\mathbf{a}}}_k^{\top }{\stackrel{\sim }{\mathbf{a}}}_k+{\stackrel{\sim }{\mathbf{b}}}_k^{\top }{\stackrel{\sim }{\mathbf{b}}}_k)\left\}\right]\\ \times exp[-\frac{1}{2}\{\sum _{j=1}^p{d}_{2,j}({\mathbf{a}}_j^{\top }{\mathbf{a}}_j)+\sum _{\ell =1}^q{d}_{3,\ell }({\mathbf{b}}_{\ell }^{\top }{\mathbf{b}}_{\ell })\left\}\right]\\ \times \prod _{i=1}^3\left[\prod _{j=1}^{m_i}\right\{{(d_{i,j})}^{\frac{{\lambda }_i}{2}}exp(-\frac{{\omega }_0}{2}{d}_{i,j})\left\}\right],\end{array}$$ (14)

where m₁ = r, m₂ = p and m₃ = q. Let π^BSRR(A, B | λ) denote the BSRR prior defined in (12). Then, for any positive λ and ω₀, we have that

$${\pi }^{\text{BSRR}}(\mathbf{A},\mathbf{B}|\mathit{\lambda })\propto \underset{\mathrm{d}}{max}\{\pi (\mathbf{A},\mathbf{B},\mathbf{d}|\mathit{\lambda })\}.$$

The proof of Lemma 2 can be shown by differentiating Eq. (14) with respect to d_i,j's, letting them to be zero, and finding the solutions of the equations. Based on Lemma 2, we introduce the following hierarchical Bayesian representation of the sparse reduced-rank regression model (HBSRR),

$$f(\mathbf{Y}|\mathbf{A},\mathbf{B})\propto exp(-\frac{1}{2}{\Vert \mathbf{Y}-\mathbf{X}\mathbf{A}{\mathbf{B}}^{\top }\Vert }_F^2),$$ (15)

$$\pi (\mathbf{A},\mathbf{B}|\mathbf{d})\propto exp\{-\frac{1}{2}({\Vert \mathbf{A}{\mathbf{D}}_1^{1/2}\Vert }_F^2+{\Vert {\mathbf{D}}_2^{1/2}\mathbf{A}\Vert }_F^2)\}\times exp\{-\frac{1}{2}({\Vert \mathbf{B}{\mathbf{D}}_1^{1/2}\Vert }_F^2+{\Vert {\mathbf{D}}_3^{1/2}\mathbf{B}\Vert }_F^2)\},$$ (16)

$$\begin{array}{c}{\mathbf{D}}_i^{1/2}=\text{diag}(\sqrt{d_{i,1}},\dots ,\sqrt{d_{i,m_i}}),i=1,2,3,\\ \pi (\mathbf{d}|\mathit{\lambda })\alpha \prod _{i=1}^3[\prod _{j=1}^{m_i}\{{(d_{i,j})}^{\frac{{\lambda }_i}{2}}exp(-\frac{{\omega }_0}{2}{d}_{i,j}\left)\right\}],\end{array}$$ (17)

where m₁ = r, m₂ = p, m₃ = q, d = {diag(D₁), diag(D₂), diag(D₃)}, λ = (λ₁, λ₂, λ₃), and d_i,mi > 0 (i = 1, 2, 3). Let (Â_mode, B̂_mode, d̂_mode) be the mode of the induced posterior π(A, B, d | Y, λ) from the above hierarchical model. Recall that Ĉ_BSRRindicates the BSRR estimator. Then, using Lemma 2, it is straightforward to show that Ĉ_BSRR = Â_mode (B̂_mode)^⊤, almost surely. This enables us to easily find Ĉ_BSRR using the HBSRR. Since all full conditional distributions of the HBSRR are well-known distributions such as Gaussian and gamma, the estimation procedure can be conducted by standard Bayesian estimation algorithms.

4. Bayesian analysis

Since our posterior distribution is complex, the Bayesian inference procedure requires the implementation of iterated conditional modes (ICM) algorithm [6] or Markov chain Monte Carlo (MCMC) sampling techniques, to obtain Bayes estimators such as posterior mode, posterior mean, or credible set. We derive full conditional distributions from the joint posterior of HBSRR. Then, we describe the implementation of ICM and MCMC, and discuss the determination of the tuning parameter from a Bayesian perspective.

4.1. Full conditionals

To derive the full conditionals of the HBSRR, we write

$$\begin{array}{c}{\Vert \mathbf{Y}-\mathbf{X}\mathbf{A}{\mathbf{B}}^{\top }\Vert }_F^2=\text{trace}\{(\mathbf{Y}-{\mathbf{X}}_{(\stackrel{\sim }{j})}{\mathbf{A}}_{(j)}{\mathbf{B}}^{\top }\left){(\mathbf{Y}-{\mathbf{X}}_{(\overline{j})}{\mathbf{A}}_{(j)}{\mathbf{B}}^{\top })}^{\top }\right\}\\ +\text{trace}({\stackrel{\sim }{\mathbf{x}}}_j{\mathbf{a}}_j^{\top }{\mathbf{B}}^{\top }\mathbf{B}{\mathbf{a}}_j{\stackrel{\sim }{\mathbf{x}}}_j^{\top })-2\text{trace}\left\{{\stackrel{\sim }{\mathbf{x}}}_j{\mathbf{a}}_j^{\top }{\mathbf{B}}^{\top }{(\mathbf{Y}-{\mathbf{X}}_{(\stackrel{\sim }{j})}{\mathbf{A}}_{(j)}{\mathbf{B}}^{\top })}^{\top }\right\}\\ =\text{trace}\left\{\right(\mathbf{Y}-{\mathbf{X}}_{(\stackrel{\sim }{j})}{\mathbf{A}}_{(j)}{\mathbf{B}}^{\top }\left){(\mathbf{Y}-{\mathbf{X}}_{(\stackrel{\sim }{j})}{\mathbf{A}}_{(j)}{\mathbf{B}}^{\top })}^{\top }\right\}\\ +{\mathbf{a}}_j^{\top }{\mathbf{B}}^{\top }\mathbf{B}{\mathbf{a}}_j{\stackrel{\sim }{\mathbf{x}}}_j^{\top }{\stackrel{\sim }{\mathbf{x}}}_j-2{\mathbf{a}}_j^{\top }{\mathbf{B}}^{\top }{(\mathbf{Y}-{\mathbf{X}}_{(\stackrel{\sim }{j})}{\mathbf{A}}_{(j)}{\mathbf{B}}^{\top })}^{\top }{\stackrel{\sim }{\mathbf{x}}}_j.\end{array}$$ (18)

Here we use the notation C_(j) to denote the submatrix of a generic matrix C by deleting its j^th row, and C_(j̃) by deleting its j^th column. Using (15), (16) and (18), the full conditional distribution of a_j (j = 1,…, p) is determined to be

$${\mathbf{a}}_j|\text{Others}\stackrel{\mathit{\text{ind}}}{~}{\mathcal{N}}_r({\mathit{\mu }}_j^{\mathbf{A}},{\sum }_j^{\mathbf{A}}),$$ (19)

where

$$\begin{array}{c}{\mathit{\mu }}_j^{\mathbf{A}}={\sum }_j^A{\mathbf{B}}^{\top }{(\mathbf{Y}-{\mathbf{X}}_{(\stackrel{\sim }{j})}{\mathbf{A}}_{(j)}{\mathbf{B}}^{\top })}^{\top }{\stackrel{\sim }{\mathbf{x}}}_j,\\ {\sum }_j^A={({\mathbf{B}}^{\top }\mathbf{B}{\stackrel{\sim }{\mathbf{x}}}_j^{\top }{\stackrel{\sim }{\mathbf{x}}}_j+{\mathbf{D}}_1+d_{2,j}{\mathbf{I}}_r)}^{-1},\end{array}$$

with I_r denoting the r × r identity matrix. Similar to (18), we have

$${\Vert \mathbf{Y}-\mathbf{X}\mathbf{A}{\mathbf{B}}^{\top }\Vert }_F^2=\text{trace}\left[{\{{\mathbf{Y}}_{(\stackrel{\sim }{\ell })}-\mathbf{X}\mathbf{A}{({\mathbf{B}}^{\top })}_{(\stackrel{\sim }{\ell })}\}}^{\top }\right\{{\mathbf{Y}}_{(\stackrel{\sim }{\ell })}-\mathbf{X}\mathbf{A}{({\mathbf{B}}^{\top })}_{(\stackrel{\sim }{\ell })}\left\}\right]+{\mathbf{b}}_{\ell }^{\top }{(\mathbf{X}\mathbf{A})}^{\top }\mathbf{X}\mathbf{A}{\mathbf{b}}_{\ell }-2{\mathbf{b}}_{\ell }^{\top }{(\mathbf{X}\mathbf{A})}^{\top }{\stackrel{\sim }{\mathbf{y}}}_{\ell }+{\stackrel{\sim }{\mathbf{y}}}_{\ell }^{\top }{\stackrel{\sim }{\mathbf{y}}}_{\ell }.$$

The full conditional distribution of b_ℓ, for ℓ = 1,…, q, is given by

$${\mathbf{b}}_{\ell }|\text{Others}\stackrel{\mathit{\text{ind}}}{~}{\mathcal{N}}_r({\mathit{\mu }}_{\ell }^{\mathbf{B}},{\sum }_{\ell }^{\mathbf{B}}),$$ (20)

where

$$\begin{array}{c}{\mathit{\mu }}_{\ell }^B={\sum }_{\ell }^{\mathbf{b}}{(\mathbf{X}\mathbf{A})}^{\top }{\stackrel{\sim }{\mathbf{y}}}_{\ell },\\ {\sum }_{\ell }^B={\{{(\mathbf{X}\mathbf{A})}^{\top }\mathbf{X}\mathbf{A}+{\mathbf{D}}_1+d_{3,\ell }{\mathbf{I}}_r\}}^{-1}.\end{array}$$

From (16) and (17), it is straightforward to show that the full conditionals for elements of D are written as

$$\begin{array}{c}{d}_{1,k}|\text{Others}\stackrel{\mathit{\text{ind}}}{~}\mathcal{G}(\frac{{\lambda }_1}{2},\frac{{\omega }_0+{\stackrel{\sim }{\mathbf{a}}}_k^T{\stackrel{\sim }{\mathbf{a}}}_k+{\stackrel{\sim }{\mathbf{b}}}_k^T{\stackrel{\sim }{\mathbf{b}}}_k}{2}),\\ d_{2,j}|\text{Others}\stackrel{\mathit{\text{ind}}}{~}\mathcal{G}(\frac{{\lambda }_2}{2},\frac{{\omega }_0+{\mathbf{a}}_j^T{\mathbf{a}}_j}{2}),\\ d_{3,\ell }|\text{Others}\stackrel{\mathit{\text{ind}}}{~}\mathcal{G}(\frac{{\lambda }_3}{2},\frac{{\omega }_0+{\mathbf{b}}_{\ell }^T{b}_{\ell }}{2}),\end{array}$$

where k = 1,…, r, j = 1,…, p, and ℓ = 1, …, q.

4.2. Iterated conditional modes

All full conditionals in Section 4.1 are well-known distributions (normal or gamma distribution), and the modes are thus well-known. Consequently, using the full conditionals, we construct the following ICM algorithm to find the BSRR estimate Ĉ_BSRR:

Algorithm 1 ICM algorithm for ĈBSRR

Step 1) Set initial values (Â, B̂, d̂) = (A(0), B(0), d(0)).

Step 2) Update (Â, B̂, d̂) = (A(t+1), B(t+1), d(t+1)) by

$\begin{array}{c}{\mathbf{a}}_j^{(t+1)}\leftarrow {\{{({\mathbf{B}}^{(t)})}^{\top }{\mathbf{B}}^{(t)}{\stackrel{\sim }{\mathbf{x}}}_j^{\top }{\stackrel{\sim }{\mathbf{x}}}_j+{\mathbf{D}}_1^{(t)}+d_{2,j}^{(t)}{\mathbf{I}}_r\}}^{-1}\\ \times {({\mathbf{B}}^{(t)})}^{\top }{\{\mathbf{Y}-{\mathbf{X}}_{(\stackrel{\sim }{j})}{\mathbf{A}}_{(j)}^{(t)}{({\mathbf{B}}^{(t)})}^{\top }\}}^{\top }{\stackrel{\sim }{\mathbf{x}}}_j,\end{array}$

${\mathbf{b}}_{\ell }^{(t+1)}\leftarrow {\{{(\mathbf{X}{\mathbf{A}}^{(t+1)})}^{\top }\mathbf{X}{\mathbf{A}}^{(t+1)}+{\mathbf{D}}_1^{(t)}+d_{3,\ell }^{(t)}{\mathbf{I}}_r\}}^{-1}{(\mathbf{X}{\mathbf{A}}^{(t+1)})}^{\top }{\stackrel{\sim }{\mathbf{y}}}_{\ell },$

$d_{1,k}^{(t+1)}\leftarrow {\lambda }_1{\{{\omega }_0+{({\stackrel{\sim }{\mathbf{a}}}_k^{(t+1)})}^{\top }{\stackrel{\sim }{\mathbf{a}}}_k^{(t+1)}+{({\stackrel{\sim }{\mathbf{b}}}_k^{(t+1)})}^{\top }{\stackrel{\sim }{\mathbf{b}}}_k^{(t+1)}\}}^{-1},$

$d_{2,j}^{(t+1)}\leftarrow {\lambda }_2{\{{\omega }_0+{({\mathbf{a}}_j^{(t+1)})}^{\top }{\mathbf{a}}_j^{(t+1)}\}}^{-1},$

$d_{3,\ell }^{(t+1)}\leftarrow {\lambda }_3{\{{\omega }_0+{({\mathbf{b}}_{\ell }^{(t+1)})}^{\top }{\mathbf{b}}_{\ell }^{(t+1)}\}}^{-1}$

for k = 1, …, r; j = 1, …, p; ℓ = 1, …, q.

Step 3) Repeat step 2 until convergence.

Step 4) Return ĈBSRR = ÂB̂⊤.

To set an initial value (A⁽⁰⁾, B⁽⁰⁾, d⁽⁰⁾), we propose to utilize the OLS estimate Ĉ_ols = (X^⊤X)⁻X^⊤Y. Let Ĉ_ols = USV^⊤ be the sigular value decomposition of Ĉ_ols. Then, (A⁽⁰⁾, B⁽⁰⁾, d⁽⁰⁾) can be defined as

$$\begin{array}{c}{\mathbf{A}}^{(0)}=\mathbf{U}{\mathbf{S}}^{1/2},{\mathbf{B}}^{(0)}=\mathbf{V}{\mathbf{S}}^{1/2},\\ d_{1,k}^{(0)}={\lambda }_1{\{{\omega }_0+{({\stackrel{\sim }{\mathbf{a}}}_k^{(0)})}^{\top }{\stackrel{\sim }{\mathbf{a}}}_k^{(0)}+{({\stackrel{\sim }{\mathbf{b}}}_k^{(0)})}^{\top }{\stackrel{\sim }{\mathbf{b}}}_k^{(0)}\}}^{-1},\\ d_{2,j}^{(0)}={\lambda }_2{\{{\omega }_0+{({\mathbf{a}}_j^{(0)})}^{\top }{\mathbf{a}}_j^{(0)}\}}^{-1}\\ d_{3,\ell }^{(0)}={\lambda }_3{\{{\omega }_0+{({\mathbf{b}}_{\ell }^{(0)})}^{\top }{\mathbf{b}}_{\ell }^{(0)}\}}^{-1},\end{array}$$

for k = 1, …, r, j = 1, …, p and ℓ = 1, …, q.

4.3. Posterior sampling

Using the HBSRR, we introduce indirect sampling method to obtain the posterior samples for C, so that they can be used to construct the credible set for the BSRR estimate Ĉ_BSRR. Recall that (Â_mode, B̂_mode, d̂_mode) denotes the posterior mode of the HBSRR. Then,

$$\begin{array}{c}{\widehat{\mathbf{C}}}_{\text{BSRR}}=arg\underset{\mathbf{C}=\mathbf{A}{\mathbf{B}}^{\top }}{max}\{\underset{\mathbf{d}}{max}\pi (\mathbf{A},\mathbf{B},\mathbf{d}|\mathbf{Y},\mathit{\lambda })\}\\ =arg\underset{\mathbf{C}=\mathbf{A}{\mathbf{B}}^{\top }}{max}\{f(\mathbf{Y}|\mathbf{A},\mathbf{B})\pi (\mathbf{A},\mathbf{B}|{\widehat{\mathbf{d}}}_{\text{mode}})\}.\end{array}$$

Consequently, we can obtain the posterior sample of C from the following posterior

$$\pi (\mathbf{A},\mathbf{B}|\mathbf{Y},{\widehat{\mathbf{d}}}_{\text{mode}})\propto f(\mathbf{Y}|\mathbf{A},\mathbf{B})\pi (\mathbf{A},\mathbf{B}|{\widehat{\mathbf{d}}}_{\text{mode}}).$$

Note that d̂_mode can be obtained by the proposed ICM algorithm in the previous section. First, to generate MCMC samples from the above posterior distribution of (A, B), we consider a Gibbs sampler that iterates through the following steps:

1. update a_j for j = 1, …, p;

2. update b_ℓ for ℓ= 1, …, q.

The explicit forms of full conditionals of a_j and b_ℓ, respectively, are given in (19) and (20). In each Gibbs step, we update a_j and b_ℓ by generating samples from

$$\begin{array}{c}{\mathbf{a}}_j|\mathbf{Y},{\mathbf{A}}_{(j)},\mathbf{B},{\widehat{\mathbf{d}}}_{\text{mode}}\sim {\mathcal{N}}_r({\mathit{\mu }}_{\mathbf{A},j},{\sum }_{\mathbf{A},j})\\ {\mathit{\mu }}_{\mathbf{A},j}={\sum }_{\mathbf{A},j}{\mathbf{B}}^{\top }{(\mathbf{Y}-{\mathbf{X}}_{(\stackrel{\sim }{j})}{\mathbf{A}}_{(j)}{\mathbf{B}}^{\top })}^{\top }{\stackrel{\sim }{\mathbf{x}}}_j,\\ {\sum }_{\mathbf{A},j}={({\mathbf{B}}^{\top }\mathbf{B}{\stackrel{\sim }{\mathbf{x}}}_j^{\top }{\stackrel{\sim }{\mathbf{x}}}_j+{\widehat{\mathbf{D}}}_1+{\widehat{d}}_{2,j}{\mathbf{I}}_r)}^{-1}\text{for}j=1,\dots ,p,\\ {\mathbf{b}}_{\ell }|\mathbf{Y},\mathbf{A},{\widehat{\mathbf{d}}}_{\text{mode}}\sim {\mathcal{N}}_r({\mathit{\mu }}_{\mathbf{B},\ell },{\sum }_{\mathbf{B},\ell }),\\ {\mathit{\mu }}_{\mathbf{B},\ell }={\sum }_{\mathbf{B},\ell }{(\mathbf{X}\mathbf{A})}^{\top }{\stackrel{\sim }{\mathbf{y}}}_{\ell },\\ {\sum }_{\mathbf{B},\ell }={\{{(\mathit{X}\mathit{A})}^{\top }\mathit{X}\mathit{A}+{\widehat{\mathbf{D}}}_1+{\widehat{d}}_{3,j}{\mathbf{I}}_r\}}^{-1}\text{for}\ell =1,\dots ,q,\end{array}$$

where d̂_mode = {diag(D̂₁), diag(D̂₂), diag(D̂₃)}. Let ${\{{\mathbf{A}}^i,{\mathbf{B}}^i\}}_{i=1}^N$ be a set of obtained MCMC samples from the above sampling procedure. Then a set of posterior samples for C can be obtained by $\mathcal{S}={\{{\mathbf{C}}^i:{\mathbf{C}}^i={\mathbf{A}}^i{({\mathbf{B}}^i)}^{\top }\}}_{i=1}^N$.

4.4. Tuning parameter selection

In practice, we are usually interested in selecting a tuning parameter λ = (λ₁, λ₂, λ₃) (or hyperparameter) from the set of candidates ℒ = {λ₁, …, λ_K}. We assume that there is no preferred model, i.e., π(λ_k) = 1/K for k = 1, …, K. In general, the tuning parameter is determined via a grid search strategy from a lower bound λ_L = (0⁺, 0⁺, 0⁺) to a given upper bound λ_U which is the smallest value to induce the marginal null model (i.e., all estimates are zero). Hence, the set ℒ is well-defined. Let m(Y | λ_k) = ∫ f(Y | A, B)Π(dA, dB | λ_k) be a marginal likelihood for a given λ_k. Then, we can show that m(Y | λ_k) is proportional to the posterior probability of λ_k given Y, that is

$$\prod ({\mathit{\lambda }}_k|\mathbf{Y})=\frac{m(\mathbf{Y}|{\mathit{\lambda }}_k)\pi ({\mathit{\lambda }}_k)}{{\sum }_{k=1}^{K}m(\mathbf{Y}|{\mathit{\lambda }}_k)\pi ({\mathit{\lambda }}_k)}\propto m(\mathbf{Y}|{\mathit{\lambda }}_k).$$ (21)

From the above viewpoint, we define the optimal λ_k* such that

$${\mathit{\lambda }}_{k^{\ast }}=arg\underset{{\mathit{\lambda }}_k\epsilon \mathcal{L}}{max}\{m(\mathbf{Y}|{\mathit{\lambda }}_k)\}.$$

Let C be the p × q coefficient matrix with rank(C) = r*. Without loss of generality, suppose that the first p* rows and q* columns of C are non-zero and the remaining rows and columns of C are zero. Then, the matrix C can be decomposed as

$$\mathbf{C}=\left(\frac{{\mathbf{I}}_{r^{\ast }}}{\frac{{\mathbf{C}}_A}{{\mathrm{O}}_A}}\right)({\mathbf{C}}_B|{\mathbf{O}}_B),$$

where I_r*. denotes the identity matrix of order r*, C_A is a (p* −r*) × r* nonzero matrix, C_B is a r* × q* nonzero matrix, and O_A is the (p−p*) × r* zero matrix, and O_B is the r* × (q − q*) zero matrix. The key to above parameterization of C is that the matrix C_A and C_B are uniquely determined. It can be seen that for given (r*, p*, q*), the number of free parameters in C is dim(C_A) + dim(C_B) = r*(p* + q*−r*). Suppose that a given tuning parameter λ = λ_k results in an estimator with (r_k, p_k, q_k). Define θ = {vec(C_A)^⊤, vec(C_B)^⊤}^⊤ such that θ ∈ Θ ⊂ ℝ^r_k(p_k+q_k−^r_k), where vec(⋅) denotes the vectorization of a matrix. Then, the marginal likelihood can be rewritten as

$$\begin{array}{c}m(\mathbf{Y}|{\mathit{\lambda }}_k)={\int }_{\mathrm{\Theta }}f(\mathbf{Y}|\mathit{\theta })\pi (\mathit{\theta }|{\mathit{\lambda }}_k)d\mathit{\theta }\\ ={\int }_{\mathrm{\Theta }}exp\{(n{q}_k)g_n(\mathit{\theta }|\mathbf{Y},{\mathit{\lambda }}_k)\}d\mathit{\theta },\end{array}$$ (22)

where g_n(θ | Y, λ) = (nq_k)⁻¹ {ln f(Y | θ) + ln π (θ | λ_k)}. Let θ̂ be the mode of g_n(θ | Y, λ). By the Laplace approximation, the marginal likelihood in (22) can be expressed as

$$m(\mathbf{Y}|\mathit{\lambda })=\frac{{(2\pi /n{q}_k)}^{\frac{r_k(p_k+q_k-r_k)}{2}}}{{|-G_n(\widehat{\mathit{\theta }})|}^{1/2}}exp\{(n{q}_k)g_n(\widehat{\mathit{\theta }}|\mathbf{Y},{\mathit{\lambda }}_k)\}\{1+O_p(n^{-1})\},$$ (23)

where

$$G_n(\widehat{\mathit{\theta }})={\frac{{\partial }^2{g}_n(\theta |\mathbf{Y},\mathit{\lambda })}{\partial \mathit{\theta }\partial {\mathit{\theta }}^{\top }}|}_{\mathit{\theta }=\widehat{\mathit{\theta }}}.$$

By taking the logarithm of the formula (23) and ignoring the term of O(1) and higher order terms, we have the following approximation of log marginal likelihood

$$ln\{m(\mathbf{Y}|{\mathit{\lambda }}_k)\}\approx lnf(\mathbf{Y}|\widehat{\mathit{\theta }})-\frac{r_k(p_k+q_k-r_k)}{2}ln(n{q}_k).$$ (24)

Let Ĉ_k be the BRRR estimate for given λ_k. Then, by substituting it in (24) and multiplying by −2, (24) reduces to the BRRR version of Bayesian information criterion [35],

$$\text{BIC}({\mathit{\lambda }}_k)=-2lnf(\mathbf{Y}|{\widehat{\mathbf{C}}}_k)+r_k(p_k+q_k-r_k)ln(n{q}_k).$$ (25)

According (21), we know that minimizing the BIC corresponds to maximizing the posterior probability of λ_k given Y. Hence, we regard the tuning parameter λ_* as the optimum if λ_* = argmin_λk∈ℒ BIC(λ_k).

5. Posterior consistency

In Bayesian analysis, the posterior consistency assures that the posterior converges to point mass at the true parameter as more data are collected [13, 15, 21]. Here, we discuss the posterior consistency for the proposed BSRR method, following Armagan et al. [3]. We allow the number of predictors p to grow with sample size n, and the number of true non-zero coefficients p* is assumed to be finite. Henceforth, we denote p as p_n. Similarly the response matrix Y and predictor matrix X are denoted by Y_n and X_n, respectively. Unlike p_n, the number of response variables q is assumed to be fixed in our analysis.

Suppose that, given X_n and C*, Y_n is generated from

$${\mathbf{Y}}_n={\mathbf{X}}_n{\mathbf{C}}^{\ast }+E,$$

where ${\mathbf{e}}_i\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{~}{\mathcal{N}}_q(\mathbf{0},\sum )$ with a positive definite matrix Σ (assumed to be known) and C* is a (p_n × q) matrix such that card $(\{j:{{\mathbf{c}}^{\ast }}_j^{\top }{{\mathbf{c}}^{\ast }}_j\ne 0\})=p^{\ast }$, card $(\{\ell :{{\stackrel{\sim }{\mathbf{c}}}_{\ell }^{\ast }}^{\top }{\stackrel{\sim }{\mathbf{c}}}_{\ell }^{\ast }\ne 0\})=q^{\ast }$ and rank(C*) = r*. Further, we make the following assumptions.

1. p_n = o(n), but p* < ∞ and q* ≤ q < ∞.

2. 0 < S_min < lim inf_n→∞ S_{n, min}/√n ≤ lim sup_n→∞ S_{n, max}/√n < S_max < ∞, where S_{n, min} and S_{n, max} denote the smallest and the largest singular values of X, respectively.

3. ${\text{Sup}}_{(j,\ell )}(c_{j\ell }^{\ast })<\infty$, where $c_{j\ell }^{\ast }$ indicates the (j, ℓ)^th element of C*.

Our main results are presented in Theorems 3 and 4 below.

Theorem 3. Under assumptions I and II, if the prior Π(A, B) satisfies the following condition:

$$\prod \{(\mathbf{A},\mathbf{B}):{\Vert \mathbf{A}{\mathbf{B}}^{\top }-{\mathbf{C}}^{\ast }\Vert }_F<\frac{\mathrm{\Delta }}{n^{\frac{\rho }{2}}}\}>exp(-dn),$$

for all $0<\mathrm{\Delta }<{\epsilon }^2{S}_{min}^2/(48S_{max}^2)$ and $0<d<\epsilon S_{min}^2/(32{\tau }_{max})-3\mathrm{\Delta }{S}_{max}/(2{\tau }_{min})$ and some ρ > 0, where τ_min and τ_max denote, respectively, the smallest and the largest eigenvalue of Σ, then the posterior of (A, B) induced by the prior Π(A, B) is strongly consistent, i.e., for any ε > 0,

$$\prod \{(\mathbf{A},\mathbf{B}):{\Vert \mathbf{C}-{\mathbf{C}}^{\ast }\Vert }_F>\epsilon ,\mathbf{C}=\mathbf{A}{\mathbf{B}}^{\top }|{\mathbf{Y}}_n\}\to 0\mathit{\text{almost surely}},$$

as n → ∞.

Theorem 4. Under assumptions I, II and III, the prior defined in (12) yields a strongly consistent posterior if ${\lambda }_i={\delta }_i{n}^{\rho /2}\sqrt{p_n}$ ln n for finite δ_i > 0, i = 1, 2, 3.

In Theorem 3, we establish a sufficient condition on a prior distribution in order to achieve posterior consistency. Theorem 4 then shows that our BSRR prior in (12) satisfies the sufficient condition in Theorem 3, and consequently, our BSRR method possesses the desirable posterior consistency property. The proofs of both theorems are shown in the supplementary materials.

6. Simulation studies

To examine the performance of our BSRR method, we conduct Monte Carlo experiments under several possible scenarios. For purposes of comparison, we also consider the following two reduced priors:

$${\pi }^{\text{RR}}(\mathbf{A},\mathbf{B}|{\lambda }_1,{\lambda }_2):={\pi }^{\text{BSRR}}(\mathbf{A},\mathbf{B}|\mathit{\lambda },{\lambda }_3=0),$$ (26)

$${\pi }^{\text{RC}}(\mathbf{A},\mathbf{B}|{\lambda }_2,{\lambda }_3):={\pi }^{\text{BSRR}}(\mathbf{A},\mathbf{B}|\mathit{\lambda },{\lambda }_1=0).$$ (27)

We denote the Bayesian methods using (26) and (27) as RR (Row-wise-sparse and Reduced-rank) method and RC (Row-and-Column-wise sparse) method, respectively. Our BSRR method aims to recover all the low-dimensional structures in A1–A3, but RR and RC methods, respectively, do not consider the column-wise sparsity of C in A3 and the reduced rank structure of C in A1. Therefore, the RR method is analogous to the joint rank and predictor selection methods proposed by Chen and Huang [12] and Bunea et al. [9]. The RR and RC methods can be derived from BSRR method with setting λ₃ = 0 and λ₁ = 0 in (13), respectively. Hence, the BSRR estimate Ĉ_BSRR as well as RR and RC estimates, Ĉ_RR and Ĉ_RC, are obtained by the proposed algorithm in Section 4.2. Similarly, the unknown tuning parameter λ for each model is estimated by the proposed BIC in (25).

We generate data from the multivariate regression model Y = XC + E. For the n × p design matrix X, its n rows are independently generated from 𝒩_p(0, Γ), where Γ = (Γ_ij)_p×p with Γ_ij = (0.5)^|i−j|. The p × q coefficient matrix C is defined as $\mathbf{C}={\sum }_{k=1}^{r^{\ast }}{s}_k{\mathbf{C}}_k$, where s_k = 5 + (k − 1)ȳ15/r*ɉ; the entries of C_k are all zero expect in its upper left p* × q* submatrix, which is generated by ${\mathbf{z}}_1{\mathbf{z}}_2^{\top }/({\Vert {\mathbf{z}}_1\Vert }_2{\Vert {\mathbf{z}}_2\Vert }_2)$, where z₁ ∈ ℝ^p*, z₂ ∈ ℝ^q*, and all their entries are i.i.d samples from uniform([−1, −0.3] ∪ [0.3,1]). The rows of the noise matrix E are independently generated from 𝒩_p(0, Σ_e), where Σ_e = (Σ_ij)_q×q with Σ_ij = σ²(0.5)^|i−j| and σ² is chosen according to the signal to noise ratio(SNR) defined by s_r* (XC)/s₁(P_XE) with P_X = X(X^⊤X)⁻X^⊤.

In the first scenario, we generate models of moderate dimensions (i.e., p, q < n) in three setups:

• (a1) p = q = 25, n = 50, r* = 3, p* = 10, q* = 10. This setup favors our BSRR method.

• (a2) p = q = 25, n = 50, r* = 3, p* = 10, q* = 25. As all the responses are revelent in the model, this setup favors the RR method.

• (a3) p = q = 25, n = 50, r* = 10, p* = 10, q* = 10. This favors the RC method, which does not enforce rank reduction.

In the second scenario, we generate high-dimensional data (i.e., p, q > n) using similar settings as above,

• (b1) p = 200, q = 170, n = 50, r* = 3, p* = 10, q* = 10.

• (b2) p = 200, q = 170, n = 50, r* = 3, p* = 10, q* = 170.

• (b3) p = 200, q = 170, n = 50, r* = 10, p* = 10, q* = 10.

The estimation accuracy is measured by the following three mean squared errors (MSEs):

$$\begin{array}{c}{\text{MSE}}_{\text{est}}=100{\Vert \widehat{\mathbf{C}}-\mathbf{C}\Vert }_F^2/(\mathcal{P}q),\\ {\text{MSE}}_{\text{pred}}=100{\Vert \mathbf{X}\widehat{\mathbf{C}}-\mathbf{X}\mathbf{C}\Vert }_F^2/(nq),\\ {\text{MSE}}_{dim}=100{\Vert \mathrm{s}(\widehat{\mathbf{C}})-\mathbf{s}(\mathbf{C})\Vert }^2/min(p,q),\end{array}$$

where s(C) denotes the vector of singular values for a matrix C. To assess the variable selection performance, we use false positive rate (FPR) and false negative rate (FNR) such that FPR% = 100FP/(TN + FP) and FNR% = 100FN/(TP + FN), where TP, FP, TN and FN denote the numbers of true nonzeros, false nonzeros, true zeros and false zeros, respectively. The rank selection performance is evaluated by the percentage of correct rank identification (CRI%). All measurements are estimated by the Monte Carlo method with 500 replications.

Tables 1 and 2 summarize the simulation results. As expected, in the cases (a1) and (b1), where rank reduction, predictor selection and response selection are all preferable, the BSRR method performs much better than the other two reduced methods. In cases (a2) and (b2), rank reduction and predictor selection are preferable while response selection is not necessary. The performance of the BSRR method is very similar to the RR method which assumes the correct model structure. In the cases (a3) and (b3), rank reduction becomes unnecessary when response and predictor selections are performed. While the BSRR method slightly underestimates the true rank (r* = 10), its variable selection performance (FPR and FNR) is comparable to that of the RC method which assumes the correct model structure. The results are consistent for different SNR levels. Therefore, our BSRR approach provides a flexible and unified way for simultaneously exploring rank reduction, predictor selection and response selection.

Table 1.

Summary of the simulation results for examples (a1)–(a3).

Case	SNR	Method	MSEest	MSEpred	MSEdim	FPR%	FPR%	CRI%	r̂
(a1)	0.50	BSRR	7.42	109.63	17.59	0.41	0.24	100.00	3.00
		RR	14.74	220.78	49.13	28.91	0.26	98.80	3.01
		RC	16.26	212.42	198.35	3.53	0.02	0.00	9.73
	0.75	BSRR	2.73	40.11	5.40	0.02	0.02	100.00	3.00
		RR	5.14	81.80	10.06	28.55	0.06	100.00	3.00
		RC	5.50	71.34	64.30	0.43	0.00	0.00	9.30
	1.00	BSRR	1.48	21.69	2.85	0.00	0.02	100.00	3.00
		RR	2.60	42.38	3.93	28.57	0.00	100.00	3.00
		RC	2.89	37.46	33.57	0.06	0.00	0.00	9.07
(a2)	0.50	BSRR	17.00	255.77	50.98	0.01	0.68	100.00	3.00
		RR	17.48	259.70	56.60	0.01	0.44	100.00	3.00
		RC	38.16	470.79	535.99	0.03	0.01	0.00	10.00
	0.75	BSRR	6.01	96.44	9.77	0.00	0.24	100.00	3.00
		RR	6.15	97.63	11.59	0.00	0.08	100.00	3.00
		RC	16.43	204.44	232.50	0.00	0.02	0.00	10.00
	1.00	BSRR	3.06	50.14	3.70	0.00	0.10	100.00	3.00
		RR	3.12	50.62	4.32	0.00	0.04	100.00	3.00
		RC	8.93	112.27	128.20	0.00	0.00	0.00	10.00
(a3)	0.50	BSRR	0.03	0.38	0.15	0.36	0.00	18.20	9.10
		RR	0.07	0.87	0.12	28.69	0.00	23.00	9.16
		RC	0.03	0.36	0.07	1.64	0.00	98.00	9.98
	0.75	BSRR	0.02	0.19	0.10	0.80	0.00	19.40	9.12
		RR	0.03	0.40	0.09	28.82	0.00	21.40	9.14
		RC	0.01	0.15	0.03	3.50	0.00	98.20	9.98
	1.00	BSRR	0.01	0.12	0.09	1.73	0.00	20.00	9.12
		RR	0.02	0.24	0.08	29.16	0.00	21.20	9.13
		RC	0.01	0.09	0.02	4.38	0.00	98.40	9.98

Table 2.

Summary of the simulation results for examples (b1)–(b3).

Case	SNR	Method	MSEest	MSEpred	MSEdim	FPR%	FNR%	CRI%	r̂
(b1)	0.50	BSRR	0.03	3.12	0.47	0.00	0.06	98.00	3.00
		RR	0.35	44.87	21.49	4.72	0.08	30.40	3.69
		RC	0.06	6.67	5.68	0.04	0.02	0.00	9.14
	0.75	BSRR	0.01	1.37	0.23	0.00	0.04	99.20	3.00
		RR	0.15	19.28	8.68	4.72	0.02	27.20	3.73
		RC	0.02	2.29	1.90	0.00	0.02	0.00	8.66
	1.00	BSRR	0.01	0.79	0.17	0.00	0.00	99.80	3.00
		RR	0.07	9.97	3.54	4.72	0.02	51.20	3.49
		RC	0.01	1.22	0.94	0.00	0.00	0.00	8.20
(b2)	0.50	BSRR	0.30	44.79	4.78	0.00	0.79	99.60	3.00
		RR	0.31	45.36	6.04	0.00	0.16	99.40	3.00
		RC	1.29	128.08	168.35	0.00	0.03	0.00	10.00
	0.75	BSRR	0.12	18.79	0.82	0.00	0.38	99.20	3.00
		RR	0.12	18.77	1.09	0.00	0.00	99.20	3.00
		RC	0.56	55.61	72.10	0.00	0.07	0.00	10.00
	1.00	BSRR	0.07	10.44	0.27	0.00	0.26	99.60	3.00
		RR	0.07	10.36	0.36	0.00	0.00	99.40	3.00
		RC	0.30	30.43	39.46	0.00	0.06	0.00	10.00
(b3)	0.50	BSRR	0.00	0.02	0.01	0.02	0.00	19.40	9.13
		RR	0.00	0.21	0.02	4.68	0.00	27.20	9.22
		RC	0.00	0.01	0.00	0.00	0.00	98.20	9.98
	0.75	BSRR	0.00	0.01	0.01	0.03	0.00	19.20	9.13
		RR	0.00	0.10	0.01	4.67	0.00	24.60	9.19
		RC	0.00	0.01	0.00	0.00	0.00	98.40	9.98
	1.00	BSRR	0.00	0.01	0.01	0.02	0.00	19.20	9.13
		RR	0.00	0.06	0.01	4.66	0.00	22.40	9.16
		RC	0.00	0.00	0.00	0.00	0.00	98.60	9.99

7. Yeast cell cycle data

Transcription factors (TFs), also called sequence-specific DNA binding proteins, regulate the transcription of genes from DNA to mRNA by binding specific DNA sequences. In order to understand the regulatory mechanism, it is important to reveal the network structure between TFs and their target genes. The network structure can be formulated using the multivariate regression model in (1), where the row and column of the response matrix, respectively, correspond to genes and samples (arrays, tissue types, time points), and the design matrix includes the binding information representing the strength of interaction between TFs and the target genes. The regression coefficient matrix then describes actual transcription factor activities of TFs for genes. In practice, many TFs are not actually related to the genes and there exists dependency among the samples due to the design of experiment.

Here, we analyze an Yeast cell cycle data [14] using BSRR. The dataset is available in the spls package in R. The response matrix Y consists of 542 cell-cycle-regulated genes from an α factor arrest method, where mRNA levels are measured at every 7 minutes during 119 minutes, i.e., n = 542 and q = 18. The 542 × 106 predictor matrix X contains the binding information of the target genes for a total of 106 TFs, where Chromatin immunoprecipitation (ChIP) for the 542 genes was performed on each of these 106 TFs. In our analyses, Y and X are centered.

We apply the BSRR method to the dataset. We use the proposed BIC to choose the tuning parameter and obtain λ̂₁ = 5, λ̂₂ = 1.5 and λ̂₃ = 1.5. As a result, 26 TFs are identified at 17 time points (105 min is eliminated) with the estimated rank r̂ = 4. Fig. 2 displays the obtained parameter estimates and 95% credible bands for randomly selected 4 TFs among the 26 TFs; see figures in the supplementary materials for all TFs. The same data set was also analyzed by the adaptive SRRR method of Chen and Huang [12]. In the adaptive SRRR, 32 TFs were identified at 18 time points with the optimal rank r̂ = 4 determined by a cross validation method. Among their selected 32 TFs, 21 TFs were also identified by our BSRR method. To compare variable selection performance between two methods, we define the following two models:

Figure 2.

The parameter estimates and 95% credible bands for randomly selected 4 TFs from the BSRR, where x-axis indicates time (min) and y-axis indicates estimated coefficients.

$$\begin{array}{c}{M}^1:\mathbf{Y}={\mathbf{X}}_1{\mathbf{C}}_1+\mathbf{E};\\ M^2:\mathbf{Y}={\mathbf{X}}_2{\mathbf{C}}_2+\mathbf{E};\end{array}$$

where X₁ contains the information of the 542 genes for the 32 TFs identified by the adaptive SRRR, X₂ contains the information of the 542 genes for the 26 TFs identified by BSRR, C₁ is the 32 × 18 matrix with rank(C₁) = 4, and C₂ is the 26 × 18 matrix with rank(C₂) = 4. To conduct a fair comparison, we consider the following reduced rank priors in Geweke [20] for the models M¹ and M², respectively:

$${\pi }^1({\mathbf{A}}_1,{\mathbf{B}}_1)\propto exp\{-\frac{\tau }{2}({\Vert {\mathbf{A}}_1\Vert }_F^2+{\Vert {\mathbf{B}}_1\Vert }_F^2)\}\mathrm{s}.\mathrm{t}.{\mathbf{C}}_1={\mathbf{A}}_1{\mathbf{B}}_1^{\top },$$ (28)

$${\pi }^2({\mathbf{A}}_2,{\mathbf{B}}_2)\propto exp\{-\frac{\tau }{2}({\Vert {\mathbf{A}}_2\Vert }_F^2+{\Vert {\mathbf{B}}_2\Vert }_F^2)\}\mathrm{s}.\mathrm{t}.{\mathbf{C}}_2={\mathbf{A}}_2{\mathbf{B}}_2^{\top },$$ (29)

where A₁ is a 32 × 4 matrix, B₁ is an 18 × 4 matrix, A₂ is an 26 × 4 matrix, B₂ is a 18 × 4 matrix and we set τ = 0.0001 to be a non-informative (flat) prior, so that the parameter estimates are determined nearly by the observations (Y₁, X₁) and (Y₂, X₂). As the Bayesian model selection criterion, using the priors in (28) and (29), we utilize the deviance information criteria (DIC) defined by

$$\begin{array}{cc}{\text{DIC}}_1=& -4E_{{\mathbf{A}}_1,{\mathbf{B}}_1|\mathbf{Y},{\mathbf{X}}_1}[ln\{f(\mathbf{Y}|{\mathbf{X}}_1,{\mathbf{C}}_1={\mathrm{A}}_1{\mathbf{b}}_1^{\top })\}]\\ & +2ln\left\{f\right(\mathbf{Y}|{\mathbf{X}}_1,{\mathbf{C}}_1=\overline{{\mathbf{A}}_1{\mathbf{B}}_1^{\top }})\},\\ {\text{DIC}}_2=& -4E_{{\mathbf{A}}_2,{\mathbf{B}}_2|\mathbf{Y},{\mathbf{X}}_2}[ln\{f(\mathbf{Y}|{\mathbf{X}}_2,{\mathbf{C}}_2={\mathbf{A}}_2{\mathbf{B}}_2^{\top })\}]\\ & +2ln\left\{f\right(\mathbf{Y}|{\mathbf{X}}_2,{\mathbf{C}}_2=\overline{{\mathbf{A}}_2{\mathbf{B}}_2^{\top }})\},\end{array}$$

where $\overline{\mathbf{A}{\mathbf{B}}^{\top }}$ denotes the posterior mean. If DIC₁ > DIC₂, then it implies that the model M² is more strongly supported by the given data than the model M¹. Let ${\{{\mathbf{A}}_1^i,{\mathbf{B}}_1^i\}}_{i=1}^N$ and ${\{{\mathbf{A}}_2^i,{\mathbf{B}}_2^i\}}_{i=1}^N$ be MCMC samples from the posteriors π¹(A₁, B₁ | Y, X₁) and π²(A₂, B₂ | Y, X₂), respectively. Note that the MCMC samples can be easily generated from multivariate normal distributions by using the Gibbs sampler. Define ${\{{\mathbf{C}}_m^i:{\mathbf{C}}_m^i={\mathbf{A}}_m^i{({\mathbf{B}}_m^i)}^{\top }\}}_{i=1}^N$, for m = 1, 2. Then the DIC can be estimated by the following Monte Carlo estimator:

$$\begin{array}{c}{\widehat{\text{DIC}}}_1=-4\{\frac{1}{N}\sum _{i=1}^{N}lnf(\mathbf{Y}|{\mathbf{X}}_1,{\mathbf{C}}_1^i)\}+2lnf(\mathbf{Y}|{\mathbf{X}}_1,\frac{1}{N}\sum _{i=1}^N{\mathbf{C}}_1^i),\\ {\widehat{\text{DIC}}}_2=-4\{\frac{1}{N}\sum _{i=1}^{N}lnf(\mathbf{Y}|{\mathbf{X}}_2,{\mathbf{C}}_2^i)\}+2lnf(\mathbf{Y}|{\mathbf{X}}_2,\frac{1}{N}\sum _{i=1}^N{\mathbf{C}}_2^i).\end{array}$$

Based on 1,000 MCMC samples (after 1,000 burn-in iterations) with 100 replication, we obtain $\widehat{{\text{DIC}}_1}=19824.46$ and $\widehat{{\text{DIC}}_2}=19784.03$ with Monte Carlo errors 1.29 and 1.14, respectively. Since $\widehat{{\text{DIC}}_1}>\widehat{{\text{DIC}}_2}$, this result supports the model M². Consequently, this implies that our BSRR method has better variable selection performance than the adaptive SRRR for Yeast cell cycle data. Recall that the response at 105 min was eliminated in the BSRR method. Table 3 displays the parameter estimates and 95% credible intervals (CIs) at 105 min from the model M². Since all CIs include zero, this demonstrates that the response elimination at 105 min in the BSRR is valid. In other words, none of TFs activates at 105 min.

Table 3.

Parameter estimates (Est) with 95% credible intervals (CIs) at 105 min from the model M².

TF	Est	CIs	TF	Est	CIs
ACE2	−0.03	(−0.13, 0.10)	RME1	0.05	(−0.08, 0.17
ARG81	−0.05	(−0.19, 0.09)	RTG3	0.01	(−0.09, 0.11)
FKH2	0.06	(−0.01, 0.13)	SFP1	−0.05	(−0.15, 0.05)
HIR1	0.10	(−0.06, 0.26)	SOK2	−0.03	(−0.07, 0.02)
HIR2	0.07	(−0.06, 0.21)	STB1	0.01	(−0.05, 0.06)
IME4	−0.02	(−0.13, 0.08)	STE12	−0.04	(−0.21, 0.12)
MBP1	−0.04	(−0.13, 0.05)	SWI4	0.02	(−0.03, 0.08)
MCM1	−0.03	(−0.11, 0.04)	SWI5	−0.06	(−0.21, 0.09)
MET4	0.06	(−0.02, 0.13)	SWI6	−0.01	(−0.08, 0.07)
NDD1	0.05	(−0.08, 0.17)	YAP7	0.03	(−0.04, 0.10)
NRG1	0.02	(−0.06, 0.11)	YFL044C	0.04	(−0.06, 0.15)
PHD1	−0.02	(−0.09, 0.04)	YJL206C	0.08	(−0.05, 0.22)
REB1	0.03	(−0.03, 0.09)	ZAP1	−0.03	(−0.15, 0.09)

8. Discussion

We have developed a Bayesian sparse and low rank regression method, which achieves simultaneous rank reduction and predictor/response selection. There are many directions for future research. We have mainly focused on the ℓ₀ type sparsity-inducing penalties to construct prior distribution. The method can be extended to use other forms of penalties for inducing diverse lower-dimensional structures. The low-rank structure induces dependency among the response variables, and hence the error correlation structure is not explicitly considered in the current work. Incorporating the variance component into our model might improve the efficiency of the coefficient estimation. In a Bayesian framework, this can be accomplished by assigning an appropriate prior on the variance component; the choice of the prior should be carefully treated due to the lack of unimodality of the posterior [31]. In practice, the response variables could be binary or counts. It is thus pressing to utilize a general likelihood function with the proposed BSRR prior. The proposed ICM algorithm converges relatively fast, and in each iteration the main cost is to inverse a matrix of dimension min(n, p) owning to Woodbury matrix identity. However, this approach would still be inefficient when both p and n are extremely large. One way is to conduct some pre-screening procedure [17, 19] before implementation of the proposed method. It would also be interesting to study on-line learning and the divide-and-conquer strategies of the proposed model. We have established the posterior consistency of the proposed sparse and low-rank estimation method under a high-dimensional asymptotic regime, which characterizes the behavior of the posterior distribution when the number of predictors p_n increases with the sample size n. The theoretical analysis of the Bayesian (point) estimator itself could be of interest rather than the entire posterior distribution [1].