Bayesian tensor-on-tensor regression with efficient computation

Abstract

We propose a Bayesian tensor-on-tensor regression approach to predict a multidimensional array (tensor) of arbitrary dimensions from another tensor of arbitrary dimensions, building upon the Tucker decomposition of the regression coefficient tensor. Traditional tensor regression methods making use of the Tucker decomposition either assume the dimension of the core tensor to be known or estimate it via cross-validation or some model selection criteria. However, no existing method can simultaneously estimate the model dimension (the dimension of the core tensor) and other model parameters. To fill this gap, we develop an efficient Markov Chain Monte Carlo (MCMC) algorithm to estimate both the model dimension and parameters for posterior inference. Besides the MCMC sampler, we also develop an ultra-fast optimization-based computing algorithm wherein the maximum a posteriori estimators for parameters are computed, and the model dimension is optimized via a simulated annealing algorithm. The proposed Bayesian framework provides a natural way for uncertainty quantification. Through extensive simulation studies, we evaluate the proposed Bayesian tensor-on-tensor regression model and show its superior performance compared to alternative methods. We also demonstrate its practical effectiveness by applying it to two real-world datasets, including facial imaging data and 3D motion data.

1. INTRODUCTION

Multi-dimensional arrays, also called tensors, are widely used to represent data with complex structures in different fields such as genomics, neuroscience, computer vision, and graph analysis. For example, a multi-tissue experiment (Wang et al., 2019) collects gene expression data in different tissues from different individuals, leading to three-dimensional arrays (Genes × Tissues × Individuals). Other notable examples include magnetic resonance imaging data (MRI, three-dimensional arrays), functional MRI (fMRI) data (four-dimensional arrays), and facial images (four-dimensional arrays) (Vasilescu and Terzopoulos, 2002; Hasan et al., 2011; Guhaniyogi and Spencer, 2021). In this paper, we focus on the task of tensor-on-tensor regression that predicts one multi-dimensional tensor from another multi-dimensional tensor, e.g., predicting gene expression across multiple tissues for multiple individuals from their clinical/omics data with tensor structures.

One simple approach dealing with tensor-on-tensor regression is to turn tensors into vectors, and then apply classic regression methods. However, such a treatment introduces high-dimensional unstructured vectors and destroys the correlation structure of data, resulting in a huge number of parameters to be estimated and potentially significant loss of information. For example, to predict a response tensor of dimensions $N\times Q_1\times Q_2$ from a predictor tensor of dimensions $N\times P_1\times P_2$, the classic linear regression method requires estimating $P_1\times P_2\times Q_1\times Q_2$ parameters, which may cause overfitting or computational issues, especially when the number of parameters is larger than the sample size $N$.

To reduce the number of free parameters while preserving the correlation structure in modeling tensor data, tensor decomposition techniques have been widely applied (Kolda and Bader, 2009). The two most commonly-used tensor decomposition methods are the PARAFAC/CANDECOMP (CP) decomposition (Harshman, 1970) and Tucker decomposition (Tucker, 1966). The CP decomposition reconstructs a tensor as a linear combination of rank-1 tensors, each one of which is represented as the outer product of a number of vectors. On the other hand, the Tucker decomposition factorizes a tensor into a small core tensor and a set of matrices along each dimension. Both decomposition methods are able to reduce model dimensionality to a manageable size and make parameter estimation more efficient. Compared to CP decomposition, Tucker decomposition allows a more flexible correlation structure processed by the core tensor and the freedom in choosing different orders, making it useful in estimating data with skewed dimensions (Li et al., 2013). In fact, CP decomposition is a special case of Tucker decomposition with the core tensor being superdiagonal.

There is a rich literature on regression methods treating tensors as either predictors or responses in both frequentist and Bayesian statistics. Guo et al. (2012) and Zhou et al. (2013) proposed tensor regression models to predict scalar outcomes from tensor predictors by assuming that the coefficient tensor has a low rank CP decomposition. Li et al. (2013) later extended the framework by employing Tucker decomposition for the coefficient tensor, and demonstrated that Tucker decomposition is more suitable to deal with tensor predictors of skewed dimensions and gains better accuracy in neuroimaging data analysis. Guhaniyogi et al. (2017) proposed a Bayesian approach to regression with a scalar response on tensor predictors by developing a multiway Dirichlet generalized double Pareto prior on tensor margins after applying CP decomposition to the coefficient tensor. Miranda et al. (2018) developed a Bayesian tensor partition regression model using a generalized linear model with a sparse inducing normal mixture prior to learn the relationship between a matrix response (clinical outcomes) and a tensor predictor (imaging data). Li and Zhang (2017) proposed a parsimonious regression model with tensor response and vector predictors adopting a generalized sparsity principle based on Tucker decomposition. To detect neuronal activation in fMRI experiments, Guhaniyogi and Spencer (2021) developed a Bayesian regression approach with a tensor response on scalar predictors by introducing a novel multiway stick breaking shrinkage prior distribution on tensor structured regression coefficients.

There exist many scientific applications that require methods for predicting a tensor response from another tensor predictor. One typical example in fMRI studies is to detect brain regions activated by an external stimulus or condition (Zhang et al., 2015). Hoff (2015) proposed a tensor-on-tensor bilinear regression framework to handle a special case where the tensor predictor has the same dimension as the tensor response making use of Tucker decomposition. Billio et al. (2018) introduced a Bayesian tensor autoregressive model to tackle tensor-on-tensor regression, and used CP decomposition to provide parsimonious parametrization. Lock (2018) proposed to predict a tensor response from another tensor predictor by assuming that the coefficient tensor has a low-rank CP factorization. Gahrooei et al. (2021) extended the work of Lock (2018) to allow multiple tensor inputs under the Tucker decomposition framework.

Despite advances in methods development for dealing with tensor data, there are some limitations in the aforementioned methods. First, tensor-on-tensor regression methods based on CP decomposition, e.g., Lock (2018), require both the response tensor and the predictor tensor to have the same rank in CP decomposition, making them restrictive when the response and predictor tensors have different ranks. Second, the rank in CP decomposition and the dimension of the core tensor in Tucker decomposition (i.e., model dimension) are essential for statistical inference in tensor-on-tensor regression models. However, they are either assumed known or estimated via cross-validation (Gahrooei et al., 2021) or some model selection criteria, such as Bayesian information criterion (Guhaniyogi and Spencer, 2021). To our best knowledge, there is no existing method that can simultaneously estimate the model dimension and parameters.

In this paper, we develop a novel Bayesian approach for tensor-on-tensor regression based on Tucker decomposition of the coefficient tensor. The main contributions of this work are threefold. First, our Bayesian framework is built upon the flexible Tucker decomposition so that the response tensor and the predictor tensor can have different dimensions in the core tensor. Second, we propose an efficient Markov chain Monte Carlo (MCMC) algorithm to simultaneously estimate the model dimension (the dimension of the core tensor) and parameters. The resulting posterior inference naturally offers us characterization of uncertainty in parameter estimation and prediction. Third, as an alternative to MCMC, we develop an ultra-fast computing algorithm, in which the maximum a posteriori (MAP) estimators for parameters are computed and meanwhile the dimension of the core tensor is optimized via a simulated annealing (SA) algorithm (Kirkpatrick et al., 1983).

The rest of the article is organized as follows. We start with introducing some preliminaries in Section 2. Section 3 describes the proposed Bayesian tensor-on-tensor regression model. We develop an efficient MCMC algorithm to simultaneously estimate the model dimension and parameters in Section 4. An optimization-based ultra-fast computational algorithm for inference is described in Section 5. Section 6 evaluates the proposed approach via simulation studies and comparisons to alternative methods. Section 7 provides real data analyses on facial imaging data and 3D motion data. Section 8 concludes with a discussion.

2. PRELIMINARIES

2.1. Notations

We begin with introducing notations and operations that will be used throughout the paper. We use uppercase blackboard bold characters $\left(\mathbb{X}\right)$ to denote tensors, bold uppercase characters $\left(\mathbf{X}\right)$ to denote matrices, and bold lowercase characters (a) to denote vectors. The order of a tensor is the number of dimensions. For example, $\mathbb{X}\in {\mathbb{R}}^{I_1\times I_2\times \cdots \times I_N}$ denotes an $N$th order tensor, where $I_n$ denotes the dimension of the $n$th mode, $n=1,\dots ,N$. The $i$th entry of a vector $\mathit{a}$ is denoted as $a_i$; the element $(i,\mathrm{ }j)$ of a matrix $\mathbf{X}$ is denoted as $X_{ij}$; and the entries of a tensor are defined by indices enclosed in square brackets: ${\mathbb{X}}_{\left[i_1,\cdots ,i_N\right]}$, where $i_n\in \left\{1,\cdots ,I_n\right\}$ for $n\in \{1,\cdots N\}$. The $n$th element in a sequence of matrices or vectors is denoted by a subscript in parenthesis. For example, ${\mathbf{X}}_{\left(n\right)}$ denotes the $n$th matrix in a sequence of matrices, and ${\mathbf{x}}_{\left(n\right)}$ denotes the $n$th vector in a sequence of vectors.

The vectorization of a tensor $\mathbb{X}\in {\mathbb{R}}^{I_1\times I_2\times \cdots \times I_N}$ transforms an $N$th order tensor into a column vector $vec\mathbb{X}$ such that the entry ${\mathbb{X}}_{\left[i_1,\cdots ,i_N\right]}$ maps to the $j$th entry of $vec\mathbb{X}$, that is

$${\mathbb{X}}_{\left[i_1,\cdots ,i_N\right]}=\mathit{\text{vec}}{\mathbb{X}}_j,$$ (1)

where $j=1+{\sum }_{k=1}^N\left(i_k-1\right){\prod }_{l=1}^{k-1}{I}_l$. Similarly, $vec\mathbf{X}$ is used to denote the vectorization of a matrix $\mathbf{X}\in {\mathbb{R}}^{I_1\times I_2}$ when $N=2$ in (1). Matricization, also known as unfolding, is the process of transforming a tensor into a matrix. The mode-$n$ matricization of a tensor $\mathbb{X}\in {\mathbb{R}}^{I_1\times I_2\times \cdots \times I_N}$ is denoted by ${\mathbb{X}}_{\left(n\right)}\in {\mathbb{R}}^{I_n\times J}$ where $J={\prod }_{k\ne n}{I}_k$. The entry ${\mathbb{X}}_{\left[i_1,\cdots ,i_N\right]}$ of $\mathbb{X}$ maps to the $\left(i_n,j\right)$ element of the resulting matrix ${\mathbb{X}}_{\left(n\right)}$, where

$$j=1+\sum _{\begin{array}{c}k=1\\ k\ne n\end{array}}^N\left(i_k-1\right)J_k\hspace{1em}\text{with}\hspace{1em}{J}_k=\prod _{\begin{array}{c}l=1\\ l\ne n\end{array}}^{k-1}{I}_l.$$

A more general treatment of the tensor matricization is defined as follows. Let $ℛ=\left\{r_1,\cdots ,r_L\right\}$ and $𝒞=\left\{c_1,\cdots ,c_M\right\}$ be two sets of indices such that $ℛ\cup 𝒞=\{1,\cdots ,N\}$ and $ℛ\cap 𝒞=\mathrm{\varnothing }$. Then the matricized tensor can be specified by ${\mathbb{X}}_{(ℛ\times 𝒞)}\in {\mathbb{R}}^{J\times K}$, where $J={\prod }_{n\in ℛ}{I}_n$ and $K={\prod }_{n\in 𝒞}{I}_n$. And the entry ${\mathbb{X}}_{\left[i_1,\cdots ,i_N\right]}$ maps to the $(j,\mathrm{ }k)$ element of the matrix ${\mathbb{X}}_{(ℛ\times 𝒞)}$, that is

$${\mathbb{X}}_{\left[i_1,\cdots ,i_N\right]}={\left({\mathbb{X}}_{(ℛ\times 𝒞)}\right)}_{jk},$$ (2)

where

$$j=1+\sum _{l=1}^L\left[\left(i_{r_l}-1\right)\prod _{l^{\prime }=1}^{l-1}{I}_{r_{l^{\prime }}}\right],$$

and

$$k=1+\sum _{m=1}^M\left[\left(i_{c_m}-1\right)\prod _{m^{\prime }=1}^{m-1}{I}_{c_{m^{\prime }}}\right].$$

The Kronecker product of matrices $\mathbf{U}\in {\mathbb{R}}^{I\times J}$, and $\mathbf{V}\in {\mathbb{R}}^{K\times L}$ is denoted by $\mathbf{U}\otimes \mathbf{V}$ with the detailed definition and properties shown in A.1. The product of a tensor and a matrix in mode $n$ is defined as the $n$-mode product. The $n$-mode product of $\mathbb{X}\in {\mathbb{R}}^{I_1\times I_2\times \cdots \times I_N}$ with a matrix $\mathbf{U}\in {\mathbb{R}}^{J\times I_n}$ is denoted by $\mathbb{X}{\times }_n\mathbf{U}$, resulting in a new tensor $\mathbb{Y}\in {\mathbb{R}}^{I_1\times \cdots \times I_{n-1}\times J\times I_{n+1}\times \cdots \times I_N}$ where the $\left[i_1,\cdots i_{n-1},j,i_{n+1},\cdots i_N\right]$ entry is defined by

$${\mathbb{Y}}_{\left[i_1,\cdots ,i_{n-1},j,i_{n+1},\cdots ,i_N\right]}=\sum _{i_n=1}^{I_n}{\mathbb{X}}_{\left[i_1,\cdots ,i_N\right]}{U}_{j{i}_n}.$$

An important fact regarding the $n$-mode product is that given matrices $\mathbf{U}\in {\mathbb{R}}^{J_1\times I_n},\mathrm{ }\mathbf{V}\in {\mathbb{R}}^{J_2\times I_m}$ with $m\ne n$, and tensor $\mathbb{X}\in {\mathbb{R}}^{I_1\times I_2\times \cdots \times I_N}$, then

$$\mathbb{X}{\times }_n\mathbf{U}{\times }_m\mathbf{V}=\left(\mathbb{X}{\times }_n\mathbf{U}\right){\times }_m\mathbf{V}=\left(\mathbb{X}{\times }_m\mathbf{V}\right){\times }_n\mathbf{U}.$$

For two tensors $\mathbb{X}\in {\mathbb{R}}^{I_1\times \cdots \times I_N\times P_1\times \cdots \times P_L}$, and $\mathbb{Y}\in {\mathbb{R}}^{P_1\times \cdots \times P_L\times J_1\times \cdots \times J_M}$, the contracted tensor product $⟨\mathbb{X},\mathbb{Y}{⟩}_L$ is defined as

$$\mathbb{Z}=⟨\mathbb{X},\mathbb{Y}{⟩}_L\in {\mathbb{R}}^{I_1\times \cdots \times I_N\times J_1\times \cdots \times J_M}$$

with

$${\mathbb{Z}}_{\left[i_1,\cdots ,i_N,j_1,\cdots ,j_M\right]}=\sum _{p_1=1}^{P_1}\cdots \sum _{p_L=1}^{P_L}{\mathbb{X}}_{\left[i_1,\cdots ,i_N,p_1,\cdots ,p_L\right]}{\mathbb{Y}}_{\left[p_1,\cdots ,p_L,j_1,\cdots ,j_M\right]}.$$

It can be shown that for two matrices $\mathbf{U}\in {\mathbb{R}}^{I\times P}$ and $\mathbf{V}\in {\mathbb{R}}^{P\times J}$, the contracted product $⟨\mathbf{U},\mathrm{ }\mathbf{V}{⟩}_1$ is equivalent to the standard matrix product $\mathbf{U}\mathbf{V}$. Therefore, the contracted product of two tensors can be regarded as an extension of the usual matrix product to higher-order operands.

2.2. Tucker decomposition

The proposed Bayesian tensor-on-tensor model is built upon Tucker decomposition (Tucker, 1966), which decomposes a tensor $\mathbb{B}\in {\mathbb{R}}^{I_1\times I_2\times \cdots \times I_N}$ into a core tensor $\mathbb{G}$ and a set of factor matrices ${\mathbf{A}}_{\left(n\right)},n=1,\dots ,N$, denoted by

$$\mathbb{B}=⟦\mathbb{G};{\mathbf{A}}_{\left(1\right)},{\mathbf{A}}_{\left(2\right)},\cdots ,{\mathbf{A}}_{\left(N\right)}⟧.$$

Or equivalently,

$$\mathbb{B}=\mathbb{G}{\times }_1{\mathbf{A}}_{\left(1\right)}{\times }_2{\mathbf{A}}_{\left(2\right)}\cdots {\times }_N{\mathbf{A}}_{\left(N\right)},$$

with $\mathbb{G}\in {\mathbb{R}}^{J_1\times \cdots \times J_N}$ being the core tensor and ${\mathbf{A}}_{\left(n\right)}\in {\mathbb{R}}^{I_n\times J_n}$ being the factor matrix in mode $n$, for $n=1,\cdots ,N$, forming a sequence of matrices. The order of $\mathbb{G}$ can be the same as the order of $\mathbb{B}$, but more often, we are interested in compressing the information of $\mathbb{B}$ to a smaller size of $\mathbb{G}$ than $\mathbb{B}$. CP decomposition (Harshman, 1970) is a special case of Tucker decomposition wherein the core tensor is superdiagonal.

3. A BAYESIAN TENSOR-ON-TENSOR REGRESSION MODEL

Our task is to predict a tensor response $\mathbb{Y}\in {\mathbb{R}}^{N\times Q_1\times \cdots \times Q_M}$ from a tensor predictor $\mathbb{X}\in {\mathbb{R}}^{N\times P_1\times \cdots \times P_L}$. We propose a Bayesian tensor-on-tensor regression framework by extending the standard multivariate linear regression model from matrices to tensors:

$$\mathbb{Y}=⟨\mathbb{X},\mathbb{B}{⟩}_L+\mathbb{E},$$ (3)

where $\mathbb{B}\in {\mathbb{R}}^{P_1\times \cdots \times P_L\times Q_1\times \cdots \times Q_M}$ denotes the coefficient tensor. In this paper, we assume that each element of $\mathbb{E}\in {\mathbb{R}}^{N\times Q_1\times \cdots \times Q_M}$ follows $N\left(0,{\sigma }^2\right)$ independently for illustration simplicity. More flexible covariance structures are discussed in Appendix A.6. The first $L$ modes of $\mathbb{B}$ contract the dimensions of $\mathbb{X}$ and the last $M$ modes of $\mathbb{B}$ match the modes of $\mathbb{Y}$. For each of the $N$ observations, there are a total of $\left({\prod }_{m=1}^M{Q}_m\right)$ responses and $\left({\prod }_{l=1}^L{P}_l\right)$ predictors. The model can be reformulated into a matrix form as follows,

$${\mathbb{Y}}_{\left(1\right)}={\mathbb{X}}_{\left(1\right)}{\mathbb{B}}_{(𝒫\times 𝒬)}+{\mathbb{E}}_{\left(1\right)},$$ (4)

where $𝒫=\{1,\cdots ,L\}$, $𝒬=\{L+1,\cdots ,L+M\}$, and each row of ${\mathbb{E}}_{\left(1\right)}$ independently follows $N\left(0,{\sigma }^2{\mathbf{I}}_d\right)$ with ${\mathbf{I}}_d$ being an identity matrix of dimension $\left({\prod }_{m=1}^M{Q}_m\right)$. From the equivalence of (3) and (4), it is clear that our model (3) supports linear relations between responses and predictors for each observation.

If $\mathbb{B}$ is unconstrained, the estimation of $\mathbb{B}$ can be obtained by conducting separate ordinary least squares (OLS) regressions for each of the ${\prod }_{m=1}^M{Q}_m$ responses in ${\mathbb{Y}}_{\left(1\right)}$ over ${\mathbb{X}}_{\left(1\right)}$ by equation (4) in the frequentist framework. However, the solution is not well-defined if the number of observations $N$ is less than the number of responses ${\prod }_{m=1}^M{Q}_m$. Even if the solution is well-defined, separate OLS of equation (4) does not consider the correlation structure within the response $\mathbb{Y}$ and predictor $\mathbb{X}$ and between them. Moreover, the total number of parameters in this case is ${\prod }_{l=1}^L{P}_l{\prod }_{m=1}^M{Q}_m$, which can be computationally challenging due to its gigantic size.

In our model, we assume that $\mathbb{B}$ follows the Tucker decomposition defined by

$$\mathbb{B}=⟦\mathbb{G};{\mathbf{U}}_{\left(1\right)},\cdots ,{\mathbf{U}}_{\left(L\right)},{\mathbf{V}}_{\left(1\right)},\cdots ,{\mathbf{V}}_{\left(M\right)}⟧,$$ (5)

where $\mathbb{G}$ denotes the core tensor of dimensions $R_1\times \cdots \times R_L\times S_1\times \cdots \times S_M,\mathrm{ }{\mathbf{U}}_{\left(l\right)}\in {\mathbb{R}}^{P_l\times R_l}$ denotes the factor matrix corresponding to the mode $l$ in $\mathbb{B}$ for $l=1,\cdots ,L$, and ${\mathbf{V}}_{\left(m\right)}\in {\mathbb{R}}^{Q_m\times S_m}$ denotes the factor matrix corresponding to the mode $L+m$ in $\mathbb{B}$ for $m=1,\cdots ,M$.

We complete the proposed Bayesian tensor-on-tensor regression model by assigning priors to the core tensor $\mathbb{G},\mathrm{ }{\left\{{\mathbf{U}}_{\left(l\right)}\right\}}_{l=1}^L,\mathrm{ }{\left\{{\mathbf{V}}_{\left(m\right)}\right\}}_{m=1}^M$, and ${\sigma }^2$. For the core tensor $\mathbb{G}$, we consider a normal prior, that is $vec\mathbb{G}~N\left({\mathit{\mu }}_G,{\mathbf{\Sigma }}_G\right)$ where ${\mathbf{\Sigma }}_G$ is diagonal. For factor matrices ${\left\{{\mathbf{U}}_{\left(l\right)}\right\}}_{l=1}^L$ and ${\left\{{\mathbf{V}}_{\left(m\right)}\right\}}_{m=1}^M$, taking ${\mathbf{U}}_{\left(l\right)}$ and ${\mathbf{V}}_{\left(m\right)}$ as examples, we assign normal priors for $vec{\mathbf{U}}^{\left(l\right)}$ and $vec{\mathbf{V}}^{\left(m\right)}$. That is, $vec{\mathbf{U}}_{\left(l\right)}~N\left({\mathit{\mu }}_{U_l},{\mathbf{\Sigma }}_{U_l}\right)$, and $vec{\mathbf{V}}_{\left(m\right)}~N\left({\mathit{\mu }}_{V_m},{\mathbf{\Sigma }}_{V_m}\right)$, where ${\mathbf{\Sigma }}_{U_l}$ and ${\mathbf{\Sigma }}_{V_m}$ are diagonal matrices. Usually we choose ${\mathit{\mu }}_{U_l}={\mathit{\mu }}_U,\mathrm{ }{\mathit{\mu }}_{V_m}={\mathit{\mu }}_V,\mathrm{ }{\mathbf{\Sigma }}_{U_l}={\mathbf{\Sigma }}_U$, and ${\mathbf{\Sigma }}_{V_m}={\mathbf{\Sigma }}_V$ for all $l=1,\cdots ,L$ and $m=1,\cdots ,M$. Lastly we assign an inverse gamma prior distribution for ${\sigma }^2:{\sigma }^2~IG(\alpha ,\mathrm{ }\beta )$. A discussion on other choices of prior distributions and covariance structures is provided in Appendix A.6.

4. POSTERIOR INFERENCE

We conduct posterior inference using Markov chain Monte Carlo (MCMC) simulations. Given the dimension $\mathit{\theta }=\left(R_1,\dots ,R_L,S_1,\cdots ,S_M\right)$ of the core tensor, we update ${\mathbf{U}}_{\left(l\right)},\mathrm{ }{\mathbf{V}}_{\left(m\right)},\mathrm{ }\mathbb{G}$, and ${\sigma }^2$ using Gibbs sampling transition probabilities for posterior updates, the details of which will be given in Section 4.1. The posterior update of the core tensor dimension $\mathit{\theta }$ is challenging since the dimensions of ${\mathbf{U}}_{\left(l\right)},\mathrm{ }{\mathbf{V}}_{\left(m\right)}$, and $\mathbb{G}$ change when $\mathit{\theta }$ varies. A reversible jump (RJ) MCMC (Green, 1995) algorithm is a natural choice for such a trans-dimensional update, however, it is difficult to construct a practicable RJ scheme due to the high-dimensionality of the problem. To address this challenge, we will develop an efficient Metropolis-Hastings (MH) algorithm to update $\mathit{\theta }$ building upon the idea of fractional Bayes factor (Lee et al., 2016; O’Hagan, 1995) in Section 4.2. The R code for the proposed algorithms with demonstrating examples can be found at GitHub.

4.1. Posterior inference given the dimension of the core tensor

Given the dimension $\mathit{\theta }=\left(R_1,\cdots ,R_L,S_1,\cdots ,S_M\right)$ of the core tensor $\mathbb{G}$, we derive the full conditional posterior distributions of ${\left\{{\mathbf{U}}_{\left(l\right)}\right\}}_{l=1}^L,\mathrm{ }{\left\{{\mathbf{V}}_{\left(m\right)}\right\}}_{m=1}^M,\mathrm{ }\mathbb{G}$, and ${\sigma }^2$ in closed forms. Without loss of generality, we first derive the full conditional posterior distribution of ${\mathbf{U}}_{\left(1\right)}$. The full conditional posterior distributions of $\left\{{\mathbf{U}}_{\left(2\right)},\cdots ,{\mathbf{U}}_{\left(L\right)}\right\}$ can be derived in the same manner.

By properties of $n$-mode product of tensor and Tucker decomposition, we have

$$\mathbb{B}=\mathbb{G}{\times }_2{\mathbf{U}}_{\left(2\right)}\cdots {\times }_L{\mathbf{U}}_{\left(L\right)}{\times }_{L+1}{\mathbf{V}}_{\left(1\right)}\cdots {\times }_{L+M}{\mathbf{V}}_{\left(M\right)}{\times }_1{\mathbf{U}}_{\left(1\right)}.$$

Let ${\mathbb{B}}_{(-)}$ denote $\mathbb{G}{\times }_2{\mathbf{U}}_{\left(2\right)}\cdots {\times }_L{\mathbf{U}}_{\left(L\right)}{\times }_{L+1}{\mathbf{V}}_{\left(1\right)}\cdots {\times }_{L+M}{\mathbf{V}}_{\left(M\right)}$, then ${\mathbb{B}}_{(-)}\in {\mathbb{R}}^{R_1\times P_2\times \cdots \times P_L\times Q_1\times \cdots \times Q_M}$, and

$$\mathbb{B}={\mathbb{B}}_{(-)}{\times }_1{\mathbf{U}}_{\left(1\right)}.$$

We denote the contracted product of ${⟨{\mathbb{B}}_{(-)},\mathbb{X}⟩}_{P_2,\dots ,P_N}$ by a new tensor called $\mathbb{C}$, then tensor $\mathbb{C}\in {\mathbb{R}}^{R_1\times N\times P_1\times Q_1\times \cdots Q_M}$. By tensor matricization of $\mathbb{C}$ into ${\mathbb{C}}_{(ℛ\times 𝒞)}\in {\mathbb{R}}^N{\prod }_{m=1}^M{Q}_m\times R_1{P}_1$, where $ℛ=\left\{N,Q_1,\cdots ,Q_M\right\}$, and $𝒞=\left\{R_1,P_1\right\}$, we can rewrite our model (3) as follows:

$$vec\mathbb{Y}={\mathbb{C}}_{\left(ℛ\times 𝒞\right)}\times vec{\mathbf{U}}_{\left(1\right)}+vec\mathbb{E}.$$ (6)

The proof for equation (6) is given in Appendix A.2.

Following (6), we can easily derive that the full conditional posterior distribution of $vec{\mathbf{U}}_{\left(1\right)}$ is normally distributed:

$$p\left(vec{\mathbf{U}}_{\left(1\right)}\mid vec\mathbb{Y},\mathbb{X},{\sigma }^2,{\mathbf{V}}_{\left(m\right)},{\mathbf{U}}_{\left(l\right)}l\ne 1\right)~N\left({\mathit{\mu }}_U^{\prime },{\mathbf{\Sigma }}_U^{\prime }\right),$$ (7)

where

$$\begin{gathered} {\mathbf{\Sigma }}_U^{\prime }={\left(\frac{{\mathbb{C}}_{(ℛ\times 𝒞)}^T{\mathbb{C}}_{(ℛ\times 𝒞)}}{{\sigma }^2}+{\mathbf{\Sigma }}_U^{-1}\right)}^{-1}, \\ {\mathit{\mu }}_U^{\prime }={\mathbf{\Sigma }}_U^{\prime }\left(\frac{{\mathbb{C}}_{(ℛ\times 𝒞)}^{T}vec\mathbb{Y}}{{\sigma }^2}+{\mathbf{\Sigma }}_U^{-1}{\mathit{\mu }}_U\right). \end{gathered}$$

Figure 2 presents an illustration of updating ${\mathbf{U}}_{\left(1\right)}$.

Figure 2.

Illustration of updating ${\mathbf{U}}_{\left(1\right)}$.

We then derive the conditional distributions of ${\mathbf{V}}_{\left(m\right)}$ given ${\sigma }^2,{\left\{{\mathbf{U}}_{\left(l\right)}\right\}}_{l=1}^L,{\mathbf{V}}_{\left(k\right)}$ for $k\ne m$, and $\mathbb{G}$. Without loss of generality, we derive the full conditional posterior distribution of ${\mathbf{V}}_{\left(1\right)}$ below.

Denote the contracted product of the tensor $\mathbb{G}{\times }_1{\mathbf{U}}_{\left(1\right)}\cdots {\times }_L{\mathbf{U}}_{\left(L\right)}{\times }_{L+2}{\mathbf{V}}_{\left(2\right)}\cdots {\times }_{L+M}{\mathbf{V}}_{\left(M\right)}$ and the tensor $\mathbb{X}$ by a new tensor $\mathbb{D}$, where $\mathbb{D}\in {\mathbb{R}}^{N\times S_1\times Q_2\times \cdots \times Q_M}$. We then matricize $\mathbb{D}$ into a matrix ${\mathbb{D}}_{(ℛ\times 𝒞)}\in {\mathbb{R}}^N{\prod }_{m=2}^M{Q}_m\times S_1$ and write

$${\mathbb{Y}}_{\left(2\right)}={\mathbf{V}}_{\left(1\right)}\times {\left({\mathbb{D}}_{(ℛ\times 𝒞)}\right)}^T+{\mathbb{E}}_{\left(2\right)},$$ (8)

where ${\mathbb{Y}}_{\left(2\right)}\in {\mathbb{R}}^{Q_1\times N}{\prod }_{m=2}^M{Q}_m$ is the mode-2 matricization of tensor $\mathbb{Y}$. The proof of equation (8) can be found in Appendix A.3. Let $\tilde{\mathbf{Y}}={\mathbb{Y}}_{\left(2\right)}^T$. Given that ${\mathbf{V}}_{\left(1\right)}$ follows a normal distribution with a diagonal covariance matrix, we can rewrite (8) as

$$vec\tilde{\mathbf{Y}}=\left({\mathbf{I}}_{Q_1}\otimes {\mathbb{D}}_{(ℛ\times 𝒞)}\right)\times vec{\mathbf{V}}_{\left(1\right)}^T+vec{\left({\mathbb{E}}_{\left(2\right)}\right)}^T,$$

where ${\mathbf{I}}_{Q_1}$ denotes an identity matrix of size $Q_1$. Given that the prior distribution of $vec{\mathbf{V}}_{\left(1\right)}$ is a normal $N\left({\mathit{\mu }}_V,{\mathbf{\Sigma }}_V\right)$ with diagonal ${\mathbf{\Sigma }}_V$, the prior distribution of $vec{\mathbf{V}}_{\left(1\right)}^T$ is also a normal distribution $N\left({\tilde{\mathit{\mu }}}_V,{\tilde{\mathbf{\Sigma }}}_V\right)$ with a diagonal covariance matrix. Then the full conditional posterior distribution of $vec{{\mathbf{V}}_{\left(1\right)}}^T$ is normally distributed:

$$p\left(vec{\mathbf{V}}_{\left(1\right)}^T\mid vec\tilde{\mathbf{Y}},\mathbb{X},{\sigma }^2,{\mathbf{U}}_{\left(l\right)},{\mathbf{V}}_{\left(m\right)}m\ne 1\right)~N\left({\tilde{\mathit{\mu }}}_V^{\prime },{\tilde{\mathbf{\Sigma }}}_V^{\prime }\right),$$ (9)

where

$$\begin{gathered} {\tilde{\mathbf{\Sigma }}}_V^{\prime }={\left(\frac{{\left({\mathbf{I}}_{Q_1}\otimes {\mathbb{D}}_{(ℛ\times 𝒞)}\right)}^T\left({\mathbf{I}}_{Q_1}\otimes {\mathbb{D}}_{(ℛ\times 𝒞)}\right)}{{\sigma }^2}+{\tilde{\mathbf{\Sigma }}}_V^{-1}\right)}^{-1}, \\ {\tilde{\mathit{\mu }}}_V^{\prime }={\tilde{\mathbf{\Sigma }}}_V^{\prime }\left(\frac{{\left({\mathbf{I}}_{Q_1}\otimes {\mathbb{D}}_{(ℛ\times 𝒞)}\right)}^{T}vec\tilde{\mathbf{Y}}}{{\sigma }^2}+{\tilde{\mathbf{\Sigma }}}_V^{-1}{\tilde{\mathit{\mu }}}_V\right). \end{gathered}$$ (10)

The posterior distribution of the core tensor $\mathbb{G}$ is more complex than the posterior of ${\mathbf{U}}_l^{\prime }s$ and ${\mathbf{V}}_m^{\prime }s$. First, we have

$$\begin{gathered} {\mathbb{Y}}_{\left(1\right)}={\mathbb{X}}_{\left(1\right)}{\mathbb{B}}_{(𝒫\times 𝒬)}+{\mathbb{E}}_{\left(1\right)} \\ ={\mathbb{X}}_{\left(1\right)}\left({\mathbf{U}}_{\left(L\right)}\otimes \cdots \otimes {\mathbf{U}}_{\left(1\right)}\right){\mathbb{G}}_{(ℛ\times 𝒞)}\times {\left({\mathbf{V}}_{\left(M\right)}\otimes \cdots \otimes {\mathbf{V}}_{\left(1\right)}\right)}^T+{\mathbb{E}}_{\left(1\right)}. \end{gathered}$$ (11)

Let $\mathbf{U}$ denote $\left({\mathbf{U}}_{\left(L\right)}\otimes \cdots \otimes {\mathbf{U}}_{\left(1\right)}\right)$, and $\mathrm{ }\mathbf{V}$ denote $\left({\mathbf{V}}_{\left(M\right)}\otimes \cdots \otimes {\mathbf{V}}_{\left(1\right)}\right)$, we have

$${\mathbb{Y}}_{\left(1\right)}=\left({\mathbb{X}}_{\left(1\right)}\mathbf{U}\right){\mathbb{G}}_{(ℛ\times 𝒞)}{\mathbf{V}}^T+{\mathbb{E}}_{\left(1\right)}.$$ (12)

If we further let $\widetilde{\tilde{\mathbf{Y}}}$ denote ${\mathbb{Y}}_{\left(1\right)}\mathbf{V}{\left({\mathbf{V}}^T\mathbf{V}\right)}^{-1}$, and $\widetilde{\tilde{\mathbf{E}}}$ denote ${\mathbb{E}}_{\left(1\right)}\mathbf{V}{\left({\mathbf{V}}^T\mathbf{V}\right)}^{-1}$, we can rewrite (12) as

$$\widetilde{\tilde{\mathbf{Y}}}=\left({\mathbb{X}}_{\left(1\right)}\mathbf{U}\right){\mathbb{G}}_{(ℛ\times 𝒞)}+\widetilde{\tilde{\mathbf{E}}},$$

or

$$vec\widetilde{\tilde{\mathbf{Y}}}=\left({\mathbf{I}}_S\otimes \left({\mathbb{X}}_{\left(1\right)}\mathbf{U}\right)\right)vec{\mathbb{G}}_{\left(ℛ\times 𝒞\right)}+vec\widetilde{\tilde{\mathbf{E}}},$$

where ${\mathbf{I}}_S$ denotes an $S\times S$ identity matrix with $S={\prod }_{m=1}^M{S}_m$, and $vec\widetilde{\tilde{\mathbf{E}}}$ is normally distributed with mean 0 and block diagonal covariance matrix ${\sigma }^2{\left({\mathbf{V}}^T\mathbf{V}\right)}^{-1}\otimes {\mathbf{I}}_N$. Then $\mathit{\text{vec}}\widetilde{\tilde{\mathbf{Y}}}~N\left({\mathit{\mu }}_{\widetilde{\tilde{\mathit{Y}}}},{\mathbf{\Sigma }}_{\widetilde{\tilde{\mathit{Y}}}}\right)$ with

$$\begin{gathered} {\mathit{\mu }}_{\widetilde{\tilde{\mathit{Y}}}}=\left({\mathbf{I}}_S\otimes \left({\mathbb{X}}_{\left(1\right)}\mathbf{U}\right)\right)\times vec{\mathbb{G}}_{(ℛ\times 𝒞)}, \\ {\mathbf{\Sigma }}_{\widetilde{\tilde{\mathit{Y}}}}={\sigma }^2{\left({\mathbf{V}}^T\mathbf{V}\right)}^{-1}\otimes {\mathbf{I}}_N. \end{gathered}$$

Given that the prior distribution of $vec\mathbb{G}$ is $N\left({\mathit{\mu }}_G,{\mathbf{\Sigma }}_G\right)$ with a diagonal covariance ${\mathbf{\Sigma }}_G,\mathrm{ }vec{\mathbb{G}}_{(ℛ\times 𝒞)}$ is also normally distributed with ${\tilde{\mathit{\mu }}}_G$ and diagonal covariance ${\tilde{\mathbf{\Sigma }}}_G$ by rearranging elements of ${\mathit{\mu }}_G$ and ${\mathbf{\Sigma }}_G$. Then the full conditional posterior distribution of $vec{\mathbb{G}}_{(ℛ\times 𝒞)}$ is a normal distribution with

$$\begin{gathered} {\tilde{\mathit{\mu }}}_G^{\prime }={\tilde{\mathbf{\Sigma }}}_G^{\prime }\left({\left({\mathbf{I}}_S\otimes \left({\mathbb{X}}_{\left(1\right)}\mathbf{U}\right)\right)}^T{\left({\mathbf{\Sigma }}_{\widetilde{\tilde{\mathbf{Y}}}}\right)}^{-1}vec\widetilde{\tilde{\mathbf{Y}}}+{\tilde{\mathbf{\Sigma }}}_G^{-1}{\tilde{\mathit{\mu }}}_G\right), \\ {\tilde{\mathbf{\Sigma }}}_G^{\prime }={\left({\left({\mathbf{I}}_S\otimes \left({\mathbb{X}}_{\left(1\right)}\mathbf{U}\right)\right)}^T{\left({\mathbf{\Sigma }}_{\widetilde{\tilde{\mathbf{Y}}}}\right)}^{-1}\left({\mathbf{I}}_S\otimes \left({\mathbb{X}}_{\left(1\right)}\mathbf{U}\right)\right)+{\left({\tilde{\mathbf{\Sigma }}}_G\right)}^{-1}\right)}^{-1}. \end{gathered}$$ (13)

Lastly, deriving the full conditional posterior distribution of ${\sigma }^2$ is straightforward:

$$p\left({\sigma }^2\mid \mathbb{Y},\mathbb{X},{\left\{{\mathbf{U}}_{\left(l\right)}\right\}}_{l=1}^L,{\left\{{\mathbf{V}}_{\left(m\right)}\right\}}_{m=1}^M,\mathbb{G}\right)~IG\left({\alpha }^{\prime },{\beta }^{\prime }\right),$$ (14)

where ${\alpha }^{\prime }=\alpha +\frac{NQ}{2},\mathrm{ }{\beta }^{\prime }=\beta +\frac{{∥\mathbb{Y}-⟨\mathbb{X},\mathbb{B}{⟩}_L∥}_F^2}{2}$ with $\mathbb{B}$ defined in (5), and $Q={\prod }_{m=1}^M{Q}_m$.

4.2. Updating the model dimension

In this subsection, we show how to simultaneously update the dimension of the core tensor and estimate model parameters. Denote $\mathit{\theta }=\left(R_1,\dots ,R_L,S_1,\cdots ,S_M\right)$, and we assign a prior distribution $\pi \left(\mathit{\theta }\right)$ to $\mathit{\theta }$, in this study, a uniform distribution over all candidates of $\mathit{\theta }$. Since conditional posterior distribution of $\mathit{\theta }$ is not in closed form, we employ a trans-dimensional Metropolis-Hastings (MH) sampler to update $\mathit{\theta }$. The most challenging task is to design a good proposal distribution that can result in a reasonable acceptance rate given the fact that the dimensions of ${\left\{{\mathbf{U}}_{\left(l\right)}\right\}}_{l=1}^L,\mathrm{ }{\left\{{\mathbf{V}}_{\left(m\right)}\right\}}_{m=1}^M$, and $\mathbb{G}$ change when $\mathit{\theta }$ varies.

To address the challenge, we construct our proposal distribution building upon the idea of fractional Bayes factor (O’Hagan, 1995; Lee et al., 2016). Assuming that at iteration $t-1$ of the MCMC sampler ${\mathit{\theta }}^{(t-1)}=\left(R_1^{(t-1)},\cdots ,R_L^{(t-1)},S_1^{(t-1)}\cdots ,S_M^{(t-1)}\right)$, at iteration $t$ we generate a candidate $\tilde{\mathit{\theta }}$ from the “neighbor” of ${\mathit{\theta }}^{(t-1)}$ defined as $O\left({\mathit{\theta }}^{(t-1)}\right):=\left\{\tilde{\mathit{\theta }}\in \mathbf{\Theta }:{∥{\mathit{\theta }}^{(t-1)}-\tilde{\mathit{\theta }}∥}_{L_1}=1\right\}$, where $\mathbf{\Theta }$ is the parameter space for $\mathit{\theta }$. In this work, we propose to generate $\tilde{\mathit{\theta }}$ uniformly over all candidates in $O\left({\mathit{\theta }}^{(t-1)}\right)$, denoted by $q\left(\tilde{\mathit{\theta }}\mid {\mathit{\theta }}^{(t-1)}\right)$. To calculate the acceptance rate of the proposed $\tilde{\mathit{\theta }}$ in the MH step, we denote $\mathit{\xi }=\left({\left\{{\mathbf{U}}_{\left(l\right)}\right\}}_{l=1}^L,{\left\{{\mathbf{V}}_{\left(m\right)}\right\}}_{m=1}^M,\mathbb{G},{\sigma }^2\right)$, and write the likelihood function as the multiplication of two parts:

$$p\left(\mathbb{Y}∣\mathit{\xi },\mathit{\theta }\right)=\underset{\text{training }}{\underbrace{p(\mathbb{Y}\mid \mathit{\xi },\mathit{\theta }{)}^b}}\times \underset{\text{testing }}{\underbrace{p(\mathbb{Y}\mid \mathit{\xi },\mathit{\theta }{)}^{\left(1-b\right)}}},$$

where $b$ is small and $0<b<1$. The key idea here is to utilize a fraction $b$ of the data as the training data to propose new parameters $\mathit{\xi }$ associated with $\tilde{\mathit{\theta }}$ so that the new values can be accepted with a reasonable acceptance probability. In practice, we usually choose a small value of $b$, e.g., between 0.01 and 0.1, to obtain a reasonable acceptance rate. In particular, we update ${\mathit{\theta }}^{\left(t\right)}$ as follows:

• Generate $\tilde{\mathit{\theta }}$ from $q\left(\cdot \mid {\mathit{\theta }}^{(t-1)}\right)$.

• Generate $\tilde{\mathit{\xi }}$ from a distribution proportional to

  $$p(\mathbb{Y}\mid \tilde{\mathit{\xi }},\tilde{\mathit{\theta }}{)}^b\times p\left(\tilde{\mathit{\xi }}∣\tilde{\mathit{\theta }}\right),$$

  which is the posterior of $\mathit{\xi }$ based on the training portion conditional on $\tilde{\mathit{\theta }}$. The detailed conditional posterior distributions are shown in Appendix A.4.
• Generate $\tilde{\tilde{\xi }}$ from a distribution proportional to

  $$p{\left(\mathbb{Y}\mid \widetilde{\tilde{\mathit{\xi }}},{\mathit{\theta }}^{(t-1)}\right)}^b\times p\left(\widetilde{\tilde{\mathit{\xi }}}\mid {\mathit{\theta }}^{(t-1)}\right),$$

  which is the posterior of $\mathit{\xi }$ based on the training portion conditional on ${\mathit{\theta }}^{(t-1)}$.
• Accept $\tilde{\mathit{\theta }}$ with probability $\mathrm{m}\mathrm{i}\mathrm{n}\left(1,A\left(\tilde{\mathit{\theta }},{\mathit{\theta }}^{(t-1)}\right)\right.$ where

  $$A\left(\tilde{\mathit{\theta }},{\mathit{\theta }}^{(t-1)}\right)=\frac{\pi \left(\tilde{\mathit{\theta }}\right)q\left({\mathit{\theta }}^{(t-1)}\mid \tilde{\mathit{\theta }}\right)}{\pi \left({\mathit{\theta }}^{(t-1)}\right)q\left(\tilde{\mathit{\theta }}\mid {\mathit{\theta }}^{(t-1)}\right)}\times \underset{\left(\mathrm{*}\right)}{\underbrace{\frac{p(\mathbb{Y}\mid \tilde{\mathit{\xi }},\tilde{\mathit{\theta }}{)}^{(1-b)}}{p{\left(\mathbb{Y}\mid \widetilde{\tilde{\mathit{\xi }}},{\mathit{\theta }}^{(t-1)}\right)}^{(1-b)}}}}.$$ (15)

The (*) part in equation (15) coincides with the fractional Bayes factor given in O’Hagan (1995). The detailed proof is given in Appendix A.5.

We summarize the full MCMC sampler in Algorithm 1. For predictive inference, it is straightforward based on the posterior samples obtained from Algorithm 1. Given new data ${\mathbb{X}}_{\mathit{\text{new}}}$ of $\tilde{N}$ samples, we can easily sample ${\mathbb{Y}}_{\mathit{\text{new}}}$ predictions according to

$$vec{\stackrel{\mathbb{ˆ}}{\mathbb{Y}}}_{new}~N\left(\mathit{\text{vec}}\left({⟨{\mathbb{X}}_{new},\stackrel{\mathbb{ˆ}}{\mathbb{B}}⟩}_L\right),{\hat{\sigma }}^2{\mathbf{I}}_{\tilde{N}Q}\right),$$ (16)

where $Q={\prod }_{m=1}^M{Q}_m$, and $\stackrel{\mathbb{ˆ}}{\mathbb{B}}$ is calculated by (5) using post-burn-in samples of $\mathbb{G},{\left\{{\mathbf{U}}_{\left(l\right)}\right\}}_{l=1}^L,{\left\{{\mathbf{V}}_{\left(m\right)}\right\}}_{m=1}^M$, and ${\sigma }^2$.

5. FAST COMPUTING ALGORITHM

In practice, the proposed MCMC sampler involves generating samples from high-dimensional conditional posterior distributions at each iteration, which can be time-consuming. In this section, we propose an ultra-fast optimization-based computing algorithm as an alternative for posterior inference using the maximum a posteriori probability (MAP) estimators. Given the dimension of the core tensor, the MAP estimators of ${\left\{{\mathbf{U}}_{\left(l\right)}\right\}}_{l=1}^L,{\left\{{\mathbf{V}}_{\left(m\right)}\right\}}_{m=1}^M,\mathbb{G}$, and ${\sigma }^2$ can be computed in closed forms. Specifically, the MAP estimator of ${\mathbf{U}}_{\left(1\right)}$ is

$$vec{\mathbf{U}}_{\left(1\right)}^{MAP}={\left({\mathbb{C}}_{(ℛ\times 𝒞)}^T{\mathbb{C}}_{(ℛ\times 𝒞)}+{\sigma }^2{\mathbf{\Sigma }}_U^{-1}\right)}^{-1}{\mathbb{C}}_{(ℛ\times 𝒞)}^{T}vec\mathbb{Y},$$ (17)

when ${\mathit{\mu }}_U=0$. And if we further set ${\mathbf{\Sigma }}_U$ to be an identity matrix, the result is exactly the solution to the ridge linear regression problem

$$\text{arg}\underset{{\mathbf{U}}_{\left(1\right)}}{\mathrm{m}\mathrm{i}\mathrm{n}}{∥\mathbb{Y}-⟨\mathbb{X},\mathbb{B}{⟩}_L∥}_F^2-\lambda {∥{\mathbf{U}}_{\left(1\right)}∥}_2^2,$$

with $\lambda ={\sigma }^2$ and $\mathbb{B}=⟦\mathbb{G};{\mathbf{U}}_{\left(1\right)},\cdots ,{\mathbf{U}}_{\left(L\right)},{\mathbf{V}}_{\left(1\right)},\cdots ,{\mathbf{V}}_{\left(M\right)}⟧$. Similarly, assuming ${\mathit{\mu }}_V=0$, and ${\mathit{\mu }}_G=0$, the MAP estimators of ${\mathbf{V}}_{\left(1\right)}$ and the core tensor $\mathbb{G}$ are given below:

$$vec{\mathbf{V}}_{\left(1\right)}^T{ }^{MAP}={\left({\mathbf{I}}_{Q_1}\otimes \left({\mathbb{D}}_{(ℛ\times 𝒞)}^T{\mathbb{D}}_{(ℛ\times 𝒞)}\right)+{\sigma }^2{\tilde{\mathbf{\Sigma }}}_V^{-1}\right)}^{-1}\times \left({\mathbf{I}}_{Q_1}\otimes {\mathbb{D}}_{(ℛ\times 𝒞)}^T\right) vec\tilde{\mathbf{Y}},$$ (18)

and

$$vec{\mathbb{G}}_{(ℛ\times 𝒞)}^{MAP}={\left(\left({\mathbf{V}}^T\mathbf{V}\right)\otimes \left({\left({\mathbb{X}}_{\left(1\right)}\mathbf{U}\right)}^T\left({\mathbb{X}}_{\left(1\right)}\mathbf{U}\right)\right)+{\sigma }^2{\tilde{\mathbf{\Sigma }}}_G^{-1}\right)}^{-1}\times \left(\left({\mathbf{V}}^T\mathbf{V}\right)\otimes {\left({\mathbb{X}}_{\left(1\right)}\mathbf{U}\right)}^T\right)vec\widetilde{\tilde{\mathbf{Y}}},$$ (19)

with the same notations given in Section 4.1. And the MAP estimator of ${\sigma }^2$ is given by

$${\left({\sigma }^2\right)}^{MAP}=\frac{{\beta }^{\prime }}{{\alpha }^{\prime }-1}$$ (20)

with ${\alpha }^{\prime }=\alpha +\frac{NQ}{2},\mathrm{ }{\beta }^{\prime }=\beta +\frac{{∥\mathbb{Y}-⟨\mathbb{X},\mathbb{B}{⟩}_L∥}_F^2}{2}$, and $Q={\prod }_{m=1}^M{Q}_m$. We remark that the unregularized least square results in Lock (2018) is a special case of our MAP results above, when flat priors are given to ${\mathbf{U}}_{\left(l\right)}$’s and ${\mathbf{V}}_{\left(m\right)}$’s, and the core tensor is fixed to be a superdiagonal tensor.

By using MAP estimators of ${\left\{{\mathbf{U}}_{\left(l\right)}\right\}}_{l=1}^L,\mathrm{ }{\left\{{\mathbf{V}}_{\left(m\right)}\right\}}_{m=1}^M$ and $\mathbb{G}$ instead of generating samples from high dimensional posterior distributions, the problem of choosing the optimal dimension of the core tensor becomes a discrete optimization problem to find the tuple of parameters over an $(L+M)$-dimensional grid of parameters $\mathit{\theta }:=\left(R_1,\cdots ,R_L,S_1,\cdots ,S_M\right)\in \mathrm{\Theta }$ that minimizes some loss function. In this work, we use the Bayesian information criterion (BIC) as the loss function.

To solve the discrete optimization problem, we adopt simulated annealing (SA) algorithm (Kirkpatrick et al., 1983), which is a metaheuristic to approximate global optimum in a large search space. Starting from the current guess ${\mathit{\theta }}^{\left(t\right)}$ at iteration $t$, we uniformly generate $\tilde{\mathit{\theta }}$ from $O\left({\mathit{\theta }}^{\left(t\right)}\right)$. The probability of accepting the new candidate $\tilde{\mathit{\theta }}$ is $A\left(\tilde{\mathit{\theta }},{\mathit{\theta }}^{\left(t\right)}\right)$ defined as follows:

$$A\left(\tilde{\mathit{\theta }},{\mathit{\theta }}^{\left(t\right)}\right)=\left\{\begin{array}{ll}1& \text{if }BIC\left(\tilde{\mathit{\theta }}\right)<BIC\left({\mathit{\theta }}^{\left(t\right)}\right),\\ \mathrm{e}\mathrm{x}\mathrm{p}\left(\frac{BIC\left({\mathit{\theta }}^{\left(t\right)}\right)-BIC\left(\tilde{\mathit{\theta }}\right)}{\zeta \left(t\right)}\right)& \text{otherwise,}\end{array}\right.$$ (21)

where $\zeta \left(t\right)$, a function of the iteration $t$, is the usual temperature parameter in standard SA algorithm. Two commonlyused choices for $\zeta \left(t\right)$ are $\zeta \left(t\right)={\gamma }^t{\zeta }_0$ (Dosso and Oldenburg, 1991), and $\zeta \left(t\right)=\frac{{\zeta }_0}{\mathrm{l}\mathrm{o}\mathrm{g}(1+t)}$ (Geman and Geman, 1984), where ${\zeta }_0$ is the initial temperature. The detailed procedure of the fast computing algorithm is shown in Algorithm 2.

As will be shown in numerical studies of Section 6, the proposed Algorithm 2 performs well in terms of parameter estimation, and saves around 90% of the running time in most simulation cases. For example, the average running time of the BayTensor MCMC algorithm on a problem with $\mathbb{X}\in {\mathbb{R}}^{100\times 16\times 12}$ and $\mathbb{Y}\in {\mathbb{R}}^{100\times 10\times 8}$ is around 23 minutes on a personal laptop, whereas the BayTensor Fast algorithm takes less than 3 minutes when executed on the same machine. We also remark that we only obtain a point estimate for parameter estimation from Algorithm 2 since it is an optimization-based method. If one wants uncertainty quantification of parameters, we suggest to use the estimated optimal dimension of the core tensor from Algorithm 2, and conduct the MCMC sampler introduced in Section 4.1, i.e., update model parameters given the estimated dimension of the core tensor. Such a treatment is able to not only save the computing time for updating the dimension of the core tensor in Algorithm 1, but also quantify parameter uncertainties.

6. SIMULATION STUDIES

We evaluate the proposed Bayesian tensor-on-tensor regression model and the computational algorithms through extensive simulation studies and comparisons to alternative methods.

6.1. Simulation setup

Assume that the response tensor is a three-way array $\mathbb{Y}\in {\mathbb{R}}^{N\times Q_1\times Q_2}$ and the predictor tensor is another three-way array $\mathbb{X}\in {\mathbb{R}}^{N\times P_1\times P_2}$. The sample size is set to $N=100$. We set $P_1=16,\mathrm{ }{P}_2=12,\mathrm{ }{Q}_1=10$, and $Q_2=8$. We generate simulated data as follows:

• Generate $\mathbb{X}\in {\mathbb{R}}^{N\times P_1\times P_2}$.

• Generate ${\mathbf{U}}_{\left(l\right)}$ and ${\mathbf{V}}_{\left(m\right)}$ for $l=1,\cdots ,L$ and $m=1,\cdots ,M$, each with independent $N(0,\mathrm{ }1)$ entries.

• Generate $\mathbb{G}$ with ${\theta }^{\mathrm{*}}$ dimensions, and independent $N(0,\mathrm{ }1)$ entries.

• Generate $\mathbb{E}\in {\mathbb{R}}^{N\times Q_1\times Q_2}$.

• Calculate $\mathbb{Y}=⟨\mathbb{X},\mathbb{B}{⟩}_L+c\mathbb{E}$, where

  $$\mathbb{B}=⟦\mathbb{G};{\mathbf{U}}_{\left(1\right)},\cdots ,{\mathbf{U}}_{\left(L\right)},{\mathbf{V}}_{\left(1\right)},\cdots ,{\mathbf{V}}_{\left(M\right)}⟧,$$

  and $c$ is a scaling parameter for defining the signal-to-noise (SNR) ratio such that

  $$\frac{{∥⟨\mathbb{X},\mathbb{B}{⟩}_L∥}_F^2}{c^2\parallel \mathbb{E}{\parallel }_F^2}=\mathrm{S}\mathrm{N}\mathrm{R}.$$

We consider three different setups for generating the predictor tensor $\mathbb{X}$ and error $\mathbb{E}$. In the first setup, all entries of $\mathbb{X}$ and $\mathbb{E}$ are independently and identically generated from a normal distribution $N(0,\mathrm{ }1)$, called uncorrelated-normal setup. To test the performance of our algorithms when the simulated data generation model is different from the data fitting model, in the second setup all entries of $\mathbb{X}$ are independently and identically generated from a normal $N(0,\mathrm{ }1)$, and entries of $\mathbb{E}$ are independently and identically generated from a student-t distribution with a degree of freedom 3. We call this uncorrelated-t setup. In the third setup, we consider the entries for each observation of tensor $\mathbb{X}$ to have a correlated structure, which can be seen in some real-world applications such as spatial and temporal data (Lankao et al., 2008). We call it correlated setup. Specifically, for each of the $N$ observations of $\mathbb{X}$, denoted by ${\mathbf{X}}_{\left(n\right)}\in {\mathbb{R}}^{P_1\times P_2},n=1,\cdots ,N$, the correlation between the $(i,\mathrm{ }j)$ entry and $(k,\mathrm{ }l)$ entry of ${\mathbf{X}}_{\left(n\right)}$ is $e^{-r\sqrt{|i-k{|}^2+|j-l{|}^2}}$, where $i,\mathrm{ }k=1,\cdots ,P_1,\mathrm{ }j,\mathrm{ }l=1,\cdots ,P_2$, and $r>0$.

In each of the setups, we consider two cases for the simulated true dimension of the core tensor $\mathbb{G}:{\theta }^{\mathrm{*}}=(3,\mathrm{ }3,\mathrm{ }3,\mathrm{ }3)$ and (4, 4, 2, 2), and two cases for SNR = 2 and 5, yielding a total of four cases. We conduct 50 repeated experiments by generating 50 replicated datasets for each case, and apply the proposed Bayesian tensor-on-tensor method using both the MCMC sampler (BayTensor MCMC) in Algorithm 1 and the fast computing Algorithm 2 (BayTensor Fast) for each dataset.

To evaluate the performance of different methods, for each dataset in each case we generate 5 new datasets with $N_{\mathit{\text{new}}}=1000$ observations. The new observations are given by

$${\mathbb{Y}}_{\mathit{\text{new }}}={⟨{\mathbb{X}}_{\mathit{\text{new}}},\mathbb{B}⟩}_L+c{\mathbb{E}}_{\mathit{\text{new,}}}$$

with ${\mathbb{X}}_{\mathit{\text{new}}}$ and ${\mathbb{E}}_{\mathit{\text{new}}}$ generated in the same way as $\mathbb{X}$ and $\mathbb{E}$. We then calculate the relative prediction error (RPE), defined as the average prediction error for the 5 new datasets:

$$\mathbf{R}\mathbf{P}\mathbf{E}=\frac{1}{5}\sum _{i=1}^5\frac{{∥{\mathbb{Y}}_{\mathit{\text{new}}}^{\left(i\right)}-{\stackrel{\mathbb{ˆ}}{\mathbb{Y}}}_{\mathit{\text{new}}}^{\left(i\right)}∥}_F^2}{{∥{\mathbb{Y}}_{\mathit{\text{new}} }^{\left(i\right)}∥}_F^2},$$

where ${\stackrel{\mathbb{ˆ}}{\mathbb{Y}}}_{\mathit{\text{new}}}^{\left(i\right)}$ is the predicted value for the dataset $i,\mathrm{ }i=1,\dots ,5$. For BayTensor MCMC and BayTnesor Fast algorithms, we also calculate the “Dimension Recovery” rate defined by (the number of experiments where recovered dimension equals the true core tensor dimension) / (total number of experiments) for each simulation setup.

For comparison, we consider four alternative methods:

• The tensor-on-tensor regression method based on the CP decomposition from Lock (2018), denoted as the CP method. As the CP method needs to pre-define or estimate the rank $R$, we run their algorithm with different $R$’s and report the result under the optimal $R$ value, defined as the one yielding the smallest RPE.

• Multivariate linear regression method after turning tensors to vectors to solve ${\mathbb{B}}_{(𝒫\times 𝒬)}$ in equation (4), denoted as the OLS method.

• Bayesian multivariate linear regression method with the rescaled spike and slab algorithm prior (Ishwaran and Rao, 2005) for inducing sparsity after turning tensors to vectors, denoted as the SAS (Spike-And-Slab) method.

• Bayesian multivariate linear regression method with the horseshoe prior (Carvalho et al., 2010) for inducing shrinkage after turning tensors to vectors, denoted as the HS (horseshoe) method.

6.2. Simulation results: uncorrelated setup

We first apply the proposed BayTensor MCMC with $b=0.05$, BayTensor Fast, and four alternative methods (including the CP method, the OLS method, the SAS method, and the HS method) to datasets under the uncorrelated-normal setup. For the CP method, we run the algorithm with $R=1,\cdots ,6$. The reason why we choose 6 as the upper bound is that when $R=6$, the total number of parameters in the CP method is 276 which is larger than the true number of parameters 212 for the ${\mathit{\theta }}^{\mathrm{*}}=(4,\mathrm{ }4,\mathrm{ }2,\mathrm{ }2)$ case and 219 for the ${\mathit{\theta }}^{\mathrm{*}}=(3,\mathrm{ }3,\mathrm{ }3,\mathrm{ }3)$ case. The CP method with $R=6$ yields the smallest RPE under both cases in all repeated simulations. Full results from the CP method with different $R$ values are shown in Appendix A.9.

Table 1 reports the means and standard deviations (sd) of RPEs averaging over 50 replicated experiments for each of the four cases under the six methods. No matter when the information in $\mathbb{X}$ and $\mathbb{Y}$ is balanced (i.e., $\left.{\theta }^{\mathrm{*}}=(3,\mathrm{ }3,\mathrm{ }3,\mathrm{ }3)\right)$ or skewed to some modes (i.e., ${\theta }^{\mathrm{*}}=(4,\mathrm{ }4,\mathrm{ }2,\mathrm{ }2)$), BayTensor MCMC and BayTensor Fast always yield smaller prediction RPEs than the CP method, while the OLS method, the SAS method and the HS method all yield much larger RPEs. For example, when ${\theta }^{\mathrm{*}}=(4,\mathrm{ }4,\mathrm{ }2,\mathrm{ }2)$ with SNR = 2, BayTensor MCMC and BayTensor Fast have comparable results with mean RPEs being 0.350 and 0.357 respectively, while the CP method has a slightly larger mean RPE of 0.377, and the OLS method, the SAS method, and the HS method all have much larger mean RPE of 1.016, 0.970, and 1.114 respectively. The superior performance of the Bayesian tensor-on-tensor regression method over the CP method comes from the flexibility of Tucker decomposition that permits different orders along different modes of the core tensor. From Table 1, we can also see that a larger signal to noise ratio leads to lower RPEs. In particular, when SNR = 5, all methods, except for the HS method, have lower RPEs compared to the results from cases where SNR = 2.

Table 1.

Mean RPE with SD on uncorrelated-normal data

RPE (SD)	BayTensor MCMC	BayTensor Fast	CP Method	OLS Method	SAS Method	HS Method
$\theta * = (4, 4, 2, 2)$, SNR=2	0.350 (0.013)	0.357 (0.026)	0.377 (0.014)	1.016 (0.034)	0.970 (0.025)	1.114 (0.310)
$\theta * = (3, 3, 3, 3)$, SNR=2	0.343 (0.004)	0.351 (0.015)	0.391 (0.016)	1.023 (0.031)	0.963 (0.020)	1.079 (0.153)
$\theta * = (4, 4, 2, 2)$, SNR=5	0.173 (0.001)	0.185 (0.034)	0.209 (0.046)	0.751 (0.030)	0.946 (0.038)	1.182 (0.410)
$\theta * = (3, 3, 3, 3)$, SNR=5	0.172 (0.001)	0.181 (0.037)	0.222 (0.016)	0.748 (0.023)	0.934 (0.031)	1.136 (0.240)

As the proposed BayTensor MCMC method and BayTensor Fast method can simultaneously estimate the dimension of the core tensor and other model parameters, we next report the empirical probabilities that the true dimension of the core tensor can be recovered by them in 50 replicated experiments for all 4 cases, as shown in Table 2. BayTensor MCMC has higher recovering rates than BayTensor Fast in all cases. And we can better recover the core tensor dimension with a higher SNR. Another observation is that both BayTensor MCMC and BayTensor Fast have higher recovering rates when ${\theta }^{\mathrm{*}}=(3,\mathrm{ }3,\mathrm{ }3,\mathrm{ }3)$ compared with the cases where ${\theta }^{\mathrm{*}}=(4,\mathrm{ }4,\mathrm{ }2,\mathrm{ }2)$.

Table 2.

Core tensor dimension recovery and number of model parameters on uncorrelated-normal data

	BayTensor MCMC		BayTensor Fast
	Dimension Recovery	# Parameters (SD)	Dimension Recovery	# Parameters (SD)
$\theta * = (4, 4, 2, 2)$, SNR=2	80%	207 (15)	46%	189 (14)
$\theta * = (3, 3, 3, 3)$, SNR=2	98%	218 (6)	54%	193 (18)
$\theta * = (4, 4, 2, 2)$, SNR=5	96%	215 (14)	64%	209 (16)
$\theta * = (3, 3, 3, 3)$, SNR=5	100%	219 (0)	76%	215 (16)

We also present the average numbers of parameters required by BayTensor MCMC and BayTensor Fast in 50 replicated experiments in Table 2. BayTensor MCMC and BayTensor Fast usually require smaller numbers of parameters than the CP method. For example, when ${\theta }^{\mathrm{*}}=(4,\mathrm{ }4,\mathrm{ }2,\mathrm{ }2)$ with SNR = 2, BayTensor MCMC requires 207 parameters on average, and BayTensor Fast requires 196 parameters on average. In contrast, the CP method with $R=6$ has 276 parameters. When ${\theta }^{\mathrm{*}}=(3,\mathrm{ }3,\mathrm{ }3,\mathrm{ }3)$ with SNR = 2, the average numbers of parameters are 218, 193, and 276 for BayTensor MCMC, BayTensor Fast, and the CP method respectively. We can see that both BayTensor MCMC and BayTensor Fast yield higher dimension recovery rates than the CP method with smaller numbers of parameters.

To quantify the prediction uncertainty, for each test dataset under each simulation setup, we collect 1000 post-burn-in samples from BayTensor MCMC and generate the empirical posterior predictive distribution for each element of ${\mathbb{Y}}_{\mathit{\text{new}}}$. The symmetric credible intervals are then calculated. For cases with ${\mathit{\theta }}^{\mathrm{*}}=(4,\mathrm{ }4,\mathrm{ }2,\mathrm{ }2)$, the empirical coverage rates for 95% credible interval are 0.944 (0.013) and 0.945 (0.012) for SNR = 2 and SNR = 5 respectively. And for cases with ${\mathit{\theta }}^{\mathrm{*}}=(3,\mathrm{ }3,\mathrm{ }3,\mathrm{ }3)$, the empirical coverage rates for 95% credible interval are 0.950 (0.011) and 0.950 (0.011) for SNR = 2 and SNR = 5 respectively. Figure 3(a) plots the predictive posterior estimations with 95% credible intervals for a randomly-selected sample of ${\mathbb{Y}}_{\mathit{\text{new}}}$ under a randomly-selected test case when ${\mathit{\theta }}^{\mathrm{*}}=(3,\mathrm{ }3,\mathrm{ }3,\mathrm{ }3)$ and SNR = 5. Figure 3(b) plots the empirical predictive distributions from BayTensor MCMC for 8 randomly selected elements from the same sample of ${\mathbb{Y}}_{\mathit{\text{new}}}$ shown in Figure 3(a).

Figure 3.

Uncertainty quantification for a randomly selected case where ${\mathit{\theta }}^{\mathrm{*}}=(3,\mathrm{ }3,\mathrm{ }3,\mathrm{ }3)$ and SNR = 5. (a): An example of prediction with estimation uncertainty. Elements of ${\mathbb{Y}}_{[1::]}$ are sorted for visualization. (b): Empirical predictive distributions from BayTensor MCMC for randomly selected elements from ${\mathbb{Y}}_{\mathit{\text{new}}}$ with the simulation truths shown as red lines.

We then apply all six methods to datasets under the uncorrelated-t setup. The results are very similar to those under the uncorrelated-normal setup, suggesting that our proposed algorithms have reasonably good performance under model mis-specification when the data fitting model is different from the data generating model. The RPEs and dimension recovery results are presented in Table A.1 and Table A.2 of Appendix A.7, respectively.

6.3. Simulation results: correlated setup

We then apply the proposed BayTensor MCMC method with $b=0.05$, the BayTensor Fast method, the CP method, the OLS method, the SAS method, and the HS method to datasets under the correlated setup. For the CP method, we run the algorithm with $R=1,\cdots ,6$. The CP method with $R=6$ yields the smallest RPE under all cases except for the case of ${\mathit{\theta }}^{\mathrm{*}}=(4,\mathrm{ }4,\mathrm{ }2,\mathrm{ }2)$ and SNR = 2 where the optimal $R=5$. Full results from the CP method with different $R$ values are shown in Appendix A.9. For each replicated dataset, we compute the RPEs under all four methods. The means and standard deviations (sd) of RPEs averaging over the 50 replicated datasets for all 4 cases are presented in Table 3.

Table 3.

Mean RPE with SD on correlated data

RPE (SD)	BayTensor MCMC	BayTensor Fast	CP Method	OLS Method	SAS Method	HS Method
$\theta * = (4, 4, 2, 2)$, SNR=2	0.344 (0.004)	0.350 (0.006)	0.370(0.011)	0.886 (0.086)	0.542 (0.115)	1.380 (0.540)
$\theta * = (3, 3, 3, 3)$, SNR=2	0.344 (0.004)	0.348 (0.006)	0.369(0.007)	0.893 (0.065)	0.532 (0.121)	1.366 (0.783)
$\theta * = (4, 4, 2, 2)$, SNR=5	0.173 (0.002)	0.175 (0.003)	0.188(0.006)	0.492 (0.086)	0.397 (0.141)	1.466 (0.648)
$\theta * = (3, 3, 3, 3)$, SNR=5	0.173 (0.008)	0.173 (0.003)	0.187(0.006)	0.497 (0.059)	0.384 (0.150)	1.479 (1.003)

The prediction RPE results under the correlated setup are analogous to the results under the uncorrelated setup. To summarize, the RPE results from Baytensor MCMC and BayTensor Fast are comparable in all cases. And both of them have better RPEs than the CP method followed by the SAS method, and the OLS method. The HS performs the worst in terms of yielding the highest RPEs in all cases. And when the signal to noise ratio increases from 2 to 5, the prediction accuracies are improved under all methods, except for the HS method.

Table 4 shows the empirical probabilities of dimension recovery and the average numbers of parameters required by BayTensor MCMC and BayTensor Fast in 50 replicated experiments. In all 4 cases, BayTensor MCMC and BayTensor Fast both require a smaller number of parameters than the CP method. And in terms of recovering the dimension of the core tensor, BayTensor MCMC has higher empirical probabilities of recovering the true dimension than BayTensor Fast in all cases. And the recovering probabilities in the cases where ${\mathit{\theta }}^{\mathrm{*}}=(3,\mathrm{ }3,\mathrm{ }3,\mathrm{ }3)$ are higher than those in cases where ${\mathit{\theta }}^{\mathrm{*}}=(4,\mathrm{ }4,\mathrm{ }2,\mathrm{ }2)$ for both methods. We also observe that the recovering probabilities under the correlated setup are smaller than those under the uncorrelated setup for both the BayTensor MCMC and BayTensor Fast.

Table 4.

Core tensor dimension recovery and number of model parameters on correlated data

	BayTensor MCMC		BayTensor Fast
	Dimension Recovery	# Parameters (SD)	Dimension Recovery	# Parameters (SD)
$\theta * = (4, 4, 2, 2)$, SNR=2	72%	207 (17)	36%	196 (16)
$\theta * = (3, 3, 3, 3)$, SNR=2	86%	214 (14)	32%	164 (20)
$\theta * = (4, 4, 2, 2)$, SNR=5	96%	212 (7)	44%	188 (15)
$\theta * = (3, 3, 3, 3)$, SNR=5	96%	218 (6)	56%	196 (15)

7. REAL DATA ANALYSES

We demonstrate the usefulness of the proposed Bayesian tensor-on-tensor regression approach by applying it to two real-world datasets: labeled faces in the wild database (Huang et al. (2007)) and multi-person motion (UMPM) benchmark (Van der Aa et al., 2011).

7.1. Facial image data

We apply the proposed Bayesian tensor-on-tensor regression approach, with $b=0.05$, to predict different attributes of facial images, such as smiling and nose size, using the labeled faces in the wild database (Huang et al. (2007)). The database collects more than 13,000 face images, each of which has been labeled with the name of the person pictured, often a celebrity. There can be multiple images for one person.

We use the estimation results from the attribute classifiers developed in Kumar et al. (2011) for response $\mathbb{Y}$, resulting in a total of 73 describable attributes. These attributes can be categorized into characteristics that describe a person, an expression, or an accessory. The attributes are all given as continuous variables, with a higher value denoting a more obvious characteristic. The proposed approach is applied to predict the 73 attributes from a given facial image $\mathbb{X}$. In this work, we use the frontalized version of facial images (Hassner et al., 2015), which show only forward-facing faces obtained by rotating, scaling, and cutting original facial images. The frontalized images are highly aligned, allowing for appearances to be easily compared across faces. Each frontalized image contains 90 × 90 pixels, with each pixel giving color intensities for red, green, and blue, resulting in a 90 × 90 × 3 tensor for each image. We randomly sample 1000 images. Thus the predictor tensor $\mathbb{X}$ is of dimensions 1000 × 90 × 90 × 3, and the response tensor $\mathbb{Y}$ is of dimensions 1000 × 73. We center the tensor for each image by subtracting the mean of tensors for all images. Another randomly-sampled 1000 images are used as a validation set, in other words, ${\mathbb{X}}_{new}$ is of dimensions 1000 × 90 × 90 × 3, and ${\mathbb{Y}}_{new}$ is of dimensions 1000 × 73.

We apply the BayTensor MCMC in Algorithm 1 and BayTensor Fast in Algorithm 2 to the dataset and conduct inference. For comparison, we also apply the CP method proposed by Lock (2018) to the same dataset. For the CP method, we choose $R=15$ and $\lambda ={10}^5$ since these values yielded the best prediction performance, as reported in Lock (2018). BayTensor MCMC estimates the dimension of the core tensor based on the posterior mode to be (5, 2, 3, 5) with a total of 1154 parameters, resulting in an RPE of 0.375. And BayTensor Fast yields an RPE of 0.446. In contrast, the CP method has 3840 parameters and results in a higher RPE, 0.477. To summarize, the proposed Bayesian tensor-on-tensor regression approach is able to reduce predictive errors with a smaller number of parameters due to the flexibility of the Tucker decomposition that permits different orders along different modes.

Next we report the prediction uncertainty, which is a natural byproduct of the proposed Bayesian framework. We collect 500 post-burn-in MCMC samples and compute the posterior predictive distribution along with credible intervals for each of the 73 attributes for each image. The empirical coverage rate for 95% credible interval is 0.930, and for 90% credible interval is 0.889. Figure 4 shows an example of a test image, and plots the corresponding posterior predictive values for four randomly-selected characteristics.

Figure 4.

An example of a test image and the corresponding posterior predictive values for four selected characteristics.

7.2. Utrecht multi-person motion (UMPM) data

We then apply the proposed Bayesian tensor-on-tensor regression model, with $b=0.05$, to the multi-person motion (UMPM) benchmark (Van der Aa et al., 2011) that contains temporally synchronized video sequences from multiple viewpoints and human motion capture data. Each video is of length 30 to 60 seconds, with resolution of 644 × 484 pixels at 50 fps (frames per second). Motion capture (Mo-Cap) data contain 3D positions of 37 markers at 100 fps for each subject of interest.

To evaluate the performance of our model on 3D motion data, we consider two scenarios, namely ‘chair’ and ‘triangle’ from the UMPM dataset. In the scenario ‘chair’, we have a sequence of 2570 sample images in which the subject of interest starts walking in a circle, finds the chair, sits on the chair and stands up multiple times with different postures. In the scenario ‘triangle’, we have a sequence of 2471 sample images in which the subject walks by following the path of a triangle within a circular area. In our data analysis, the input data was a grayscale image sequence from the front camera with resolution downsized to 32 × 24 pixels, forming a 3-order predictor tensor (i.e. frames × width × height). The response data containing 3D positions of 37 markers is first downsampled to 50fps to match the video, and is then presented as a 3-order tensor (i.e. samples × 3D position × markers). For each scenario, we run 10 repeated experiments, and randomly sample 200 images from the sequence in each experiment to form the predictor tensor data $\mathbb{X}$ of dimensions 200 × 32 × 24 and response tensor data $\mathbb{Y}$ of dimensions 200 × 3 × 37. The remaining images are used as testing data ${\mathbb{X}}_{\mathit{\text{new}}}$ and ${\mathbb{Y}}_{\mathit{\text{new}}}$.

We apply BayTensor MCMC, BayTensor Fast, and the CP method to the datasets. For scenario ‘chair’, the mean prediction RPE of BayTensor MCMC is 0.176 (sd: 0.009) while that of BayTensor Fast and the CP method are 0.206 (sd: 0.053) and 0.254 (sd: 0.064) respectively. The mean number of parameters for BayTensor MCMC is 752 (sd: 58), for BayTensor Fast, the number is 798 (sd: 132), and for the CP method, with $R=10$, the number of parameters is 960. For scenario ‘triangle’, BayTensor MCMC yields a mean RPE of 0.243 (sd: 0.049) with the mean number of parameters being 685 (sd: 127). BayTensor Fast, with the mean number of parameters being 798 (sd: 132), has a slightly larger RPE of 0.252 (sd: 0.028). In contrast, the CP method, with $R=10$, has 960 parameters and results in a worse RPE, 0.346 (sd: 0.049). To summarize, with a smaller number of estimating parameters, the proposed Bayesian tensor-on-tensor regression model is able to yield smaller predictive errors compared to the CP method. And this improvement comes from the flexibility of Tucker decomposition.

To demonstrate the predictive uncertainty, we generate 500 post-burn-in posterior samples, produce the empirical posterior predictive distribution of 3D position for each of the 37 markers, and calculate the symmetric credible intervals. The empirical coverage rate for 95% credible interval is 0.957 for scenario ‘chair’, and 0.931 for scenario ‘triangle’. Figure 5 shows an example of estimation results for 3D position of the #3 marker by BayTensor MCMC method with 95% credible intervals, BayTensor Fast, and the CP method.

Figure 5.

Predictions of x, y, z-position of marker 3 by Bayesian tensor-on-tensor regression method with 95% credible intervals, the fast computing method, and the CP method. The underlying truths are shown in red.

8. DISCUSSION

We developed a Bayesian tensor-on-tensor regression model to predict one tensor response from another tensor predictor, building upon the flexible Tucker decomposition of the coefficient tensor so that the response tensor and predictor tensor can have different dimensions in the core tensor. For posterior inference, we proposed an efficient MCMC algorithm to simultaneously estimate the model dimension and model parameters. In addition, we developed an ultrafast computing algorithm wherein the MAP estimators of model parameters are computed given the model dimension, and then an SA algorithm is used to find the optimal model dimension. Both simulation studies and real data analyses show that the performance of our Bayesian tensor-on-tensor regression model benefits from the flexibility in the core tensor structure compared to alternative methods. And the fast computing algorithm yielded comparable prediction results to the MCMC sampler, meanwhile saved a significant amount of computing time. As the fast computing algorithm is only guaranteed to converge to a local optimum, in practice it can be repeatedly applied with different initial values to search for the global optimum. To our best knowledge, this work represents the first effort in literature to simultaneously estimate the model dimension and parameters in tensor-on-tensor regression setup under a fully Bayesian framework.

There are several interesting future directions. First, instead of assigning normal priors to factor matrices, the proposed framework can easily incorporate other priors, such as sparsity-inducing priors (Guhaniyogi et al., 2017; Miranda et al., 2018), for applications where the sparsity is desired. More general covariance structures for $\mathbb{E}$ will also be considered. Second, this work only considers one tensor predictor. Some applications may require to predict one tensor response from multiple tensor predictors or mixed-type predictors including tensors, matrices, and vectors. We will extend the proposed model to handle more flexible predictors. Third, the proposed BayTensor Fast algorithm provides a general algorithmic framework for estimating the model dimension as well as model parameters. In this work, we consider the SA algorithm and BIC. As a future direction, we plan to explore alternative optimization methods beyond the SA and target functions beyond the BIC. Finally, the computational cost of the proposed MCMC sampler comes as a price for prediction accuracy and uncertainty quantification compared to the proposed fast computing algorithm. To improve the efficiency of posterior computation, we plan to take advantage of certain advanced MCMC techniques, e.g., stochastic variational inference (Hoffman et al., 2013) and pseudo-marginal Metropolis-Hastings algorithms (Andrieu and Roberts, 2009).

Figure 1.

Illustration of Tucker decomposition. Here, the core tensor is of dimension (4, 3, 3).

APPENDIX

A.1. A brief review of matrix Kronecker product

The Kronecker product is a matrix operation that is important in showing posterior distribution of parameters in this paper, and we will briefly review it here.

The Kronecker product of matrices $\mathbf{A}\in {\mathbb{R}}^{I\times J}$, and $\mathbf{B}\in {\mathbb{R}}^{K\times L}$ is denoted by $\mathbf{A}\otimes \mathbf{B}$. And the result is of size $\left(IK\right)\times \left(JL\right)$ defined by

$$\mathbf{A}\otimes \mathbf{B}=\left[\begin{array}{cccc}{a}_{11}\mathbf{B}& a_{12}\mathbf{B}& \cdots & a_{1J}\mathbf{B}\\ a_{21}\mathbf{B}& a_{22}\mathbf{B}& \cdots & a_{2J}\mathbf{B}\\ ⋮& ⋮& \ddots & ⋮\\ a_{I1}\mathbf{B}& a_{I2}\mathbf{B}& \cdots & a_{IJ}\mathbf{B}\end{array}\right].$$

Some of the properties of Kronecker product are proved useful for this paper. See the detailed proofs of these properties in Kolda (2006)

$$\left(\mathbf{A}\otimes \mathbf{B}\right)\left(\mathbf{C}\otimes \mathbf{D}\right)=\mathbf{A}\mathbf{C}\otimes \mathbf{B}\mathbf{D}.$$

Let $\mathbb{X}\in {\mathbb{R}}^{I_1\times I_2\times \cdots \times I_N}$, and $𝒩=\{1,\cdots ,N\}$. Let ${\mathbf{A}}_{\left(n\right)}\in {\mathbb{R}}^{I_n\times J_n}$ be a sequence matrices for all $n\in 𝒩$. Let the ordered sets $ℛ=\left\{r_1,\cdots ,r_L\right\}$ and $𝒞=\left\{c_1,\cdots ,c_M\right\}$ be a partition of $𝒩$, then if

$$\mathbb{X}=\mathbb{Y}{\times }_1{\mathbf{A}}_{\left(1\right)}{\times }_2{\mathbf{A}}_{\left(2\right)}\cdots {\times }_N{\mathbf{A}}_{\left(N\right)}$$

we have

$${\mathbb{X}}_{(ℛ\times 𝒞)}=\left({\mathbf{A}}_{\left(r_L\right)}\otimes \cdots \otimes {\mathbf{A}}_{\left(r_1\right)}\right){\mathbb{Y}}_{(ℛ\times 𝒞)}\times {\left({\mathbf{A}}_{\left(c_M\right)}\otimes \cdots \otimes {\mathbf{A}}_{\left(c_1\right)}\right)}^T.$$

Consequently, if ${\mathbf{A}}_{\left(n\right)}\in {\mathbb{R}}^{I_n\times J_n}$ for all $n\in 𝒩$, then for any specific $n\in 𝒩$ if we have

$$\mathbb{X}=\mathbb{Y}{\times }_1{\mathbf{A}}_{\left(1\right)}{\times }_2{\mathbf{A}}_{\left(2\right)}\cdots {\times }_N{\mathbf{A}}_{\left(N\right)},$$

and then

$${\mathbb{X}}_{\left(n\right)}={\mathbf{A}}_{\left(n\right)}{\mathbb{Y}}_{\left(n\right)}\times {\left({\mathbf{A}}_{\left(N\right)}\otimes \cdots \otimes {\mathbf{A}}_{(n+1)}\otimes {\mathbf{A}}_{(n-1)}\otimes \cdots \otimes {\mathbf{A}}_{\left(1\right)}\right)}^T.$$ (22)

A.2. Proof for equation (6)

By properties of n-mode product of tensor and Tucker decomposition, we have

$$\mathbb{B}=\mathbb{G}{\times }_2{\mathbf{U}}_{\left(2\right)}\cdots {\times }_L{\mathbf{U}}_{\left(L\right)}{\times }_{L+1}{\mathbf{V}}_{\left(1\right)}\cdots {\times }_{L+M}{\mathbf{V}}_{\left(M\right)}{\times }_1{\mathbf{U}}_{\left(1\right)}.$$

Let ${\mathbb{B}}_{(-)}$ denote $\mathbb{G}{\times }_2{\mathbf{U}}_{\left(2\right)}\cdots {\times }_L{\mathbf{U}}_{\left(L\right)}{\times }_{L+1}{\mathbf{V}}_{\left(1\right)}\cdots {\times }_{L+M}{\mathbf{V}}_{\left(M\right)}$, then ${\mathbb{B}}_{(-)}\in {\mathbb{R}}^{R_1\times P_2\times \cdots \times P_L\times Q_1\times \cdots \times Q_M}$, and

$$\mathbb{B}={\mathbb{B}}_{(-)}{\times }_1{\mathbf{U}}_{\left(1\right)}.$$

Therefore,

$${\mathbb{B}}_{\left[p_1,\cdots ,p_L,q_1,\cdots ,q_M\right]}=\sum _{r_1=1}^{R_1}\left({\left({\mathbb{B}}_{(-)}\right)}_{\left[r_1,p_2,\cdots ,p_L,q_1,\cdots ,q_M\right]}{\mathbf{U}}_{\left(1\right)r_1{p}_1}\right).$$

And

$$\begin{gathered} {\hat{\mathbb{Y}}}_{\left[n,q_1,\cdots ,q_M\right]}={\left({\langle \mathbb{X},\mathbb{B}\rangle }_L\right)}_{\left[n,q_1,\cdots ,q_M\right]} \\ =\sum _{p_1=1}^{P_1}\cdots \sum _{p_L=1}^{P_L}{\mathbb{B}}_{\left[p_1,\cdots ,p_L,q_1,\cdots ,q_M\right]}{\mathbb{X}}_{\left[n,p_1,\cdots ,p_L\right]} \\ =\sum _{p_1=1}^{P_1}\cdots \sum _{p_L=1}^{P_L}\sum _{r_1=1}^{R_1}({\left({\mathbb{B}}_{(-)}\right)}_{\left[r_1,p_2,\cdots ,p_L,q_1,\cdots ,q_M\right]}\times {\text{U}}_{\left(1\right)r_1{p}_1}{\mathbb{X}}_{\left[n,p_1,\cdots ,p_L\right]}) \\ =\sum _{r_1=1}^{R_1}\sum _{p_1=1}^{P_1}\left(\sum _{p_2=1}^{P_2}\cdots \sum _{p_L=1}^{P_L}{\left({\mathbb{B}}_{(-)}\right)}_{\left[r_1,p_2,\cdots ,p_L,q_1,\cdots ,q_M\right]}{\mathbb{X}}_{\left[n,p_1,\cdots ,p_L\right]}\right)U_{\left(1\right)r_1{p}_1} \\ =\sum _{r_1=1}^{R_1}\sum _{p_1=1}^{P_1}{\left({\mathbb{B}}_{(-)},{\mathbb{X}}_{P_2,\cdots ,P_N}\right)}_{\left[n,p_1,r_1,q_1,\cdots ,q_M\right]}{U}_{\left(1\right)r_1{p}_1}. \end{gathered}$$

And further,

$$vec\stackrel{\mathbb{ˆ}}{\mathbb{Y}}={\mathbb{C}}_{(ℛ\times 𝒞)}\times vec{\mathbf{U}}_{\left(1\right)}.$$

If we denote the contracted product of ${⟨{\mathbb{B}}_{(-)},\mathbb{X}⟩}_{P_2,\cdots ,P_N}$ by a new tensor called $\mathbb{C}$, then tensor $\mathbb{C}\in {\mathbb{R}}^{R_1\times N\times P_1\times Q_1\times \cdots Q_M}$. And matricize tensor $\mathbb{C}$ to ${\mathbb{C}}_{(ℛ\times 𝒞)}\in {\mathbb{R}}^N{\prod }_{m=1}^M{Q}_m\times R_1{P}_1$, where $ℛ=\left\{N,Q_1,\cdots ,Q_M\right\}$, and $𝒞=\left\{R_1,P_1\right\}$. We have

$$vec\mathbb{Y}={\mathbb{C}}_{(ℛ\times 𝒞)}\times vec{\mathbf{U}}_{\left(1\right)}+vec\mathbb{E}.$$

A.3. Proof for equation (8)

We denote tensor ${\mathbb{D}}_{(-)}=\mathbb{G}{\times }_1{\mathbf{U}}_{\left(1\right)}\cdots {\times }_L{\mathbf{U}}_{\left(L\right)}{\times }_{L+2}{\mathbf{V}}_{\left(2\right)}\cdots {\times }_{L+M}{\mathbf{V}}_{\left(M\right)}$. And then the contracted product of tensor ${\mathbb{D}}_{(-)}$ and $\mathbb{X}$ is a new tensor denoted as $\mathbb{D}$. Then

$$\mathbb{B}={\mathbb{D}}_{(-)}{\times }_{L+1}{\mathbf{V}}_{\left(1\right)}.$$

Then

$${\mathbb{B}}_{\left[p_1,\cdots ,p_L,q_1,\cdots ,q_M\right]}=\sum _{s_1=1}^{S_1}{\mathbb{D}}_{(-)\left[p_1,\cdots ,p_L,s_1,q_2,\cdots ,q_M\right]}{\mathbf{V}}_{\left(1\right)s_1{q}_1},$$

and

$$\begin{gathered} {\hat{\mathbb{Y}}}_{\left[n,q_1,\cdots ,q_M\right]}={\left({\langle \mathbb{X},\mathbb{B}\rangle }_L\right)}_{\left[n,q_1,\cdots ,q_M\right]} \\ =\sum _{p_1=1}^{P_1}\cdots \sum _{p_L=1}^{P_L}{\mathbb{B}}_{\left[p_1,\cdots ,p_L,q_1,\cdots ,q_M\right]}{\mathbb{X}}_{\left[n,p_1,\cdots ,p_L\right]} \\ =\sum _{p_1=1}^{P_1}\cdots \sum _{p_L=1}^{P_L}\sum _{s_1=1}^{S_1}({\left({\mathbb{D}}_{(-)}\right)}_{\left[p_1,\cdots ,p_L,s_1,q_2,\cdots ,q_M\right]}\times V_{\left(1\right)s_1{q}_1{\mathbb{X}}_{\left[n,p_1,\cdots ,p_L\right]})} \\ =\sum _{s_1=1}^{S_1}\sum _{p_1=1}^{P_1}\cdots \sum _{p_L=1}^{P_L}\left({\left({\mathbb{D}}_{(-)}\right)}_{\left[p_1,\cdots ,p_L,s_1,q_2,\cdots ,q_M\right]}{\mathbb{X}}_{\left[n,p_1,\cdots ,p_L\right]}\right)V_{\left(1\right)s_1{q}_1}. \end{gathered}$$ (23)

By equation (23), we have

$$\stackrel{\mathbb{ˆ}}{\mathbb{Y}}=\mathbb{D}{\times }_{L+1}{\mathbf{V}}_{\left(1\right)}.$$ (24)

Combine (22) and equation (24) resulting in

$${\mathbb{Y}}_{\left(2\right)}={\mathbf{V}}_{\left(1\right)}\times {\left({\mathbb{D}}_{(ℛ\times 𝒞)}\right)}^T+{\mathbb{E}}_{\left(2\right)},$$

where ${\mathbb{Y}}_{\left(2\right)}\in {\mathbb{R}}^{Q_1\times N}{\prod }_{m=2}^M{Q}_m$ is the matricization of tensor $\mathbb{Y}$ and ${\mathbb{D}}_{(ℛ\times 𝒞)}\in {\mathbb{R}}^N{\prod }_{m=2}^M{Q}_m\times S_1$ is the matricization of tensor $\mathbb{D}$.

A.4. Conditional posterior distributions given training fraction $\mathit{b}$

Given the dimension $\mathit{\theta }=\left(R_1,\cdots ,R_L,S_1,\cdots ,S_M\right)$ of the core tensor $\mathbb{G}$, and the training fraction $b$, we first derive the full conditional posterior distributions of ${\left\{{\mathbf{U}}_{\left(l\right)}\right\}}_{l=1}^L,\mathrm{ }{\left\{{\mathbf{V}}_{\left(m\right)}\right\}}_{m=1}^M,\mathbb{G},{\sigma }^2$ in closed forms. Without loss of generality, we first derive the full conditional posterior distribution of ${\mathbf{U}}_{\left(1\right)}$. The full conditional posterior distributions of $\left\{{\mathbf{U}}_{\left(2\right)},\cdots ,{\mathbf{U}}_{\left(L\right)}\right\}$ can be derived in the same manner.

By equation (6), we have

$$vec\mathbb{Y}={\mathbb{C}}_{(ℛ\times 𝒞)}\times vec{\mathbf{U}}_{\left(1\right)}+vec\mathbb{E},$$

where $ℛ=\left\{N,Q_1,\cdots ,Q_M\right\}$, and $𝒞=\left\{R_1,P_1\right\}$. And combined with the idea that ${\mathbf{U}}_{\left(1\right)}$ is from a distribution proportional to $p{\left(\mathbb{Y}\mid {\left\{{\mathbf{U}}_{\left(l\right)}\right\}}_{l=1}^L,{\left\{{\mathbf{V}}_{\left(m\right)}\right\}}_{m=1}^M,\mathbb{G},{\sigma }^2,\mathit{\theta }\right)}^b\times p\left({\mathbf{U}}_{\left(1\right)}\mid \right.\mathit{\theta })$, we have, the posterior distribution of $vec{\mathbf{U}}_{\left(1\right)}$ given all other parameters and $b$ is normal distribution. That is

$$vec{\mathbf{U}}_{\left(1\right)}~N\left({\mathit{\mu }}_U^{\prime },{\mathbf{\Sigma }}_U^{\prime }\right)$$ (25)

where,

$$\begin{gathered} {\mathbf{\Sigma }}_U^{\prime }={\left(\frac{b\times {\mathbb{C}}_{(ℛ\times 𝒞)}^T{\mathbb{C}}_{(ℛ\times 𝒞)}}{{\sigma }^2}+{\mathbf{\Sigma }}_U^{-1}\right)}^{-1}, \\ {\mathit{\mu }}_U^{\prime }={\mathbf{\Sigma }}_U^{\prime }\left(\frac{b\times {\mathbb{C}}_{(ℛ\times 𝒞)}^{T}vec\mathbf{Y}}{{\sigma }^2}+{\mathbf{\Sigma }}_U^{-1}{\mathit{\mu }}_U\right). \end{gathered}$$

We then derive the conditional posterior distributions of ${\mathbf{V}}_{\left(m\right)}$ given ${\sigma }^2,{\left\{{\mathbf{U}}_{\left(l\right)}\right\}}_{l=1}^L,{\mathbf{V}}_{\left(k\right)}$ for $k\ne m$, and $\mathbb{G}$. Without loss of generality, we derive the full conditional posterior distribution of ${\mathbf{V}}_{\left(1\right)}$ below.

Denote the contracted product of the tensor $\mathbb{G}{\times }_1{\mathbf{U}}_{\left(1\right)}\cdots {\times }_L{\mathbf{U}}_{\left(L\right)}{\times }_{L+2}{\mathbf{V}}_{\left(2\right)}\cdots {\times }_{L+M}{\mathbf{V}}_{\left(M\right)}$ and tensor $\mathbb{X}$ by a new tensor $\mathbb{D}$, where $\mathbb{D}\in {\mathbb{R}}^{N\times S_1\times Q_2\times \cdots \times Q_M}$. We then matricize $\mathbb{D}$ into a matrix ${\mathbb{D}}_{(ℛ\times 𝒞)}\in {\mathbb{R}}^N{\prod }_{m=2}^M{Q}_m\times S_1$ and write

$${\mathbb{Y}}_{\left(2\right)}={\mathbf{V}}_{\left(1\right)}\times {\left({\mathbb{D}}_{(ℛ\times 𝒞)}\right)}^T+{\mathbb{E}}_{\left(2\right)},$$ (26)

where ${\mathbb{Y}}_{\left(2\right)}\in {\mathbb{R}}^{Q_1\times N}{\prod }_{m=2}^M{Q}_m$ is the matricization of tensor $\mathbb{Y}$. Let $\tilde{\mathbf{Y}}={\mathbb{Y}}_{\left(2\right)}^T$, given ${\mathbf{V}}_{\left(1\right)}$ follows a normal distribution with a diagonal covariance matrix, we can rewrite (26) as

$$vec\tilde{\mathbf{Y}}=\left({\mathbf{I}}_{Q_1}\otimes {\mathbb{D}}_{(ℛ\times 𝒞)}\right)\times vec{\mathbf{V}}_{\left(1\right)}^T+vec{\left({\mathbb{E}}_{\left(2\right)}\right)}^T,$$

where ${\mathbf{I}}_{Q_1}$ denotes an identity matrix of size $Q_1$. Given the prior distribution of $vec{\mathbf{V}}_{\left(1\right)}$ is a normal $N\left({\mathit{\mu }}_V,{\mathbf{\Sigma }}_V\right)$ with diagonal ${\mathbf{\Sigma }}_V$, the prior distribution of $vec{\mathbf{V}}_{\left(1\right)}^T$ is also a normal distribution $N\left({\tilde{\mathit{\mu }}}_V,{\tilde{\mathbf{\Sigma }}}_V\right)$ with a diagonal covariance matrix. Then the full conditional posterior distribution of $vec{\mathbf{V}}_{\left(1\right)}{ }^T$ is also normally distributed:

$$p\left(vec{\mathbf{V}}_{\left(1\right)}^T\mid vec\mathbb{Y},\mathbb{X},{\sigma }^2,{\mathbf{U}}_{\left(l\right)},{\mathbf{V}}_{\left(m\right)}m\ne 1\right)~N\left({\tilde{\mathit{\mu }}}_U^{\prime },{\tilde{\mathbf{\Sigma }}}_U^{\prime }\right),$$ (27)

where

$$\begin{gathered} {\tilde{\mathbf{\Sigma }}}_V^{\prime }={\left(\frac{b\times {\left({\mathbf{I}}_{Q_1}\otimes {\mathbb{D}}_{(ℛ\times 𝒞)}\right)}^T\left({\mathbf{I}}_{Q_1}\otimes {\mathbb{D}}_{(ℛ\times 𝒞)}\right)}{{\sigma }^2}+{\tilde{\mathbf{\Sigma }}}_V^{-1}\right)}^{-1} \\ {\tilde{\mathit{\mu }}}_V^{\prime }={\tilde{\mathbf{\Sigma }}}_V^{\prime }\left(\frac{b\times {\left({\mathbf{I}}_{Q_1}\otimes {\mathbb{D}}_{(ℛ\times 𝒞)}\right)}^{T}vec\tilde{\mathbf{Y}}}{{\sigma }^2}+{\tilde{\mathbf{\Sigma }}}_V^{-1}{\tilde{\mathit{\mu }}}_V\right). \end{gathered}$$ (28)

By analogous procedures, we have the posterior distribution of $vec{\mathbb{G}}_{(ℛ\times 𝒞)}$ is also normal distribution with

$$\begin{gathered} {\mu }_G^{\prime }={\Sigma }_G^{\prime }\left({\left(I_S\otimes \left({\mathbb{X}}_{\left(1\right)}U\right)\right)}^T{\left({\Sigma }_{\tilde{Y}}\right)}^{-1}\left(b\times vec\tilde{Y}\right)+{\Sigma }_G^{-1}{\mu }_G\right) \\ {\text{Σ}}_G^{\prime }={\left(b\times {\left(I_S\otimes \left({\mathbb{X}}_{\left(1\right)}U\right)\right)}^T{\left({\Sigma }_{\tilde{Y}}\right)}^{-1}\left(I_S\otimes \left({\mathbb{X}}_{\left(1\right)}U\right)\right)+{\left({\Sigma }_G\right)}^{-1}\right)}^{-1}, \end{gathered}$$ (29)

where ${\mathbf{I}}_S$ denotes an $S\times S$ identity matrix with $S={\prod }_{m=1}^M{S}_m$. And for ${\sigma }^2$, the posterior given all other parameters and $b$ also follows inverse gamma distribution. That is

$${\sigma }^2~IG\left({\alpha }^{\prime },{\beta }^{\prime }\right)$$ (30)

with ${\alpha }^{\prime }=\alpha +\frac{b\times NQ}{2},{\beta }^{\prime }=\beta +\frac{b\times {∥\mathbb{Y}-⟨\mathbb{X},\mathbb{B}{⟩}_L∥}_F^2}{2}$, and $Q={\prod }_{m=1}^M{Q}_m$.

A.5. Proof for equation (15)

The Fractional Bayes Factor, Eq(11) of O’Hagan (1995), is given by

$$B_b(\mathbb{Y})=\underset{(**)}{\underbrace{\frac{\int p(\tilde{\mathit{\xi }}|\tilde{\mathit{\theta }})p(\mathbb{Y}|\tilde{\mathit{\xi }},\tilde{\mathit{\theta }})d\tilde{\mathit{\xi }}}{\int p(\tilde{\mathit{\xi }}|\tilde{\mathit{\theta }})p{(\mathbb{Y}|\tilde{\mathit{\xi }},\tilde{\mathit{\theta }})}^{b}d\tilde{\mathit{\xi }}}}}\times \frac{{\int }^{\text{}} p\left(\widetilde{\tilde{\mathit{\xi }}}\mid {\mathit{\theta }}^{\left(t-1\right)}\right)p{\left(\mathbb{Y}\mid \widetilde{\tilde{\mathit{\xi }}},{\mathit{\theta }}^{\left(t-1\right)}\right)}^{b}d\widetilde{\tilde{\mathit{\xi }}}}{{\int }^{\text{}} p(\stackrel{}{\widetilde{\tilde{\mathit{\xi }}}\mid {\mathit{\theta }}^{\left(t-1\right)})p\left(\mathbb{Y}\mid \widetilde{\tilde{\mathit{\xi }}},{\mathit{\theta }}^{\left(t-1\right)}\right)d\widetilde{\tilde{\mathit{\xi }}}}.}.$$

We note that

$$\left(**\right)=\frac{\int \stackrel{\left(\mathrm{*}\mathrm{*}\mathrm{*}\right)}{\overbrace{p(\tilde{\mathit{\xi }}\mid \tilde{\mathit{\theta }})p(\mathbb{Y}\mid \tilde{\mathit{\xi }},\tilde{\mathit{\theta }}{)}^b}}p(\mathbb{Y}\mid \tilde{\mathit{\xi }},\tilde{\mathit{\theta }}{)}^{(1-b)}d\tilde{\mathit{\xi }}}{\int p(\tilde{\mathit{\xi }}\mid \tilde{\mathit{\theta }})p(\mathbb{Y}\mid \tilde{\mathit{\xi }},\tilde{\mathit{\theta }}{)}^{b}d\tilde{\mathit{\xi }}}.$$

By similar techniques in O’Hagan (1995), we can rewrite $p(\mathbb{Y}\mid \tilde{\mathit{\xi }},\tilde{\mathit{\theta }}{)}^b$ as $p\left({\mathbb{Y}}^{\prime }\mid \tilde{\mathit{\xi }},\tilde{\mathit{\theta }}\right)$ that is the likelihood based on ${\mathbb{Y}}^{\prime }$ which is the training proportion $b$ of data $\mathbb{Y}$. And rewrite (***) as

$$\begin{gathered} \left(\mathrm{*}\mathrm{*}\mathrm{*}\right)=p(\tilde{\mathit{\xi }}\mid \tilde{\mathit{\theta }})p\left({\mathbb{Y}}^{\prime }\mid \tilde{\mathit{\xi }},\tilde{\mathit{\theta }}\right) \\ =p\left(\tilde{\mathit{\xi }}\mid {\mathbb{Y}}^{\prime },\tilde{\mathit{\theta }}\right)\int p(\tilde{\mathit{\xi }}\mid \tilde{\mathit{\theta }})p\left({\mathbb{Y}}^{\prime }\mid \tilde{\mathit{\xi }},\tilde{\mathit{\theta }}\right)d\tilde{\mathit{\xi }} \\ =p\left(\tilde{\mathit{\xi }}\mid {\mathbb{Y}}^{\prime },\tilde{\mathit{\theta }}\right)\int p(\tilde{\mathit{\xi }}\mid \tilde{\mathit{\theta }})p(\mathbb{Y}\mid \tilde{\mathit{\xi }},\tilde{\mathit{\theta }}{)}^{b}d\tilde{\mathit{\xi }}. \end{gathered}$$

Then $B_b\left(\mathbb{Y}\right)$ becomes

$$B_b\left(\mathbb{Y}\right)=\frac{\int p\left(\tilde{\mathit{\xi }}\mid {\mathbb{Y}}^{\prime },\tilde{\mathit{\theta }}\right)p(\mathbb{Y}\mid \tilde{\mathit{\xi }},\tilde{\mathit{\theta }}{)}^{(1-b)}d\tilde{\mathit{\xi }}}{\int p\left(\widetilde{\tilde{\mathit{\xi }}}\mid {\mathbb{Y}}^{\prime },{\mathit{\theta }}^{(t-1)}\right)p{\left(\mathbb{Y}\mid \widetilde{\tilde{\mathit{\xi }}},{\mathit{\theta }}^{(t-1)}\right)}^{(1-b)}d\widetilde{\tilde{\mathit{\xi }}}}.$$ (31)

And (*) in (15) is evaluating the Fractional Bayes Factor $B_b\left(\mathbb{Y}\right)$ at one sample of $\tilde{\mathit{\xi }}$ and $\widetilde{\tilde{\mathit{\xi }}}$ instead of integrating out as in (31).

A.6. More general covariance structures for prior distributions and noise

In the main manuscript, we assign normal priors with diagonal covariance matrices for ${\left\{{\mathbf{U}}_{\left(l\right)}\right\}}_{l=1}^L,{\left\{{\mathbf{V}}_{\left(m\right)}\right\}}_{m=1}^M,\mathbb{G}$, and show that the full conditional posterior distributions can be derived in closed-forms. When the covariance matrices, ${\mathbf{\Sigma }}_U,\mathrm{ }{\mathbf{\Sigma }}_V$, and ${\mathbf{\Sigma }}_G$ have general structures, we can still derive the full conditional posteriors in closed-forms by rearranging the elements of covariance matrices according to the way we unfold matrices $\mathbf{U},\mathrm{ }\mathbf{V}$ and tensor $\mathbb{G}$. For example, given that the prior of $vec{\mathbf{V}}_{\left(1\right)}$ is a normal distribution with arbitrary covariance matrix ${\mathbf{\Sigma }}_V$, to update $vec{\mathbf{V}}_{\left(1\right)}^T$, we need the covariance matrix of $vec{\mathbf{V}}_{\left(1\right)}^T$ by rearranging the elements of ${\mathbf{\Sigma }}_V$, and then the posterior distribution of $vec{\mathbf{V}}_{\left(1\right)}^T$ is given in the same formula as shown in equations (9) and (10). Similarly, to update $\mathbb{G}$, we need the covariance matrix of $vec{\mathbb{G}}_{ℛ\times 𝒞}$, denoted as ${\tilde{\mathbf{\Sigma }}}_G$, which can be accessed by rearranging the elements of ${\mathbf{\Sigma }}_G$, the covariance matrix of $vec\mathbb{G}$.

For the noise term $vec\mathbb{E}$, we assume that $vec\mathbb{E}$ follows a normal distribution with a diagonal covariance matrix ${\sigma }^2{\mathbf{I}}_{NQ}$ in the main manuscript. Here we consider an alternative construction: $vec\mathbb{E}~N\left(0,{\mathbf{\Sigma }}_E\right)$ with ${\mathbf{\Sigma }}_E=\mathbf{\Sigma }\otimes {\mathbf{I}}_N$ in the form of Kronecker product, and an Inverse-Wishart prior to $\mathbf{\Sigma }$, i.e., $\mathbf{\Sigma }~IW(\mathbf{\Psi },\nu )$. Under such a prior setup, the posterior updates of $vec{\mathbf{U}}_{\left(l\right)},\mathrm{ }vec{\mathbf{V}}_{\left(s\right)},\mathrm{ }vec\mathbb{G}$, and $\mathbf{\Sigma }$ are still in closed forms.

Specifically, the full conditional posterior distribution of $vec{\mathbf{U}}_{\left(1\right)}$ is normally distributed:

$$p\left(vec{\mathbf{U}}_{\left(1\right)}\mid vec\mathbb{Y},\mathbb{X},{\mathbf{\Sigma }}_E,{\mathbf{V}}_{\left(m\right)},{\mathbf{U}}_{\left(l\right)}l\ne 1\right)~N\left({\mathit{\mu }}_U^{\prime },{\mathbf{\Sigma }}_U^{\prime }\right),$$

where

$$\begin{gathered} {\mathbf{\Sigma }}_U^{\prime }={\left({\mathbb{C}}_{(ℛ\times 𝒞)}^T{\mathbf{\Sigma }}_E^{-1}{\mathbb{C}}_{(ℛ\times 𝒞)}+{\mathbf{\Sigma }}_U^{-1}\right)}^{-1}, \\ {\mathit{\mu }}_U^{\prime }={\mathbf{\Sigma }}_U^{\prime }\left({\mathbb{C}}_{(ℛ\times 𝒞)}^T{\mathbf{\Sigma }}_E^{-1}vec\mathbb{Y}+{\mathbf{\Sigma }}_U^{-1}{\mathit{\mu }}_U\right). \end{gathered}$$

And the full conditional posterior distribution of $vec{{\mathbf{V}}_{\left(1\right)}}^T$ is normally distributed:

$$p\left(vec{\mathbf{V}}_{\left(1\right)}^T\mid vec\tilde{\mathbf{Y}},\mathbb{X},{\mathbf{\Sigma }}_E,{\mathbf{U}}_{\left(l\right)},{\mathbf{V}}_{\left(m\right)}m\ne 1\right)~N\left({\tilde{\mathit{\mu }}}_V^{\prime },{\tilde{\mathbf{\Sigma }}}_V^{\prime }\right),$$

where

$$\begin{gathered} {\tilde{\mathbf{\Sigma }}}_V^{\prime }={\left({\left({\mathbf{I}}_{Q_1}\otimes {\mathbb{D}}_{(ℛ\times 𝒞)}\right)}^T{\tilde{\mathbf{\Sigma }}}^{-1}\left({\mathbf{I}}_{Q_1}\otimes {\mathbb{D}}_{(ℛ\times 𝒞)}\right)+{\tilde{\mathbf{\Sigma }}}_V^{-1}\right)}^{-1}, \\ {\tilde{\mathit{\mu }}}_V^{\prime }={\tilde{\mathbf{\Sigma }}}_V^{\prime }\left({\left({\mathbf{I}}_{Q_1}\otimes {\mathbb{D}}_{(ℛ\times 𝒞)}\right)}^T{\tilde{\mathbf{\Sigma }}}^{-1}vec\tilde{\mathbf{Y}}+{\tilde{\mathbf{\Sigma }}}_V^{-1}{\tilde{\mathit{\mu }}}_V\right). \end{gathered}$$

and $\tilde{\mathbf{\Sigma }}$ is the covariance matrix of $vec\left({\mathbb{E}}_{\left(2\right)}^T\right)$ by rearranging elements of ${\mathbf{\Sigma }}_E$.

The derivation of the full conditional posterior of $vec{\mathbb{G}}_{(ℛ\times 𝒞)}$ is relatively more complex compared to the posteriors of $\mathbf{U}$ and $\mathbf{V}$. Starting from

$$vec\widetilde{\tilde{\mathbf{Y}}}=\left({\mathbf{I}}_S\otimes \left({\mathbb{X}}_{\left(1\right)}\mathbf{U}\right)\right)vec{\mathbb{G}}_{(ℛ\times 𝒞)}+vec\tilde{\mathbf{E}},$$

where $\widetilde{\tilde{\mathbf{E}}}={\mathbb{E}}_{\left(1\right)}\mathbf{V}{\left({\mathbf{V}}^T\mathbf{V}\right)}^{-1}$, we can write $vec\widetilde{\tilde{\mathbf{E}}}=\left(\left({\left({\mathbf{V}}^T\mathbf{V}\right)}^{-1}{\mathbf{V}}^T\right)\otimes {\mathbf{I}}_N\right)\mathrm{ }vec\mathbb{E}$. Then the covariance matrix of $vec\widetilde{\tilde{\mathbf{E}}}$ becomes

$$\widetilde{\tilde{\mathrm{\Sigma }}}=\tilde{\mathbf{V}}{\mathbf{\Sigma }}_E{\tilde{\mathbf{V}}}^T,$$

where $\tilde{\mathbf{V}}=\left(\left({\left({\mathbf{V}}^T\mathbf{V}\right)}^{-1}{\mathbf{V}}^T\right)\otimes {\mathbf{I}}_N\right)$. The full conditional posterior distribution of $vec{\mathbb{G}}_{(ℛ\times 𝒞)}$ is a normal distribution with

$${\tilde{\mathit{\mu }}}_G^{\prime }={\tilde{\mathbf{\Sigma }}}_G^{\prime }\left({\left({\mathbf{I}}_S\otimes \left({\mathbb{X}}_{\left(1\right)}\mathbf{U}\right)\right)}^T{\widetilde{\tilde{\mathrm{\Sigma }}}}^{-1}vec\widetilde{\tilde{\mathrm{Y}}}+{\tilde{\mathbf{\Sigma }}}_G^{-1}{\tilde{\mathit{\mu }}}_G\right),$$

$${\tilde{\mathbf{\Sigma }}}_G^{\prime }={\left({\left({\mathbf{I}}_S\otimes \left({\mathbb{X}}_{\left(1\right)}\mathbf{U}\right)\right)}^T{\widetilde{\tilde{\mathrm{\Sigma }}}}^{-1}\left({\mathbf{I}}_S\otimes \left({\mathbb{X}}_{\left(1\right)}\mathbf{U}\right)\right)+{\left({\tilde{\mathbf{\Sigma }}}_G\right)}^{-1}\right)}^{-1}$$

Table A.1.

Mean RPE with SD on uncorrelated-t data

RPE (SD)	BayTensor MCMC	BayTensor Fast	CP Method	OLS Method	SAS Method	HS Method
$\theta * = (4, 4, 2, 2)$, SNR=2	0.353 (0.017)	0.374 (0.097)	0.407 (0.037)	1.019 (0.029)	0.968 (0.024)	1.058 (0.137)
$\theta * = (3, 3, 3, 3)$, SNR=2	0.344 (0.004)	0.350 (0.016)	0.415 (0.032)	1.014 (0.032)	0.958 (0.025)	1.109 (0.281)
$\theta * = (4, 4, 2, 2)$, SNR=5	0.173 (0.002)	0.208 (0.125)	0.235 (0.035)	0.747 (0.028)	0.949 (0.030)	1.104 (0.227)
$\theta * = (3, 3, 3, 3)$, SNR=5	0.172 (0.001)	0.182 (0.038)	0.256 (0.038)	0.747 (0.024)	0.931 (0.035)	1.173 (0.388)

Table A.2.

Core tensor dimension recovery and number of model parameters on uncorrelated-t data

	BayTensor MCMC		BayTensor Fast
	Dimension Recovery	# Parameters (SD)	Dimension Recovery	# Parameters (SD)
$\theta * = (4, 4, 2, 2)$, SNR=2	72%	205 (19)	42%	188 (29)
$\theta * = (3, 3, 3, 3)$, SNR=2	96%	218 (6)	58%	205 (26)
$\theta * = (4, 4, 2, 2)$, SNR=5	90%	213 (10)	68%	195 (31)
$\theta * = (3, 3, 3, 3)$, SNR=5	100%	219 (0)	82%	213 (24)

Lastly, the full conditional posterior of $\mathbf{\Sigma }$ is an Inverse-Wishart distribution with $\tilde{\nu }=N+\nu$, and $\tilde{\mathbf{\Psi }}={\mathbf{S}}^T\mathbf{S}+\mathbf{\Psi }$ where $\mathbf{S}={\mathbb{Y}}_{\left(1\right)}-{\mathbb{X}}_{\left(1\right)}{\mathbb{B}}_{(𝒫\times 𝒬)}$.

A.7. Simulation results: uncorrelated-t setup

We present the detailed results under the proposed BayTensor MCMC, BayTensor Fast, and four alternative methods (including the CP method, the OLS method, the SAS method, and the HS method) to datasets under the uncorrelated-t setup.

The prediction RPE results are presented in Table A.1. The RPE results under Baytensor MCMC and BayTensor Fast are comparable in all cases. And both of them have better RPEs than the CP method followed by the SAS method, and the OLS method. The HS performs the worst in terms of yielding the highest RPEs in all cases. And when the signal to noise ratio increases from 2 to 5, the prediction accuracies are improved under all methods, except for the HS method.

Table A.2 shows the empirical probabilities of dimension recovery and the average numbers of parameters required by BayTensor MCMC and BayTensor Fast in 50 replicated experiments. In all 4 cases, BayTensor MCMC and BayTensor Fast both require a smaller number of parameters than the CP method. And in terms of recovering the dimension of the core tensor, BayTensor MCMC has higher empirical probabilities of recovering the true dimension than BayTensor Fast in all cases. And the recovering probabilities in the cases where ${\mathit{\theta }}^{\mathrm{*}}=(3,\mathrm{ }3,\mathrm{ }3,\mathrm{ }3)$ are higher than those in cases where ${\mathit{\theta }}^{\mathrm{*}}=(4,\mathrm{ }4,\mathrm{ }2,\mathrm{ }2)$ for both methods.

We also observe that the RPEs under uncorrelated-t setups are larger than those under uncorrelated-normal setups for the BayTensor methods and the CP method. This is expected because the fitting model is different from the data generation model. However, the magnitudes of RPE increase under BayTensor algorithms are small. Moreover, the dimension recovery rates and number of parameters are also similar to those under uncorrelated-normal setups. These results indicate that the proposed BayTensor algorithms have reasonably good performance when the fitting model is misspecified.

A.8. Traceplots of the estimated number of parameters associated with $\mathit{\theta }$ in simulation study

We plot the estimated number of parameters associated with ${\theta }^{\left(t\right)}$ versus iteration $t$ in the first 100 iterations of BayTensor MCMC algorithm for some randomlyselected experiments under the uncorrelated-normal simulation setup in Figure A.1, and under the correlated setup in Figure A.2.

A.9. Full simulation study results of CP method with different ranks

For each case of the simulation study, we tried the CP method with CP-rank $R$ values from 1 to 6. Note that when $R=5,\mathrm{ }6$, the total number of parameters in CP method are 230 and 276 respectively which are larger than the true number of parameters (212 for ${\mathit{\theta }}^{\mathrm{*}}=(4,\mathrm{ }4,\mathrm{ }2,\mathrm{ }2)$ cases and 219 for ${\mathit{\theta }}^{\mathrm{*}}=(3,\mathrm{ }3,\mathrm{ }3,\mathrm{ }3)$ cases). We calculate the RPE results with different $R$ values for all 50 repeated experiments under all simulation setups. And means and standard deviations of RPEs averaging over the 50 repeated experiments for all 8 simulation setups are shown in Table A.3.

Figure A.1.

The estimated number of parameters associated with ${\theta }^{\left(t\right)}$ versus iteration $t$ in the first 100 iterations of BayTensor MCMC algorithm for some randomly-selected experiments under the uncorrelated-normal simulation setup. The true number of parameters is shown by the red line. (a): SNR = 2, ${\theta }^{\mathrm{*}}=(4,\mathrm{ }4,\mathrm{ }2,\mathrm{ }2)$. (b): SNR = 2, ${\theta }^{\mathrm{*}}=(3,\mathrm{ }3,\mathrm{ }3,\mathrm{ }3)$. (c): SNR = 5, ${\theta }^{\mathrm{*}}=(4,\mathrm{ }4,\mathrm{ }2,\mathrm{ }2).$ (d): SNR = 5, ${\theta }^{\mathrm{*}}=(3,\mathrm{ }3,\mathrm{ }3,\mathrm{ }3)$.

Table A.3.

Mean RPE with SD for the CP method with different R values

RPE (SD)	$\mathit{R}\mathbf{ }=\mathbf{ }1$	$\mathit{R}\mathbf{ }=\mathbf{ }2$	$\mathit{R}\mathbf{ }=\mathbf{ }3$	$\mathit{R}\mathbf{ }=\mathbf{ }4$	$\mathit{R}\mathbf{ }=\mathbf{ }5$	$\mathit{R}\mathbf{ }=\mathbf{ }6$
Uncorrelated Data
$\theta * = (4, 4, 2, 2)$, SNR=2	0.771(0.102)	0.627(0.087)	0.529(0.095)	0.463(0.061)	0.409(0.036)	0.377(0.014)
$\theta * = (3, 3, 3, 3)$, SNR=2	0.780(0.081)	0.623(0.061)	0.535(0.048)	0.466(0.038)	0.422(0.024)	0.391(0.016)
$\theta * = (4, 4, 2, 2)$, SNR=5	0.718(0.137)	0.530(0.109)	0.392(0.072)	0.308(0.052)	0.241(0.039)	0.209(0.046)
$\theta * = (3, 3, 3, 3)$, SNR=5	0.718(0.095)	0.527(0.074)	0.414(0.060)	0.320(0.044)	0.266(0.032)	0.222(0.016)
Correlated Data
$\theta * = (4, 4, 2, 2)$, SNR=2	0.533(0.152)	0.423(0.095)	0.382(0.043)	0.371(0.021)	0.370(0.011)	0.377(0.014)
$\theta * = (3, 3, 3, 3)$, SNR=2	0.514(0.120)	0.429(0.070)	0.391(0.037)	0.374(0.019)	0.370(0.010)	0.369(0.007)
$\theta * = (4, 4, 2, 2)$, SNR=5	0.407(0.170)	0.266(0.104)	0.219(0.056)	0.198(0.034)	0.190(0.014)	0.188(0.006)
$\theta * = (3, 3, 3, 3)$, SNR=5	0.413(0.187)	0.280(0.085)	0.230(0.047)	0.205(0.026)	0.194(0.015)	0.187(0.006)

Figure A.2.

The estimated number of parameters associated with ${\theta }^{\left(t\right)}$ versus iteration $\mathrm{t}$ in the first 100 iterations of BayTensor MCMC algorithm for some randomly-selected experiments under the correlated simulation setup. The true number of parameters is shown by the red line. (a): SNR = 2, ${\theta }^{\mathrm{*}}=(4,\mathrm{ }4,\mathrm{ }2,\mathrm{ }2)$. (b): SNR = 2, ${\theta }^{\mathrm{*}}=(3,\mathrm{ }3,\mathrm{ }3,\mathrm{ }3)$. (c): SNR = 5, ${\theta }^{\mathrm{*}}=(4,\mathrm{ }4,\mathrm{ }2,\mathrm{ }2).$ (d): SNR = 5, ${\theta }^{\mathrm{*}}=(3,\mathrm{ }3,\mathrm{ }3,\mathrm{ }3)$.