Tensor canonical correlation analysis

Abstract

Canonical correlation analysis (CCA) is a multivariate analysis technique for estimating a linear relationship between two sets of measurements. Modern acquisition technologies, for example, those arising in neuroimaging and remote sensing, produce data in the form of multidimensional arrays or tensors. Classic CCA is not appropriate for dealing with tensor data due to the multidimensional structure and ultrahigh dimensionality of such modern data. In this paper, we present tensor CCA (TCCA) to discover relationships between two tensors while simultaneously preserving multidimensional structure of the tensors and utilizing substantially fewer parameters. Furthermore, we show how to employ a parsimonious covariance structure to gain additional stability and efficiency. We delineate population and sample problems for each model and propose efficient estimation algorithms with global convergence guarantees. Also we describe a probabilistic model for TCCA that enables the generation of synthetic data with desired canonical variates and correlations. Simulation studies illustrate the performance of our methods.

1 |. INTRODUCTION

Canonical correlation analysis (CCA) is a classic statistical method for identifying associations between two sets of measurements (Hotelling, 1936). Specifically, CCA identifies a pair of coefficient vectors, one for each set of measurements, such that the correlation between the corresponding linear combinations of variables from each set is maximized. By default, CCA applies when each observation consists of a pair of vector covariates. In many modern data analysis problems, however, each observation may consist more generally of a pair of multidimensional arrays or tensors. For example, in imaging genetics, to identify genetic variants that can best capture and explain phenotypic variations in brain function and structure, Stein et al. (2010) studied p = 448,293 single-nucleotide polymorphisms and q = 31,622 voxels in brain images, on n = 740 individuals. A naive approach to dealing with tensor-valued data would be to reshape tensor covariates into vectors and then apply standard CCA. There are, however, two serious drawbacks to doing so. First, structural information in tensors is discarded through vectorization. Second, the resulting vectors consist of a prohibitively large number of parameters. In the imaging genetics problem (Stein et al., 2010), vectorizing the voxel intensity measurements disregards the spatial correlation among neighbouring voxels. Moreover, applying standard CCA to vectors of single-nucleotide polymorphism covariates and vectorized brain images would require estimating nearly half a million parameters using fewer than a thousand observations.

In light of these issues, there have been extensions of CCA to handle special cases of tensor-valued data (Lee & Choi, 2007; Wang, 2010; Yan, Zheng, Zhou, & Zhao, 2012; Gang, Yong, Yan-Lei, & Jing, 2011; Lu, 2013; Wang, Yan, Sun, Zhao, & Fu, 2016). Although they have exhibited good empirical performance in some applications, there remains no clear population models underlying these sample based heuristics. To address this gap in the literature, we introduce a novel statistical model for tensor CCA (TCCA). We summarize at a high level our formulation and its contributions:

• We propose a TCCA population model that imposes the CANDECOMP/PARAFAC (CP) decomposition (Carroll & Chang, 1970; Harshman, 1970) structure on canonical tensors. This population model enforces model parsimony and enables efficient estimation.

• We propose a refinement of TCCA, which assumes a separable covariance structure (scTCCA). This refinement enables efficient estimation of large covariance matrices of tensor-valued data.

• We derive convenient representations of the covariance between linear combinations of two random tensors under the unstructured and separable covariance structured assumptions.

• We develop efficient estimation algorithms for TCCA and scTCCA, both based on block coordinate ascent, which leverage these efficient representations. Each step of both algorithms solves a substantially lower dimensional CCA problem; thus, both algorithms can be easily implemented using any standard solvers for the CCA problem. Moreover, we prove global convergence guarantees of both estimation algorithms under modest regularity conditions.

• We develop simple modifications to the TCCA and scTCCA estimation algorithms to incorporate recovery of sparse canonical correlation tensors to improve interpretability of the estimated models.

• Finally, we extend the probabilistic interpretation of CCA by Bach and Jordan (2006) to TCCA. This extension leads to a probabilistic model for generating datasets with specified canonical correlation and variates.

The remainder of the paper is organized as follows. In Section 2, we review tensor notation and basic operations used in this paper. In Sections 3 and 4, we propose our two TCCA methods: TCCA and scTCCA. In Section 5, we describe a modification of the estimation algorithms for TCCA and scTCCA for sparse models. In Section 6, we introduce the probabilistic TCCA model. In Section 7, we describe the numerical experiment results. In Section 8, we conclude and highlight directions for future work.

2 |. NOTATION AND PRELIMINARIES

We review basic operations on matrices and tensors invoked throughout this paper, adopting the terminology and notation in Kolda and Bader (2009). Throughout the paper, we use lowercase letters to indicate scalars, bold lowercase letters to indicate vectors, bold capital characters to indicate matrices, and bold calligraphic capital characters to indicate tensors. We will also use the shorthand [n] to denote an index set {1, …, n}.

Tensors can be considered generalizations of scalars, vectors, and matrices. Let $\mathcal{X}$ represent an D-dimensional tensor in ${ℝ}^{p_1\times \dots \times p_D}$. The tensor $\mathcal{X}$ has order D, its number of dimensions or modes. For example, vectors are tensors of order one and have one mode. Matrices are tensors of order two and have two modes. We denote an element of $\mathcal{X}$ by ${\text{x}}_{{\iota }_1{\iota }_2,\dots ,{\iota }_D}$, where ι_i ∈ [p_i] and i ∈ [D]. Fibres are the generalization of matrix rows and columns to higher order tensors. A fibre is defined by fixing the index of every dimension except one dimension. Mode i fibres are p_i-dimensional vectors extracted from $\mathcal{X}$ by fixing all the indices $\left({\iota }_1,\dots ,{\iota }_{l-1},{\iota }_{i+1},\dots ,{\iota }_D\right)$ except the ith one ι_i. For example, columns of a matrix are Mode 1 fibres, and rows of a matrix are Mode 2 fibres.

It is often useful to reshape a tensor into a matrix. Reordering a tensor into a matrix is referred to as matricization. The mode i matricization of a tensor $\mathcal{X}\in {ℝ}^{p_1\times \dots \times p_D}$, denoted ${\mathbf{X}}_{(i)}\in {ℝ}^{p_i\times p_{-i}}$ with $p_{-i}={\mathrm{\Pi }}_{k=1,k\ne i}^D{p}_k$, arranges the mode i fibres as the columns of the matrix X_(i). In a mode i matricization, the tensor element ${\text{x}}_{{\iota }_1,\dots ,{\iota }_D}$ is mapped to the matrix element of X_(i). with index (ι_i j), where $j=1+{\sum }_{k=1,k\ne i}^D\left({\iota }_k-1\right)J_k$ with

$$J_k=\{\begin{array}{ll}1\hfill & \text{if}k=1\text{or}\text{if}k=2\text{and}i=1.\hfill \\ \prod _{k=1,k^{\prime }\ne i}^{k-1}{p}_{k^{\prime }}\hfill & \text{otherwise.}\hfill \end{array}$$

Reordering a tensor into a vector is referred to as vectorization. We first describe vectorization of a matrix before describing vectorization of a general tensor. The vectorization of a matrix X is denoted by vec(X) and is the vector obtained by stacking the columns of X on top of each other. The vectorization of the mode i matricization of a tensor $\mathcal{X}$ in turn is denoted as x_(i) = vec(X_(i))). We then define the vectorization of a tensor $\mathcal{X}$, denoted by vec($\mathcal{X}$), as the vectorization of its Mode 1 matricization, namely, vec(X₍₁₎). When unambiguous from context, we will often denote the vectorization of a tensor $\mathcal{X}$ by its corresponding bold lowercase $\mathcal{X}$.

The inner product of two tensors of compatible dimensions $\mathcal{X}$, $\tilde{\mathcal{X}}\in {ℝ}^{p_1\times \dots \times p_D}$ is the sum of the product of their entries, namely,

$$\langle \mathcal{X},\tilde{\mathcal{X}}\rangle =\sum _{{\iota }_1=1}^{p_1}\dots \sum _{{\iota }_D=1}^{p_D}{\mathbf{x}}_{{\iota }_1,\dots ,{\iota }_D}{\tilde{\mathbf{x}}}_{{\iota }_1,\dots ,{\iota }_D}.$$

The mode i product of a tensor $\tilde{\mathcal{X}}\in {ℝ}^{p_1\times \dots \times p_D}$ with a matrix $\cup \in {ℝ}^{J\times p_d}$ is denoted by $\mathcal{X}{\times }_i\cup$ and is the tensor of size $p_1\times \dots \times p_{i-1}\times J\times p_{i+1}\times \dots \times p_D$ with elements

$${\left(\mathcal{X}{\times }_i\cup \right)}_{{\iota }_1,\dots ,{\iota }_{i-1},j{\iota }_{i+1},\dots ,{\iota }_D}=\sum _{{\iota }_i=1}^{p_i}{x}_{{\iota }_1{\iota }_2,\dots ,{\iota }_D}{U}_{j_{{\iota }_i}}.$$

Finally, we review three kinds of matrix products, as well as one definition of matrix division, that will be used throughout the paper.

• For two matrices $\mathbf{A}\in {ℝ}^{p_1\times p_2}$ and $\mathbf{B}\in {ℝ}^{q_1\times q_2}$, the Kronecker product is the p₁q₁-by-p₂q₂ matrix,

  $$\mathbf{A}\otimes \mathbf{B}=\left(\begin{array}{ccc}{a}_{11}\mathbf{B}& \dots & a_{1p_2}\mathbf{B}\\ ⋮& \ddots & ⋮\\ a_{p_1}\mathbf{B}& \dots & a_{p_1{p}_2}\mathbf{B}\end{array}\right).$$
• For two matrices $\mathbf{A}=\left(\begin{array}{lll}{a}_1\hfill & \dots \hfill & a_{p_2}\hfill \end{array}\right)\in {ℝ}^{p_1\times p_2}$ and $B=\left(\begin{array}{lll}{b}_1\hfill & \dots \hfill & b_{p_2}\hfill \end{array}\right)\in {ℝ}^{q_1\times p_2}$ that have the same number of columns p₂, the Khatri-Rao product is the p₁q₁-by-p₂ matrix,

  $$A\odot B=\left({\text{a}}_1\otimes {\text{b}}_{\text{1}}{\text{a}}_{\text{2}}\otimes {\text{b}}_2\dots {\text{a}}_{{\text{p}}_{\text{2}}}\otimes {\text{b}}_{{\text{p}}_{\text{2}}}\right).$$

  which is a column-wise Kronecker product of A and B.

• For two matrices A and B of the same size, the Hadamard product is the element-wise product A * B = {a_ijb_ij}. Because the Hadamard product commutes, we use *_i A_i to denote A₁ * … * A_m = A_π(1)* … * A_π(m) for any permutation π.

• Finally, for two matrices A and B of the same size, the Hadamard quotient is the element-wise quotient A ⊘ B = {a_ij/b_ij}.

3 |. TENSORCANONICAL CORRELATION ANALYSIS

3.1 |. Population TCCA

Let $\mathcal{X}\in {ℝ}^{p_1\times \dots \times p_{D_x}}$ and $\mathcal{Y}\in {ℝ}^{q_1\times \dots \times q_{D_{\text{y}}}}$ be two random tensors of order D_x and D_y, respectively. We denote the vectorizations of $\mathcal{X}$ and $\mathcal{Y}$ by x and y, respectively. Denote by ${\mathrm{\Sigma }}_{\text{x}}$ and ${\mathrm{\Sigma }}_{\text{y}}$ the covariances of x and y, respectively. Denote by ${\mathrm{\Sigma }}_{\text{x,y}}$ the covariance between x and y. Let $\mathcal{V}\in {ℝ}^{p_1\times \dots \times p_{D_x}}$, and $\mathcal{W}\in {ℝ}^{q_1\times \dots \times q_{D_{\text{y}}}}$ be constant tensors, and let $\rho (\mathcal{V},\mathcal{W})$ denote the correlation between the two linear combinations $\langle \mathcal{X},\mathcal{V}\rangle$ and $\langle \mathcal{Y},\mathcal{W}\rangle$ namely,

$$\rho (\mathcal{V},\mathcal{W})=\frac{\mathrm{Cov}(\langle \mathcal{X},\mathcal{V}\rangle ,\langle \mathcal{Y},\mathcal{W}\rangle )}{\sqrt{\mathrm{Var}(\langle \mathcal{X},\mathcal{V}\rangle )}\sqrt{\mathrm{Var}(\langle \mathcal{Y},\mathcal{W}\rangle )}}=\frac{{\mathbf{v}}^{\top }{\mathrm{\Sigma }}_{\mathbf{x},\mathbf{y}}\mathbf{w}}{\sqrt{{\mathbf{v}}^{\top }{\mathrm{\Sigma }}_{\mathbf{x}}\mathbf{v}}\sqrt{{\mathbf{w}}^{\top }{\mathrm{\Sigma }}_{\mathbf{y}}\mathbf{w}}}.$$ (1)

The pair $(\mathcal{V},\mathcal{W})$ that maximizes ρ are the canonical tensors, and the optimal is the canonical coefficient. Maximizing the objective in Equation (1) presents two challenges: (a) high dimensionality of optimization variables $\mathcal{V}$ and $\mathcal{W}$ and (b) the estimation of the huge covariance matrices ${\mathrm{\Sigma }}_{\text{x}}$ and ${\mathrm{\Sigma }}_{\text{y}}$ and the cross-covariance matrix ${\mathrm{\Sigma }}_{\text{x,y}}$. We will address challenge (b) by imposing a separable covariance structure in Section 4.

To address challenge (a), we impose the parsimonious CANDECOMP/PARAFAC (CP), or Kruskal, representation on the canonical tensors. The CP representation generalizes the idea of representing a matrix as the sum of Rank 1 matrices to representing a tensor as the sum of Rank 1 tensors. An order-D tensor $\mathcal{X}\in {ℝ}^{p_1\times \dots \times p_D}$ is Rank 1 if it can be expressed as the outer product of D vectors a⁽¹⁾, a⁽²⁾, …, a^(D), namely, $\mathcal{X}={\mathbf{a}}^{(1)}\circ {\mathbf{a}}^{(2)}\circ \dots \circ {\mathbf{a}}^{(D)}$, where the binary operator o denotes the vector outer product. Thus, the (ι₁, ι₂,…, ι_D )th element of $\mathcal{X}$ is ${\mathbf{x}}_{{\iota }_1{\iota }_2\dots {\iota }_{\text{D}}}={\text{a}}_{{\iota }_1}^{(1)}{\text{a}}_{{\iota }_2}^{(2)}\dots {\text{a}}_{{\iota }_0}^{(\text{D})}$. A rank-R tensor can be written as the sum of R Rank 1 tensors, namely,

$$\mathcal{X}=\left[A_1,\dots ,A_D\right]=\sum _{r=1}^R{a}_r^{(1)}\circ \dots \circ a_r^{(D)},$$

where $A_i=\left(\begin{array}{llll}{a}_1^{(1)}\hfill & a_2^{(i)}\hfill & \dots \hfill & a_R^{(i)}\hfill \end{array}\right)\in {ℝ}^{p_i\times R}$ denotes the mode i factor matrix. We use the Kruskal notation $[\dots ]$ to concisely summarize the sum.

Thus, instead of searching over the space of all order-D_x and order-D_y tensor pairs $(\mathcal{V},\mathcal{W})$, we limit our search to tensors of rank-R_x and rank-R_y,

$$\mathcal{V}=\left[{\text{V}}_1,\dots ,{\text{V}}_{{\text{D}}_{\text{x}}}\right],{\text{V}}_{\text{i}}\in {ℝ}^{\text{P}\times {\text{R}}_x}\text{,i}\in \left[{\text{D}}_{\text{x}}\right]\text{,}\mathcal{W}=\left[{\text{W}}_1,\dots ,{\text{W}}_{{\text{D}}_y}\right],{\text{W}}_{\text{j}}\in {ℝ}^{{\text{q}}_{\text{j}}}{}^{\times {\text{R}}_y},\text{j}\in \left[{\text{D}}_{\text{y}}\right].$$ (2)

As we will see later, this parameterization makes progress towards alleviating the burden of estimating a huge covariance matrix. Note that

$$\langle \mathcal{X},\mathcal{V}\rangle =\langle \mathcal{X},\sum _{r=1}^{R_x}{\mathbf{w}}_r^{(1)}\circ \dots \circ {\mathbf{w}}_r^{\left(D_x\right)}\rangle =\sum _{r=1}^{R_x}x{\times }_1{\mathbf{w}}_r^{(1)}{\times }_2\dots {\times }_{D_x}{\mathbf{w}}_r^{\left(D_x\right)}$$

$$\langle \mathcal{Y},\mathcal{W}\rangle =\langle \mathcal{Y},\sum _{r=1}^{R_Y}{\mathbf{w}}_r^{(1)}\circ \dots \circ {\mathbf{w}}_r^{\left(D_y\right)}\rangle =\sum _{r=1}^{R_y}\mathcal{Y}{\times }_1{\mathbf{w}}_r^{(1)}{\times }_2\dots {\times }_{D_y}{\mathbf{w}}_r^{\left(D_y\right)}.$$

Thus, we seek to maximize the correlation between a rank-R_x multilinear form in $\mathcal{X}$ and a rank-R_y multilinear form in $\mathcal{Y}$. Multiway information is preserved, and the dimensionality is reduced from an exponential number of parameters ${\mathrm{\Pi }}_{i=1}^{D_x}{p}_i+{\mathrm{\Pi }}_{j=1}^{D_y}{q}_j$ to a linear number of parameters $R_x{\sum }_{i=1}^{D_x}{p}_i+R_y{\sum }_{j=1}^{D_y}{q}_j$. Note that the ranks (R_x, R_y) here are not the number of canonical tensor pairs being sought. In this paper, we focus on obtaining only the top canonical tensor pair $(\mathcal{V},\mathcal{W})$, which have ranks R_x and R_y.

The following representations of $\text{Var}(\langle \mathcal{X},\mathcal{Y}\rangle )$, $\text{Var}(\langle \mathcal{Y},\mathcal{W}\rangle )$, and $\mathrm{Cov}(\langle \mathcal{X},\mathcal{Y}\rangle ,\langle \mathcal{Y},\mathcal{N}\rangle )$ in terms of a CP decomposition are keys to our estimation algorithms. Its proof is in the Supporting Information.

Proposition 1

Let $\mathcal{X}\in {ℝ}^{p_1\times \dots \times p_{D_x}}$ and $\mathcal{Y}\in {ℝ}^{q_1\times \dots \times q_{D_{\text{y}}}}$ be two random tensors and $\mathcal{V}=\left[{\text{V}}_1,\dots ,{\text{V}}_{{\text{D}}_{\text{x}}}\right]$ and $\mathcal{W}=\left[{\text{W}}_1,\dots ,{\text{W}}_{{\text{D}}_{\text{y}}}\right]$ be two constant tensors of the same size as $\mathcal{X}$and $\mathcal{Y}$ respectively. Define

$${\mathbf{V}}_{(-1)}=\left[{\mathbf{V}}_{D_x}\odot \dots \odot {\mathbf{V}}_{i+1}\odot {\mathbf{V}}_{i-1}\odot \dots \odot {\mathbf{V}}_1\right]\otimes {\mathbf{I}}_{p_i},$$ (3)

$${\mathbf{W}}_{(-j)}=\left[{\mathbf{W}}_{D_x}\odot \dots \odot {\mathbf{W}}_{j+1}\odot {\mathbf{W}}_{j-1}\odot \dots \odot {\mathbf{W}}_1\right]\otimes {\mathbf{I}}_{q_j},$$ (4)

and let ${\mathrm{\Sigma }}_{x_{(j)}}$ denote the covariance of x_(i) = vec(X_(i)). Define ${\mathrm{\Sigma }}_{{\mathbf{y}}_{(j)}}$ and ${\mathrm{\Sigma }}_{{\mathbf{x}}_0,{\mathbf{y}}_{(j}{}_{)}}$ analogously. Then

$$\mathrm{Var}(\langle \mathcal{X},\mathcal{V}\rangle )={\mathbf{v}}_i^{\top }{\mathbf{V}}_{(-i)}^{\top }{\mathrm{\Sigma }}_{{\mathbf{x}}_{(i)}}{\mathbf{V}}_{(-i)}{\mathbf{v}}_i$$

$$\text{Var}(\langle \mathcal{Y},\mathcal{W}\rangle )={\mathbf{w}}_j^{\top }{\mathbf{W}}_{(-j)}^{\top }{\mathrm{\Sigma }}_{y_{(j)}}{\mathbf{W}}_{(-j)}{\mathbf{w}}_j$$

$$Cov(\langle \mathcal{X},\mathcal{V}\rangle ,\langle \mathcal{Y},\mathcal{W}\rangle )={\mathbf{v}}_i^{\top }{\mathbf{V}}_{(-i)}^{\top }{\mathrm{\Sigma }}_{{\mathbf{x}}_{(j)},{\mathbf{y}}_{(j)}}{\mathbf{W}}_{(-j)}{\mathbf{w}}_j$$

for any i ∈ [D_x] and j ∈ [D_y], where v_i = vec(V_i) and Wj = vec(W_j).

3.2 |. Sample TCCA

Suppose we observe N pairs of i.i.d. tensor data $\left({\mathcal{X}}_n,{\mathcal{Y}}_n\right)$, and we estimate $\mathcal{V}$ and $\mathcal{W}$ by solving the optimization problem

$$\text{maximize}\widehat{\rho }(\mathcal{V},\mathcal{W})\equiv \frac{{\mathbf{v}}^{\top }{\widehat{\mathrm{\Sigma }}}_{\mathbf{x},\mathbf{y}}\mathbf{W}}{\sqrt{{\mathbf{v}}^{\top }{\widehat{\mathrm{\Sigma }}}_{\mathbf{x}}\mathbf{v}}\sqrt{{\mathbf{w}}^{\top }{\widehat{\mathrm{\Sigma }}}_{\mathbf{y}}\mathbf{w}}}.$$ (5)

where ${\widehat{\mathrm{\Sigma }}}_{\mathbf{x}}$, ${\widehat{\mathrm{\Sigma }}}_{\mathbf{y}}$ and ${\widehat{\mathrm{\Sigma }}}_{\mathbf{x},\mathbf{y}}$ are sample estimates of the corresponding covariances. Recall that CCA models can be estimated numerically by computing the solution to the following generalized eigenvalue problem:

$$\left(\begin{array}{cc}0& {\widehat{\mathrm{\Sigma }}}_{x,y}\\ {\widehat{\mathrm{\Sigma }}}_{y,x}& 0\end{array}\right)\left(\begin{array}{c}v\\ w\end{array}\right)=\rho \left(\begin{array}{cc}{\widehat{\mathrm{\Sigma }}}_x& 0\\ 0& {\widehat{\mathrm{\Sigma }}}_y\end{array}\right)\left(\begin{array}{c}v\\ w\end{array}\right).$$

This problem is guaranteed to have a solution if and only/ if the covariance matrices ${\widehat{\mathrm{\Sigma }}}_{\mathbf{x}}$ and ${\widehat{\mathrm{\Sigma }}}_{\mathbf{y}}$ are nonsingular. In practice, the sample size N is smaller than the size of ${\widehat{\mathrm{\Sigma }}}_{\mathbf{x}}$ and ${\widehat{\mathrm{\Sigma }}}_{\mathbf{y}}$, $\left({\mathrm{\Pi }}_{i=1}^{{\text{D}}_x}{p}_i\right)\left(\times {\mathrm{\Pi }}_{i=1}^{{\text{D}}_x}{p}_i\right),\left({\mathrm{\Pi }}_{j=1}^{{\text{D}}_x}{q}_j\right)\times \left({\mathrm{\Pi }}_{j=1}^{{\text{D}}_y}{q}_j\right)$, respectively. Therefore, the sample covariance matrices ${\widehat{\mathrm{\Sigma }}}_{\mathbf{x}}$ and ${\widehat{\mathrm{\Sigma }}}_{\mathbf{y}}$ are singular, and a solution cannot be obtained. As a remedy, several regularized estimation methods for obtaining nonsingular sample covariance matrices (Vinod, 1976; Ledoit & Wolf, 2004; González, Déjean, Martin, & Baccini, 2008; Ledoit & Wolf, 2012; Cai & Yuan, 2012; Bickel & Levina, 2008; 2008; Kubokawa et al., 2013; Srivastava & Reid, 2012) can be used.

If we take $\text{Var}(\langle \mathcal{X},\mathcal{V}\rangle )$, $\text{Var}(\langle \mathcal{Y},\mathcal{W}\rangle )$ and $\mathrm{Cov}(\langle \mathcal{X},\mathcal{V}\rangle ,\langle \mathcal{Y},\mathcal{W}\rangle )$ to be ${\mathbf{v}}^T{\widehat{\mathrm{\Sigma }}}_{\text{x}}\mathbf{v}$, ${\mathbf{w}}^T{\widehat{\mathrm{\Sigma }}}_{\text{y}}\mathbf{w}$, and ${\mathbf{v}}^{\top }{\widehat{\mathrm{\Sigma }}}_{\text{x},\text{y}}\mathbf{w}$ respectively, then Proposition 1 suggests a block coordinate ascent algorithm where we update the factor matrices in pairs (V_i, W_j), for different combinations of i ∈ [D_x] and j ∈ [D_y]. To update the pair (V_i, W_j), we solve the following problem:

$$\text{maximize}{\mathbf{v}}_i^{\top }{\mathbf{V}}_{(-i)}^{\top }{\widehat{\mathrm{\Sigma }}}_{{\mathbf{x}}_{(i)},{\mathbf{y}}_{(j)}}{\mathbf{W}}_{(-j)}{\mathbf{w}}_j\text{subject}\text{to}{\mathbf{v}}_i^{\top }{\mathbf{V}}_{(-i)}^{\top }{\widehat{\mathrm{\Sigma }}}_{{\mathbf{x}}_{(i)}}{\mathbf{V}}_{(-i)}{\mathbf{v}}_i=1,{\mathbf{w}}_j^{\top }{\mathbf{W}}_{(-j)}^{\top }{\widehat{\mathrm{\Sigma }}}_{{\mathbf{y}}_{(j)}}{\mathbf{W}}_{(-j)}{\mathbf{w}}_j=1,$$ (6)

which is a substantially smaller optimization problem over p_iR_x + q_jR_y variables compared with any alternative “all-at-once” strategies to iteratively optimize over all $\left({\mathrm{\Pi }}_{i=1}^{D_{\text{x}}}{p}_i\right)\left(\times {\mathrm{\Pi }}_{i=1}^{D_{\text{x}}}{p}_i\right)+\left({\mathrm{\Pi }}_{j=1}^{D_{\text{y}}}{q}_j\right)\times \left({\mathrm{\Pi }}_{j=1}^{D_{\text{y}}}{q}_i\right)$ parameters simultaneously. Problem (6) includes ${\widehat{\mathrm{\Sigma }}}_{{\mathbf{x}}_{(i)}}$, a permuted version of ${\widehat{\mathrm{\Sigma }}}_{\mathbf{x}}$ with size $\left({\mathrm{\Pi }}_{i=1}^{D_{\text{x}}}{p}_i\right)\left(\times {\mathrm{\Pi }}_{i=1}^{D_{\text{x}}}{p}_i\right)$. However, we work on the “compressed” covariance matrix

$${\mathbf{V}}_{(-i)}^{\top }{\widehat{\mathrm{\Sigma }}}_{{\mathbf{x}}_{(i)}}{\mathbf{V}}_{(-i)}\in {ℝ}^{p_i{R}_x\times p_i{R}_x},$$

which is likely to be full rank when N ≥ p_iR_x instead of a singular matrix ${\widehat{\mathrm{\Sigma }}}_{{\mathbf{x}}_{(i)}}$. This approach enables us to solve the generalized eigenvalue problem with matrices that are (p_iR_x) + (q_jR_y)-by-(p_iR_x) + (q_jR_y). Algorithm 1 summarizes the estimation procedure that comes with the following convergence guarantees. The proof is in the Supporting Information.

Proposition 2

If the matrices ${\widehat{\mathrm{\Sigma }}}_{\mathbf{x}}$ and ${\widehat{\mathrm{\Sigma }}}_y$ are nonsingular, then the limit Points of the iterate sequence generated by Algorithm 1 are canonical tensors of the sample TCCA Problem.

Algorithm 1.

TCCA, with (i) assumptions on CP structure on canonical tensors $(\mathcal{V},\mathcal{W})$ and (ii) no additional assumptions on the structures of the covariances $\mathrm{Var}(\mathrm{vec}(\mathcal{X}))$ and $\mathrm{Var}(\mathrm{vec}(\mathcal{Y}))$

Initialize $v_i^{(0)}$ and $w_j^{(0)}$, for i ∈ [Dx] and j ∈ [Dy]

t ← 0

repeat

Select (i, j) ∈ [Dx] × [Dy]

${\mathbf{V}}_{(-i)}^{(t)}\leftarrow \left[{\mathbf{v}}_{D_x}^{(t)}\odot \cdots \odot {\mathbf{V}}_{i+1}^{(t)}\odot {\mathbf{V}}_{i-1}^{(t)}\odot \cdots \odot {\mathbf{V}}_1^{(t)}\right]\otimes {\mathbf{I}}_{p_i}$

${\mathbf{W}}_{(-j)}^{(t)}\leftarrow \left[{\mathbf{W}}_{D_y}^{(t)}\odot \cdots \odot {\mathbf{W}}_{j+1}^{(t)}\odot {\mathbf{W}}_{j-1}^{(t)}\odot \cdots \odot {\mathbf{W}}_1^{(t)}\right]\otimes {\mathbf{I}}_{q_j}$

${\mathbf{C}}_{\mathbf{x}}^{(\text{t})}\leftarrow {\mathbf{V}}_{(-i)}^{(t)T}{\widehat{\mathbf{\Sigma }}}_{{\mathbf{x}}_{(i)}}{\mathbf{V}}_{(-i)}^{(t)}$

${\mathbf{C}}_{\mathbf{y}}^{(\text{t})}\leftarrow {\mathbf{W}}_{(-j)}^{(t)\top }{\widehat{\mathbf{\Sigma }}}_{y_{(j)}}{\mathbf{W}}_{(-j)}^{(t)}$

${\mathbf{C}}_{\mathbf{x},\mathbf{y}}^{(t)}\leftarrow {\mathbf{V}}_{(-i)}^{(t)T}{\widehat{\mathbf{\Sigma }}}_{{\mathbf{x}}_{(i)}}{\mathbf{y}}_{(j)}{\mathbf{W}}_{(-j)}^{(t)}$

Generalized eigen-decomposition: $\left(\begin{array}{cc}0& C_{x,y}^{(t)}\\ C_{xy}^{(t)T}& 0\end{array}\right)\left(\begin{array}{c}{w}_i^{(t+1)}\\ w_j^{(t+1)}\end{array}\right)={\rho }^{(t+1)}\left(\begin{array}{cc}{C}_x^{(t)}& 0\\ 0& C_y^{(t)}\end{array}\right)\left(\begin{array}{c}{w}_i^{(t+1)}\\ w_j^{(t+1)}\end{array}\right)$

t ← t + l

until ρ(t) converges

4 |. TCCA WITH SEPARABLE COVARIANCE STRUCTURE

4.1 |. Population TCCA with separable covariances

Hoff (2011) proposed the array normal distribution with separable covariance structure. Separable marginal covariances are defined as

$${\mathrm{\Sigma }}_{\text{x}}=\mathrm{Var}(\mathbf{x})={\mathrm{\Sigma }}_{\text{x},{\text{D}}_{\text{x}}}\otimes \dots \otimes {\mathrm{\Sigma }}_{\text{x},1}\text{and}{\mathrm{\Sigma }}_{\text{y}}=\mathrm{Var}(\text{y})={\mathrm{\Sigma }}_{\text{y},{\text{D}}_{\text{y}}}\otimes \dots \otimes {\mathrm{\Sigma }}_{\text{y},1},$$ (7)

where ${\mathrm{\Sigma }}_{\text{x},j}\in {ℝ}^{p_i\times p_i}$ for i ∈ [D_x] and ${\mathrm{\Sigma }}_{\text{y},j}\in {ℝ}^{q_j\times q_j}$ for j ∈ [D_y]. Then the overall covariance of population model is

$$\mathrm{Var}\left(\begin{array}{l}\mathbf{X}\hfill \\ \mathbf{y}\hfill \end{array}\right)=\left(\begin{array}{cc}{\mathrm{\Sigma }}_{\mathbf{x},{\mathbf{D}}_x}\otimes \dots \otimes {\mathrm{\Sigma }}_{\mathbf{x},1}& {\mathrm{\Sigma }}_{\mathbf{x},\mathbf{y}}\\ {\mathrm{\Sigma }}_{y_{\mathbf{x}}}& {\mathrm{\Sigma }}_{\mathbf{y},{\mathbf{D}}_y}\otimes \dots \otimes {\mathrm{\Sigma }}_{\mathbf{y},1}\end{array}\right).$$ (8)

Intuitively, ${\mathrm{\Sigma }}_{\mathbf{x},i}$ summarizes the covariance along the mode i fibres of tensor $\mathcal{X}$ and ${\mathrm{\Sigma }}_{y,j}$ summarizes the covariance along the mode j fibres of tensor $\mathcal{Y}$. The following result shows a representation of $\text{Var}(\langle \mathcal{X},\mathcal{V}\rangle )$ and $\text{Var}(\langle \mathcal{Y},\mathcal{W}\rangle )$ in presence of separable covariance structure (7) that we will leverage in our estimation algorithm.

Proposition 3

Let $\mathcal{X}\in {ℝ}^{p_1\times \dots \times p_{D_x}}$ and $\mathcal{Y}\in {ℝ}^{q_1\times \dots \times q_{D_{\text{y}}}}$ be two random tensors admitting the separable covariance structure (7) and $\mathcal{V}=\left[v_1,\dots ,{\mathbf{v}}_{D_{\text{x}}}\right]$ and $\mathcal{W}=\left[{\mathbf{W}}_1,\dots ,{\mathbf{W}}_{D_y}\right]$ be two constant tensors. Define

$${\text{H}}_{\text{x},-i}={*}_{i^{\prime }\ne i}\left({\text{V}}_{i^{\prime }}^{\top }{\mathrm{\Sigma }}_{\text{x},i^{\prime }}{\text{V}}_{i^{\prime }}\right)\text{and}{\text{H}}_{\text{y},-j}={*}_{j^{\prime }\ne j}\left({\text{W}}_{j^{\prime }}^{\top }{\mathrm{\Sigma }}_{y,j^{\prime }}{\text{W}}_{j^{\prime }}\right).$$

Then

$$\mathrm{Var}(\langle \mathcal{X},\mathcal{V}\rangle )={\mathbf{v}}_i^{\top }\left({\mathbf{H}}_{\mathbf{x},-i}\otimes {\mathrm{\Sigma }}_{\mathbf{x},i}\right){\mathbf{v}}_i\text{and}\mathrm{Var}(\langle \mathcal{Y},\mathcal{W}\rangle )={\mathbf{w}}_j^{\top }\left({\mathbf{H}}_{\mathbf{y},-j}\otimes {\mathrm{\Sigma }}_{{\mathbf{y}}_j}\right){\mathbf{w}}_j,$$

for any i ∈ [D_x] and j ∈ [D_y], where v_i = vec(V_i) and w_j = vec(W_j).

With the separable covariance structure and the CP structure on canonical tensors $\left(\mathcal{V},\mathcal{W}\right)$, the objective function of TCCA population model (1) greatly simplifies. Note that the separable covariance structure may not hold for real data, in which case the covariance estimates are biased. Despite this drawback, this parsimonious structure is worth considering due to the stability that it can impart by reducing estimation variance.

4.2 |. Sample TCCA with separable covariances

Given data $\left({\mathcal{X}}_{\mathrm{n}},{\mathcal{Y}}_{\text{n}}\right)$, n ∈ [N], the goal is to maximize the sample canonical correlation (5) under assumptions that (a) $\mathcal{V}$ and $\mathcal{W}$ have the CP decomposition structure and (b) $\mathcal{X}$ and $\mathcal{Y}$ admit the separable covariance structure. We follow the same strategy as sample TCCA in Section 3.2, updating parameters in pairs of factor matrices (V_i, W_j). To update the pair (V_i, W_j), we solve the subproblem

$$\text{maximize}{\mathbf{v}}_{\text{i}}^{\top }{\mathbf{V}}_{(-\text{i})}^{\top }{\widehat{\mathrm{\Sigma }}}_{{\mathbf{x}}_{(\text{i})},{\mathbf{y}}_{(\text{j})}}{\mathbf{W}}_{(-\text{j})}{\mathbf{w}}_{\text{j}},\text{subject}\text{to}{\mathbf{v}}_{\mathbf{i}}^{\top }\left({\mathbf{H}}_{\mathbf{x},-\mathbf{i}}\otimes {\widehat{\mathrm{\Sigma }}}_{\mathbf{x},\mathbf{i}}\right){\mathbf{v}}_{\mathbf{i}}=1,{\mathbf{w}}_{\mathbf{j}}^{\top }\left({\mathbf{H}}_{\mathbf{y},-\mathbf{j}}\otimes {\widehat{\mathrm{\Sigma }}}_{\mathbf{y},\mathbf{j}}\right){\mathbf{w}}_{\mathbf{j}}=1.$$

where H_x,_−i and H_y,−j, defined in Proposition 3, are evaluated at current iterates V_i′ where i′ ≠ i, and W_j′, where j′ ≠ j.

By assuming the separable covariance structure, Proposition 3 enables us to greatly simplify the variance calculations for $\mathrm{Var}(\langle \mathcal{X},\mathcal{V}\rangle )$ and $\mathrm{Var}(\langle \mathcal{Y},\mathcal{W}\rangle )$. Note that the calculation of ${\text{V}}_{(-i)}^{\top }{\widehat{\mathrm{\Sigma }}}_{{\text{x}}_0}{\text{V}}_{(-i)}$ and ${\text{W}}_{(-j)}^{\top }{\widehat{\mathrm{\Sigma }}}_{{\text{y}}_{(j)}}{\text{W}}_{(-j)}$ in the subproblem of the sample TCCA costs ${\left(R_{\text{x}}{\prod }_{i=1}^{D_{\text{x}}}{p}_i\right)}^2+{\left(R_{\text{y}}{\prod }_{j=1}^{D_{\text{y}}}{q}_j\right)}^2$ flops. In contrast, the calculation of matrices ${\text{H}}_{\text{x},-i}\otimes {\widehat{\mathrm{\Sigma }}}_{\text{x},j}$ and ${\text{H}}_{\text{y},-j}\otimes {\widehat{\mathrm{\Sigma }}}_{\text{y},\text{j}}$ in the subproblem of the sample TCCA with the separable covariance structure only costs (R_xp_i)² + (R_yq_j)² flops. Algorithm 2 summarizes the estimation procedure under the separable covariance assumption (7). Like Algorithm 1, Algorithm 2 also comes with convergence guarantees. The proof is in the Supporting Information.

Proposition 4

If the matrices ${\widehat{\mathrm{\Sigma }}}_{\mathbf{x}}$ and ${\widehat{\mathrm{\Sigma }}}_y$ are nonsingular, then the limit points of the iterate sequence generated by Algorithm 2 are canonical tensors of the sample TCCA problem with separable covariances.

Algorithm 2.

TCCA for two tensors of modes D_x and D_y, respectively, assuming (i) CP structure on canonical correlation tensors $\left(\mathcal{V},\mathcal{W}\right)$ and (ii) separable covariances $\mathrm{Var}(\mathrm{vec}(\mathcal{X}))$ and $\mathrm{Var}(\mathrm{vec}(\mathcal{Y}))$

Initialize ${\mathbf{V}}_i^{(0)},i\in \left[D_x\right]$

${\mathbf{W}}_j^{(0)},j\in \left[D_y\right]$

${\mathbf{H}}_{\mathbf{x}}^{(0)}\leftarrow {*}_i\left({\mathbf{V}}_i^{(0)}{}^T{\widehat{\mathbf{\Sigma }}}_{\mathbf{x},i}{\mathbf{V}}_i^{(0)}\right)$

${\mathbf{H}}_{\mathbf{y}}^{(0)}\leftarrow {*}_i\left({\mathbf{W}}_j^{(0)}{}^{\top }{\widehat{\mathrm{\Sigma }}}_{y,j}{\mathbf{W}}_j^{(0)}\right)$

t ← 0

repeat

Select (i, j) ∈ [Dx] × [Dy]

${\mathbf{H}}_{\mathbf{x},-i}^{(t)}\leftarrow {\mathbf{H}}_{\mathbf{x}}^{(t)}\tau \left({\mathbf{V}}_i^{(t)}{}^{\text{T}}{\widehat{\mathbf{\Sigma }}}_{\mathbf{x},i}{\mathbf{V}}_i^{(t)}\right)$

${\mathbf{C}}_{\mathbf{x}}^{(t)}\leftarrow {\mathbf{H}}_{\mathbf{x},-i}^{(t)}\otimes {\widehat{\mathbf{\Sigma }}}_{\mathbf{x},i}$

${\mathbf{H}}_{\mathbf{y},-j}^{(t)}\leftarrow {\mathbf{H}}_{\mathbf{y}}^{(t)}\tau \left({\mathbf{W}}_j^{(t)}{}^T{\widehat{\mathrm{\Sigma }}}_{\mathbf{y},j}{\mathbf{W}}_j^{(t)}\right)$

${\mathbf{C}}_{\mathbf{y}}^{(t)}\leftarrow {\mathbf{H}}_{\mathbf{y},-j}^{(t)}\otimes {\widehat{\mathrm{\Sigma }}}_{\mathbf{y},j}$

${\mathbf{V}}_{(-i)}^{(t)}\leftarrow {\mathbf{V}}_{D_{\text{x}}}^{(t)}\odot \cdots \odot {\mathbf{V}}_{i+1}^{(t)}\odot {\mathbf{V}}_{i-1}^{(t)}\odot \cdots \odot {\mathbf{V}}_1^{(t)}$

${\mathbf{W}}_{(-j)}^{(t)}\leftarrow {\mathbf{W}}_{D_{\text{y}}}^{(t)}\odot \cdots \odot {\mathbf{W}}_{j+1}^{(t)}\odot {\mathbf{W}}_{j-1}^{(t)}\odot \cdots \odot {\mathbf{W}}_1^{(t)}$

${\mathbf{C}}_{\mathbf{x}y}^{(\text{t})}\leftarrow \left({\mathbf{V}}_{-i}^{{(\text{t})}^T}\otimes {\mathbf{I}}_{p_i}\right){\widehat{\mathbf{\Sigma }}}_{{\mathbf{x}}_{(i)},{\mathbf{y}}_{(j)}}\left({\mathbf{W}}_{-j}^{{(\text{t})}^{\text{T}}}\otimes {\mathbf{I}}_{q_j}\right)$

Solve following generalized eigenvalue decomposition, $\left[\begin{array}{cc}\mathbf{0}& {\text{C}}_{\text{xy}}^{(\text{t})}\\ {\text{C}}_{\text{xy}}^{(\text{t})}& \mathbf{0}\end{array}\right]\left[\begin{array}{c}{\text{v}}_i^{(\text{t}+1)}\\ {\text{W}}_j^{(\text{t}+1)}\end{array}\right]={\rho }^{(\text{t}+1)}\left[\begin{array}{cc}{\text{C}}_{\text{x}}^{(\text{t})}& \mathbf{0}\\ \mathbf{0}& {\text{C}}_{\text{y}}^{(\text{t})}\end{array}\right]\left[\begin{array}{c}{\text{V}}_i^{(\text{t}+1)}\\ {\text{W}}_j^{(\text{t}+1)}\end{array}\right]$

${\mathbf{H}}_{\mathbf{x}}^{(t+1)}\leftarrow {\mathbf{H}}_{\mathbf{x},-i}^{(t)}*\left({\mathbf{V}}_i^{{(t+1)}^T}{\widehat{\mathbf{\Sigma }}}_{\mathbf{x},i}{\mathbf{V}}_i^{(t+1)}\right)$

${\mathbf{H}}_{\mathbf{y}}^{(t+1)}\leftarrow {\mathbf{H}}_{\mathbf{y},-j}^{(t)}*\left({\mathbf{W}}_i^{{(t+1)}^T}{\widehat{\mathbf{\Sigma }}}_{\mathbf{y},j}{\mathbf{W}}_j^{(t+1)}\right)$

t ← t + 1

until ρ(t) converges

4.2.1 |. Estimation of separable covariance matrices

Algorithm 2 relies on the sample estimate of separable covariances ${\widehat{\mathrm{\Sigma }}}_{\mathbf{x},i}$ and ${\widehat{\mathrm{\Sigma }}}_{\text{y},j}$, and the unstructured cross-covariance ${\widehat{\mathrm{\Sigma }}}_{\mathbf{x},\text{y}}$. The following lemma will be useful in computing a consistent estimator for covariance matrices with the separable structure.

Lemma 1

If a random tensor $\mathcal{X}\in {ℝ}^{p_1\times \dots \times p_{D_x}}$ has mean zero and separable covariance $\text{Var}(\mathbf{x})={\mathrm{\Sigma }}_{\mathbf{x},D_{\text{x}}}\otimes \dots \otimes {\mathrm{\Sigma }}_{\mathbf{x},1}$, then

$$\mathbb{E}\left({\mathbf{X}}_{(i)}{\mathbf{X}}_{(i)}^{\top }\right)=\left(\prod _{i^{\prime }\ne i}tr\left({\mathrm{\Sigma }}_{\mathbf{x},i^{\prime }}\right)\right){\mathrm{\Sigma }}_{\mathbf{x},i}\mathrm{and}\mathbb{E}\left(\Vert \mathbf{x}{\Vert }_2^2\right)=\prod _{i=1}^{D_{\text{x}}}tr\left({\mathrm{\Sigma }}_{\mathbf{x},i}\right).$$

Proof. For the first identity, see Hoff, 2011 (2011, Proposition 2.1). For the second identity, $\mathbb{E}\left(\Vert \mathbf{x}{\Vert }_2^2\right)=\mathrm{tr}(\mathrm{Var}(\mathbf{x}))=\mathrm{tr}\left({\mathrm{\Sigma }}_{\mathbf{x},{\mathbf{D}}_x}\otimes \dots \otimes {\mathrm{\Sigma }}_{\mathbf{x},1}\right)={\mathrm{\Pi }}_i\mathrm{tr}\left({\mathrm{\Sigma }}_{\mathbf{x},1}\right)$. □

Given N i.i.d. observations (x_n, y_n), the estimators ${\widehat{r}}_x=\frac{1}{N}\sum _{n=1}^N{\Vert {\mathbf{x}}_n-\overline{\mathbf{x}}\Vert }_2^2$ and ${\widehat{r}}_y=\frac{1}{N}\sum _{n=1}^N{\Vert {\mathbf{y}}_n-\overline{\mathbf{y}}\Vert }_2^2$ consistently estimate ${\mathrm{\Pi }}_{i=1}^{D_{\text{x}}}\mathrm{tr}\left({\mathrm{\Sigma }}_{\text{x},i}\right)$ and ${\prod }_{j=1}^{D_{\text{y}}}\mathrm{tr}\left({\mathrm{\Sigma }}_{\text{y},j}\right)$, respectively, where $\overline{x}$ and $\overline{y}$ are the sample means of the vectorized tensors x_n and y_n. We propose the following covariance estimators:

$${\widehat{\mathrm{\Sigma }}}_{\text{x},i}=\frac{1}{N{r}_{\text{x}}^{\left(D_{\text{x}}-1\right)/D_{\text{x}}}}\sum _{n=1}^N\left(x_{n(i)}-{\overline{x}}_{(i)}\right){\left(x_{n(i)}-{\overline{x}}_{(i)}\right)}^{\top },{\widehat{\mathrm{\Sigma }}}_{y,j}=\frac{1}{N{\widehat{r}}_y^{\left(D_y-1\right)/D_y}}\sum _{n=1}^N\left(y_{n(j)}-{\overline{y}}_{(j)}\right){\left(y_{n(j)}-{\overline{y}}_{(j)}\right)}^T,$$

$${\widehat{\mathrm{\Sigma }}}_{\text{x},\text{y}}=\frac{1}{N}\sum _{n=1}^N\left({\text{x}}_n-\overline{x}\right){\left({\text{y}}_n-\overline{y}\right)}^T,$$

where i ∈ [D_x] and j ∈ [D_y]. The vectors x_n(i) and ${\overline{x}}_{\text{(i)}}$ denote the mode i vectorization of the nth observation and the sample mean of the mode i vectorized tensors x_n, respectively. The vectors y_n(j) and ${\overline{y}}_{\text{(j)}}$ denote analogous vectorizations.

Unfortunately, the separable covariance structure (8) is not identifiable in the individual ${\mathrm{\Sigma }}_{\text{x},i}$ due to scaling indeterminacy. Therefore, ${\mathrm{\Sigma }}_{\text{x},i}$ cannot be consistently estimated. Note, however, that we do not need to consistently estimate the individual ${\mathrm{\Sigma }}_{\text{x},i}$ in order to consistently estimate their Kronecker product. To see this, note that by Slutsky’s theorem,

$${\widehat{\mathrm{\Sigma }}}_{\text{x},D_{\text{x}}}\otimes \dots \otimes {\widehat{\mathrm{\Sigma }}}_{\text{x},1}\to \frac{{\left(\prod _i\mathrm{tr}\left({\mathrm{\Sigma }}_{\text{x},j}\right)\right)}^{D_{\text{x}}-1}}{(\prod _i\mathrm{tr}{\left({\mathrm{\Sigma }}_{\text{x},j}\right)}^{D_{\text{x}}-1}}{\mathrm{\Sigma }}_{\text{x},D_{\text{x}}}\otimes \dots \otimes {\mathrm{\Sigma }}_{\text{x},1}={\mathrm{\Sigma }}_{\text{x},D_{\text{x}}}\otimes \dots \otimes {\mathrm{\Sigma }}_{\text{x},1}$$

consistently estimates Var(x).

Note that Hoff (2011) proposes an iterative algorithm for finding the maximum likelihood estimation (MLE) on the basis of the array normal assumption. The maximum likelihood estimation may improve upon the above estimates when data actually come from an array normal distribution.

5 |. SPARSE TCCA

We may also recover sparse canonical tensors for both TCCA and scTCCA to enhance the interpretability of the estimated canonical tensors. Following the iterative thresholding strategy introduced by Ma (202013) for sparse principal component analysis, by Wang, Gu, Ning, and Liu (2015) for sparse expectation-maximization algorithms, and by Tan et al. (2018) for generalized eigenvalue problems, we incorporate a hard-thresholding step in Algorithms 1 and 2, as follows:

$${\mathbf{w}}_i^{(t+1)}\leftarrow {\mathrm{\Theta }}_{\lambda }\left({\mathbf{w}}_i^{(t+1)}\right)\text{and}{\mathbf{w}}_j^{(t+1)}\leftarrow {\mathrm{\Theta }}_{\lambda }\left({\mathbf{w}}_j^{(t+1)}\right),$$

where Θ_λ(v) performs element-wise hard-thresholding on v, namely, the ith element of Θ_λ(v) is v_i if |v_i| > λ and 0 otherwise. In the simulation studies of Section 7, we employ fivefold cross-validation with a grid point search to choose the tuning parameter λ, following Tan, Wang, Liu, and Zhang (2018). Instead of a grid point search, a random search would be another choice (Bergstra & Bengio, 2012).

6 |. PROBABILISTIC MODEL FOR TCCA

Bach and Jordan (2006) give a probabilistic interpretation for the classic CCA, which enables us to simulate data with desired canonical vectors and canonical correlations. For TCCA without any assumption on the covariance structure, it is the same as the regular CCA by treating vectorized canonical tensors as canonical vectors. In this section, we first discuss how to generate data from given d canonical correlations ${\rho }_d=\left({\rho }_1,\dots ,{\rho }_d\right)$ and their corresponding canonical vectors in columns of matrices (V_d, W_d), and then we extend it to TCCA with the separable covariance assumption. Let ${\mathrm{\Sigma }}_{\text{x}}$ and ${\mathrm{\Sigma }}_{\text{y}}$ be two covariance matrices such that ${\text{V}}_d^{\top }{\mathrm{\Sigma }}_{\text{x}}{\text{V}}_d={\text{W}}_d^{\top }{\mathrm{\Sigma }}_{\text{y}}{\text{W}}_d={\text{I}}_{\text{d}}$. Define two linear transformations $A_x={\mathrm{\Sigma }}_x{V}_d{M}_x$ and $A_y={\mathrm{\Sigma }}_y{W}_d{M}_y$, where ${\text{M}}_{\text{x}},{\text{M}}_{\text{y}}\in {ℝ}^{d\times d}$ are arbitrary matrices such that ${\text{M}}_{\text{x}}{\text{M}}_{\text{y}}^{\top }=\mathrm{diag}\left({\rho }_d\right)$ and their spectral norms are less 1. We consider the latent factor model

$$\mathbf{z}\sim N\left(\mathbf{0},{\mathbf{I}}_d\right)$$

$$\mathbf{x}\mid \mathbf{Z}=\mathbf{z}\sim N\left({\mathbf{A}}_{\mathbf{x}}\mathbf{z}+{\mathbf{\mu }}_{\mathbf{x}},{\mathrm{\Sigma }}_{\mathbf{x}}-{\mathbf{A}}_{\mathbf{x}}{\mathbf{A}}_{\mathbf{x}}^{\top }\right)$$

$$\mathbf{y}\mid \mathbf{Z}=\mathbf{z}\sim N\left({\mathbf{A}}_{\mathbf{y}}\mathbf{z}+{\mathbf{\mu }}_{\mathbf{y}},{\mathrm{\Sigma }}_{\mathbf{y}}-{\mathbf{A}}_{\mathbf{y}}{\mathbf{A}}_{\mathbf{y}}^{\top }\right).$$

The joint distribution of (x, y) is

$$\left(\begin{array}{l}\mathbf{x}\hfill \\ \mathbf{y}\hfill \end{array}\right)\sim N\left(\left[\begin{array}{l}{\mu }_x\hfill \\ {\mu }_y\hfill \end{array}\right],\left[\begin{array}{c}{\mathrm{\Sigma }}_x{\mathrm{\Sigma }}_{xy}\\ {\mathrm{\Sigma }}_{xy}^{\top }{\mathrm{\Sigma }}_y\end{array}\right]\right),$$

where ${\mathrm{\Sigma }}_{\text{xy}}={\mathrm{\Sigma }}_{\text{x}}{\text{V}}_{\text{d}}\mathrm{diag}\left({\rho }_{\text{d}}\right){\text{W}}_{\text{d}}^{\top }{\mathrm{\Sigma }}_{\text{y}}$.

Now, we discuss how to construct ${\mathrm{\Sigma }}_{\text{x}}$ and ${\mathrm{\Sigma }}_{\text{y}}$ from (V_d, W_d), which is not described in Bach and Jordan (2006). Let V_d = Q_xR_x be the thin QR decomposition of V_d. Then

$${\mathrm{\Sigma }}_x=Q_x{R}_x^{-T}{R}_x^{-1}{Q}_x^T+T_x\left(I_p-Q_x{Q}_x^T\right)T_x^T$$

satisfies ${\mathbf{V}}_d^{\top }{\mathrm{\Sigma }}_x{\mathbf{V}}_d={\text{I}}_d$ for arbitrary $T_x\in {ℝ}^{p\times p}$. Similarly, let W_d = Q_yR_y be the thin QR decomposition of W_d. Then

$${\mathrm{\Sigma }}_y=Q_y{R}_y^{-T}{R}_y^T{Q}_y^T+T_y\left(I_q-Q_y{Q}_y^T\right)T_y^T$$

satisfies ${\mathbf{W}}_d^{\top }{\mathrm{\Sigma }}_y{\mathbf{W}}_d={\mathbf{I}}_d$ for arbitrary ${\text{T}}_{\text{y}}\in {ℝ}^{q\times q}$. In this notation, the joint covariance is

$$\text{Var}\left(\begin{array}{l}\text{x}\hfill \\ \text{y}\hfill \end{array}\right)=\left(\begin{array}{cc}{\text{Q}}_{\text{x}}{\text{R}}_{\text{x}}^{-\text{T}}{\text{R}}_{\text{x}}^{\text{T}}{\text{Q}}_{\text{x}}^{\text{T}}& {\text{Q}}_{\text{x}}{\text{R}}_{\text{x}}^{-\text{T}}\mathrm{diag}\left({\rho }_d\right){\text{R}}_{\text{y}}^{\text{T}}{\text{Q}}_{\text{y}}^{\text{T}}\\ {\text{Q}}_{\text{y}}{\text{R}}_{\text{y}}^{-T}\mathrm{diag}\left({\rho }_d\right){\text{R}}_{\text{y}}^{\text{T}}{\text{Q}}_{\text{y}}^{\text{T}}& {\text{Q}}_{\text{x}}{\text{R}}_{\text{x}}^{-\text{T}}{\text{R}}_{\text{y}}^{\text{T}}{\text{Q}}_{\text{y}}^{\text{T}}\end{array}\right)+\left(\begin{array}{cc}{\text{T}}_{\text{x}}\left({\text{I}}_{\text{p}}-{\text{Q}}_{\times }{\text{Q}}_{\text{x}}^{\top }\right){\text{T}}_{\text{x}}^T& 0\\ 0& {\text{T}}_{\text{y}}\left({\text{I}}_q-{\text{Q}}_{\text{y}}{\text{Q}}_{\text{y}}^{\text{T}}\right){\text{T}}_{\text{y}}^{\text{T}}\end{array}\right).$$ (9)

T_x and T_y are free parameters that adjust the noise level in x and y, respectively. The normal generative model is

$$\mathbf{z}\sim N\left(0,I_d\right)$$

$$\mathbf{x}\mid \mathbf{z}=\mathbf{z}\sim N\left({\mathbf{Q}}_{\mathbf{x}}{\mathbf{R}}_{\mathbf{x}}^{-T}{\mathbf{M}}_{\mathbf{x}}\mathbf{z}+{\mathbf{\mu }}_{\mathbf{x}},{\mathrm{\Sigma }}_{\mathbf{x}}\right)$$

$$\mathbf{y}\mid \mathbf{z}=\mathbf{z}\sim N\left({\mathbf{Q}}_{\mathbf{y}}{\mathbf{R}}_{\mathbf{y}}^{-T}{\mathbf{M}}_{\mathbf{y}}\mathbf{z}+{\mathbf{\mu }}_{\mathbf{y}},{\mathrm{\Sigma }}_{\mathbf{y}}\right)$$

where ${\mathrm{\Sigma }}_{\text{x}}$ and ${\mathrm{\Sigma }}_{\text{y}}$ are as in Equation (9). The normality is not essential for construction of this covariance structure.

The separable covariance structure brings complication. To account for this parsimonious structure in the generative model, we construct marginal factor matrices ${\mathrm{\Sigma }}_{\text{x,i}}$ and ${\mathrm{\Sigma }}_{\text{y,j}}$ that satisfy ${\mathbf{V}}_i^{\top }{\mathrm{\Sigma }}_{\mathbf{x},i}{\mathbf{V}}_i=R_x^{1/D_x}{\mathbf{I}}_{R_x}$ and ${\mathbf{W}}_j^{\top }{\mathrm{\Sigma }}_{y,j}{\mathbf{W}}_j=R_y^{1/D_Y}{\mathbf{y}}_{R_y}$. Let ${\text{V}}_i={\text{Q}}_{x,i}{\text{R}}_{x,j}$ and ${\text{W}}_j={\text{Q}}_{y,j}{\text{R}}_{y,j}$ be the thin QR decompositions. Then

$${\mathrm{\Sigma }}_{x,i}=R_x^{-D_x}{Q}_{x,j}{R}_{x,y}^{-T}{R}_{x,j}^{-1}{Q}_{x,j}^T+T_{x,j}\left(I_{D_i}-Q_{x,i}{Q}_{x,j}^T\right)T_{x,j}^T$$

$${\mathrm{\Sigma }}_{y,j}=R_y^{-D_y}{Q}_{y,j}{R}_{y,j}^{-T}{R}_{y,j}^{-1}{Q}_{y,j}^T+T_{y,j}\left(I_{q_j}-Q_{y,j}{Q}_{y,j}^T\right)T_{y,j}^T$$

satisfy the conditions with arbitrary ${\mathbf{T}}_{x,i}\in {ℝ}^{p_i\times p_i}$ and ${\mathbf{T}}_{y,j}\in {ℝ}^{q_j\times q_j}$. And we have the desired property

$$\begin{gathered} \mathrm{vec}{\left(\mathbf{V}\right)}^T\left({\mathrm{\Sigma }}_{\mathbf{x},D_x}\otimes \dots \otimes {\mathrm{\Sigma }}_{\mathbf{x},1}\right)\mathrm{vec}(\mathbf{V}) \\ =1_{R_x}^T{\left(V_{D_x}\odot \dots \odot V_1\right)}^T\left({\mathrm{\Sigma }}_{x,D_x}\otimes \dots \otimes {\mathrm{\Sigma }}_{x,1}\right)\left(V_{D_x}\odot \dots \odot V_1\right)1_{R_x} \\ =1_{R_x}^{\top }{\left({*}_i{V}_i^{\top }{\mathrm{\Sigma }}_{x,i}{V}_i\right)}_{1_x} \\ =R_x^{-1}{1}_{R_x}^T{I}_{R_x}{1}_{R_x} \\ =1. \end{gathered}$$

Similarly, $\mathrm{vec}{(\mathbf{W})}^T\left({\mathrm{\Sigma }}_{\mathbf{y},D_y}\otimes \dots \otimes {\mathrm{\Sigma }}_{\mathbf{y},1}\right)\mathrm{vec}(\mathbf{W})=1$.

7 |. NUMERICAL EXPERIMENTS

We use the generative model described in Section 6 to evaluate the methods discussed in this paper: classic CCA, TCCA, scTCCA, sparse TCCA, and sparse TCCA with the separable covariance. We assess these methods on their ability to recover the true latent population parameters $(\mathcal{V},\mathcal{W})$ used to generate i.i.d. samples of tensor data pairs $\left({\mathcal{X}}_n,{\mathcal{Y}}_n\right)$ for n ∈ [1000]. In all examples, the true $\mathcal{V}$ is a vector of length 100 with six entries set to 1 and the rest to 0. We use three different latent $\mathcal{W}\in {ℝ}^{64\times 64}$ shown in Figure 1: ${\mathcal{W}}_{\text{rectangle}}$, ${\mathcal{W}}_{\text{cross}}$ and ${\mathcal{W}}_{\text{butterfly}}$. White pixels indicate values of 1, and black pixels indicate values of 0. The rectangle and cross-population canonical tensors ${\mathcal{W}}_{\text{rectangle}}$ and ${\mathcal{W}}_{\text{cross}}$ are low rank; specifically, they are Rank 1 and Rank 2, respectively. The butterfly population canonical tensor ${\mathcal{W}}_{\text{butterfly}}$ is a high rank. At the first glance, these illustrative examples do not come across as challenging estimation problems in the high dimension-low sample size regime, but if we were to vectorize the data and perform CCA, the number of parameters to fit is 100 + 64⁴ = 4196 whereas the sample size is 1,000. Because the number of parameters exceeds the sample size, the sample covariance matrix is singular. Consequently, we add a small ridge term 10⁻³| to sample covariance matrices so that the generalized eigenvalue problem has a unique solution. The code for generating the simulation results is provided in the Supporting Information.

FIGURE 1.

True latent population $\mathcal{W}$ canonical tensors for numerical experiments[dummy]

7.1 |. Evaluation criteria and selection of tuning parameter

Canonical tensors can only be estimated up to a scaling factor. Thus, we measure estimation accuracy by the angle between population canonical tensors used to generate the data $(\mathcal{V},\mathcal{W})$ and estimated canonical tensors $(\widehat{\mathcal{V}},\widehat{\mathcal{W}})$, as follows:

$$\angle (\mathcal{V},\widehat{\mathcal{V}})=\frac{\langle \text{v},\widehat{\text{V}}\rangle }{\Vert \text{v}{\Vert }_2\Vert \widehat{\text{V}}{\Vert }_2},\text{and}\angle (\mathcal{W},\widehat{\mathcal{W}})=\frac{\langle \text{w},\widehat{\text{W}}\rangle }{\Vert \text{w}{\Vert }_2\Vert \widehat{\text{W}}{\Vert }_2}.$$

The angles can take on values from −1 to 1 where an angle closer to 1 indicates better recovery of the true canonical tensors.

We use k-fold cross-validation to perform model selection, for example, selection of the tensor rank or the sparsity level. We split our data into K equally sized groups; for k ∈ [K], we estimate a pair $({\mathcal{V}}_{-k},{\mathcal{W}}_{-k})$ on all but the kth fold of data for a sequence of models of varying complexity, ${\mathcal{M}}_1,{\mathcal{M}}_2,\dots$, where model ${\mathcal{M}}_1$ has a fixed pair of tensor ranks R_x and R_y and sparsity inducing parameter λ. We denote the fitted canonical correlation using model ${\mathcal{M}}_1$ by ${\widehat{\rho }}_{-k}\left({\mathcal{V}}_{-k},{\mathcal{W}}_{-k};{\mathcal{M}}_l\right)$ Using $({\mathcal{V}}_{-k},{\mathcal{W}}_{-k})$, we compute the canonical correlation on the held out jth fold, which we denote ${\widehat{\rho }}_{-j}\left({\mathcal{V}}_{-k},{\mathcal{W}}_{-k}\right)$. We choose the model that minimizes the average discrepancy between the empirical canonical correlations on the training sets and testing sets (Waaijenborg, Verselewel de Witt Hamer, & Zwinderman, 2008).

$$\widehat{\mathcal{M}}={\mathcal{M}}_l\arg \min \frac{1}{\text{K}}\sum _{k=1}^K\left|{\widehat{\rho }}_k\left({\mathcal{V}}_{-k},{\mathcal{W}}_{-k}\right)-{\widehat{\rho }}_{-k}\left({\mathcal{V}}_{-k},{\mathcal{W}}_{-k};{\mathcal{M}}_l\right)\right|.$$

We then use all the data to estimate $(\mathcal{V},\mathcal{W})$ using model $\widehat{\mathcal{M}}$.

7.2 |. Results

Figure 2 compares estimation accuracy of various methods over 100 replicates. First of all, we observe that the various versions of TCCA outperform the CCA on the vectorized data for all three choices of the latent canonical tensor $\mathcal{W}$. Second, we notice that there appears to be some overfitting in the case of rectangle and cross problems. In the rectangle problem, where the population canonical tensor ${\mathcal{W}}_{\text{rectangle}}$ is a Rank 1 tensor, all four TCCA methods show the best performance when we fix the rank R_y for estimating $\mathcal{W}$ to be 1, and the performance deteriorates as higher ranks are used. The same trend can be seen in the result of the cross problem where the population canonical tensor ${\mathcal{W}}_{\text{cross}}$ is a Rank 2 tensor. All TCCA methods give smaller values of angle when R_y = 3 is used compared with when R_y = 2 is used. In contrast, in the case of the butterfly image, the rank of population canonical tensor ${\mathcal{W}}_{\text{butterfly}}$ is much greater than 3. Thus, we do not expect an overfitting problem as well as good estimation performance as in the low-rank cases. We confirm that in Figure 2, calculated angles from the butterfly problem are lower than those from rectangles or cross problem.

FIGURE 2.

Angles between the recovered canonical vector/tensors and the true canonical vector/tensors. CCA, canonical correlation analysis; scTCCA, tensor canonical correlation analysis with separable covariance; spscTCCA, sparse tensor canonical correlation analysis with separable covariance; spTCCA, sparse tensor canonical correlation analysis; TCCA, tensor canonical correlation analysis

Another interesting observation is that results from TCCA methods with a separable covariance structure show better performance when we use a higher value for the rank parameter R_y than the true value than do other two models in cross and butterfly problems. This results implies that assuming a separable covariance structure improves the estimation accuracy in the two problems. This can be explained by the true image $\mathcal{X}$, which has a symmetric structure. Due to this special structure, we possibly expect improved performance from the more parsimonious model. However, this structured model does not have much effect on the rectangle problem. This may be because the true canonical tensor $\mathcal{W}$ already possess relatively few parameters. Note that the rectangle image is also symmetric but Rank 1, which has very few parameters.

Table 1 shows computation times taken for each method. In most cases, the computation times of TCCA methods are better or comparable with those of the CCA method. Especially, there is huge improvement when the underlying true canonical tensor has a low rank. Also, sparse models take less time than do nonsparse models, and separable covariance structure models take less time than do the models without the assumption for computation in general, which is expected.

TABLE 1.

Computation time: The mean run times are reported in seconds with the standard deviation in parentheses

Problem	Rank	CCA	TCCA	spTCCA	scTCCA	spscTCCA
Rectangle	1	22.90 (3.62)	7.52 (1.10)	7.36 (0.95)	3.14 (0.55)	3.09 (0.44)
	2	22.80 (3.18)	15.32 (2.51)	15.12 (2.17)	11.03 (2.03)	30.11 (8.78)
	3	21.85 (3.36)	24.07 (3.91)	18.61 (3.14)	15.89 (2.95)	37.09 (12.09)
Cross	1	18.49 (2.32)	8.05 (1.34)	7.78 (1.40)	3.39 (0.53)	13.19 (12.14)
	2	17.33 (2.43)	17.31 (3.32)	15.13 (5.02)	6.96 (1.11)	33.55 (12.78)
	3	17.10 (2.55)	24.44 (3.79)	54.86 (25.73)	10.04 (1.84)	9.63 (2.05)
Butterfly	1	18.92 (3.48)	7.08 (1.60)	61.18 (28.58)	2.90 (0.64)	31.66 (6.36)
	2	18.23 (2.92)	20.01 (4.53)	30.10 (14.87)	9.89 (2.15)	48.57 (13.14)
	3	19.99 (2.55)	32.27 (4.68)	26.09 (3.89)	11.84 (2.06)	13.58 (5.89)

Note. Experiments were performed on a computer cluster consisting of machines with Intel Xeon-based processors (I5 or I7 processors) with RAM ranging from 1,600 to 2,100 MHz. Abbreviations: CCA, canonical correlation analysis; scTCCA, tensor canonical correlation analysis with separable covariance; spscTCCA, sparse tensor canonical correlation analysis with separable covariance; spTCCA, sparse tensor canonical correlation analysis; TCCA, tensor canonical correlation analysis

8 |. DISCUSSION

We have proposed a TCCA approach for finding the relationship between linear combinations of two tensor datasets. Our method combines the classic CCA approach and the low-rank tensor decomposition to reduce the vast dimensionality of tensor parameters. The proposed estimation algorithm scales well with the tensor data size and is easy to implement using existing statistical software. Our algorithms also support convergence guarantees, properties arguably lacking in the alternatives in the literature. We briefly highlight some open problems for further investigation.

In our illustrative examples, we did not incorporate a procedure for selecting the rank parameter. A main motivation for the low-rank formulation of the canonical tensors is reduced computation. Thus, one may wish to choose as high a rank parameter as one’s computational budget allows. Nonetheless, we leave for future work a more principled approach to rank selection. One promising angle of pursuit would be to leverage the equivalence between CCA and a least squares problem (Sun, Ji, & Ye, 2008) and then derive an information criterion, a strategy commonly used to select the rank parameter in several recently proposed tensor estimation procedures (Zhou et al., 2013; Sun et al., 2017; Sun & Li, 2019).

An additional direction for future work is to generalize our TCCA framework to handle the analysis of multiple tensor datasets. There are numerous proposed methods to extend CCA for multiple vector-measurement datasets (Carroll, 1968; Kettenring, 1971; Hanafi, 2007; Witten & Tibshirani, 2009; Luo et al., 2015) but not for multiple tensor datasets to the best of our knowledge.

Another problem not tackled in the paper includes verifying the consistency of the proposed estimation procedures. Our clear specification of the population models provides a framework for studying the consistency property in both the large n fixed p (Zhou et al., 2013) and the large n diverging p settings (Zhang, Li, Zhou, Zhou, & Shen, 2019). Finally, we are currently investigating our methods on a real dataset.