Tensor-tensor algebra for optimal representation and compression of multiway data

Significance

Many real-world data are inherently multidimensional; however, often data are processed as two-dimensional arrays (matrices), even if the data are naturally represented in higher dimension. The common practice of matricizing high-dimensional data is due to the ubiquitousness and strong theoretical foundations of matrix algebra. Various tensor-based approximations have been proposed to exploit high-dimensional correlations. While these high-dimensional techniques have been effective in many applications, none have been theoretically proven to outperform matricization generically. In this study, we propose matrix-mimetic, tensor-algebraic formulations to preserve and process data in its native, multidimensional format. For a general family of tensor algebras we prove the superiority of optimal truncated tensor representations to traditional matrix-based representations with implications for other related tensorial frameworks.

Abstract

With the advent of machine learning and its overarching pervasiveness it is imperative to devise ways to represent large datasets efficiently while distilling intrinsic features necessary for subsequent analysis. The primary workhorse used in data dimensionality reduction and feature extraction has been the matrix singular value decomposition (SVD), which presupposes that data have been arranged in matrix format. A primary goal in this study is to show that high-dimensional datasets are more compressible when treated as tensors (i.e., multiway arrays) and compressed via tensor-SVDs under the tensor-tensor product constructs and its generalizations. We begin by proving Eckart–Young optimality results for families of tensor-SVDs under two different truncation strategies. Since such optimality properties can be proven in both matrix and tensor-based algebras, a fundamental question arises: Does the tensor construct subsume the matrix construct in terms of representation efficiency? The answer is positive, as proven by showing that a tensor-tensor representation of an equal dimensional spanning space can be superior to its matrix counterpart. We then use these optimality results to investigate how the compressed representation provided by the truncated tensor SVD is related both theoretically and empirically to its two closest tensor-based analogs, the truncated high-order SVD and the truncated tensor-train SVD.

1. Introduction

A. Overview.

Following the discovery of the spectral decomposition by Lagrange in 1762, the singular value decomposition (SVD) was discovered independently by Beltrami and Jordan in 1873 and 1874, respectively. Further generalization of the decomposition came independently by Sylvester in 1889 and by Autonne in 1915 (1). Perhaps the most notable theoretical results associated with the decomposition is due to Eckart and Young, who provided the first optimality proof of the decomposition back in 1936 (2).

The use of SVD in data analysis is ubiquitous. From a statistical point of view, the singular vectors of a (mean-subtracted) data matrix represent the principal component directions, i.e., the directions in which there is maximum variance corresponding to the largest singular values. However, the SVD is historically well-motivated by spectral analysis of linear operators. In typical data analysis tasks, matrices are treated as rectangular arrays of data which may not correspond directly to either statistical interpretations or representations of linear transforms. Hence, the prevalence of the SVD in data analysis applications requires further investigation.

The utility of the SVD in the context of data analysis is due to two key factors: the aforementioned Eckart–Young theorem (also known as the Eckart–Young–Minsky theorem) and the fact that the SVD (or in some cases a partial decomposition or high-fidelity approximation) can be efficiently computed relative to the matrix dimensions and/or desired rank of the partial decomposition (see, e.g., ref. 3 and references therein). Formally, the Eckart–Young theorem offers the solution to the problem of finding the best (in the Frobenius norm or 2-norm) rank-$k$ approximation to a matrix with rank greater than $k$ in terms of the first $k$ terms of the SVD expansion of that matrix. The theorem implies, in some informal sense, that the majority of the informational content is captured by the dominant singular subspaces (i.e., the span of the singular vectors corresponding to the largest singular values), opening the door to compression, efficient representation, denoising, and so on. The SVD motivates modeling data as matrices, even in cases where a more natural model is a high-dimensional array (a tensor)—a process known as matricization.

Intuitively, there is an inherent disadvantage of matricization of data which can naturally be represented as a tensor. For example, a grayscale image is intrinsically represented as a matrix of numbers, but a video is natively represented as a tensor since there is an additional dimension: time. Preservation of the dimensional integrity of the data can be imperative for subsequent analysis, e.g. to account for high-dimensional correlations embedded in the structures in which the data are organized. Even so, in practice, there is often a surprising dichotomy between the data representation and the algebraic constructs employed for its processing. Thus, in the last century there have been efforts to define decomposition of tensorial structures, e.g. CANDECOMP/PARAFAC (CP) (4–6), Tucker (7), higher-order SVD (HOSVD) (8), and Tensor-Train (9). However, none of the aforementioned decompositions offers an Eckart–Young-like optimality result.

In this study, we attempt to close this gap by proving a general Eckart–Young optimality theorem for a tensor-truncated representation. An Eckart–Young-like theorem must revolve around some tensor decomposition and a metric. We consider tensor decompositions built around the idea of the t-product presented in ref. 10 and the tensor-tensor product extensions of a similar vein in ref. 11. In refs. 10 and 11 the authors define tensor-tensor products between third-order tensors and corresponding algebra in which notions of identity, orthogonality, and transpose are all well-defined. For the t-product in ref. 10, the authors show there exists an Eckart–Young type of optimality result in the Frobenius norm. Other popular tensor decompositions [e.g., Tucker, HOSVD, CP, Tensor-Train SVD (TT-SVD) (4, 7–9, 12)] do not lend themselves easily to a similar analysis, either because direct truncation does not lead to an optimal lower-term approximation or because they lack analogous definitions of orthogonality and energy preservation.

In this paper, for the generalization of the t-product, the ${\star }_{\mathrm{\text{M}}}$-product (11), we prove two Eckart–Young optimality results. Importantly, we prove the superiority of the corresponding compressed tensor representation to the matrix compressed representation of the same data. This is a theoretical proof of superiority of compression of multiway data using tensor decomposition as opposed to treating the data as a matrix. Leveraging our algebraic framework, we show that the HOSVD can be interpreted as a special case of the class of tensor-tensor products we consider. As such, we are able to apply our tensor Eckart–Young results to explain why truncation of the HOSVD will, in general, not give an optimal compressed representation when compared to the proposed truncated tensor SVD approach. Moreover, we show how our Eckart–Young results can be applied in the context of comparison of the proposed tensor-tensor framework to a truncated TT-SVD approximation.

The ${\star }_{\mathrm{\text{M}}}$-product-based algebra is orientation-dependent, meaning that the ability to compress the data in a meaningful way depends on the orientation of the tensor (i.e., how the data are organized in the tensor). For example, a collection of $\ell$ grayscale images of size $m\times n$ can be placed into a tensor as lateral slices or be rotated first and then placed as lateral slices, and so on. In many applications, there are good reasons to keep the second, lateral, dimension fixed (i.e., representing time, total number of samples, etc.), but in others there may be no obvious merit to preferentially treat one of the other two dimensions. Thus, we also consider variants of the t-product approach that offer optimal approximations to the tensorized data without handling one spatial orientation differently than another.

We primarily limit the discussion to third-order tensors, though in the final section we discuss how the ideas generalize to higher-order as well. Indeed, the potential for even greater compression gain for higher-order representations of the data exists, provided the data have higher-dimensional correlations to be exploited.

B. Paper Organization.

In Section 2, we give background notation and definitions. In Section 3, we define decompositions, the t-SVDM and its variant, the t-SVDMII, and prove an Eckart–Young-like theorem for each. In Section 5, we employ these theorems to prove the superior representation of the t-SVDM and t-SVDMII compared to the matrix SVD. To provide intuition, Section 4 and Section 5 discuss when and why some data are more amenable to optimal representation through the use of the truncated t-SVDM than through the matrix SVD. In Section 6, we relate the ${\star }_{\mathrm{\text{M}}}$-framework to other related tensorial frameworks, including HOSVD and TT-SVD. Section 7 discusses multisided tensor compression. Section 8 contains a numerical study and highlights extensions to higher-order data. A summary and future work are the subjects of Section 9.

2. Background

For the purposes of this paper, a tensor is a multidimensional array, and the order of the tensor is defined as the number of dimensions of this array. As we are concerned with third-order tensors throughout most of the discussion, we limit our notation and definitions to the third-order case here and generalize to higher order in the final section.

A. Notation and Indexing

A third-order tensor $A$ is an object in ${\mathbb{C}}^{m\times p\times n}$. Its Frobenius norm, ${\Vert A\Vert }_F$, is analogous to the matrix case, that is, ${\Vert A\Vert }_F^2={\sum }_{i,j,k}|A_{i,j,k}{|}^2$. We use MATLAB notation for entries: $A_{i,j,k}$ denotes the entry at row $i$ and column $j$ of the matrix going $k$ “inward.” The fibers of tensor $A$ are defined by fixing two indices. Of note are the tube fibers, written as $A_{i,j,:}$ or ${\mathbf{a}}_{i,j}$, $i=1\hspace{0.17em}:\hspace{0.17em}m,j=1\hspace{0.17em}:\hspace{0.17em}p$. A slice of a third-order tensor $A$ is a two-dimensional array defined by fixing one index. Of particular note are the frontal and lateral slices, as depicted in Fig. 1. The $i^{th}$ frontal slice is expressed as $A_{:,:,i}$ and also referenced as ${\mathbf{A}}^{\left(i\right)}$ for convenience in later definitions. The $j^{th}$ lateral slice would be $A_{:,j,:}$ or equivalently expressed as ${\overrightarrow{A}}_j$.

Fig. 1.

Fibers and slices of $m\times p\times n$ tensor $A$. Left to right: Frontal slices, denoted either ${\mathbf{A}}^{\left(i\right)}$ or $A_{:,:,i}$; lateral slices denoted ${\overrightarrow{A}}_j$ or $A_{:,j,:}$; tube fibers ${\mathbf{a}}_{ij}$ or $A_{i,j,:}$.

Some other notation that we use for convenience are the vec and reshape operators that map matrices to vectors by column unwrapping, and vice versa:

$$\mathbf{\text{a}}=\mathtt{\text{vec}}\left(\mathbf{A}\right)\in {\mathbb{C}}^{mn}\leftrightarrow \mathbf{A}=\mathtt{\text{reshape}}\left(\mathbf{\text{a}},\left[m,n\right]\right).$$

We can also define invertible mappings between $m\times n$ matrices and $m\times 1\times n$ tensors by twisting and squeezing* (13): i.e., $\mathbf{X}\in {\mathbb{C}}^{m\times n}$ is related to $\overrightarrow{X}\in {\mathbb{C}}^{m\times 1\times n}$ via

$$\overrightarrow{X}=\mathtt{\text{twist}}\left(\mathbf{X}\right)\text{and}\mathbf{X}=sq\left(\overrightarrow{X}\right).$$

The mode-1, mode-2, and mode-3 unfoldings of $A\in {\mathbb{C}}^{m\times p\times n}$ are $m\times pn$, $p\times mn$, and $n\times mp$, respectively, and are given by

$$\begin{array}{ccc}{A}_{\left(1\right)}\hspace{1em}\hfill & ≔\hspace{1em}\hfill & \left[{\mathbf{A}}^{\left(1\right)},\dots ,{\mathbf{A}}^{\left(n\right)}\right]\hfill \\ A_{\left(2\right)}\hspace{1em}\hfill & ≔\hspace{1em}\hfill & \left[{\left({\mathbf{A}}^{\left(1\right)}\right)}^{\top },\dots ,{\left({\mathbf{A}}^{\left(n\right)}\right)}^{\top }\right]\hfill \\ A_{\left(3\right)}\hspace{1em}\hfill & ≔\hspace{1em}\hfill & \left[sq{\left(A_{:,1,:}\right)}^{\top },sq{\left(A_{:,2,:}\right)}^{\top },\dots ,sq{\left(A_{:,p,:}\right)}^{\top }\right]\hfill \end{array}.$$ [1]

These are useful in defining modewise tensor-matrix products (see ref. 12). For example, $A{\times }_3\mathbf{M}$ for a $r\times n$ matrix $\mathbf{M}$ is equivalent to computing the matrix-matrix product $\mathbf{M}{A}_{\left(3\right)}$, which is an $r\times mp$ matrix, and then reshaping the result to an $m\times p\times r$ tensor.

The HOSVD (8) can be expressed as

$$A=C{\times }_1\mathbf{U}{\times }_2\mathbf{V}{\times }_3\mathbf{W},$$ [2]

where $\mathbf{U},\mathbf{V},\mathbf{W}$ are the matrices containing the left singular vectors, corresponding to nonzero singular values, of the matrix SVDs of $A_{\left(1\right)},A_{\left(2\right)},A_{\left(3\right)}$, respectively. For an $m\times p\times n$ tensor, $\mathbf{U}$ would be $m\times r_1$, $\mathbf{V}$, $p\times r_2$, and $\mathbf{W}\hspace{0.17em}n\times r_3$, where $r_1,r_2,r_3$ are the ranks of the three respective unfoldings. Correspondingly, we say the tensor has HOSVD rank $\left(r_1,r_2,r_3\right)$. The $r_1\times r_2\times r_3$ core tensor is given by $C≔A{\times }_1{\mathbf{U}}^{\mathrm{H}}{\times }_2{\mathbf{V}}^{\mathrm{H}}{\times }_3{\mathbf{W}}^{\mathrm{H}}$. While the columns of the factor matrices are orthonormal, the core need not be diagonal, and its entries need not be nonnegative. In practice, compression is achieved by truncating to an HOSVD rank $\mathbf{k}=\left(k_1,k_2,k_3\right)$, but unlike the matrix case such truncation does not lead to an optimal truncated approximation in a norm sense. It does, however, offer a quasi-optimal approximation in the following sense: ${\Vert A-B_{\mathbf{k}}\Vert }_F\le \sqrt{d}{\Vert A-B_{*}\Vert }_F$, which means that if there exists an optimal solution $B_{*}$ with a small error then a truncated HOSVD $B_{\mathbf{k}}$ will yield a bounded approximation to it (14–16).

If $\mathbf{a},\mathbf{b},\mathbf{c}$ are length $m,n,p$ vectors, respectively, then $B≔\mathbf{u}○\mathbf{v}○\mathbf{w}$ (i.e., $B_{i,j,k}=u_i{v}_j{w}_k$) is called a rank-1 tensor. A CP (4–6) decomposition of a tensor $A$ is an expression as a sum of rank-1 outer-products:

$$A={\sum }_{i=1}^r{s}_i\left({\mathbf{u}}^{\left(i\right)}○{\mathbf{v}}^{\left(i\right)}○{\mathbf{w}}^{\left(i\right)}\right)=:\left[\left[\mathbf{S};\mathbf{U},\mathbf{V},\mathbf{W}\right]\right],$$

where the factor matrices $\mathbf{U},\mathbf{V},\mathbf{W}$ have the ${\mathbf{u}}^{\left(i\right)},{\mathbf{v}}^{\left(i\right)},{\mathbf{w}}^{\left(i\right)}$ as their columns, respectively, and $\mathbf{S}$ is a diagonal matrix containing the weights. If $r$ is minimal, then $r$ is said to be the rank of the tensor; that is, $r$ is the tensor rank. Although it is known that $r\le min\left(np,mp,nm\right)$, determining the rank of a tensor is an NP-hard problem (17). Also, the factor matrices need not have orthonormal columns, nor be full rank.^† .

However, if we are given a set of factor matrices with $k$ columns such that $A=\left[\left[\mathbf{U},\mathbf{V},\mathbf{W}\right]\right]$ holds, this is still a CP decomposition, even if we do not know if $k$ corresponds to, or is bigger than, the rank. In other words, given a CP decomposition where the factor matrices have $k$ columns, all we know is that the rank of the tensor represented by this decomposition has tensor rank at most $k$.

In the remaining subsections of this section, we provide background on a tensor-tensor product representation recently developed in the literature (10, 11, 13). The primary goal of this paper is to derive provably optimal (i.e., minimal, in the Frobenius norm) approximations to tensors under this matrix mimetic framework. We compare these approximations to those derived by processing the same data in matrix form and show links between our approximations and to direct tensorial factorizations that offer a notion of orthogonality, the HOSVD and TT-SVD.

B. A Family of Tensor-Tensor Products

The first closed multiplicative operation between a pair of third-order tensors of appropriate dimension was given in ref. 10. That operation was named the t-product, and the resulting linear algebraic framework is described in refs. 10 and 13. In ref. 11, the authors followed the theme from ref. 10 by describing a new family of tensor-tensor products, called ${\star }_{\mathrm{\text{M}}}$-product, and gave the associated algebraic framework. As the presentation in ref. 11 includes the t-product as one example, we will introduce the class of tensor-tensor products of interest and at times throughout the paper highlight the t-product as a special case.

Let $\mathbf{M}$ be any invertible $n\times n$ matrix, and $A\in {\mathbb{C}}^{m\times p\times n}$. We will use hat notation to denote a tensor in the transform domain specified by $\mathbf{M}$:

$$\hat{A}≔A{\times }_3\mathbf{M},$$

where, since $\mathbf{M}$ is $n\times n$, $\hat{A}$ has the same dimension as $A$. Importantly, $\hat{A}$ corresponds to applying $\mathbf{M}$ along all tube fibers, although it is implemented according to the definition of computing the matrix-matrix product $\mathbf{M}{A}_{\left(3\right)}$ and reshaping the result. The “hat” notation should be understood in context relative to the $\mathbf{M}$ applied.

Algorithm 1:

Algorithm $A{\star }_{\mathrm{\text{M}}}B$ for invertible $\mathbf{M}$ from ref. 11

INPUT: $A\in {\mathbb{C}}^{m\times p\times n}$, $B\in {\mathbb{C}}^{p\times \ell \times n}$, invertible $\mathbf{M}\in {\mathbb{C}}^{n\times n}$

1: Define $\hat{A}≔A{\times }_3\mathbf{M}$, $\hat{B}≔B{\times }_3\mathbf{M}$

2: for $i=1,\dots ,n$ do

3: ${\hat{C}}_{:,:,i}={\hat{A}}_{:,:,i}{\hat{B}}_{:,:,i}$

4: end for

5: Define-$C=\hat{C}{\times }_3{\mathbf{M}}^{-1}.\text{Now}C=A{\star }_{\mathrm{\text{M}}}B\in {\mathbb{C}}^{m\times \ell \times n}$

From ref. 11, we define the ${\star }_{\mathrm{\text{M}}}$-product between $A\in {\mathbb{C}}^{m\times p\times n}$ and $B\in {\mathbb{C}}^{p\times r\times n}$ through the steps in Algorithm 1. The inner for-loop (steps 2 through 4) is the facewise product, denoted $\hat{C}=\hat{A}△\hat{B}$, and is embarrassingly parallelizable since the matrix-matrix products in the loop are independent. Step 1 (and 5) could in theory also be performed in $p$ independent blocks of matrix-matrix products (or matrix solves, in the case of ${\times }_3{\mathbf{M}}^{-1}$ to avoid inverse computation).

Choosing $\mathbf{M}$ as the (unnormalized) DFT matrix and comparing to the algorithm in the previous section, we see that $A{\star }_{\mathrm{\text{M}}}B$ effectively reduce to the t-product operation, $A^{*}B$, defined in ref. 10. Thus, the t-product is a special instance of products from the ${\star }_{\mathrm{\text{M}}}$-product family.

C. Tensor Algebraic Framework

Now we can introduce the remaining parts of the framework. For the t-product, the linear algebraic framework is in refs. 10 and 13. However, as the t-product is a special case of the ${\star }_{\mathrm{\text{M}}}$ as noted above, we will elucidate here the linear algebraic framework as described in ref. 11, with pointers to refs. 10 and 13 so we can cover all cases.

There exists a notion of conjugate transposition and an identity element:

Definition 2.1 (Conjugate Transpose): Given $A\in {\mathbb{C}}^{m\times p\times n}$ its $p\times m\times n$ conjugate transpose under ${\star }_{\mathrm{\text{M}}}\hspace{0.17em}{A}^{\mathrm{H}}$ is defined

$${\left({\hat{A}}^{\mathrm{H}}\right)}_{:,:,i}={\left({\hat{A}}_{:,:,i}\right)}^{\mathrm{H}},\hspace{1em}i=1,\dots ,n.$$

As noted in ref. 11, this definition ensures the multiplication reversal property for the Hermitian transpose under ${\star }_{\mathrm{\text{M}}}$: $A^{\mathrm{H}}{\star }_{\mathrm{\text{M}}}{B}^{\mathrm{H}}={\left(B{\star }_{\mathrm{\text{M}}}A\right)}^{\mathrm{H}}.$ This definition is consistent with the t-product transpose given in ref. 10 when ${\star }_{\mathrm{\text{M}}}$ is defined by the DFT matrix.

Definition 2.2 (Identity Tensor; Unit-Normalized Slices): The $m\times m\times n$ identity tensor $I$ satisfies $A{\star }_{\mathrm{\text{M}}}I=A=I{\star }_{\mathrm{\text{M}}}A$ for $A\in {\mathbb{C}}^{m\times m\times n}$. For invertible $\mathbf{M}$, this tensor always exists. Each frontal slice of $\hat{I}=I{\times }_3\mathbf{M}$ is an identity matrix. If $\overrightarrow{B}$ is $m\times 1\times n$, and ${\overrightarrow{B}}^{\mathrm{H}}{\star }_{\mathrm{\text{M}}}\overrightarrow{B}$ is the $1\times 1\times n$ identity tensor under ${\star }_{\mathrm{\text{M}}}$, we say the $\overrightarrow{B}$ is a unit-normalized tensor slice.

The concept of unitary and orthogonal tensors is now straightforward:

Definition 2.3 (Unitary/Orthogonality): Two $m\times 1\times n$ tensors, $\overrightarrow{A},\overrightarrow{B}$, are called ${\star }_{\mathrm{\text{M}}}$-orthogonal slices if ${\overrightarrow{A}}^{\mathrm{H}}{\star }_{\mathrm{\text{M}}}\overrightarrow{B}$ is the tube fiber $\mathbf{\text{0}}$. If $Q\in {\mathbb{C}}^{m\times m\times n}$ ($Q\in {\mathbb{R}}^{m\times m\times n}$) is called ${\star }_{\mathrm{\text{M}}}$-unitary,^‡ (${\star }_{\mathrm{\text{M}}}$-orthogonal), if

$$Q^{\mathrm{H}}{\star }_{\mathrm{\text{M}}}Q=I=Q{\star }_{\mathrm{\text{M}}}{Q}^{\mathrm{H}},$$

where $H$ is replaced by transpose for real-valued tensors. Note that $I$ must be the one defined under ${\star }_{\mathrm{\text{M}}}$ as well and that tube fibers on the diagonal correspond to unit-normalized slices ${\overrightarrow{Q}}_j^{\mathrm{H}}{\star }_{\mathrm{\text{M}}}{\overrightarrow{Q}}_j$ and off-diagonals to tube fibers formed from products ${\overrightarrow{Q}}_j^{\mathrm{H}}{\star }_{\mathrm{\text{M}}}{\overrightarrow{Q}}_k$ with $k\ne j$.

3. Tensor ${\star }_{\mathrm{\text{M}}}$*M-SVDs and Optimal Truncated Representation

As noted in the previous section and described in more detail in ref. 11, any invertible matrix $\mathbf{M}$ can be used to define a valid tensor-tensor product. However, in this paper we will focus on a specific class of matrices $\mathbf{M}$ for which unitary invariance under the Frobenius norm is preserved, as we discuss in the following. We then use this feature to develop Eckart–Young theory for our tensor decompositions later in this section.

A. Unitary Invariance

Unitary invariance of the Frobenius norm of real-valued orthogonal tensors under the t-product was shown in ref. 10. Here, we prove a more general result.

Theorem 3.1. With the choice of $n\times n\hspace{0.17em}\mathbf{M}=c\mathbf{W}$ for unitary (orthogonal) $\mathbf{W}$, and nonzero $c$, assume $Q$ is $m\times m\times n$ and ${\star }_{\mathrm{\text{M}}}$-unitary (${\star }_{\mathrm{\text{M}}}$-orthogonal). Then

$${\Vert Q{\star }_{\mathrm{\text{M}}}B\Vert }_F={\Vert B\Vert }_F,B\in {\mathbb{C}}^{m\times k\times n}.$$

Likewise, if $B\in {\mathbb{C}}^{p\times m\times n}$, ${\Vert B{\star }_{\mathrm{\text{M}}}Q\Vert }_F={\Vert B\Vert }_F$.

Proof.

Suppose $\mathbf{M}=c\mathbf{W}$ where $\mathbf{W}$ is unitary. Then, ${\mathbf{M}}^{-1}=\frac{1}{c}{\mathbf{W}}^{\mathrm{H}}$. Next,

$${\Vert \hat{B}\Vert }_F={\Vert B{\times }_3\mathbf{M}\Vert }_F={\Vert c\mathbf{W}{B}_{\left(3\right)}\Vert }_F=c{\Vert B_{\left(3\right)}\Vert }_F=c{\Vert B\Vert }_F.$$ [3]

Let $C=Q{\star }_{\mathrm{\text{M}}}B$. Using [3],

$$\begin{array}{cc}\hfill {\Vert B\Vert }_F^2=\frac{1}{|c{|}^2}{\Vert \hat{B}\Vert }_F^2& =\frac{1}{|c{|}^2}{\sum }_{i=1}^p{\Vert {\hat{Q}}_{:,:,i}{\hat{B}}_{:,:,i}\Vert }_F^2\hfill \\ \hfill & =\frac{1}{|c{|}^2}{\Vert \hat{C}\Vert }_F^2={\Vert C\Vert }_F^2={\Vert Q{\star }_{\mathrm{\text{M}}}B\Vert }_F^2,\hfill \end{array}$$

as each ${\hat{Q}}_{:,:,i}$ is unitary. The other direction is similar.

We now have the framework we need to describe tensor SVDs induced by a fixed, ${\star }_{\mathrm{\text{M}}}$ operator. These were defined and existence was proven in ref. 10 for the t-product over real-valued tensors and in ref. 11 for ${\star }_{\mathrm{\text{M}}}$ more generally.

Definition 3.2 (10, 11): Let $A$ be a $m\times p\times n$ tensor. The (full) ${\star }_{\mathrm{\text{M}}}$ tensor SVD (t-SVDM) of $A$ is

$$A=U{\star }_{\mathrm{\text{M}}}S{\star }_{\mathrm{\text{M}}}{V}^{\mathrm{H}}={\sum }_{i=1}^r{U}_{:,i,:}{\star }_{\mathrm{\text{M}}}{S}_{i,i,:}{\star }_{\mathrm{\text{M}}}{V}_{:,i,:}^{\mathrm{H}},$$ [4]

where $U\in {\mathbb{R}}^{m\times m\times n}$, $V\in {\mathbb{R}}^{p\times p\times n}$ are ${\star }_{\mathrm{\text{M}}}-unitary$, and $S\in {\mathbb{R}}^{m\times p\times n}$ is a tensor whose frontal slices are diagonal (such a tensor is called f-diagonal), and $r\le min\left(m,p\right)$ is the number of nonzero tubes in $S$. When $\mathbf{M}$ is the DFT matrix, this reduces to the t-product-based t-SVD introduced in ref. 10.

Note that Definition 3.2 implies each frontal slice of $\hat{A}$ has rank less than or equal to $r$. Clearly, if $m>p$, from the second equality we can get a reduced t-SVDM, by restricting $U$ to have only $p$ orthonormal lateral slices, and $S$ to be $p\times p\times n$, as opposed to the full representation. Similarly, if $p>m$, we only need to keep the $m\times m\times n$ portion of $S$ and the $m$ columns of $V$ to obtain the same representation. An illustration of the decomposition is presented in Fig. 2, Top.

Fig. 2.

(Top) Illustration of the tensor SVD. (Bottom) Example showing different truncations across the different SVDs of the faces, based on Algorithm 3.

Algorithm 2:

Full t-SVDM from ref. 11

INPUT: $A\in {\mathbb{C}}^{m\times p\times n}$, invertible $\mathbf{M}\in {\mathbb{C}}^{n\times n}$

1: $\hat{A}\leftarrow A{\times }_3\mathbf{M}$

2: for $i=1,\dots ,n$ do

3: $\left[{\hat{U}}_{:,:,i},{\hat{S}}_{:,:,i},{\hat{V}}_{:,:,i}\right]=\mathtt{\text{svd}}\left({\hat{A}}_{:,:,i}\right)$

End for

4: $U=\hat{U}{\times }_3{\mathbf{M}}^{-1},\hspace{0.17em}S=\hat{S}{\times }_3{\mathbf{M}}^{-1},\hspace{0.17em}V=\hat{V}{\times }_3{\mathbf{M}}^{-1}$

Independent of the choice of $\mathbf{M}$, the components of the t-SVDM are computed in transform space. We describe the full t-SVDM in Algorithm 2. As noted, the t-SVDM above was proposed already in ref. 11. However, when we restrict the class of $\mathbf{M}$ to nonzero multiples of unitary or orthogonal matrices, we can now derive an Eckart–Young theorem for tensors in general form. To do so, we first give a new corollary for the restricted class of $\mathbf{M}$ considered.

Corollary 3.3. Assume $\mathbf{M}=c\mathbf{W}$, where $c\ne 0$, and $\mathbf{W}$ is unitary. Then given the t-SVDM of $A$ over ${\star }_{\mathrm{\text{M}}}$ defined in Definition 3.2,

$${\Vert A\Vert }_F^2={\Vert S\Vert }_F^2={\sum }_{i=1}^{min\left(p,m\right)}{\Vert S_{i,i,:}\Vert }_F^2.$$

Moreover, ${\Vert S_{\mathrm{1,1},:}\Vert }_F^2\ge {\Vert S_{\mathrm{2,2},:}\Vert }_F^2\ge \dots$.

Proof: The proof of the first equality follows from Theorem 3.1, the second from the definition of Frobenius norm. To prove the ordering property, use the shorthand for each singular tube fiber as ${\mathbf{s}}_i≔S_{i,i,:}$, and note using [3] that

$${\Vert {\mathbf{s}}_i\Vert }_F^2=\frac{1}{c}{\Vert {\widehat{\mathbf{s}}}_i\Vert }_F^2=\frac{1}{c}{\sum }_{j=1}^n{\left({\widehat{\sigma }}_i^{\left(j\right)}\right)}^2,$$

where we have used ${\widehat{\sigma }}_i^{\left(j\right)}$ to denote the $i^{th}$ largest singular value of the $j^{th}$ frontal face of $\hat{S}$. However since ${\widehat{\sigma }}_i^{\left(j\right)}\ge {\widehat{\sigma }}_{i+1}^{\left(j\right)}$, the result follows.

This observation gives rise to a new definition.

Definition 3.4: We refer to $r$ in the t-SVDM Definition 3.2 (see the second equality in [4]) as the t-rank,^§ the number of nonzero singular tubes in the t-SVDM.

We can also extend the idea of multirank in ref. 13 to the ${\star }_{\mathrm{\text{M}}}$ general case:

Definition 3.5: The multirank of $A$ under ${\star }_{\mathrm{\text{M}}}$ is the vector $\mathit{\rho }$ such that its $i^{th}$ entry ${\rho }_i$ denotes the rank of the $i^{th}$ frontal slice of $\hat{A}$; that is, ${\rho }_i=\text{rank}\left({\hat{A}}_{:,:,i}\right)$.

Notice that a tensor with multirank $\mathit{\rho }$ must have t-rank equal to $\underset{i=1,\dots ,n}{max}{\rho }_i$.

Definition 3.6: The implicit rank under ${\star }_{\mathrm{\text{M}}}$ of $A_{\mathit{\rho }}$ is $r={\sum }_{i=1}^n{\rho }_i$.

Note that $S$ is uniquely defined in Definition 3.2, thus the t-rank and multirank of a tensor are also unique.

B. Eckart–Young Theorem for Tensors

The key aspect that has made the t-product-based t-SVD so instrumental in many applications (see, for example, refs. 18–21) is the tensor Eckart–Young theorem proven in ref. 10 for real-valued tensors under the t-product. In loose terms, truncating the t-product-based t-SVDM gives an optimal low t-rank approximation in the Frobenius norm. An Eckart–Young theorem for the ${\star }_{\mathrm{\text{M}}}$ operator was not provided in ref. 11. We give a proof below for the special case that we have been considering in which $\mathbf{M}$ is a multiple of a unitary matrix.

Theorem 3.7. Define $A_k=U_{:,1:k,:}{\star }_{\mathrm{\text{M}}}{S}_{1:k,1:k,:}{\star }_{\mathrm{\text{M}}}{V}_{:,1:k,:}^{\mathrm{H}}$, where $\mathbf{M}$ is a nonzero multiple of a unitary matrix. Then, $A_k$ is the best Frobenius norm approximation^¶ over the set $\mathrm{\Gamma }=\left\{C=X{\star }_{\mathrm{\text{M}}}Y|X\in {\mathbb{C}}^{m\times k\times n},Y\in {\mathbb{C}}^{k\times p\times n}\right\},$ the set of all t-rank $k$ tensors under ${\star }_{\mathrm{\text{M}}}$, of the same dimensions as $A$. The squared error is ${\Vert A-A_k\Vert }_F^2={\sum }_{i=k+1}^r{\Vert {\mathbf{s}}_i\Vert }_F^2$, where $r$ is the t-rank of $A$.

Proof: The squared error result follows easily from the results in the previous section. Now let $B=X{\star }_{\mathrm{\text{M}}}Y$. ${\Vert A-B\Vert }_F^2=\frac{1}{c}{\Vert \hat{A}-\hat{B}\Vert }_F^2=\frac{1}{c}{\sum }_{i=1}^n{\Vert {\hat{A}}_{:,:,i}-{\hat{B}}_{:,:,i}\Vert }_F^2.$ By definition, ${\hat{B}}_{:,:,i}$ is a rank-k outer product ${\hat{X}}_{:,:,i}{\hat{Y}}_{:,:,i}$. The best rank-k approximation to ${\hat{B}}_{:,:,i}$ is ${\hat{U}}_{:,1:k,i}{\hat{S}}_{1:k,1:k,i}{\hat{V}}_{:,1:k,i}^{\mathrm{H}}$, so ${\Vert {\hat{A}}_{:,:,i}-{\hat{U}}_{:,1:k,i}{\hat{S}}_{1:k,1:k,i}{\hat{V}}_{:,1:k,i}^{\mathrm{H}}\Vert }_F^2\le {\Vert {\hat{A}}_{:,:,i}-{\hat{B}}_{:,:,i}\Vert }_F^2$, and the result follows.

In ref. 18, the authors used the Eckart–Young result for the t-product for compression of facial data and a PCA-like approach to recognition. They also gained additional compression in an algorithm they called the t-SVDII (only for the t-product on real-valued tensors), which, although not described as such in that paper, is effectively reducing the multirank for further compression. Here, we provide the theoretical justification for the t-SVDII approach in ref. 18 while simultaneously extending the result to the ${\star }_{\mathrm{\text{M}}}$-product family restricted to $\mathbf{M}$ being a nonzero multiple of a unitary matrix.

Theorem 3.8. Given the t-SVDM of $A$ under ${\star }_{\mathrm{\text{M}}}$, define $A_{\mathit{\rho }}$, to be the approximation having multirank $\mathit{\rho }$: that is,

$${\left({\hat{A}}_{\mathit{\rho }}\right)}_{:,:,i}={\hat{U}}_{:,1:{\rho }_i,i}{\hat{S}}_{1:{\rho }_i,1:{\rho }_i,i}{\hat{V}}_{:,1:{\rho }_i,i}^{\mathrm{H}}.$$

Then $A_{\mathit{\rho }}$ is the best multirank $\mathit{\rho }$ approximation to $A$ in the Frobenius norm and

$${\Vert A-A_{\mathit{\rho }}\Vert }_F^2={\sum }_{i=1}^n{\sum }_{k=1}^{r_i}{\left({\widehat{\sigma }}_{{\rho }_i+k}^{\left(i\right)}\right)}^2,$$

where $r_i$ denotes the rank of the $i^{th}$ frontal face of $\hat{A}$.

Proof: Follows similarly to the above, and is omitted.

To use this in practice, we generalize the idea of the t-SVDII in ref. 18 to the ${\star }_{\mathrm{\text{M}}}$-product for $\mathbf{M}$ a multiple of a unitary matrix. First, we need a suitable method to choose $\mathit{\rho }$. We know

$${\Vert \hat{A}\Vert }_F^2={\sum }_{i=1}^n{\sum }_{j=1}^{r_i}{\left({\widehat{\sigma }}_j^{\left(i\right)}\right)}^2,$$

where $r_i$ is the rank of the $i^{th}$ frontal face of $\hat{A}$. Thus, there are $K≔{\sum }_{i=1}^n{r}_i\le nmin\left(m,p\right)$ total nonzero singular values. Let us order the ${\left({\widehat{\sigma }}_j^{\left(i\right)}\right)}^2$ values in descending order and put them into a vector of length $K$. We find the first index $J\le K$ such that $\left({\sum }_{i=1}^J{v}_i\right)/{\Vert \hat{A}\Vert }_F^2>\gamma$. Keeping $J$ total terms thus implies an approximation of energy $\gamma$. Then let $\tau =\sqrt{v_J}$—this will be the value of the singular value that is the smallest one which we should include in the approximation. We run back through the $n$ faces, and for face $i$ we keep only the ${\rho }_i$ singular 3-tuples such that ${\widehat{\sigma }}_j^{\left(i\right)}\ge \tau$. In other words, the RE in our approximation is given by

$$\frac{{\sum }_{i=1}^n{\sum }_{j=1}^{{\rho }_i}{\left({\widehat{\sigma }}_j^{\left(i\right)}\right)}^2}{{\Vert \hat{A}\Vert }_F^2}\approx \gamma .$$

The pseudo-code is given in Algorithm 3, and a cartoon illustration of the output is given in Fig. 2.

Algorithm 3:

Return t-SVDMII under ${\star }_{\mathrm{\text{M}}}$, $\mathit{\rho }$ to meet energy constraint

INPUT: $A$, $\mathbf{M}$ a multiple of unitary matrix; desired energy $\gamma \in \left(\mathrm{0,1}\right]$.

1: Compute t-SVDM of $A$.

2: Concatenate $\left(\left({\hat{S}}_{j,j,i}\right){.}^2\right)$ for all $i,j$ into a vector $\mathbf{v}$.

3: $\mathbf{v}\leftarrow \text{sort}\left(\mathbf{v}{,}^{\prime }{\text{descend}}^{\prime }\right)$.

4: Let $\mathbf{w}$ be the vector of cumulative sums: i.e., ${\mathbf{w}}_k={\sum }_{i=1}^k{\mathbf{v}}_i$

5: Find the first index $J$ such that ${\mathbf{w}}_J/{\Vert \widehat{S}\Vert }_F^2>\gamma$.

6: Define $\tau ≔{\mathbf{v}}_J$.

7: for $i=1,\dots ,n$ do

8: Set ${\rho }_i$ as number of singular values for ${\hat{A}}_{:,:,i}$ greater or equal to $\tau$.

9: Keep only the $m\times {\rho }_i\hspace{0.17em}{\hat{U}}_{:,1:{\rho }_i,i}$ and ${\hat{G}}_{\mathit{\rho }}≔{\hat{S}}_{1:{\rho }_i,1:{\rho }_i,i}{\hat{V}}_{:,1:{\rho }_i,i}^{\mathrm{H}}$.

End for

In Section 5, we compare our Eckart–Young results for tensors with the corresponding matrix approximations obtained by using the matrix-based Eckart–Young theorem, but first we need a few results that show what structure these tensor approximations inherit from $\mathbf{M}$.

4. Latent Structure

To understand why the proposed tensor decompositions are efficient at compression and feature extraction we investigate the latent structure induced by the algebra in which we operate. We shall also capitalize on this structural analysis in the next section’s proofs stating how the proposed t-SVDM and t-SVDMII decompositions can be used to devise superior approximations compared to their matrix counterparts.

If $\mathbf{v},\mathbf{c}\in {\mathbb{C}}^{1\times 1\times n}$, from Algorithm 1 we have

$$\mathbf{v}{\star }_{\mathrm{\text{M}}}\mathbf{c}=\left(\left(\mathbf{v}{\times }_3\mathbf{M}\right)\right.\odot \left(\left(\mathbf{c}{\times }_3\mathbf{M}\right)\right){\times }_3{\mathbf{M}}^{-1},$$

where $\odot$ indicates pointwise scalar products on each face. Using the definitions, this expression is tantamount to

$$\begin{array}{c}\hfill \mathtt{\text{twist}}\left[{\left(\left({\mathbf{M}}^{-1}\mathtt{\text{diag}}\left(\mathbf{M}\hspace{0.17em}{\mathbf{c}}_{\left(3\right)}\right)\mathbf{M}\hspace{0.17em}{\mathbf{v}}_{\left(3\right)}\right.\right)}^{\top }\right]\\ \hfill \hspace{1em}=\mathtt{\text{twist}}\left[{{\mathbf{v}}_{\left(3\right)}}^{\top }{\mathbf{M}}^{\top }\mathtt{\text{diag}}\left(\widehat{\mathbf{c}}\right){\mathbf{M}}^{-\top }\right].\end{array}$$ [5]

Note that $\mathbf{M}{\mathbf{c}}_{\left(3\right)}$ is mathematically equivalent to forming the tube fiber $\widehat{\mathbf{c}}$, and $diag$ applied to a tube fiber works analogously to $diag$ applied to that tube fiber’s column vector equivalent. Further, the transpose is a real-valued transpose, stemming from the definition of the mode-3 product.

Let $\overrightarrow{B}$ be any element in ${\mathbb{C}}^{m\times 1\times n}$ and consider computing the product $\overrightarrow{Q}≔\overrightarrow{B}{\star }_{\mathrm{\text{M}}}\mathbf{c}$. In ref. 11 it was shown that the $j^{th}$ tube fiber entry in $\overrightarrow{Q}$ is effectively the product of the tubes ${\overrightarrow{B}}_{j,1,:}{\star }_{\mathrm{\text{M}}}\mathbf{c}$. From [5] we have

$$sq\left(\overrightarrow{Q}\right)=sq\left(\overrightarrow{B}\right)\left({\mathbf{M}}^{\top }\mathtt{\text{diag}}\left(\widehat{\mathbf{c}}\right){\mathbf{M}}^{-\top }\right).$$ [6]

The matrix in the parentheses on the right is an element of the space of all matrices ${\mathcal{X}}_{\mathbf{M}}=\left\{\mathbf{X}:\mathbf{X}={\mathbf{M}}^{\top }\mathbf{D}{\mathbf{M}}^{-\top }\right\}$, where $\mathbf{D}$ is diagonal. This brings us to a major result.

Theorem 4.1. Given $A=U{\star }_{\mathrm{\text{M}}}\underset{C}{\underbrace{S{\star }_{\mathrm{\text{M}}}{V}^{\mathrm{H}}}}={\sum }_{i=1}^t{U}_{:,i,:}{\star }_{\mathrm{\text{M}}}{C}_{i,:,:}$. Then, using $\mathbf{R}\left[\mathbf{v}\right]≔{\mathbf{M}}^{\top }diag\left(\widehat{\mathbf{v}}\right){\mathbf{M}}^{-\top }$

$$sq\left(A_{:,k,:}\right)={\sum }_{i=1}^{t}sq\left(U_{:,i,:}\right)\mathbf{R}\left[C_{i,k,:}\right],={\sum }_{i=1}^t{\mathbf{U}}_i\mathbf{R}\left[C_{i,k,:}\right].$$ [7]

Thus, each lateral slice of $A$ is a weighted combination of “basis” matrices given by ${\mathbf{U}}_i≔sq\left(U_{:,i,:}\right)$, but the weights, instead of being scalars, are matrices $\mathbf{R}\left[C_{i,k,:}\right]$ from the matrix algebra induced by the choice of $\mathbf{M}$. For $\mathbf{M}$, the DFT matrix, the matrix algebra is the algebra of circulants.

5. Tensors and Optimal Approximations

In ref. 18, the claim was made that a t-SVD to $k$ terms could be superior to a matrix SVD based compression to $k$ terms. Here, we offer a formal proof then discuss the relative meaning of $k$. Then, in the next section, we discuss what can be done to obtain further compression.

A. Theory: T-rank vs. Matrix Rank

Let us assume that our data are a collection of $\ell$, $m\times n$ matrices ${\mathbf{D}}_i,i=1,\dots \ell$. For example, ${\mathbf{D}}_i$ might be a grayscale image, or it might be the values of a function discretized on a two-dimensional uniform grid. Let ${\mathbf{d}}_i=\mathtt{\text{vec}}\left({\mathbf{D}}_i\right)$, so that ${\mathbf{d}}_i$ has length $mn$.

We put samples into a matrix (tensor) from left to right:

$$\begin{array}{ll}\hfill \mathbf{A}& =\left[{\mathbf{d}}_1,\dots ,{\mathbf{d}}_{\ell }\right]\in {\mathbb{C}}^{mn\times \ell }\hfill \\ \hfill A& =\left[\mathtt{\text{twist}}\left({\mathbf{D}}_i\right),\dots ,\mathtt{\text{twist}}\left({\mathbf{D}}_{\ell }\right)\right]\in {\mathbb{C}}^{m\times \ell \times n}.\hfill \end{array}$$

Thus, $\mathbf{A},A$ represent the same data, just in different formats. It is first instructive to consider in what ways the t-rank, $t$, of $A$ and the matrix rank $r$ of $\mathbf{A}$ are related. Then, we will move on to relating the optimal t-rank $k$ approximation of $A$ with the optimal rank-$k$ approximation to $\mathbf{A}$.

Theorem 5.1. The t-rank, $t$, of $A$ is less than or equal to the rank, $r$, of $\mathbf{A}$. Additionally, since $t\le min\left(m,\ell \right)$, if $m<r$, then $t<r$.

Proof: The problem’s dimensions necessitate $t\le min\left(m,\ell \right)$ and $r\le \ell$. Let $\mathbf{A}=\mathbf{G}{\mathbf{H}}^{\top }$ be a rank-r factorization of $\mathbf{A}$ such that $\mathbf{G}$ is $mn\times r$ and ${\mathbf{H}}^{\top }$ is $r\times \ell$. From the fact $\hat{A}=A{\times }_3\mathbf{M}$, we can show the $m\times \ell$-sized $i^{th}$ frontal face of $\hat{A}$ satisfies

$$\begin{array}{cc}\hfill {\hat{A}}_{:,:,i}={\sum }_{j=1}^n{m}_{ij}{A}_{:,:,j}& ={\sum }_{j=1}^n{m}_{ij}{\mathbf{G}}_{\left(j-1\right)m+jm,:}{\mathbf{H}}^{\top }\hfill \\ \hfill & =\left({\sum }_{j=1}^n{m}_{ij}{\mathbf{G}}_{\left.j-1\right)m+jm,:}\right){\mathbf{H}}^{\top }.\hfill \end{array}$$ [8]

Clearly, the rank of this frontal slice is bounded above by $min\left(m,r\right)$ since this is the maximal rank of the matrix in parentheses. Then, the singular values of the matrix ${\hat{A}}_{:,:,i}$ satisfy ${\widehat{\sigma }}_1^{\left(i\right)}\ge {\widehat{\sigma }}_2^{\left(i\right)}\ge \dots {\widehat{\sigma }}_{r_i}^{\left(i\right)}$, where $r_i\le min\left(m,r\right)$. As ${\hat{S}}_{j,j,i}={\widehat{\sigma }}_j^{\left(i\right)}$, $S_{j,j,:}={\hat{S}}_{j,j,:}{\times }_3{\mathbf{M}}^{-1}$ for a particular value $j$ will be a nonzero tube fiber iff for any of the $i=1:n$ at least one ${\widehat{\sigma }}_j^{\left(i\right)}$ is nonzero. There can be at most $min\left(m,r\right)$ nonzero tube fibers, so $t\le min\left(m,r\right)$.

Note that the proof was independent of the choice of invertible $\mathbf{M}$. In particular, it holds for $\mathbf{M}=\mathbf{I}$. This means that the act of “folding” the data matrix into a tensor may provide a reduced rank approximation (a rank-$r$ matrix goes to a t-rank $<r$ tensor). A nonidentity choice of $\mathbf{M}$, though, may reveal $t\ll r$. To make the idea concrete, let us consider an example.

Example 5.2: Let $\mathbf{U}\in {\mathbb{R}}^{n\times n}$ invertible, and ${\mathbf{c}}_i\in {\mathbb{R}}^n$, $i=1,\dots ,p$ with $p\le n$ be a set of independent vectors. Define $A$ such that $A_{:,i,:}=\mathtt{\text{twist}}\left(\mathbf{U}\mathtt{\text{circ}}\left({\mathbf{c}}_i\right)\right)$. The t-rank is 1.

On the other hand, with $\mathbf{Z}$ the circulant downshift matrix,

$$\mathbf{A}=\left(\mathbf{I}\otimes \mathbf{U}\right)diag\left(\mathbf{I},\mathbf{Z},\dots ,{\mathbf{Z}}^{n-1}\right)\left[\begin{array}{cccc}{\mathbf{c}}_1\hfill & {\mathbf{c}}_2\hfill & \cdots \hfill & {\mathbf{c}}_p\hfill \\ {\mathbf{c}}_1\hfill & {\mathbf{c}}_2\hfill & \cdots \hfill & {\mathbf{c}}_p\hfill \\ ⋮\hfill & ⋮\hfill & ⋮\hfill & ⋮\hfill \\ {\mathbf{c}}_1\hfill & {\mathbf{c}}_2\hfill & \cdots \hfill & {\mathbf{c}}_p\hfill \end{array}\right].$$

The rank of the $\mathbf{A}$ therefore is $p$. If $\mathbf{U}$ and $\mathbf{C}$ have orthonormal columns, we can show ${\Vert \mathbf{A}-{\mathbf{A}}_k\Vert }_F^2=\left(p-k\right)n$ for any $k<n$.

B. Theory: Comparison of Optimal Approximations

In this subsection we compare the quality of approximations obtained by truncating the matrix SVD of the data matrix vs. truncating the t-SVDM of the same data as a tensor. We again assume that $mn>\ell$ and that $\mathbf{A}$ has rank $r\le \ell$.

Let $\mathbf{A}=\mathbf{U}\mathbf{\Sigma }{\mathbf{V}}^{\top }$ be the matrix SVD of $\mathbf{A}$, and denote its best rank-$k$, $k<r$ approximation according to

$${\mathbf{A}}_k≔{\mathbf{U}}_{:,1:k}{\mathbf{C}}_{1:k,:}\Rightarrow {\left({\mathbf{A}}_k\right)}_{:,j}={\sum }_{i=1}^k{\mathbf{U}}_{:,i}{c}_{ij},$$ [9]

for $j=1,\dots ,k$ where $\mathbf{C}=\mathbf{\Sigma }{\mathbf{V}}^{\top }$. Finally, we need the following matrix version of [9] which we reference in the proof:

$$\mathtt{\text{reshape}}\left({\left({\mathbf{A}}_k\right)}_{:,j},\left[m,n\right]\right)={\sum }_{i=1}^k\mathtt{\text{reshape}}\left({\mathbf{U}}_{:,i},\left[m,n\right]\right)c_{ij}.$$ [10]

Theorem 5.3. Given $\mathbf{A}$, $A$ as defined above, with $\mathbf{A}$ having rank $r$ and $A$ having t-rank $t$, let ${\mathbf{A}}_k$ denote the best rank-$k$ matrix approximation to $\mathbf{A}$ in the Frobenius norm, where $k\le r$. Let $A_k$ denote the best t-rank-$k$ tensor approximation under ${\star }_{\mathrm{\text{M}}}$, where $\mathbf{M}$ is a multiple of a unitary matrix, to $A$ in the Frobenius norm. Then

$${\Vert S_{k+1:t,k+1:t,:}\Vert }_F={\Vert A-A_k\Vert }_F\le {\Vert \mathbf{A}-{\mathbf{A}}_k\Vert }_F.$$

Proof: Consider [10]. The multiplication by the scalar $c_{ij}$ in the sum is equivalent to multiplication from the right by $c_{ij}\mathbf{I}$. However, since $\mathbf{M}=c\mathbf{W}$ for unitary $\mathbf{W}$, we have $c_{ij}\mathbf{I}={\mathbf{M}}^{\top }\mathtt{\text{diag}}\left(c_{ij}\mathbf{e}\right){\mathbf{M}}^{-\top }$, where $\mathbf{e}$ is the vector of all ones. Define the tube fiber $C_{i,j,:}$ from the matrix-vector product $c_{ij}{\mathbf{M}}^{-1}\mathbf{e}$ oriented into the third dimension. Then, $c_{ij}\mathbf{I}=\mathbf{R}\left[C_{i,j,:}\right]$. Now we observe that [10] can be equivalently expressed as

$$\mathtt{\text{reshape}}\left({\left({\mathbf{A}}_k\right)}_{:,j}\right)={\sum }_{i=1}^k\mathtt{\text{reshape}}\left({\mathbf{U}}_{:,i}\right)\mathbf{R}\left[{\mathbf{c}}_{ij}\right].$$ [11]

These can be combined into a tensor equivalent

$$Z_k≔{\sum }_{i=1}^k{Q}_{:,i,:}{\star }_{\mathrm{\text{M}}}{C}_{i,:,:}=Q{\star }_{\mathrm{\text{M}}}C\text{where}$$

$$\begin{array}{cc}\hfill {\left(Z_k\right)}_{:,j,:}& =\mathtt{\text{twist}}\left(\mathtt{\text{reshape}}\left({\left({\mathbf{A}}_k\right)}_{:,j},\left[m,n\right]\right)\right),\hfill \\ \hfill Q_{:,i,:}& =\mathtt{\text{twist}}\left(\mathtt{\text{reshape}}\left({\mathbf{U}}_{:,i},\left[m,n\right]\right)\right).\hfill \end{array}$$

Since ${\hat{C}}_{:,:,i}={\hat{\mathbf{\Sigma }}}_{1:k,1:k}{\hat{\mathbf{V}}}_{:,1:k}^{\top }$, the t-rank of $C$ is $k$. The t-rank of $Q$ must also not be smaller than $k$, by Theorem 5.1.

Thus, given the definition of $A_k$ as the minimizer over all such $k$-term “outer-products” under ${\star }_{\mathrm{\text{M}}}$, it follows that

$$\begin{array}{cc}\hfill {\Vert S_{k+1:t,k+1:t,:}\Vert }_F& ={\Vert A-A_k\Vert }_F\hfill \\ \hfill & \le {\Vert A-Z_k\Vert }_F={\Vert \mathbf{A}-{\mathbf{A}}_k\Vert }_F.\hfill \end{array}$$

Here is an example showing strict inequality is possible. Additional supporting examples are in the numerical results.

Example 5.4: Given $\mathbf{M}$ the Haar wavelet matrix, and let

$$\mathbf{A}=\left[\begin{array}{cc}1\hfill & 1\hfill \\ 1\hfill & 4\hfill \\ 0\hfill & 0\hfill \\ 0\hfill & -3\hfill \end{array}\right],\hspace{1em}\text{with}{\mathbf{A}}_1={\sigma }_1{\mathbf{u}}_1{\mathbf{v}}_1^{\top }.$$

It is easily shown that ${\Vert \mathbf{A}-{\mathbf{A}}_1\Vert }_F=\sqrt{{\sigma }_2}=1$. Setting $A_1=U_{:,1,:}{\star }_{\mathrm{\text{M}}}{S}_{\mathrm{1,1},:}{\star }_{\mathrm{\text{M}}}{V}_{:,1,:}^{\top }$, we observe

$$\begin{array}{cc}\hfill {\Vert A-A_1\Vert }_F& ={\Vert S_{\mathrm{2,2},:}\Vert }_F\hfill \\ \hfill & ={‖{\mathbf{M}}^{\top }\left[\begin{array}{c}0\hfill \\ {\widehat{\sigma }}_2^{\left(2\right)}\hfill \end{array}\right]‖}_F={\widehat{{\sigma }_2}}^{\left(2\right)}\approx 0.59236<1.\hfill \end{array}$$

In the next subsection we discuss the level of approximation provided by the output of Algorithm 3 by relating it back to the truncated t-SVDM and also to truncated matrix SVD. First, we need a way to relate storage costs.

Theorem 5.5. Let $A_k$ be the t-SVDM t-rank $k$ approximation to $A$, and suppose its implicit rank is $r$. Define $\mu ={\Vert A_k\Vert }_F^2/{\Vert A\Vert }_F^2$. There exists $\gamma \le \mu$ such that the t-SVDMII approximation, $A_{\mathit{\rho }}$, obtained for this $\gamma$ in Algorithm 3, has implicit rank less than or equal to an implicit rank of $A_k$ and

$${\Vert A-A_{\mathit{\rho }}\Vert }_F\le {\Vert A-A_k\Vert }_F\le {\Vert \mathbf{A}-{\mathbf{A}}_k\Vert }_F.$$

Proof: From Theorem 5.3 that ${\Vert A-A_k\Vert }_F^2=\frac{1}{c}{\sum }_{j=1}^n{\sum }_{i=k+1}^n{\left({\widehat{\sigma }}_i^{\left(j\right)}\right)}^2$. The proof is by construction using $A_k$ as the starting point; that is, assume ${\rho }_i=k$ for each frontal slice. Let ${\widehat{\sigma }}_{*}$ be the largest singular value we truncated; that is,

$${\widehat{\sigma }}_{*}=\underset{i=1,\dots ,n}{max}{\sigma }_{{\rho }_i+1}^{\left(i\right)}.$$

Let $C=\left\{{\widehat{\sigma }}_j^{\left(i\right)}<{\widehat{\sigma }}_{*}|1\le j\le {\rho }_i,\forall i\right\}$ be the union of all of the singular values that were kept in the approximation and smaller than ${\widehat{\sigma }}_{*}$. If $C$ is empty, then $A_{\mathit{\rho }}=A_k$ and we are done.

Otherwise, ${\widehat{\sigma }}_{*}$ was larger than at least one of the singular values we kept. Let $i_{*}$ be the index of the frontal slice containing ${\widehat{\sigma }}_{*}$ with corresponding rank ${\rho }_{i_{*}}$. Let $c_{\mathrm{m}\mathrm{i}\mathrm{n}}$ be the smallest element contained in $C$ and, by definition, $c_{\mathrm{m}\mathrm{i}\mathrm{n}}<{\widehat{\sigma }}_{*}$. Let $i_{\mathrm{m}\mathrm{i}\mathrm{n}}$ be the index of the frontal slice containing $c_{\mathrm{m}\mathrm{i}\mathrm{n}}$ with corresponding rank ${\rho }_{i_{\mathrm{m}\mathrm{i}\mathrm{n}}}$. Note that $c_{\mathrm{m}\mathrm{i}\mathrm{n}}$ must be the smallest singular value kept from frontal slice $i_{\mathrm{m}\mathrm{i}\mathrm{n}}$. Set ${\rho }_{i_{*}}\leftarrow {\rho }_{i_{*}}+1$ and ${\rho }_{i_{\mathrm{m}\mathrm{i}\mathrm{n}}}\leftarrow {\rho }_{i_{\mathrm{m}\mathrm{i}\mathrm{n}}}-1$. Then, the error ${\Vert A-A_{\mathit{\rho }}\Vert }_F^2$ has changed by an amount $-{\widehat{\sigma }}_{*}^2+c_{\mathrm{m}\mathrm{i}\mathrm{n}}^2$.

In practice, we can decrease the implicit rank further. For convenience, assume the elements of $C$ are labeled in increasing order $\left(c_1\le c_2\le c_3\dots \right)$. Let ${\pi }_p={\sum }_{j=1}^p{c}_j^2$ for $p$ less than or equal to the cardinality of $C$. Choose the largest value of $p$ such that ${\pi }_p<{\widehat{\sigma }}_{*}^2$. Again, set ${\rho }_{i_{*}}\leftarrow {\rho }_{i_{*}}+1$ and reduce the $p$ values of ${\rho }_i$ that correspond to $c_1,\dots ,c_p$. Then, the error ${\Vert A-A_{\mathit{\rho }}\Vert }_F^2$ has changed by an amount $-{\widehat{\sigma }}_{*}^2+{\pi }_p$ and the implicit rank has decreased by $p-1$.

C. Storage Comparisons

Let us suppose that $\kappa$ is the truncation parameter for the tensor approximation and $k$ is the truncation parameter for the matrix approximation. Table 1 gives a comparison of storage for the methods we have discussed so far. Note that for the t-SVDMII it is necessary to work only in the transform domain, as moving back to the spatial domain would cause fill and unnecessary storage.

Table 1.

Comparison of storage costs for approximations of an $m\times p\times n$ tensor $A$

	Basis storage		Coefficients storage	Total implicit storage
${\mathbf{U}}_k$	$kmn$	${\mathbf{C}}_k={\mathbf{S}}_k{\mathbf{V}}_k^{\top }$	$kp$	${\mathbf{A}}_k$	$k\left(mn+p\right)$
$U_{\kappa }$	$\kappa mn$	$C_{\kappa }=S_{\kappa }{\star }_{\mathrm{\text{M}}}{V}_{\kappa }^{\top }$	$\kappa pn$	$A_k$	$\kappa \left(mn+pn\right)$ + st[$\mathbf{M}$]
${\hat{U}}_{\mathit{\rho }}$	$rm$	${\hat{C}}_{\mathit{\rho }}={\hat{S}}_{\mathit{\rho }}△{\hat{V}}_{\mathit{\rho }}^{\top }$	$rp$	$A_{\mathit{\rho }}$	$r\left(m+p\right)$ + st[$\mathbf{M}$]

Top-to-bottom: a $k$-term truncated matrix SVD expansion, a $\kappa$-term truncated t-SVDM expansion, and a $\mathit{\rho }$-multirank t-SVDMII. Recall that for the t-SVDMII, we store terms in the transform domain and the implicit rank for $A_{\mathit{\rho }}$ is $r={\sum }_{i=1}^n{\rho }_i$. The notation $st\left[\mathbf{M}\right]$ refers to the implicit storage of $\mathbf{M}$, described below.

In practice, we can omit the storage costs of $\mathbf{M}$ when using fast transform techniques, such as the DCT, DFT, or discrete wavelet transform. In the case where $\mathbf{M}$ cannot be applied by fast transform the storage for $\mathbf{M}$ for t-SVDM is bounded by the number of nonzeros in $\mathbf{M}$ for t-SVDM. As we show in Section 5D, if $\mathit{\rho }$ has many zeros (that is, many faces in the transform domain do not contribute singular values above the threshold), we can reduce the storage to $\mathtt{\text{nnz}}\left(\mathit{\rho }\right)n$.

Discussion

If we assume we do not need to store the transformation matrix $\mathbf{M}$, then if $\kappa =k$, Theorem 5.3 says the approximation error of the t-SVD is at least as good as the corresponding matrix approximation. In applications where we only need to store the basis terms, e.g., to do projections, the basis for the tensor approximation is better in a relative error sense than the basis for the matrix case, for the same storage. However, unless $n=1$, if we need to store both the basis and the coefficients, we will need more storage for the tensor case if we need to take $\kappa =k$. Fortunately, in practice ${\Vert A-A_{\kappa }\Vert }_F\le {\Vert \mathbf{A}-{\mathbf{A}}_k\Vert }_F$ for $\kappa <k$. Indeed, we already showed an example (see Example 5.2) where the error is zero for $\kappa =1$, but $k$ had to be much larger to achieve exact approximation. If $\frac{\kappa }{k}<\frac{m+\frac{p}{n}}{m+p}$, then the total implicit storage of the tensor approximation of $\kappa$ terms is less than the total storage for the matrix case of $k$ terms.

Compared to the matrix SVD, the t-SVDMII approach can provide compression for at least as good, or better, an approximation level as indicated by the theorem. Of course, $\mathbf{M}$ should be an appropriate one given the latent structure in the data. The t-SVDMII approach allows us to account for the more “important” features (e.g., low frequencies and multidimensional correlations) and therefore impose a larger truncation on the corresponding frontal faces because those features contribute more to the global approximation. Truncation of t-SVDM by a single truncation index, on the other hand, effectively treats all features equally and always truncates each ${\hat{A}}_{:,:,i}$ to $k$ terms, which depending on the choice of $\mathbf{M}$ may not be as good. This is demonstrated in Section 8.

6. Comparison to Other Tensor Decompositions

Now, we compare the ${\star }_{\mathrm{\text{M}}}$ decompositions to other types of tensor representations as described in Section 1.

A. Comparison to truncated HOSVD

In this section, we wish to show how truncated HOSVD (tr-HOSVD) can be expressed using a ${\star }_{\mathrm{\text{M}}}$-product. Then, we can compare our truncated results to the tr-HOSVD. Other truncation strategies than the one discussed below could be considered, e.g. ref. 22, but are outside the scope of this study.

The tr-HOSVD is formed by truncating to $k_1,k_2,k_3$ columns, respectively, the factor matrices $\mathbf{Q},\mathbf{W},\mathbf{Z}$ and forming the $k_1\times k_2\times k_3$ core tensor as

$$C_{\mathbf{k}}≔A{\times }_1{\mathbf{Q}}_{:,1:k_1}^{\top }{\times }_2{\mathbf{W}}_{:,1:k_2}^{\top }{\times }_3{\mathbf{Z}}_{:,1:k_3}^{\top },$$

where $\mathbf{k}$ denotes the triple $\left(k_1,k_2,k_3\right)$. The tr-HOSVD approximation then is

$$A_{\mathbf{k}}=C_{\mathbf{k}}{\times }_1{\mathbf{Q}}_{:,1:k_1}{\times }_2{\mathbf{W}}_{:,1:k_2}{\times }_3{\mathbf{Z}}_{:,1:k_3}.$$

We now prove the following theorem that shows the tr-HOSVD can be represented under ${\star }_{\mathrm{\text{M}}}$ when $\mathbf{M}={\mathbf{Z}}^{\top }$.

Theorem 6.1. Define the $n\times n$ matrix $\mathbf{M}$ as $\mathbf{M}={\mathbf{Z}}^{\top }$ (since $\mathbf{Z}$ is unitary, it follows that ${\mathbf{M}}^{-1}=\mathbf{Z}$), and define $m\times k_1\times n$ and $p\times k_2\times n$ tensors in the transform space according to

$${\hat{Q}}_{:,:,i}=\mathbf{Q},\hspace{1em}\text{and}\hspace{1em}{\hat{W}}_{:,:,i}=\mathbf{W}\hspace{1em}\text{for}\hspace{1em}i=1,\dots ,n.$$

Then $Q=\hat{Q}{\times }_3{\mathbf{M}}^{-1},W=\hat{W}{\times }_3{\mathbf{M}}^{-1}$ and it is easy to show that $Q$, $W$ are unitary tensors. Define $\hat{P}$ as the $p\times p\times n$ tensor with identity matrices on faces 1 to $k_3$ and 0 matrices from faces $k_3+1$ to $n$.

Let $C=Q_{:,1:k_1,:}^{\top }{\star }_{\mathrm{\text{M}}}A{\star }_{\mathrm{\text{M}}}{W}_{:,1:k_2,:}$. Then

$$A_{\mathbf{k}}=Q_{:,1:k_1,:}{\star }_{\mathrm{\text{M}}}C{\star }_{\mathrm{\text{M}}}{W}_{:,1:k_2,:}^{\top }{\star }_{\mathrm{\text{M}}}P.$$

Proof: First consider

$$\begin{array}{cc}\hfill C& ≔A{\times }_1{\mathbf{Q}}_{:,1:k_1}^{\top }{\times }_2{\mathbf{W}}_{:,1:k_2}^{\top }{\times }_3{\mathbf{Z}}^{\top }\hfill \\ \hfill & =\left(A{\times }_3{\mathbf{Z}}^{\top }\right){\times }_1{\mathbf{Q}}_{:,1:k_1}^{\top }{\times }_2{\mathbf{W}}_{:,1:k_2}^{\top }\hfill \\ \hfill & =\hat{A}{\times }_1{\mathbf{Q}}_{:,1:k_1}^{\top }{\times }_2{\mathbf{W}}_{:,1:k_2}^{\top },\hfill \end{array}$$ [12]

using properties of modewise products (see [12]). From the definitions of the modewise product the $i^{th}$ face of $C$ as defined via [12] is ${\mathbf{Q}}_{:,1:k_1}^{\top }{\hat{A}}_{:,:,i}{\mathbf{W}}_{:,1:k_2}$. However, this means that we can equivalently represent $C$ as $C=Q_{:,1:k_1,:}^{\top }{\star }_{\mathrm{\text{M}}}A{\star }_{\mathrm{\text{M}}}{W}_{:,1:k_2,:}$.

Now $B≔Q_{:,1:k_1,:}{\star }_{\mathrm{\text{M}}}C{\star }_{\mathrm{\text{M}}}{W}_{:,1:k_2,:}^{\top }$ implies ${\hat{B}}_{:,:,i}={\mathbf{Q}}_{:,1:k_1}{\hat{C}}_{:,:,i}{\mathbf{W}}_{:,1:k_2}^{\top },i=1,\dots ,n$. However, since ${\hat{C}}_{:,:,i}={\mathbf{Q}}_{:,1:k_1}^{\top }{\hat{A}}_{:,:,i}{\mathbf{W}}_{:,1:k_2}$ for $i=1,\dots ,n$, we only need to zero-out the last $k_3+1:n$ frontal slices of $B$ to get to $A_{\mathbf{k}}$, which we do by taking the ${\star }_{\mathrm{\text{M}}}$-product with $P$ on the right.

Note that in our theorem we assume $\mathbf{Z}$ is square, but the transformation $\mathbf{Z}$ is effectively truncated by postmultiplying by $P$, which allows for the ${\star }_{\mathrm{\text{M}}}$-product presentation of the tr-HOSVD. Thus, we now can compare the theoretical results from tr-HOSVD to our truncated methods.

Theorem 6.2. Given the tr-HOSVD approximation $A_{\mathbf{k}}$, for $\kappa ≔min\left(k_1,k_2\right)$,

$${\Vert A-A_{\kappa }\Vert }_F\le {\Vert A-A_{\mathbf{k}}\Vert }_F$$

with equality only if $A_{\mathbf{k}}=A_{\kappa }$.

Proof: Note that the t-ranks of $Q$ and $W{\star }_{\mathrm{\text{M}}}P$ are $k_1$ and $k_2$, respectively. Since $C$ is $k_1\times k_2\times n$, its t-rank cannot exceed $\kappa ≔min\left(k_1,k_2\right)$. As such, we know $A_{\mathbf{k}}$ can be written as a sum of $\kappa$ outer-products of tensors under ${\star }_{\mathrm{\text{M}}}$, and the result follows given the optimality of $A_{\kappa }$.

Corollary 6.3. Given the tr-HOSVD approximation $A_{\mathbf{k}}$, for $\kappa ≔min\left(k_1,k_2\right)$ there exists $\gamma$ such that $A_{\mathit{\rho }}$ returned by Algorithm 3 has implicit rank less than or equal to $A_{\kappa }$ and

$${\Vert A-A_{\mathit{\rho }}\Vert }_F\le {\Vert A-A_{\kappa }\Vert }_F\le {\Vert A-A_{\mathbf{k}}\Vert }_F.$$

Note this is independent of the choice of $k_3$: The best tr-HOSVD approximation will occur when $k_3=n$.

While this has theoretical value, it begs the question of whether or not $\mathbf{Z}$ needs to be stored explicitly. When $\mathbf{M}$ is a matrix that can be applied quickly without explicit storage, such as a discrete cosine transform, this is not a consideration. We will say more about this at the end of this section.

B. Comparison to TT-SVD

The TT-SVD was introduced in ref. 9. In recent years, the (truncated) TT-SVD has been used successfully for compressed representations of matrix operators in high-dimensional spaces and for data compression.

For a third-order tensor, the first step of the TT-SVD algorithm is to perform a truncated matrix SVD of one unfolding. Thus, the results depend upon the choice of unfolding. In order to compare with our method, we shall choose the mode-2 unfolding.^# We note that ${\mathbf{A}}^{\top }=A_{\left(2\right)}$, and since $\mathbf{A}=\mathtt{\text{unfold}}\left(A\right)=\mathbf{U}\mathbf{S}{\mathbf{V}}^{\top }$ has been used already, we observe $A_{\left(2\right)}=\mathbf{V}\mathbf{S}{\mathbf{U}}^{\top }$ is the SVD of the mode-2 unfolding. The TT-SVD algorithm proceeds as follows for third-order tensors:

1. Truncate $\mathbf{V}\left(\mathbf{S}{\mathbf{U}}^{\top }\right)$, to $k$ terms, keep ${\mathbf{V}}_k$ (typically, as a $1\times p\times k$ tensor). Let $\mathbf{C}≔{\mathbf{S}}_k{\mathbf{U}}_k^{\top }$, and note it is $k\times mn$.

2. Let $\mathcal{L}\left(\mathbf{C}\right)$ denote $\mathbf{C}$ reshaped to an $mk\times n$ matrix. Compute the SVD $\mathcal{L}\left(\mathbf{C}\right)=\mathbf{W}\mathbf{D}{\mathbf{Q}}^{\top }$ and truncate to $q$ terms. The $mk\times q\hspace{0.17em}{\mathbf{W}}_q$ is folded into a $k\times m\times q$ tensor, and the remaining ${\mathbf{E}}_q≔{\mathbf{D}}_q{\mathbf{Q}}_q^{\top }$ is stored as a $q\times n\times 1$ tensor.

The truncations $k,q$ are chosen based on a user-defined threshold $ϵ$ such that ${\Vert A-\tilde{A}\Vert }_F\le ϵ{\Vert A\Vert }_F$, where $\tilde{A}$ is the TT-SVD approximation. The storage cost of this implicit tensor approximation is $pk+mkq+qn$ numbers.

Theorem 6.4. Let $\tilde{A}$ denote the $\left(k,q\right)$ truncated TT-SVD approximation, where $k,q$ correspond to the truncation indices that satisfy the user input threshold $ϵ$. Then,

$${\Vert A-A_k\Vert }_F^2\le {\Vert A-\tilde{A}\Vert }_F^2,$$

with strict inequality if $q$ is less than the rank of $\mathcal{L}\left(\mathbf{C}\right)$. Furthermore, there exists $\rho$ such that

$${\Vert A-A_{\mathit{\rho }}\Vert }_F^2\le {\Vert A-\tilde{A}\Vert }_F^2.$$

Proof: Let $A_{\left(2\right),k}={\mathbf{V}}_k\mathbf{C}$ and let $r_1$ be the rank of $\mathbf{C}$ and $r_2$ be the rank of $\mathcal{L}\left(\mathbf{C}\right)$. The steps in the algorithm imply that ${\tilde{A}}_{\left(2\right)}={\mathbf{V}}_k{\mathcal{L}}^{-1}\left({\mathbf{E}}_q\right)$. Thus,

$$\begin{array}{ll}\hfill & {\Vert A_{\left(2\right)}-{\tilde{A}}_{\left(2\right)}\Vert }_F^2={\Vert A_{\left(2\right)}-A_{\left(2\right),k}+A_{\left(2\right),k}-{\tilde{A}}_{\left(2\right)}\Vert }_F^2\hfill \\ \hfill & \hspace{1em}={‖\left[\begin{array}{c}\hfill \mathbf{C}-{\mathcal{L}}^{-1}\left({\mathbf{E}}_q\right)\hfill \\ \hfill {\mathbf{S}}_{k+1:r_1,k+1:r_1}{\mathbf{U}}_{:,k+1:r_1}^{\top }\hfill \end{array}\right]‖}_F^2={\sum }_{i=k+1}^{r_1}{\sigma }_i^2+{\Vert \mathcal{L}\left(\mathbf{C}\right)-{\mathbf{E}}_q\Vert }_F^2\hfill \\ \hfill & \hspace{1em}={\Vert \mathbf{A}-{\mathbf{A}}_k\Vert }_F^2+{\sum }_{j=q+1}^{r_2}{d}_i^2,\hfill \end{array}$$

by employing unitary-invariance of the Frobenius norm.

Applying Theorem 5.5,

$$\begin{array}{ll}\hfill {\Vert A-A_{\mathit{\rho }}\Vert }_F^2& \le {\Vert A-A_k\Vert }_F\le {\Vert \mathbf{A}-{\mathbf{A}}_k\Vert }_F^2+{\sum }_{j=q+1}^{r_2}{d}_i^2\hfill \\ \hfill & ={\Vert A_{\left(2\right)}-{\tilde{A}}_{\left(2\right)}\Vert }_F^2={\Vert A-\tilde{A}\Vert }_F^2.\hfill \end{array}$$

The inequality is strict if $q<r_2$; it is also strict independent of $q$ if $\Vert A-A_k{\Vert }_F<\Vert \mathbf{A}-{\mathbf{A}}_k{\Vert }_F$ which, as noted earlier, is achievable and is almost always the case in practice.

C. Approximation in CP Form

The purpose of the interpretation of our decomposition in CP form is with an eye toward an implicit compressed representation discussed in the next subsection. Consider the formation of $A_{\mathit{\rho }}$ in Algorithm 3. In step 10, we keep ${\rho }_i$ terms in the matrix SVD for frontal slice $i$ of $\hat{A}$. So

$${\hat{A}}_{\mathit{\rho }}={\sum }_{i=1}^n{\sum }_{j=1}^{{\rho }_i}{\hat{U}}_{:,j,i}○{\hat{V}}_{:,j,i}○{\widehat{\sigma }}_j^{\left(i\right)}{\mathbf{e}}_i,$$ [13]

and ${\mathbf{e}}_i$ is the $i^{th}$ column of the $n\times n$ identity matrix. However, $A_{\mathit{\rho }}={\hat{A}}_{\mathit{\rho }}{\times }_3{\mathbf{M}}^{-1}$. Thus,

$$A_{\mathit{\rho }}={\sum }_{i=1}^n{\sum }_{j=1}^{{\rho }_i}{\widehat{\sigma }}_j^{\left(i\right)}\left({\hat{U}}_{:,j,i}○{\hat{V}}_{:,j,i}○\left({\mathbf{M}}^{-1}{\mathbf{e}}_i\right)\right).$$

Next, concatenate vectors to form three matrices of $r$ columns as follows. Let $\mathbf{U}≔\left[{\hat{U}}_{:,1:{\rho }_1,1},\dots ,{\hat{U}}_{:,1:{\rho }_n,n}\right]$ similarly for $\mathbf{V}$. Let

$$\mathbf{W}≔\left[\underset{{\rho }_1}{\underbrace{{\tilde{\mathbf{M}}}_{:,1},\dots ,{\tilde{\mathbf{M}}}_{:,1}}},\dots ,\underset{{\rho }_n}{\underbrace{{\tilde{\mathbf{M}}}_{:,n},\dots ,{\tilde{\mathbf{M}}}_{:,n}}}\right],$$ [14]

where $\tilde{\mathbf{M}}={\mathbf{M}}^{-1}$, so that $\mathbf{W}$ also has $r$ columns, with repeats as dictated by the ${\rho }_i$. Let $\mathbf{S}$ contain the ${\hat{\sigma }}_{{\rho }_i}^{\left(j\right)}$. Then

$$A_{\mathit{\rho }}=\left[\left[\mathbf{S};\mathbf{U},\mathbf{V},\mathbf{W}\right]\right].$$

Note that if $\mathbf{M}$ is an orthogonal (unitary) matrix then ${\mathbf{M}}^{-1}={\mathbf{M}}^{\top }$ (${\mathbf{M}}^{\mathrm{H}}$ for complex $\mathbf{M}$) the elements of the rightmost matrix are columns of the transpose of $\mathbf{M}$ and hence have unit norm. The columns of $\mathbf{U},\mathbf{V}$ also have unit norm.

D. Discussion: Storage of the Transformation Matrix

From [14], only columns $i$ of $\tilde{\mathbf{M}}$ for which ${\rho }_i>0$ need to be stored (along with a storage-negligible vector of integer pointers) so storage drops to $O\left(n\left|\mathit{\rho }\right|\right)$; see Table 2. In the numerical results on hyperspectral data, this fact, in combination with Corollary 6.3, allows the t-SVDMII to give results superior to the tr-HOSVD in both error and storage.

Table 2.

Summary of st$\left[\mathbf{M}\right]$

	t-SVDM	t-SVDMII
Fast transform $\mathbf{M}$	0	0
Unstructured $\mathbf{M}$	nnz($\mathbf{M}$)	$min$ (nnz$\left(\mathit{\rho }\right)n$, nnz($\mathbf{M}$))

7. Multisided Tensor Compression

Consider the mapping ${\mathbb{C}}^{m\times p\times n}\to {\mathbb{C}}^{n\times p\times m}$ induced by matrix-transposing (without conjugation) each of the $p$ lateral slices.^∥ We use a superscript of $\mathrm{P}$ to denote such a permuted tensor:

$$A^{\mathrm{\text{P}}}=\mathtt{\text{permute}}\left(A,\left[\mathrm{3,2,1}\right]\right),\hspace{1em}{\left(A^{\mathrm{\text{P}}}\right)}^{\mathrm{\text{P}}}=A.$$

In this section, we define techniques for compression using both orientations of the lateral slices in order to ensure a more balanced approach to the compression of the data.

A. Optimal Convex Combinations

Given $A=U{\star }_{\mathrm{\text{M}}}S{\star }_{\mathrm{\text{M}}}{V}^H$ and $A^{\mathrm{\text{P}}}=W{\star }_{\mathrm{\text{B}}}D{\star }_{\mathrm{\text{B}}}{Q}^H$, compress each and form

$$\alpha \left(U_{k_1}{\star }_{\mathrm{\text{M}}}{S}_{k_1}{\star }_{\mathrm{\text{M}}}{V}_{k_1}^H\right)+\left(1-\alpha \right){\left(W_{k_2}{\star }_{\mathrm{\text{B}}}{D}_{k_2}{\star }_{\mathrm{\text{B}}}{Q}_{k_2}^H\right)}^{\mathrm{\text{P}}}.$$

Observe that $\text{unfold}\left(A^{\mathrm{\text{P}}}\right)=\mathbf{P}\mathbf{A}$, where $\mathbf{A}=\text{unfold}\left(A\right)$ as before and $\mathbf{P}$ denotes a stride permutation matrix. Since $\mathbf{P}$ is orthogonal, the singular values and right singular vectors of $\mathbf{A}$ are the same as those of $\mathbf{P}\mathbf{A}$, and the left singular vectors are row permuted by $\mathbf{P}$. For a truncation parameter $r$,

$$\Vert A^{\mathrm{\text{P}}}-{\left(A^{\mathrm{\text{P}}}\right)}_r{\Vert }_F\le \Vert \mathbf{A}-{\mathbf{A}}_r{\Vert }_F.$$

It follows that for $\beta ≔1-\alpha$

$$\begin{array}{cc}\hfill \Vert A-{\left(\alpha A_{k_1}+\beta {\left(A_{k_2}^{\mathrm{\text{P}}}\right)}^{\mathrm{\text{P}}}\right)\Vert }_F& \le \alpha \Vert \mathbf{A}-A_{k_1}{\Vert }_F+\beta \Vert \mathbf{A}-A_{k_2}{\Vert }_F\hfill \\ \hfill & \le \Vert \mathbf{A}-{\mathbf{A}}_{min\left(k_1,k_2\right)}{\Vert }_F.\hfill \end{array}$$

Similarly for the optimal t-SVDMII approximations, $A_{\mathit{\rho },\mathit{\delta }}≔\alpha A_{\mathit{\rho }}+\beta {\left(A_{\mathit{\delta }}^{\mathrm{\text{P}}}\right)}^{\mathrm{\text{P}}}$, where $\mathit{\rho },\mathit{\delta }$ are the multiindices for each orientation, respectively, which may have been determined with different energy levels. From Theorem 3.8,

$${\Vert A-A_{\mathit{\rho },\mathit{\delta }}\Vert }_F^2={\alpha }^2{\sum }_{i=1}^n{\sum }_{j={\rho }_i+1}^{r_i}{\left({\widehat{\sigma }}_j^{\left(i\right)}\right)}^2+{\beta }^2{\sum }_{k=1}^m{\sum }_{j={\delta }_k+1}^{{\tilde{r}}_k}{\left({\tilde{\sigma }}_j^{\left(k\right)}\right)}^2,$$

where $r_i$ is the rank of ${\hat{A}}_{:,:,i},i=1:n$ under ${\star }_{\mathrm{\text{M}}}$, ${\tilde{r}}_k$ is the rank of ${\widehat{A^{\mathrm{p}}}}_{:,:,k}$ under ${\star }_{\mathrm{\text{B}}}$ and ${\tilde{\sigma }}_j^{\left(k\right)}$ are for $\widehat{A^{\mathrm{p}}}$ under ${\star }_{\mathrm{\text{B}}}$ as well.

B. Sequential Compression

Sequential compression is also possible, as described in ref. 23. A second tensor SVD is applied to $C^{\mathrm{p}}$, making each step locally optimal. Due to space constraints, we will not elaborate on this further here.

8. Numerical Examples

In the following discussion, the compression ratio (CR) is defined as the number of floating point numbers needed to store the uncompressed data divided by the number of floating point numbers needed to store the compressed representation (in its implicit form). Thus, the larger the ratio, the better the compression. The relative error (RE) is the ratio of the Frobenius-norm difference between the original data and the approximation over the Frobenius-norm of the original data.

A. Compression of Yale B Data

In this section, we show the power of compression for the t-SVDMII approach with appropriate choice of $\mathbf{M}$, that is, $\mathbf{M}$ exploits structure inherent in the data. We create a third-order tensor from the Extended Yale B face database (24) by putting the training images in as lateral slices in the tensor. Then, we apply Algorithm 3, varying $\mathit{\rho }$, for four different choices of $\mathbf{M}$: We choose $\mathbf{M}$ as a random orthogonal matrix, we use $\mathbf{M}$ as an orthogonal wavelet matrix, we use $\mathbf{M}$ as the unnormalized DCT matrix, and we form $\mathbf{M}$ in a data-driven approach by taking the transpose of the left factor matrix $\mathbf{Z}$ of the mode-3 unfolding. We have chosen to use a random orthogonal matrix in this experiment to show that the compression power is relative to structure that is induced through the choice of $\mathbf{M}$, so we do not expect, nor do we observe, value in choosing $\mathbf{M}$ to be random. In Fig. 3, we plot the CR against the RE in the approximation. We observe that for RE on the order of 10 to 15% the margins in compression achieved by the t-SVDMII for both the DCT, the wavelet transform, and the data-driven transform vs. treating the data in either matrix form, or in choosing a transform that—like the matrix case—does not exploit structure in the data, are quite large. To evaluate only compressibility associated with the transformation we do not count the storage of $\mathbf{M}$.

Fig. 3.

Illustration of the compressive power of the t-SVDIIM in Algorithm 3 for appropriate choices of $\mathbf{M}$. Far more compression is achieved using either the DCT or wavelet transform to define ${\star }_{\mathrm{\text{M}}}$, as well as defining ${\star }_{\mathrm{\text{M}}}$ in a data-driven approach since they capitalize on structural features in the data.

B. Video Frame Data

For this experiment, we use video data available in MATLAB.** The video consists of 120, $120\times 160$ frames in grayscale. The camera is positioned near one spot in the road, and cars travel on that road (more or less from the top to the bottom as the frames progress), so the only changes per frame are cars entering and disappearing.

We compare the performance of our truncated t-SVDMII, for $\mathbf{M}$ being the DCT matrix, against the truncated matrix and truncated HOSVD approximations. We orient the frames as transposed lateral slices in the tensor to confine the change in one lateral slice to another to rows where there is car movement.^†† Thus, $A$ is $160\times 120\times 120$.

With both the truncated t-SVDMII and truncated matrix SVD approaches we can truncate based on the same energy value. Thus, we get RE in our respective approximations with about the same value, and then we can compare the relative compression. Alternatively, we can fix our energy value and compute our truncated t-SVDMII and find its CR. Then, we can compute the truncated matrix SVD approximation with similar CR and compare its RE to the tensor-based approximation. We show some results for each of these two ways of comparison.

There are many ways of choosing the truncation 3-tuple for HOSVD. Trying to choose a 3-tuple that has a comparable relative approximation to our approach would be cumbersome and would still leave ambiguities in the selection process. Thus, we employ two truncation methods that yield an approximation best matching the CRs of our tensor approximation. The indices are chosen as follows: 1) Compress only on the second mode (i.e., change $k_2$, fix $k_1=m,k_3=n$) and 2) choose truncation parameters on dimensions such that the modewise compression to dimension ratios are about the same. The second option amounts to looping over $k_2=1,\dots ,p$, setting $k_1=\lfloor \frac{k_{2}m}{n}\rfloor$, and $k_3=k_2$. Thus, it is possible to compute the CR for the tr-HOSVD based on the dimension in advance to find the closest match to the desired compression levels. The results are given in Table 3.

Table 3.

Video experiment results

	t-SVDMII	Matrix		tr-HOSVD
	${\gamma }_1=.998$	${\gamma }_1$	CR	$\left(m,k_2,n\right)$	$\left(k_1,k_2,k_2\right)$
CR	4.76	1.83	4.76	4.95	4.90
RE	0.044	0.045	0.093	0.098	0.065
	${\gamma }_2=.996$	${\gamma }_1$	CR	$\left(m,k_2,n\right)$	$\left(k_1,k_2,k_2\right)$
CR	10.10	2.54	10.87	10.75	10.42
RE	0.063	0.064	0.120	0.125	0.090

Matrix-based compression can be for predefined relative energy $\gamma$ (which affects RE) or set to achieve desired compression, so we performed both. For experiment 1 with ${\gamma }_1$, the $k_2$ and $\left(k_1,k_2,k_3\right)$ values that gave the same compression results were 25 and (92, 69, 69), respectively; for the second experiment these were 11 and (70, 53, 53), respectively. Truncations for the matrix case were 65 and 25 for the ${\gamma }_1$ experiment and 47 and 11 for the second.

We can also visualize the impact of the compression schemes. In Fig. 4 we give the corresponding reconstructed representations of frame 10 and 54 for the four methods under comparable CR for the second $\gamma$ (columns 2 and 4 through 6, second row-block of the table, i.e., the results corresponding to the most compression). Cars disappear altogether and/or artifacts make it appear as though cars may be in the frame when they are not—at these compression levels, the matrix and tr-HOSVD suffer from a ghosting effect.

Fig. 4.

Various reconstructions for frames 10 and 54 (Left and Right). Top to bottom: Original, tr-tSVDMII with ${\gamma }_2=0.996$, tr-Matrix for same ${\gamma }_2$, tr-HOSVD $\left(\mathrm{70,53,53}\right)$, and tr-HOSVD$\left(m,25,n\right)$. See Table 3 for numerical results.

C. Hyperspectral Image Compression

We compare the performance of the t-SVDMII and the truncated HOSVD based on our results in Corollary 6.3 on hyperspectral data. The hyperspectral data comes from flyover images of the Washington, DC mall with a sensor that measured 210 bands of visible infrared spectra (25). After removing the opaque atmospheric levels the data consist of 191 images corresponding to different wavelengths and each image is of size 307 × 1,280.

To reduce the computational cost for this comparison, and to allow for equal truncation in both spatial dimensions, we resize the images using MATLAB imresize to $300\times 300$ images. We store the data in a tensor $A$ of size $300\times 300\times 191$, where the third dimension contains the wavelengths. We choose this orientation because we use $\mathbf{M}={\mathbf{Z}}^{\top }$, where $\mathbf{Z}$ contains the left singular vectors of $A_{\left(3\right)}$. We can expect a high correlation and hence high compressibility (i.e., many ${\rho }_i$ will be 0) along the third dimension as they correspond to exactly the same spatial location at each wavelength.

For a fair comparison, we relate the t-SVDMII, the tr-HOSVD, and the TT-SVD via the truncation parameter $\kappa$ of the truncated t-SVDM $A_{\kappa }$ using $\mathbf{M}={\mathbf{Z}}^{\top }$. For the t-SVDMII based on Theorem 5.5 we use the energy parameter $\gamma ={\Vert A_{\kappa }\Vert }_F^2/{\Vert A\Vert }_F^2$. For the tr-HOSVD based on Theorem 6.2 we use a variety of multilinear truncations $\left(\kappa ,k_2,k_3\right)$ and $\left(k_1,\kappa ,k_3\right)$ for various choices of $k_1,k_2\le \kappa$ and $k_3$. For the TT-SVD based on Theorem 6.4 we choose the accuracy threshold $ϵ$ such that the truncation of the first unfolding is $\kappa$. Note we first permute the tensor $A$ so that the first TT-SVD unfolding is the mode-2 unfolding. For completeness, we include the matrix SVD ${\mathbf{A}}_{\kappa }$ and apply Theorem 5.3. We display the RE-vs.-CR results in Fig. 5.

Fig. 5.

Performance of various representations the hyperspectral data compression. Each shape represents a different method. The methods are related by the truncation parameter $\kappa$ and each color represents a different choice of $\kappa$. The HOSVD results depicted by the squares correspond to multilinear rank $\left(\kappa ,\kappa ,n\right)$ and maximally compresses the first two dimensions such that Theorem 6.2 applies and does not compress in the third dimension to minimize the RE. The HOSVD results depicted by the asterisks are many different choices of multilinear rank $\left(k_1,k_2,k_3\right)$ such that Theorem 6.2 applies. For each $\kappa$, we depict all possible combinations of $k_3=\mathrm{10,30},\dots ,190$ and either $k_1=\kappa$ and $k_2$ is any larger truncation value, or vice versa. The RE depends most on the value of $\kappa$ and the CR depends most on the value of $k_3$. This is due to the orientation and structure of the data; the third dimension is highly compressible and thus truncating $k_3$ does not significantly worsen the approximation. Comparing shapes, the t-SVDMII produces the best results, clearly outperforming all cases of the SVD and HOSVD and outperforming the TT-SVD, most noticeably for better approximations (lower RE). Comparing colors, for a fixed value of $\kappa$ the t-SVDMII produces the smallest RE, providing empirical support for the theory.

The results use the t-SVDMII storage method described in Section 6D. Even when counting the storage of $\mathbf{M}$, the t-SVDMII outperforms the matrix SVD and the tr-HOSVD and is highly competitive with the TT-SVD and outperforms for larger values of $\kappa$ (i.e., less truncation, but better approximation quality). For example, if we compare the t-SVDMII result with $\kappa =250$ (dark gray diamond) with the TT-SVD result with $\kappa =290$ (light gray circle), we see that the t-SVDMII produces a better approximation for roughly the same CR. The color-coded results in Fig. 5 follow directly from writing the tr-HOSVD in our ${\star }_{\mathrm{\text{M}}}$-framework and the optimality condition in Theorem 6.2 and follow from relating the TT-SVD to the matrix SVD and using our Eckart–Young result in Theorem 5.3. Specifically, we see that for each choice of $\kappa$ the t-SVDMII approximation has a smaller RE than all other methods.

We also note that for the HOSVD in Fig. 5 a large number of combinations of multirank parameters $\left(k_1,k_2,k_3\right)$ are possible and can drastically change the approximation–compression relationship. Finding a reasonable tr-HOSVD multirank via trial and error may not be feasible in practice. In comparison, there is only one parameter for the t-SVDMII and TT-SVD which impacts the approximation–compression relationship more predictably, particularly because the approximation quality is known a priori.

D. Extension to Four Dimensions and Higher

Although the algorithms and optimality results were described for third-order tensors, the algorithmic approach and optimality theory can be extended to higher-order tensors since the definitions of the tensor-tensor products extend to higher-order tensors in a recursive fashion, as shown in ref. 26. Ideas based on its truncation for compression and high-order tensor completion and robust PCA can be found in the literature (19, 27). As noted in ref. 11, a similar recursive construct can be used for higher-order tensors for the ${\star }_{\mathrm{\text{M}}}$-product, or different combinations of transform-based products can be used along different modes. We give the four-dimensional t-SVDMII algorithm in ref. 23. Due to space constraints, we do not discuss this further here.

9. Conclusions and Ongoing Work

We have proved Eckart–Young theorems for the t-SVDM and t-SVDMII. Importantly, we showed that the truncated t-SVDM and t-SVDMII representations give better approximation to the data than the corresponding matrix-based SVD compression. Although the superiority of tensor-based compression has been observed by many in the literature, our result proves this theoretically. By interpreting the HOSVD in the ${\star }_{\mathrm{\text{M}}}$-framework, we developed the relationship between the HOSVD and the t-SVDM and t-SVDMII and then applied our Eckart–Young results to show how our tensor approximations improve on the Frobenius norm error. We were also able to apply our Eckart–Young theorems to compare Frobenius norm error with that of a TT-SVD approximation.

We briefly considered both multisided compression and extensions of our work to higher-order tensors. In future work, means for optimizing $\alpha$, $k$, and $j$ such that the upper bound on the error is minimized while minimizing the total storage will be investigated.

The choice of $\mathbf{M}$ defining the tensor-tensor product should be tailored to the data for best compression. Therefore, consideration as for how to best design $\mathbf{M}$ to suit the dataset shall be pursued in future work.

Data Availability

There are no data underlying this work.