On the interconnection between the higher-order singular values of real tensors

Abstract

A higher-order tensor allows several possible matricizations (reshapes into matrices). The simultaneous decay of singular values of such matricizations has crucial implications on the low-rank approximability of the tensor via higher-order singular value decomposition. It is therefore an interesting question which simultaneous properties the singular values of different tensor matricizations actually can have, but it has not received the deserved attention so far. In this paper, preliminary investigations in this direction are conducted. While it is clear that the singular values in different matricizations cannot be prescribed completely independent from each other, numerical experiments suggest that sufficiently small, but otherwise arbitrary perturbations preserve feasibility. An alternating projection heuristic is proposed for constructing tensors with prescribed singular values (assuming their feasibility). Regarding the related problem of characterising sets of tensors having the same singular values in specified matricizations, it is noted that orthogonal equivalence under multilinear matrix multiplication is a sufficient condition for two tensors to have the same singular values in all principal, Tucker-type matricizations, but, in contrast to the matrix case, not necessary. An explicit example of this phenomenon is given.

Introduction and problem statement

A space ${\mathbb{R}}^{n_1}\otimes \cdots \otimes {\mathbb{R}}^{n_d}$ of higher-order tensors is isomorphic to many different matrix spaces of the form $({⨂}_{j\in t}{\mathbb{R}}^{n_j})\otimes ({⨂}_{i\notin t}{\mathbb{R}}^{n_i})$ where $t⊊\{1,\cdots ,d\}$, $|t|\ge 1$. Concretely, when identifying tensors with d-dimensional arrays of coordinates with respect to an orthonormal tensor product basis, such an isomorphism is realized by reshaping the array into a matrix. The directions in t indicate the multi-indices for the rows of the resulting matrix, while the other directions are used for the columns. All these different matricizations (also called unfoldings or reshapes in the literature) of the tensor carry some spectral information in form of their singular value decompositions.

For subsets of t that are part of a dimension partition tree, the column spaces of the corresponding matricizations satisfy certain nestedness properties that form the basis for important subspace based low-rank tensor decompositions like the Tucker format [22], the hierarchical Tucker (HT) format  [7, 9], or the tensor train (TT) format [18, 19]. As a by-product, the ranks $r_t$ of the corresponding matricizations, that is, the number of nonzero singular values, are estimated as $r_t\le r_{t_1}\cdots r_{t_s}$, where $t=t_1\cup \cdots \cup t_s$ is a disjoint partition. In contrast, the interconnections between the singular values themselves have not been studied so far.

At first sight, the singular values of different matricizations could be considered as unnatural or artificial characteristics for tensors, as they ignore their multilinear nature. However, as it turns out, they provide crucial measures for the approximability of tensors in the aforementioned low-rank subspace formats. In the pioneering work [3] the higher-order singular value decomposition has been defined, and it has been shown how it can be practically used to obtain quasi-optimal low-rank approximations in the Tucker format with full error control. The approximation is obtained by an orthogonal projection on the tensor product of subspaces spanned by the dominating singular vectors of the corresponding matricizations in ${\mathbb{R}}^{n_j}\otimes ({⨂}_{i\ne j}{\mathbb{R}}^{n_i})$ (i.e. corresponding to the choices $t=\{j\}$ for $j=1,\cdots ,d$). An upper bound of the squared error is then given by the sum of squares of all deleted singular values in all directions. Later, variants of such truncation procedures have been obtained for the TT format  [16, 17] and the HT format [7] with similar error bounds, but involving singular values of some other matricizations of the tensor.

Building on these available, quasi-optimal bounds for low-rank approximations via higher-order versions of SVD truncation, it is understandable that quite some theorems have been stated making simultaneous assumptions on the singular values of certain matricizations of a tensor. This concerns stability of low-rank ODE integrators [12, 14], local convergence of optimization algorithms [20], or approximability by low-rank tensor formats [1], to name a few. Assumed properties of interest are decay rate of and gaps between the singular values, for instance. A principal task would then be to give alternative descriptions of classes of tensors satisfying such assumptions to prevent tautological results or, in worst case, void statements. But this task has turned out to be notoriously difficult. For tensors arising from function discretization, some qualitative statements about the decay of singular values can be obtained from their regularity using explicit analytic approximation techniques by tensor products of (trigonometric) polynomials or wavelets, exponential sum, or cross approximation; see [8, 21] and references therein. But these qualitative implications on the decay of singular values obtained from explicit separable approximations can rarely be made quantitatively precise, for instance, if they contain unknown constants, and also provide little insight on the actual interconnection between different matricizations.

In its purest form the question we are interested in is very simple to state. Given prescribed singular values for some matricizations (having, e.g., some favourable properties), does there exist at all a tensor having these singular values? For a matrix this is of course very easy to answer by simply constructing a diagonal matrix. For tensors it turns out to be quite difficult, and seems to depend on how many matricizations are simultaneously considered.

In the present paper we study this and related questions for the singular values related to the classical higher-order SVD, that is, the singular values of the principal, Tucker-type matricizations that separate single indices $t=\{j\}$. We call the collection of the d corresponding singular value vectors the higher-order singular values, see Definition 1.1 below. Compared to other systems of matricizations, this framework is the historically most important. It also appears to be the simplest, partly because it is, at least to some extent, very conveniently possible to manipulate the principal matricizations simultaneously via multilinear matrix multiplications. Yet, even in this case, the obtained results remain fragmentary and far from complete. Nevertheless, we consider them as valuable first steps toward future investigations of this important and fascinating subject. Our contributions are as follows.

• We show that not all configurations of higher-order singular values can be feasible. The proof is nonconstructive (Sect. 3.1).

• However, conducted numerical experiments suggest that the singular values for different matricizations are, except for degenerate situations, locally independent from each other. That is, in the neighbourhood of a tensor it is possible to slightly perturb, say, only the singular values of the first matricization, while maintaining the singular values of the other ones. This is fundamentally different from the matrix case, since the singular values of a matrix are always the same as the ones of its transpose. However, currently this remains an unproved conjecture Sect. 3.2).

• We propose the method of alternating projections as a heuristic to construct (approximately) tensors with prescribed singular values in certain matricizations (Sect. 3.3).

• The higher-order SVD (HOSVD) is a generalization of the SVD from matrices to tensors. The role of the diagonal matrix of singular values is replaced by the core tensor in the HOSVD, representing the normal form under orthogonal equivalence, and characterized by slice-wise orthogonality properties. We show manifold properties of the set of these core tensors (called HOSVD tensors) in the case of strictly decreasing and positive higher-order singular values (Sect. 2.2).

• We provide an example of two $2\times 2\times 2$ tensors having the same singular values in all three principal matricizations without being orthogonally equivalent (Sect. 2.4).

In this paper we consider real tensors for convenience. Although most concepts seem to generalize to the complex case, some care would be required, e.g., when switching from smooth manifolds to analytic ones.

The rest of this section is devoted to the precise statements of the considered problems. They require some amount of definitions and notation, which will be introduced first.

Preliminaries, definitions, notation

Let $d\ge 3$ and $n_1,\cdots ,n_d\in \mathbb{N}$. We consider the d-fold tensor product space ${\mathbb{R}}^{n_1}\otimes \cdots \otimes {\mathbb{R}}^{n_d}$ as isomorphic to the space ${\mathbb{R}}^{n_1\times \cdots \times n_d}$ of real $n_1\times \cdots \times n_d$ arrays (sometime called hyper-matrices). The entries $X_{i_1,\cdots ,i_d}$ of $\mathbf{X}\in {\mathbb{R}}^{n_1\times \cdots \times n_d}$ will be indexed by multi-indices $(i_1,\cdots ,i_d)$, with every $i_j$ taking values between 1 and $n_j$. For convenience, it will be assumed

$$\begin{array}{c}\hfill n_1\ge n_2\ge \cdots \ge n_d\end{array}$$

throughout the paper. Furthermore, we set

$$\begin{array}{c}\hfill n_j^c=\prod _{i\ne j}{n}_i.\end{array}$$

A tensor $\mathbf{X}\in {\mathbb{R}}^{n_1\times \cdots \times n_d}$ admits d principal matricizations [10, 11]

$$\begin{array}{c}\hfill M_{\mathbf{X}}^{(j)}\in {\mathbb{R}}^{n_j\times n_j^c}\end{array}$$

in which the $i_j$-th row contains all entries $X_{i_1,\cdots ,i_d}$ with fixed $i_j$, arranged in some fixed ordering with respect to the remaining multi-indices. The choice of that ordering is not important for our purposes. The matricizations realize the isomorphism between the tensor space ${\mathbb{R}}^{n_1}\otimes \cdots \otimes {\mathbb{R}}^{n_d}$ and the matrix spaces ${\mathbb{R}}^{n_j\times n_j^c}$. Note that for tensors $\mathbf{X}$ of order $d=2$ (that is, matrices), the matricizations are given by $M_{\mathbf{X}}^{(1)}=\mathbf{X}$ and $M_{\mathbf{X}}^{(2)}={\mathbf{X}}^{\mathsf{T}}$ (up to permutation).

It will further be convenient to have a notation for the Gram matrix of $M_{\mathbf{X}}^{(j)}$, which will be

$$\begin{array}{c}\hfill G_{\mathbf{X}}^{(j)}=M_{\mathbf{X}}^{(j)}{\left(M_{\mathbf{X}}^{(j)}\right)}^{\mathsf{T}}.\end{array}$$

By $\mathrm{O}(n)$ we denote the group of real orthogonal $n\times n$ matrices, by $I_n$ the $n\times n$ identity matrix. Each of the matrices $M_{\mathbf{X}}^{(j)}$ admits a singular value decomposition

$$\begin{array}{c}\hfill M_{\mathbf{X}}^{(j)}=U_{\mathbf{X}}^{(j)}{\mathrm{\Sigma }}_{\mathbf{X}}^{(j)}{\left(V_{\mathbf{X}}^{(j)}\right)}^{\mathsf{T}},\end{array}$$ 1.1

where $U_{\mathbf{X}}^{(j)}\in \mathrm{O}(n_j)$, ${(V_{\mathbf{X}}^{(j)})}^{\mathsf{T}}{V}_{\mathbf{X}}^{(j)}=I_{n_j}$, and ${\mathrm{\Sigma }}_{\mathbf{X}}^{(j)}$ is a diagonal matrix containing the mode-j singular values ${\mathit{\sigma }}_1^{(j)}\ge {\mathit{\sigma }}_2^{(j)}\ge \cdots {\mathit{\sigma }}_{n_j}^{(j)}\ge 0$ as diagonal elements. We denote ${\mathrm{\Lambda }}_{\mathbf{X}}^{(j)}={({\mathrm{\Sigma }}_{\mathbf{X}}^{(j)})}^2$. The diagonal entries of ${\mathrm{\Lambda }}_{\mathbf{X}}^{(j)}$ are the ordered eigenvalues of $G_{\mathbf{X}}^{(j)}$.

Definition 1.1

Let $\mathbf{X}\in {\mathbb{R}}^{n_1\times \cdots \times n_d}$.

1. For $j=1,\cdots ,d$, the vector

   $$\begin{array}{c}\hfill {\mathit{\sigma }}_{\mathbf{X}}^{(j)}=\mathrm{diag}\left({\mathrm{\Sigma }}_{\mathbf{X}}^{(j)}\right)=\left({\mathit{\sigma }}_1^{(j)},\cdots ,{\mathit{\sigma }}_{n_j}^{(j)}\right)\in {\mathbb{R}}_{+}^{n_j}\end{array}$$

   is called the vector of mode-j singular values. The tuple

   $$\begin{array}{c}\hfill {\mathbf{\Sigma }}_{\mathbf{X}}=\left({\mathit{\sigma }}_{\mathbf{X}}^{(1)},\cdots ,{\mathit{\sigma }}_{\mathbf{X}}^{(d)}\right)\in {\mathbb{R}}_{+}^{n_1}\times \cdots \times {\mathbb{R}}_{+}^{n_d}\end{array}$$

   is called the set of higher-order singular values of the tensor $\mathbf{X}$.
2. Correspondingly, for $j=1,\cdots ,d$, the vector

   $$\begin{array}{c}\hfill {\mathit{\lambda }}_{\mathbf{X}}^{(j)}=\mathrm{diag}\left({\mathrm{\Lambda }}_{\mathbf{X}}^{(j)}\right)=\left({\left({\mathit{\sigma }}_1^{(j)}\right)}^2,\cdots ,{\left({\mathit{\sigma }}_{n_j}^{(j)}\right)}^2\right)\in {\mathbb{R}}_{+}^{n_j}\end{array}$$

   is called the vector of mode-j Gramian eigenvalues. The tuple

   $$\begin{array}{c}\hfill {\mathbf{\Lambda }}_{\mathbf{X}}=\left({\mathit{\lambda }}_{\mathbf{X}}^{(1)},\cdots ,{\mathit{\lambda }}_{\mathbf{X}}^{(d)}\right)\end{array}$$

   is called the set of higher-order Gramian eigenvalues of the tensor $\mathbf{X}$.

3. The multilinear rank of the tensor $\mathbf{X}$ is the tuple ${\mathbf{r}}_{\mathbf{X}}=(r^{(1)},\cdots ,r^{(d)})$ with $r^{(j)}=\mathrm{rank}(M_{\mathbf{X}}^{(j)})=\mathrm{rank}(G_{\mathbf{X}}^{(j)})$ being equal to the number of nonzero entries of ${\mathit{\sigma }}_{\mathbf{X}}^{(j)}$.

4. The tensor $\mathbf{X}$ is called non-singular, if ${\mathbf{r}}_{\mathbf{X}}=(n_1,\cdots ,n_d)$.

We note that for matrices the definition of ‘non-singular’ coincides with the usual definition (in particular, it enforces $n_1=n_2$). In general, the following is true.

Proposition 1.2

There exists a non-singular tensor in ${\mathbb{R}}^{n_1\times \cdots \times n_d}$ if and only if the following compatibility conditions hold:

$$\begin{array}{c}\hfill n_j\le n_j^c,j=1,2,\cdots ,d.\end{array}$$ 1.2

In this case the set of non-singular tensors is open and dense in ${\mathbb{R}}^{n_1\times \cdots \times n_d}$.

Proof

Consider j fixed. By isomorphy and known results on matrices, it is clear that the set of all $\mathbf{X}$ with $M_{\mathbf{X}}^{(j)}$ being of rank $n_j$ is not empty, open, and dense if and only if $n_j\le n_j^c$. The set of non-singular tensors is the intersection of these sets for $j=1,\cdots ,d$. As such, it is also open and dense. $\square$

Let ${\Vert ·\Vert }_F$ denote the Frobenius norm of matrices and tensors, and ${\Vert ·\Vert }_2$ the standard Euclidean norm for vectors. Since matricization of a tensor is an isometric isomorphism in Frobenius norm, and since it holds $\Vert M_{\mathbf{X}}^{(j)}{\Vert }_F={\Vert {\mathit{\sigma }}_{\mathbf{X}}^{(j)}\Vert }_2$, an obvious observation for higher-order singular values is

$$\begin{array}{c}\hfill {\Vert {\mathit{\sigma }}_{\mathbf{X}}^{(1)}\Vert }_2=\cdots ={\Vert {\mathit{\sigma }}_{\mathbf{X}}^{(d)}\Vert }_2={\Vert \mathbf{X}\Vert }_F^{}.\end{array}$$

Therefore, we can focus in the following on tensors $\mathbf{X}$ on the unit sphere

$$\begin{array}{c}\hfill \mathcal{S}=\left\{\mathbf{X}\in {\mathbb{R}}^{n_1\times \cdots \times n_d}:{\Vert \mathbf{X}\Vert }_F=1\right\},\end{array}$$

and hence higher-order singular values $\mathbf{\Sigma }$ in the set

$$\begin{array}{c}\hfill {\mathfrak{S}}_{\ge }={\mathfrak{S}}_{\ge }(n_1,\cdots ,n_d)=S_{\ge }^{(1)}\times \cdots \times S_{\ge }^{(d)},\end{array}$$

where $S_{\ge }^{(j)}$ denotes the set of all nonnegative, decreasingly ordered vectors on the Euclidean unit sphere in ${\mathbb{R}}^{n_j}$. For most results, however, it will be necessary to further restrict to the set ${\mathcal{S}}^{\ast }$ of non-singular tensors having strictly decreasing mode-j singular values in every direction j. Therefore, we also introduce the notation

$$\begin{array}{c}\hfill {\mathfrak{S}}_{>}={\mathfrak{S}}_{>}(n_1,\cdots ,n_d)=S_{>}^{(1)}\times \cdots \times S_{>}^{(d)}\subset {\mathfrak{S}}_{\ge },\end{array}$$

where each $S_{>}^{(j)}$ contains the unit norm vectors in ${\mathbb{R}}^{n_j}$ with strictly decreasing and strictly positive entries. Then

$$\begin{array}{c}\hfill {\mathcal{S}}^{\ast }=\{\mathbf{X}\in \mathcal{S}:{\mathbf{\Sigma }}_{\mathbf{X}}\in {\mathfrak{S}}_{>}\}.\end{array}$$

Note that we do not introduce a notation for the slightly larger set of all non-singular tensors in $\mathcal{S}$. The main technical advantage of tensors in ${\mathcal{S}}^{\ast }$ is that all principal unfoldings admit essentially unique singular value decompositions.

The following two facts are useful to know, and follow immediately from the matrix case.

Proposition 1.3

The function $\mathbf{X}\mapsto {\mathbf{\Sigma }}_{\mathbf{X}}$ is continuous on $\mathcal{S}$. Assuming (1.2), the set ${\mathcal{S}}^{\ast }$ is relatively open and dense in $\mathcal{S}$.

Proof

The continuity of ${\mathbf{\Sigma }}_{\mathbf{X}}$ as a function of $\mathbf{X}$ follows by isomorphy to ${\mathbb{R}}^{n_j\times n_j^c}$ from the continuity of each ${\mathit{\sigma }}_{\mathbf{X}}^{(j)}$ as a function of $M_{\mathbf{X}}^{(j)}$. The proof that ${\mathcal{S}}^{\ast }$ is relatively open and dense in $\mathcal{S}$ is analogous to the proof of Proposition 1.2. $\square$

Problem statement

Regarding the higher-order singular values of tensors a principle question of interest is the following one.

Problem 1.4

(Feasible higher-order singular values) Given $\mathbf{\Sigma }\in {\mathfrak{S}}_{\ge }$, does there exist a tensor $\mathbf{X}\in \mathcal{S}$ such that ${\mathbf{\Sigma }}_{\mathbf{X}}=\mathbf{\Sigma }$?

Such $\mathbf{\Sigma }$ will be called a feasible. We define

$$\begin{array}{c}\hfill \mathfrak{F}=\mathfrak{F}(n_1,\cdots ,n_d)=\{\mathbf{\Sigma }\in {\mathfrak{S}}_{\ge }:\mathbf{\Sigma }\text{is feasible}\}.\end{array}$$

In this generality, Problem 1.4 appears to be quite hard, and will not be satisfactorily solved in this article. At least, as a first result, we are able to show that not all $\mathbf{\Sigma }\in {\mathfrak{S}}_{\ge }$ are feasible: $\mathfrak{F}\ne {\mathfrak{S}}_{\ge }$. The argument, however, is non-constructive, see Sect. 3.1.

A relaxed question of a more qualitative nature is the following one.

Problem 1.5

(Properties of $\mathfrak{F}$) What are the topological properties of the set $\mathfrak{F}$ as a subset of ${\mathfrak{S}}_{\ge }$? Does it, for instance, have positive (relative) Lebesgue measure?

Numerical experiments with random tensors seem to indicate that the answer to the second question could be positive when $d\ge 3$, but we are not able to prove it. So it remains a conjecture. In fact, we conjecture that for every $\mathbf{X}$ in ${\mathcal{S}}^{\ast }$ it holds that ${\mathbf{\Sigma }}_{\mathbf{X}}$ is an interior point of $\mathfrak{F}$, see Sect. 3.2. A striking implication of this conjecture is that given $\mathbf{X}\in {\mathcal{S}}^{\ast }$, its high-order singular values in different directions can be perturbed independently from each other without loosing feasibility (local independence of high-order singular values). In Sect. 3.3 we will present a heuristic approach to do this using an alternating projection method, which seems to work quite reliably for small perturbations, although we are currently neither able to prove its convergence nor that limit points must have the desired property.

To approach Problems 1.4 and 1.5, it seems useful to also study the following problem, which is of some interest in itself.

Problem 1.6

(Tensors with same higher-order singular values) Given $\mathbf{X}\in \mathcal{S}$, characterize sets of tensors having the same singular values ${\mathbf{\Sigma }}_{\mathbf{X}}$ as $\mathbf{X}$.

The corresponding equivalence classes for tensors in $\mathcal{S}$ and ${\mathcal{S}}^{\ast }$ are denoted by

$$\begin{array}{c}\hfill {\mathcal{S}}_{\mathbf{X}}=\{\mathbf{Y}\in \mathcal{S}:{\mathbf{\Sigma }}_{\mathbf{Y}}={\mathbf{\Sigma }}_{\mathbf{X}}\},{\mathcal{S}}_{\mathbf{X}}^{\ast }=\{\mathbf{Y}\in {\mathcal{S}}^{\ast }:{\mathbf{\Sigma }}_{\mathbf{Y}}={\mathbf{\Sigma }}_{\mathbf{X}}\}.\end{array}$$

The next Sect. 2 provides some results related to Problem 1.6. It is observed that orbits of orthogonally equivalent tensors provide trivial examples of subsets of tensors having the same higher-order singular values. However, other than in the matrix case, their dimension is too small to provide a complete description. Via the tool of HOSVD tensors, which serve as normal forms in the orbits of orthogonally equivalent tensors, we are able to construct an example of two tensors with the same higher-order singular values that are not orthogonally equivalent.

Tensors with the same higher-order singular values

In this section we focus on equivalence classes of tensors having the same higher-order singular values.

Orthogonally equivalent tensors

We recall a fact from matrices: Two rectangular matrices $X,Y\in {\mathbb{R}}^{n\times n^{\prime }}$, $n\le n^{\prime }$, have the same singular values, if and only if they are orthogonally equivalent, that is, if there exists $U\in \mathrm{O}(n)$ and $V\in \mathrm{O}(n^{\prime })$ such that

$$\begin{array}{c}\hfill UX{V}^{\mathsf{T}}=Y.\end{array}$$

This definition of orthogonal equivalence can be generalized to tensors using the multilinear matrix multiplication, see, e.g., [5, 13]. We consider the product unitary group

$$\begin{array}{c}\hfill \mathrm{O}(n_1\times \cdots \times n_d)=\mathrm{O}(n_1)\times \mathrm{O}(n_2)\times \cdots \times \mathrm{O}(n_d).\end{array}$$

The left action $(U^{(1)},\cdots ,U^{(d)})·\mathbf{X}$ of this group on ${\mathbb{R}}^{n_1\times \cdots \times n_d}$ is defined as the canonical action of the tensor product operator $U^{(1)}\otimes \cdots \otimes U^{(d)}$ on ${\mathbb{R}}^{n_1}\otimes \cdots \otimes {\mathbb{R}}^{n_d}$ in the sense that

$$\begin{array}{c}\hfill (U^{(1)},\cdots ,U^{(d)})·\left(\sum _{k=1}^r{x}_k^{(1)}\otimes \cdots \otimes x_k^{(d)}\right)=\sum _{k=1}^r{U}^{(1)}{x}_k^{(1)}\otimes \cdots \otimes U^{(d)}{x}_k^{(d)}.\end{array}$$

In terms of matricizations, in a slight abuse of notation, we note that

$$\begin{array}{c}\hfill M_{(U^{(1)},\cdots ,U^{(d)})·\mathbf{X}}^{(j)}=U^{(j)}{M}_{\mathbf{X}}^{(j)}{\left(U^{(1)}\otimes \cdots \otimes U^{(j-1)}\otimes U^{(j+1)}\otimes \cdots \otimes U^{(d)}\right)}^{\mathsf{T}},\end{array}$$ 2.1

cf. [8, Lemma 5.6]. In particular, since $U^{(1)}\otimes \cdots \otimes U^{(j-1)}\otimes U^{(j+1)}\otimes \cdots \otimes U^{(d)}\in \mathrm{O}(n_j^c)$, it holds

$$\begin{array}{c}\hfill G_{(U^{(1)},\cdots ,U^{(d)})·\mathbf{X}}^{(j)}=U^{(j)}{G}_{\mathbf{X}}^{(j)}{(U^{(j)})}^{\mathsf{T}}\end{array}$$ 2.2

for $j=1,\cdots ,d$. For matrices ($d=2$), these formulas define orthogonal equivalence, which motivates the following generalization.

Definition 2.1

(see [5]) Two tensors $\mathbf{X},\mathbf{Y}\in {\mathbb{R}}^{n_1\times \cdots \times n_d}$ are called orthogonally equivalent, if there exists $(U^{(1)},\cdots ,U^{(d)})\in \mathrm{O}(n_1\times \cdots \times n_d)$ such $\mathbf{Y}=(U^{(1)},\cdots ,U^{(d)})·\mathbf{X}$.

From (2.2), we draw a trivial but important conclusion.

Proposition 2.2

If two tensors are orthogonally equivalent, then they have the same higher-order singular values.

In particular, the orbit of each $\mathbf{X}$ under the group action contains only tensors with identical higher-order singular values.

Proposition 2.3

Let $\mathbf{X}\in {\mathcal{S}}^{\ast }$. Then the orbit $\mathrm{O}(n_1\times \cdots \times n_d)·\mathbf{X}$ is a locally smoothly embedded submanifold of ${\mathcal{S}}^{\ast }$ of dimension

$$\begin{array}{c}\hfill dim(\mathrm{O}(n_1\times \cdots \times n_d)·\mathbf{X})=dim(\mathrm{O}(n_1\times \cdots \times n_d))=\sum _{j=1}^d\frac{1}{2}{n}_j(n_j-1).\end{array}$$

Proof

We write $\mathrm{O}$ instead of $\mathrm{O}(n_1\times \cdots \times n_d)$. Consider the canonical map ${\mathit{\theta }}_{\mathbf{X}}:\mathrm{O}\to {\mathcal{S}}^{\ast }$, $(U^{(1)},\cdots ,U^{(d)})\mapsto (U^{(1)},\cdots ,U^{(d)})·\mathbf{X}$, whose image is $\mathrm{O}·\mathbf{X}$. Since ${\mathit{\theta }}_{\mathbf{X}}$ is of constant rank [6, §16.10.2] and easily shown to be locally injective (uniqueness of left singular vectors up to sign flipping for $\mathbf{X}\in {\mathcal{S}}^{\ast }$), it is already an immersion [6, §16.8.8.(iv)]. The result is now standard, see, e.g., [6, §16.8.8.(ii)]. $\square$

For $\mathbf{X}\in \mathcal{S}\backslash {\mathcal{S}}^{\ast }$ the dimension of $\mathrm{O}(n_1\times \cdots \times n_d)·\mathbf{X}$ can be smaller than $dim\mathrm{O}(n_1\times \cdots \times n_d)$. Note that we did not attempt to prove or disprove that the orbits are globally embedded submanifolds.

HOSVD tensors

The compact Lie group $\mathrm{O}(n_1\times \cdots \times n_d)$ acts freely on ${\mathcal{S}}^{\ast }$. It also acts properly (since it is compact and acts continuously). By a general theorem (e.g. [6, § 16.10.3]), the quotient manifold ${\mathcal{S}}^{\ast }/\mathrm{O}(n_1\times \cdots \times n_d)$ of equivalence classes exists, and the canonical mapping ${\mathcal{S}}^{\ast }\to {\mathcal{S}}^{\ast }/\mathrm{O}(n_1\times \cdots \times n_d)$ is a submersion. A concrete realization of this abstract quotient manifold is the set ${\mathcal{H}}^{\ast }$ of regular HOSVD tensors which is now introduced.

Given $\mathbf{Y}$, let $U_{\mathbf{Y}}^{(j)}$ denote a matrix of left singular vectors of $M_{\mathbf{Y}}^{(j)}$ as in  (1.1). By (2.1), $\mathbf{X}=({(U_{\mathbf{Y}}^{(1)})}^{\mathsf{T}},\cdots ,{(U_{\mathbf{Y}}^{(d)})}^{\mathsf{T}})·\mathbf{Y}$ has the matricizations

$$\begin{array}{c}\hfill M_{\mathbf{X}}^{(j)}={\mathrm{\Sigma }}_{\mathbf{Y}}^{(j)}{\left(V_{\mathbf{Y}}^{(j)}\right)}^{\mathsf{T}}\left(U_{\mathbf{Y}}^{(1)}\otimes \cdots \otimes U_{\mathbf{Y}}^{(j-1)}\otimes U_{\mathbf{Y}}^{(j+1)}\otimes \cdots \otimes U_{\mathbf{Y}}^{(d)}\right).\end{array}$$

In particular, the rows of ${(V_{\mathbf{Y}}^{(j)})}^{\mathsf{T}}(U_{\mathbf{Y}}^{(1)}\otimes \cdots \otimes U_{\mathbf{Y}}^{(j-1)}\otimes U_{\mathbf{Y}}^{(j+1)}\otimes \cdots \otimes U_{\mathbf{Y}}^{(d)})$ are right singular vectors of $M_{\mathbf{X}}^{(j)}$, the left singular vectors are unit vectors, and the singular values are the same as of $\mathbf{Y}$, that is, ${\mathrm{\Sigma }}_{\mathbf{X}}^{(j)}={\mathrm{\Sigma }}_{\mathbf{Y}}^{(j)}$. Hence $\mathbf{X}$ has the specific property that

$$\begin{array}{c}\hfill G_{\mathbf{X}}^{(j)}=M_{\mathbf{X}}^{(j)}{\left(M_{\mathbf{X}}^{(j)}\right)}^{\mathsf{T}}={\left({\mathrm{\Sigma }}_{\mathbf{X}}^{(j)}\right)}^2={\mathrm{\Lambda }}_{\mathbf{X}}^{(j)},j=1,\cdots ,d,\end{array}$$ 2.3

is a diagonal matrix of decreasing eigenvalues. The reverse relation

$$\begin{array}{c}\hfill \mathbf{Y}=\left(U_{\mathbf{Y}}^{(1)},\cdots ,U_{\mathbf{Y}}^{(d)}\right)·\mathbf{X}\end{array}$$

between $\mathbf{X}$ and $\mathbf{Y}$ is called the higher-order singular value decomposition (HOSVD) of $\mathbf{Y}$ and has been introduced by De Lathauwer et al. [3].

Definition 2.4

Tensors satisfying (2.3) are called HOSVD tensors. The subset of HOSVD tensors in $\mathcal{S}$ is denoted by $\mathcal{H}$, and the subset of HOSVD tensors in ${\mathcal{S}}^{\ast }$ by ${\mathcal{H}}^{\ast }=\mathcal{H}\cap {\mathcal{S}}^{\ast }$.

HOSVD tensors can be regarded as representatives of orbits $\mathrm{O}(n_1\times \cdots \times n_d)·\mathbf{X}$ of orthogonally equivalent tensors. For $\mathbf{X}\in {\mathcal{S}}^{\ast }$, the representatives are essentially unique as stated next. Here it is instructive to note that for square matrices, the set ${\mathcal{H}}^{\ast }$ consists of regular diagonal matrices with strictly decreasing diagonal entries.

Proposition 2.5

Let $\mathbf{X},\mathbf{Y}\in {\mathcal{H}}^{\ast }$ be two HOSVD tensors. If $\mathbf{X}$ and $\mathbf{Y}$ are orthogonally equivalent, that is, $\mathbf{Y}=(U^{(1)},\cdots ,U^{(d)})·\mathbf{X}$, then the $U^{(j)}$ must be diagonal orthogonal matrices (i.e. with values $\pm 1$ on the diagonal).

The proof is immediate from (2.2),  (2.3), and the uniqueness of orthogonal diagonalization up to sign flipping in the case of mutually distinct eigenvalues. Comparing with the explicit form  (2.1), we see that the action of $(U^{(1)},\cdots ,U^{(d)})·\mathbf{X}$ with diagonal $U^{(j)}$ with $\pm 1$ entries results in some sign flipping pattern for the entries of $\mathbf{X}$. This provides the following, sometimes useful necessary condition.

Proposition 2.6

If two HOSVD tensors $\mathbf{X},\mathbf{Y}\in {\mathcal{H}}^{\ast }$ are orthogonally equivalent, then $|X_{i_1,\cdots ,i_d}|=|Y_{i_1,\cdots ,i_d}|$. In particular, $\mathbf{X}$ and $\mathbf{Y}$ have the same zero pattern.

We now turn to the manifold properties of ${\mathcal{H}}^{\ast }$.

Theorem 2.7

The set ${\mathcal{H}}^{\ast }$ is a smooth embedded submanifold of ${\mathcal{S}}^{\ast }$ of dimension

$$\begin{array}{c}\hfill dim({\mathcal{H}}^{\ast })=n_1\cdots n_d-\sum _{j=1}^d\frac{1}{2}{n}_j(n_j-1)=dim({\mathcal{S}}^{\ast })-dim(\mathrm{O}(n_1\times \cdots \times n_d)).\end{array}$$

Proof

The formal setting is as follows. We denote by ${\mathbb{R}}_{\text{sym}}^{n_j\times n_j}$ the space of symmetric $n_j\times n_j$ matrices, by ${\mathbb{R}}_{\text{sym},0}^{n_j\times n_j}$ the subspace with zeros on the diagonal, by ${\mathit{\pi }}_j(Z)=Z-\mathrm{diag}(Z)$ the orthogonal projection from ${\mathbb{R}}_{\text{sym}}^{n_j\times n_j}$ onto ${\mathbb{R}}_{\text{sym},0}^{n_j\times n_j}$, and $p={\mathit{\pi }}_1\times \cdots \times {\mathit{\pi }}_d$. Consider

$$\begin{array}{c}\hfill g:{\mathcal{S}}^{\ast }\to {\mathbb{R}}_{\text{sym}}^{n_1\times n_1}\times \cdots \times {\mathbb{R}}_{\text{sym}}^{n_d\times n_d},\mathbf{X}\mapsto (G_{\mathbf{X}}^{(1)},\cdots ,G_{\mathbf{X}}^{(d)}).\end{array}$$ 2.4

Then $f=p\circ g$ is a smooth map from ${\mathcal{S}}^{\ast }$ to ${\mathbb{R}}_{\text{sym},0}^{n_1\times n_1}\times \cdots \times {\mathbb{R}}_{\text{sym},0}^{n_d\times n_d}$, and, by Definition 2.4, we have

$$\begin{array}{c}\hfill {\mathcal{H}}^{\ast }=f^{-1}(0).\end{array}$$

Since $dim({\mathbb{R}}_{\text{sym},0}^{n_1\times n_1}\times \cdots \times {\mathbb{R}}_{\text{sym},0}^{n_d\times n_d})=dim(\mathrm{O}(n_1\times \cdots \times n_d))={\sum }_{j=1}^d\frac{1}{2}{n}_j(n_j-1),$ the assertion will follow from the regular value theorem, if we show that $f^{\prime }(\mathbf{X})$ is surjective for every $\mathbf{X}\in {\mathcal{H}}^{\ast }$. To prove the latter, we show that the range of $f^{\prime }(\mathbf{X})$ contains the spaces ${\mathcal{W}}_j=\{0\}\times \cdots \times \{0\}\times {\mathbb{R}}_{\text{sym},0}^{n_j\times n_j}\times \{0\}\times \cdots \times \{0\}$ for $j=1,\cdots ,d$. We demonstrate this for $j=1$. Consider the map

$$\begin{array}{c}\hfill \mathit{\phi }:\mathrm{O}(n_1)\to {\mathcal{S}}^{\ast },U\mapsto (U,I_{n_2},\cdots ,I_{n_d})·\mathbf{X}.\end{array}$$

For brevity, we set $I=I_{n_1}$. Since $\mathit{\phi }(I)=\mathbf{X}$, it follows from the chain rule that the range of $f^{\prime }(\mathbf{X})$ contains the range of ${(f\circ \mathit{\phi })}^{\prime }(I)$. We show that the latter equals ${\mathcal{W}}_1$. Since $\mathbf{X}\in {\mathcal{H}}^{\ast }$, we have $(f\circ \mathit{\phi })(U)=p(U{\mathrm{\Lambda }}_{\mathbf{X}}^{(1)}{U}^{\mathsf{T}},{\mathrm{\Lambda }}_{\mathbf{X}}^{(2)},\cdots ,{\mathrm{\Lambda }}_{\mathbf{X}}^{(d)})$. Further noting that the tangent space to $\mathrm{O}(n_1)$ at I is the space ${\mathbb{R}}_{\text{skew}}^{n_1\times n_1}$ of skew-symmetric $n_1\times n_1$ matrices, we see that

$$\begin{array}{c}\hfill {(f\circ \mathit{\phi })}^{\prime }(I):{\mathbb{R}}_{\text{skew}}^{n_1\times n_1}\to {\mathcal{W}}_1,H\mapsto p(H{\mathrm{\Lambda }}_{\mathbf{X}}^{(1)}+{\mathrm{\Lambda }}_{\mathbf{X}}^{(1)}{H}^{\mathsf{T}},0,\cdots ,0).\end{array}$$

As $dim({\mathcal{W}}_1)=\frac{1}{2}{n}_1(n_1-1)=dim({\mathbb{R}}_{\text{skew}}^{n_1\times n_1})$, it is enough to show that ${(f\circ \mathit{\phi })}^{\prime }(I)$ is injective in order to finish the proof. This now follows from the fact that, by definition of ${\mathcal{H}}^{\ast }$, the diagonal entries of ${\mathbf{\Lambda }}_{\mathbf{X}}^{(1)}$ are strictly decreasing, as it implies that $H{\mathrm{\Lambda }}_{\mathbf{X}}^{(1)}+{\mathrm{\Lambda }}_{\mathbf{X}}^{(1)}{H}^{\mathsf{T}}=H{\mathrm{\Lambda }}_{\mathbf{X}}^{(1)}-{\mathrm{\Lambda }}_{\mathbf{X}}^{(1)}H$ cannot be diagonal for skew-symmetric $H\ne 0$. This, however, is equivalent to injectivity of ${(f\circ \mathit{\phi })}^{\prime }(I)$ as given above. $\square$

Remark 2.8

In our definition (2.3) of HOSVD tensors we required the diagonal elements of $G_{\mathbf{X}}^{(j)}$ to be decreasing. This has advantages and drawbacks. One advantage are the narrower uniqueness properties leading to the practical condition in Proposition 2.6. A disadvantage is that it is more difficult to design HOSVD tensors “by hand” as in Sect. 2.4. Alternatively, one may define a set $\tilde{\mathcal{H}}$ by just requiring the $G_{\mathbf{X}}^{(j)}$ to be diagonal. Then for every $\mathbf{X}\in \tilde{\mathcal{H}}$ we have $(P^{(1)},\cdots ,P^{(d)})·\mathbf{X}\in \mathcal{H}$, where $P^{(j)}$ are permutation matrices that sort the diagonal entries of $G_{\mathbf{X}}^{(j)}$ accordingly. For mutually distinct eigenvalues the choice of $P^{(j)}$ is unique. The corresponding set ${\tilde{\mathcal{H}}}^{\ast }$ is therefore the finite disjoint union of sets $(P^{(1)},\cdots ,P^{(d)})·{\mathcal{H}}^{\ast }$ over all $P^{(j)}$, and as such also an embedded submanifold of ${\mathcal{S}}^{\ast }$.

Degrees of freedom

A principal challenge in understanding the interconnection between higher-order singular values of tensors arises from the fact that, in contrast to the matrix case, the converse statement of Proposition 2.2 is in general not true when $d\ge 3$. Tensors may have the same higher-order singular values without being orthogonally equivalent. This can be seen from the following heuristic.

The set ${\mathcal{S}}^{\ast }$ is open and dense in $\mathcal{S}$ by Proposition 1.3, and therefore is a smooth manifold of dimension

$$\begin{array}{c}\hfill dim({\mathcal{S}}^{\ast })=(n_1\cdots n_d)-1.\end{array}$$

The set ${\mathfrak{S}}_{>}$ is an open subset of Cartesian products of spheres and hence of dimension

$$\begin{array}{c}\hfill dim({\mathfrak{S}}_{>})=(n_1+\cdots +n_d)-d.\end{array}$$

Therefore, given $\mathbf{X}\in {\mathcal{S}}^{\ast }$, we expect the set ${\mathcal{S}}_{\mathbf{X}}^{\ast }$ of tensors having the same higher-order singular values as $\mathbf{X}$ to be at least of “dimension”

$$\begin{array}{c}\hfill dim({\mathcal{S}}^{\ast })-dim({\mathfrak{S}}_{>})=(n_1\cdots n_d)-(n_1+\cdots +n_2)+(d-1).\end{array}$$

When $d\ge 3$, by Proposition 2.3, this set cannot only consist of tensors that are orthogonally equivalent to $\mathbf{X}$.1 In fact, for large d, the orthogonally equivalent tensors will only be a very “low-dimensional” subset of ${\mathcal{S}}_{\mathbf{X}}^{\ast }$.

A non-equivalent example

The previous considerations suggest that there must exist tensors having the same higher-order singular values without being orthogonally equivalent. We construct here an example of size $2\times 2\times 2$ using Proposition 2.6. Let us shortly count the degrees of freedom in this situation. The Euclidean unit sphere $\mathcal{S}$ is of dimension seven, the set ${\mathfrak{S}}_{\ge }$ of potential tuples of higher-order singular values is of dimension three, while orbits $\mathrm{O}(2\times 2\times 2)·\mathbf{X}$ of orthogonally equivalent tensors are of dimension at most three, too. This indicates for every $\mathbf{X}\in \mathcal{S}$ an at least one-dimensional set of non-equivalent tensor with same higher-order singular values.

Using a common slice-wise notation of tensors, we consider (currently not normalized)

$$\begin{array}{c}\hfill \mathbf{X}=\left[\begin{array}{cc}-2\hfill & 1\hfill \\ 1\hfill & 1\hfill \end{array}\right|\left.\begin{array}{cc}-1\hfill & -1\hfill \\ -1\hfill & 0\hfill \end{array}\right].\end{array}$$

The three matricizations are $M_{\mathbf{X}}^{(1)}=\left[\begin{array}{cccc}-2& 1& -1& -1\\ 1& 1& -1& 0\end{array}\right]$, $M_{\mathbf{X}}^{(2)}=\left[\begin{array}{cccc}-2& 1& -1& -1\\ 1& 1& -1& 0\end{array}\right]$, and $M_{\mathbf{X}}^{(3)}=\left[\begin{array}{cccc}-2& 1& 1& 1\\ -1& -1& -1& 0\end{array}\right]$. In all three matricizations the rows are orthogonal, and the norm of the first row is larger than the norm of the second one. This shows that $\mathbf{X}$ is a HOSVD tensor. Its squared higher-order singular values are

$$\begin{array}{c}\hfill {\mathbf{\Lambda }}_{\mathbf{X}}=\left[{\mathit{\lambda }}_{\mathbf{X}}^{(1)},{\mathit{\lambda }}_{\mathbf{X}}^{(2)},{\mathit{\lambda }}_{\mathbf{X}}^{(3)}\right]=\left[\left(\begin{array}{c}7\\ 3\end{array}\right),\left(\begin{array}{c}7\\ 3\end{array}\right),\left(\begin{array}{c}7\\ 3\end{array}\right)\right].\end{array}$$

In particular, $\mathbf{X}/{\Vert \mathbf{X}\Vert }_F\in {\mathcal{S}}^{\ast }$. As a second tensor consider

$$\begin{array}{c}\hfill \mathbf{Y}=\left[\begin{array}{cc}3/\sqrt{2}\hfill & -1\hfill \\ -1\hfill & -1/\sqrt{2}\hfill \end{array}\right|\left.\begin{array}{cc}1\hfill & 1/\sqrt{2}\hfill \\ 1/\sqrt{2}\hfill & 1\hfill \end{array}\right].\end{array}$$

One checks again that all three matricizations $M_{\mathbf{Y}}^{(1)}=\left[\begin{array}{cccc}3/\sqrt{2}& -1& 1& 1/\sqrt{2}\\ -1& -1/\sqrt{2}& 1/\sqrt{2}& 1\end{array}\right]$, $M_{\mathbf{Y}}^{(2)}=\left[\begin{array}{cccc}3/\sqrt{2}& -1& 1& 1/\sqrt{2}\\ -1& -1/\sqrt{2}& 1/\sqrt{2}& 1\end{array}\right]$, and $M_{\mathbf{Y}}^{(3)}=\left[\begin{array}{cccc}3/\sqrt{2}& -1& -1& -1/\sqrt{2}\\ 1& 1/\sqrt{2}& 1/\sqrt{2}& 1\end{array}\right]$ have orthogonal rows with squared row norms

$$\begin{array}{c}\hfill {\mathbf{\Lambda }}_{\mathbf{Y}}=\left[{\mathit{\lambda }}_{\mathbf{Y}}^{(1)},{\mathit{\lambda }}_{\mathbf{Y}}^{(2)},{\mathit{\lambda }}_{\mathbf{Y}}^{(3)}\right]=\left[\left(\begin{array}{c}7\\ 3\end{array}\right),\left(\begin{array}{c}7\\ 3\end{array}\right),\left(\begin{array}{c}7\\ 3\end{array}\right)\right].\end{array}$$

This shows that $\mathbf{X}/{\Vert \mathbf{X}\Vert }_F$ and $\mathbf{Y}/{\Vert \mathbf{Y}\Vert }_F$ are two HOSVD tensors in ${\mathcal{S}}^{\ast }$ with the same set of higher-order singular values. By Proposition 2.6, they are not orthogonally equivalent.

The set of feasible configurations

The set $\mathfrak{F}=\mathfrak{F}(n_1,\cdots ,n_d)\subseteq {\mathfrak{S}}_{\ge }$ of feasible configurations has been defined in (1.2). In this section we investigate this set. A simple observation worth to mention is that $\mathfrak{F}$ is closed. This follows from Proposition 1.3 and the compactness of $\mathcal{S}$.

Not all configurations are feasible

When $d=2$, we know that the singular values of a matrix and its transpose are the same, so trivially not all configurations for ${\mathit{\sigma }}_{\mathbf{X}}^{(1)}$ and ${\mathit{\sigma }}_{\mathbf{X}}^{(2)}$ are possible. The formal statement, using the introduced notation, reads, with $n_1\ge n_2$,

$$\begin{array}{c}\hfill \mathfrak{F}(n_1,n_2)=\left\{\mathbf{\Sigma }=({\mathit{\sigma }}^{(1)},{\mathit{\sigma }}^{(2)})\in {\mathfrak{S}}_{\ge }(n_1,n_2):{\mathit{\sigma }}^{(1)}=({\mathit{\sigma }}^{(2)},0,\cdots ,0)\right\}\ne {\mathfrak{S}}_{\ge }.\end{array}$$

In fact, $\mathfrak{F}$ is an $n_2$-dimensional subset in the $(n_1+n_2)$-dimensional set ${\mathfrak{S}}_{\ge }(n_1,n_2)$.

This known phenomenon in the matrix case can be used to give a qualitative proof that also for higher-order tensors not all configurations are feasible. To start, we recall a fact on the HOSVD from the literature. Let $\mathbf{X}$ have the left singular vector matrices $U_{\mathbf{X}}^{(j)}$ (column-wise ordered by decreasing singular values), and multilinear rank $\mathbf{r}=(r_1,\cdots ,r_d)$. Then we can write the “economic” HOSVD as

$$\begin{array}{c}\hfill \mathbf{X}=\left({\widehat{U}}_{\mathbf{X}}^{(1)},\cdots ,{\widehat{U}}_{\mathbf{X}}^{(d)}\right)·\mathbf{C},\end{array}$$

where ${\widehat{U}}_{\mathbf{X}}^{(j)}$ contains only the first $r_j$ columns of $U_{\mathbf{X}}^{(j)}$, and the core tensor $\mathbf{C}$ is of size $r_1\times \cdots \times r_d$. The multilinear matrix product here corresponds to the action of the tensor product operator ${\widehat{U}}_{\mathbf{X}}^{(1)}\otimes \cdots \otimes {\widehat{U}}_{\mathbf{X}}^{(d)}$ on ${\mathbb{R}}^{r_1}\otimes \cdots \otimes {\mathbb{R}}^{r_d}$, the explicit formulas are similar to (2.1). Note that if $\mathbf{X}$ is non-singular, $\mathbf{C}$ is just an HOSVD tensor in the orthogonally equivalent orbit of $\mathbf{X}$ as defined above. The key observation in the general case is that $\mathbf{C}$ is non-singular in ${\mathbb{R}}^{r_1\times \cdots \times r_d}$, and its higher-order singular values in every direction are given by the nonzero higher-order singular values of $\mathbf{X}$ [3].

Based on this fact, we can first give trivial examples of singular tensors for which the nonzero singular values in different directions are not independent of each other.

Lemma 3.1

Let $\mathbf{X}$ have multilinear rank $\mathbf{r}=(r_1,\cdots ,r_d)$. Assume $r_j=1$ for $j\ge 3$. Then $r_1=r_2$ and $\{{\mathit{\sigma }}_1^{(1)},\cdots ,{\mathit{\sigma }}_{r_1}^{(1)}\}=\{{\mathit{\sigma }}_1^{(2)},\cdots ,{\mathit{\sigma }}_{r_2}^{(2)}\}$.

Proof

Let $\mathbf{C}\in {\mathbb{R}}^{r_1\times \cdots \times r_d}$ be the economic HOSVD core tensor of $\mathbf{X}$. The matricizations $M_{\mathbf{C}}^{(j)}$ for $j\ge 3$ are just row vectors and have only one singular value which equals the Frobenius norm of $\mathbf{X}$. On the other hand, we have $M_{\mathbf{C}}^{(1)}={(M_{\mathbf{C}}^{(2)})}^{\mathsf{T}}$ (up to possible permutations), which implies the result. $\square$

Since tensors with $r_j=1$ for $j\ge 3$ considered in the previous lemma are naturally identified as elements of ${\mathbb{R}}^{n_1}\otimes {\mathbb{R}}^{n_2}$, that is, as matrices, the previous statement may appear rather odd at first. However, using a perturbation argument, it leads to a non-constructive proof that non-feasible configurations for higher-order singular values do exist even in the non-singular case. In fact, these configurations are of positive volume within ${\mathfrak{S}}_{\ge }$.

Theorem 3.2

For $n_1\ge n_2$, consider ${\mathit{\sigma }}^{(1)}\in S_{\ge }^{(1)}$ and ${\mathit{\sigma }}^{(2)}\in S_{\ge }^{(2)}$ such that

$$\begin{array}{c}\hfill {\mathit{\sigma }}^{(1)}\ne ({\mathit{\sigma }}^{(2)},0,\cdots ,0),\end{array}$$ 3.1

where the number of appended zeros on the right side equals $n_1-n_2$. Let further ${\mathcal{O}}_{\mathit{ϵ}}^{(j)}$, $j=3,\cdots ,d$, denote neighbourhoods of ${\mathbf{e}}_1^{(j)}=(1,0,\cdots ,0)$ in $S_{\ge }^{(j)}$ of diameter at most $\mathit{ϵ}>0$ (w.r.t. some norm). For $j\in \{1,2\}$, let ${\mathcal{O}}_{\mathit{ϵ}}^{(j)}$ be similar neighbourhoods, but of ${\mathit{\sigma }}^{(1)}$ and ${\mathit{\sigma }}^{(2)}$, respectively. Then there exists $\mathit{ϵ}>0$ such that

$$\begin{array}{c}\hfill \mathfrak{F}\cap \left({\mathcal{O}}_{\mathit{ϵ}}^{(1)}\times \cdots \times {\mathcal{O}}_{\mathit{ϵ}}^{(d)}\right)=\mathrm{\varnothing },\end{array}$$

that is, no $\mathbf{\Sigma }\in {\mathcal{O}}_{\mathit{ϵ}}^{(1)}\times \cdots \times {\mathcal{O}}_{\mathit{ϵ}}^{(d)}$ is feasible.

Proof

Assume to the contrary that for every n there exists a tensor ${\mathbf{X}}_n\in \mathcal{S}$ such that ${\mathbf{\Sigma }}_{{\mathbf{X}}_n}\in {\mathcal{O}}_{1/n}^{(1)}\times \cdots \times {\mathcal{O}}_{1/n}^{(d)}$. The sequence of ${\mathbf{X}}_n$ has a convergent subsequence with a limit $\mathbf{X}\in \mathcal{S}$. By Lemma 1.3, $\mathbf{X}$ has higher-order singular values ${\mathbf{\Sigma }}_{\mathbf{X}}=({\mathit{\sigma }}^{(1)},{\mathit{\sigma }}^{(2)},{\mathbf{e}}_1^{(3)},\cdots ,{\mathbf{e}}_1^{(d)})$. Now Lemma 3.1 applies, but is in contradiction to (3.1). $\square$

Remark 3.3

The condition (3.1) can hold in two cases: (i) the number of nonzero singular values in direction one and two are the same ($r_1=r_2$), but the singular values themselves are not, or (ii) $r_1\ne r_2$. The second case has some interesting implications for rectangular tensors. Assume for instance $n_1\ne n_2$. Then by Theorem 3.2 there cannot exist normalized non-singular tensors in ${\mathbb{R}}^{n_1\times \cdots \times n_d}$ for which the singular value vectors ${\mathit{\sigma }}^{(j)}$ in directions $j=3,\cdots ,d$ are arbitrarily close to the corresponding unit vector ${\mathbf{e}}^{(j)}$. This surprising connection between mode sizes of the tensor and location of the singular value vectors is not obvious, especially given the fact that almost every tensor is non-singular (assuming (1.2)).

A conjecture on interior points

For $d=2$ we have seen that $\mathfrak{F}(n_1,n_2)$ is a set of measure zero within ${\mathfrak{S}}_{\ge }(n_1,n_2)$, even when $n_1=n_2$. One question is whether this is also true for higher-order tensors. Remarkably, the following experiment suggests that this does not need to be the case.

We generate random $2\times 2\times 2$ tensors $\mathbf{X}$ of Frobenius norm one.2 With probability one, the higher-order singular values ${\mathbf{\Sigma }}_{\mathbf{X}}=({\mathit{\sigma }}_{\mathbf{X}}^{(1)},{\mathit{\sigma }}_{\mathbf{X}}^{(2)},{\mathit{\sigma }}_{\mathbf{X}}^{(3)})$ are elements of ${\mathfrak{S}}_{>}(2,2,2)$, which is a set of three dimensions and therefore can be visualized. We simply make the identification,

$$\begin{array}{c}\hfill {\mathrm{\Sigma }}_{\mathbf{X}}\mapsto \left({\mathit{\sigma }}_1^{(1)},{\mathit{\sigma }}_1^{(2)},{\mathit{\sigma }}_1^{(3)}\right)\in {\left[\frac{1}{\sqrt{2}},1\right]}^3,\end{array}$$

that is, we project on the first coordinate of each singular value vector. In Fig. 1 we see these projected points for 10,000 random examples, and their convex hull computed with a Matlab integrated Delauney triangulation.

Fig. 1.

Visualizing higher-order singular values in the $2\times 2\times 2$ case. Plotted are the vectors $({\mathit{\sigma }}_1^{(1)},{\mathit{\sigma }}_1^{(2)},{\mathit{\sigma }}_1^{(3)})$ containing the largest singular values of the three directions for 10,000 random tensors $\mathbf{X}\in \mathcal{S}$. They seem to form a three-dimensional connected set. Hence, the corresponding set of ${\mathbf{\Sigma }}_{\mathbf{X}}^{}=({\mathit{\sigma }}_{\mathbf{X}}^{(1)},{\mathit{\sigma }}_{\mathbf{X}}^{(1)},{\mathit{\sigma }}_{\mathbf{X}}^{(1)})$ should be of positive volume in the three-dimensional set ${\mathfrak{S}}_{\ge }$

As the resulting point cloud appears three-dimensional, we suppose that the set of feasible configurations is also three-dimensional. But one can also verify in the plot that not all configurations are feasible. Above we made use of the fact that ${\mathit{\sigma }}^{(j)}=(1,0)$ (Tucker rank in the direction j equals one) implies ${\mathit{\sigma }}^{(i)}={\mathit{\sigma }}^{(k)}$ for $i,k\ne j$. This can be seen in the picture as the convex polytope intersects the hyperplanes $x=1$, $y=1$ and $z=1$ in single one-dimensional facets of 45 degree.

We are led to the following conjecture.

Conjecture 3.4

When $d\ge 3$, and given the compatibility condition (1.2), the set $\mathfrak{F}(n_1,\cdots ,n_d)$ has positive (relative) volume in ${\mathfrak{S}}_{\ge }(n_1,\cdots ,n_d)$.

In fact, the following seems likely (under the same assumptions).

Conjecture 3.5

For generic $\mathbf{X}\in \mathcal{S}$, ${\mathbf{\Sigma }}_{\mathbf{X}}$ is a (relative) interior point of $\mathfrak{F}(n_1,\cdots ,n_d)$ within ${\mathfrak{S}}_{\ge }(n_1,\cdots ,n_d)$.

Remark 3.6

During revision of the paper, a possible strategy to prove this conjecture has been revealed. It is based on the observation that ${\mathbf{\Sigma }}_{\mathbf{X}}$ is a relative interior point of $\mathfrak{F}(n_1,\cdots ,n_d)$ if and only if the map $g(\mathbf{X})=(G_{\mathbf{X}}^{(1)},\cdots ,G_{\mathbf{X}}^{(d)})$ (that has already been considered in (2.4)) is locally surjective when regarded as a map from the unit sphere $\mathcal{S}$ to the Cartesian product of hyperplanes $\{A^{(j)}\in {\mathbb{R}}_{\text{sym}}^{n_j\times n_j}:\mathrm{tr}(A^{(j)})=1\}$. In other words, one has to show that the rank of the derivative $g^{\prime }(\mathbf{X})$, when restricted to the tangent space $T_{\mathbf{X}}\mathcal{S}$, equals the maximum possible value $\mathit{\alpha }=\left({\sum }_{j=1}^d\frac{1}{2}{n}_j(n_j+1)\right)-d$. A sufficient condition for this is that $g^{\prime }(\mathbf{X})$ is of rank $\mathit{\alpha }+1$ on ${\mathbb{R}}^{n_1\times \cdots \times n_d}$. However, as $g^{\prime }(\mathbf{X})$ depends polynomially on the entries of $\mathbf{X}$, the function $\mathbf{X}\mapsto \mathrm{rank}(g^{\prime }(\mathbf{X}))$ achieves its maximum value for almost all $\mathbf{X}$. Since it is bounded by $\mathit{\alpha }+1$, it is therefore enough to find a single tensor $\mathbf{X}$ for which rank $\mathit{\alpha }+1$ is achieved. In this way, one can validate Conjecture 3.5 for different configurations of $n_1,\cdots ,n_d$ by constructing random $\mathbf{X}$ and evaluating the rank of $g^{\prime }(\mathbf{X})$ numerically. A rigorous proof would have to confirm this numerical rank for “simple” candidates $\mathbf{X}$, which we were able to do for $2\times 2\times 2$ tensors so far. This approach shall be subject of a future work.

Alternating projection method

Even in the case that one would be given the information that a configuration $\mathbf{\Sigma }=({\mathit{\sigma }}^{(1)},\cdots ,{\mathit{\sigma }}^{(d)})\in {\mathfrak{S}}_{\ge }$ is feasible, the question remains how to construct a corresponding tensor. Note that the suggested strategy to prove Conjecture 3.5 by showing full rank of (2.4) may not provide an explicit way for perturbing singular values in single directions.

A (currently) heuristic approach can be taken via the method of alternating projections. It is based on an alternative viewpoint on Problem 1.4: Given ${\mathit{\sigma }}^{(j)}\in S_{\ge }^{(j)}$ for $j=1,\cdots ,d$, the configuration $\mathbf{\Sigma }=({\mathit{\sigma }}^{(1)},\cdots ,{\mathit{\sigma }}^{(d)})$ is feasible, if and only if there exists a tensor $\mathbf{X}$ such that

$$\begin{array}{c}\hfill \mathbf{X}\in \bigcap _{j=1}^d{\mathcal{M}}_{{\mathit{\sigma }}^{(j)}}^{(j)},\end{array}$$ 3.2

where ${\mathcal{M}}_{\mathit{\sigma }}^{(j)}$ denotes the set of all tensors with mode-j singular values $\mathit{\sigma }$. More concretely,

$$\begin{array}{c}\hfill {\mathcal{M}}_{\mathit{\sigma }}^{(j)}=\left\{\mathbf{X}:M_{\mathbf{X}}^{(j)}=U\mathrm{diag}(\mathit{\sigma })V^{\mathsf{T}},U\in \mathrm{O}(n_j),V^{\mathsf{T}}V=I_{n_j}\right\}.\end{array}$$

The method of alternating projections tries to find $\mathbf{X}$ satisfying (3.2) by successively projecting on the sets ${\mathcal{M}}_{{\mathit{\sigma }}^{(j)}}^{(j)}$. It hence takes the form

$$\begin{array}{c}\hfill {\mathbf{X}}_{k+1}=\left({\mathrm{\Pi }}_{{\mathit{\sigma }}^{(d)}}^{(d)}\circ \cdots \circ {\mathrm{\Pi }}_{{\mathit{\sigma }}^{(1)}}^{(1)}\right)({\mathbf{X}}_k),\end{array}$$ 3.3

where ${\mathrm{\Pi }}_{\mathit{\sigma }}^{(j)}:{\mathbb{R}}^{n_1\times \cdots \times n_d}\to {\mathcal{M}}_{\mathit{\sigma }}^{(j)}$ is a metric projection on the set ${\mathcal{M}}_{\mathit{\sigma }}^{(j)}$, that is, ${\mathrm{\Pi }}_{\mathit{\sigma }}^{(j)}(\mathbf{Y})$ returns a best approximation of $\mathbf{Y}$ in the set ${\mathcal{M}}_{\mathit{\sigma }}^{(j)}$. A best approximation in Frobenius norm can be obtained by simply replacing the singular values of $M_{\mathbf{Y}}^{(j)}$ with $\mathit{\sigma }$:

$$\begin{array}{c}\hfill M_{{\mathrm{\Pi }}_{\mathit{\sigma }}^{(j)}(\mathbf{Y})}^{(j)}=U_{\mathbf{Y}}^{(j)}\mathrm{diag}\left(\mathit{\sigma }\right){(V_{\mathbf{Y}}^{(j)})}^{\mathsf{T}}.\end{array}$$ 3.4

Moreover, if ${\mathit{\sigma }}_{\mathbf{Y}}^{(j)}\in S_{>}^{(j)}$, this best approximation is unique. To prove these assertions, note that the best approximation problem in Frobenius norm is equivalent to maximizing the trace of $U\mathrm{diag}(\mathit{\sigma })V^{\mathsf{T}}{(M_{\mathbf{Y}}^{(j)})}^{\mathsf{T}}$ over all $U\in \mathrm{O}(n_j)$ and $V\in {\mathbb{R}}^{n_j^c}$, $V^{\mathsf{T}}V=I_{n_j}$. The von Neumann trace inequality  [15, 23] states that the upper bound for this quantity is ${\mathit{\sigma }}^{\mathsf{T}}{\mathit{\sigma }}_{\mathbf{Y}}^{(j)}$ . Moreover, equality is achieved at U, V if and only if $M_{\mathbf{Y}}^{(j)}=U\mathrm{diag}({\mathit{\sigma }}_{\mathbf{Y}}^{(j)})V^{\mathsf{T}}$, see [4, Remark 1.2]. Hence $U=U_{\mathbf{Y}}^{(j)}$ and $V=V_{\mathbf{Y}}^{(j)}$ are unique in the case that ${\mathit{\sigma }}_{\mathbf{Y}}^{(j)}\in S_{>}^{(j)}$.

Although the interpretation as an alternating projection method is nice, we remark that the multiplication by $U_{\mathbf{Y}}^{(j)}$ in  (3.4) could be omitted in practice. It is an easy induction to show that in this case an orthogonally equivalent sequence of tensors would be produced.

Even assuming that intersection points exist, we are currently not able to provide local or global convergence results for the alternating projection method  (3.3). Instead, we confine ourselves with three numerical illustrations.

Recovering a feasible configuration

To obtain a feasible configuration $\mathbf{\Sigma }$, we create a norm-one tensor $\mathbf{X}$ and take its higher-order singular values, $\mathbf{\Sigma }={\mathbf{\Sigma }}_{\mathbf{X}}$. Then we run the iteration (3.3) starting from a random initialization, and measure the errors $\Vert {\mathit{\sigma }}_{{\mathbf{X}}_{k+1}}^{(j)}-{\mathit{\sigma }}_{\mathbf{X}}^{(j)}{\Vert }_2$ (Euclidean norm) after every full cycle of projections. Since ${\mathrm{\Pi }}_{{\mathit{\sigma }}^{(3)}}^{(3)}$ is applied last, the singular values in direction three are always correct at the time the error is measured. The question is whether also the singular values in the other directions converge to the desired target values. The left plot in Fig. 2 shows one typical example of error curves observed in this kind of experiment in ${\mathbb{R}}^{30\times 30\times 30}$. We see that the sequence ${\mathbf{\Sigma }}_{{\mathbf{X}}_k}$ converges to $\mathbf{\Sigma }$, hence every cluster point of the sequence ${\mathbf{X}}_k$ will have the desired higher-order singular values. So far, we have no theoretical explanation for the shifted peaks occurring in the curves.

Fig. 2.

Recovering a feasible configuration in ${\mathbb{R}}^{30\times 30\times 30}$ via alternating projection. Left errors of singular value vectors. Right maximum difference between absolute values of entries of HOSVD core tensors of iterates and the generating tensor that provided the feasible configuration. As it does not go to zero, the limiting tensor is not orthogonally equivalent to the generating tensor

Since our initial guess is random, we do not expect that the generating $\mathbf{X}$ or an orthogonally equivalent tensor will be recovered. To verify this, we make use of Proposition 2.6 and measure the error ${max}_{i_1,\cdots ,i_d}||{\widehat{X}}_{i_1,\cdots ,i_d}|-|{({\widehat{X}}_{k+1})}_{i_1,\cdots ,i_d}||$ after every loop, where ${\widehat{\mathbf{X}}}_{k+1}$ and $\widehat{\mathbf{X}}$ are HOSVD representatives in the corresponding orbits of orthogonal equivalence. The right plot in Fig.  shows this error curve, and we can see it does not tend to zero. By Proposition 2.6, the limiting tensor is hence not orthogonally equivalent to $\mathbf{X}$. Since this behaviour was observed being typical, the alternating projection method can be suggested as a practical procedure to construct tensors having the same higher-order singular values without being orthogonally equivalent.

Experiments with tensors of order $d=4$ and larger lead to similar results, but they quickly become computationally expensive as SVDs of large matrices have to be calculated.

Perturbation of a feasible configuration

To support Conjecture 3.5, we now consider random perturbations

$$\begin{array}{c}\hfill {\mathbf{\Sigma }}_{\mathit{ϵ}}=\left({\mathit{\sigma }}_{\mathit{ϵ}}^{(1)},\cdots ,{\mathit{\sigma }}_{\mathit{ϵ}}^{(d)}\right)=\mathbf{\Sigma }+\mathcal{O}(\mathit{ϵ})\in {\mathcal{S}}_{\ge }\end{array}$$

of a known feasible configuration $\mathbf{\Sigma }\in \mathfrak{F}\cap {\mathfrak{S}}_{>}$ (obtained again from a random tensor $\mathbf{X}\in {\mathcal{S}}^{\ast }$).3 According to the conjecture, we expect that for small $\mathit{ϵ}$ the configuration ${\mathbf{\Sigma }}_{\mathit{ϵ}}$ is also feasible, so a corresponding tensor may be found by the alternating projection method  (3.3). This can be verified in numerical experiments. The left plot in Fig. 3 shows the errors $\Vert {\mathit{\sigma }}_{{\mathbf{X}}_{k+1}}^{(j)}-{\mathit{\sigma }}_{\mathit{ϵ}}^{(j)}{\Vert }_F$ for one experiment in ${\mathbb{R}}^{10\times 10\times 10\times 10}$ using $\mathit{ϵ}={10}^{-3}$.

Fig. 3.

Experiments in ${\mathbb{R}}^{10\times 10\times 10\times 10}$. Left perturbation of a given feasible configuration $\mathbf{\Sigma }$ by $\mathcal{O}(\mathit{ϵ})$ with $\mathit{ϵ}={10}^{-3}$. Right infeasible configuration ${\mathit{\sigma }}^{(j)}=(1,0,\cdots ,0)+\mathcal{O}({\mathit{ϵ}}_j)$ obtained using ${\mathit{ϵ}}_3={\mathit{ϵ}}_4={10}^{-6}$, but ${\mathit{ϵ}}_1={\mathit{ϵ}}_2={10}^{-3}$

Infeasible configuration

When conducting our experiments with the alternating projection method, we made the experience that with high probability even a randomly generated configuration will be feasible. Indeed, Fig. 1 supports this in the $2\times 2\times 2$ case, as the feasible configurations seem to make up a rather large fraction in ${\mathfrak{S}}_{\ge }(2,2,2)$.

To construct an infeasible configuration we therefore mimic the proof of Theorem 3.2: we generate ${\mathit{\sigma }}^{(j)}$ as $(1,0,\cdots ,0)+\mathcal{O}({\mathit{ϵ}}_j)$ (as described in Footnote 3), where we use very small ${\mathit{ϵ}}_j$ for $j\ge 3$, e.g., ${\mathit{ϵ}}_j={10}^{-6}$. By the arguments presented above this should also enforce ${\mathit{\sigma }}^{(1)}$ to be close to ${\mathit{\sigma }}^{(2)}$ to ensure feasibility. To impede this, we use larger ${\mathit{ϵ}}_1$ and ${\mathit{ϵ}}_2$ instead, e.g., ${\mathit{ϵ}}_1={\mathit{ϵ}}_2={10}^{-3}$ (an alternative would be to generate ${\mathit{\sigma }}^{(1)}$ and ${\mathit{\sigma }}^{(2)}$ completely random). Our results suggest that this indeed results in an infeasible configuration. Accordingly, the alternating projection method fails. The right plot in Fig. 3 shows the outcome of one experiment, again in ${\mathbb{R}}^{10\times 10\times 10\times 10}$.