Sparse and Low-rank Tensor Estimation via Cubic Sketchings

Abstract

In this paper, we propose a general framework for sparse and low-rank tensor estimation from cubic sketchings. A two-stage non-convex implementation is developed based on sparse tensor decomposition and thresholded gradient descent, which ensures exact recovery in the noiseless case and stable recovery in the noisy case with high probability. The non-asymptotic analysis sheds light on an interplay between optimization error and statistical error. The proposed procedure is shown to be rate-optimal under certain conditions. As a technical by-product, novel high-order concentration inequalities are derived for studying high-moment sub-Gaussian tensors. An interesting tensor formulation illustrates the potential application to high-order interaction pursuit in high-dimensional linear regression.

I. Introduction

The rapid advance in modern scientific technology gives rise to a wide range of high-dimensional tensor data [1, 2]. Accurate estimation and fast communication/processing of tensor-valued parameters are crucially important in practice. For example, a tensor-valued predictor which characterizes the association between brain diseases and scientific measurements becomes the point of interest [3, 4, 5]. Another example is the tensor-valued image acquisition algorithm that can considerably reduce the number of required samples by exploiting the compressibility property of signals [6, 7].

The following tensor estimation model is widely considered in recent literatures,

$$y_i=\langle {\mathcal{J}}^{*},{\mathfrak{X}}_i\rangle +{ϵ}_i,i=1,\dots ,n.$$ (I.1)

Here, ${\mathfrak{X}}_i$ and ϵ_i are the measurement tensor and the noise, respectively. The goal is to estimate the unknown tensor ${\mathcal{J}}^{*}$ from measurements ${\left\{y_i,{\mathfrak{X}}_i\right\}}_{i=1}^n$. A number of specific settings with varying forms of ${\mathfrak{X}}_i$ have been studied, e.g., tensor completion [8, 9, 10, 11, 12, 13, 14, 15], tensor regression [5, 3, 4, 16, 17, 18, 19], multi-task learning [20], etc.

In this paper, we focus on the case that the measurement tensor can be written in a cubic sketching form. For example, ${\mathfrak{X}}_i=x_i\circ x_i\circ x_i$ or ${\mathfrak{X}}_i=u_i\circ v_i\circ w_i$, depending on whether ${\mathcal{J}}^{*}$ is symmetric or not. The cubic sketching form of ${\mathfrak{X}}_i$ is motivated by a number of applications.

• Interaction effect estimation: High-dimensional high-order interaction models have been considered under a variety of settings [21, 22, 23, 24]. By writing ${\mathfrak{X}}_i=x_i\circ x_i\circ x_i$, we find that the interaction model has an interesting tensor representation (see left panel of Figure 1) which allows us to estimate high-order interaction terms using tensor techniques. This is in contrast with the existing literature that mostly focused on pair-wise interactions due to the model complexity and computational difficulties. More detailed discussions will be provided in Section V.

• High-order imaging/video compression: High-order imaging/video compression is an important task in modern digital imaging with various applications (see right panel of Figure 1), such as hyper-spectral imaging analysis [25] and facial imaging recognition [26]. One could use Gaussian ensembles for compression such that each entry of ${\mathfrak{X}}_i$ is i.i.d. randomly generated [3, 16, 17]. In contrast, the non-symmetric cubic sketchings, i.e., ${\mathfrak{X}}_i=u_i\circ v_i\circ w_i$, reduce the memory storage from O(np₁p₂p₃) to O(n(p₁ +p₂ +p₃)) (n is the sample size and (p₁,p₂,p₃) is the tensor dimension), but still preserve the optimal statistical rate. More detailed discussions will be provided in Section VI.

In practice, the total number of measurements n is considerably smaller than the number of parameters in the unknown tensor ${\mathcal{J}}^{*}$, due to all kinds of restrictions such as time and storage. Fortunately, a variety of high-dimensional tensor data possess intrinsic structures, such as low-rankness [2] and sparsity [27]. This could highly reduce the effective dimension of the parameter and make the accurate estimation possible. Please refer to (III.2) and (VI.2) for low-rankness and sparsity assumptions.

Fig. 1.

Illustration for interaction reformulation and tensor image/video compression

In this paper, we propose a computationally efficient non-convex optimization approach for sparse and low-rank tensor estimation via cubic-sketchings. Our procedure is two-stage:

1. obtain an initial estimate via the method of tensor moment (motivated by high-order Stein’s identity), and then apply sparse tensor decomposition to the initial estimate to output a warm start;

2. use a thresholded gradient descent to iteratively refine the warm start in each tensor mode until convergence.

Theoretically, we carefully characterize the optimization and statistical errors at each iteration step. The output estimate is shown to converge in a geometric rate to an estimation with minimax optimal rate in statistical error (in terms of tensor Frobenius norm). In particular, after a logarithmic number of iterations, whenever $n\gtrsim K^2{(s\text{ log}(ep/s))}^{\frac{3}{2}}$, the proposed estimator $\widehat{\mathcal{J}}$ achieves

$${\Vert \widehat{\mathcal{J}}-{\mathcal{J}}^{*}\Vert }_F^2\le C{\sigma }^2\frac{Ks\text{ log}(p/s)}{n}$$ (I.2)

with high probability, where s, K, p, and σ² are the sparsity, rank, dimension, and noise level, respectively. We further establish the matching minimax lower bound to show that (I.2) is indeed optimal over a large class of sparse low-rank tensors. Our optimality result can be further extended to the non-sparse case (such as tensor regression [3, 17, 28, 29]) – to the best of our knowledge, this is the first statistical rate optimality result in both sparse and non-sparse low-rank tensor regressions.

The above theoretical analyses are non-trivial due to the non-convexity of the empirical risk function, and the need to develop some new high-order sub-Gaussian concentration inequalities. Specifically, the empirical risk function in consideration satisfies neither restricted strong convexity (RSC) condition nor sparse eigenvalue (SE) condition in general. Thus, many previous results, such as the one based on local optima analysis [30, 31, 17], are not directly applicable. Moreover, the structure of cubic-sketching tensor leads to high-order products of sub-Gaussian random variables. Thus, the matrix analysis based on Hoeffding-type or Bernstein-type concentration inequality [32, 33] will lead to sub-optimal statistical rate and sample complexity. This motivates us to develop new high-order concentration inequalities and sparse tensor-spectral-type bound, i.e., Lemmas 1 and 2 in Section IV-C. These new technical results are obtained based on the careful partial truncation of high-order products of sub-Gaussian random variables and the argument of bounded ψ_α-norm [34], and may be of independent interest.

The literature on low-rank matrix estimation methods, e.g., the spectral method and nuclear norm minimization [35, 36, 37], is also related to this work. However, our cubic sketching model is by-no-means a simple extension from matrix estimation problems. In general, many related concepts or methods for matrix data, such as singular value decomposition, are problematic to apply in the tensor framework [38, 39]. It is also found that simple unfolding or matricizing of tensors may lead to suboptimal results due to the loss of structural information [40]. Technically, the tensor nuclear norm is NP-hard to even approximate [9, 10, 41], and thus the method to handle tensor low-rankness is distinct from the matrix.

The rest of the paper is organized as follows. Section II provides preliminaries on notation and basic knowledge of tensor. A two-stage method for symmetric tensor estimation is proposed in Section III, with the corresponding theoretical analysis given in Section IV. A concrete application to high-order interaction effect models is described in Section V. The non-symmetric tensor estimation model is introduced and discussed in Section VI. Numerical analysis is provided in Section VII to support the proposed procedure and theoretical results of this paper. Section VIII discusses extensions to higher-order tensors. The proofs of technical results are given in supplementary materials.

II. Preliminary

Throughout the paper, vector, matrix, and tensor are denoted by boldface lower-case letters (e.g., x,y), boldface upper-case letters (e.g., X,Y), and script letters (e.g., $\mathcal{X}$, $\mathcal{Y}$), respectively. For any set A, let |A| be the cardinality. The diag(x) is a diagonal matrix generated by x. For two vectors x and y, x ∘ y is the outer product. Define ‖x‖_q ≔ (|x₁|^q + ⋯ + |x_p|^q)^1/q. We also define the l₀ quasi-norm by ‖x‖₀ = #{j : x_j ≠ 0} and l_∞ norm by max_1≤j≤p |x_j|. Denote the set {1,2,…,n} by [n]. Let e_j be the canonical vectors, whose j-th entry equals to 1 and all other entries equal to zero. For any two sequences ${\left\{a_n\right\}}_{n=1}^{\infty }$, ${\left\{b_n\right\}}_{n=1}^{\infty }$, we say $a_n=\mathcal{O}\left(b_n\right)$ if there exists some positive constant C₀ and sufficiently large n₀ such that |a_n| ≤ C₀b_n for all n ≥ n₀. We also write a_n ≍ b_n if there exists C,c > 0 such that ca_n ≤ b_n ≤ Ca_n for all n ≥ 1. Additionally, C₁,C₂, …, c₁,c₂, … are generic constants, whose actual values may be different from line to line.

We next introduce notations and operations on the matrix. For matrices $A=\left[a_1,\dots ,a_J\right]\in {ℝ}^{I\times J}$ and $B=\left[b_1,\dots ,b_L\right]\in {ℝ}^{K\times L}$, their Kronecker product is defined as a (IK)-by-(JL) matrix A ⊗ B = [a₁ ⊗ B ⋯a_J ⊗ B], where a_j ⊗ B = (a_j1B^⊤,…,a_jIB^⊤)^⊤. If A and B have the same number of columns J = L, the Khatri-Rao product is defined as $A\odot B=\left[a_1\circ b_1,a_2\circ b_2,\cdots ,a_J\circ b_J\right]\in {ℝ}^{IK\times J}$. If the matrices A and B are of the same dimension, the Hadamard product is their element-wise matrix product, such that (A*B)_ij = A_ij ·B_ij. For matrix $X=\left[x_1\cdots x_n\right]\in {ℝ}^{m\times n}$, we also denote the vectorization $\text{vec}(X)=\left(x_1^{\top },\dots ,x_n^{\top }\right)\in {ℝ}^{1\times mn}$ and column-wise ${\mathcal{l}}_2$ norms as $\text{Norm}(X)=\left({\Vert x_1\Vert }_2,\dots ,{\Vert x_n\Vert }_2\right)\in {ℝ}^{1\times n}$.

In the end, we focus on tensor notation and relevant operations. Interested readers are referred to [2] for more details. Suppose $\mathcal{X}\in {ℝ}^{p_1\times p_2\times p_3}$ is an order-3 tensor. Then the (i,j,k)-th element of $\mathcal{X}$ is denoted by ${[\mathcal{X}]}_{ijk}$. The successive tensor multiplication with vectors $u\in {ℝ}^{p_2}$, $v\in {ℝ}^{p_3}$ is denoted by $\mathcal{X}{\times }_{2}u{\times }_{3}v={\sum }_{j\in \left[p_2\right],l\in \left[p_3\right]}{u}_j{v}_l{\mathcal{X}}_{[:,j,l]}\in {ℝ}^{p_1}$. We say $\mathcal{X}\in {ℝ}^{p_1\times p_2\times p_3}$ is rank-one if it can be written as the outer product of three vectors, i.e., $\mathcal{X}=x_1\circ x_2\circ x_3$ or ${[\mathcal{X}]}_{ijk}=x_{1i}{x}_{2j}{x}_{3k}$ for all i,j,k. Here “∘” represents the vector outer product. We say $\mathcal{X}$ is symmetric if ${[\mathcal{X}]}_{ijk}={[\mathcal{X}]}_{ikj}={[\mathcal{X}]}_{jik}={[\mathcal{X}]}_{jki}={[\mathcal{X}]}_{kij}={[\mathcal{X}]}_{kji}$ for all i,j,k. Then, $\mathcal{X}$ is rank-one and symmetric if and only if it can be decomposed as $\mathcal{X}=x\circ x\circ x$ for some vector x.

More generally, we may decompose a tensor as the sum of rank one tensors as follows,

$$\mathcal{X}=\sum _{k=1}^K{\eta }_k{x}_{1k}\circ x_{2k}\circ x_{3k},$$ (II.1)

where ${\eta }_k\in ℝ$, $x_{1k}\in {\mathbb{S}}^{p_1-1}$, $x_{2k}\in {\mathbb{S}}^{p_2-1}$, $x_{3k}\in {\mathbb{S}}^{p_3-1}$. This is the so-called CANDECOMP/PARAFAC, or CP decomposition [2] with CP-rank being defined as the minimum number K such that (II.1) holds. Then, ${\left\{x_{1k}\right\}}_{k=1}^K,{\left\{x_{2k}\right\}}_{k=1}^K,{\left\{x_{3k}\right\}}_{k=1}^K$ are called factors along first, second and third mode. Note that factors are normalized as unit vectors to guarantee the uniqueness of decomposition, and η = {η₁,…,η_K} plays an analogous role of singular values in matrix value decomposition here. Several tensor norms also need to be introduced. The tensor Frobenius norm and tensor spectral norm are defined respectively as

$$\begin{gathered} \Vert \mathcal{X}{\Vert }_F=\sqrt{\sum _{i=1}^{p_1}\sum _{j=1}^{p_2}\sum _{k=1}^{p_3}{\mathcal{X}}_{ijk}^2} \\ \Vert \mathcal{X}{\Vert }_{op}:=\underset{u\in {ℝ}^{p_1},v\in {ℝ}^{p_2},w\in {ℝ}^{p_3}}{\text{sup}}\frac{|\langle \mathcal{X},u\circ v\circ w\rangle |}{\Vert u{\Vert }_2\Vert v{\Vert }_2\Vert w{\Vert }_2}, \end{gathered}$$ (II.2)

where $\langle \mathcal{X},\mathcal{Y}\rangle ={\sum }_{i,j,k}{\mathcal{X}}_{ijk}{\mathcal{Y}}_{ijk}$. Clearly, $\Vert \mathcal{X}{\Vert }_F^2=\langle \mathcal{X},\mathcal{X}\rangle$. We also consider the following sparse tensor spectral norm,

$$\Vert \mathcal{X}{\Vert }_s:=\underset{\begin{array}{c}\Vert a\Vert =\Vert b\Vert =\Vert c\Vert =1\\ \text{max}\left\{\Vert a{\Vert }_0,\Vert b{\Vert }_0,\Vert c{\Vert }_0\right\}\le s\end{array}}{\text{sup}}|\langle \mathcal{X},a\circ b\circ c\rangle |.$$ (II.3)

By definition, $\Vert \mathcal{X}{\Vert }_s\le \Vert \mathcal{X}{\Vert }_{op}$. Suppose $\mathcal{X}=x_1\circ x_2\circ x_3$ and $\mathcal{Y}=y_1\circ y_2\circ y_3$ are two rank-one tensors. Then it is easy to check $\Vert \mathcal{X}{\Vert }_F={\Vert x_1\Vert }_2{\Vert x_2\Vert }_2{\Vert x_3\Vert }_2$ and $\langle \mathcal{X},\mathcal{Y}\rangle =\left(x_1^{\top }{y}_1\right)\left(x_2^{\top }{y}_2\right)\left(x_3^{\top }{y}_3\right)$.

III. Symmetric Tensor Estimation Via Cubic Sketchings

In this section, we focus on the estimation of sparse and low-rank symmetric tensors,

$$\begin{gathered} y_i=\langle {\mathcal{J}}^{*},{\mathfrak{X}}_i\rangle +{ϵ}_i, \\ {\mathcal{X}}_i=x_i\circ x_i\circ x_i\in {ℝ}^{p\times p\times p},i=1,\dots ,n, \end{gathered}$$ (III.1)

where x_i are random vectors with i.i.d. standard normal entries. As previously discussed, the tensor parameter ${\mathcal{J}}^{*}$ often satisfies certain low-dimensional structures in practice, among which the factor-wise sparsity and low-rankness [16] commonly appear. We thus assume ${\mathcal{J}}^{*}$ is CP rank-K for K ≪ p and the corresponding factors are sparse,

$$\begin{gathered} {\mathcal{J}}^{*}=\sum _{k=1}^K{\eta }_k^{*}{\beta }_k^{*}\circ {\beta }_k^{*}\circ {\beta }_k^{*} \\ ,\text{with }{\Vert {\beta }_k^{*}\Vert }_2=1,{\Vert {\beta }_k^{*}\Vert }_0\le s,\forall k\in [K]. \end{gathered}$$ (III.2)

The CP low-rankness has been widely assumed in literature for its nice scalability and simple formulation [5, 25, 18]. Different from the matrix factor analysis, we do not assume the tensor factors ${\beta }_k^{*}$ here are orthogonal. On the other hand, since the low-rank tensor estimation is NP-hard in general [42], we will introduce an incoherence condition in the forthcoming Condition 3 to ensure that the correlation among different factors ${\beta }_k^{*}$ is not too strong. Such a condition has been used in recent literature on tensor data analysis [43], compressed sensing [44], matrix decomposition [45], and dictionary learning [46].

Based on observations ${\left\{y_i,{\mathfrak{X}}_i\right\}}_{i=1}^n$, we propose to estimate ${\mathcal{J}}^{*}$ via minimizing the empirical squared loss since the close-form gradient provides computational convenience,

$$\widehat{\mathcal{J}}=\underset{\mathcal{J}}{\text{argmin}}\mathcal{L}(\mathcal{J})\text{ subject to }\mathcal{J}\text{ is sparse and low-rank},$$ (III.3)

where

$$\begin{gathered} \mathcal{L}(\mathcal{J})=\mathcal{L}\left({\eta }_k,{\beta }_1,\dots ,{\beta }_K\right)=\frac{1}{n}\sum _{i=1}^n{\left(y_i-\langle \mathcal{J},{\mathfrak{X}}_i\rangle \right)}^2 \\ =\frac{1}{n}\sum _{i=1}^n{\left(y_i-\sum _{k=1}^K{\eta }_k{\left(x_i^{\top }{\beta }_k\right)}^3\right)}^2. \end{gathered}$$ (III.4)

Equivalently, (III.3) can be written as,

$$\begin{gathered} \underset{{\eta }_k,{\beta }_k}{\text{min}}\frac{1}{n}\sum _{i=1}^n{\left(y_i-\sum _{k=1}^K{\eta }_k{\left(x_i^{\top }{\beta }_k\right)}^3\right)}^2, \\ \text{s}.\text{t}.{\Vert {\beta }_k\Vert }_2=1,{\Vert {\beta }_k\Vert }_0\le s,\text{ for }k\in [K]. \end{gathered}$$ (III.5)

Clearly, (III.5) is a non-convex optimization problem. To solve it, we propose a two-stage method as described in the next two subsections.

A. Initialization

Due to the non-convexity of (III.5), a straightforward implementation of many local search algorithms, such as gradient descent and alternating minimization, may easily get trapped into local optimums and result in sub-optimal statistical performance. Inspired by recent advances of spectral method (e.g., EM algorithm [47], phase retrieval [48], and tensor SVD [39]), we propose to evaluate an initial estimate $\left\{{\eta }_k^{(0)},{\beta }_k^{(0)}\right\}$ via the method of moment and sparse tensor decomposition (a variant of high-order spectral method) in the following Steps 1 and 2, respectively. The pseudo-code is given in Algorithm 1.

Step 1: Unbiased Empirical Moment Estimator.

Construct the empirical moment-based estimator ${\mathcal{T}}_s$,

$${\mathcal{T}}_s:=\frac{1}{6}[\frac{1}{n}\sum _{i=1}^n{y}_i{x}_i\circ x_i\circ x_i-\sum _{j=1}^p\left(m_1\circ e_j\circ e_j+e_j\circ m_1\circ e_j+e_j\circ e_j\circ m_1\right)],$$ (III.6)

where $m_1:=\frac{1}{n}\sum _{i=1}^n{y}_i{x}_i$, e_j is the canonical vector.

Based on Lemma 4, ${\mathcal{T}}_s$ is an unbiased estimator of ${\mathcal{J}}^{*}$. The construction of (III.6) is motivated by the high-order Stein’s identity ([49]; also see Theorem 7 for a complete statement). Intuitively speaking, based on the third-order score function of a Gaussian random vector $x:{\mathcal{S}}_3(x)=x\circ x\circ x-{\sum }_{j=1}^p\left(x\circ e_j\circ e_j+e_j\circ x\circ e_j+e_j\circ e_j\circ x\right)$, we can construct the unbiased estimator of ${\mathcal{J}}^{*}$ by properly choosing a continuously differentiable function in high-order Stein’s identity. See the proof of Lemma 4 for details.

Step 2: Sparse Tensor Decomposition.

Based on the method of moment estimator obtained in Step 1, we further obtain good initialization for the factors $\left\{{\eta }_k^{(0)},{\beta }_k^{(0)}\right\}$ via truncation and alternating rank-1 power iterations [27, 50],

$${\mathcal{T}}_s\approx \sum _{k=1}^K{\eta }_k^{(0)}{\beta }_k^{(0)}\circ {\beta }_k^{(0)}\circ {\beta }_k^{(0)}.$$

Note that the tensor power iterations recover one rank-1 component per time. To identify all rank-1 components, we generate a large number of different initialization vectors, implement a clustering step, and choose the centroids as the estimates in the initialization stage. This scheme originally appears in tensor decomposition literature [43, 50], although our problem setting and proof techniques are very different. This procedure is also very different from the matrix setting since the rank-1 component in singular value decomposition is mutually orthogonal, but we do not enforce the exact orthogonality here for ${\mathcal{J}}^{*}$.

More specifically, we first choose a large integer M ≫ K and generate M starting vectors ${\left\{b_m^{(0)}\right\}}_{m=1}^M\in {ℝ}^p$ through sparse SVD as described in Algorithm 3. Then for each $b_m^{(0)}$, we apply the following truncated power updates for l = 0,…

$${\tilde{b}}_m^{(l+1)}=\frac{{\mathcal{T}}_s{\times }_2{b}_m^{(l)}{\times }_3{b}_m^{(l)}}{{\Vert {\mathcal{T}}_s{\times }_2{b}_m^{(l)}{\times }_3{b}_m^{(l)}\Vert }_2},b_m^{(l+1)}=\frac{T_d\left({\tilde{b}}_m^{(l+1)}\right)}{{\Vert T_d\left({\tilde{b}}_m^{(l+1)}\right)\Vert }_2},$$

where ×₂, ×₃ are tensor multiplication operators defined in Section II and T_d(x) is a truncation operator that sets all but the largest d entries in absolute values to zero for any vector x. It is noteworthy that the symmetry of ${\mathcal{T}}_s$ implies

$${\mathcal{T}}_s{\times }_2{b}_m^{(l)}{\times }_3{b}_m^{(l)}={\mathcal{T}}_s{\times }_1{b}_m^{(l)}{\times }_3{b}_m^{(l)}={\mathcal{T}}_s{\times }_1{b}_m^{(l)}{\times }_2{b}_m^{(l)}.$$

This means the multiplications along different modes are the same. We run power iterations till its convergence, and denote b_m as the outcome. Finally, we apply K-means to partition ${\left\{b_m\right\}}_{m=1}^M$ into K clusters, let the centroids of the output clusters be ${\left\{{\beta }_k^{(0)}\right\}}_{k=1}^K$, and calculate ${\eta }_k^{(0)}={\mathcal{T}}_s{\times }_1{\beta }_k^{(0)}{\times }_2{\beta }_k^{(0)}{\times }_3{\beta }_k^{(0)}\text{ for }k\in [K]$.

B. Thresholded Gradient Descent

After obtaining a warm start in the first stage, we propose to apply the thresholded gradient descent to iteratively refine the solution to the non-convex optimization problem (III.5). Specifically, denote $X=\left(x_1,\dots ,x_n\right)\in {ℝ}^{p\times n}$, $y={\left(y_1,\dots ,y_n\right)}^{\top }\in {ℝ}^n$, $\eta ={\left({\eta }_1,\dots ,{\eta }_K\right)}^{\top }\in {ℝ}^K$, and $B=\left({\beta }_1,\dots ,{\beta }_K\right)\in {ℝ}^{p\times K}$. Since $\mathcal{L}(B,\eta )=\mathcal{L}(\mathcal{J})$, we let ${\nabla }_B\mathcal{L}(B,\eta )=\left({\nabla }_{{\beta }_1}\mathcal{L}{(B,\eta )}^{\top },\dots ,{\nabla }_{{\beta }_K}\mathcal{L}{(B,\eta )}^{\top }\right)\in {ℝ}^{1\times pK}$, be the gradient function with respect to B. Based on the detailed calculation in Lemma A.1, ${\nabla }_B\mathcal{L}(B,\eta )$ can be written as

$$\begin{gathered} {\nabla }_B\mathcal{L}(B,\eta )=\frac{6}{n}{\left[{\left\{{\left(B^{\top }X\right)}^{\top }\right\}}^3\eta -y\right]}^{\top }* \\ {\left[{\left({\left\{{\left(B^{\top }X\right)}^{\top }\right\}}^2\odot {\eta }^{\top }\right)}^{\top }\odot X\right]}^{\top }, \end{gathered}$$ (III.7)

where {(B^⊤X)^⊤}³ and {(B^⊤X)^⊤}² are entry-wise cubic and squared matrices of (B^⊤X)^⊤. Define φ_h(x) as the thresholding function with a level h that satisfies the following minimal assumptions:

$$\begin{gathered} |{\phi }_h(x)-x|\le h,\forall x\in ℝ, \\ \text{and }{\phi }_h(x)=0,\text{ when }|x|\le h. \end{gathered}$$ (III.8)

Many widely used thresholding schemes, such as hard thresholding H_h(x) = xI_(|x|>h), soft-thresholding S_h(x) = sign(x)max(|x| − h,x), satisfy (III.8). With a slight abuse of notation, we further define the vector thresholding function as ${\phi }_h(x)=\left({\phi }_h\left(x_1\right),\dots ,{\phi }_h\left(x_p\right)\right)\text{ for }x\in {ℝ}^p$.

The initial estimates η⁽⁰⁾ and B⁽⁰⁾ will be updated by thresholded gradient descent in two steps summarized in Algorithm 2. It is noteworthy that only B is updated in Step 3, while η will be updated in Step 4 after finishing the update of B.

Step 3: Updating B via Thresholded Gradient descent.

We update B^(t) via thresholded gradient descent,

$$\text{vec}\left(B^{(t+1)}\right)={\phi }_{\frac{\mu h\left(B^{(t)}\right)}{\varphi }}\left(\text{vec}\left(B^{(t)}\right)-\frac{\mu }{\varphi }{\nabla }_B\mathcal{L}\left(B^{(t)},{\eta }^{(0)}\right)\right).$$ (III.9)

Here,

• μ is the step size and $\varphi ={\sum }_{i=1}^n{y}_i^2/n$ serves as an approximation for ${\left({\sum }_{k=1}^K{\eta }_k^{*}\right)}^2$ (see Lemma 15);

• $h(B)\in {ℝ}^{1\times K}$ is the thresholding level defined as

  $$h(B)=\sqrt{\frac{4\text{log }np}{n^2}}{\left[{\left\{{\left\{{\left(B^{\top }X\right)}^{\top }\right\}}^3{\eta }^{(0)}-y\right\}}^2\right]}^{\top }*{\left\{{\left\{{\left(B^{\top }X\right)}^{\top }\right\}}^2\odot {\eta }^{(0)\top }\right\}}^2.$$

Step 4: Updating η via Normalization.

We normalize each column of B^(T) and estimate the weight parameter as

$$\begin{gathered} \widehat{B}={\left({\widehat{\beta }}_1,\dots ,{\widehat{\beta }}_K\right)}^{\top } \\ =\left(\frac{{\beta }_1^{(T)}}{{\Vert {\beta }_1^{(T)}\Vert }_2},\dots ,\frac{{\beta }_K^{(T)}}{{\Vert {\beta }_K^{(T)}\Vert }_2}\right), \\ \widehat{\eta }={\left({\widehat{\eta }}_1,\dots ,{\widehat{\eta }}_K\right)}^{\top } \\ ={\left({\eta }_1^{(0)}{\Vert {\beta }_1^{(T)}\Vert }_2^3,\dots ,{\eta }_K^{(0)}{\Vert {\beta }_K^{(T)}\Vert }_2^3\right)}^{\top }. \end{gathered}$$ (III.10)

The final estimator for ${\mathcal{J}}^{*}$ is

$$\widehat{\mathcal{J}}=\sum _{k=1}^K{\widehat{\eta }}_k{\widehat{\beta }}_k\circ {\widehat{\beta }}_k\circ {\widehat{\beta }}_k.$$

Remark 1 (Stochastic Thresholded Gradient descent). The evaluation of the gradient (III.7) requires $\mathcal{O}\left(np{K}^2\right)$ operations at each iteration and can be computationally intense for large n or p. To economize the computational cost, a stochastic version of thresholded gradient descent algorithm can be easily carried out by sampling a subset of summand functions (III.7) at each iteration. This will accelerate the procedure especially in the case of large-scale settings. See Section P2 for details.

IV. Theoretical Analysis

In this section, we establish the geometric convergence rate in optimization error and minimax optimal rate in statistical error of the proposed symmetric tensor estimator.

A. Assumptions

We first introduce the assumptions for theoretical analysis. Conditions 1–3 are on the true tensor parameter ${\mathcal{J}}^{*}$ and Conditions 4–5 are on the measurement scheme. Specifically, the first condition ensures the model identifiability for CP-decomposition.

Condition 1 (Uniqueness of CP-decomposition). The CP-decomposition in (III.2) is unique in the sense that if there exists another CP-decomposition ${\mathcal{J}}^{*}={\sum }_{k=1}^{K^{\prime }}{\eta }_k^{{*}^{\prime }}{\beta }_k^{{*}^{\prime }}\circ {\beta }_k^{{*}^{\prime }}\circ {\beta }_k^{{*}^{\prime }}$, it must have K = K′ and be invariant up to a permutation of {1,…,K}.

For technical purposes, we introduce the following conditions to regularize the CP-decomposition of ${\mathcal{J}}^{*}$. Similar assumptions were imposed in recent tensor literature, e.g., [3, 27] and Assumption 1.1 (A4) [51].

Condition 2 (Parameter space). The CP-decomposition ${\mathcal{J}}^{*}={\sum }_{k=1}^K{\eta }_k^{*}{\beta }_k^{*}\circ {\beta }_k^{*}\circ {\beta }_k^{*}$ satisfies

$${\Vert {\mathcal{J}}^{*}\Vert }_{op}\le C{\eta }_{\text{max}}^{*},K=\mathcal{O}(s),R={\eta }_{\text{max}}^{*}/{\eta }_{\text{min}}^{*}\le C^{\prime }$$ (IV.1)

for some absolute constants C,C′, where ${\eta }_{\text{min}}^{*}={\text{min}}_k{\eta }_k^{*}$ and ${\eta }_{\text{max}}^{*}={\text{max}}_k{\eta }_k^{*}$. Recall that s is the sparsity of ${\beta }_k^{*}$.

Remark 2. In Condition 2, R plays a similar role as a “condition number.” This assumption means that the tensor is “well-conditioned,” i.e., each rank-1 component is roughly of the same size.

As shown in the seminal work of [42], the estimation of low-rank tensors can be NP-hard in general. Hence, we impose the following incoherence condition.

Condition 3 (Parameter incoherence). The true tensor components are incoherent such that

$$\Gamma :=\underset{1\le k_1\ne k_2\le K}{\text{max}}|\langle {\beta }_{k_1}^{*},{\beta }_{k_2}^{*}\rangle |\le \text{min}\left\{C^{\prime \prime }{K}^{-\frac{3}{4}}{R}^{-1},s^{-\frac{1}{2}}\right\},$$

where R is the singular value ratio defined in (IV.1) and C″ is some small constant.

Remark 3. The preceding incoherence condition has been widely used in different scenarios in recent high-dimensional research, such as tensor decomposition [27, 50], compressed sensing [44], matrix decomposition [45], and dictionary learning [46]. It can be also viewed as a relaxation of orthogonality: if $\left\{{\beta }_1^{*},\dots ,{\beta }_K^{*}\right\}$ are mutually orthogonal, Γ equals zero. We can show from both theory (Lemma 28 in the supplementary materials) and simulation (Section VII) that the low-rank tensor ${\mathcal{J}}^{*}$ induced by (III.2) satisfies the incoherence condition with high probability, if the component vectors ${\beta }_k^{*}$ are randomly generated, say from Gaussian distribution.

We also introduce the following conditions on noise distribution.

Condition 4 (Sub-exponential noise). The noise ${\left\{{ϵ}_i\right\}}_{i=1}^n$ are i.i.d. randomly generated with mean 0 and variance σ² satisfying $0<\sigma <C{\sum }_{k=1}^K{\eta }_k^{*}\cdot \left({ϵ}_i/\sigma \right)$ is sub-exponential distributed, i.e., there exists constant C_ϵ > 0 such that ${\left({ϵ}_i/\sigma \right)}_{{\psi }_1}:={\text{sup}}_{p\ge 1}{p}^{-1}{\left(\mathbb{E}{\left|{ϵ}_i/\sigma \right|}^p\right)}^{1/p}\le C_{ϵ}$ and is independent of ${\left\{{\mathfrak{X}}_i\right\}}_{i=1}^n$.

The sample complexity condition is crucial for our algorithm especially in the initialization stage. Ignoring any polylog factors, Condition 5 is even weaker than the sparse matrix estimation case (n ≳ s²) in [48].

Condition 5 (Sample complexity).

$$n\ge C^{\prime \prime \prime }{K}^2{(s\text{ log}(ep/s))}^{\frac{3}{2}}{\text{log}}^{4}n.$$

B. Main Theoretical Results

Our main Theorem 1 shows that based on a proper initializer, the output of the proposed procedure can achieve optimal estimation error rate after a sufficient number of iterations. Here, we define the contraction parameter

$$0<\kappa =1-32\mu K^{-2}{R}^{-\frac{8}{3}}<1$$

and also denote ${\mathcal{E}}_1=4K{\eta }_{\text{max}}^{*\frac{2}{3}}{\epsilon }_0^2$ and ${\mathcal{E}}_2=C_0{\eta }_{\text{min}}^{*-\frac{4}{3}}/16$ for some C₀ > 0.

Theorem 1 (Statistical and Optimization Errors). Suppose Conditions 3–5 hold, $|\text{supp}\left({\beta }_k^{(0)}\right)|\lesssim s$, and the initial estimator ${\left\{{\beta }_k^{(0)},{\eta }_k^{(0)}\right\}}_{k=1}^K$ satisfy

$$\underset{1\le k\le K}{\text{max}}\left\{{\Vert {\beta }_k^{(0)}-{\beta }_k^{*}\Vert }_2,|{\eta }_k^{(0)}-{\eta }_k^{*}|\right\}\lesssim K^{-1}$$ (IV.2)

with probability at least $1-\mathcal{O}(1/n)$. Assume the step size μ ≤ μ₀, where μ₀ is defined in (A.14). Then, the output of the thresholded gradient descent update in (III.9) satisfies:

• For any t = 0,1,2,…, the factor-wise estimator satisfies

  $$\sum _{k=1}^K{\Vert \sqrt[3]{{\eta }_k^{(0)}}{\beta }_k^{(t+1)}-\sqrt[3]{{\eta }_k^{*}}{\beta }_k^{*}\Vert }_2^2\le {\mathcal{E}}_1{\kappa }^t+{\mathcal{E}}_2\frac{{\sigma }^{2}s\text{ log }p}{n}$$ (IV.3)

  with probability at least $1-\mathcal{O}(tKs/n)$.
• When the total number of iterations is no smaller than

  $$T^{*}=\left(\text{log}\left(\frac{n}{{\sigma }^{2}s\text{ log }p}\vee 1\right)+\text{log}\frac{{\mathcal{E}}_1}{{\mathcal{E}}_2}\right)/\text{log }{\kappa }^{-1},$$ (IV.4)

  there exists a constant C₁ (independent of K,s,p,n,σ²) such that the final estimator $\widehat{\mathcal{J}}={\sum }_{k=1}^K{\eta }_k^{(0)}{\beta }_k^{\left(T^{*}\right)}\circ {\beta }_k^{\left(T^{*}\right)}\circ {\beta }_k^{\left(T^{*}\right)}$ satisfies

  $${\Vert \widehat{\mathcal{J}}-{\mathcal{J}}^{*}\Vert }_F^2\le \frac{C_1{\sigma }^{2}Ks\text{ log }p}{n}$$ (IV.5)

  with probability at least $1-\mathcal{O}\left(T^{*}Ks/n\right)$.

Remark 4. The error bound (IV.3) can be decomposed into an optimization error ${\mathcal{E}}_1{\kappa }^t$ (which decays with a geometric rate as iterations) and a statistical error ${\mathcal{E}}_2\frac{{\sigma }^{2}s\text{ log }p}{n}$ (which does not decay as iterations). In the special case that σ = 0, $\widehat{\mathcal{J}}$ exactly recover ${\mathcal{J}}^{*}$ with high probability.

The next theorem shows that Steps 1 and 2 of Algorithm 1 provides a good initializer required in Theorem 1.

Theorem 2 (Initialization Error). Recall $\Gamma ={\text{max}}_{1\le k_1\ne k_2\le K}|{\beta }_{k_1}^{*},{\beta }_{k_2}^{*}|$. Suppose the number of initializations $L\ge K^{C_3{\gamma }^{-4}}$, where γ is a constant defined in (A.11). Given that Conditions 1–4 hold, the initial estimator obtained from Steps 1–2 with a truncation level s ≤ d ≤ Cs satisfies

$$\underset{1\le k\le K}{\text{max}}\left\{{\Vert {\beta }_k^{(0)}-{\beta }_k^{*}\Vert }_2,|{\eta }_k^{(0)}-{\eta }_k^{*}|\right\}\le C_{2}KR{\delta }_{n,p,s}+\sqrt{K}{\Gamma }^2$$ (IV.6)

and

$$\left|\text{supp}\left({\beta }_k^{(0)}\right)\right|\lesssim s$$

with probability at least 1 − 5/n, where

$${\delta }_{n,p,s}={(\text{log }n)}^3\left(\sqrt{\frac{s^3{\text{log}}^3(ep/s)}{n^2}}+\sqrt{\frac{s\text{ log}(ep/s)}{n}}\right).$$ (IV.7)

Moreover, if the sample complexity condition 5 holds, then the above bound satisfies (IV.2).

Remark 5 (Interpretation of initialization error). The upper bound of (IV.6) consists of two terms that correspond to the approximation error of ${\mathcal{T}}_s$ to ${\mathcal{J}}^{*}$ and the incoherence among ${\beta }_k^{*}$’s, respectively. Especially, the former converges to zero as n grows while the latter does not.

The proof of Theorems 1 and 2 are postponed to Section C-D in the supplementary materials. The combination of Theorems 1 and 2 immediately yields the following upper bound for the final estimator, which is one main result of this paper.

Theorem 3 (Upper Bound). Suppose Conditions 1 – 5 hold, s ≤ d ≤ Cs. After T* iterations, there exists a constant C₁ not depending on K,s,p,n,σ², such that the proposed procedure yields

$${\Vert \widehat{\mathcal{J}}-{\mathcal{J}}^{*}\Vert }_F^2\le \frac{C_1{\sigma }^{2}Ks\text{ log }p}{n}$$ (IV.8)

with probability at least $1-\mathcal{O}\left(T^{*}Ks/n\right)$, where T* is defined in (IV.4).

The above upper bound turns out to match the minimax lower bound for a large class of sparse and low-rank tensors.

Theorem 4 (Lower Bound). Consider the following class of sparse and low-rank tensors,

$${\mathcal{F}}_{p,K,s}=\left\{\begin{array}{c}\mathcal{J}={\sum }_{k=1}^K{\eta }_k{\beta }_k\circ {\beta }_k\circ {\beta }_k,\\ \mathcal{J}:\text{ where }{\Vert {\beta }_k\Vert }_0\le s,\text{ for }k\in [K],\\ \mathcal{J}\text{ satisfies Conditions }1-3\end{array}\right\}.$$ (IV.9)

Suppose that ${\left\{{\mathfrak{X}}_i\right\}}_{i=1}^n$ are i.i.d standard normal cubic sketchings with i.i.d. N(0,σ²) noise in (III.1), p ≥ 20s, and s ≥ 4. We have the following lower bound result,

$$\underset{\tilde{\mathcal{T}}}{\text{inf}}\underset{\mathcal{T}\in {\mathcal{F}}_{p,K,s}}{\text{sup}}\mathbb{E}\Vert \tilde{\mathcal{J}}-\mathcal{J}{\Vert }_F^2\ge c{\sigma }^2\frac{Ks\text{ log}(ep/s)}{n}.$$

The proof of Theorem 4 is deferred to Section E in the supplementary materials. Combining Theorems 3 and 4, we immediately obtain the following minimax-optimal rate for sparse and low-rank tensor estimation with cubic sketchings when logp ≍ log(p/s):

$$\underset{\tilde{\mathcal{T}}}{\text{inf}}\underset{{\mathcal{T}}^{*}\in {\mathcal{F}}_p,K,s}{\text{sup}}\mathbb{E}{\Vert \tilde{\mathcal{J}}-{\mathcal{J}}^{*}\Vert }_F^2\asymp {\sigma }^2\frac{Ks\text{ log}(ep/s)}{n}.$$ (IV.10)

The rate in (IV.10) sheds light upon the effect of dimension p, noise level σ², sparsity s, sample size n and rank K to the estimation performance.

Remark 6. Recently, Li, Haupt, and Woodruff [29] studied the optimal sketching for the low-rank tensor regression and gave an near-optimal sketching complexity with a sharp (1 + ε)-worse-case error bound. Different from the framework of [29] that focuses on a deterministic setting, we study a probabilistic model with random observation noises, propose a new algorithm, and studied the minimax optimal rate of estimation errors. In addition, [5, 16, 17] considered different types of convex/non-convex algorithms for low-rank tensor regression with statistical assumptions. To our best knowledge, we are the first to achieve an optimal rate in estimation error based on polynomial-time algorithms for the tensor regression problem.

Remark 7 (Non-sparse low-rank tensor estimation via cubic-sketchings). When the low-rank tensor ${\mathcal{J}}^{*}$ is not necessarily sparse, i.e.,

$${\mathcal{J}}^{*}\in {\mathcal{F}}_{p,K}=\left\{\mathcal{J}:\begin{array}{l}\mathcal{J}=\sum _{k=1}^K{\eta }_k{\beta }_k\circ {\beta }_k\circ {\beta }_k,\hfill \\ \mathcal{J}\text{ satisfies Conditions }1-3\hfill \end{array}\right\},$$

we can apply the proposed procedure with all the truncation/thresholding steps removed. If $n\ge \mathcal{O}\left(p^{3/2}\right)$, we can use similar arguments of Theorems 1–3 to show that the estimator $\widehat{{\mathcal{J}}^{\prime }}$ satisfies

$${\Vert \widehat{{\mathcal{J}}^{\prime }}-{\mathcal{J}}^{*}\Vert }_F^2\lesssim \frac{{\sigma }^{2}Kp}{n}$$ (IV.11)

for any ${\mathcal{J}}^{*}\in {\mathcal{F}}_{p,K}$ with high probability. Furthermore, similar arguments of Theorem 4 imply that the rate in (IV.11) is minimax optimal.

Remark 8 (Comparison with existing matrix results). Our cubic sketching tensor results are far more than extensions of the existing matrix ones. For example, [32, 33] studied the low-rank matrix recovery via rank-1 projections: $y_i=x_i^{\top }T{x}_i+{ϵ}_i$ and proposed the convex nuclear norm minimization methods. The theoretical properties of their estimate are analyzed under a ${\mathcal{l}}_1/{\mathcal{l}}_2\text{-RIP}$ or Restricted Uniform Boundedness condition (RUB). However, the tensor nuclear norm is computationally infeasible and one can check that our cubic sketching framework does not satisfy RIP or RUB conditions in general following the arguments in [48, 52]. Thus, these previous results cannot be directly applied.

In addition, the analysis of gradient updates for the tensor case is significantly more complicated than the matrix case. First, it requires high-order concentration inequalities for the tensor case since the cubic-sketching tensor leads to high-order products of sub-Gaussian random variables (see Section IV-C for details). The necessity of high-order expansions in the analysis of gradient updates for the tensor case also significantly increases the hardness of the problem. To ensure the geometric convergence, we need much more subtle analysis comparing to the ones in the matrix case [52].

C. Key Lemmas: High-order Concentration Inequalities

As mentioned earlier, one major challenge for theoretical analysis of cubic sketching is to handle heavy tails of high-order Gaussian moments. One can only handle up-to second moments of sub-Gaussian random variables by directly applying the Hoeffding’s or Bernstein’s concentration inequalities. Therefore, we need to develop the following high-order concentration inequalities as technical tools: Lemma 1 characterizes the tail bounds for the sum of sub-Gaussian products, and Lemma 2 provides the concentration inequalities for Gaussian cubic sketchings. The proofs of Lemmas 1 and 2 are given in Section B.

Lemma 1 (Concentration inequality for sum of sub-Gaussian products). Suppose $X_i={\left(x_{1i}^{\top },\dots ,x_{mi}^{\top }\right)}^{\top }\in {ℝ}^{m\times p},i\in [n]$ are n i.i.d random matrices. Here, suppose x_ij, the j-th row of X_i, is an isotropic sub-Gaussian vector, i.e., $\mathbb{E}{x}_{ij}=0$ and Cov(x_ij) = I. Then for any vectors $a=\left(a_1\dots ,a_n\right)\in {ℝ}^n$, ${\left\{{\beta }_j\right\}}_{j=1}^m\subseteq {ℝ}^p$, and 0 < δ < 1, we have

$$\begin{gathered} \left|\sum _{i=1}^n{a}_i\prod _{j=1}^m\left(x_{ij}^{\top }{\beta }_j\right)-\mathbb{E}\left(\sum _{i=1}^n{a}_i\prod _{j=1}^m\left(x_{ij}^{\top }{\beta }_j\right)\right)\right| \\ \le C\prod _{i=1}^m{\Vert {\beta }_j\Vert }_2\left(\Vert a{\Vert }_{\infty }{\left(\text{log }{\delta }^{-1}\right)}^{m/2}+\Vert a{\Vert }_2{\left(\text{log }{\delta }^{-1}\right)}^{1/2}\right) \end{gathered}$$

with probability at least 1 – δ for some constant C.

Note that in Lemma 1, each X_i does not necessarily have independent entries, even though ${\left\{X_i\right\}}_{i=1}^n$ are independent matrices. Building on Lemma 1, Lemma 2 provides a generic spectral-type concentration inequality that can be used to quantify the approximation error of ${\mathcal{T}}_s$ introduced in Step 1 of the proposed procedure.

Lemma 2 (Concentration inequality for Gaussian cubic sketchings). Suppose ${\left\{x_{1i}\right\}}_{i=1}^n\stackrel{iid}{~}\mathcal{N}\left(0,I_{p_1}\right)$, ${\left\{x_{2i}\right\}}_{i=1}^n\stackrel{iid}{~}\mathcal{N}\left(0,I_{p_2}\right)$, ${\left\{x_{3i}\right\}}_{i=1}^n\stackrel{iid}{~}\mathcal{N}\left(0,I_{p_3}\right)$, ${\beta }_1\in {ℝ}^{p_1}$, ${\beta }_2\in {ℝ}^{p_2}$, ${\beta }_3\in {ℝ}^{p_3}$ are fixed vectors.

• Define $M_{\text{nsy}}=\frac{1}{n}{\sum }_{i=1}^n\langle x_{1i}\circ x_{2i}\circ x_{3i},{\beta }_1\circ {\beta }_2\circ {\beta }_3\rangle x_{1i}\circ x_{2i}\circ x_{3i}$. Then $\mathbb{E}\left(M_{\text{nsy }}\right)={\beta }_1\circ {\beta }_2\circ {\beta }_3$ and

  $${\Vert M_{\text{nsy}}-\mathbb{E}\left(M_{\text{nsy}}\right)\Vert }_s\le C{(\text{log }n)}^3(\sqrt{\frac{s^3{\text{log}}^3(ep/s)}{n^2}}+\sqrt{\frac{s\text{ log}(ep/s)}{n}}){\Vert {\beta }_1\Vert }_2{\Vert {\beta }_2\Vert }_2{\Vert {\beta }_3\Vert }_2,$$

  with probability at least 1 − 10/n³ − 1/p.
• Define $M_{\text{sym}}=\frac{1}{n}{\sum }_{i=1}^n\langle x_{1i}\circ x_{1i}\circ x_{1i},{\beta }_1\circ {\beta }_1\circ {\beta }_1\rangle x_{1i}\circ x_{1i}\circ x_{1i}$. Then $\mathbb{E}\left(M_{\text{sym}}\right)=6{\beta }_1\circ {\beta }_1\circ {\beta }_1+3{\sum }_{m=1}^p\left({\beta }_1\circ e_m\circ e_m+e_m\circ {\beta }_1\circ e_m+e_m\circ e_m\circ {\beta }_1\right)$ and

  $$\begin{gathered} {\Vert M_{\text{sym}}-\mathbb{E}\left(M_{\text{sym}}\right)\Vert }_s \\ \le C{(\text{log }n)}^3\left(\sqrt{\frac{s^3{\text{log}}^3(ep/s)}{n^2}}+\sqrt{\frac{s\text{ log}(ep/s)}{n}}\right){\Vert {\beta }_1\Vert }_2^3, \end{gathered}$$

  with probability at least 1 − 10/n³ − 1/p.

Here, C is an absolute constant and ‖ · ‖_s is the sparse tensor spectral norm defined in (II.3).

V. Application To High-Order Interaction Effect Models

In this section, we study the high-order interaction effect model in the cubic sketching framework. Specifically, we consider the following three-way interaction model

$$y_l={\xi }_0+\sum _{i=1}^p{\xi }_i{z}_{li}+\sum _{i,j=1}^p{\gamma }_{ij}{z}_{li}{z}_{lj}+\sum _{i,j,k=1}^p{\eta }_{ijk}{z}_{li}{z}_{lj}{z}_{lk}+{ϵ}_l,$$ (V.1)

for l = 1,…,n. Here ξ, γ, and η are coefficients for the main effect, pairwise interaction, and triple-wise interaction, respectively. More importantly, (V.1) can be reformulated into the following tensor form (also see the left panel of Figure 1)

$$y_l=\langle \mathcal{B},x_l\circ x_l\circ x_l\rangle +{ϵ}_l,l=1,\dots ,n,$$ (V.2)

where $x_l={\left(1,z_l^{\top }\right)}^{\top }\in {ℝ}^{p+1}$ and $\mathcal{B}\in {ℝ}^{(p+1)\times (p+1)\times (p+1)}$ is a tensor parameter corresponding to coefficients in the following way:

$$\{\begin{array}{l}{\mathcal{B}}_{[0,0,0]}={\xi }_0,\hfill \\ {\mathcal{B}}_{[1:p,1:p,1:p]}={\left({\eta }_{ijk}\right)}_{1\le i,j,k\le p},\hfill \\ {\mathcal{B}}_{[0,1:p,1:p]}={\mathcal{B}}_{[1:p,0,1:p]}={\mathcal{B}}_{[1:p,1:p,0]}={\left({\gamma }_{ij}/3\right)}_{1\le i,j\le p},\hfill \\ {\mathcal{B}}_{[0,0,1:p]}={\mathcal{B}}_{[0,1:p,0]}={\mathcal{B}}_{[1:p,0,0]}={\left({\xi }_i/3\right)}_{1\le i\le p}.\hfill \end{array}$$ (V.3)

We provide the following justification for assuming the tensorized coefficient $\mathcal{B}$ is low-rank and sparse. First, in modern applications, such as the biomedical research [53], the response is often driven by a small portion of coefficients and a small number of factors, leading to a highly entry-wise sparse and low-rank $\mathcal{B}$. Second, [54] suggested that it is suitable to model entry-wise sparse and low-enough rank tensors as arising from sparse loadings. Therefore, we assume $\mathcal{B}$ is CP rank-K with s-sparse factors:

$$\mathcal{B}=\sum _{k=1}^K{\eta }_k{\beta }_k\circ {\beta }_k\circ {\beta }_k,{\Vert {\beta }_k\Vert }_0\le s,$$

where K,s ≪ p. Then the number of parameters in (V.4), K(p + 1), is significantly smaller than (p + 1)³, the total number of parameters in the original three-way interaction effect model (V.1), which makes the consistent estimation of $\mathcal{B}$ possible in the high-dimensional case. In this case, (V.2) can be written as

$$y_l=\langle \sum _{k=1}^K{\eta }_k{\beta }_k\circ {\beta }_k\circ {\beta }_k,x_l\circ x_l\circ x_l\rangle +{ϵ}_l,$$ (V.4)

where l ∈ [n], ‖β_k‖₂ = 1, ‖β_k‖₀ ≤ s,k ∈ [K].

By assuming $z_l\stackrel{iid}{~}{N}_p\left(0,I_p\right)$, the high-order interaction effect model (V.2) reduces to the symmetric tensor estimation model (III.1), except one slight difference that the first coordinate of x_l, i.e., the intercept, is always 1. To accommodate this difference, we only need to adjust the initial unbiased estimate in the above two-step procedure. Let

$${\mathcal{T}}_s=\frac{1}{6n}\sum _{l=1}^n{y}_l{x}_l\circ x_l\circ x_l-\frac{1}{6}\sum _{j=1}^p\left(a\circ e_j\circ e_j+e_j\circ a\circ e_j+e_j\circ e_j\circ a\right),$$ (V.5)

where $a=\frac{1}{n}{\sum }_{l=1}^n{y}_l{x}_l$. Then we construct the empirical moment-based initial tensor Ts′ as

• For i,j,k ≠ 0, ${\mathcal{T}}_{s^{\prime }[i,j,k]}={\mathcal{T}}_{s[i,j,k]}$, ${\mathcal{T}}_{s^{\prime }[i,j,0]}={\mathcal{T}}_{s[i,j,0]}$, ${\mathcal{T}}_{s^{\prime }[0,j,k]}={\mathcal{T}}_{s[0,j,k]}$, and ${\mathcal{T}}_{s^{\prime }}[i,0,k]={\mathcal{T}}_{s[i,0,k]}$.

• For i ≠ 0, ${\mathcal{T}}_{s^{\prime }[0,0,i]}={\mathcal{T}}_{s^{\prime }[0,i,0]}={\mathcal{T}}_{s^{\prime }[i,0,0]}=\frac{1}{3}{\mathcal{T}}_{s[0,0,i]}-\frac{1}{6}\left({\sum }_{k=1}^p{\mathcal{T}}_{s[k,k,i]}-(p+2)a_i\right)$.

• ${\mathcal{T}}_{s^{\prime }[0,0,0]}=\frac{1}{2p-2}\left({\sum }_{k=1}^p{\mathcal{T}}_{s[0,k,k]}-(p+2){\mathcal{T}}_{s[0,0,0]}\right)$.

Lemma 5 shows that ${\mathcal{T}}_{s^{\prime }}$ is an unbiased estimator for $\mathcal{B}$.

The theoretical results in Section IV imply the following upper and lower bounds for the three-way interaction effect estimation.

Corollary 1. Suppose z₁,…,z_n are i.i.d. standard Gaussian random vectors and $\mathcal{B}$ satisfies Conditions 1, 2 and 3. The output, denoted as $\widehat{\mathcal{B}}$, from the proposed Algorithms 1 and 2 based on ${\mathcal{T}}_{s^{\prime }}$ satisfies

$$\Vert \widehat{\mathcal{B}}-\mathcal{B}{\Vert }_F^2\le C\frac{{\sigma }^{2}Ks\text{ log }p}{n}$$ (V.6)

with high probability. On the other hand, considering the following class of $\mathcal{B}$,

$${\mathcal{F}}_{p+1,K,s}=\left\{\begin{array}{c}\mathcal{B}={\sum }_{k=1}^K{\eta }_k{\beta }_k\circ {\beta }_k\circ {\beta }_k\\ \mathcal{B}:\text{ where }{\Vert {\beta }_k\Vert }_0\le s,\text{ for }k\in [K],\\ \mathcal{B}\text{ satisfies Conditions }1-3,\end{array}\right\}.$$

Then the following lower bound holds,

$$\underset{\widehat{\mathcal{B}}}{\text{inf}}\underset{\mathcal{B}\in {\mathcal{F}}_{p+1,K,s}}{\text{sup}}\mathbb{E}\Vert \widehat{\mathcal{B}}-\mathcal{B}{\Vert }_F^2\ge C\frac{{\sigma }^{2}Ks\text{ log }p}{n}.$$

VI. Non-Symmetric Tensor Estimation Model

In this section, we extend the previous results to the non-symmetric tensor case. Specifically, we have ${\mathcal{J}}^{*}\in {ℝ}^{p_1\times p_2\times p_3}$ and

$$y_i=\langle {\mathcal{T}}^{*},{\mathcal{X}}_i\rangle +{ϵ}_i,{\mathcal{X}}_i=u_i\circ v_i\circ w_i,i\in [n],$$ (VI.1)

where $u_i\in {ℝ}^{p_1}$, $v_i\in {ℝ}^{p_2}$, $w_i\in {ℝ}^{p_3}$ are random vectors with i.i.d. standard normal entries. Again, we assume ${\mathcal{J}}^{*}$ is sparse and low-rank in a similar sense that

$$\begin{gathered} {\mathcal{J}}^{*}=\sum _{k=1}^K{\eta }_k^{*}{\beta }_{1k}^{*}\circ {\beta }_{2k}^{*}\circ {\beta }_{3k}^{*}, \\ {\Vert {\beta }_{1k}^{*}\Vert }_2={\Vert {\beta }_{2k}^{*}\Vert }_2={\Vert {\beta }_{3k}^{*}\Vert }_2=1, \\ \text{max}\left\{{\Vert {\beta }_{1k}^{*}\Vert }_0,{\Vert {\beta }_{2k}^{*}\Vert }_0,{\Vert {\beta }_{3k}^{*}\Vert }_0\right\}\le s. \end{gathered}$$ (VI.2)

Denote

• B₁ = (β₁₁, ⋯, β_1K), B₂ = (β₂₁, ⋯, β_2K), B₃ = (β₃₁, ⋯, β_3K),

• U = (u₁,…,u_n), V = (v₁,…,v_n), W = (w₁,…,w_n), η = (η₁,…,η_k)^⊤,y = (y₁,…,y_n)^⊤.

Then, the empirical risk function can be written compactly as

$$\mathcal{L}\left(B_1,B_2,B_3,\eta \right)=\frac{1}{n}{\Vert \left(U^{\top }{B}_1\right)*\left(V^{\top }{B}_2\right)*\left(W^{\top }{B}_3\right)\cdot \eta -y\Vert }_2^2.$$ (VI.3)

Since (VI.3) is non-convex but fortunately tri-convex in terms of B₁, B₂, and B₃, we develop a block-wise thresholded gradient descent algorithm as detailed below. The complete algorithm is deferred to Section O1 in the supplementary materials.

Step 1: (Method of Tensor Moments)

Construct the empirical moment-based estimator

$$\mathcal{T}:=\frac{1}{n}\sum _{i=1}^n{y}_i{u}_i\circ v_i\circ w_i\in {ℝ}^{p_1\times p_2\times p_3}$$ (VI.4)

to which sparse tensor decomposition is applied for initialization.

Step 2: (Block-wise Gradient Descent)

Lemma 17 shows that the gradient function for (VI.3) with respect to B₁ can be written as

$${\nabla }_{B_1}\mathcal{L}\left(B_1,B_2,B_3,\eta \right)=D^{\top }{\left(C_1^{\top }\odot U\right)}^{\top }\in {ℝ}^{1\times \left(p_{1}K\right)},$$ (VI.5)

where $D={\left(B_1^{\top }U\right)}^{\top }*{\left(B_2^{\top }V\right)}^{\top }*{\left(B_3^{\top }W\right)}^{\top }\eta -y$ and $C_1={\left(B_2^{\top }V\right)}^{\top }*{\left(B_3^{\top }W\right)}^{\top }\odot {\eta }^{\top }$. For t = 1, …, T, we fix $B_2^{(t)}$, $B_3^{(t)}$ and update $B_1^{(t+1)}$ via block-wise thresholded gradient descent,

$$\text{vec}\left(B_1^{(t+1)}\right)={\phi }_{\frac{\mu h\left(B_1^{(t)}\right)}{\varphi }}(\text{vec}\left(B_1^{(t)}\right)-\frac{\mu }{\varphi }{\nabla }_{B_1}\mathcal{L}\left(B_1^{(t)},B_2^{(t)},B_3^{(t)},\eta \right)),$$

where $\varphi ={\sum }_{i=1}^n{y}_i^2/n$, μ is the step size, and $h\text{(}B\text{)=}\sqrt{\frac{4\text{log}np}{n^2}{\left\{D^2\right\}}^{\top }\left\{C^2\right\}}$. The updates of B₂,B₃ are similar.

The theoretical analysis for the non-symmetric case is different from the symmetric one in two folds. First, the non-symmetric cubic sketching tensor is formed by three Gaussian vectors rather than one, which leads to many differences in the calculation of high-order moments. Second, the CP-decomposition of non-symmetric tensor ${\mathcal{J}}^{*}$ (VI.2) forms a tri-convex optimization. At this point, the standard convex analysis for vanilla gradient descent [55] could be applied given a proper initialization.

With the regularity conditions detailed in Section O1, we present the theoretical results for non-symmetric tensor estimation as follows.

Theorem 5 (Upper Bound). Suppose Conditions 6 – 9 hold and n ≳ (slog(p₀/s))^3/2, where p₀ = max{p₁,p₂,p₃}. For any t = 0,1,2,…, the output of Algorithm O1 satisfies

$$\sum _{k=1}^K\sum _{j=1}^3{\Vert \sqrt[3]{{\eta }_k}{\beta }_{jk}^{(t+1)}-\sqrt[3]{{\eta }_k^{*}}{\beta }_{jk}^{*}\Vert }_2^2\le {\mathcal{O}}_p\left({\kappa }^t+\frac{{\sigma }^{2}s\text{ log }{p}_0}{n}\right)$$

for some 0 < κ < 1. When the total number of iterations is no smaller than $\text{log}\left(\frac{n}{{\sigma }^{2}s\text{log}{p}_0}\vee 1\right)/\text{log}{\kappa }^{-1}$, the final estimator $\widehat{\mathcal{J}}$ satisfies

$${\Vert \widehat{\mathcal{J}}-{\mathcal{J}}^{*}\Vert }_F^2\le {\mathcal{O}}_p\left(\frac{{\sigma }^{2}Ks\text{ log }{p}_0}{n}\right).$$

Theorem 6 (Lower Bound). Consider the class of incoherent sparse and low-rank tensors $\mathcal{F}=\{\mathcal{J}:\mathcal{J}={\sum }_{k=1}^K{\beta }_{1k}\circ {\beta }_{2k}\circ {\beta }_{3k},{\Vert {\beta }_{i,k}\Vert }_0\le s\text{ for }i=1,2,3,k=1,\dots ,K\}$. If ${\left\{{\mathfrak{X}}_i\right\}}_{i=1}^n$ are i.i.d standard normal cubic sketchings, $ϵ\stackrel{iid}{~}N\left(0,{\sigma }^2\right)$, min{p₁,p₂,p₃} ≥ 20s, and s ≥ 4, we have

$$\underset{\widehat{\mathcal{J}}}{\text{inf}}\underset{\mathcal{J}\in \mathcal{F}}{\text{sup}}\mathbb{E}\Vert \widehat{\mathcal{J}}-\mathcal{J}{\Vert }_F^2\ge \frac{C{\sigma }^{2}sK\text{ log}\left(e\cdot p_0/s\right)}{n}.$$ (VI.6)

Theorems 5 and 6 imply that the proposed algorithm achieves a minimax-optimal rate of estimation error in the class of $\mathcal{F}$ as long as log(p₀) ≍ log(p₀/s).

VII. Numerical Results

In this section, we investigate the effect of noise level, CP-rank, sample size, dimension, and sparsity on the estimation performance by simulation studies. We also investigate the numerical performance of the proposed algorithm when the incoherence assumption required in the theoretical analysis fails to hold.

In each setting, we generate ${\mathcal{J}}^{*}={\sum }_{k=1}^K{\beta }_k^{*}\circ {\beta }_k^{*}\circ {\beta }_k^{*}$, where $\left|\text{supp}\left({\beta }_k^{*}\right)\right|=s$, the support of ${\beta }_k^{*}$ is uniformly selected from {1,…,p}, and the nonzero entries of ${\beta }_k^{*}$ are drawn randomly from standard normal distribution. Then, we calculate ${\eta }_k^{*}\leftarrow {\Vert {\beta }_k^{*}\Vert }_2^3$ and normalize ${\beta }_k^{*}\leftarrow {\beta }_k^{*}/{\Vert {\beta }_k^{*}\Vert }_2$. The cubic sketchings ${\left\{{\mathfrak{X}}_i\right\}}_{i=1}^n$ are generated as ${\mathfrak{X}}_i=x_i\circ x_i\circ x_i$ and $x_i\stackrel{iid}{~}N(0,1)$. The noise satisfies ${\left\{{ϵ}_i\right\}}_{i=1}^n\stackrel{iıd}{~}N\left(0,{\sigma }^2\right)$ or $\text{Laplace}(0,\sigma /\sqrt{2})$. Additionally, we adopt the following stopping rules in iterations: (1) the initialization iteration (Step 2 in Algorithm 1) is stopped if ${\Vert b_m^{(l+1)}-b_m^{(l)}\Vert }_2\le {10}^{-6}$; (2) the gradient update iteration (Step 3 in Algorithm 2) is stopped if ‖B^(T+1) − B^(T) ‖_F ≤ 10⁻⁶. The numerical results are based on 200 repetitions unless otherwise specified. The code was written in R and implemented on an Intel Xeon-E5 processor with 64 GB of RAM.

First, we consider the percentage of successful recovery in the noiseless case. Let K = 3, s/p = 0.3, p = 30 or 50, so that the total number of unknown parameters in ${\mathcal{J}}^{*}$ is 2.7 × 10⁴ or 1.25 × 10⁵. The sample size n ranges from 500 to 6000. Each recovery is called “successful” if the relative error ${\Vert \widehat{\mathcal{J}}-{\mathcal{J}}^{*}\Vert }_F/{\Vert {\mathcal{J}}^{*}\Vert }_F<{10}^{-4}$. We report the average successful recovery rate in Figure VII. We can see from Figure VII that the empirical relation among successful recovery, dimension, and sample size is consistent with the theoretical results in Section IV.

We then move to the noisy case. Select K = 3, s/p = 0.3, p ∈ {30, 50}, ${\left\{{ϵ}_i\right\}}_{i=1}^n\stackrel{iid}{~}N\left(0,{\sigma }^2\right)$. We consider two scenarios: (1) sample size n = 6000, 8000, or 10000, s/p = 0.3, the noise level σ varies from 0 to 200; (2) noise level σ = 200, sample size n varies from 4000 to 10000, p = 30, s/p = 0.1,0.3,0.5. The estimation errors in terms of ${\Vert \widehat{\mathcal{J}}-{\mathcal{J}}^{*}\Vert }_F/{\Vert {\mathcal{J}}^{*}\Vert }_F$ in these two scenarios are plotted in Figures 3 and 4, respectively. These results show that the proposed procedure achieves a good performance – Algorithms 1 and 2 yield more accurate estimation with smaller variance σ² and/or large value of sample size n.

Fig. 3.

Estimation error under different noise levels. Left panel: p = 30, right panel: p = 50

Fig. 4.

Estimation error under different dimension/sample ratio (n/p³). Left panel: initial estimation error, right panel: final estimation error

Next, we demonstrate that the low-rank tensor parameter ${\mathcal{J}}^{*}$ with randomly generated factors ${\beta }_k^{*}$ satisfies the incoherence condition 3 with high probability. Set the CP-rank K = 3 and the sparsity level s/p = 0.3 with the dimension p ranging from 10 to 2000. We compute the incoherence parameter Γ defined in Condition 3. The left panel of Figure 5 shows that the incoherence parameter Γ decays in a polynomial rate as s grows, which matches the bound in Condition 3. Recall a theoretical justification on this point is also provided in Lemma 28.

Fig. 5.

Left panel: incoherence parameter Γ with varying sparsity. Here, the red line corresponds to the rate $\sqrt{s}$ required in the theoretical analysis. Right panel: average relative estimation error for tensors with varying incoherence.

We further examine the performance of the proposed algorithm when the incoherence condition required in the theoretical analysis fails to hold. Specifically, we set the CP-rank K = 3, p = 30, and the sparsity level s/p = 0.3. We construct enormous copies of tensor parameter ${\mathcal{J}}_j^{*}$ with i.i.d. standard normal factor vectors ${\beta }_k^{*}$. For each ${\mathcal{J}}_j^{*}$, we calculate the incoherence Γ_j defined in Condition 3, then manually pick 40 ${\mathcal{J}}_{j^{\prime }}^{*}$ such that

$$0.01\cdot \left(j^{\prime }-1\right)\le {\Gamma }_{j^{\prime }}\le 0.01\cdot j^{\prime }\text{ for }{j}^{\prime }=\{1,2,\dots ,40\}.$$

In this way, we obtain a set of tensor parameters $\left\{{\mathcal{J}}_{j^{\prime }}^{*}\right\}$ with incoherence uniformly varying from 0 to 0.4. The right panel of Figure 5 plots the relative error for estimating ${\mathcal{J}}^{*}$ based on observations from cubic sketchings of ${\mathcal{J}}_{j^{\prime }}^{*}$ based on 1000 repetitions. We can see that the proposed algorithm achieves small relative errors even when the true factors are highly coherent.

Moreover, we consider a setting with Laplacian noise. Suppose ${\left\{{ϵ}_i\right\}}_{i=1}^n\stackrel{iid}{~}Lap(\sigma )$ with density $f(x)=\frac{1}{\sigma }\text{exp}(-2\left|x\right|/\sigma )$. With n = 3000, p = 30, and varying values of σ, the average estimation error and its comparison with Gaussian noise setting are provided in Figure 6. We note that the estimation errors under Laplace noise are slightly higher than those under Gaussian noise.

Fig. 6.

Comparison of estimation errors between Laplace error and Gaussian error

We also compare the estimation errors of initial and final estimators for different ranks and sample sizes. Set K = 3,p = 30,s/p = 0.3 and consider the noiseless setting. It is clear from Figure 7 that the initialization error decays sufficiently, but does not converge to zero as sample size n grows. This result matches our theoretical findings in Theorem 2: as discussed in Remark 5, the initial stage may yield an inconsistent estimator due to the incoherence among β_k’s. We also evaluate and compare the estimation errors for both initial and final estimators. From the right panel of Figure 7, we can see that the final estimator is more stable and accurate compared to the initial one, which illustrates the merit of thresholded gradient descent step of the proposed procedure.

Fig. 7.

Log relative estimation error of initial estimation error (left panel) and initialization/final estimation error (right panel)

Finally, we compare the performance of the proposed method with the alternating least square (ALS)-based tensor regression method [3]. We specifically consider two schemes for the initialization of ALS: (a) $\left\{{\beta }_k^{(0)}\right\}$ are i.i.d. standard Gaussian (cold start), and (b) $\left\{{\beta }_k^{(0)}\right\}$ are generated from the proposed Algorithm 1 (warm start). Setting K = 2, s/p = 0.2, p = 30, ${\left\{{ϵ}_i\right\}}_{i=1}^n\stackrel{iid}{~}N\left(0,{200}^2\right)$, we apply both the proposed procedure and the ALS-based algorithm and record the average estimation errors with standard deviations for both initial and final estimators. From the result in Table VII, one can see the proposed algorithm significantly outperforms the ALS under both cold and warm start schemes. The main reason is pointed out in Remark 8: the cubic sketching setting possesses distinct aspects compared with the i.i.d. random Gaussian sketching setting, so that the method proposed by [3] does not exactly fit here.

VIII. Discussions

This paper focuses on the third order tensor estimation via cubic sketchings. Moreover, all results can be extended to the higher-order case via high-order sketchings. To be specific, suppose

$$y_i=\langle {\mathcal{J}}^{*},x_i^{\otimes m}\rangle +{ϵ}_i,i=1,\dots ,n,$$

where ${\mathcal{J}}^{*}\in {\left({ℝ}^p\right)}^{\otimes m}$ is an order-m, sparse, and low-rank tensor. In order to estimate ${\mathcal{J}}^{*}$ based on ${\left\{y_i,x_i\right\}}_{i=1}^n$, one can first construct the order-m moment-based estimator using a generalized version of Theorem 7 and the fact that the score functions ${\mathcal{S}}_m(x)={(-1)}^m{\nabla }^{m}p(x)/p(x)$ for the density function p(x) satisfy a nice recursive equation:

$${\mathcal{S}}_m(x):=-{\mathcal{S}}_{m-1}(x)\circ \nabla \text{log }p(x)-\nabla {\mathcal{S}}_{m-1}(x).$$

Then, one can similarly perform high-order sparse tensor decomposition and thresholded gradient descent to estimate ${\mathcal{J}}^{*}$. On the theoretical side, we can show if mild conditions hold and n ≥ C(logn)^m(slogp)^m/2, the proposed procedure achieves

$${\Vert \tilde{\mathcal{T}}-{\mathcal{T}}^{*}\Vert }_F^2\lesssim {\sigma }^2\frac{Kms\text{ log}(p/s)}{n}$$

with high probability. The minimax optimality can be shown similarly.

Fig. 2.

Successful rate of recovery with varying sample size

TABLE I.

Estimation Error And Standard Deviation (In Subscript) Of The Proposed Method And Als-Based Method

Sample size	ours	warm start	cold start	initial
n = 4000	4.020.13	32.821.79	37.781.23	38.031.74
n = 5000	4.020.13	32.342.34	36.962.10	33.711.78
n = 6000	1.770.09	22.221.21	59.973.40	25.571.48

Biography

Botao Hao received B.S. degree from the School of Mathematics, Nankai University, China, in 2014, and Ph.D. from the Department of Statistics, Purdue University, USA, 2019. He is currently a postdoctoral researcher in Department of Electrical Engineering, Princeton University, USA.

Anru Zhang received the Ph.D. degree from University of Pennsylvania, Philadelphia, PA, in 2015, and B.S. degree from Peking University, Beijing, China, in 2010. He is currently an assistant professor in Statistics at the University of Wisconsin-Madison, Madison, WI. His current research interests include high-dimensional statistical inference, tensor data analysis, statistical learning theory, dimension reduction, and convex/non-convex optimization.

Guang Cheng received BA degree in Economics from Tsinghua University, China, in 2002, and PhD degree from University of Wisconsin–Madison in 2006. He then joined Dept of Statistics at Duke University as Visiting Assistant Professor and Postdoc Fellow in SAMSI. He is currently Professor in Statistics at Purdue University, directing Big Data Theory research group, whose main goal is to develop computationally efficient inferential tools for big data with statistical guarantees.

Appendix

This appendix contains five parts: (1) Sections A–B provide detailed proofs for empirical moment estimator and concentration results; (2) Sections C–N provide additional proofs for the main theoretical results of this paper; (4) Section O covers the pseudo-code, conditions and main proofs of non-symmetric tensor estimation; (5) Section P discusses the matrix form of gradient function and stochastic gradient descent; (6) Section Q provides several technical lemmas and their proofs.

A. Moment Calculation

We first introduce three lemmas to show that the empirical moment based tensors (III.6), (V.5), and (VI.4) are all unbiased estimators for the target low-rank tensor in the corresponding scenarios. Detail proofs of three lemmas are postponed to Sections G1, G2 and G3 in the supplementary materials.

Lemma 3 (Unbiasedness of moment estimator under non-symmetric sketchings). For non-symmetric tensor estimation model (VI.1) & (VI.2), define the empirical moment-based tensor $\mathcal{T}$ by

$$\mathcal{T}:=\frac{1}{n}\sum _{i=1}^n{y}_i{u}_i\circ v_i\circ w_i.$$

Then $\mathcal{T}$ is an unbiased estimator for ${\mathcal{J}}^{*}$, i.e.,

$$\mathbb{E}(\mathcal{T})=\sum _{k=1}^K{\eta }_k^{*}{\beta }_{1k}^{*}\circ {\beta }_{2k}^{*}\circ {\beta }_{3k}^{*}.$$

The extension to the symmetric case is non-trivial due to the dependency among three identical sketching vectors. We borrow the idea of high-order Stein’s identity, which was originally proposed in [49]. To fix the idea, we present only third order result for simplicity. The extension to higher-order is straightforward.

Theorem 7 (Third-order Stein’s Identity, [49]). Let $x\in {ℝ}^p$ be a random vector with joint density function p(x). Define the third order score function ${\mathcal{S}}_3(x):{ℝ}^p\to {ℝ}^{p\times p\times p}$ as ${\mathcal{S}}_3(x)=-{\nabla }^{3}p(x)/p(x)$. Then for continuously differentiable function $G(x):{ℝ}^p\to ℝ$, we have

$$\mathbb{E}\left[G(x)\cdot {\mathcal{S}}_3(x)\right]=\mathbb{E}\left[{\nabla }^{3}G(x)\right].$$ (A.1)

In general, the order-m high-order score function is defined as

$${\mathcal{S}}_m(x)={(-1)}^m\frac{{\nabla }^{m}p(x)}{p(x)}.$$

Interestingly, the high-order score function has a recursive differential representation

$${\mathcal{S}}_m(x):=-{\mathcal{S}}_{m-1}(x)\circ \nabla \text{log }p(x)-\nabla {\mathcal{S}}_{m-1}(x),$$ (A.2)

with ${\mathcal{S}}_0(x)=1$. This recursive form is helpful for constructing unbiased tensor estimator under symmetric cubic sketchings. Note that the first order score function ${\mathcal{S}}_1(x)=-\nabla \text{log}p(x)$ is the same as score function in Lemma 26 (Stein’s lemma [56]). The proof of Theorem 7 relies on iteratively applying the recursion representation of score function (A.2) and the first-order Stein’s lemma (Lemma 26). We provide the detailed proof in Section F for the sake of completeness.

In particular, if x follows a standard Gaussian vector, each order score function can be calculated based on (A.2) as follows,

$$\begin{array}{l}{\mathcal{S}}_1(x)=x,{\mathcal{S}}_2(x)=x\circ x-I_{d\times d},\\ {\mathcal{S}}_3(x)=x\circ x\circ x-\sum _{j=1}^p\left(x\circ e_j\circ e_j+e_j\circ x\circ e_j+e_j\circ e_j\circ x\right).\end{array}$$ (A.3)

Interestingly, if we let $G(x)={\sum }_{k=1}^K{\eta }_k^{*}{\left(x^{\top }{\beta }_k^{*}\right)}^3$, then

$$\frac{1}{6}{\nabla }^{3}G(x)=\sum _{k=1}^K{\eta }_k^{*}{\beta }_k^{*}\circ {\beta }_k^{*}\circ {\beta }_k^{*},$$ (A.4)

which is exactly ${\mathcal{J}}^{*}$. Connecting this fact with (A.1), we are able to construct the unbiased estimator in the following lemma through high-order Stein’s identity.

Lemma 4 (Unbiasedness of moment estimator under symmetric sketchings). Consider the symmetric tensor estimation model (III.1) & (IV.9). Define the empirical first-order moment $m_1:=\frac{1}{n}{\sum }_{i=1}^n{y}_i{x}_i$. If we further define an empirical third-order-moment-based tensor ${\mathcal{T}}_s$ by

$${\mathcal{T}}_s:=\frac{1}{6}[\frac{1}{n}\sum _{i=1}^n{y}_i{x}_i\circ x_i\circ x_i-\sum _{j=1}^p\left(m_1\circ e_j\circ e_j+e_j\circ m_1\circ e_j+e_j\circ e_j\circ m_1\right)],$$

then

$$\mathbb{E}\left({\mathcal{T}}_s\right)=\sum _{k=1}^K{\eta }_k^{*}{\beta }_k^{*}\circ {\beta }_k^{*}\circ {\beta }_k^{*}.$$

Proof. Note that y_i = G(x_i) + ϵ_i. Then we have

$$\mathbb{E}\left(\frac{1}{n}\sum _{i=1}^n{y}_i{\mathcal{S}}_3(x)\right)=\mathbb{E}\left(\frac{1}{n}\sum _{i=1}^n\left(G\left(x_i\right)+{ϵ}_i\right){\mathcal{S}}_3\left(x_i\right)\right),$$

where ${\mathcal{S}}_3(x)$ is defined in (A.3). By using the conclusion in Theorem 7 and the fact (A.4), we obtain

$$\mathbb{E}\left({\mathcal{T}}_s\right)=\mathbb{E}\left(\frac{1}{6n}\sum _{i=1}^n{y}_i{\mathcal{S}}_3(x)\right)=\sum _{k=1}^K{\eta }_k^{*}{\beta }_k^{*}\circ {\beta }_k^{*}\circ {\beta }_k^{*},$$

since ϵ_i is independent of x_i. This ends the proof. ■

Although the interaction effect model (V.1) is still based on symmetric sketchings, we need much more careful construction for the moment-based estimator, since the first coordinate of the sketching vector is always constant 1. We give such an estimator in the following lemma.

Lemma 5 (Unbiasedness of moment estimator in interaction model). For interaction effect model (V.1), construct the empirical moment based tensor ${\mathcal{T}}_{s^{\prime }}$ as following

• For i,j,k ≠ 06, ${\mathcal{T}}_{s^{\prime }[i,j,k]}={\mathcal{T}}_{s[i,j,k]}$. And ${\mathcal{T}}_{s^{\prime }[i,j,0]}={\mathcal{T}}_{s[i,j,0]}$, ${\mathcal{T}}_{s^{\prime }[0,j,k]}={\mathcal{T}}_{s[0,j,k]}$, ${\mathcal{T}}_{s^{\prime }[i,0,k]}={\mathcal{T}}_{s[i,0,k]}$.

• For i ≠ 0, ${\mathcal{T}}_{s^{\prime }[0,0,i]}={\mathcal{T}}_{s^{\prime }[0,i,0]}={\mathcal{T}}_{s^{\prime }[i,0,0]}=\frac{1}{3}{\mathcal{T}}_{s[0,0,i]}-\frac{1}{6}\left({\sum }_{k=1}^p{\mathcal{T}}_{s[k,k,i]}-(p+2)a_i\right)$.

• ${\mathcal{T}}_{s^{\prime }[0,0,0]}=\frac{1}{2p-2}\left({\sum }_{k=1}^p{\mathcal{T}}_{s[0,k,k]}-(p+2){\mathcal{T}}_{s[0,0,0]}\right)$

The ${\mathcal{T}}_{s^{\prime }}$ is an unbiased estimator for $\mathcal{B}$ i.e.,

$$\mathbb{E}\left({\mathcal{T}}_{s^{\text{'}}}\right)=\sum _{k=1}^K{\eta }_k{\beta }_k\circ {\beta }_k\circ {\beta }_k.$$

B. Proofs of Lemmas 1 and 2: Concentration Inequalities

We aim to prove Lemmas 1 and 2 in this subsection. These two lemmas provide key concentration inequalities of the theoretical analysis for the main result. Before going into technical details, we introduce a quasi-norm called ψ_α-norm.

Definition 1 (ψ_α-norm [34]). The ψ_α-norm of any random variable X and α > 0 is defined as

$$\Vert X{\Vert }_{{\psi }_{\alpha }}:=\text{inf}\left\{C\in (0,\infty ):\mathbb{E}\left[\text{exp}{(|X|/C)}^{\alpha }\right]\le 2\right\}.$$

Particularly, a random variable who has a bounded ψ₂-norm or bounded ψ₁-norm is called sub-Gaussian or sub-exponential random variable, respectively. Next lemma provides an upper bound for the p-th moment of sum of random variables with bounded ψ_α-norm.

Lemma 6. Suppose X₁,…,X_n are n independent random variables satisfying ${\Vert X_i\Vert }_{{\psi }_{\alpha }}\le b$ with α > 0, then for all $a=\left(a_1,\dots ,a_n\right)\in {ℝ}^n$ and p ≥ 2,

$${\left(\mathbb{E}{\left|\sum _{i=1}^n{a}_i{X}_i-\mathbb{E}\left(\sum _{i=1}^n{a}_i{X}_i\right)\right|}^p\right)}^{\frac{1}{p}}\le \{\begin{array}{ll}{C}_1(\alpha )b\left(\sqrt{p}\Vert a{\Vert }_2+p^{1/\alpha }\Vert a{\Vert }_{\infty }\right),\hfill & \text{ if }0<\alpha <1;\hfill \\ C_2(\alpha )b\left(\sqrt{p}\Vert a{\Vert }_2+p^{1/\alpha }\Vert a{\Vert }_{{\alpha }^{*}}\right),\hfill & \text{ if }\alpha \ge 1.\hfill \end{array}$$ (A.5)

where 1/α* + 1/α = 1, C₁(α),C₂(α) are some absolute constants only depending on α.

If 0 < α < 1, (A.5) is a combination of Theorem 6.2 in [57] and the fact that the p-th moment of a Weibull variable with parameter α is of order p^1/α. If α ≥ 1, (A.5) follows from a combination of Corollaries 2.9 and 2.10 in [58]. Continuing with standard symmetrization arguments, we reach the conclusion for general random variables. When α = 1 or 2, (A.5) coincides with standard moment bounds for a sum of sub-Gaussian and sub-exponential random variables in [59]. The detailed proof of Lemma 6 is postponed to Section H.

When 0 < α < 1, by Chebyshev’s inequality, one can obtain the following exponential tail bound for the sum of random variables with bounded ψ_α-norm. This lemma generalizes the Hoeffding-type concentration inequality for sub-Gaussian random variables (see, e.g. Proposition 5.10 in [59]), and Bernstein-type concentration inequality for sub-exponential random variables (see, e.g. Proposition 5.16 in [59]).

Lemma 7. Suppose 0 < α < 1, X₁,…,X_n are independent random variables satisfying ${\Vert X_i\Vert }_{{\psi }_{\alpha }}\le b$. Then there exists absolute constant C(α) only depending on α such that for any $a=\left(a_1,\dots ,a_n\right)\in {ℝ}^n$ and 0 < δ < 1/e²,

$$\left|\sum _{i=1}^n{a}_i{X}_i-\mathbb{E}\left(\sum _{i=1}^n{a}_i{X}_i\right)\right|\le C(\alpha )b\Vert a{\Vert }_2{\left(\text{log}{\delta }^{-1}\right)}^{1/2}+C(\alpha )b\Vert a{\Vert }_{\infty }{\left(\text{log}{\delta }^{-1}\right)}^{1/\alpha },$$

with probability at least 1 − δ.

Proof. For any t > 0, by Markov’s inequality,

$$\begin{gathered} \mathbb{P}\left(\left|\sum _{i=1}^n{a}_i{X}_i-\mathbb{E}\left(\sum _{i=1}^n{a}_i{X}_i\right)\right|\ge t\right) \\ =\mathbb{P}\left({\left|\sum _{i=1}^n{a}_i{X}_i-\mathbb{E}\left(\sum _{i=1}^n{a}_i{X}_i\right)\right|}^p\ge t^p\right) \\ \le \frac{\mathbb{E}{\left|{\sum }_{i=1}^n{a}_i{X}_i-\mathbb{E}\left({\sum }_{i=1}^n{a}_i{X}_i\right)\right|}^p}{t^p} \\ \le \frac{C{(\alpha )}^p{b}^p{\left(\sqrt{p}\Vert a{\Vert }_2+p^{1/\alpha }\Vert a{\Vert }_{\infty }\right)}^p}{t^p}, \end{gathered}$$

where the last inequality is from Lemma 6. We set t such that $\text{exp}(-p)=C{(\alpha )}^p{b}^p{\left(\sqrt{p}\Vert a{\Vert }_2+p^{1/\alpha }\Vert a{\Vert }_{\infty }\right)}^p/t^p$. Then for p ≥ 2,

$$\left|\sum _{i=1}^n{a}_i{X}_i-\mathbb{E}\left(\sum _{i=1}^n{a}_i{X}_i\right)\right|\le eC(\alpha )b\left(\sqrt{p}\Vert a{\Vert }_2+p^{1/\alpha }\Vert a{\Vert }_{\infty }\right)$$

holds with probability at least 1 − exp(−p) Letting δ = exp(−p), we have that for any 0 < δ < 1/e²,

$$\left|\sum _{i=1}^n{a}_i{X}_i-\mathbb{E}\left(\sum _{i=1}^n{a}_i{X}_i\right)\right|\le C(\alpha )b\left(\Vert a{\Vert }_2{\left(\text{log}{\delta }^{-1}\right)}^{1/2}+\Vert a{\Vert }_{\infty }{\left(\text{log}{\delta }^{-1}\right)}^{1/\alpha }\right),$$

holds with probability at least 1 − δ. This ends the proof. ■

The next lemma provides an upper bound for the product of random variables in ψ_α-norm.

Lemma 8 (ψ_α for product of random variables). Suppose X₁,…,X_m are m random variables (not necessarily independent) with ψ_α-norm bounded by ${\Vert X_j\Vert }_{{\psi }_{\alpha }}\le K_j$. Then the ψ_α/m-norm of ${\prod }_{j=1}^m{X}_j$ is bounded as

$${\Vert \prod _{j=1}^m{X}_j\Vert }_{{\psi }_{\alpha /m}}\le \prod _{j=1}^m{K}_j.$$

Proof. For any ${\left\{x_j\right\}}_{j=1}^m$ and α > 0, by using the inequality of arithmetic and geometric means we have

$${\left(\left|\prod _{j=1}^m\frac{x_j}{K_j}\right|\right)}^{\alpha /m}={\left(\prod _{j=1}^m{\left|\frac{x_j}{K_j}\right|}^{\alpha }\right)}^{1/m}\le \frac{1}{m}\sum _{j=1}^m{\left|\frac{x_j}{K_j}\right|}^{\alpha }.$$

Since exponential function is a monotone increasing function, it shows that

$$\text{exp}{\left(\left|\prod _{j=1}^m\frac{x_j}{K_j}\right|\right)}^{\alpha /m}\le \text{exp}\left(\frac{1}{m}\sum _{j=1}^m{\left|\frac{x_j}{K_j}\right|}^{\alpha }\right)={\left(\prod _{j=1}^m\text{exp}\left({\left|\frac{x_j}{K_j}\right|}^{\alpha }\right)\right)}^{1/m}\le \frac{1}{m}\sum _{j=1}^m\text{exp}\left({\left|\frac{x_j}{K_j}\right|}^{\alpha }\right).$$ (A.6)

From the definition of ψ_α-norm, for j = 1,2,…,m, each individual X_j has

$$\mathbb{E}\left(\text{exp}{\left(\frac{\left|X_j\right|}{K_j}\right)}^{\alpha }\right)\le 2.$$ (A.7)

Putting (A.6) and (A.7) together, we obtain

$$\mathbb{E}\left[\text{exp}{\left(\left|\frac{\prod _{j=1}^m{X}_j}{\prod _{j=1}^m{K}_j}\right|\right)}^{\alpha /m}\right]=\mathbb{E}\left[\text{exp}{\left(\left|\prod _{j=1}^m\frac{X_j}{K_j}\right|\right)}^{\alpha /m}\right]\le \frac{1}{m}\sum _{j=1}^m\mathbb{E}\left[\text{exp}{\left(\left|\frac{X_j}{K_j}\right|\right)}^{\alpha }\right]\le 2.$$

Therefore, we conclude that the ψ_α/m-norm of ${\prod }_{j=1}^m{X}_j$ is bounded by ${\prod }_{j=1}^m{K}_j$.

Proof of Lemma 1. Note that for any j = 1,2,…,m, the ψ₂-norm of $X_j^{\top }{\beta }_j$ is bounded by ‖β_j‖₂ [59]. According to Lemma 8, the ψ_2/m-norm of ${\prod }_{j=1}^m\left(X_j^{\top }{\beta }_j\right)$ is bounded by ${\prod }_{j=1}^m{\Vert {\beta }_j\Vert }_2$. Directly applying Lemma 7, we reach the conclusion.

Proof of Lemma 2. We first focus on the non-symmetric version and the proof follows three steps:

1. Truncate the first coordinate of x_1i, x_2i, x_3i by a carefully chosen truncation level;

2. Utilize the high-order concentration inequality in Lemma 20 at order three;

3. Show that the bias caused by truncation is negligible.

With slightly abuse of notations, we denote a, x, y etc. as their first coordinate of a, x, y etc. Without loss of generality, we assume p ≔ max{p₁,p₂,p₃}. By unitary invariance, we assume β₁ = β₂ = β₃ = e₁, where e₁ = (1,0,…,0)^⊤.

Then, it is equivalent to prove

$$\begin{gathered} {\Vert M_{\text{nsy}}-\mathbb{E}\left(M_{\text{nsy}}\right)\Vert }_s \\ ={\Vert \frac{1}{n}\sum _{i=1}^n{x}_{1i}{x}_{2i}{x}_{3i}{x}_{1i}\circ x_{2i}\circ x_{3i}-e_1\circ e_1\circ e_1\Vert }_s \\ \le C{(\text{log}n)}^3\left(\sqrt{\frac{s^3{\text{log}}^3(p/s)}{n^2}}+\sqrt{\frac{s\text{log}(p/s)}{n}}\right). \end{gathered}$$

Suppose $x_1~\mathcal{N}\left(0,I_{p_1}\right),x_2~\mathcal{N}\left(0,I_{p_2}\right),x_3~\mathcal{N}\left(0,I_{p_3}\right)$ and ${\left\{x_{1i},x_{2i},x_{3i}\right\}}_{i=1}^n$ are n independent samples of {x₁, x₂, x₃}. And define a bounded event ${\mathcal{G}}_n$ for the first coordinate and its corresponding population version,

$${\mathcal{G}}_n=\left\{\underset{i}{\text{max}}\left\{\left|x_{1i}\right|,\left|x_{2i}\right|,\left|x_{3i}\right|\right\}\le M\right\},$$

$$\mathcal{G}=\left\{\text{max}\left\{\left|x_1\right|,\left|x_2\right|,\left|x_3\right|\right\}\le M\right\},$$

where M is a large constant to be specified later. Let $\Vert M_{\text{nsy}}-\mathbb{E}\left(M_{\text{nsy }}\right){\Vert }_s$ upper bounded by M₁ + M₂ where

$$M_1={\Vert \frac{1}{n}\sum _{i=1}^n{x}_{1i}{x}_{2i}{x}_{3i}{x}_{1i}\circ x_{2i}\circ x_{3i}-\mathbb{E}\left(x_1{x}_2{x}_3{x}_1\circ x_2\circ x_3\mid \mathcal{G}\right)\Vert }_s$$

and

$$M_2={\Vert \mathbb{E}\left(x_1{x}_2{x}_3{x}_1\circ x_2\circ x_3\mid \mathcal{G}\right)-e_1\circ e_1\circ e_1\Vert }_s.$$

We will prove M₂ that is negligible in terms of convergence rate of M₁.

Bounding M₁. For simplicity, we define $x_1^{\prime }=x_1|\mathcal{G}$, $x_2^{\prime }=x_2\left|\mathcal{G},x_3^{\prime }=x_3\right|\mathcal{G}$, and ${\left\{x_{1i}^{\prime },x_{2i}^{\prime },x_{3i}^{\prime }\right\}}_{i=1}^n$ are n independent samples of $\left\{x_1^{\prime },x_2^{\prime },x_3^{\prime }\right\}$. According to the law of total probability, we have

$$\mathbb{P}\left(M_1\ge t\right)\le \mathbb{P}\left({\mathcal{G}}_n^c\right)+\mathbb{P}\left(M_{11}\ge t\right),$$

where

$$M_{11}={\Vert \frac{1}{n}\sum _{i=1}^n{x}_{1i}^{\prime }{x}_{1i}^{\prime }\circ x_{3i}^{\prime }{x}_{2i}^{\prime }\circ x_{i1}^{\prime }{x}_{3i}^{\prime }-\mathbb{E}\left(x_1^{\prime }{x}_1^{\prime }\circ x_2^{\prime }{x}_2^{\prime }\circ x_3^{\prime }{x}_3^{\prime }\right)\Vert }_s.$$

According to Lemma 22, the entry of $x_{1i}^{\prime }{x}_{1i}^{\prime },x_{2i}^{\prime }{x}_{2i}^{\prime },x_{3i}^{\prime }{x}_{3i}^{\prime }$ are sub-Gaussian random variable with ψ₂-norm M². Applying Lemma 20, we obtain

$$\mathbb{P}\left(M_{11}\ge C_1{M}^6{\delta }_{n,s}\right)\le \frac{1}{p},$$

where δ_n,s = ((slog(p/s))³/n²)^1/2 + (slog(p/s)/n)^1/2.

On the other hand,

$$\mathbb{P}\left({\mathcal{G}}_n^c\right)\le 3\sum _{i=1}^n\mathbb{P}\left(\left|x_{1i}\right|\ge M\right)\le 3n{e}^{1-C_2{M}^2}$$

Putting the above bounds together, we obtain

$$\mathbb{P}\left(M_1\ge C_1{M}^6{\delta }_{n,s}\right)\le \frac{1}{p}+3n{e}^{1-C_2{M}^2}.$$

By setting $M=2\sqrt{\text{log}n/C_2}$, the bound of M₁ reduces to

$$\mathbb{P}\left(M_1\ge \frac{64C_1}{C_2^3}{\delta }_{n,s}{(\text{log}n)}^3\right)\le \frac{1}{p}+\frac{3e}{n^3}.$$ (A.8)

Bounding M₂. From the definitions of M₂ and sparse spectral norm,

$$M_2={\Vert \mathbb{E}\left(x_1{x}_2{x}_3{x}_1\circ x_2\circ x_3\mid \mathcal{G}\right)-e_1\circ e_1\circ e_1\Vert }_s=\underset{\mathcal{D}(a,b)}{\text{sup}}\left|\mathbb{E}\left(x_1{x}_2{x}_3\left(x_1^{\top }a\right)\left(x_2^{\top }b\right)\left(x_3^{\top }c\right)\mid \mathcal{G}\right)-a_1{b}_1{c}_1\right|.$$

where

$$\mathcal{D}=\left\{\Vert a{\Vert }_2=\Vert b{\Vert }_2=\Vert c{\Vert }_2=1,\text{max}\left\{\Vert a{\Vert }_0,\Vert b{\Vert }_0,\Vert c{\Vert }_0\right\}\le s\right\}.$$

Since x_1j is independent of x_1k for any j ≠ k, $\mathbb{E}\left(x_1\left(x_1^{\top }\varrho a\right)\mid \mathcal{G}\right)=\mathbb{E}\left(x_1^2{a}_1\mid \mathcal{G}\right)$. Similar results hold for x₂,x₃. Then we have

$$M_2=\underset{\mathcal{D}}{\text{sup}}\left|a_1{b}_1{c}_1\right|\left|\mathbb{E}\left(x_1^2{x}_2^2{x}_3^2\mid \mathcal{G}\right)-1\right|\le \mathbb{E}\left(x_1^2{x}_2^2{x}_3^2\mid \mathcal{G}\right)-1|=|\mathbb{E}\left(x_1^2||x_1\mid \le M\right)\mathbb{E}\left(x_2^2||x_2\mid \le M\right)\mathbb{E}\left(x_3^2||x_3\mid \le M\right)-1\mid .$$

By the basic property of Gaussian random variable, we can show

$$1\ge \mathbb{E}\left(x_i^2||x_i\mid \le M\right)\ge 1-2M{e}^{-M^2/2},i=1,2,3.$$

Plugging them into M₂, we have

$$\begin{gathered} M_2\le |{\left(1-2M{e}^{-M^2/2}\right)}^3-1| \\ \le |12M^2{e}^{-M^2}-6M{e}^{-M^2/2}-8M^3{e}^{-3M^2/2}| \\ \le |26M^3{e}^{-M^2/2}|, \end{gathered}$$

where the last inequality holds for a large M > 0. By the choice of $M=2\sqrt{\text{log}n/C_2}$, we have $M_2\le 208/C_2^{3/2}{(\text{log}n)}^{\frac{3}{2}}/n^2$ for some constant C₂. When n is large, this rate is negligible comparing with (A.8)

Bounding M: We put the upper bounds of M₁ and M₂ together. After some adjustments for absolute constant, it suffices to obtain

$$M_1+M_2\le C{(\text{log}n)}^3\left(\sqrt{\frac{s^3{\text{log}}^3(p/s)}{n^2}}+\sqrt{\frac{s\text{log}(p/s)}{n}}\right),$$

with probability at least 1 − 10/n³ − 1/p. This concludes the proof of non-symmetric part. The proof of symmetric part remains similar and thus is omitted here. ■

C. Proof of Theorem 2: Initialization Effect

Theorem 2 gives an approximation error upper bound for the sparse-tensor-decomposition-based initial estimator. In Step I of Section III-A, the original problem can be reformatted to a version of tensor denoising:

$${\mathcal{T}}_s={\mathcal{J}}^{*}+\mathcal{E},\text{ where }\mathcal{E}={\mathcal{T}}_s-\mathbb{E}\left({\mathcal{T}}_s\right).$$ (A.9)

The key difference between our model (A.9) and recent works [50, 27] is that E arises from empirical moment approximation, rather than the random observation noise considered in [50] and [27]. Next lemma gives an upper bound for the approximation error. The proof of Lemma 9 is deferred to Section I.

Lemma 9 (Approximation error of ${\mathcal{T}}_s$). Recall that $\mathcal{E}={\mathcal{T}}_s-\mathbb{E}\left({\mathcal{T}}_s\right)$, where ${\mathcal{T}}_s$ is defined in (III.6). Suppose Condition 4 is satisfied and s ≤ d ≤ Cs. Then

$$\Vert \mathcal{E}{\Vert }_{s+d}\le 2C_1\sum _{k=1}^K{\eta }_k^{*}\left(\sqrt{\frac{s^3{\text{log}}^3(p/s)}{n^2}}+\sqrt{\frac{s\text{log}(p/s)}{n}}\right){(\text{log}n)}^4,$$ (A.10)

with probability at least 1 − 5/n for some uniform constant C₁.

Next we denote the following quantity for simplicity,

$$\gamma =C_2\text{min}\left\{\frac{R^{-1}}{6}-\frac{\sqrt{K}}{s},\frac{R^{-1}}{4\sqrt{5}}-\frac{2}{\sqrt{s}}{\left(1+\sqrt{\frac{K}{s}}\right)}^2\right\},$$ (A.11)

where R is the singular value ratio, K is the CP-rank, s is the sparsity parameter, Γ is the incoherence parameter and C₂ is uniform constant.

Next lemma provides theoretical guarantees for sparse tensor decomposition method.

Lemma 10. Suppose that the symmetric tensor denoising model (A.9) satisfies Conditions 1, 2 and 3 (i.e., the identifiability, parameter space and incoherence). Assume the number of initializations $L\ge K^{C_3{\gamma }^{-4}}$ and the number of iterations $N\ge C_4\text{log}\left(\gamma /\left(\frac{1}{{\eta }_{\text{min}}^{*}}\Vert \mathcal{E}{\Vert }_{s+d}+\sqrt{K}{\Gamma }^2\right)\right)$ for constants C₃,C₄, the truncation parameter s ≤ d ≤ Cs. Then the sparse-tensor-decomposition-based initialization satisfies

$$\text{max}\left\{{\Vert {\beta }_k^{(0)}-{\beta }_k^{*}\Vert }_2,|{\eta }_k^{(0)}-{\eta }_k^{*}|\right\}\le \frac{C_4}{{\eta }_{\text{min}}^{*}}\Vert \mathcal{E}{\Vert }_{s+d}+\sqrt{K}{\Gamma }^2,$$ (A.12)

for any k ∈ [K]

The proof of Lemma 10 essentially follows Theorem 3.9 in [27], we thus omit the detailed proof here. The upper bound in (A.12) contains two terms: $\frac{C_4}{{\eta }_{\text{min}}^{*}}\Vert \mathcal{E}{\Vert }_{s+d}$ and $\sqrt{K}{\Gamma }^2$, which are due to the empirical moment approximation and the incoherence among different β_k, respectively.

Although the sparse tensor decomposition is not optimal in statistical rate, it does offer a reasonable initial estimation provided enough samples. Equipped with (A.10) and Condition 2, the right side of (A.12) reduces to

$$\frac{C_4}{{\eta }_{\text{min}}^{*}}\Vert \mathcal{E}{\Vert }_{s+d}+\sqrt{K}{\Gamma }^2\le 2C_1{C}_{4}KR(\sqrt{\frac{s^3{\text{log}}^3(p/s)}{n^2}}+\sqrt{\frac{s\text{log}(p/s)}{n}}){(\text{log}n)}^4+\sqrt{K}{\Gamma }^2,$$

with probability at least 1−5/n. Denote C₀ = 4·2160·C₁C₄. Using Conditions 3 and 5, we reach the conclusion that

$$\text{max}\left\{{\Vert {\beta }_k^{(0)}-{\beta }_k^{*}\Vert }_2,|{\eta }_k^{(0)}-{\eta }_k^{*}|\right\}\le K^{-1}{R}^{-2}/2160,$$

with probability at least 1 − 5/n.

D. Proof of Theorem 1: Gradient Update

We first introduce the following lemma to illustrate the improvement of one step thresholded gradient update under suitable conditions. The error bound includes two parts: the optimization error that describes one step effect for gradient update, and the statistical error that reflects the random noise effect. The proof of Lemma 11 is given in Section J. For notation simplicity, we drop the superscript of $\left\{{\beta }_k^{(t)},{\eta }_k\right\}$ in the following proof.

Lemma 11. Let t ≥ 0 be an integer. Suppose Conditions 1–5 hold and $\left\{{\beta }_k^{(t)},{\eta }_k\right\}$ satisfies the following upper bound

$$\sum _{k=1}^K{\Vert \sqrt[3]{{\eta }_k}{\beta }_k^{(t)}-\sqrt[3]{{\eta }_k^{*}}{\beta }_k^{*}\Vert }_2^2\le 4K{\eta }_{\text{max}}^{*\frac{2}{3}}{\epsilon }_0^2,\underset{k\in [K]}{\text{max}}|{\eta }_k-{\eta }_k^{*}|\le {\epsilon }_0,$$ (A.13)

with probability at least $1-\mathcal{O}(K/n)$, where ${\epsilon }_0=K^{-1}{R}^{-\frac{4}{3}}/2160$. As long as the step size μ satisfies

$$0<\mu \le {\mu }_0=\frac{32R^{-20/3}}{3K{[220+270K]}^2},$$ (A.14)

then $\left\{{\beta }_k^{(t+1)}\right\}$ can be upper bounded as

$$\sum _{k=1}^K{\Vert \sqrt[3]{{\eta }_k}{\beta }_k^{(t+1)}-\sqrt[3]{{\eta }_k^{*}}{\beta }_k^{*}\Vert }_2^2\le \underset{\text{optimization error}}{\underbrace{\left(1-32\mu K^{-2}{R}^{-\frac{8}{3}}\right)\sum _{k=1}^n{\Vert \sqrt[3]{{\eta }_k}{\beta }_k^{(t)}-\sqrt[3]{{\eta }_k^{*}}{\beta }_k^{*}\Vert }_2^2}}+\underset{\text{statistical error }}{\underbrace{2C_0{\mu }^2{K}^{-2}{R}^{-\frac{8}{3}}{\eta }_{\text{min}}^{*-\frac{4}{3}}\frac{{\sigma }^{2}s\text{log}p}{n}}},$$

with probability at least $1-\mathcal{O}(Ks/n)$.

In order to apply Lemma 11, we prove that the required condition (A.13) holds at every iteration step t by induction. When t = 0, by (IV.2) and Condition 2,

$${\Vert {\beta }_k^{(0)}-{\beta }_k^{*}\Vert }_2\le {\epsilon }_0,|{\eta }_k-{\eta }_k^{*}|\le {\epsilon }_0,\text{ for }k\in [K],$$

holds with probability at least $1-\mathcal{O}(1/n)$. Since the initial estimator output by first stage is normalized, i.e., ${\Vert {\beta }_k^{(0)}\Vert }_2={\Vert {\beta }_k^{*}\Vert }_2=1$, by triangle inequality we have

$$\begin{gathered} {\Vert \sqrt[3]{{\eta }_k}{\beta }_k^{(0)}-\sqrt[3]{{\eta }_k^{*}}{\beta }_k^{*}\Vert }_2 \\ \le {\Vert \sqrt[3]{{\eta }_k}{\beta }_k^{(0)}-\sqrt[3]{{\eta }_k^{*}}{\beta }_k^{(0)}+\sqrt[3]{{\eta }_k^{*}}{\beta }_k^{(0)}-\sqrt[3]{{\eta }_k^{*}}{\beta }_k^{*}\Vert }_2 \\ \le |\sqrt[3]{{\eta }_k}-\sqrt[3]{{\eta }_k^{*}}|+\sqrt[3]{{\eta }_k^{*}}{\Vert {\beta }_k^{(0)}-{\beta }_k^{*}\Vert }_2. \end{gathered}$$

Note that

$$|\sqrt[3]{{\eta }_k}-\sqrt[3]{{\eta }_k^{*}}|\le \frac{{\epsilon }_0}{{\left(\sqrt[3]{{\eta }_k}\right)}^2+\sqrt[3]{{\eta }_k{\eta }_k^{*}}+{\left(\sqrt[3]{{\eta }_k^{*}}\right)}^2}\le {\epsilon }_0\sqrt[3]{{\eta }_k^{*}}.$$

This implies

$${\Vert \sqrt[3]{{\eta }_k}{\beta }_k^{(0)}-\sqrt[3]{{\eta }_k^{*}}{\beta }_k^{*}\Vert }_2\le 2\sqrt[3]{{\eta }_k^{*}}{\epsilon }_0,$$

with probability at least $1-\mathcal{O}(1/n)$. Taking the summation over k ∈ [K], we have

$$\sum _{k=1}^K{\Vert \sqrt[3]{{\eta }_k}{\beta }_k^{(0)}-\sqrt[3]{{\eta }_k^{*}}{\beta }_k^{*}\Vert }_2^2\le \sum _{k=1}^{K}4{\eta }_k^{*\frac{2}{3}}{\epsilon }_0^2\le 4K{\eta }_{\text{max}}^{*\frac{2}{3}}{\epsilon }_0^2,$$

with probability at least $1-\mathcal{O}(K/n)$, which means (A.13) holds for t = 0.

Suppose (A.13) holds at the iteration step t − 1, which implies

$$\begin{gathered} \sum _{k=1}^K{\Vert \sqrt[3]{{\eta }_k}{\beta }_k^{(t)}-\sqrt[3]{{\eta }_k^{*}}{\beta }_k^{*}\Vert }_2^2 \\ \le \left(1-32\mu K^{-2}{R}^{-\frac{8}{3}}\right)\sum _{k=1}^K{\Vert \sqrt[3]{{\eta }_k}{\beta }_k^{(t-1)}-\sqrt[3]{{\eta }_k^{*}}{\beta }_k^{*}\Vert }_2^2+\mu 2C_0{K}^{-2}{R}^{-\frac{8}{3}}{\eta }_{\text{min}}^{*\frac{4}{3}}\frac{{\sigma }^{2}s\text{log}p}{n} \\ \le 4K{\eta }_{\text{max}}^{*\frac{2}{3}}{\epsilon }_0^2-\mu (128K{R}^{-\frac{8}{3}}{\eta }_{\text{max}}^{*\frac{2}{3}}{\epsilon }_0^2-2C_0{K}^{-2}{R}^{-\frac{8}{3}}{\eta }_{\text{min}}^{*\frac{4}{3}}\frac{{\sigma }^{2}s\text{log}p}{n}). \end{gathered}$$

Since Condition 5 automatically implies

$$\frac{n}{s\text{log}p}\ge \frac{C_0{\sigma }^2{R}^{-\frac{2}{3}}{\eta }_{\text{min}}^{*\frac{2}{3}}K}{64{\epsilon }_0^2},$$

for a sufficiently large C₀, we can obtain

$$\sum _{k=1}^K{\Vert \sqrt[3]{{\eta }_k}{\beta }_k^{(t)}-\sqrt[3]{{\eta }_k^{*}}{\beta }_k^{*}\Vert }_2^2\le 4K{\eta }_{\text{max}}^{*\frac{2}{3}}{\epsilon }_0^2.$$

By induction, (A.13) holds at each iteration step.

Now we are able to use Lemma 11 recursively to complete the proof. Repeatedly using Lemma 11, we have for t = 1, 2, …,

$$\sum _{k=1}^K{\Vert \sqrt[3]{{\eta }_k}{\beta }_k^{(t+1)}-\sqrt[3]{{\eta }_k^{*}}{\beta }_k^{*}\Vert }_2^2\le {\left(1-32\mu K^{-2}{R}^{-\frac{8}{3}}\right)}^t\sum _{k=1}^K{\Vert \sqrt[3]{{\eta }_k}{\beta }_k^{(0)}-\sqrt[3]{{\eta }_k^{*}}{\beta }_k^{*}\Vert }_2^2+\frac{C_0{\eta }_{\text{min}}^{*-\frac{4}{3}}{\sigma }^{2}s\text{log}p}{16},$$

with probability at least $1-\mathcal{O}(tKs/n)$. This concludes the first part of Theorem 1.

When the total number of iterations is no smaller than

$$T^{*}=\frac{\text{log}\left(C_3{\eta }_{\text{min}}^{*-4/3}{\sigma }^{2}s\text{log}p\right)-\text{log}\left(64{\eta }_{\text{max}}^{*2/3}K{\epsilon }_{0}n\right)}{\text{log}\left(1-32\mu K^{-2}{R}^{-8/3}\right)},$$

the statistical error will dominate the whole error bound in the sense that

$$\sum _{k=1}^K\Vert \sqrt[3]{{\eta }_k}{\beta }_k^{\left(T^{*}\right)}-\sqrt[3]{{\eta }_k^{*}}{\beta }_k^{*}{\Vert }_2^2\le \frac{C_3{\eta }_{\text{min}}^{*-\frac{4}{3}}}{8}\frac{{\sigma }^{2}s\text{log}p}{n},$$ (A.15)

with probability at least $1-\mathcal{O}\left(T^{*}Ks/n\right)$.

The next lemma shows that the Frobenius norm distance between two tensors can be bounded by the distances between each factors in their CP decomposition. The proof of this lemma is provided in Section K.

Lemma 12. Suppose $\mathcal{J}$ and ${\mathcal{J}}^{*}$ have CP-decomposition $\mathcal{J}={\sum }_{k=1}^K{\eta }_k{\beta }_k\circ {\beta }_k\circ {\beta }_k$ and ${\mathcal{J}}^{*}={\sum }_{k=1}^K{\eta }_k^{*}{\beta }_k^{*}\circ {\beta }_k^{*}\circ {\beta }_k^{*}$. If $|{\eta }_k-{\eta }_k^{*}|\le c$, then

$${\Vert \mathcal{J}-{\mathcal{J}}^{*}\Vert }_F^2\le 9(1+c)\left(\sum _{k=1}^K{\Vert \sqrt[3]{{\eta }_k}{\beta }_k-\sqrt[3]{{\eta }_k^{*}}{\beta }_k^{*}\Vert }_2^2\right)\left(\sum _{k=1}^K{\left(\sqrt[3]{{\eta }_k^{*}}\right)}^4\right)$$

Denote $\widehat{\mathcal{J}}={\sum }_{k=1}^K{\eta }_k{\beta }_k^{\left(T^{*}\right)}\circ {\beta }_k^{\left(T^{*}\right)}\circ {\beta }_k^{\left(T^{*}\right)}$. Combing (A.15) and Lemma 12, we have

$$\begin{gathered} \Vert \widehat{\mathcal{J}}-{\mathcal{J}}^{*}{\Vert }_F^2\le 9\left(1+{\epsilon }_0\right)\frac{C_3{\eta }_{\text{min}}^{*\frac{4}{3}}}{8}\frac{{\sigma }^{2}s\text{log}p}{n}K{\eta }_{\text{max}}^{*\frac{4}{3}}, \\ =\frac{9C_{3}R}{4}\frac{{\sigma }^{2}Ks\text{log}p}{n}, \end{gathered}$$

with probability at least $1-\mathcal{O}(TKs/n)$. By setting C₁ = 9C₂/4, we complete the proof of Theorem 1.

E. Proofs of Theorems 4 and 6: Minimax Lower Bounds

We first consider the proof for Theorem 6 on non-symmetric tensor estimation. Without loss of generality we assume p = max{p₁,p₂,p₃}. We uniformly randomly generate ${\left\{{\Omega }^{(k,m)}\right\}}_{\begin{array}{c}m=1,\dots ,M\\ k=1,\dots ,K\end{array}}$ as MK subsets of {1,…,p} with cardinality of s. Here M > 0 is a large integer to be specified later. Then we construct ${\left\{{\beta }^{(k,m)}\right\}}_{\begin{array}{c}m=1,\dots ,M\\ k=1,\dots ,K\end{array}}\subseteq {ℝ}^p$ as

$${\beta }_j^{(k,m)}=\{\begin{array}{ll}\sqrt{\lambda },\hfill & \text{ if }j\in {\Omega }^{(k,m)};\hfill \\ 0,\hfill & \text{ if }j\notin {\Omega }^{(k,m)}.\hfill \end{array}$$

λ > 0 will also be specified a little while later. Clearly, ${\Vert {\beta }^{\left(k,m_1\right)}-{\beta }^{\left(k,m_2\right)}\Vert }_2^2\le 2s\lambda$ for any 1 ≤ k ≤ K, 1 ≤ m₁,m₂ ≤ M hyper-geometric distribution: $\mathbb{P}\left(|{\Omega }^{\left(k,m_1\right)}\cap {\Omega }^{\left(k,m_2\right)}|=t\right)=\frac{\left(\begin{array}{l}a\hfill \\ t\hfill \end{array}\right)\left(\begin{array}{l}p-a\hfill \\ a-t\hfill \end{array}\right)}{\left(\begin{array}{l}p\hfill \\ a\hfill \end{array}\right)}$.

Let

$$w^{\left(k,m_1,m_2\right)}=|{\Omega }^{\left(k,m_1\right)}\cap {\Omega }^{\left(k,m_2\right)}|,$$ (A.16)

then for any s/2 ≤ t ≤ s,

$$\mathbb{P}\left(w^{\left(k,m_1,m_2\right)}=t\right)=\frac{\frac{s\cdots (s-t+1)}{t!}\cdot \frac{(p-s)\cdots (p-2s+t+1)}{(s-t)!}}{\frac{p\cdots (p-s+1)}{s!}}\le \left(\begin{array}{l}s\hfill \\ t\hfill \end{array}\right)\cdot {\left(\frac{s}{p-s+1}\right)}^t\le 2^s{\left(\frac{s}{p-s+1}\right)}^t\le {\left(\frac{4s}{p-s+1}\right)}^t.$$

Thus, if η > 0, the moment generating function of $w^{\left(k,m_1,m_2\right)}-\frac{s}{2}$ satisfies

$$\begin{gathered} \mathbb{E}\text{exp}\left(\eta \left(w^{\left(k,m_1,m_2\right)}-\frac{s}{2}\right)\right) \\ \le \text{exp}(0)\cdot \mathbb{P}\left(w^{\left(k,m_1,m_2\right)}\le \frac{s}{2}\right)+\sum _{t=\lfloor s/2\rfloor +1}^s\text{exp}\left(\eta \left(t-\frac{s}{2}\right)\right)\cdot \mathbb{P}\left(w^{\left(k,m_1,m_2\right)}=t\right) \\ \le 1+\sum _{t=\lfloor s/2\rfloor +1}^s(4s/(p-s+1){)}^t\text{exp}(\eta (t-s/2)) \\ =1+{\left(\frac{4s}{p-s+1}\right)}^{\lfloor s/2\rfloor +1}\sum _{t=0}^{s-\lfloor s/2\rfloor -1}{\left(\frac{4s}{p-s+1}\right)}^t\text{exp}(\eta (t+\lfloor s/2\rfloor +1-s/2)) \\ \stackrel{(*)}{\le }1+\left(\frac{4s}{p-s+1}\right)\sum _{t=0}^{s/2s-\lfloor s/2\rfloor -1}{\left(\frac{4s{e}^{\eta }}{p-s+1}\right)}^t \\ =1+{\left(\frac{4s}{p-s+1}\right)}^{s/2}\frac{1-{\left(4s{e}^{\eta }/(p-s+1)\right)}^{s-\lfloor s/2\rfloor }}{1-4s{e}^{\eta }/(p-s+1)} \\ <1+{\left(4s/(p-s+1)\right)}^{s/2}\frac{1}{1-4s/(p-s+1)\cdot e^{\eta }}. \end{gathered}$$

Here, (*) is due to η > 0 and ⌊s/2⌋ + 1 ≥ s/2. By setting η = log((p − s + 1)/(8s)), we have

$$\begin{gathered} \mathbb{P}\left(\sum _{k=1}^K{w}^{\left(k,m_1,m_2\right)}\ge \frac{3sK}{4}\right) \\ =\mathbb{P}\left(\sum _{k=1}^K{w}^{\left(k,m_1,m_2\right)}-\frac{sK}{2}\ge \frac{sK}{4}\right) \\ \le \frac{\mathbb{E}\text{exp}\left(\eta \left(\sum _{k=1}^K{w}^{\left(k,m_1,m_2\right)}-\frac{sK}{2}\right)\right)}{\text{exp}\left(\eta \cdot \frac{sK}{4}\right)} \\ =\frac{\prod _{k=1}^K\mathbb{E}\text{exp}\left(\eta \left(w^{\left(k,m_1,m_2\right)}-\frac{s}{2}\right)\right)}{\text{exp}\left(\eta \cdot \frac{sK}{4}\right)} \\ \le {\left(1+{(4s/(p-s+1))}^{s/2}\cdot 2\right)}^K*\text{exp}\left(-\frac{sK}{4}\text{log}\left(\frac{p-s+1}{8s}\right)\right). \end{gathered}$$ (A.17)

Since p ≥ 20s and s ≥ 4, we have

$$\begin{gathered} {\left(1+2{(4s/(p-s+1))}^{s/2}\right)}^K \\ \le \text{exp}\left(K\text{log}\left(1+2{\left(\frac{4}{p/s-1}\right)}^{s/2}\right)\right) \\ \le \text{exp}\left(K\text{log}\left(1+2{\left(\frac{4}{19}\right)}^2\right)\right) \\ \le \text{exp}(K\cdot 0.085)\le \text{exp}(sK\text{log}(p/s)\cdot 0.0144), \\ \text{exp}\left(-\frac{sK}{4}\text{log}\left(\frac{p-s+1}{8s}\right)\right) \\ =\text{exp}\left(-\frac{sK\text{log}(p/s)}{4}+\frac{sK}{4}\text{log}(8p/(p-s+1))\right) \\ \le \text{exp}\left(-\frac{sK\text{log}(p/s)}{4}+\frac{sK}{4}\text{log}(8\cdot 19/20)\right) \\ \le \text{exp}(-sK\text{log}(p/s)\cdot 0.08). \end{gathered}$$

Combining the two inequalities above, we have.

$$\begin{gathered} {\left(1+{(4s/(p-s+1))}^{s/2}\cdot 2\right)}^K\text{exp}\left(-\frac{sK}{4}\text{log}\left(\frac{p-s+1}{8s}\right)\right) \\ \le \text{exp}\left(-c_{0}sK\text{log}(p/s)\right) \end{gathered}$$

for c₀ = 1/20.

Next we choose M = ⌊exp(c₀/2 · sK log(p/s))⌋. Note that

$$\begin{gathered} {\Vert {\beta }^{\left(k,m_1\right)}-{\beta }^{\left(k,m_2\right)}\Vert }_2^2 \\ =\lambda \cdot \left(|{\Omega }^{\left(k,m_1\right)}\backslash {\Omega }^{\left(k,m_2\right)}|+|{\Omega }^{\left(k,m_2\right)}\backslash {\Omega }^{\left(k,m_1\right)}|\right) \\ =\lambda \left(|{\Omega }^{\left(k,m_1\right)}|+|{\Omega }^{\left(k,m_2\right)}|-2|{\Omega }^{\left(k,m_1\right)}\cap {\Omega }^{\left(k,m_2\right)}|\right) \\ =2\lambda \left(s-|{\Omega }^{\left(k,m_1\right)}\cap {\Omega }^{\left(k,m_2\right)}|\right)\stackrel{(\text{A}.16)}{=}2\lambda \left(s-w^{\left(k,m_1,m_2\right)}\right), \end{gathered}$$

then we further have

$$\begin{gathered} \mathbb{P}\left(\sum _{k=1}^K{\Vert {\beta }^{\left(k,m_1\right)}-{\beta }^{\left(k,m_2\right)}\Vert }_2^2\ge \frac{sK\lambda }{2},\forall 1\le m_1<m_2\le M\right) \\ =\mathbb{P}\left(\sum _{k=1}^{K}2\lambda \left(s-w^{\left(k,m_1,m_2\right)}\right)\ge \frac{sK\lambda }{2},\forall 1\le m_1<m_2\le M\right) \\ =\mathbb{P}\left(\sum _{k=1}^K{w}^{\left(k,m_1,m_2\right)}\le \frac{3K}{4},\forall 1\le m_1<m_2\le M\right) \\ \stackrel{(\text{A}.17)}{\ge }1-\frac{M(M-1)}{2}\text{exp}\left(-c_{0}sK\text{log}(p/s)\right) \\ >1-M^2\text{exp}\left(-c_{0}sK\text{log}(p/s)\right)\ge 0, \end{gathered}$$

which means there are positive probability that ${\left\{{\beta }^{(k,m)}\right\}}_{\begin{array}{c}k=1,\dots ,K\\ m=1,\dots ,M\end{array}}$ satisfy

$$\begin{gathered} \frac{sK\lambda }{2}\le \underset{1\le m_1<m_2\le M}{\text{min}}\sum _{k=1}^K{\Vert {\beta }^{\left(k,m_1\right)}-{\beta }^{\left(k,m_2\right)}\Vert }_2^2 \\ \le \underset{1\le m_1<m_2\le M}{\text{max}}\sum _{k=1}^K{\Vert {\beta }^{\left(k,m_1\right)}-{\beta }^{\left(k,m_2\right)}\Vert }_2^2\le 2sK\lambda . \end{gathered}$$ (A.18)

For the rest of the proof, we fix ${\left\{{\beta }^{(k,m)}\right\}}_{\begin{array}{c}k=1,\dots ,K\\ m=1,\dots ,M\end{array}}$ to be the set of vectors satisfying (A.18).

Next, recall the canonical basis $e_k=(0,\dots ,\stackrel{k-\text{th}}{\overbrace{1}},0,\cdots ,0)\in {ℝ}^p$. Define

$${\mathcal{J}}^{(m)}=\sum _{k=1}^K{\beta }^{(k,m)}\circ e_k\circ e_k,1\le m\le M.$$

For each tensor ${\mathcal{J}}^{(m)}$ and n i.i.d. Gaussian sketches $u_i,v_i,w_i\in {ℝ}^p$, we denote the response

$$y^{(m)}={\left\{y_i^{(m)}\right\}}_{i=1}^n,y_i^{(m)}=u_i\circ v_i\circ w_i,{\mathcal{T}}^{(m)}+{ϵ}_i,$$

where ${ϵ}_i\stackrel{iid}{~}N\left(0,{\sigma }^2\right)$. Clearly, y (y^(m), u, v, w) follows a joint distribution, which may vary based on different values of m.

In this step, we analyze the Kullback-Leibler divergence between different distribution pairs:

$$D_{KL}\left(\left(y^{\left(m_1\right)},u,v,w\right),\left(y^{\left(m_2\right)},u,v,w\right)\right):={\mathbb{E}}_{\left(y^{\left(m_1\right)},u,v,w\right)}\text{log}\left(\frac{p\left(y^{\left(m_1\right)},u,v,w\right)}{p\left(y^{\left(m_2\right)},u,v,w\right)}\right).$$

Note that conditioning on fixed values of u, v, w,

$$y_i^{(m)}~N\left(\sum _{k=1}^K\left({\beta }^{(k,m)\top }{u}_i\right)\cdot \left(e^{(k)\top }{v}_i\right)\cdot \left(e^{(k)\top }{w}_i\right),{\sigma }^2\right).$$

By the KL-divergence formula for Gaussian distribution,

$$\begin{gathered} {\mathbb{E}}_{\left(y^{\left(m_1\right)},u,v,w\right)}\left(\frac{p\left(y^{\left(m_1\right)},u,v,w\right)}{p\left(y^{\left(m_2\right)},u,v,w\right)}\mid u,v,w\right) \\ =\frac{1}{2}\sum _{i=1}^n(\sum _{k=1}^K\left({\left({\beta }^{\left(k,m_1\right)}-{\beta }^{\left(k,m_2\right)}\right)}^{\top }{u}_i\right)* \\ {\left(e^{(k)\top }{v}_i\right)\left(e^{(k)\top }{w}_i\right))}^2{\sigma }^{-2}. \end{gathered}$$

Therefore, for any m₁ ≠ m₂,

$$\begin{gathered} D_{KL}\left(\left(y^{\left(m_1\right)},u,v,w\right),\left(y^{\left(m_2\right)},u,v,w\right)\right) \\ ={\mathbb{E}}_{u,v,w}\frac{1}{2}\sum _{i=1}^n\left(\sum _{k=1}^K{\left({\beta }^{\left(k,m_1\right)}-{\beta }^{\left(k,m_2\right)}\right)}^{\top }{u}_i\right){\left(e^{(k)\top }{v}_i\right)\left(e^{(k)\top }{w}_i\right))}^2{\sigma }^{-2} \\ =\frac{{\sigma }^{-2}}{2}\sum _{i=1}^n\sum _{k=1}^K{\mathbb{E}}_u{\left({\left({\beta }^{\left(k,m_1\right)}-{\beta }^{\left(k,m_2\right)}\right)}^{\top }{u}_i\right)}^2{\mathbb{E}}_v{\left(e^{(k)\top }{v}_i\right)}^2{\mathbb{E}}_w{\left(e^{(k)\top }{w}_i\right)}^2 \\ =\frac{n{\sigma }^{-2}}{2}\sum _{k=1}^K{\Vert {\beta }^{\left(k,m_1\right)}-{\beta }^{\left(k,m_2\right)}\Vert }_2^2\le {\sigma }^{-2}nKs\lambda . \end{gathered}$$

Meanwhile, for any 1 ≤ m₁ < m₂ ≤ M,

$$\begin{gathered} {\Vert {\mathcal{J}}^{\left(m_1\right)}-{\mathcal{J}}^{\left(m_2\right)}\Vert }_F={\Vert \sum _{k=1}^K\left({\beta }^{\left(k,m_1\right)}-{\beta }^{\left(k,m_2\right)}\right)\circ e^{(k)}\circ e^{(k)}\Vert }_F \\ =\sqrt{\sum _{k=1}^K{\Vert {\beta }^{\left(k,m_1\right)}-{\beta }^{\left(k,m_2\right)}\Vert }_2^2}\ge \sqrt{\frac{sK\lambda }{2}}. \end{gathered}$$

By generalized Fano’s Lemma (see, e.g., [60]),

$$\underset{\widehat{\mathcal{J}}}{\text{inf}}\underset{\mathcal{J}\in \mathcal{F}}{\text{sup}}\mathbb{E}\Vert \widehat{\mathcal{J}}-\mathcal{J}{\Vert }_F\ge \sqrt{\frac{sK\lambda }{2}}\left(1-\frac{{\sigma }^{-2}nKs\lambda +\text{log}2}{\text{log}M}\right).$$

Finally we set $\lambda =\frac{c{\sigma }^2}{n}\text{log}(p/s)$ for some small constant c > 0, then

$$\underset{\widehat{\mathcal{J}}}{\text{inf}}\underset{\mathcal{J}\in \mathcal{F}}{\text{sup}}\mathbb{E}\Vert \widehat{\mathcal{T}}-\mathcal{J}{\Vert }_F^2\ge {\left(\underset{\widehat{\mathcal{J}}}{\text{inf}}\underset{\mathcal{J}\in \mathcal{F}}{\text{sup}}\mathbb{E}\Vert \widehat{\mathcal{J}}-\mathcal{J}{\Vert }_F\right)}^2\ge \frac{c{\sigma }^{2}sK\text{log}(p/s)}{n}.$$

which has finished the proof of Theorem 6.

For the proof for Theorem 4, without loss of generality we assume K is a multiple of 3. We first partition {1,…,p} into two subintervals: I₁ = {1,…,p−K/3},I₂ = {p−K/3+1,…,p}, randomly generate ${\left\{{\Omega }^{(k,m)}\right\}}_{\begin{array}{c}m=1,\dots ,M\\ k=1,\dots ,K/3\end{array}}\subseteq {ℝ}^p$ as (MK/3) subsets of {1, …, p – K/3} and construct ${\left\{{\beta }^{(k,m)}\right\}}_{\begin{array}{c}m=1,\dots ,M\\ k=1,\dots ,K\end{array}}\subseteq {ℝ}^{p-K/3}$ as

$${\beta }^{(k,m)}=\{\begin{array}{ll}\sqrt{\lambda },\hfill & \text{ if }j\notin {\Omega }^{(k,m)};\hfill \\ 0,\hfill & \text{ if }j\notin {\Omega }^{(k,m)}.\hfill \end{array}$$

With M = exp(csK log(p/s)) and similar techniques as previous proof, one can show there exists positive possibility that

$$\frac{sK\lambda }{6}\le \underset{1\le m_1<m_2\le M}{\text{min}}\sum _{k=1}^{K/3}{\Vert {\beta }^{\left(k,m_1\right)}-{\beta }^{\left(k,m_2\right)}\Vert }_2^2\le \underset{1\le m_1<m_2\le M}{\text{max}}\sum _{k=1}^{K/3}{\Vert {\beta }^{\left(k,m_1\right)}-{\beta }^{\left(k,m_2\right)}\Vert }_2^2\le \frac{2sK}{3}\lambda .$$

We then construct the following candidate symmetric tensors by blockwise design,

$${\mathcal{T}}^{(m)}=\{\begin{array}{l}{\mathcal{T}}_{\left[I_1,I_2,I_2\right]}^{(m)}={\sum }_{k=1}^{K/3}{\beta }^{(k,m)}\circ e^{(k)}\circ e^{(k)},\hfill \\ {\mathcal{T}}_{\left[I_2,I_1,I_2\right]}^{(m)}={\sum }_{k=1}^{K/3}{e}^{(k)}\circ {\beta }^{(k,m)}\circ e^{(k)},\hfill \\ {\mathcal{T}}_{\left[I_2,I_2,I_1\right]}^{(m)}={\sum }_{k=1}^{K/3}{e}^{(k)}\circ e^{(k)}\circ {\beta }^{(k,m)},\hfill \\ {\mathcal{T}}_{\left[I_1,I_1,I_1\right]}^{(m)},{\mathcal{T}}_{\left[I_1,I_1,I_2\right]}^{(m)},\hfill \\ {\mathcal{T}}_{\left[I_1,I_2,I_1\right]}^{(m)},{\mathcal{T}}_{\left[I_2,I_1,I_1\right]}^{(m)},{\mathcal{T}}_{\left[I_2,I_2,I_2\right]}^{(m)}\text{ are all zeros. }\hfill \end{array}$$

Then we can see for any $u\in {ℝ}^p$,

$$\langle {\mathcal{T}}^{(m)},u\circ u\circ u\rangle =3\sum _{k=1}^{K/3}\left({\beta }^{(k,m)\top }{u}_{I_1}\right)\cdot {\left(e^{(k)\top }{u}_{I_2}\right)}^2.$$

The rest of the proof essentially follows from the proof of Theorem 6. ■

F. Proof of Theorem 7: High-order Stein’s Lemma

The proof of this theorem follows from the one of Theorem 6 in [49]. For the sake of completeness, we restate the detail here. Applying the recursion representation of score function (A.2), we have

$$\begin{gathered} \mathbb{E}\left[G(x){\mathcal{S}}_3(x)\right] \\ =\mathbb{E}\left[G(x)\left(-{\mathcal{S}}_2(x)\circ {\nabla }_x\text{log}p(x)-{\nabla }_x{\mathcal{S}}_2(x)\right)\right] \\ =-\mathbb{E}\left[G(x){\mathcal{S}}_2(x)\circ {\nabla }_x\text{log}p(x)\right]-\mathbb{E}\left[G(x){\nabla }_x{\mathcal{S}}_2(x)\right)]. \end{gathered}$$

Then, we apply the first-order Stein’s lemma (see Lemma 26) on function $G(x){\mathcal{S}}_2(x)$ and obtain

$$\begin{gathered} \mathbb{E}\left[G(x){\mathcal{S}}_3(x)\right] \\ =\mathbb{E}\left[{\nabla }_x\left(G(x){\mathcal{S}}_2(x)\right)\right]-\mathbb{E}\left[G(x){\nabla }_x{\mathcal{S}}_2(x)\right)] \\ =\mathbb{E}\left[{\nabla }_{x}G(x){\mathcal{S}}_2(x)+{\nabla }_x{\mathcal{S}}_2(x)G(x)\right]-\mathbb{E}\left[G(x){\nabla }_x{\mathcal{S}}_2(x)\right)] \\ =\mathbb{E}\left[{\nabla }_{x}G(x){\mathcal{S}}_2(x)\right]. \end{gathered}$$

Repeating the above argument two more times, we reach the conclusion. ■

G. Proofs of Lemmas 3, 4, and 5: Moment Calculation

In this subsection, we present the detail proofs of moment calculation, including non-symmetric case, symmetric case, and interaction model.

1). Proof of Lemma 3:

By the definition of {y_i} in (VI.1) & (VI.2), we have

$$\begin{gathered} \mathbb{E}\left(\frac{1}{n}\sum _{i=1}^n{y}_i{u}_i\circ v_i\circ w_i\right)=\mathbb{E}\left(\frac{1}{n}\sum _{i=1}^n{ϵ}_i{u}_i\circ v_i\circ w_i\right) \\ +\mathbb{E}\left(\frac{1}{n}\sum _{i=1}^n\sum _{l_{k-1}}^K{\eta }_k^{*}\left({\beta }_{1k}^{*\top }{u}_i\right)\left({\beta }_{2k}^{*\top }{v}_i\right)\left({\beta }_{3k}^{*\top }{w}_i\right)u_i\circ v_i\circ w_i\right). \end{gathered}$$ (A.19)

First, we observe $\mathbb{E}\left({ϵ}_i{u}_i\circ v_i\circ w_i\right)=0$ due to the independence between ϵ_i and {u_i,v_i,w_i}. Then, we consider a single component from a single observation

$$M=\mathbb{E}\left(\left({\beta }_{1k}^{*\top }{u}_i\right)\left({\beta }_{2k}^{*\top }{v}_i\right)\left({\beta }_{3k}^{*\top }{w}_i\right)u_i\circ v_i\circ w_i\right),i\in [n],k\in [K].$$

For notation simplicity, we drop the subscript i for i-th observation and k for k-th component such that

$$M=\mathbb{E}\left(\left({\beta }_1^{*\top }u\right)\left({\beta }_2^{*\top }v\right)\left({\beta }_3^{*\top }w\right)u\circ v\circ w\right)\in {ℝ}^{p_1\times p_2\times p_3}.$$ (A.20)

Each entry of M can be calculated as follows

$$\begin{gathered} M_{ijk}=\mathbb{E}\left(\left({\beta }_1^{*\top }u\right)\left({\beta }_2^{*\top }v\right)\left({\beta }_3^{*\top }w\right)u_i{v}_j{w}_k\right) \\ =\mathbb{E}\left(\left({\beta }_{1i}^{*}{u}_i+\sum _{m\ne i}{\beta }_{1m}^{*}{u}_m\right)u_i\right) \\ \times \mathbb{E}\left(\left({\beta }_{2j}^{*}{u}_i+\sum _{m\ne j}{\beta }_{2m}^{*}{v}_m\right)v_j\right) \\ \times \mathbb{E}\left(\left({\beta }_{3k}^{*}{w}_k+\sum _{m\ne k}{\beta }_{3m}^{*}{w}_m\right)w_k\right) \\ ={\beta }_{1i}^{*}{\beta }_{2j}^{*}{\beta }_{3k}^{*}, \end{gathered}$$

which implies M = β₁ ∘ β₂ ∘ β₃. Combining with n observations and K components, we can obtain

$$\mathbb{E}(\mathcal{T})=\frac{1}{n}\sum _{i=1}^n\sum _{k=1}^K{\eta }_k^{*}{\beta }_{1k}\circ {\beta }_{2k}\circ {\beta }_{3k}.$$

This finished our proof. ■

2). Proof of Lemma 4:

In this subsection, we provide an alternative and more direct proof for Lemma 4. We consider a similar single component of (A.20) but with a symmetric structure, namely, $M_s=\mathbb{E}\left({\left({\beta }^{*}{}^{\top }x\right)}^{3}x\circ x\circ x\right)$. Based on the symmetry of both underlying tensor and sketchings, we will verify the following three cases:

• When i = j = k, then

  $$\begin{gathered} M_{s_{iii}}=\mathbb{E}{\left({\beta }_i^{*}{x}_i+\sum _{m\ne i}{\beta }_m^{*}{x}_m\right)}^3{x}_i^3 \\ =\mathbb{E}({\beta }_i^{*3}{x}_i^3+3{\beta }_i^{*2}{x}_i^2\left(\sum _{m\ne i}{\beta }_m^{*}{x}_m\right)+3{\beta }_i^{*}{x}_i{\left(\sum _{m\ne i}{\beta }_m^{*}{x}_m\right)}^2+{\left(\sum _{m\ne i}{\beta }_m^{*}{x}_m\right)}^3)x_i^3 \\ =15{\beta }_i^{*3}+9{\beta }_i^{*}\sum _{m\ne i}{\beta }_m^{*2}=9{\beta }_i^{*}+6{\beta }_i^{*3}. \end{gathered}$$

  The last equation is due to ‖β*‖₂ = 1.
• When i ≠ j ≠ k, then

  $$\begin{gathered} M_{s_{ijk}}=\mathbb{E}{\left({\beta }_i^{*}{x}_i+{\beta }_j^{*}{x}_j+{\beta }_k^{*}{x}_k\right)}^3{x}_i{x}_j{x}_k \\ =6{\beta }_i^{*}{\beta }_j^{*}{\beta }_k^{*}. \end{gathered}$$
• When i = j ≠ k, then

  $$\begin{gathered} M_{s_{iik}}=\mathbb{E}{\left({\beta }_i^{*}{x}_i+{\beta }_k^{*}{x}_k+\sum _{m\ne i,k}{\beta }_m^{*}{x}_m\right)}^3{x}_i^2{x}_k \\ =9{\beta }_i^{*2}{\beta }_k^{*}+3{\beta }_k^{*3}+3{\beta }_k^{*}\left(\sum _{m\ne i,k}{\beta }_m^{*2}\right) \\ =9{\beta }_i^{*2}{\beta }_k^{*}+3{\beta }_k^{*}\left(\sum _{m\ne i}{\beta }_m^{*2}\right) \\ =3{\beta }_k^{*}+6{\beta }_i^{*2}{\beta }_k^{*}. \end{gathered}$$

Therefore, it is sufficient to calculate M_s by

$$M_s=3\sum _{k=1}^K{\eta }_k^{*}\left(\sum _{m=1}^p{\beta }_k^{*}\circ e_m\circ e_m+e_m\circ {\beta }_k^{*}\circ e_m+e_m\circ e_m\circ {\beta }_k^{*}\right)+6\sum _{k=1}^K{\eta }_k^{*}{\beta }_k^{*}°{\beta }_k^{*}°{\beta }_k^{*}.$$

The first term is the bias term due to correlations among symmetric sketchings. Denote $M_1=\frac{1}{n}{\sum }_{i=1}^n{y}_i{x}_i$ and note that $\mathbb{E}\left(\frac{1}{n}{\sum }_{i=1}^n{y}_i{x}_i\right)=3{\sum }_{k=1}^K{\eta }_k^{*}{\beta }_k^{*}$ Therefore, the empirical first-order moment M₁ could be used to remove the bias term as follows

$$\begin{gathered} \mathbb{E}\left(M_s-\sum _{m=1}^p\left(M_1\circ e_m\circ e_m+e_m\circ M_1\circ e_m+e_m\circ e_m\circ M_1\right)\right) \\ =6\sum _{k=1}^K{\eta }_k^{*}{\beta }_k^{*}\circ {\beta }_k^{*}\circ {\beta }_k^{*}. \end{gathered}$$

This finishes our proof. ■

3). Proof of Lemma 5:

As before, consider a single component first. For notation simplicity, we drop the subscript l for l-th observation and k for k-th component. Since each component is normalized, the entry-wise expectation of (β^⊤x)³ x ∘ x ∘ x can be calculated as

$$\begin{array}{l}{\left[\mathbb{E}{\left({\beta }^{\top }x\right)}^{3}x\circ x\circ x\right]}_{0,0,0}=3{\beta }_0-2{\beta }_0^3\hfill \\ {\left[\mathbb{E}{\left({\beta }^{\top }x\right)}^{3}x\circ x\circ x\right]}_{0,0,i}=3{\beta }_i\hfill \\ {\left[\mathbb{E}{\left({\beta }^{\top }x\right)}^{3}x\circ x\circ x\right]}_{0,i,i}=6{\beta }_0{\beta }_i^2+3{\beta }_0\hfill \\ {\left[\mathbb{E}{\left({\beta }^{\top }x\right)}^{3}x\circ x\circ x\right]}_{0,i,j}=6{\beta }_0{\beta }_i{\beta }_j\hfill \\ {\left[\mathbb{E}{\left({\beta }^{\top }x\right)}^{3}x\circ x\circ x\right]}_{i,i,i}=6{\beta }_i^3+9{\beta }_i\hfill \\ {\left[\mathbb{E}{\left({\beta }^{\top }x\right)}^{3}x\circ x\circ x\right]}_{i,i,j}=6{\beta }_i^2{\beta }_j+3{\beta }_j\hfill \\ {\left[\mathbb{E}{\left({\beta }^{\top }x\right)}^{3}x\circ x\circ x\right]}_{i,j,k}=6{\beta }_i{\beta }_j{\beta }_k.\hfill \end{array}$$

Due to the symmetric structure and non-randomness of first coordinate, there are bias appearing for each entry. For i,j,k ≠ 0, we could use ${\sum }_{m=1}^p\left(a\circ e_m\circ e_m+e_m\circ a\circ e_m+e_m\circ e_m\circ a\right)$ to remove the bias as shown in the previous proof of Lemma 4. For the subscript involving 0, the following two calculations work for removing the bias,

$$\begin{array}{l}\mathbb{E}\left(\frac{1}{3}{\mathcal{T}}_s-\frac{1}{6}\left(\sum _{k=1}^p{\mathcal{T}}_{s,[k,k,i]}-(p+1)a_i\right)\right)={\beta }_0^2{\beta }_i.\hfill \\ \mathbb{E}\left(\frac{1}{2p-2}\left(\sum _{k=1}^p{\mathcal{T}}_{s[0,k,k]}-(p+2){\mathcal{T}}_{s[0,0,0]}\right)\right)={\beta }_0^3.\hfill \end{array}$$

This ends the proof. ■

H. Proof of Lemma 6

Recall the ‖X‖_ψα is defined in Definition 1. Without loss of generality, we assume ‖X‖_ψα = 1 and $\mathbb{E}{X}_i=0$ throughout this proof. Let β = (log2)i/α and Z_i = (|X_i| = − β)₊, where (x)₊ = x if x ≥ 0 and (x)₊ = 0 if else. For notation simplicity, we define $\Vert X{\Vert }_p={\left(\mathbb{E}|X{|}^p\right)}^{1/p}$ for a random variable X. The following step is to estimate the moment of linear combinations of variables ${\left\{X_i\right\}}_{i=1}^n$.

According to the symmetrization inequality (e.g., Proposition 6.3 of [61]), we have

$${\Vert \sum _{i=1}^n{a}_i{X}_i\Vert }_p\le 2{\Vert \sum _{i=1}^n{a}_i{\epsilon }_i{X}_i\Vert }_p=2{\Vert \sum _{i=1}^n{a}_i{\epsilon }_i|X_i|\Vert }_p,$$ (A.21)

where ${\left\{{\epsilon }_i\right\}}_{i=1}^n$ are independent Rademacher random variables and we notice that ε_iX_i and ε_i|X_i| are identically distributed. Moreover, if |X_i| ≥ β, the definition of Z_i implies that |X_i| = Z_i + β. And if |X_i| < β, we have Z_i = 0. Thus, we have |X_i| ≤ Z_i + β at any time and it leads to

$$2{\Vert \sum _{i=1}^n{a}_i{\epsilon }_i|X_i|\Vert }_p\le 2{\Vert \sum _{i=1}^n{a}_i{\epsilon }_i\left(\beta +Z_i\right)\Vert }_p.$$ (A.22)

By triangle inequality,

$$2{\Vert \sum _{i=1}^n{a}_i{\epsilon }_i\left(\beta +Z_i\right)\Vert }_p\le 2{\Vert \sum _{i=1}^n{a}_i{\epsilon }_i{Z}_i\Vert }_p+2{\Vert \sum _{i=1}^n{a}_i{\epsilon }_i\beta \Vert }_p.$$ (A.23)

Next, we will bound the second term of the RHS of (A.23). In particular, we will utilize Khinchin-Kahane inequality, whose formal statement is included in Lemma 27 for the sake of completeness. From Lemma 27 we have

$${\Vert \sum _{i=1}^n{a}_i{\epsilon }_i\beta \Vert }_p\le {\left(\frac{p-1}{2-1}\right)}^{1/2}{\Vert \sum _{i=1}^n{a}_i{\epsilon }_i\beta \Vert }_2\le \beta \sqrt{p}{\Vert \sum _{i=1}^n{a}_i{\epsilon }_i\Vert }_2.$$ (A.24)

Since ${\left\{{\epsilon }_i\right\}}_{i=1}^n$ are independent Rademacher random variables, some simple calculations implies

$${\left(\mathbb{E}{\left(\sum _{i=1}^n{\epsilon }_i{a}_i\right)}^2\right)}^{1/2}$$ (A.25)

$$\begin{array}{l}={\left(\mathbb{E}\left(\sum _{i=1}^n{\epsilon }_i^2{a}_i^2+2\sum _{1\le i<j\le n}{\epsilon }_i{\epsilon }_j{a}_i{a}_j\right)\right)}^{1/2}\hfill \\ ={\left(\sum _{i=1}^n{a}_i^2\mathbb{E}{\epsilon }_i^2+2\sum _{1\le i<j\le n}{a}_i{a}_j\mathbb{E}{\epsilon }_i\mathbb{E}{\epsilon }_j\right)}^{1/2}\hfill \\ ={\left(\sum _{i=1}^n{a}_i^2\right)}^{1/2}=\Vert \mathit{a}{\Vert }_2.\hfill \end{array}$$ (A.26)

Combining inequalities (A.22)–(A.25),

$$2{\Vert \sum _{i=1}^n{a}_i{\epsilon }_i|X_i|\Vert }_p\le 2{\Vert \sum _{i=1}^n{a}_i{\epsilon }_i{Z}_i\Vert }_p+2\beta \sqrt{p}\Vert a{\Vert }_2.$$ (A.27)

Let ${\left\{Y_i\right\}}_{i=1}^n$ are independent symmetric random variables satisfying $\mathbb{P}\left(|Y_i|\ge t\right)=\text{exp}\left(-t^{\alpha }\right)$ for all t ≥ 0. Then we have

$$\begin{array}{l}\mathbb{P}\left(Z_i\ge t\right)\hfill \\ \le \mathbb{P}\left(|X_i|\ge t+\beta \right)=\mathbb{P}\left(\text{exp}\left({|X_i|}^{\alpha }\right)\ge \text{exp}\left({(t+\beta )}^{\alpha }\right)\right)\hfill \\ \le \mathbb{E}\left(\text{exp}{\left(|X_i|\right)}^{\alpha }\right)\cdot \text{exp}\left(-{(t+\beta )}^{\alpha }\right)\le 2\text{exp}\left(-{(t+\beta )}^{\alpha }\right)\hfill \\ \le 2\text{exp}\left(-t^{\alpha }-{\beta }^{\alpha }\right)=\mathbb{P}\left(|Y_i|\ge t\right),\hfill \end{array}$$

which implies

$${\Vert \sum _{i=1}^n{a}_i{\epsilon }_i{Z}_i\Vert }_p\le {\Vert \sum _{i=1}^n{a}_i{\epsilon }_i{Y}_i\Vert }_p={\Vert \sum _{i=1}^n{a}_i{Y}_i\Vert }_p,$$ (A.28)

since ε_iY_i and Y_i have the same distribution due to symmetry. Combining (A.27) and (A.28) together, we reach

$${\Vert \sum _{i=1}^n{a}_i{X}_i\Vert }_p\le 2\beta \sqrt{p}\Vert a{\Vert }_2+2{\Vert \sum _{i=1}^n{a}_i{Y}_i\Vert }_p.$$ (A.29)

For 0 < α < 1, it follows Lemma 25 that

$${\Vert \sum _{i=1}^n{a}_i{Y}_i\Vert }_p\le C_1(\alpha )\left(\sqrt{p}\Vert a{\Vert }_2+p^{1/\alpha }\Vert a{\Vert }_{\infty }\right),$$ (A.30)

where C₁(α) is some absolute constant only depending on α.

For α ≥ 1, we will combine Lemma 24 and the method of the integration by parts to pass from tail bound result to moment bound result. Recall that for every non-negative random variable X, integration by parts yields the identity

$$\mathbb{E}X={\int }_0^{\infty }\mathbb{P}(X\ge t)dt.$$

Applying this to $X={|{\sum }_{i=1}^n{a}_i{Y}_i|}^p$ and changing the variable t = t^p, then we have

$$\begin{array}{l}\text{E}{|\sum _{i=1}^n{a}_i{Y}_i|}^p\hfill \\ ={\int }_0^{\infty }\mathbb{P}\left(|\sum _{i=1}^n{a}_i{Y}_i|\ge t\right)p{t}^{p-1}dt\hfill \\ \le {\int }_0^{\infty }2\text{exp }\left(-c\text{min}\left(\frac{t^2}{\Vert a{\Vert }_2^2},\frac{t^{\alpha }}{\Vert a{\Vert }_{\alpha *}^{\alpha }}\right)\right)p{t}^{p-1}dt,\hfill \end{array}$$ (A.31)

where the inequality is from Lemma 24 for all p ≥ 2 and 1/α + 1/α* = 1. In this following, we bound the integral in three steps:

1. If $\frac{t^2}{\Vert a{\Vert }_2^2}\le \frac{t^{\alpha }}{\Vert a{\Vert }_{\alpha *}^{\alpha }}$, (A.31) reduces to

   $$\mathbb{E}{\left|\sum _{i=1}^n{a}_i{Y}_i\right|}^p\le 2p{\int }_0^{\infty }\text{exp}\left(-c\frac{t^2}{\Vert a{\Vert }_2^2}\right))t^{p-1}dt.$$

   Letting $t^{\prime }=c{t}^2/\Vert a{\Vert }_2^2$, we have

   $$\begin{array}{l}2p{\int }_0^{\infty }\text{exp}\left(-c\frac{t^2}{\Vert a{\Vert }_2^2}\right))t^{p-1}dt\hfill \\ =\frac{p\Vert a{\Vert }_2^p}{c^{p/2}}{\int }_0^{\infty }{e}^{-t^{\prime }}{t}^{\prime p/2-1}d{t}^{\prime }\hfill \\ =\frac{p\Vert a{\Vert }_2^p}{c^{p/2}}\Gamma \left(\frac{p}{2}\right)\le \frac{p\Vert a{\Vert }_2^p}{c^{p/2}}{\left(\frac{p}{2}\right)}^{p/2},\hfill \end{array}$$

   where the second equation is from the density of Gamma random variable. Thus,

   $${\left(\mathbb{E}{\left|\sum _{i=1}^n{a}_i{Y}_i\right|}^p\right)}^{\frac{1}{p}}\le \frac{p^{1/p}}{{(2c)}^{1/2}}\sqrt{p}\Vert a{\Vert }_2\le \frac{\sqrt{2}}{\sqrt{c}}\sqrt{p}\Vert a{\Vert }_2.$$ (A.32)
2. If $\frac{t^2}{\Vert a{\Vert }_2^2}>\frac{t^{\alpha }}{\Vert a{\Vert }_{\alpha *}^{\alpha }}$, (A.31) reduces to

   $$\mathbb{E}{\left|\sum _{i=1}^n{a}_i{Y}_i\right|}^p\le 2p{\int }_0^{\infty }\text{exp}\left(-c\frac{t^{\alpha }}{\Vert a{\Vert }_{\alpha *}^{\alpha }}\right))t^{p-1}dt.$$

   Letting $t^{\prime }=c{t}^{\alpha }/\Vert a{\Vert }_{\alpha *}^{\alpha }$, we have

   $$\begin{array}{l}2p{\int }_0^{\infty }\text{exp}\left(-c\frac{t^{\alpha }}{\Vert a{\Vert }_{\alpha *}^{\alpha }}\right))t^{p-1}dt\hfill \\ =\frac{2p\Vert a{\Vert }_{\alpha *}^p}{\alpha c^{p/\alpha }}{\int }_0^{\infty }{e}^{-t^{\prime }}{t}^{\prime p/\alpha -1}d{t}^{\prime }\hfill \\ =\frac{2}{\alpha }\frac{p\Vert a{\Vert }_{\alpha *}^p}{c^{p/\alpha }}\Gamma \left(\frac{p}{\alpha }\right)\le \frac{2p\Vert a{\Vert }_{\alpha *}^p}{{\alpha }^{p/\alpha }}{\left(\frac{p}{\alpha }\right)}^{p/\alpha }.\hfill \end{array}$$

   Thus,

   $${\left(\mathbb{E}{\left|\sum _{i=1}^n{a}_i{Y}_i\right|}^p\right)}^{\frac{1}{p}}\le \frac{2p^{1/p}}{{(c\alpha )}^{1/\alpha }}{p}^{1/\alpha }\Vert a{\Vert }_{\alpha *}\le \frac{4}{{(c\alpha )}^{1/\alpha }}{p}^{1/\alpha }\Vert a{\Vert }_{\alpha *}.$$ (A.33)
3. Overall, we have the following by combining (A.32) and (A.33),

   $${\left(\mathbb{E}{\left|{\sum }_{i=1}^n{a}_i{Y}_i\right|}^p\right)}^{\frac{1}{p}}\le \text{max}\left(\sqrt{\frac{2}{c}},\frac{4}{{(c\alpha )}^{1/\alpha }}\right)\left(\sqrt{p}\Vert a{\Vert }_2+p^{1/\alpha }\Vert a{\Vert }_{\alpha *}\right).$$

   After denoting $C_2(\alpha )=\text{max}\left(\sqrt{\frac{2}{c}},\frac{4}{{(c\alpha )}^{1/\alpha }}\right)$, we reach

   $${\Vert \sum _{i=1}^n{a}_i{Y}_i\Vert }_p\le C_2(\alpha )\left(\sqrt{p}\Vert a{\Vert }_2+p^{1/\alpha }\Vert a{\Vert }_{\alpha *}\right).$$ (A.34)

Since 0 < β < 1, the conclusion can be reached by combining (A.29),(A.30) and (A.34). ■

I. Proof of Lemma 9

Firstly, let us consider the non-symmetric perturbation error analysis. According to Lemma 3, the exact form of $\mathcal{E}=\mathcal{T}-\mathbb{E}(\mathcal{T})$ is given by

$$\mathcal{E}=\frac{1}{n}\sum _{i=1}^n{y}_i{u}_i\circ v_i\circ w_i-\sum _{k=1}^K{\eta }_k^{*}{\beta }_{1k}^{*}\circ {\beta }_{2k}^{*}\circ {\beta }_{3k}^{*}.$$

We decompose it by a concentration term $\left({\mathcal{E}}_1\right)$ and a noise term $\left({\mathcal{E}}_2\right)$ as follows,

$$\mathcal{E}={\mathcal{E}}_1+{\mathcal{E}}_2,$$ (A.35)

where

$${\mathcal{E}}_1=\frac{1}{n}\sum _{i=1}^n\langle u_i\circ v_i\circ w_i,\sum _{k=1}^K{\eta }_k^{*}{\beta }_{1k}^{*}\circ {\beta }_{2k}^{*}\circ {\beta }_{3k}^{*}\rangle u_i\circ v_i\circ w_i-\sum _{k=1}^K{\eta }_k^{*}{\beta }_{1k}^{*}\circ {\beta }_{2k}^{*}\circ {\beta }_{3k}^{*}.$$

$${\mathcal{E}}_2=\frac{1}{n}\sum _{i=1}^n{ϵ}_i{u}_i\circ v_i\circ w_i.$$

Bounding ${\mathcal{E}}_1$: For k-th componet of ${\mathcal{E}}_1$, we denote

$${\mathcal{E}}_{1k}=\frac{1}{n}\sum _{i=1}^n\langle u_i\circ v_i\circ w_i,{\beta }_{1k}^{*}\circ {\beta }_{2k}^{*}\circ {\beta }_{3k}^{*}\rangle u_i\circ v_i\circ w_i-{\beta }_{1k}^{*}\circ {\beta }_{2k}^{*}\circ {\beta }_{3k}^{*}.$$

Define

$${\delta }_{n,p,s}={(\text{log }n)}^3(\sqrt{\frac{s^3{\text{ log}}^3(p/s)}{n^2}+\sqrt{\frac{s\text{ log}(p/s)}{n}})}.$$

By using Lemma 2 and s ≤ d ≤ Cs, it suffices to have for some absolute constant C₁₁,

$${\Vert {\mathcal{E}}_{1k}\Vert }_{s+d}\le C_{11}{\delta }_{n,p,s},$$

with probability at least 1 − 10/n³, where ‖ · ‖_s+d is the sparse tensor spectral norm defined in (II.3). Equipped with the triangle inequality, the sparse tensor spectral norm for ${\mathcal{E}}_1$ can be bounded by

$${\Vert {\mathcal{E}}_1\Vert }_{s+d}\le C_{11}{\delta }_{n,p,s}\sum _{k=1}^K{\eta }_k^{*},$$ (A.36)

with probability at least 1 − 10K/n³.

Bounding ${\mathcal{E}}_2$: Note that the random noise ${\left\{{ϵ}_i\right\}}_{i=1}^n$ is independent of sketching vector {u_i,v_i,w_i}. For fixed ${\left\{{ϵ}_i\right\}}_{i=1}^n$, applying Lemma 20, we have for some absolute constant C₁₂

$${\Vert \frac{1}{n}\sum _{i=1}^n{ϵ}_i{u}_i\circ v_i\circ w_i\Vert }_{s+d}\le C_{12}\Vert \mathit{ϵ}{\Vert }_{\infty }{C}_{11}{\delta }_{n,p,s},$$

with probability at least 1−1/p. According to Lemma 23, we have

$$\mathbb{P}\left({\Vert {\mathcal{E}}_2\Vert }_{s+d}\ge C_{12}\sigma \text{ log }n{\delta }_{n,p,s}\right)\le \frac{1}{p}+\frac{3}{n}\le \frac{4}{n}.$$ (A.37)

Bounding $\mathcal{E}$: Putting (A.36) and (A.37) together, we obtain

$$\Vert \mathcal{E}{\Vert }_{s+d}\le \left(C_{11}\sum _{k=1}^K{\eta }_k^{*}+C_{12}\sigma \text{ log }n\right){\delta }_{n,p,s},$$

with probability at least 1 − 5/n. Under Condition 9, we have

$$\Vert \mathcal{E}{\Vert }_{s+d}\le 2C_1\sum _{k=1}^K{\eta }_k^{*}{\delta }_{n,p,s}\text{ log }n,$$

with probability at least 1 − 5/n.

The perturbation error analysis for the symmetric tensor estimation model and the interaction effect model is similar since the empirical first-order moment converges much faster than the empirical third-order moment. So we omit the detailed proof here. ■

J. Proof of Lemma 11

Lemma 11 quantifies one step update for thresholded gradient update. The proof consists of two parts.

First, we evaluate an oracle estimator ${\left\{{\tilde{\beta }}_k^{(t+1)}\right\}}_{k=1}^K$ with known support information, which is defined as

$${\tilde{\beta }}_k^{(t+1)}={\phi }_{\frac{\mu }{\varphi }h\left({\beta }_k^{(t)}\right)}\left({\beta }_k^{(t)}-\frac{\mu }{\varphi }{\nabla }_k\mathcal{L}{\left({\beta }_k^{(t)}\right)}_{F^{(t)}}\right).$$ (A.38)

Here,

• $h\left({\beta }_k^{(t)}\right)$ is the k-th component of h(B^(t)) defined in (III-B)

• ${\nabla }_B\mathcal{L}(B)=\left({\nabla }_1\mathcal{L}\left({\beta }_1\right),\cdots ,{\nabla }_K\mathcal{L}\left({\beta }_K\right)\right).$

• $F^{(t)}={\cup }_{k=1}^K{F}_k^{(t)}$, where $F_k^{(t)}=\text{supp}\left({\beta }_k^{*}\right)\cup \text{supp}\left({\beta }_k^{(t)}\right)$.

• For a vector $x\in {ℝ}^p$ and a subset A ⊂ {1,…,p}, we denote $x_A\in {ℝ}^p$ by keeping the coordinates of x with indices in A unchanged, while changing all other components to zero.

We will show that ${\tilde{\beta }}_k^{(t+1)}$ converges as a geometric rate for optimization error and an optimal rate for statistical error. See Lemma 13 for details.

Second, we aim to prove that ${\tilde{\beta }}_k^{(t+1)}$ and ${\beta }_k^{(t+1)}$ are almost equivalent with high probability. See Lemma 14 for details. For simplicity, we drop the superscript of ${\beta }_k^{(t)}$, F^(t) in the following proof, and denote ${\tilde{\beta }}_k^{(t+1)}$, ${\beta }_k^{(t+1)}$ and F^(t+1) by ${\tilde{\beta }}_k^{+}$, ${\tilde{\beta }}_k^{+}$ and F⁺ respectively.

Lemma 13. Suppose Conditions 1–5 hold. Assume (A.13) is satisfied and |F| ≲ Ks. As long as the step size μ ≤ 32R^−20/3/(3K[220 + 270K]²), we obtain the upper bound for $\left\{{\tilde{\beta }}_k^{+}\right\}$,

$$\sum _{k=1}^K{\Vert \sqrt[3]{{\eta }_k}{\tilde{\beta }}_k^{+}-\sqrt[3]{{\eta }_k^{*}}{\beta }_k^{*}\Vert }_2^2\le \left(1-32\mu \frac{R^{-\frac{8}{3}}}{K^2}\right)\sum _{k=1}^K{\Vert \sqrt[3]{{\eta }_k}{\beta }_k-\sqrt[3]{{\eta }_k^{*}}{\beta }_k^{*}\Vert }_2^2+2C_3{\mu }^2{R}^{-\frac{8}{3}}{\eta }_{\text{min}}^{*-\frac{4}{3}}\frac{{\sigma }^2{K}^{-2}s\text{ log }p}{n},$$ (A.39)

with probability at least 1 − (21K² + 11K + 4Ks)/n.

The proof of Lemma 13 is postponed to the Section L. Next lemma guarantees that with high probability, ${\left\{{\beta }_k^{+}\right\}}_{k=1}^K$ is equivalent to the oracle update ${\left\{{\tilde{\beta }}_k^{+}\right\}}_{k=1}^K$ with high probability.

Lemma 14. Recall that the truncation level h(β_k) is defined as

$$h\left({\beta }_k\right)=\frac{\sqrt{4\text{ log }np}}{n}*\sqrt{\sum _{i=1}^n{\left(\sum _{k=1}^K{\eta }_k{\left(x_i^{\top }{\beta }_k\right)}^3-y_i\right)}^2{\left({\eta }_k{\left(x_i^{\top }{\beta }_k\right)}^2\right)}^2}.$$ (A.40)

If |F| ≲ Ks, we have ${\beta }_k^{+}={\tilde{\beta }}_k^{+}$ for any k ∈ [K] with probability at least 1 − (n²p)⁻¹ and F⁺ ⊂ F.

The proof of Lemma 14 is postponed to the Section L. By using Lemma 14 and induction, we have

$$F^{(t+1)}\subset \cdots F^{(1)}\subset F^{(0)}={\cup }_{k=1}^K\text{supp}\left({\beta }_k^{*}\right)\cup \text{supp}\left({\beta }_k^{(0)}\right).$$

It implies for every t, we have |F^(t)| ≲ Ks. Combining with Lemmas 13 and 14 together, we obtain with probability at least 1 − (21K² + 11K + 4Ks)/n,

$$\sum _{k=1}^K{\Vert \sqrt[3]{{\eta }_k}{\beta }_k^{+}-\sqrt[3]{{\eta }_k^{*}}{\beta }_k^{*}\Vert }_2^2\le \left(1-32\mu K^{-2}{R}^{-\frac{8}{3}}\right)\sum _{k=1}^K{\Vert \sqrt[3]{{\eta }_k}{\beta }_k-\sqrt[3]{{\eta }_k^{*}}{\beta }_k^{*}\Vert }_2^2+2C_3{\mu }^2{R}^{-\frac{8}{3}}{\eta }_{\text{min}}^{*-\frac{4}{3}}\frac{{\sigma }^2{K}^{-2}s\text{ log }p}{n},$$ (A.41)

This ends the proof. ■

K. Proof of Lemma 12

Based on the CP low-rank structure of true tensor parameter $\mathcal{J}*$, we can explicitly write down the distance between $\mathcal{J}$ and $\mathcal{J}*$ under tensor Frobenius norm as follows

$$\begin{array}{l}{\Vert \mathcal{J}-\mathcal{J}*\Vert }_F^2\hfill \\ =\sum _{i_1,i_2,i_3}{\left(\sum _{k=1}^n{\eta }_k{\beta }_{k{i}_1}{\beta }_{k{i}_2}{\beta }_{k{i}_3}-\sum _{k=1}^n{\eta }_k^{*}{\beta }_{k{i}_1}^{*}{\beta }_{k{i}_2}^{*}{\beta }_{k{i}_3}^{*}\right)}^2.\hfill \end{array}$$

For notation simplicity, denote ${\overline{\beta }}_k=\sqrt[3]{{\eta }_k}{\beta }_k,{\overline{\beta }}_k^{*}=\sqrt[3]{{\eta }_k^{*}}{\beta }_k^{*}$. Then

$$\begin{array}{l}{\Vert \mathcal{J}-\mathcal{J}*\Vert }_F^2\\ =\sum _{i_1,i_2,i_3}{\left(\sum _{k=1}^K{\overline{\beta }}_{k{i}_1}{\overline{\beta }}_{k{i}_2}{\overline{\beta }}_{k{i}_3}-\sum _{k=1}^K{\overline{\beta }}_{k{i}_1}^{*}{\overline{\beta }}_{k{i}_2}^{*}{\overline{\beta }}_{k{i}_3}^{*}\right)}^2\\ =\sum _{i_1,i_2,i_3}{\left(\sum _{k=1}^K\left({\overline{\beta }}_{k{i}_1}-{\overline{\beta }}_{k{i}_1}^{*}\right){\overline{\beta }}_{k{i}_2}^{*}{\overline{\beta }}_{k{i}_3}^{*}+\sum _{k=1}^K{\overline{\beta }}_{k{i}_1}\left({\overline{\beta }}_{k{i}_2}-{\overline{\beta }}_{k{i}_2}^{*}\right){\overline{\beta }}_{k{i}_3}^{*}+\sum _{k=1}^K{\overline{\beta }}_{k{i}_1}{\overline{\beta }}_{k{i}_2}\left({\overline{\beta }}_{k{i}_3}-{\overline{\beta }}_{k{i}_3}^{*}\right)+\sum _{k=1}^K{\overline{\beta }}_{k{i}_1}{\overline{\beta }}_{k{i}_2}\left({\overline{\beta }}_{k{i}_3}-{\overline{\beta }}_{k{i}_3}^{*}\right)\right)}^2=\text{RHS}.\end{array}$$

Since (a + b + c)² ≤ 3(a² + b² + c²), we have

$$\text{RHS}\le 3\sum _{i_1,i_2,i_3}\left[{\left(\sum _{k=1}^K\left({\overline{\beta }}_{k{i}_1}-{\overline{\beta }}_{k{i}_1}^{*}\right){\overline{\beta }}_{k{i}_2}^{*}{\overline{\beta }}_{k{i}_3}^{*}\right)}^2+{\left(\sum _{k=1}^K{\overline{\beta }}_{k{i}_1}\left({\overline{\beta }}_{k{i}_2}-{\overline{\beta }}_{k{i}_2}^{*}\right){\overline{\beta }}_{k{i}_3}^{*}\right)}^2+{\left(\sum _{k=1}^K{\overline{\beta }}_{k{i}_1}{\overline{\beta }}_{k{i}_2}\left({\overline{\beta }}_{k{i}_3}-{\overline{\beta }}_{k{i}_3}^{*}\right)\right)}^2\right].$$

Equipped with Cauchy-Schwarz inequality, RHS can be further bounded by

$$\text{RHS}\le 3\sum _{i_1,i_2,i_3}\left[\sum _{k=1}^K{\left({\overline{\beta }}_{k{i}_1}-{\overline{\beta }}_{k{i}_1}^{*}\right)}^2\sum _{k=1}^K{\overline{\beta }}_{k{i}_2}^{*2}{\overline{\beta }}_{k{i}_3}^{*2}+\sum _{k=1}^K{\left({\overline{\beta }}_{k{i}_2}-{\overline{\beta }}_{k{i}_2}^{*}\right)}^2\sum _{k=1}^K{\overline{\beta }}_{k{i}_1}^2{\overline{\beta }}_{k{i}_3}^{*2}+\sum _{k=1}^K{\left({\overline{\beta }}_{k{i}_3}-{\overline{\beta }}_{k{i}_3}^{*}\right)}^2\sum _{k=1}^K{\overline{\beta }}_{k{i}_2}^2{\overline{\beta }}_{k{i}_1}^2\right]$$

At the same time, using ${\eta }_k\le (1+c){\eta }_k^{*}$ for k ∈ [K],

$$\begin{array}{l}{\Vert \mathcal{J}-\mathcal{J}*\Vert }_F^2\\ \le 3\left[\sum _{i_1=1}^p\sum _{k=1}^K{\left({\overline{\beta }}_{k{i}_1}-{\overline{\beta }}_{k{i}_1}^{*}\right)}^2\left(\sum _{i_2=1}^p\sum _{i_3=1}^p\sum _{k=1}^K{\overline{\beta }}_{k{i}_2}^{*2}{\overline{\beta }}_{k{i}_3}^{*2}\right)+\sum _{i_2=1}^p\sum _{k=1}^K{\left({\overline{\beta }}_{k{i}_2}-{\overline{\beta }}_{k{i}_2}^{*}\right)}^2\left(\sum _{i_1=1}^p\sum _{i_3=1}^p\sum _{k=1}^K{\overline{\beta }}_{k{i}_1}^2{\overline{\beta }}_{k{i}_3}^{*2}\right)+\sum _{i_3=1}^p\sum _{k=1}^K{\left({\overline{\beta }}_{k{i}_3}-{\overline{\beta }}_{k{i}_3}^{*}\right)}^2\left(\sum _{i_2=1}^p\sum _{i_1=1}^p\sum _{k=1}^K{\overline{\beta }}_{k{i}_2}^2{\overline{\beta }}_{k{i}_1}^2\right)\right]\\ =3\left(\sum _{k=1}^K{\Vert {\overline{\beta }}_k-{\overline{\beta }}_k^{*}\Vert }_2^2\right)\left(\sum _{k=1}^K{\left(\sqrt[3]{{\eta }_k^{*}}\right)}^4+\sum _{k=1}^K{\left(\sqrt[3]{{\eta }_k^{*}}\right)}^2{\left(\sqrt[3]{{\eta }_k}\right)}^2+\sum _{k=1}^K{\left(\sqrt[3]{{\eta }_k}\right)}^4\right)\\ \le 9(1+c)\left(\sum _{k=1}^K{\Vert {\overline{\beta }}_k-{\overline{\beta }}_k^{*}\Vert }_2^2\right)\left(\sum _{k=1}^K{\left(\sqrt[3]{{\eta }_k^{*}}\right)}^4\right).\end{array}$$

For the non-symmetric tensor estimation model, we have

$${\Vert \mathcal{J}-\mathcal{J}*\Vert }_F^2=\sum _{i_1,i_2,i_3}{\left(\sum _{k=1}^K{\eta }_k{\beta }_{1k{i}_1}{\beta }_{2k{i}_2}{\beta }_{3k{i}_3}-\sum _{k=1}^K{\eta }_k^{*}{\beta }_{1k{i}_1}^{*}{\beta }_{2k{i}_2}^{*}{\beta }_{3k{i}_3}^{*}\right)}^2.$$

Following the same strategy above, we obtain

$${\Vert \mathcal{J}-\mathcal{J}*\Vert }_F^2\le 3(1+c)\left(\sum _{k=1}^K{\Vert {\overline{\beta }}_{1k}-{\overline{\beta }}_{1k}^{*}\Vert }_2^2+\sum _{k=1}^K{\Vert {\overline{\beta }}_{2k}-{\overline{\beta }}_{2k}^{*}\Vert }_2^2+\sum _{k=1}^K{\Vert {\overline{\beta }}_{3k}-{\overline{\beta }}_{3k}^{*}\Vert }_2^2)(\sum _{k=1}^K{\left(\sqrt[3]{{\eta }_k^{*}}\right)}^4\right).$$

This ends the proof. ■

L. Proof of Lemma 13

First of all, we state a lemma to illustrate the effect of weight ϕ. The proof of Lemma 15 is deferred to Section 15.

Lemma 15. Consider ${\left\{y_i\right\}}_{i=1}^n$ come from either non-symmetric tensor estimation model (VI.1) or symmetric tensor estimation model (III.1). Suppose Conditions 3–5 hold. Then $\varphi =\frac{1}{n}{\sum }_{i=1}^n{y}_i^2$ is upper and lower bounded by

$$\left(16-6{\Gamma }^3-9\Gamma \right){\left(\sum _{k=1}^K{\eta }_k^{*}\right)}^2\le \frac{1}{n}\sum _{i=1}^n{y}_i^2\le \left(16+6{\Gamma }^3+9\Gamma \right){\left(\sum _{k=1}^K{\eta }_k^{*}\right)}^2,$$

with probability at least 1 − (K² + K + 3)/n, where Γ is the incoherence parameter defined in Definition 3.

According to Lemma 15, $\frac{1}{n}{\sum }_{i=1}^n{y}_i^2$ approximates ${\left({\sum }_{k=1}^K{\eta }_k^{*}\right)}^2$ up to some constants with high probability. Moreover, we know that from (A.13), ${\text{max}}_k|{\eta }_k-{\eta }_k^{*}|\le {\epsilon }_0$ for some small ε₀. Based on those two facts described above, we replace ηk by ${\eta }_k^{*}$ and ϕ by ${\left({\sum }_{k=1}^K{\eta }_k^{*}\right)}^2$ for the sake of completeness. Note that this change could only result in some constant scale changes for final results. Similar simplification was used in matrix recovery scenario [62]. Therefore, we define the weighted estimator and weighted true parameter as ${\overline{\beta }}_k=\sqrt[3]{{\eta }_k^{*}}{\beta }_k$, ${\overline{\beta }}_k^{*}=\sqrt[3]{{\eta }_k^{*}}{\beta }_k^{*}$. Now, ${\eta }_k^{*}{\beta }_k\circ {\beta }_k\circ {\beta }_k={\overline{\beta }}_k\circ {\overline{\beta }}_k\circ {\overline{\beta }}_k$. Recall · is the loss function defined in (III.4). Correspondingly with a slight abuse of notation, define the gradient function ${\nabla }_k\mathcal{L}\left({\overline{\beta }}_k\right)$ on F as

$${\nabla }_k\mathcal{L}{\left({\overline{\beta }}_k\right)}_F=\frac{6\sqrt[3]{{\eta }_k^{*}}}{n}\sum _{i=1}^n\left(\sum _{k^{\prime }=1}^K{\left(x_{i_F}^{\top }{\overline{\beta }}_{k^{\prime }}\right)}^3-y_i\right){\left(x_{i_F}^{\top }{\overline{\beta }}_k\right)}^2{x}_{i_F},$$

and its noiseless version as

$${\nabla }_k\mathcal{L}{\left({\overline{\beta }}_k\right)}_F=\frac{6\sqrt[3]{{\eta }_k^{*}}}{n}\sum _{i=1}^n\left(\sum _{k^{\prime }=1}^K{\left(x_{i_F}^{\top }{\overline{\beta }}_{k^{\prime }}\right)}^3-\sum _{k^{\prime }=1}^K{\left(x_{i_F}^{\top }{\overline{\beta }}_{k^{\prime }}^{*}\right)}^3\right){\left(x_{i_F}^{\top }{\overline{\beta }}_k\right)}^2{x}_{i_F}.$$ (A.42)

According to the definition of thresholding function (III.8), ${\tilde{\beta }}_k^{+}$ can be written as

$${\tilde{\beta }}_k^{+}={\beta }_k-\frac{\mu }{\varphi }{\nabla }_k\mathcal{L}{\left({\overline{\beta }}_k\right)}_F+\frac{\mu }{\varphi }h\left({\overline{\beta }}_k\right){\gamma }_k,$$

where ${\gamma }_k\in {ℝ}^p$ satisfies $\text{supp}\left({\gamma }_k\right)\subset F,{\Vert {\gamma }_k\Vert }_{\infty }\le 1$ and $h\left({\overline{\beta }}_k\right)$ is defined as

$$h\left({\overline{\beta }}_k\right)=\frac{\sqrt{4\text{ log}(np)}}{n}*\sqrt{\sum _{i=1}^n{\left(\sum _{k=1}^K{\left(x_{i_F}^{\top }{\overline{\beta }}_k\right)}^3-y_i\right)}^2{\eta }_k^{*\frac{2}{3}}{\left(x_{i_F}^{\top }{\overline{\beta }}_k\right)}^2}.$$ (A.43)

Moreover, we denote $z_k={\overline{\beta }}_k-{\overline{\beta }}_k^{*}$. With a little abuse of notations, we also drop the subscript F in this section for notation simplicities.

We expand and decompose the sum of square error by three parts as follows:

$$\sum _{k=1}^K{\Vert \sqrt[3]{{\eta }_k^{*}}{\tilde{\beta }}_k^{+}-\sqrt[3]{{\eta }_k^{*}}{\beta }_k^{*}\Vert }_2^2=\sum _{k=1}^K{\Vert z_k-\frac{\mu \sqrt[3]{{\eta }_k^{*}}}{\varphi }{\nabla }_k\mathcal{L}\left({\overline{\beta }}_k\right)+\frac{\mu \sqrt[3]{{\eta }_k^{*}}}{\varphi }h\left({\overline{\beta }}_k\right){\gamma }_k\Vert }_2^2=\underset{\text{A: gradient update effect }}{\underbrace{\sum _{k=1}^K{\Vert z_k-\mu \frac{\sqrt[3]{{\eta }_k^{*}}}{\varphi }{\nabla }_k\mathcal{L}\left({\overline{\beta }}_k\right)\Vert }_2^2}}+\underset{\text{B: threshoding effect }}{\underbrace{\sum _{k=1}^K{\Vert \frac{\mu \sqrt[3]{{\eta }_k^{*}}}{\varphi }h\left({\overline{\beta }}_k\right){\gamma }_k\Vert }_2^2}}+\underset{\text{C: cross term }}{\underbrace{\sum _{k=1}^K\langle z_k-\mu \frac{\sqrt[3]{{\eta }_k^{*}}}{\varphi }{\nabla }_k\mathcal{L}\left({\overline{\beta }}_k\right),\frac{\mu \sqrt[3]{{\eta }_k}}{\varphi }h\left({\overline{\beta }}_k\right){\gamma }_k\rangle .}}$$ (A.44)

In the following proof, we will bound three parts sequentially.

1). Bounding gradient update effect:

In order to separate the optimization error and statistical error, we use the noiseless gradient ${\nabla }_k\tilde{\mathcal{L}}\left({\overline{\beta }}_k\right)$ as a bridge such that A can be decomposed as

$$A=\sum _{k=1}^K{\Vert z_k\Vert }_2^2-2\mu \sum _{k=1}^K\langle \frac{\sqrt[3]{{\eta }_k^{*}}}{\varphi }{\nabla }_k\mathcal{L}\left({\overline{\beta }}_k\right),z_k\rangle +{\mu }^2\sum _{k=1}^K{\Vert \frac{\sqrt[3]{{\eta }_k^{*}}}{\varphi }{\nabla }_k\mathcal{L}\left({\overline{\beta }}_k\right)\Vert }_2^2\le \sum _{k=1}^K{\Vert z_k\Vert }_2^2-2\mu \underset{A_1}{\underbrace{\sum _{k=1}^K\langle \frac{\sqrt[3]{{\eta }_k^{*}}}{\varphi }{\nabla }_k\tilde{\mathcal{L}}\left({\overline{\beta }}_k\right),z_k\rangle }}+2{\mu }^2\underset{A_2}{\underbrace{\sum _{k=1}^K{\Vert \frac{\sqrt[3]{{\eta }_k^{*}}}{\varphi }{\nabla }_k\tilde{\mathcal{L}}\left({\overline{\beta }}_k\right)\Vert }_2^2}}+2{\mu }^2\underset{A_3}{\underbrace{\sum _{k=1}^K{\Vert \frac{\sqrt[3]{{\eta }_k^{*}}}{\varphi }\left({\nabla }_k\tilde{\mathcal{L}}\left({\overline{\beta }}_k\right)-{\nabla }_k\mathcal{L}\left({\overline{\beta }}_k\right)\right)\Vert }_2^2}}+2\mu \underset{A_4}{\underbrace{\sum _{k=1}^K\langle z_k,\frac{\sqrt[3]{{\eta }_k^{*}}}{\varphi }\left({\nabla }_k\tilde{\mathcal{L}}\left({\overline{\beta }}_k\right)-{\nabla }_k\mathcal{L}\left({\overline{\beta }}_k\right)\right)\rangle ,}}$$ (A.45)

where A₁ and A₂ quantify the optimization error, A₃ quantifies the statistical error, and A₄ is a cross term which can be negligible comparing with the rate of the statistical error. The lower bound for A₁ and upper bound for A₂ together coincide with the verification of regularity conditions in the matrix recovery case [52].

Step One: Lower bound for A₁.

Plugging in $\varphi ={\left({\sum }_{k=1}^K{\eta }_k^{*}\right)}^2$, we have

$$K^{-2}{R}^{-\frac{2}{3}}{\eta }_{\text{max}}^{*-\frac{4}{3}}\le \frac{{\left(\sqrt[3]{{\eta }_k^{*}}\right)}^2}{\varphi }=\frac{{\left(\sqrt[3]{{\eta }_k^{*}}\right)}^2}{{\left({\sum }_{k=1}^K{\eta }_k^{*}\right)}^2}\le K^{-2}{R}^{\frac{2}{3}}{\eta }_{\text{max}}^{*-\frac{4}{3}}.$$ (A.46)

According to the definition of noiseless gradient ${\nabla }_k\tilde{\mathcal{L}}\left({\tilde{\beta }}_k\right)$ and z_k, A₁ can be expanded and decomposed sequentially by nine terms,

$$A_1\ge K^{-2}{R}^{-\frac{2}{3}}{\eta }_{\text{max}}^{*-\frac{4}{3}}[\frac{6}{n}\sum _{i=1}^n\left(\sum _{k^{\prime }=1}^{K}3\left(x_i^{\top }{z}_{k^{\prime }}\right){\left(x_i^{\top }{\overline{\beta }}_{k^{\prime }}\right)}^2\sum _{k=1}^K\left(x_i^{\top }{z}_k\right){\left(x_i^{\top }{\overline{\beta }}_k^{*}\right)}^2\right)\Leftarrow A_{11}+\frac{6}{n}\sum _{i=1}^n\left(\sum _{k^{\prime }=1}^{K}3\left(x_i^{\top }{z}_{k^{\prime }}\right){\left(x_i^{\top }{\overline{\beta }}_{k^{\prime }}\right)}^2\sum _{k=1}^{K}2{\left(x_i^{\top }{z}_k\right)}^2\left(x_i^{\top }{\overline{\beta }}_k^{*}\right)\right)\Leftarrow A_{12}+\frac{6}{n}\sum _{i=1}^n\left(\sum _{k^{\prime }=1}^{K}3\left(x_i^{\top }{z}_{k^{\prime }}\right){\left(x_i^{\top }{\overline{\beta }}_{k^{\prime }}\right)}^2\sum _{k=1}^K{\left(x_i^{\top }{z}_k\right)}^3\right)\Leftarrow A_{13}+\frac{6}{n}\sum _{i=1}^n\left(\sum _{k^{\prime }=1}^{K}3{\left(x_i^{\top }{z}_{k^{\prime }}\right)}^2\left(x_i^{\top }{\overline{\beta }}_{k^{\prime }}\right)\sum _{k=1}^K\left(x_i^{\top }{z}_k\right){\left(x_i^{\top }{\overline{\beta }}_k^{*}\right)}^2\right)\Leftarrow A_{14}+\frac{6}{n}\sum _{i=1}^n\left(\sum _{k=1}^{K}3{\left(x_i^{\top }{z}_{k^{\prime }}\right)}^2\left(x_i^{\top }{\overline{\beta }}_{k^{\prime }}\right)\sum _{k=1}^{K}2{\left(x_i^{\top }{z}_k\right)}^2\left(x_i^{\top }{\overline{\beta }}_k^{*}\right)\right)\Leftarrow A_{15}$$ (A.47)

$$+\frac{6}{n}\sum _{i=1}^n\left(\sum _{k^{\prime }=1}^{K}3{\left(x_i^{\top }{z}_{k^{\prime }}\right)}^2\left(x_i^{\top }{\overline{\beta }}_{k^{\prime }}\right)\sum _{k=1}^K{\left(x_i^{\top }{z}_k\right)}^2\left(x_i^{\top }{\overline{\beta }}_k^{*}\right)\right)\Leftarrow A_{16}+\frac{6}{n}\sum _{i=1}^n\left(\sum _{k^{\prime }=1}^{K}3{\left(x_i^{\top }{z}_{k^{\prime }}\right)}^3\sum _{k=1}^K\left(x_i^{\top }{z}_k\right){\left(x_i^{\top }{\overline{\beta }}_k^{*}\right)}^2\right)\Leftarrow A_{17}+\frac{6}{n}\sum _{i=1}^n\left(\sum _{k^{\prime }=1}^{K}3{\left(x_i^{\top }{z}_{k^{\prime }}\right)}^3\sum _{k=1}^{K}2{\left(x_i^{\top }{z}_k\right)}^2\left(x_i^{\top }{\overline{\beta }}_k^{*}\right)\right)\Leftarrow A_{18}+\frac{6}{n}\sum _{i=1}^n\left(\sum _{k^{\prime }=1}^{K}3{\left(x_i^{\top }{z}_{k^{\prime }}\right)}^3\sum _{k=1}^K{\left(x_i^{\top }{z}_k\right)}^3\right)]\Leftarrow A_{19},$$ (A.48)

where A₁₁ is the main term according to the order of ${\overline{\beta }}_k^{*}$, while A₁₂ to A₁₉ are remainder terms. The proof of lower bound for A₁₁ to A₁₉ follows two steps:

1. Calculate and lower bound the expectation of each term through Lemma A.2: high-order Gaussian moment;

2. Argue that the empirical version is concentrated around their expectation with high probability through Lemma 1: high-order concentration inequality.

Bounding A₁₁. Note that A₁₁ involves the product of dependent Gaussian vectors. This brings difficulties on both the calculation of expectations and the use of concentration inequality. According to the high-order Gaussian moment results in Lemma A.2, the expectation of A₁₁ can be calculated explicitly as

$$\mathbb{E}\left(A_{11}\right)=36\sum _{k=1}^K\sum _{k^{\prime }=1}^K{\left({\overline{\beta }}_{k^{\prime }}^{*\top }{\overline{\beta }}_k^{*}\right)}^2\left(z_{k^{\prime }}^{\top }{z}_k\right)\Leftarrow I_1+72\sum _{k=1}^K\sum _{k^{\prime }=1}^K\left({\overline{\beta }}_{k^{\prime }}^{*\top }{\overline{\beta }}_k^{*}\right)\left(z_{k^{\prime }}^{\top }{\overline{\beta }}_k^{*}\right)\left(z_k^{\top }{\overline{\beta }}_{k^{\prime }}^{*}\right)\Leftarrow I_2+108\sum _{k=1}^K\sum _{k^{\prime }=1}^K\left(z_k^{\top }{\overline{\beta }}_{k^{\prime }}\right)\left(z_{k^{\prime }}^{\top }{\overline{\beta }}_{k^{\prime }}^{*}\right)\left(z_k^{\top }{\overline{\beta }}_k^{*}\right)\Leftarrow I_3+54\sum _{k=1}^K\sum _{k^{\prime }=1}^K\left({\overline{\beta }}_{k^{\prime }}^{*\top }{\overline{\beta }}_{k^{\prime }}^{*}\right)\left(z_{k^{\prime }}^{\top }{\overline{\beta }}_k^{*}\right)\left(z_k^{\top }{z}_k\right)\Leftarrow I_4.$$ (A.49)

Note that I₁ to I₄ involve the summation of K² term. To use incoherence Condition 3, we isolate K terms with k = k′. Then, I₁ to I₄ could be lower bounded as

$$\begin{array}{l}{I}_1\ge 36{\eta }_{\text{min}}^{*4/3}\left[\sum _{k=1}^K{\Vert z_k\Vert }_2^2-{\Gamma }^2{\left(\sum _{k=1}^K{\Vert z_k\Vert }_2\right)}^2\right]\hfill \\ I_2\ge 72{\eta }_{\text{min}}^{*4/3}\left[\sum _{k=1}^K{\left(z_k^{\top }{\overline{\beta }}_k^{*}\right)}^2-\Gamma {\left(\sum _{k=1}^K{\Vert z_k\Vert }_2\right)}^2\right]\hfill \\ I_3\ge 108{\eta }_{\text{min}}^{*4/3}\left[\sum _{k=1}^K{\left(z_k^{\top }{\overline{\beta }}_k^{*}\right)}^2-\Gamma {\left(\sum _{k=1}^K{\Vert z_k\Vert }_2\right)}^2\right]\hfill \\ I_4\ge 54{\eta }_{\text{min}}^{*4/3}{\Vert \sum _{k=1}^K{z}_k\Vert }_2^2\ge 0,\hfill \end{array}$$

where Γ is the incoherence parameter. Putting the above four bounds together, they jointly provide

$$\mathbb{E}\left(A_{11}\right)\ge 36{\eta }_{\text{min}}^{*4/3}\sum _{k=1}^K{\Vert z_k\Vert }_2^2-\left(36{\eta }_{\text{min}}^{*4/3}{\Gamma }^2+180{\eta }_{\text{min}}^{*4/3}\Gamma \right){\left(\sum _{k=1}^K{\Vert z_k\Vert }_2\right)}^2.$$ (A.50)

On the other hand, repeatedly using Lemma 1, we obtain that with probability at least 1 − 1/n,

$$\left|\frac{1}{n}\sum _{i=1}^n(\left(x_i^{\top }{z}_{k^{\prime }}\right){\left(x_i^{\top }{\overline{\beta }}_{k^{\prime }}^{*}\right)}^2\left(x_i^{\top }{z}_k\right){\left(x_i^{\top }{\overline{\beta }}_k^{*}\right)}^2-\mathbb{E}\left(x_i^{\top }{z}_{k^{\prime }}\right){\left(x_i^{\top }{\overline{\beta }}_{k^{\prime }}^{*}\right)}^2\left(x_i^{\top }{z}_k\right){\left(x_i^{\top }{\overline{\beta }}_k^{*}\right)}^2)\right|\le C\frac{{(\text{log }n)}^3}{\sqrt{n}}{\left(\sqrt[3]{{\eta }_{\text{max}}^{*}}\right)}^4{\Vert z_{k^{\prime }}\Vert }_2{\Vert z_k\Vert }_2.$$

Taking the summation over k,k′ ∈ [K], it could further imply that for some absolute constant C,

$$|A_{11}-\mathbb{E}\left(A_{11}\right)|\le 18C\frac{{(\text{log }n)}^3}{\sqrt{n}}{\left(\sqrt[3]{{\eta }_{\text{max}}^{*}}\right)}^4{\left(\sum _{k=1}^K{\Vert z_k\Vert }_2\right)}^2,$$ (A.51)

with probability at least 1 − K²/n.. Combining (A.50) and (A.51), we obtain with probability at least 1 − K²/n,

$$K^{-2}{R}^{-\frac{2}{3}}{\eta }_{\text{max}}^{*-\frac{4}{3}}{A}_{11}\ge \left[36K^{-2}{R}^{-\frac{8}{3}}-K^{-\frac{3}{2}}\left(216R^{-\frac{8}{3}}\Gamma +18C\frac{{(\text{log }n)}^3}{\sqrt{n}}\right)\right]\sum _{k=1}^K{\Vert z_k\Vert }_2^2,$$ (A.52)

Where $R={\eta }_{\text{max}}^{*}/{\eta }_{\text{max}}^{*}$. Here, we use the fact Γ ≤ 1 and ${\left({\sum }_{k=1}^K{\Vert z_k\Vert }_2^2\right)}^2\le K\left({\sum }_{k=1}^K{\Vert z_k\Vert }_2^2\right)$.

Bounding A₁₂ to A₁₉: For remainder terms, we follow the same proof strategy. According to Lemma A.2, the expectation of A₁₂ can be calculated as

$$\begin{array}{l}\mathbb{E}\left(A_{12}\right)=36\sum _{k=1}^K\sum _{k^{\prime }=1}^K{\left(z_k^{\top }{\overline{\beta }}_{k^{\prime }}^{*\top }\right)}^2\left(z_{k^{\prime }}^{\top }{\overline{\beta }}_k^{*}\right)\Leftarrow I_1\\ +72\sum _{k=1}^K\sum _{k^{\prime }=1}^K\left(z_k^{\top }{\overline{\beta }}_{k^{\prime }}^{*}\right)\left({\overline{\beta }}_{k^{\prime }}^{*\top }{\overline{\beta }}_k^{*}\right)\left(z_{k^{\prime }}^{\top }{z}_k\right)\Leftarrow I_2\\ +108\sum _{k=1}^K\sum _{k^{\prime }=1}^K\left(z_k^{\top }{\overline{\beta }}_{k^{\prime }}\right)\left(z_{k^{\prime }}^{\top }{\overline{\beta }}_{k^{\prime }}^{*}\right)\left(z_k^{\top }{\overline{\beta }}_k^{*}\right)\Leftarrow I_3\\ +54\sum _{k=1}^K\sum _{k^{\prime }=1}^K\left({\overline{\beta }}_{k^{\prime }}^{*\top }{\overline{\beta }}_{k^{\prime }}^{*}\right)\left(z_{k^{\prime }}^{\top }{\overline{\beta }}_k^{*}\right)\left(z_k^{\top }{z}_k\right)\Leftarrow I_4.\end{array}$$

Let us analyze I₁ first. Under (A.13), ${\Vert z_k\Vert }_2\le {\epsilon }_0\sqrt[3]{{\eta }_k^{*}}$, it suffices to show that

$$\sum _{k=1}^K\sum _{k^{\prime }=1}^K{\left(z_k^{\top }{\overline{\beta }}_{k^{\prime }}\right)}^2\left(z_{k^{\prime }}^{\top }{\overline{\beta }}_k^{*}\right)\ge -\sum _{k=1}^K\sum _{k^{\prime }=1}^K{\Vert z_k\Vert }_2^2{\Vert {\overline{\beta }}_{k^{\prime }}^{*}\Vert }_2^2{\Vert z_k^{\prime }\Vert }_2{\Vert {\overline{\beta }}_k^{*}\Vert }_2\ge -{\eta }_{\text{max}}^{*\frac{4}{3}}{\epsilon }_0{\left(\sum _{k=1}^K{\Vert z_k\Vert }_2\right)}^2.$$

This immediately implies a lower bound for $\mathbb{E}\left(A_{12}\right)$ after we bound similarly for I₂,I₃ and I₄,

$$\mathbb{E}\left(A_{12}\right)\ge -270{\eta }_{\text{max}}^{*\frac{4}{3}}{\epsilon }_0{\left(\sum _{k=1}^K{\Vert z_k\Vert }_2\right)}^2.$$ (A.53)

By Lemma 1, we obtain for some absolute constant C,

$$K^{-2}{R}^{-\frac{2}{3}}{\eta }_{\text{max}}^{*{-}_3^4}{A}_{12}\ge K^{-2}{R}^{-\frac{2}{3}}{\eta }_{\text{max}}^{*{-}_3^4}\left[\mathbb{E}\left(A_{12}\right)-18C{\eta }_{\text{max}}^{*{-}_3^4}{\epsilon }_0{\left(\sum _{k=1}^K{\Vert z_k\Vert }_2\right)}^2\frac{{(\text{log }n)}^3}{\sqrt{n}}\right]\ge -K^{-1}{R}^{-\frac{2}{3}}{\epsilon }_0\left(270+18C\frac{{(\text{log }n)}^3}{\sqrt{n}}\right)\left(\sum _{k=1}^K{\Vert z_k\Vert }_2^2\right),$$ (A.54)

with probability at least 1−K²/n. The detail derivation is the same as in (A.52), so we omit here.

Similarly, the lower bounds of A₁₃ to A₁₉ can be derived as follows

$$K^{-\frac{1}{2}}{\eta }_{\text{max}}^{*-\frac{3}{4}}{A}_{14}\ge -K^{\frac{1}{2}}{\epsilon }_0\left(270+18C\frac{{(\text{log }n)}^3}{\sqrt{n}}\right)\left(\sum _{k=1}^K{\Vert z_k\Vert }_2^2\right)K^{-\frac{1}{2}}{\eta }_{\text{max}}^{*-\frac{3}{4}}{A}_{13},A_{15},A_{17}\ge -K^{\frac{1}{2}}{\epsilon }_0^2\left(270+18C\frac{{(\text{log }n)}^3}{\sqrt{n}}\right)\left(\sum _{k=1}^K{\Vert z_k\Vert }_2^2\right)K^{-\frac{1}{2}}{\eta }_{\text{max}}^{*-\frac{3}{4}}{A}_{16},A_{18}\ge -K^{\frac{1}{2}}{\epsilon }_0^3\left(270+18C\frac{{(\text{log }n)}^3}{\sqrt{n}}\right)\left(\sum _{k=1}^K{\Vert z_k\Vert }_2^2\right)K^{-\frac{1}{2}}{\eta }_{\text{max}}^{*-\frac{3}{4}}{A}_{19}\ge -K^{\frac{1}{2}}{\epsilon }_0^4\left(270+18C\frac{{(\text{log }n)}^3}{\sqrt{n}}\right)\left(\sum _{k=1}^K{\Vert z_k\Vert }_2^2\right).$$ (A.55)

Putting (A.52), (A.54) and (A.55) together, we have with probability at least 1 − 9K²/n,

$$A_1\ge \left[36K^{-2}{R}^{-\frac{8}{3}}-K^{-\frac{3}{2}}\left(2160R^{-\frac{3}{3}}\Gamma +18C\frac{{(\text{log }n)}^3}{\sqrt{n}}\right)-8{\epsilon }_0{K}^{-1}{R}^{-\frac{2}{3}}\left(270+18C\frac{{(\text{log }n)}^3}{\sqrt{n}}\right)\right]\left(\sum _{k=1}^K{\Vert z_k\Vert }_2^2\right).$$

For the above bound,

• When the sample size satisfies

  $$n\ge {\left(18C{K}^{1/2}{R}^{8/3}{(\text{log }n)}^3\right)}^2,$$

  we have

  $$\text{max}\left\{18K^{-\frac{3}{2}}C\frac{{(\text{log }n)}^3}{\sqrt{n}},8{\epsilon }_0{K}^{-1}{R}^{-\frac{2}{3}}18C\frac{{(\text{log }n)}^3}{\sqrt{n}}\right\}\le K^{-2}{R}^{-\frac{8}{3}}.$$
• When ε₀ ≤ K⁻¹R⁻²/2160, we have

  $$8{\epsilon }_0{K}^{-1}{R}^{-\frac{2}{3}}270\le K^{-2}{R}^{-\frac{8}{3}}.$$
• When the incoherence parameter satisfies Γ ≤ K^−1/2/216, we have

  $$K^{-\frac{3}{2}}2160R^{-\frac{8}{3}}\Gamma \le K^{-2}{R}^{-\frac{8}{3}}.$$

Note that those above conditions can be fulfilled by Conditions 3, 5 and (A.13). Thus, we are able to simplify A₁ by

$$A_1\ge 32K^{-2}{R}^{-\frac{8}{3}}\left(\sum _{k=1}^K{\Vert z_k\Vert }_2^2\right),$$ (A.56)

with probability at least 1 − 9K²/n.

Step Two: Upper bound for A₂.

We observe the fact that

$$A_2=\sum _{k=1}^K{\Vert \frac{1}{\varphi }\sqrt[3]{{\eta }_k^{*}}{\nabla }_k\tilde{\mathcal{L}}\left({\overline{\beta }}_k\right)\Vert }_2^2=\underset{w\in {\mathbb{S}}^{K_{s-1}}}{\text{sup}}{|\langle \sum _{k=1}^R\frac{\sqrt[3]{{\eta }_k^{*}}}{\varphi }{\nabla }_k\tilde{\mathcal{L}}\left({\overline{\beta }}_k\right),w\rangle |}^2,$$ (A.57)

where $\mathbb{S}$ is a unit sphere. It is equivalent to show for any $w\in {\mathbb{S}}^{Ks-1},A_2^{\prime }=|\langle {\sum }_{k=1}^K\frac{\sqrt[3]{{\eta }_k^{*}}}{\varphi }{\nabla }_k\tilde{\mathcal{L}}\left({\overline{\beta }}_k\right),w\rangle |$ is upper bounded. According to the definition of noiseless gradient (A.42), $A_2^{\prime }$ is explicitly written as

$$A_2^{\prime }=\frac{6}{n}\sum _{i=1}^n\left(\sum _{k^{\prime }=1}^K{\left(x_i^{\top }{\overline{\beta }}_{k^{\prime }}\right)}^3-\sum _{k^{\prime }=1}^K{\left(x_i^{\top }{\overline{\beta }}_{k^{\prime }}^{*}\right)}^3\right)*\left(\sum _{k=1}^K\frac{{\left(\sqrt[3]{{\eta }_k^{*}}\right)}^2}{\varphi }{\left(x_i^{\top }{\overline{\beta }}_k\right)}^2\left(x_i^{\top }w\right)\right).$$

Following by (A.46) and (A.48), similar decomposition can be made for $A_2^{\prime }$ as follows, where the only difference is that we replace one $x_i^{\top }{z}_k$ by $x_i^{\top }w$.

$$A_2^{\prime }\le K^{-2}{R}^{\frac{2}{3}}{\eta }_{\text{min}}^{*-\frac{4}{3}}\left[\frac{6}{n}\sum _{i=1}^n\left(\sum _{k^{\prime }=1}^{K}3\left(x_i^{\top }{z}_{k^{\prime }}\right){\left(x_i^{\top }{\overline{\beta }}_{k^{\prime }}\right)}^2\sum _{k=1}^K\left(x_i^{\top }w\right){\left(x_i^{\top }{\overline{\beta }}_k^{*}\right)}^2\right)\Leftarrow A_{21}^{\prime }+\frac{6}{n}\sum _{i=1}^n\left(\sum _{k^{\prime }=1}^{K}3\left(x_i^{\top }{z}_{k^{\prime }}\right){\left(x_i^{\top }{\overline{\beta }}_{k^{\prime }}\right)}^2\sum _{k=1}^{K}2\left(x_i^{\top }{z}_k\right)\left(x_i^{\top }w\right)\left(x_i^{\top }{\overline{\beta }}_k^{*}\right)\right)\Leftarrow A_{22}^{\prime }+\frac{6}{n}\sum _{i=1}^n\left(\sum _{k^{\prime }=1}^{K}3\left(x_i^{\top }{z}_{k^{\prime }}\right){\left(x_i^{\top }{\overline{\beta }}_{k^{\prime }}\right)}^2\sum _{k=1}^K{\left(x_i^{\top }{z}_k\right)}^2\left(x_i^{\top }w\right)\right)\Leftarrow A_{23}^{\prime }+\frac{6}{n}\sum _{i=1}^n\left(\sum _{k^{\prime }=1}^{K}3{\left(x_i^{\top }{z}_{k^{\prime }}\right)}^2\left(x_i^{\top }{\overline{\beta }}_{k^{\prime }}\right)\sum _{k=1}^K\left(x_i^{\top }w\right){\left(x_i^{\top }{\overline{\beta }}_k^{*}\right)}^2\right)\Leftarrow A_{24}^{\prime }+\frac{6}{n}\sum _{i=1}^n\left(\sum _{k^{\prime }=1}^{K}3{\left(x_i^{\top }{z}_{k^{\prime }}\right)}^2\left(x_i^{\top }{\overline{\beta }}_{k^{\prime }}\right)\sum _{k=1}^{K}2\left(x_i^{\top }{z}_k\right)\left(x_i^{\top }w\right)\left(x_i^{\top }{\overline{\beta }}_k^{*}\right)\right)\Leftarrow A_{25}^{\prime }+\frac{6}{n}\sum _{i=1}^n\left(\sum _{k^{\prime }-1}^{K}3{\left(x_i^{\top }{z}_{k^{\prime }}\right)}^2\left(x_i^{\top }{\overline{\beta }}_{k^{\prime }}\right)\sum _{k=1}^K\left(x_i^{\top }{z}_k\right)\left(x_i^{\top }w\right)\left(x_i^{\top }{\overline{\beta }}_k^{*}\right)\right)\Leftarrow A_{26}^{\prime }+\frac{6}{n}\sum _{i=1}^n\left(\sum _{k^{\prime }=1}^{K}3{\left(x_i^{\top }{z}_{k^{\prime }}\right)}^3\sum _{k=1}^K\left(x_i^{\top }w\right){\left(x_i^{\top }{\overline{\beta }}_k^{*}\right)}^2\right)\Leftarrow A_{27}^{\prime }+\frac{6}{n}\sum _{i=1}^n\left(\sum _{k^{\prime }=1}^{K}3{\left(x_i^{\top }{z}_{k^{\prime }}\right)}^3\sum _{k=1}^{K}2\left(x_i^{\top }{z}_k\right)\left(x_i^{\top }w\right)\left(x_i^{\top }{\overline{\beta }}_k^{*}\right)\right)\Leftarrow A_{28}^{\prime }+\frac{6}{n}\sum _{i=1}^n\left(\sum _{k^{\prime }=1}^{K}3{\left(x_i^{\top }{z}_{k^{\prime }}\right)}^3\sum _{k=1}^K{\left(x_i^{\top }{z}_k\right)}^2\left(x_i^{\top }w\right)\right)\right].\Leftarrow A_{29}^{\prime }$$

Let’s bound $A_{21}^{\prime }$ first. By using the same technique when calculating $\mathbb{E}\left(A_{11}\right)$ in (A.49), we derive an upper bound for $\mathbb{E}\left(A_{21}^{\prime }\right)$.

$$\mathbb{E}\left(A_{21}^{\prime }\right)\le 36{\eta }_{\text{max}}^{*\frac{4}{3}}\left(\sum _{k=1}^K{\Vert z_k\Vert }_2+(K-1)\sum _{k=1}^K\Gamma {\Vert z_k\Vert }_2\right)+180{\eta }_{\text{max}}^{*\frac{4}{3}}\left(\sum _{k=1}^K{\Vert z_k\Vert }_2+(K-1)\sum _{k=1}^K\Gamma {\Vert z_k\Vert }_2\right)+54{\eta }_{\text{max}}^{*\frac{4}{3}}\left(K\sum _{k=1}^K{\Vert z_k\Vert }_2\right).$$

Equipped with Lemma 2 and the definition of tensor spectral norm (II.3), it suffices to bound $A_{21}^{\prime }$ by

$$R^{\frac{2}{3}}{\eta }_{\text{min}}^{*-\frac{3}{4}}{K}^{-\frac{1}{2}}{A}_{21}^{\prime }\le K^{-2}{R}^2\left[216+54K+216K\Gamma +18CK{\delta }_{n,p,s}\right]\left(\sum _{k=1}^K{\Vert z_k\Vert }_2\right)$$

with probability at least 1−10K²/n³, where δ_n,p,s is defined in (IV.7).

The upper bounds for $A_{22}^{\prime }$ to $A_{29}^{\prime }$ follow similar forms. Combining them together, we can derive an upper bound for $A_2^{\prime }$ as follows

$$A_2^{\prime }\le K^{-2}{R}^2\left[216+270K+18CK{\delta }_{n,p,s}\right]\left(\sum _{k=1}^K{\Vert z_k\Vert }_2\right)\le K^{-2}{R}^2[220+270K]\left(\sum _{k=1}^K{\Vert z_k\Vert }_2\right),$$

with probability at least 1 − 90K²/n³, where the second inequality utilizes Condition 5. Therefore, the upper bound of A₂ is given as follows

$$A_2\le K^{-1}{R}^4{[220+270K]}^2\left(\sum _{k=1}^K{\Vert z_k\Vert }_2^2\right),$$ (A.58)

with probability at least 1 − 90K²/n³.

Step Three: Upper bound for A₃.

By the definition of noisy gradient and noiseless gradient, A₃ is explicitly written as

$$A_3=\sum _{k=1}^K{\Vert \frac{{\left(\sqrt[3]{{\eta }_k^{*}}\right)}^2}{\varphi }\frac{6}{n}\sum _{i=1}^n{ϵ}_i{\left(x_i^{\top }{\overline{\beta }}_k\right)}^2{x}_i\Vert }_2^2\le K^{-4}{R}^{\frac{4}{3}}{\eta }_{\text{min}}^{*-\frac{8}{3}}\sum _{k=1}^K{\left(\sqrt{Ks}\underset{j}{\text{max}}\frac{6}{n}\sum _{i=1}^n{ϵ}_i{\left(x_i^{\top }{\overline{\beta }}_k\right)}^2{x}_{ij}\right)}^2,$$

where the second inequality comes from (A.46). For fixed ${\left\{{ϵ}_i\right\}}_{i=1}^n$, applying Lemma 1, we have

$$|\sum _{i=1}^n{ϵ}_i{\left(x_i^{\top }{\overline{\beta }}_k\right)}^2{x}_{ij}-\mathbb{E}\left(\sum _{i=1}^n{ϵ}_i{\left(x_i^{\top }{\overline{\beta }}_k\right)}^2{x}_{ij}\right)|\le C{(\text{log }n)}^{\frac{3}{2}}\Vert \mathit{ϵ}{\Vert }_2{\Vert {\overline{\beta }}_k\Vert }_2^2,$$

with probability at least 1 − 1/n. Together with Lemma 23, we obtain for any j ∈ [Ks],

$$|\frac{6}{n}\sum _{i=1}^n{ϵ}_i{\left(x_i^{\top }{\overline{\beta }}_k\right)}^2{x}_{ij}|\le 6C{C}_0\sigma {\Vert {\overline{\beta }}_k\Vert }_2^2\frac{{(\text{log }n)}^{3/2}}{\sqrt{n}},$$

with probability at least 1 − 4/n, where σ is the noise level. According to (A.13),

$${\Vert {\overline{\beta }}_k-{\overline{\beta }}_k^{*}\Vert }_2^2\le \sum _{k=1}^K{\Vert {\overline{\beta }}_k-{\overline{\beta }}_k^{*}\Vert }_2^2\le K{\eta }_{\text{max}}^{*\frac{2}{3}}{\epsilon }_0^2,$$

which further implies ${\Vert {\overline{\beta }}_k\Vert }_2^2\le {\left(1+K^{\frac{1}{2}}{\epsilon }_0\right)}^2{\eta }_{\text{max}}^{*\frac{2}{3}}$. Equipped with union bound over j ∈ [Ks],

$$\underset{j\in [Ks]}{\text{max}}|\frac{6}{n}\sum _{i=1}^n{ϵ}_i{\left(x_i^{\top }{\overline{\beta }}_k\right)}^2{x}_{ij}|\le 6C{C}_0\sigma {\left(1+K^{\frac{1}{2}}{\epsilon }_0\right)}^2{\left(\sqrt[3]{{\eta }_{\text{max}}^{*}}\right)}^2\frac{{(\text{log }n)}^{3/2}}{\sqrt{n}},\le 6C{C}_0\sigma {\left(1+K^{\frac{1}{2}}{\epsilon }_0\right)}^2{\left(\sqrt[3]{{\eta }_{\text{max}}^{*}}\right)}^2\frac{{(\text{log }n)}^{3/2}}{\sqrt{n}},$$

with probability at least 1 − 4Ks/n. Letting $C=6C_0{(Ce)}^{-2/3}{\left(1+K^{\frac{1}{2}}{\epsilon }_0\right)}^2$,

$$A_3\le C{\eta }_{\text{min}}^{*-\frac{4}{3}}{R}^{\frac{8}{3}}{\sigma }^2{K}^{-2}\frac{s{(\text{log }n)}^3}{n},$$ (A.59)

with probability at least 1 − 4Ks/n.

Step Four: Upper bound for A₄.

This cross term can be written as

$$A_4=2\sum _{k=1}^K\frac{\mu }{\varphi }{\left(\sqrt[3]{{\eta }_k^{*}}\right)}^2\left(\frac{1}{n}\sum _{i=1}^n{ϵ}_i{\left(x_i^{\top }{\overline{\beta }}_k\right)}^2\left(x_i^{\top }{z}_k\right)\right).$$

To bound this term, we take the same step in Step Three which fixes the noise term ${\left\{{ϵ}_i\right\}}_{i=1}^n$ first. Similarly, we obtain with probability at least 1 − 4K/n,

$$A_4\le 2C\sigma \frac{{(\text{log }n)}^{\frac{3}{2}}}{\sqrt{n}}{K}^{-1}{R}^{\frac{4}{3}}{\eta }_{\text{min}}^{*-\frac{2}{3}}.$$ (A.60)

This term is negligible in terms of the order when comparing with (A.59).

Summary. Putting the bounds (A.56), (A.58), (A.59) and (A.60) together, we achieve an upper bound for gradient update effect as follows,

$$A\le \left(1-64\mu K^{-2}{R}^{-\frac{8}{3}}+2{\mu }^2{K}^{-1}{R}^4{[220+270K]}^2\right)\sum _{k=1}^K{\Vert z_k\Vert }_2^2+4\mu C{K}^{-2}{\eta }_{\text{min}}^{*-\frac{4}{3}}{R}^{\frac{8}{3}}\frac{{\sigma }^{2}s{(\text{log }n)}^3}{n},$$ (A.61)

with probability at least 1 − (18K² + 4K + 4Ks)/n. ■

2). Bounding thresholding effect:

The thresholding effect term in (A.44) can also be decomposed into optimization error and statistical error. Recall that B can be explicitly written as

$$B=\sum _{k=1}^K{\Vert \mu \frac{{\eta }_k^{*\frac{2}{3}}}{\varphi }\frac{4\sqrt{\text{log}(np)}}{n}*\sqrt{\sum _{i=1}^n{\left(\sum _{k^{\prime }=1}^K{\left(x_i^{\top }{\overline{\beta }}_{k^{\prime }}\right)}^3-y_i\right)}^2{\left(x_i^{\top }{\overline{\beta }}_k\right)}^4{\gamma }_k}\Vert }_2^2,$$

where supp(γ_k) ⊂ F_k and ‖γ_k‖_∞ ≤ 1. By using (a + b)² ≤ 2(a² + b²), we have

$$B\le {\mu }^2\frac{64Ks\text{ log }p}{n}\left(B_1+B_2\right),$$

where

$$\begin{array}{l}{B}_1=\frac{1}{n}\sum _{i=1}^n\left(\sum _{k^{\prime }=1}^K{\left(x_i^{\top }{\overline{\beta }}_{k^{\prime }}\right)}^3-\sum _{k^{\prime }=1}^K{\left(x_i^{\top }{\overline{\beta }}_{k^{\prime }}^{*}\right)}^3\right)\left(\sum _{k=1}^K\frac{{\eta }_k^{*\frac{4}{3}}}{{\varphi }^2}{\left(x_i^{\top }{\overline{\beta }}_k\right)}^4\right)\hfill \\ B_2=\frac{1}{n}\sum _{i=1}^n{ϵ}_i^2\sum _{k=1}^K\frac{{\eta }_k^{*\frac{4}{3}}}{{\varphi }^2}{\left(x_i^{\top }{\overline{\beta }}_k\right)}^4.\hfill \end{array}$$

Bounding B₁. This optimization error term shares similar structure with (A.57) but with higher order. Therefore, we follow the same idea as we did in bounding (A.57). Following by (A.46) and some basic expansions and inequalities,

$$B_1\le K^{-2}{R}^{\frac{4}{3}}{\eta }_{\text{min}}^{*-\frac{8}{3}}\frac{1}{n}\left(\sum _{k^{\prime }=1}^K{\left(x_i^{\top }{\overline{\beta }}_{k^{\prime }}\right)}^3-\sum _{k^{\prime }=1}^K{\left(x_i^{\top }{\overline{\beta }}_{k^{\prime }}^{*}\right)}^3\right)\left(\sum _{k=1}^K{\left(x_i^{\top }{\overline{\beta }}_k\right)}^4\right)\le K^{-2}{R}^{\frac{4}{3}}{\eta }_{\text{min}}^{*-\frac{8}{3}}\left[\frac{1}{n}\sum _{i=1}^n\left(\sum _{k=1}^{K}3K{\left(x_i^{\top }{z}_k\right)}^6+9K{\left(x_i^{\top }{z}_k\right)}^4{\left(x_i^{\top }{\overline{\beta }}_k^{*}\right)}^2+9K{\left(x_i^{\top }{z}_k\right)}^2{\left(x_i^{\top }{\overline{\beta }}_k^{*}\right)}^4\right)\sum _{k^{\prime }=1}^K{\left(x_i^{\top }{\overline{\beta }}_{k^{\prime }}\right)}^4\right].$$

The main term is ${\left(x_i^{\top }{z}_k\right)}^2{\left(x_i^{\top }{\overline{\beta }}_k^{*}\right)}^4$ according to the order of ${\overline{\beta }}_k^{*}$. We bound the main term first. Note that there exists some positive large constant C such that

$$\mathbb{E}\left(\frac{1}{n}{\sum }_{i=1}^n{\left(x_i^{\top }{z}_k\right)}^2{\left(x_i^{\top }{\overline{\beta }}_{k^{\prime }}^{*}\right)}^4{\left(x_i^{\top }{\overline{\beta }}_{k^{\prime }}\right)}^4\right)\le C{\Vert z_k\Vert }_2^2{\Vert {\overline{\beta }}_k^{*}\Vert }_2^4{\Vert {\overline{\beta }}_{k^{\prime }}\Vert }_2^4.$$

Together with Lemma 1 and (A.13), we have

$$\sum _{k=1}^K\sum _{k^{\prime }=1}^K\left(\frac{1}{n}\sum _{i=1}^n{\left(x_i^{\top }{z}_k\right)}^2{\left(x_i^{\top }{\overline{\beta }}_{k^{\prime }}^{*}\right)}^4{\left(x_i^{\top }{\overline{\beta }}_{k^{\prime }}\right)}^4\right)\le C\left(1+\frac{{(\text{log }n)}^5}{\sqrt{n}}\right)K^2{\eta }_{\text{max}}^{*\frac{8}{3}}{\left(1+K^{\frac{1}{2}}{\epsilon }_0\right)}^4\sum _{k=1}^K{\Vert z_k\Vert }_2^2.$$

with probability at least 1 − 3K²/n. Overall, the upper bound of B₁ takes the form

$$B_1\le K^{-2}{R}^{\frac{4}{3}}{\eta }_{\text{min}}^{*-\frac{8}{3}}\left[18C\left(1+\frac{{(\text{log }n)}^5}{\sqrt{n}}\right)*K^2{\eta }_{\text{min}}^{*-\frac{8}{3}}{\left(1+K^{\frac{1}{2}}{\epsilon }_0\right)}^4\sum _{k=1}^K{\Vert z_k\Vert }_2^2\right]\le R^{4}18C\left(1+\frac{{(\text{log }n)}^5}{\sqrt{n}}\right){\left(1+K^{\frac{1}{2}}{\epsilon }_0\right)}^4\sum _{k=1}^K{\Vert z_k\Vert }_2^2,$$ (A.62)

with probability at least 1 − 3K²/n.

Bounding B₂. We rewrite B₂ by

$$B_2=\sum _{k=1}^K\frac{{\eta }_k^{*\frac{4}{3}}}{{\varphi }^2}\left(\frac{1}{n}\sum _{i=1}^n{ϵ}_i^2{\left(x_i^{\top }{\overline{\beta }}_k\right)}^4\right).$$

For fixed ${\left\{{ϵ}_i\right\}}_{i=1}^n$, accordingly to Lemma 1, we have

$$\left|\sum _{i=1}^n{ϵ}_i^2{\left(x_i^{\top }{\overline{\beta }}_k\right)}^4-\mathbb{E}\left(\sum _{i=1}^n{ϵ}_i^2{\left(x_i^{\top }{\overline{\beta }}_k\right)}^4\right)\right|\le C{(\text{log }n)}^2{\Vert {\mathit{ϵ}}^2\Vert }_2{\Vert {\overline{\beta }}_k\Vert }_2^4.$$

Note that $\mathbb{E}\left({\left(x_i^{\top }{\overline{\beta }}_k\right)}^4\right)=3{\Vert {\overline{\beta }}_k\Vert }_2^4$. It will reduce to

$$\frac{1}{n}\sum _{i=1}^n{ϵ}_i^2{\left(x_i^{\top }{\overline{\beta }}_k\right)}^4\le \left(\frac{3}{n}\sum _{i=1}^n{ϵ}_i^2+C\frac{{(\text{log }n)}^2}{n}{\Vert {\mathit{ϵ}}^2\Vert }_2\right){\Vert {\overline{\beta }}_k\Vert }_2^4.$$

From Lemma 23, with probability at least 1 − 3/n,

$$\left|\frac{1}{n}\sum _{i=1}^n{ϵ}_i^2\right|\le C_0{\sigma }^2,\frac{1}{n}{\Vert {\mathit{ϵ}}^2\Vert }_2\le C_0\frac{{\sigma }^2}{\sqrt{n}}.$$

Combining the above two inequalities, we obtain

$$\left|\frac{1}{n}\sum _{i=1}^n{ϵ}_i^2{\left(x_i^{\top }{\overline{\beta }}_k\right)}^4\right|\le 6C_0{\sigma }^2{\Vert {\overline{\beta }}_k\Vert }_2^4,$$ (A.63)

with probability at least. Plugging in the definition of ϕ and (A.13), B₂ is upper bounded by

$$B_2\le 6C_0{\sigma }^2{\left(1+K^{\frac{1}{2}}{\epsilon }_0\right)}^4{\eta }_{\text{min}}^{*-\frac{4}{3}}{R}^{\frac{8}{3}}{K}^{-3},$$ (A.64)

with probability at least 1 − 7K/n.

Summary. Putting the bounds (A.62) and (A.64) together, we have similar upper bound for thresholded effect,

$$B\le C_2{\mu }^2{R}^4\sum _{k=1}^K{\Vert z_k\Vert }_2^2+C_3{\mu }^2{\eta }_{\text{min}}^{*-\frac{4}{3}}{R}^{\frac{8}{3}}{K}^{-2}\frac{{\sigma }^{2}s\text{ log }p}{n},$$ (A.65)

with probability at least 1 − (3K² + 7K)/n. ■

3). Ensemble:

From the definition of γ_k, it’s not hard to see actually the cross term C is equal to zero. Combining the upper bound of gradient update effect (A.61) and thresholding effect (A.65) together, we obtain

$$\sum _{k=1}^K{\Vert \sqrt[3]{{\eta }_k}{\tilde{\beta }}_k^{+}-\sqrt[3]{{\eta }_k^{*}}{\beta }_k^{*}\Vert }_2^2\le \left(1-64\mu K^{-2}{R}^{-\frac{8}{3}}+3{\mu }^2{K}^{-1}{R}^4{[220+270K]}^2\right)\left(\sum _{k=1}^K{\Vert z_k\Vert }_2^2\right)++2C_3{\mu }^2{R}^{\frac{8}{3}}{\eta }_{\text{min}}^{*-\frac{4}{3}}\frac{{\sigma }^2{K}^{-2}s\text{ log }p}{n}.$$

As long as the step size μ satisfies

$$0<\mu \le \frac{32R^{-20/3}}{3K{[220+270K]}^2},$$

we reach the conclusion

$$\sum _{k=1}^K{\Vert \sqrt[3]{{\eta }_k}{\tilde{\beta }}_k^{+}-\sqrt[3]{{\eta }_k^{*}}{\tilde{\beta }}_k^{*}\Vert }_2^2\le \left(1-32\mu K^{-2}{R}^{-\frac{8}{3}}\right)\sum _{k=1}^K{\Vert \sqrt[3]{{\eta }_k}{\beta }_k-\sqrt[3]{{\eta }_k^{*}}{\beta }_k^{*}\Vert }_2^2+2C_3{\mu }^2{R}^{-\frac{8}{3}}{\eta }_{\text{min}}^{*-\frac{4}{3}}\frac{{\sigma }^2{K}^{-2}s\text{ log }p}{n},$$ (A.66)

with probability at least 1 − 4Ks/n.

M. Proof of Lemma 14

Let us consider k-th component first. Without loss of generality, suppose F ⊂ {1,2,…,Ks}. For j = Ks + 1,…,p,

$$\frac{\partial }{\partial {\beta }_{kj}}\mathcal{L}\left({\beta }_k\right)=\frac{2}{n}\sum _{i=1}^n\left(\sum _{k=1}^K{\eta }_k{\left(x_i^{\top }{\beta }_k\right)}^3-y_i\right){\eta }_k{\left(x_i^{\top }{\beta }_k\right)}^2{x}_{ij},$$ (A.67)

and it’s not hard to see the independence between $\left\{x_i^{\top }{\beta }_k,y_i\right\}$ and x_ij. Applying standard Hoeffding’s inequality, we have with probability at least $1-\frac{1}{n^2{p}^2}$,

$$\left|\frac{\partial }{\partial {\beta }_{kj}}\mathcal{L}\left({\beta }_k\right)\right|\le \frac{\sqrt{4\text{ log}(np)}}{n}*\sqrt{\sum _{i=1}^n{\left(\sum _{k=1}^K{\eta }_k{\left(x_i^{\top }{\beta }_k\right)}^3-y_i\right)}^2{\left({\eta }_k\left(x_i^{\top }{\beta }_k\right)\right)}^2}=h\left({\beta }_k\right).$$

Equipped with union bound, with probability at least $1-\frac{1}{n^{2}p}$,

$$\underset{Ks+1\le j\le p}{\text{max}}\left|\frac{\partial }{\partial {\beta }_{kj}}\mathcal{L}\left({\beta }_k\right)\right|\le h\left({\beta }_k\right).$$

Therefore, according to the definition of thresholding function φ(x), we obtain the following equivalence,

$${\phi }_{\frac{\mu }{\varphi }h\left({\beta }_k\right)}\left({\beta }_k-\frac{\mu }{\varphi }{\nabla }_{{\beta }_k}\mathcal{L}\left({\beta }_k\right)\right)={\phi }_{\frac{\mu }{\varphi }h\left({\beta }_k\right)}\left({\beta }_k-\frac{\mu }{\varphi }{\nabla }_{{\beta }_k}\mathcal{L}{\left({\beta }_k\right)}_F\right),$$ (A.68)

holds for k ∈ [K], with probability at least $1-\frac{1}{n^{2}p}$. (A.68) also provides that $\text{supp}\left({\beta }_k^{+}\right)\subset F$ for every k ∈ [K], which further implies F⁺ ⊂ F. Now we end the proof. ■

N. Proof of Lemma 15

First, we consider symmetric case. According to the definition of ${\left\{y_i\right\}}_{i=1}^n$ from symmetric tensor estimation model (III.1), we separate the random noise ϵ_i by the following expansion,

$$\frac{1}{n}\sum _{i=1}^n{y}_i^2=\frac{1}{n}\sum _{i=1}^n{\left[\sum _{k=1}^K{\eta }_k^{*}{\left(x_i^{\top }{\beta }_k^{*}\right)}^3+{ϵ}_i\right]}^2=\underset{I_1}{\underbrace{\frac{1}{n}\sum _{i=1}^n{\left(\sum _{k=1}^K{\eta }_k^{*}{\left(x_i^{\top }{\beta }_k^{*}\right)}^3\right)}^2}}+\underset{I_2}{\underbrace{\frac{2}{n}\sum _{i=1}^n{ϵ}_i\sum _{k=1}^K{\eta }_k^{*}{\left(x_i^{\top }{\beta }_k^{*}\right)}^3}}+\underset{I_3}{\underbrace{\frac{1}{n}\sum _{i=1}^n{ϵ}_i^2}}.$$ (A.69)

Bounding I₁. We expand i-th component of I₁ as follows.

$${\left(\sum _{k=1}^K{\eta }_k^{*}{\left(x_i^{\top }{\beta }_k^{*}\right)}^3\right)}^2=\sum _{k=1}^K{\eta }_k^{*}{\left(x_i^{\top }{\beta }_k^{*}\right)}^6+2\sum _{k_i<k_j}{\eta }_{k_i}^{*}{\eta }_{k_j}^{*}{\left(x_i^{\top }{\beta }_{k_i}^{*}\right)}^3{\left(x_i^{\top }{\beta }_{k_j}^{*}\right)}^3.$$ (A.70)

As shown in Corollary A.2, the expectations of above two parts takes forms of

$$\mathbb{E}(x_i^{\top }{\beta }_k^{*}=6{\left({\beta }_{k_i}^{*\top }{\beta }_{k_j}^{*}\right)}^3+9\left({\beta }_{k_i}^{*\top }{\beta }_{k_j}^{*}\right){\Vert {\beta }_{k_i}^{*}\Vert }_2^2{\Vert {\beta }_{k_j}^{*}\Vert }_2^2\mathbb{E}{\left(x_i^{\top }{\beta }_k^{*}\right)}^6=15{\Vert {\beta }_k^{*}\Vert }_2^2.$$

Recall that ${\Vert {\beta }_k^{*}\Vert }_2=1$ for any k ∈ [K] and Condition 3 implies for any k_i ≠ k_j, $|{\beta }_{k_i}^{*\top }{\beta }_{k_j}^{*}|\le \Gamma$, where Γ is the incoherence parameter. Thus, $\mathbb{E}{\left(x_i^{\top }{\beta }_{k_i}^{*}\right)}^3{\left(x_i^{\top }{\beta }_{k_j}^{*}\right)}^3$ is upper bounded by

$$|\mathbb{E}{\left(x_i^{\top }{\beta }_{k_i}^{*}\right)}^3{\left(x_i^{\top }{\beta }_{k_j}^{*}\right)}^3|\le 6{\Gamma }^3+9\Gamma ,\text{ for any }{k}_i\ne k_j.$$ (A.71)

By using the concentration result in Lemma 1, we have with probability at least 1 − 1/n

$$|\frac{1}{n}\sum _{i=1}^n{\left(x_i^{\top }{\beta }_k^{*}\right)}^6-\mathbb{E}\left(\frac{1}{n}\sum _{i=1}^n{\left(x_i^{\top }{\beta }_k^{*}\right)}^6\right)|\le C_1\frac{{(\text{log }n)}^3}{\sqrt{n}},|\frac{1}{n}\sum _{i=1}^n{\left(x_i^{\top }{\beta }_{k_i}^{*}\right)}^3{\left(x_i^{\top }{\beta }_{k_j}^{*}\right)}^3-\mathbb{E}\left(\frac{1}{n}\sum _{i=1}^n{\left(x_i^{\top }{\beta }_{k_i}^{*}\right)}^3{\left(x_i^{\top }{\beta }_{k_j}^{*}\right)}^3\right)|\le C_1\frac{{(\text{log }n)}^3}{\sqrt{n}}.$$ (A.72)

Putting (A.70),(A.71) and (A.72) together, this essentially provides an upper bound for I₁, namely

$$\frac{1}{n}\sum _{i=1}^n{\left(\sum _{k=1}^K{\eta }_k^{*}{\left(x_i^{\top }{\beta }_k^{*}\right)}^3\right)}^2\le \left(15+6{\Gamma }^3+9\Gamma +2C_1\frac{{(\text{log }n)}^3}{\sqrt{n}}\right){\left(\sum _{k=1}^K{\eta }_k^{*}\right)}^2,$$ (A.73)

with probability at least 1 − K²/n.

Bounding I₂. Since the random noise ${\left\{{ϵ}_i\right\}}_{i=1}^n$ is of mean zero and independent of {x_i}, we have

$$\mathbb{E}\left({ϵ}_i\sum _{k=1}^K{\eta }_k^{*}{\left(x_i^{\top }{\beta }_k^{*}\right)}^3\right)=0.$$

By using the independence and Corollary 1, we have

$$\mathbb{P}\left(\frac{1}{n}\sum _{i=1}^n{ϵ}_i{\left(x_i^{\top }{\beta }_k^{*}\right)}^3\ge C_2\frac{{(\text{log }n)}^{\frac{3}{2}}}{n}\sqrt{n}\sigma \right)\le \mathbb{P}\left(\frac{1}{n}\sum _{i=1}^n{ϵ}_i{\left(x_i^{\top }{\beta }_k^{*}\right)}^3\ge C_2\frac{\sigma {(\text{log }n)}^{\frac{3}{2}}}{\sqrt{n}}|\Vert \mathit{ϵ}{\Vert }_2\le C_0\sigma \sqrt{n}\right)+\mathbb{P}\left(\Vert \mathit{ϵ}{\Vert }_2\ge C_0\sqrt{n}\sigma \right)\le \frac{1}{n}+\frac{3}{n}=\frac{4}{n}.$$

This further implies that

$$\frac{1}{n}\sum _{i=1}^n\sum _{k=1}^K{\eta }_k^{*}{\left(x_i^{\top }{\beta }_k^{*}\right)}^3{ϵ}_i\le \left(\sum _{k=1}^K{\eta }_k^{*}\right)C_2\frac{{(\text{log }n)}^{\frac{3}{2}}}{\sqrt{n}}\sigma ,$$ (A.74)

with probability at least 1 − 4K/n.

Bounding I₃. As shown in Lemma 23, the random noise ϵ_i with sub-exponential tail satisfies

$$\frac{1}{n}\sum _{i=1}^n{ϵ}_i^2\le C_3{\sigma }^2.$$ (A.75)

with probability at least 1 − 3/n.

Overall, putting (A.73), (A.74) and (A.75) together, we have with probability at least 1 − (K² + 4K + 3)/n,

$$\frac{\frac{1}{n}{\sum }_{i=1}^n{y}_i^2}{{\left({\sum }_{k=1}^K{\eta }_k^{*}\right)}^2}\le 15+6{\Gamma }^3+9\Gamma +2C_1\frac{{(\text{log }n)}^3}{\sqrt{n}}+\frac{2C_2\sigma }{\left({\sum }_{k=1}^K{\eta }_k^{*}\right)}\frac{{(\text{log }n)}^{\frac{3}{2}}}{\sqrt{n}}+\frac{C_3{\sigma }^2}{{\left({\sum }_{k=1}^K{\eta }_k^{*}\right)}^2}.$$

Under Conditions 4 & 5, the above bound reduces to

$$\frac{1}{n}\sum _{i=1}^n{y}_i^2\le \left(16+6{\Gamma }^3+9\Gamma \right){\left(\sum _{k=1}^K{\eta }_k^{*}\right)}^2$$

with probability at least 1 − (K² + 4K + 3)/n. The proof of lower bound is similar, and hence is omitted here.

Similar results will also hold for non-symmetric tensor estimation model. Throughout the proof, the only difference is that

$$\mathbb{E}{\left(u_i^{\top }{\beta }_{1k}^{*}\right)}^2{\left(v_i^{\top }{\beta }_{2k}^{*}\right)}^2{\left(w_i^{\top }{\beta }_{3k}^{*}\right)}^2=1.$$

O. Non-symmetric Tensor Estimation

1). Conditions and Algorithm:

In this subsection, we provide several essential conditions for Theorem 5 and the detail algorithm for non-symmetric tensor estimation.

Condition 6 (Uniqueness of CP-decomposition). The CP-decomposition form (VI.2) is unique in the sense that if there exists another CP-decomposition $\mathcal{J}*={\sum }_{k=1}^{K^{\prime }}{\eta }_k^{{*}^{\prime }}{\beta }_{1k}^{{*}^{\prime }}\circ {\beta }_{2k}^{{*}^{\prime }}\circ {\beta }_{3k}^{{*}^{\prime }}$, it must have K = K′ and be invariant up to a permutation of {1,…,K}.

Condition 7 (Parameter space). The CP-decomposition of $\mathcal{J}*={\sum }_{k=1}^K{\eta }_k^{*}{\beta }_{1k}^{*}\circ {\beta }_{2k}^{*}\circ {\beta }_{3k}^{*}$ satisfies

$${\Vert \mathcal{J}*\Vert }_{op}\le C_1{\eta }_{\text{max}}^{*},K=\mathcal{O}(s),\text{ and }R={\eta }_{\text{max}}^{*}/{\eta }_{\text{min}}^{*}\le C_2$$

for some absolute constants C₁,C₂.

Condition 8 (Parameter incoherence). The true tensor components are incoherent such that

$$\Gamma :=\underset{k_i\ne k_j}{\text{max}}\left\{|{\beta }_{1k_i}^{*},{\beta }_{1k_j}^{*}|,|{\beta }_{2k_i}^{*},{\beta }_{2k_j}^{*}|,|{\beta }_{3k_i}^{*},{\beta }_{3k_j}^{*}|\right\}\le C\text{min}\left\{K^{-\frac{3}{4}}{R}^{-1},s^{-\frac{1}{2}}\right\}.$$

Condition 9 (Random noise). We assume the random noise ${\left\{{ϵ}_i\right\}}_{i=1}^n$ follows a sub-exponential tail with parameter σ satisfying $0<\sigma <C{\sum }_{k=1}^K{\eta }_k^{*}$.

2). Proof of Theorem 5:

The main distinguished part of the proof for non-symmetric update is Lemma 16: one-step oracle estimator, which is parallel to Lemma 11. For the sake of completeness, we limit our attention to rank-one case and only provide the theoretical development for one-step oracle estimator in this subsection. The generalization to general rank case follows the exact same idea in the proof of symmetric update by incorporating the incoherence condition (8).

For rank-one non-symmetric tensor estimation, the model (VI.1) reduces to

$$y_i={\eta }^{*}{\beta }_1^{*}\circ {\beta }_2^{*}\circ {\beta }_3^{*},u_i\circ v_i\circ w_i+{ϵ}_i,\text{ for }i=1,\dots ,n.$$

Suppose $|\text{supp}\left({\beta }_1^{*}\right)|=s_1$, $|\text{supp}\left({\beta }_2^{*}\right)|=s_2$, $|\text{supp}\left({\beta }_3^{*}\right)|=s_3$ and denote s = max{s₁,s₂,s₃}. Define $F_j^{(t)}=\text{supp}\left({\beta }_j^{*}\right)\cup \text{supp}\left({\beta }_j^{(t)}\right)$, $F^{(t)}={\cup }_{j=1}^3{F}_j^{(t)}$ and the oracle estimator as

$${\tilde{\beta }}_1^{(t+1)}={\phi }_{\frac{\mu }{\varphi }h\left({\beta }_1^{(t)}\right)}\left({\beta }_j^{(t)}-\frac{\mu }{\varphi }{\nabla }_1\mathcal{L}{\left({\beta }_1^{(t)},{\beta }_2^{(t)},{\beta }_3^{(t)}\right)}_{F^{(t)}}\right),$$

where $h\left({\beta }_1^{(t)}\right)$ has the form of

$$\frac{\sqrt{4\text{log }np}}{n}{\left(\sum _{i=1}^n{\left(\eta \left(u_i^{\top }{\beta }_1^{(t)}\right)\left(v_i^{\top }{\beta }_2^{(t)}\right)\left(w_i^{\top }{\beta }_3^{(t)}\right)-y_i\right)}^2*{\eta }^{\frac{2}{3}}{\left(v_i^{\top }{\beta }_2^{(t)}\right)}^2{\left(w_i^{\top }{\beta }_3^{(t)}\right)}^2\right)}^{1/2},$$ (A.76)

The definitions of ${\tilde{\beta }}_2^{(t+1)}$ and ${\tilde{\beta }}_3^{(t+1)}$ are similar.

Lemma 16. Let t ≥ 0 be an integer. Suppose Conditions 6–9 hold and $\left\{{\beta }_j^{(t)},\eta \right\}$ satisfies the following upper bound

$$\underset{j=1,2,3}{\text{max}}{\Vert \sqrt[3]{\eta }{\beta }_j^{(t)}-\sqrt[3]{{\eta }^{*}}{\beta }_j^{*}\Vert }_2\le \sqrt[3]{{\eta }^{*}}{\epsilon }_0,|\eta -\eta *|\le {\epsilon }_0$$ (A.77)

with probability at least 1 − CO(1/n). Assume the step size μ satisfies 0 < μ < μ₀ for some small absolute constant μ₀ and s ≤ d ≤ Cs. Then $\left\{{\tilde{\beta }}_j^{(t+1)}\right\}$ can be upper bounded as

$$\underset{j=1,2,3}{\text{max}}{\Vert \sqrt[3]{\eta }{\tilde{\beta }}_j^{(t+1)}-\sqrt[3]{\eta *}{\beta }_j^{*}\Vert }_2\le \left(1-\frac{\mu }{12}\right)\underset{j=1,2,3}{\text{max}}{\Vert \sqrt[3]{\eta }{\beta }_j^{(t)}-\sqrt[3]{\eta *}{\beta }_j^{*}\Vert }_2+\mu \frac{3\sigma }{{\left(\sqrt[3]{\eta *}\right)}^2}\sqrt{\frac{3s\text{ log }p}{n}},$$

with probability at least 1 − 12s/n.

Proof. We focus on j = 1 first. To simplify the notation, we drop the superscript of iteration index t, and denote iteration index t+1 by +. Moreover, denote ${\overline{\beta }}_j=\sqrt[3]{\eta }{\beta }_j,{\overline{\beta }}_j^{+}=\sqrt[3]{\eta }{\beta }_j$, ${\overline{\beta }}_j^{*}=\sqrt[3]{\eta *}{\beta }_j^{*}$ for j = 1, 2, 3. Then, the gradient function is rewritten as

$${\nabla }_1\mathcal{L}\left({\overline{\beta }}_1,{\overline{\beta }}_2,{\overline{\beta }}_3\right)=\sqrt[3]{\eta }\frac{2}{n}\sum _{i=1}^n\left(\left(u_i^{\top }{\overline{\beta }}_1\right)\left(v_i^{\top }{\overline{\beta }}_2\right)\left(w_i^{\top }{\overline{\beta }}_3\right)\right)\left(v_i^{\top }{\overline{\beta }}_2\right)\left(w_i^{\top }{\overline{\beta }}_3\right)u_i.$$

According to the definition of thresholded function, ${\tilde{\beta }}_1^{+}$ can be explicitly written by

$${\tilde{\beta }}_1^{+}={\phi }_{\frac{\mu }{\varphi }\left({\overline{\beta }}_1\right)}\left({\beta }_1-\frac{\mu }{\varphi }{\nabla }_1\mathcal{L}{\left({\overline{\beta }}_1,{\overline{\beta }}_2,{\overline{\beta }}_3\right)}_F\right)={\beta }_1-\frac{\mu }{\varphi }{\nabla }_1\mathcal{L}{\left({\overline{\beta }}_1,{\overline{\beta }}_2,{\overline{\beta }}_3\right)}_F+\frac{\mu }{\varphi }h\left({\overline{\beta }}_1\right)\gamma ,$$

where $\gamma \in {ℝ}^p$, supp(γ) ⊂−F and ‖γ‖_∞ ≤ 1. Then the oracle estimation error ${\Vert \sqrt[3]{\eta }{\tilde{\beta }}_1^{+}-\sqrt[3]{\eta *}{\beta }_1^{*}\Vert }_2$ can be decomposed by the gradient update effect and the thresholded effect,

$${\Vert \sqrt[3]{\eta }{\tilde{\beta }}_1^{+}-\sqrt[3]{\eta *}{\beta }_1^{*}\Vert }_2=\underset{\text{gradient update effect }}{\underbrace{{\Vert {\overline{\beta }}_1-{\overline{\beta }}_1^{*}-\mu \frac{\sqrt[3]{\eta }}{\varphi }{\nabla }_1\mathcal{L}{\left({\overline{\beta }}_1,{\overline{\beta }}_2,{\overline{\beta }}_3\right)}_F\Vert }_2}}+\underset{\text{thresholded effect }}{\underbrace{\mu \frac{\sqrt[3]{\eta }}{\varphi }|h\left({\overline{\beta }}_1\right)|\sqrt{3s}}}.$$ (A.78)

By using the tri-convex structure of $\mathcal{L}\left({\overline{\beta }}_1,{\overline{\beta }}_2,{\overline{\beta }}_3\right)$, we borrow the analysis tool for vanilla gradient descent [55] given sufficient good initial. Following this proof strategy, we decompose the gradient update effect in (A.78) by three parts,

$${\Vert \sqrt[3]{\eta }{\tilde{\beta }}_1^{+}-\sqrt[3]{\eta *}{\beta }_1^{*}\Vert }_2\le \underset{I_1}{\underbrace{{\Vert {\overline{\beta }}_1-{\overline{\beta }}_1^{*}-\mu \frac{\sqrt[3]{\eta }}{\varphi }{\nabla }_1\tilde{\mathcal{L}}{\left({\overline{\beta }}_1,{\overline{\beta }}_2^{*},{\overline{\beta }}_3^{*}\right)}_F\Vert }_2}}+\underset{I_2}{\underbrace{\mu \frac{\sqrt[3]{\eta }}{\varphi }{\Vert {\nabla }_1\tilde{\mathcal{L}}{\left({\overline{\beta }}_1,{\overline{\beta }}_2^{*},{\overline{\beta }}_3^{*}\right)}_F-{\nabla }_1\tilde{\mathcal{L}}{\left({\overline{\beta }}_1,{\overline{\beta }}_2,{\overline{\beta }}_3\right)}_F\Vert }_2}}+\underset{I_3}{\underbrace{\mu \frac{\sqrt[3]{\eta }}{\varphi }{\Vert {\nabla }_1\tilde{\mathcal{L}}{\left({\overline{\beta }}_1,{\overline{\beta }}_2,{\overline{\beta }}_3\right)}_F-{\nabla }_1\mathcal{L}{\left({\overline{\beta }}_1,{\overline{\beta }}_2,{\overline{\beta }}_3\right)}_F\Vert }_2}}+\underset{I_4}{\underbrace{\mu \frac{\sqrt[3]{\eta }}{\varphi }|h\left({\overline{\beta }}_1\right)|\sqrt{3s},}}$$

where ${\nabla }_1\tilde{\mathcal{L}}$ is the noiseless gradient as we defined in (A.42). We will bound I₁, I₂, I₃, I₄ successively in the following four subsections. For simplicity, during the following proof, we drop the index subscript F as we did in Section L. And $\varphi ={\sum }_{i=1}^n{y}_i^2$ approximates η*² up to constant due to Lemma 15.

3). Bounding I₁:

In this section, let us denote

$$\begin{array}{l}\sqrt[3]{\eta }\tilde{\mathcal{L}}\left({\overline{\beta }}_1,{\overline{\beta }}_2^{*},{\overline{\beta }}_3^{*}\right)/\varphi =f\left({\overline{\beta }}_1\right)\hfill \\ \sqrt[3]{\eta }{\nabla }_1\tilde{\mathcal{L}}\left({\overline{\beta }}_1,{\overline{\beta }}_2^{*},{\overline{\beta }}_3^{*}\right)/\varphi =\nabla f\left({\overline{\beta }}_1\right),\hfill \end{array}$$ (A.79)

Where $\text{supp}\left(\nabla f\left({\overline{\beta }}_1\right)\right)=F$. When β₂ and β₃ are fixed, the update can be treated as a vanilla gradient descent update. The following proof follows three steps. The first two steps show that $f\left({\overline{\beta }}_1\right)$ is Lipshitz differentiable and strongly convex on the constraint set F, and the last step utilizes the classical convex gradient analysis.

Step One:

Verify $f\left({\overline{\beta }}_1\right)$ is L-Lipschitz differentiable. For any ${\overline{\beta }}_1^{(1)}$ and ${\overline{\beta }}_1^{(2)}$ whose support belong to F,

$$\nabla f\left({\overline{\beta }}_1^{(1)}\right)-\nabla f\left({\overline{\beta }}_1^{(2)}\right)=\frac{{(\sqrt[3]{\eta })}^2}{\varphi }\frac{2}{n}\sum _{i=1}^n\left(u_i^{\top }\left({\overline{\beta }}_1^{(1)}-{\overline{\beta }}_1^{(2)}\right){\left(v_i^{\top }{\overline{\beta }}_2^{*}\right)}^2{\left(w_i^{\top }{\overline{\beta }}_3^{*}\right)}^2\right)u_i.$$

Then, there exist $\pi \in {\mathbb{S}}^{s-1}$ such that

$${\Vert \nabla f\left({\overline{\beta }}_1^{(1)}\right)-\nabla f\left({\overline{\beta }}_1^{(2)}\right)\Vert }_2=\frac{{(\sqrt[3]{\eta })}^2}{\varphi }|\frac{1}{n}\sum _{i=1}^n\left(u_i^{\top }\left({\overline{\beta }}_1^{(1)}-{\overline{\beta }}_1^{(2)}\right){\left(v_i^{\top }{\overline{\beta }}_2^{*}\right)}^2{\left(w_i^{\top }{\overline{\beta }}_3^{*}\right)}^2\right)u_i^{\top }\pi |.$$

Applying Lemma 2 with multiplying $\left({\overline{\beta }}_1^{(1)}-{\overline{\beta }}_1^{(2)}\right)\circ {\overline{\beta }}_2^{*}\circ {\overline{\beta }}_3^{*}$ it shows

$$|\sum _{i=1}^n\left[\left(u_i^{\top }\left({\overline{\beta }}_1^{(1)}-{\overline{\beta }}_1^{(2)}\right)\left(u_i^{\top }\pi \right){\left(v_i^{\top }{\overline{\beta }}_2^{*}\right)}^2{\left(w_i^{\top }{\overline{\beta }}_3^{*}\right)}^2\right)\right]|\le \left(1+{\delta }_{n,p,s}\right){\Vert {\overline{\beta }}_1^{(1)}-{\overline{\beta }}_1^{(2)}\Vert }_2{\eta }^{*\frac{4}{3}},$$

with probability at least 1 − 10/n³, where δ_n,p,s is defined in (IV.7). Under Condition (5) with some constant adjustments, we obtain

$${\Vert \nabla f\left({\overline{\beta }}_1^{(1)}\right)-\nabla f\left({\overline{\beta }}_1^{(2)}\right)\Vert }_2\le \frac{57}{16}{\Vert {\overline{\beta }}_1^{(1)}-{\overline{\beta }}_1^{(2)}\Vert }_2.$$ (A.80)

with probability at least 1 − 10/n³. Therefore, $f\left({\overline{\beta }}_1\right)$ is Lipschitz differentiable with Lipschitz constant $L=\frac{57}{8}$.

Step Two:

Verify $f\left({\overline{\beta }}_1\right)$ is α-strongly convex. It is equivalent to prove that ${\nabla }^{2}f\left({\overline{\beta }}_1\right)\succcurlyeq m{\mathbb{I}}_p$. Based on the inequality (3.3.19) in [63], it shows that

$${\lambda }_{\text{min}}\left({\nabla }^2\left(f\left({\overline{\beta }}_1\right)\right)\right)\ge {\lambda }_{\text{min}}\left(\mathbb{E}\left({\nabla }^{2}f\left({\overline{\beta }}_1\right)\right)\right)-{\lambda }_{\text{max}}({\nabla }^{2}f\left({\overline{\beta }}_1\right)-\mathbb{E}\left({\nabla }^{2}f\left({\overline{\beta }}_1\right)\right).$$ (A.81)

The lower bound of ${\lambda }_{\text{min}}\left({\nabla }^2\left(f\left({\overline{\beta }}_1\right)\right)\right)$ breaks into two parts: an lower bound for ${\lambda }_{\text{min}}\left(\mathbb{E}\left({\nabla }^{2}f\left({\overline{\beta }}_1\right)\right)\right)$, and an upper bound for ${\lambda }_{\text{max}}({\nabla }^{2}f\left({\overline{\beta }}_1\right)-\mathbb{E}\left({\nabla }^{2}f\left({\overline{\beta }}_1\right)\right)$. The Hessian matrix of $f\left({\overline{\beta }}_1\right)$ is given by

$${\nabla }^{2}f\left({\overline{\beta }}_1\right)=\frac{{(\sqrt[3]{\eta })}^2}{\varphi }\frac{2}{n}\sum _{i=1}^n{\left(v_1^{\top }{\overline{\beta }}_2^{*}\right)}^2{\left(w_i^{\top }{\overline{\beta }}_3^{*}\right)}^2{u}_i{u}_i^{\top }.$$

Since u_i,v_i,w_i are independent with each other, we have $\mathbb{E}\left({\nabla }^{2}f\left({\overline{\beta }}_1\right)\right)=2\mathbb{I}$, which implies ${\lambda }_{\text{min}}\left(\mathbb{E}\left({\nabla }^{2}f\left({\overline{\beta }}_1\right)\right)\right)\ge 2$. On the other hand,

$${\lambda }_{\text{max}}\left({\nabla }^{2}f\left({\overline{\beta }}_1\right)-\mathbb{E}\left({\nabla }^{2}f\left({\overline{\beta }}_1\right)\right)\right)={\Vert {\nabla }^{2}f\left({\overline{\beta }}_1\right)-\mathbb{E}\left({\nabla }^{2}f\left({\overline{\beta }}_1\right)\right)\Vert }_2\le a^{\top }\left({\nabla }^{2}f\left({\overline{\beta }}_1\right)-\mathbb{E}\left({\nabla }^{2}f\left({\overline{\beta }}_1\right)\right)\right)b=\frac{2}{n}\sum _{i=1}^n{\left(v_i^{\top }{\overline{\beta }}_2^{*}\right)}^2{\left(w_i^{\top }{\overline{\beta }}_3^{*}\right)}^2\left(u_i^{\top }a\right)\left(u_i^{\top }b\right)-\mathbb{E}\left(\sum _{i=1}^n{\left(v_i^{\top }{\overline{\beta }}_2^{*}\right)}^2{\left(w_i^{\top }{\overline{\beta }}_3^{*}\right)}^2\left(u_i^{\top }a\right)\left(u_i^{\top }b\right)\right){\eta }^{*-\frac{4}{3}}.$$

where $a,b\in {\mathbb{S}}^{s-1}$. Equipped with Lemma 2, it yields that with probability at least 1 − 10/n³,

$${\lambda }_{\text{max}}\left({\nabla }^{2}f\left({\overline{\beta }}_1\right)-\mathbb{E}\left({\nabla }^{2}f\left({\overline{\beta }}_1\right)\right)\right)\le 2{\delta }_{n,s,p}.$$

Together with the lower bound of ${\lambda }_{\text{min}}\left(\mathbb{E}\left({\nabla }^{2}f\left({\overline{\beta }}_1\right)\right)\right)$, we have

$${\lambda }_{\text{min}}\left({\nabla }^{2}f\left({\overline{\beta }}_1\right)\right)\ge 2-2{\delta }_{n,p,s},$$

Under Condition 5, the minimum eigenvalue of Hessian matrix ${\nabla }^{2}f\left({\overline{\beta }}_1\right)$ is lower bounded by $\frac{19}{10}$ with probability at least 1−10/n³. This guarantees that $f\left({\overline{\beta }}_1\right)$ is strongly-convex with $\alpha =\frac{19}{10}$.

Step Three:

Combining the Lipschitz condition, strongly-convexity and Lemma 3.11 in [55], it shows that

$$\left(\nabla f\left({\overline{\beta }}_1\right)-\nabla f{\left({\overline{\beta }}_1^{*}\right)}^{\top }\right)\left({\overline{\beta }}_1-{\overline{\beta }}^{*}\right)\ge \frac{\alpha L}{\alpha +L}{\Vert {\overline{\beta }}_1-{\overline{\beta }}_1^{*}\Vert }_2^2+\frac{1}{\alpha +L}{\Vert \nabla f\left({\overline{\beta }}_1\right)-\nabla f\left({\overline{\beta }}_1^{*}\right)\Vert }_2^2.$$

Since the gradient vanishes at the optimal point, the above inequality times 2μ simplifies to

$$-2\mu \nabla f{\left({\overline{\beta }}_1\right)}^{\top }\left({\overline{\beta }}_1-{\overline{\beta }}_1^{*}\right)\le -\frac{2\mu \alpha L}{\alpha +L}{\Vert {\overline{\beta }}_1-{\overline{\beta }}_1^{*}\Vert }_2^2-\frac{2\mu }{\alpha +L}{\Vert \nabla f\left({\overline{\beta }}_1\right)\Vert }_2^2.$$ (A.82)

Now it’s sufficient to bound ${\Vert {\overline{\beta }}_1-{\overline{\beta }}_1^{*}-\mu \nabla f\left({\overline{\beta }}_1\right)\Vert }_2$ as follows

$${\Vert {\overline{\beta }}_1-{\overline{\beta }}_1^{*}-\mu \nabla f\left({\overline{\beta }}_1\right)\Vert }_2^2={\Vert {\overline{\beta }}_1^t-{\overline{\beta }}_1^{*}\Vert }_2^2+{\mu }^2{\Vert \nabla f\left({\overline{\beta }}_1\right)\Vert }_2^2-2\mu \nabla f{\left({\overline{\beta }}_1\right)}^{\top }\left({\overline{\beta }}_1-{\overline{\beta }}^{*}\right)\le \left(1-2\mu \frac{\alpha L}{\alpha +L}\right){\Vert {\overline{\beta }}_1-{\overline{\beta }}_1^{*}\Vert }_2^2+\mu \left(\mu -\frac{2}{\alpha +L}\right){\Vert \nabla f\left({\overline{\beta }}_1\right)\Vert }_2^2.$$

where L,α are Lipschitz constant and strongly convexity parameter, respectively. If $\mu <\frac{80}{361}$, the last term can be neglected and we obtain the desired upper bound,

$${\Vert {\overline{\beta }}_1-{\overline{\beta }}_1^{*}-\mu \frac{\sqrt[3]{\eta }}{\varphi }{\nabla }_1\tilde{\mathcal{L}}\left({\overline{\beta }}_1,{\overline{\beta }}_2^{*},{\overline{\beta }}_3^{*}\right)\Vert }_2\le (1-3\mu ){\Vert {\overline{\beta }}_1-{\overline{\beta }}_1^{*}\Vert }_2,$$ (A.83)

with probability 1 − 20/n³. This ends the proof. ■

4). Bounding I₂:

For simplicity, we write $z_1={\overline{\beta }}_1-{\overline{\beta }}_1^{*}$, $z_2={\overline{\beta }}_2-{\overline{\beta }}_2^{*}$, $z_3={\overline{\beta }}_3-{\overline{\beta }}_2^{*}$. By the definition of noiseless gradient, it suffices to decompose I₂ by

$${\eta }^{-\frac{1}{3}}{\Vert {\nabla }_1\tilde{\mathcal{L}}\left({\overline{\beta }}_1,{\overline{\beta }}_2^{*},{\overline{\beta }}_3^{*}\right)-{\nabla }_1\tilde{\mathcal{L}}\left({\overline{\beta }}_1,{\overline{\beta }}_2,{\overline{\beta }}_3\right)\Vert }_2\le {\Vert \frac{1}{n}\sum _{i=1}^n\left(u_i^{\top }{\overline{\beta }}_1\right)\left(v_i^{\top }{\overline{\beta }}_2\right)\left(v_i^{\top }{z}_2\right)\left(w_i^{\top }{\overline{\beta }}_3\right)\left(w_i^{\top }{z}_3\right)u_i\Vert }_2+{\Vert \frac{1}{n}\sum _{i=1}^n\left(u_i^{\top }{\overline{\beta }}_1\right)\left(v_i^{\top }{\overline{\beta }}_2\right)\left(v_i^{\top }{z}_2\right)\left(w_i^{\top }{\overline{\beta }}_3\right)\left(w_i^{\top }{\overline{\beta }}_3^{*}\right)u_i\Vert }_2+{\Vert \frac{1}{n}\sum _{i=1}^n\left(u_i^{\top }{\overline{\beta }}_1\right)\left(v_i^{\top }{\overline{\beta }}_2^{*}\right)\left(v_i^{\top }{\overline{\beta }}_2\right)\left(w_i^{\top }{\overline{\beta }}_3\right)\left(w_i^{\top }{z}_3\right)u_i\Vert }_2+{\Vert \frac{1}{n}\sum _{i=1}^n\left(u_i^{\top }{z}_1\right)\left(v_i^{\top }{\overline{\beta }}_2^{*}\right)\left(v_i^{\top }{z}_2\right)\left(w_i^{\top }{\overline{\beta }}_3^{*}\right)\left(w_i^{\top }{z}_3\right)u_i\Vert }_2+{\Vert \frac{1}{n}\sum _{i=1}^n\left(u_i^{\top }{z}_1\right)\left(v_i^{\top }{\overline{\beta }}_2^{*}\right)\left(v_i^{\top }{z}_2\right){\left(w_i^{\top }{\overline{\beta }}_3^{*}\right)}^2{u}_i\Vert }_2+{\Vert \frac{1}{n}\sum _{i=1}^n\left(u_i^{\top }{z}_1\right){\left(v_i^{\top }{\overline{\beta }}_2^{*}\right)}^2\left(w_i^{\top }{\overline{\beta }}_3^{*}\right)\left(w_i^{\top }{z}_3\right)u_i\Vert }_2.$$

Repeatedly using Lemma 2, we obtain

$${\eta }^{-\frac{1}{3}}{\Vert {\nabla }_1\tilde{\mathcal{L}}\left({\overline{\beta }}_1,{\overline{\beta }}_2^{*},{\overline{\beta }}_3^{*}\right)-{\nabla }_1\tilde{\mathcal{L}}\left({\overline{\beta }}_1,{\overline{\beta }}_2,{\overline{\beta }}_3\right)\Vert }_2\le \left(1+{\delta }_{n,p,s}\right)\left[{\left(1+{\epsilon }_0\right)}^3{\epsilon }_0+{\left(1+{\epsilon }_0\right)}^3+{\left(1+{\epsilon }_0\right)}^3+{\epsilon }_0^2+2{\epsilon }_0\right]{\eta }^{*\frac{4}{3}}\underset{j}{\text{max}}{\Vert z_j\Vert }_2\le \frac{5}{2}\left(1+{\delta }_{n,p,s}\right){\eta }^{*\frac{4}{3}}\le \frac{5}{2}\left(1+{\delta }_{n,p,s}\right){\eta }^{*\frac{4}{3}}{\text{max}}_j{\Vert z_j\Vert }_2,$$

for sufficiently small ε₀ with probability at least 1 − 60/n³. Under Condition 5, it suffices to get

$$\frac{\sqrt[3]{\eta }}{\varphi }{\Vert {\nabla }_1\tilde{\mathcal{L}}\left({\overline{\beta }}_1,{\overline{\beta }}_2,{\overline{\beta }}_3\right)-{\nabla }_1\tilde{\mathcal{L}}\left({\overline{\beta }}_1,{\overline{\beta }}_2^{*},{\overline{\beta }}_3^{*}\right)\Vert }_2\le \frac{8}{3}\underset{j}{\text{max}}{\Vert {\overline{\beta }}_j-{\overline{\beta }}_j^{*}\Vert }_2,$$ (A.84)

with probability at least 1 − 6/n.

5). Bounding I₃:

I₃ quantifies the statistical error. By the definition of noiseless gradient and noisy gradient, we have

$$\frac{\sqrt[3]{\eta }}{\varphi }{\Vert {\nabla }_1\tilde{\mathcal{L}}\left({\overline{\beta }}_1,{\overline{\beta }}_2,{\overline{\beta }}_3\right)-{\nabla }_1\mathcal{L}\left({\overline{\beta }}_1,{\overline{\beta }}_2,{\overline{\beta }}_3\right)\Vert }_2=\frac{{(\sqrt[3]{\eta })}^2}{\varphi }{\Vert \frac{2}{n}\sum _{i=1}^n{ϵ}_i\left(v_i^{\top }{\overline{\beta }}_2\right)\left(w_i^{\top }{\overline{\beta }}_3\right)u_i\Vert }_2.$$

The proof of this part essentially coincides with the proof for symmetric tensor estimation. Combining Lemmas 1 and 23, we have

$$|\frac{2}{n}\sum _{i=1}^n{ϵ}_i\left(v_i^{\top }{\overline{\beta }}_2\right)\left(w_i^{\top }{\overline{\beta }}_3\right)u_{ij}|\le C{\left(1+{\epsilon }_0\right)}^2{\eta }^{*\frac{2}{3}}\sigma \frac{{(\text{log}n)}^{\frac{3}{2}}}{\sqrt{n}},$$

with probability at least 1 − 4/n. Applying union bound over 3s coordinates, it suffices to get

$$\mathbb{P}\left(\underset{j\in [3s]}{\text{max}}|\frac{1}{n}\sum _{i=1}^n{ϵ}_i\left(v_i^{\top }{\overline{\beta }}_2\right)\left(w_i^{\top }{\overline{\beta }}_3\right)u_{ij}|\ge C{\left(1+{\epsilon }_0\right)}^2{\eta }^{*-\frac{2}{3}}\sigma \frac{{(\text{log}n)}^{\frac{3}{2}}}{\sqrt{n}}\right)\le \frac{12s}{n}.$$

Therefore, we reach

$$\frac{\sqrt[3]{\eta }}{\varphi }{\Vert {\nabla }_1\tilde{\mathcal{L}}\left({\overline{\beta }}_1,{\overline{\beta }}_2,{\overline{\beta }}_3\right)-{\nabla }_1\mathcal{L}\left({\overline{\beta }}_1,{\overline{\beta }}_2,{\overline{\beta }}_3\right)\Vert }_2\le 2C{\eta }^{*-\frac{2}{3}}\sigma \sqrt{\frac{3s{(\text{log }n)}^3}{n}},$$

with probability at least 1 − 12s/n.

6). Bounding I₄:

According to the definition of thresholding level h(β₁) in (A.76), we can bound the square as follows,

$$\frac{{(\sqrt[3]{\eta })}^2}{{\varphi }^2}{h}^2\left({\overline{\beta }}_1\right)=\frac{{(\sqrt[3]{\eta })}^4}{{\varphi }^2}\frac{4\text{ log }np}{n^2}\sum _{i=1}^n{\left(\left(u_i^{\top }{\overline{\beta }}_1\right)\left(v_i^{\top }{\overline{\beta }}_2\right)\left(w_i^{\top }{\overline{\beta }}_3\right)-\left(u_i^{\top }{\overline{\beta }}_1^{*}\right)\left(v_i^{\top }{\overline{\beta }}_2^{*}\right)\left(w_i^{\top }{\overline{\beta }}_3^{*}\right)-{ϵ}_i\right)}^2{\left(v_i^{\top }{\overline{\beta }}_2\right)}^2{\left(w_i^{\top }{\overline{\beta }}_3\right)}^2$$

Based on the basic inequality (a + b)² ≤ 2(a² + b²), we have

$${\left(\left(u_i^{\top }{\overline{\beta }}_1\right)\left(v_i^{\top }{\overline{\beta }}_2\right)\left(w_i^{\top }{\overline{\beta }}_3\right)-\left(u_i^{\top }{\overline{\beta }}_1^{*}\right)\left(v_i^{\top }{\overline{\beta }}_2^{*}\right)\left(w_i^{\top }{\overline{\beta }}_3^{*}\right)-{ϵ}_i\right)}^2\le 2{\left(\left(u_i^{\top }{\overline{\beta }}_1\right)\left(v_i^{\top }{\overline{\beta }}_2\right)\left(w_i^{\top }{\overline{\beta }}_3\right)-\left(u_i^{\top }{\overline{\beta }}_1^{*}\right)\left(v_i^{\top }{\overline{\beta }}_2^{*}\right)\left(w_i^{\top }{\overline{\beta }}_3^{*}\right)\right)}^2+2{ϵ}_i^2.$$

Denote I₁ and I₂ corresponding to optimization error and statistical error,

$$\begin{array}{l}{I}_1=\frac{{(\sqrt[3]{\eta })}^4}{{\varphi }^2}\frac{4\text{ log }np}{n^2}\sum _{i=1}^n\left(\left(u_i^{\top }{\overline{\beta }}_1\right)\left(v_i^{\top }{\overline{\beta }}_2\right)\left(w_i^{\top }{\overline{\beta }}_3\right)-\left(u_i^{\top }{\overline{\beta }}_1^{*}\right)\left(v_i^{\top }{\overline{\beta }}_2^{*}\right)\left(w_i^{\top }{\overline{\beta }}_3^{*}\right)\right)\hfill \\ I_2=\frac{{(\sqrt[3]{\eta })}^4}{{\varphi }^2}\frac{4\text{ log }np}{n^2}\sum _{i=1}^n{ϵ}_i^2{\left(v_i^{\top }{\overline{\beta }}_2\right)}^2{\left(w_i^{\top }{\overline{\beta }}_3\right)}^2.\hfill \end{array}$$

Next, I₁ is decomposed by some high-order polynomials as follows

$$I_1=\frac{{(\sqrt[3]{\eta })}^4}{{\varphi }^2}\frac{4\text{ log }np}{n^2}\left(\sum _{i=1}^n{\left(u_i^{\top }{z}_1\right)}^2{\left(v_i^{\top }{z}_2\right)}^2{\left(w_i^{\top }{z}_3\right)}^2{\left(v_i^{\top }{\overline{\beta }}_2\right)}^2{\left(w_i^{\top }{\overline{\beta }}_3\right)}^2+\sum _{i=1}^n{\left(u_i^{\top }{z}_1\right)}^2{\left(v_i^{\top }{z}_2\right)}^2{\left(w_i^{\top }{\overline{\beta }}_3^{*}\right)}^2{\left(v_i^{\top }{\overline{\beta }}_2\right)}^2{\left(w_i^{\top }{\overline{\beta }}_3\right)}^2+\sum _{i=1}^n{\left(u_i^{\top }{z}_1\right)}^2{\left(v_i^{\top }{\overline{\beta }}_2^{*}\right)}^2{\left(w_i^{\top }{z}_3\right)}^2{\left(v_i^{\top }{\overline{\beta }}_2\right)}^2{\left(w_i^{\top }{\overline{\beta }}_3\right)}^2+\sum _{i=1}^n{\left(u_i^{\top }{z}_1\right)}^2{\left(v_i^{\top }{\overline{\beta }}_2^{*}\right)}^2{\left(w_i^{\top }{\overline{\beta }}_3^{*}\right)}^2{\left(v_i^{\top }{\overline{\beta }}_2\right)}^2{\left(w_i^{\top }{\overline{\beta }}_3\right)}^2+\sum _{i=1}^n{\left(u_i^{\top }{\overline{\beta }}_1^{*}\right)}^2{\left(v_i^{\top }{\overline{\beta }}_2^{*}\right)}^2{\left(w_i^{\top }{z}_3\right)}^2{\left(v_i^{\top }{\overline{\beta }}_2\right)}^2{\left(w_i^{\top }{\overline{\beta }}_3\right)}^2+\sum _{i=1}^n{\left(u_i^{\top }{\overline{\beta }}_1^{*}\right)}^2{\left(v_i^{\top }{z}_2\right)}^2{\left(w_i^{\top }{\overline{\beta }}_3^{*}\right)}^2{\left(v_i^{\top }{\overline{\beta }}_2\right)}^2{\left(w_i^{\top }{\overline{\beta }}_3\right)}^2+\sum _{i=1}^n{\left(u_i^{\top }{\overline{\beta }}_1^{*}\right)}^2{\left(v_i^{\top }{z}_2\right)}^2{\left(w_i^{\top }{z}_3\right)}^2{\left(v_i^{\top }{\overline{\beta }}_2\right)}^2{\left(w_i^{\top }{\overline{\beta }}_3\right)}^2\right).$$ (A.85)

Each term contains the product of Gaussian random vectors form up to power ten. For the first term, by using Lemma 1,

$$\frac{1}{n}\sum _{i=1}^n{\left(u_i^{\top }{z}_1\right)}^2{\left(v_i^{\top }{z}_2\right)}^2{\left(w_i^{\top }{z}_3\right)}^2{\left(v_i^{\top }{\overline{\beta }}_2\right)}^2{\left(w_i^{\top }{\overline{\beta }}_3\right)}^2\le {\left(1+{\epsilon }_0\right)}^4{\epsilon }_0^4\left(1+C\frac{{(\text{log }n)}^5}{\sqrt{n}}\right){\eta }^{*\frac{8}{3}}\underset{j=1,2,3}{\text{max}}{\Vert z_j\Vert }_2^2,$$

with probability at least 1−1/n. Similar bounds holds for other terms. As long as n ≥ C log¹⁰ n, we have with probability at least 1 − 7/n,

$$I_1\le \frac{7\text{ log }p}{n}\underset{j=1,2,3}{\text{max}}{\Vert {\overline{\beta }}_j-{\overline{\beta }}_j^{*}\Vert }_2^2.$$ (A.86)

Now we turn to bound I₂. For fixed {ϵ_i}, we have,

$$|\sum _{i=1}^n{ϵ}_i^2{\left(v_i^{\top }{\overline{\beta }}_2\right)}^2{\left(w_i^{\top }{\overline{\beta }}_3\right)}^2-\sum _{i=1}^n{ϵ}_i^2{\Vert {\overline{\beta }}_2\Vert }_2^2{\Vert {\overline{\beta }}_3\Vert }_2^2|\le C{(\text{log }n)}^2{\Vert {\mathit{ϵ}}^2\Vert }_2{\Vert {\overline{\beta }}_2\Vert }_2^2{\Vert {\overline{\beta }}_3\Vert }_2^2.$$

with probability at least 1 − n⁻¹. Combining with Lemma 23,

$$I_2\le 4{\sigma }^2{\eta }^{*\frac{4}{3}}\frac{\text{log }p}{n}.$$ (A.87)

Putting (A.86) and (A.87) together, the thresholded effect can be bound by

$$\frac{\sqrt[3]{\eta }}{\varphi }|h\left({\beta }_1\right)|\le \sqrt{\frac{7\text{log }np}{n}}\underset{j=1,2,3}{\text{max}}{\Vert {\overline{\beta }}_j-{\overline{\beta }}_j^{*}\Vert }_2+\frac{2\sigma }{{\left(\sqrt[3]{{\eta }^{*}}\right)}^2}\sqrt{\frac{\text{log }np}{n}},$$ (A.88)

with probability at least 1 − 8/n, provided n ≳ (logn)¹⁰. ■

7). Summary:

Putting the upper bounds (A.83), (A.84) and (A.88) together, we obtain that if step size μ satisfies 0 < μ < μ₀ for some small μ₀,

$${\Vert \sqrt[3]{\eta }{\tilde{\beta }}_1^{+}-\sqrt[3]{\eta *}{\beta }_1^{*}\Vert }_2\le \left(1-\frac{\mu }{12}\right)\underset{j=1,2,3}{\text{max}}{\Vert {\overline{\beta }}_j-{\overline{\beta }}_j^{*}\Vert }_2+\mu \frac{3\sigma }{{\left(\sqrt[3]{\eta *}\right)}^2}\sqrt{\frac{3s\text{ log }p}{n}},$$

with probability at least 1−12s/n. This finishes our proof. ■

P. Matrix Form Gradient and Stochastic Gradient descent

1). Matrix Formulation of Gradient:

In this section, we provide detail derivations for (III.7) and (VI.5).

Lemma A.1. Let $\eta =\left({\eta }_1,\dots ,{\eta }_K\right)\in {ℝ}^{K\times 1},X=\left(x_1,\dots ,x_n\right)\in {ℝ}^{p\times n}$ and $B=\left({\beta }_1,\dots ,{\beta }_K\right)\in {ℝ}^{p\times K}$. The gradient of symmetric tensor estimation empirical risk function (III.5) can be written in a matrix form as follows

$${\nabla }_B\mathcal{L}(B,\eta )=\frac{6}{n}{\left[{\left({\left(B^{\top }X\right)}^{\top }\right)}^3\eta -y\right]}^{\top }{\left[{\left({\left({\left(B^{\top }X\right)}^{\top }\right)}^2\odot {\eta }^{\top }\right)}^{\top }\odot X\right]}^{\top }.$$

Proof. First let’s have a look at the gradient for k-th component,

$$\nabla {\mathcal{L}}_k\left({\beta }_k\right)=\frac{6}{n}\left(\sum _{k=1}^K{\eta }_k{\left(x_i^{\top }{\beta }_k\right)}^3-y_i\right){\eta }_k\left(x_i^{\top }{\beta }_k\right)x_i\in {ℝ}^{p\times 1},$$

for k = 1,…,K. Correspondingly, each part can be written as a matrix form,

$${\left({(\underset{K\times n}{\underbrace{B^{\top }X}})}^{\top }\right)}^3\eta -y\in {ℝ}^{n\times 1}{\left({\left({\left(B^{\top }X\right)}^{\top }\right)}^2\odot {\eta }^{\top }\right)}^{\top }\odot X\in {ℝ}^{pK\times n}.$$

This implies that ${\left[{\left({\left(B^{\top }X\right)}^{\top }\right)}^3\eta -y\right]}^{\top }{\left[{\left({\left({\left(B^{\top }X\right)}^{\top }\right)}^2\odot {\eta }^{\top }\right)}^{\top }\odot X\right]}^{\top }\in {ℝ}^{1\times pK}$. Note that ${\nabla }_B\mathcal{L}(B,\eta )=\left(\nabla {\mathcal{L}}_1{\left({\beta }_1\right)}^{\top },\dots ,\nabla {\mathcal{L}}_K{\left({\beta }_K\right)}^{\top }\right)\in {ℝ}^{1\times pK}$. The conclusion can be easily derived. ■

Lemma 17. Let $\eta =\left({\eta }_1,\dots ,{\eta }_K\right)\in {ℝ}^{K\times 1}$, $U=\left(u_1,\dots ,u_n\right)\in {ℝ}^{p_1\times n}$. The gradient of non-symmetric tensor estimation empirical risk function (VI.3) can be written in a matrix form as follows

$${\nabla }_{B_1}\mathcal{L}\left(B_1,B_2,B_3,\eta \right)=D^{\top }{\left(C_1^{\top }\odot U\right)}^{\top },$$

where e_m and ${ℝ}^p$.

Proof. Recall that {*,⊙} represent Hadamard product and Khatri-Rao product respectively. Then the dimensionality of D,C₁,C₁ ⊙ U can be calculated as follows

$$\begin{array}{l}D=\underset{n\times K}{\underbrace{{\left(B_1^{\top }U\right)}^{\top }}}*\underset{n\times K}{\underbrace{{\left(B_2^{\top }V\right)}^{\top }}}*\underset{n\times K}{\underbrace{{\left(B_3^{\top }W\right)}^{\top }}}\eta -y\in {ℝ}^{n\times 1},\hfill \\ C_1={\left(B_2^{\top }V\right)}^{\top }*{\left(B_3^{\top }W\right)}^{\top }\odot {\eta }^{\top }\in {ℝ}^{n\times K},\hfill \\ C_1^{\top }\odot U\in {ℝ}^{K{p}_1\times n}.\hfill \end{array}$$

Therefore,

$${\nabla }_{B_1}\mathcal{L}\left(B_1,B_2,B_3,\eta \right)=D^{\top }{\left(C_1^{\top }\odot U\right)}^{\top }=\left({\nabla }_1\mathcal{L}{\left({\beta }_1\right)}^{\top },\dots ,{\nabla }_K\mathcal{L}{\left({\beta }_K\right)}^{\top }\right).$$

2). Stochastic Gradient descent:

Stochastic thresholded gradient descent is a stochastic approximation of the gradient descent optimization method. Note that the empirical risk function (III.5) that can be written as a sum of differentiable functions. Followed by (III.7), the gradient of (III.5) evaluated at i-th sketching {y_i,x_i} can be written as

$${\nabla }_B{\mathcal{L}}_i(B,\eta )=\left[{\left({\left(B^{\top }{x}_i\right)}^{\top }\right)}^3\eta -y_i\right]*{\left[{\left({\left({\left(B^{\top }{x}_i\right)}^{\top }\right)}^2\odot {\eta }^{\top }\right)}^{\top }\odot x_i\right]}^{\top }\in {ℝ}^{1\times pK},$$

Thus, the overall gradient ${\nabla }_B{\mathcal{L}}_i(B,\eta )$ defined in (III.7) can be expressed as a summand of ${\nabla }_B{\mathcal{L}}_i(B,\eta )$,

$${\nabla }_B{\mathcal{L}}_i(B,\eta )=\frac{1}{n}\sum _{i=1}^n{\nabla }_B{\mathcal{L}}_i(B,\eta ).$$

The thresholded step remains the same as Step 3 in Algorithm1. Then the symmetric update of stochastic thresholded gradient descent within one iteration is summarized by

$$\text{vec}\left(B^{(t+1)}\right)=\phi \frac{{\mu }_{SGD}}{\varphi }h\left(B^{(t)}\right)\left(\text{vec}\left(B^{(t)}\right)-\frac{{\mu }_{SGD}}{\varphi }{\nabla }_B{\mathcal{L}}_i\left(B^{(t)}\right)\right).$$

Q. Technical Lemmas

Lemma 18. Suppose $x\in {ℝ}^p$ is a standard Gaussian random vector. For any non-random vector $a,b,c\in {ℝ}^p$, we have the following tensor expectation calculation,

$$\mathbb{E}\left(\left(a^{\top }x\right)\left(b^{\top }x\right)\left(c^{\top }x\right)x\circ x\circ x\right)+\left(a\circ b\circ c+a\circ c\circ b+b\circ a\circ +b\circ c\circ a+c\circ b\circ a+c\circ a\circ b\right)+3\sum _{m=1}^p\left(a\circ e_m\circ e_m\left(b^{\top }c\right)+e_m\circ b\circ e_m\left(a^{\top }c\right)+e_m\circ e_m\circ c\left(a^{\top }b\right)\right),$$ (A.89)

where e_m is a canonical vector in ${ℝ}^p$.

Proof. Recall that for a standard Gaussian random variable x, its odd moments are zero and even moments are $\mathbb{E}\left(x^6\right)=15$, $\mathbb{E}\left(x^4\right)=4$. Expanding the LHS of (A.89) and comparing LHS and RHS, we will reach the conclusion. Details are omitted here. ■

Lemma 19. Suppose $u\in {ℝ}^{p_1}$, $v\in {ℝ}^{p_2}$, $w\in {ℝ}^{p_3}$ are independent standard Gaussian random vectors. For any non-random vector $a\in {ℝ}^{p_1}$, $b\in {ℝ}^{p_2}$, $c\in {ℝ}^{p_3}$, we have the following tensor expectation calculation

$$\mathbb{E}\left(\left(a^{\top }u\right)\left(b^{\top }v\right)\left(c^{\top }w\right)u\circ v\circ w\right)=a\circ b\circ c.$$ (A.90)

Proof. Due to the independence among u,v,w, the conclusion is easy to obtain by using the moment of standard Gaussian random variable. ■

Note that in the left side of (A.89), it involves an expectation of rank-one tensor. When multiplying any non-random rank-one tensor with same dimensionality, i.e., a₁ ∘ b₁ ∘ c₁, on both sides, it will facilitate us to calculate the expectation of product of Gaussian vectors, see next Lemma for details.

Lemma A.2. Suppose $x\in {ℝ}^p$ is a standard Gaussian random vector. For any non-random vector $a,b,c,d\in {ℝ}^p$, we have the following expectation calculation

$$\begin{array}{l}\mathbb{E}{\left(x^{\top }a\right)}^6=15\Vert a{\Vert }_2^6,\\ \mathbb{E}{\left(x^{\top }a\right)}^5\left(x^{\top }b\right)=15\Vert a{\Vert }_2^4\left(a^{\top }b\right),\\ \mathbb{E}{\left(x^{\top }a\right)}^4{\left(x^{\top }b\right)}^2=12\Vert a{\Vert }_2^2{\left(a^{\top }b\right)}^2+3\Vert a{\Vert }_2^4\Vert b{\Vert }_2^2,\\ \mathbb{E}{\left(x^{\top }a\right)}^3{\left(x^{\top }b\right)}^3=6{\left(a^{\top }b\right)}^3+9\left(a^{\top }b\right)\Vert a{\Vert }_2^2\Vert b{\Vert }_2^2,\\ \mathbb{E}{\left(x^{\top }a\right)}^3{\left(x^{\top }b\right)}^2\left(x^{\top }c\right)=6{\left(a^{\top }b\right)}^2\left(a^{\top }c\right)+6\left(a^{\top }b\right)\left(b^{\top }c\right)\left(a^{\top }a\right)+3\left(a^{\top }c\right)\left(b^{\top }b\right)\left(a^{\top }a\right),\\ \mathbb{E}{\left(x^{\top }a\right)}^2\left(x^{\top }b\right){\left(x^{\top }c\right)}^2\left(x^{\top }d\right)=2{\left(a^{\top }c\right)}^2\left(b^{\top }d\right)+4\left(a^{\top }c\right)\left(b^{\top }c\right)\left(a^{\top }d\right)+6\left(a^{\top }c\right)\left(a^{\top }b\right)\left(c^{\top }d\right)+3\left(c^{\top }x\right)\left(b^{\top }d\right)\left(a^{\top }a\right).\end{array}$$

Proof. Note that $\mathbb{E}\left({\left(x^{\top }a\right)}^3{\left(x^{\top }b\right)}^3\right)=\mathbb{E}\left({\left(x^{\top }a\right)}^3\langle x\circ x\circ x,b\circ b\circ b\rangle \right)$ Then we can apply the general result in Lemma 18. Comparing both sides, we will obtain the conclusion. Others part follows the similar strategy. ■

Next lemma provides a probabilistic concentration bound for non-symmetric rank-one tensor under tensor spectral norm.

Lemma 20. Suppose $X={\left(x_1^{\top },\cdots ,x_n^{\top }\right)}^{\top }$, $Y={\left(y_1^{\top },\cdots ,y_n^{\top }\right)}^{\top }$, $Z={\left(z_1^{\top },\cdots ,z_n^{\top }\right)}^{\top }$ are three n × p random matrices. The ψ₂-norm of each entry is bounded, s.t. ‖X_ij‖_ψ2 = K_x,‖Y_ij‖_ψ2 = K_y,‖Z_ij‖_ψ2 = K_z. We assume the row of X,Y, Z are independent. There exists an absolute constant C such that,

$$\begin{array}{l}\mathbb{P}\left({\Vert \frac{1}{n}\sum _{i=1}^n\left[x_i\circ y_i\circ z_i-\mathbb{E}\left(x_i\circ y_i\circ z_i\right)\right]\Vert }_s\ge C{K}_x{K}_y{K}_z{\delta }_{n,p,s}\right)\le p^{-1}.\hfill \\ \mathbb{P}\left({\Vert \frac{1}{n}\sum _{i=1}^n\left[x_i\circ x_i\circ x_i-\mathbb{E}\left(x_i\circ x_i\circ x_i\right)\right]\Vert }_s\ge C{K}_x^3{\delta }_{n,p,s}\right)\le p^{-1}.\hfill \end{array}$$

Here, ‖ · ‖_s is the sparse tensor spectral norm defined in (II.3) and ${\delta }_{n,p,s}=\sqrt{s\text{ log}(ep/s)/n}+\sqrt{s^3\text{ log}{(ep/s)}^3/n^2}$.

Proof. Bounding spectral norm always relies on the construction of the ϵ-net. Since we will bound a sparse tensor spectral norm, our strategy is to discrete the sparse set and construct the ϵ-net on each one. Let us define a sparse set ${\mathcal{B}}_0=\left\{x\in {ℝ}^p,\Vert x{\Vert }_2=1,\Vert x{\Vert }_0\le s\right\}$. And let ${\mathcal{B}}_{0,s}$ be the s-dimensional set defined by ${\mathcal{B}}_{0,s}=\left\{x\in {ℝ}^s,\Vert x{\Vert }_2=1\right\}$. Note that ${\mathcal{B}}_0$ is corresponding to s-sparse unit vector set which can be expressed as a union of subsets of dimension s by expanding some zeros, namely ${\mathcal{B}}_0=\cup {\mathcal{B}}_{0,s}$. There should be at most < such set.

Recalling the definition of sparse tensor spectral norm in (II.3), we have

$$A={\Vert \frac{1}{n}\sum _{i=1}^n\left[x_i\circ y_i\circ z_i-\mathbb{E}\left(x_i\circ y_i\circ z_i\right)\right]\Vert }_s\underset{{\chi }_1,{\chi }_2,{\chi }_3\in {\mathcal{B}}_0}{\text{sup}}|\frac{1}{n}\sum _{i=1}^n\left[\langle x_i,{\chi }_1\rangle \langle y_i,{\chi }_2\rangle \langle z_i,{\chi }_3\rangle -\mathbb{E}\left(\langle x_i,{\chi }_1\rangle \langle y_i,{\chi }_2\rangle \langle z_i,{\chi }_3\rangle \right)\right]|.$$

Instead of constructing the ϵ-net on ${\mathcal{B}}_0$, we will construct an ϵ-net for each of subsets ${\mathcal{B}}_{0,s}$. Define ${\mathcal{N}}_{{\mathcal{B}}_{0,s}}$ as the 1/2-set of ${\mathcal{B}}_{0,s}$. From Lemma 3.18 in [64], the cardinality of ${\mathcal{N}}_{0,s}$ is bounded by 5^s. By Lemma 21, we obtain

$$\begin{array}{c}\text{ sup }\\ {\chi }_1,{\chi }_2,{\chi }_3\in {\mathcal{B}}_{0,s}\end{array}|\frac{1}{n}\sum _{i=1}^n\left[\langle x_i,{\chi }_1\rangle \langle y_i,{\chi }_2\rangle \langle z_i,{\chi }_3\rangle -\mathbb{E}\left(\langle x_i,{\chi }_1\rangle \langle y_i,{\chi }_2\rangle \langle z_i,{\chi }_3\rangle \right)\right]|\le 2^3\underset{{\chi }_1,{\chi }_2,{\chi }_3\in {\mathcal{N}}_{{\mathcal{B}}_{0,a}}}{\text{sup}}|\frac{1}{n}\sum _{i=1}^n\left[\langle x_i,{\chi }_1\rangle \langle y_i,{\chi }_2\rangle \langle z_i,{\chi }_3\rangle -\mathbb{E}\left(\langle x_i,{\chi }_1\rangle \langle y_i,{\chi }_2\rangle \langle z_i,{\chi }_3\rangle \right)\right]|.$$ (A.91)

By rotation invariance of sub-Gaussian random variable, $\langle x_i,{\chi }_1\rangle$, $\langle y_i,{\chi }_2\rangle$, $\langle z_i,{\chi }_3\rangle$ are still sub-Gaussian random variables with ψ₂-norm bounded by K_x,K_y,K_z, respectively. Applying Lemma 1 and union bound over ${\mathcal{N}}_{{\mathcal{B}}_{0,s}}$, the right hand side of (A.91) can be bounded by

$$\text{RHS}\ge 8K_x{K}_y{K}_{z}C\left(\sqrt{\frac{\text{log}{\delta }^{-1}}{n}}+\sqrt{\frac{{\left(\text{log}{\delta }^{-1}\right)}^3}{n^2}}\right),$$

with probability smaller than (5^s)³δ for any 0 < δ < 1.

Lastly, taking the union bound over all possible subsets ${\mathcal{B}}_{0,s}$ yields that

$$\mathbb{P}\left(A\ge 8K_x{K}_y{K}_{z}C\left(\sqrt{\frac{\text{log}{\delta }^{-1}}{n}}+\sqrt{\frac{{\left(\text{log}{\delta }^{-1}\right)}^3}{n^2}}\right)\right)\le {\left(\frac{ep}{s}\right)}^s{\left(5^s\right)}^3\delta ={\left(\frac{125ep}{s}\right)}^s\delta .$$

Letting $p^{-1}={\left(\frac{125ep}{s}\right)}^s\delta$, we obtain with probability at least 1 − 1/p,

$$A\le C{K}_x{K}_y{K}_z\left(\sqrt{\frac{s\text{ log}(p/s)}{n}}+\sqrt{\frac{s^3{\text{log}}^3(p/s)}{n^2}}\right),$$

with some adjustments on constant C. The proof for symmetric case is similar to non-symmetric case so we omit here.

Lemma 21 (Tensor Covering Number(Lemma 4 in [65])). Let $\mathbb{N}$ be an ϵ-net for a set B associated with a norm ‖ · ‖. Then, the spectral norm of a d-mode tensor $\mathcal{A}$ is bounded by

$$\underset{x_1,\dots ,x_{d-1}\in B}{\text{sup}}{\Vert \mathcal{A}{\times }_1{x}_1\dots {\times }_{d-1}{x}_{d-1}\Vert }_2\le {\left(\frac{1}{1-\epsilon }\right)}^{d-1}\underset{x_1\cdots x_{d-1}\in \mathbb{N}}{\text{sup}}{\Vert \mathcal{A}{\times }_1{x}_1\cdots {\times }_{d-1}{x}_{d-1}\Vert }_2.$$

This immediately implies that the spectral norm of a d-mode tensor $\mathcal{A}$ is bounded by

$$\Vert \mathcal{A}{\Vert }_2\le {\left(\frac{1}{1-ϵ}\right)}^{d-1}\underset{x_1\dots x_{d-1}\in \mathcal{N}}{\text{sup}}{\Vert \mathcal{A}{\times }_1{x}_1\cdots {\times }_{d-1}{x}_{d-1}\Vert }_2,$$

where $\mathbb{N}$ is the ϵ-net for the unit sphere ${\mathbb{S}}^{n-1}$ in ${ℝ}^n$.

Lemma 22 (Sub-Gaussianess of the Product of Random Variables). Suppose X₁ is a bounded random variable with |X₁| ≤ K₁ almost surely for some K₁ and X₂ is a sub-Gaussian random variable with Orlicz norm ‖X₂‖_ψ2K₂. Then X₁X₂ is still a sub-Gaussian random variable with Orlicz norm ‖X₁X₂‖_ψ2 = K₁K₂.

Proof: Following the definition of sub-Gaussian random variable, we have

$$\mathbb{P}\left(|X_1{X}_2|>t\right)=\mathbb{P}\left(|X_2|>\frac{t}{|X_1|}\right)\le \mathbb{P}\left(|X_2|>\frac{t}{|K_1|}\right)\le \text{exp}\left(1-t^2/K_1^2{K}_2^2\right),$$

holds for all t ≥ 0. This ends the proof. ■

Lemma 23 (Tail Probability for the Sum of Sub-exponential Random Variables (Lemma A.7 in [48])). Suppose ϵ₁,…, ϵ_n are independent centered sub-exponential random variables with

$$\sigma :=\underset{1\le i\le n}{\text{max}}{\Vert {ϵ}_i\Vert }_{{\psi }_1}.$$

Then with probability at least 1 − 3/n, we have

$$\begin{gathered} |\frac{1}{n}\sum _{i=1}^n{ϵ}_i|\le C_0\sigma \sqrt{\frac{\text{log}n}{n}},\Vert \mathit{ϵ}{\Vert }_{\infty }\le C_0\sigma \text{log }n, \\ |\frac{1}{n}\sum _{i=1}^n{ϵ}_i^2|\le C_0{\sigma }^2,|\frac{1}{n}\sum _{i=1}^n{ϵ}_i^4|\le C_0{\sigma }^4, \end{gathered}$$

for some constant C₀

Lemma 24 (Tail Probability for the Sum of Weibull Distributions (Lemma 3.6 in [34])). Let α ∈ [1,2] and Y₁,…,Y_n be independent symmetric random variables satisfying. Then for every vector $a=\left(a_1,\dots ,a_n\right)\in {ℝ}^n$ and every t ≥ 0,

$$\mathbb{P}\left(|\sum _{i=1}^n{a}_i{Y}_i|\ge t\right)\le 2\text{exp}\left(-c\text{min}\left(\frac{t^2}{\Vert a{\Vert }_2^2},\frac{t^{\alpha }}{\Vert a{\Vert }_{\alpha *}^{\alpha }}\right)\right)$$

Proof. It is a combination of Corollaries 2.9 and 2.10 in [58].

Lemma 25 (Moments for the Sum of Weibull Distributions (Corollary 1.2 in [66])). Let X₁,X₂,…,X_n be a sequence of independent symmetric random variables satisfying $\mathbb{P}\left(|Y_i|\ge t\right)=\text{exp}\left(-t^{\alpha }\right)$, where 0 < α < 1. Then, for p ≥ 2 and some constant C(α) which depends only on α,

$${\Vert \sum _{i=1}^n{a}_i{X}_i\Vert }_p\le C(\alpha )\left(\sqrt{p}\Vert a{\Vert }_2+p^{1/\alpha }\Vert a{\Vert }_{\infty }\right).$$

Lemma 26 (Stein’s Lemma [56]). Let $x\in {ℝ}^d$ be a random vector with joint density function p(x). Suppose the score function ∇_x logp(x) exists. Consider any continuously differentiable function $G(x):{ℝ}^{d_x}\to ℝ$. Then, we have

$$\mathbb{E}\left[G(x)\cdot {\nabla }_x\text{log }p(x)\right]=-\mathbb{E}\left[{\nabla }_{x}G(x)\right].$$

Lemma 27 (Khinchin-Kahane Inequality (Theorem 1.3.1 in ${\left\{a_i\right\}}_{i=1}^n$ a finite non-random sequence, ${\left\{{\epsilon }_i\right\}}_{i=1}^n$ be a sequence of independent Rademacher variables and 1 < p < q < ∞. Then

$${\Vert \sum _{i=1}^n{\epsilon }_i{a}_i\Vert }_q\le {\left(\frac{q-1}{p-1}\right)}^{1/2}{\Vert \sum _{i=1}^n{\epsilon }_i{a}_i\Vert }_p.$$

Lemma 28. Suppose each non-zero element of ${\left\{x_k\right\}}_{k=1}^K$ is drawn from standard Gaussian distribution and ‖x_k‖₀ ≤ s for k ∈ [K]. Then we have for any 0 < δ ≤ 1,

$$\mathbb{P}\left(\underset{1\le k_1<k_2\le K}{\text{max}}|\langle x_{k_1},x_{k_2}\rangle |\le C\sqrt{s}\sqrt{\text{log }K+\text{log }1/\delta }\right)\ge 1-\delta ,$$

where C is some constant.

Proof. Let us denote ${\mathcal{S}}_{k_1{k}_2}\subset [1,2,\dots ,p]$ as an index set such that for any $i,j\in {\mathcal{S}}_{k_1{k}_2}$, we have $x_{k_{1}i}\ne 0$ and $x_{k_{2}j}\ne 0$. From the definition of ${\mathcal{S}}_{k_1{k}_2}$, we know that $|{\mathcal{S}}_{k_1{k}_2}|\le s$ and $x_{k_1}^{\top }{x}_{k_2}={\sum }_{j=1}^p{x}_{k_{1}j}{x}_{k_{2}j}={\sum }_{j\in {\mathcal{S}}_{k_1{k}_2}}{x}_{k_{1}j}{x}_{k_{2}j}$. We apply standard Hoeffding’s concentration inequality,

$$\mathbb{P}\left(|x_{k_1},x_{k_2}|\ge t\right)=\mathbb{P}\left(|\sum _{j\in {\mathcal{S}}_{k_1{k}_2}}{x}_{k_{1}j}{x}_{k_{2}j}|\ge t\right)\le e\text{exp}\left(-\frac{c{t}^2}{s}\right).$$

Letting ct²/s = log(1/δ), we reach the conclusion.