Determining the Number of Latent Factors in Statistical Multi-Relational Learning

Abstract

Statistical relational learning is primarily concerned with learning and inferring relationships between entities in large-scale knowledge graphs. Nickel et al. (2011) proposed a RESCAL tensor factorization model for statistical relational learning, which achieves better or at least comparable results on common benchmark data sets when compared to other state-of-the-art methods. Given a positive integer s, RESCAL computes an s-dimensional latent vector for each entity. The latent factors can be further used for solving relational learning tasks, such as collective classification, collective entity resolution and link-based clustering.

The focus of this paper is to determine the number of latent factors in the RESCAL model. Due to the structure of the RESCAL model, its log-likelihood function is not concave. As a result, the corresponding maximum likelihood estimators (MLEs) may not be consistent. Nonetheless, we design a specific pseudometric, prove the consistency of the MLEs under this pseudometric and establish its rate of convergence. Based on these results, we propose a general class of information criteria and prove their model selection consistencies when the number of relations is either bounded or diverges at a proper rate of the number of entities. Simulations and real data examples show that our proposed information criteria have good finite sample properties.

1. Introduction

Relational data is becoming ubiquitous in artificial intelligence and social network analysis. These data sets are in the form of graphs, with nodes and edges representing entities and relationships, respectively. Recently, a number of companies have developed and released their knowledge graphs, including the Google Knowledge Graph, Microsoft Bing’s Satori Knowledge Base, Yandex’s Object Answer, the Linkedln Knowledge Graph, etc. These knowledge graphs are graph-structured knowledge bases that store factual information as relationships between entities. They are created via the automatic extraction of semantic relationships from semi-structured or unstructured text (see Section II.C in Nickel et al., 2016). The data may be incomplete, noisy and contain false information. It is therefore of great importance to infer the existence of a particular relationship to improve the quality of these extracted information.

Statistical relational learning is primarily concerned with learning from relational data sets, and solving tasks such as predicting whether two entities are related (link prediction), identifying equivalent entities (entity resolution), and grouping similar entities based on their relationships (link-based clustering). Statistical relational models can be roughly divided into three categories: the relational graphical models, the latent class models and the tensor factorization models. Relational graphical models include probabilistic relational models (Getoor and Mihalkova, 2011) and Markov logic networks (MLN, Richardson and Domingos, 2006). These models are constructed via Bayesian or Markov networks. In latent class models, each entity is assigned to one of the latent classes and the probability of a relationship between entities depends on their corresponding classes. Two important examples include the stochastic block model (SBM, Nowicki and Snijders, 2001) and the infinite relational model (IRM, Kemp et al., 2006). IRM can be viewed as a nonparametric extension of SBM where the total number of clusters is not prespecified. Both models have received considerable attentions in the statistics and machine learning literature for community detection in networks.

Tensors are multidimensional arrays. Tensor factorization methods such as CANDE-COMP/PARAFAC (CP, Harshman and Lundy, 1994), Tucker (Tucker, 1966) and their extensions have found applications in a variety of fields. Kolda and Bader (2009) presented a thorough overview of tensor decompositions and their applications. Recently, tensor factorizations are being actively studied in the statistics literature and have becoming an emerging field of statistics. To name a few, Chi and Kolda (2012) developed a Poisson tensor factorization model for sparse count data. Yang and Dunson (2016) proposed a conditional tensor factorization model for high-dimensional classification with categorical predictors. Sun et al. (2017) proposed a sparse tensor decomposition method by incorporating a truncation step into the tensor power iteration step.

Relational data sets are typically expressed as (subject, predicate, object) triples and can be grouped as a third-order tensor. As a result, tensor factorization methods can be naturally applied to these data sets. Nickel (2013) proposed a RESCAL factorization model for statistical relational learning. Compared to other tensor factorization approaches such as CP and Tucker methods, RESCAL is more capable of detecting the correlations produced between multiple interconnected nodes. For relational data consisting of n entities, K types of relations, and a positive integer s, RESCAL computes an n × s factor matrix and an s × s × K core tensor. The factor matrix and the core tensor can be further used for link prediction, entity resolution and link-based clustering. Nickel et al. (2011) showed that a linear RESCAL model achieved better or comparable results on common benchmark data sets when compared to other existing methods such as MLN, DEDICOM (Harshman, 1978), IRM, CP, MRC (Kok and Domingos, 2007), etc. It was shown in Nickel and Tresp (2013) that a logistic RESCAL model could further improve the link prediction results.

Central to the empirical validity of RESCAL is the correct specification of the number of latent factors. Nickel et al. (2011) proposed to select this parameter via cross-validation. As commonly known for cross-validation methods, there’s no theoretical guarantee against overestimation. Besides, cross-validation can be computationally expensive, especially for large n and K. In the literature, model selection is less studied for tensor factorization methods. Allen (2012) and Sun et al. (2017) proposed to use Bayesian information criteria (BIC, Schwarz, 1978) for sparse CP decomposition. However, no theoretical results were provided for BIC. Indeed, we show in this paper that a BIC-type criterion may fail for the RESCAL model.

The contribution of this paper is twofold. First, we propose a general class of information criteria for the RESCAL model and prove their model selection consistency. Although we focus on the RESCAL model, our information criteria can be extended to select models for general tensor factorization methods with slight modification. The problem is nonstandard and challenging since both the factor matrix and the core tensor are not observed and need to be estimated. Besides, the model parameters are non-identifiable. Moreover, the derivation of model/tuning parameter selection consistency of information criteria usually relies on the (uniform) consistency of estimated parameters. For example, Fan and Tang (2013) derived the uniform consistency of the maximum likelihood estimators (MLEs) to prove the consistency of GIC (see Proposition 2 in that paper). Zhang et al. (2016) established the uniform consistency of the support vector machine solutions to prove the consistency of SVMIC_h (see Lemma 2 in that paper). The consistency of these estimators are due to the concavity (convexity) of the likelihood (or the empirical loss) functions. In contrast, for most tensor decomposition models including RESCAL, the likelihood (or the empirical loss) function is usually non-concave (non-convex) and may have multiple local solutions. As a result, the corresponding global maximizer (minimizer) may not be consistent even with the identifiability constraints. It remains unknown how to establish the consistency of the information criterion without consistency of the estimator. A key innovation in our analysis is to design a “proper” pseudometric and show that the global optimum is consistent under this specific pseudometric. We further establish the rate of convergence of the global optimum under this pseudometric as a function of n and K. Based on these results, we establish the consistency of our information criteria when K is either bounded or diverges at a proper rate of n. No parametric assumptions are imposed on the latent factors. Second, we introduce a scalable algorithm for estimating the parameters in the logistic RESCAL model. Despite the fact that a linear RESCAL model can be conveniently solved by an alternating least square algorithm (Nickel et al., 2011), there are lack of optimization algorithms for solving general RESCAL models. The proposed algorithm is based on the alternating direction method of multipliers (ADMM, Boyd et al., 2011) and can be implemented in a parallelized fashion.

The rest of the paper is organized as follows. We formally introduce the RESCAL model and study the parameter identifiability in Section 2. Our information criteria are presented in Section 3 and their model selection properties are investigated. Numerical examples are presented in Section 4 to examine the finite sample performance of the proposed information criteria. Section 5 concludes with a summary and discussion of future extensions. All the proofs are given in the Appendix.

2. The RESCAL Model

This section is structured as follows. We introduce the RESCAL model in Section 2.1. In Section 2.2, we study the identifiability of parameters in the model.

2.1. Model Setup

In knowledge graphs, facts can be expressed in the form of (subject, predicate, object) triples, where subject and object are entities and predicate is the relation between entities. For example, consider the following sentence from Wikipedia:

Jon Snow is a fictional character in the A Song of Ice and Fire series of fantasy novels by American author George R. R. Martin, and its television adaptation Game of Thrones.

The information contained in this message can be summarized into the following set of (subject, predicate, object) triples:

Subject	Predicate	Object
Jon Snow	character in	A Song of Ice and Fire
Jon Snow	character in	Game of Thrones
A Song of Ice and Fire	genre	novel
Game of Thrones	genre	television series
George R.R. Martin	author of	A Song of Ice and Fire
George R.R. Martin	profession	novelist

In this example, we have a total of 7 entities, 4 types of relations and 6 triples. More generally, let $\mathcal{E}=\left\{e_1,\dots ,e_n\right\}$ denote the set of all entities and $\mathcal{R}=\left\{r_1,\dots ,r_K\right\}$ denote the set of all relation types. The number of relations K is either bounded or diverges with n. Assuming non-existing triples indicate false relationships, we can construct a third-order binary tensor

$$\mathit{Y}={\{Y_{ijk}\}}_{i,j\in \{1,\dots ,n\},k\in \{1,\dots ,K\}},$$

such that

$$Y_{ijk}=\{\begin{array}{ll}1,\hfill & \text{if a triple}(e_i,r_k,e_j)\text{exists,}\hfill \\ 0,\hfill & \text{otherwise}.\hfill \end{array}$$

The RESCAL model is defined as follows. For each entity e_i, a latent vector ${\mathit{a}}_{i,0}\in {ℝ}^{s_0}$ is generated. The Y_ijk’s are assumed to be conditionally independent given all latent factors ${\{{\mathit{a}}_{i,0}\}}_{i=1}^n$. Besides, it is assumed that

$$\mathrm{Pr}(Y_{ijk}=1|{\{{\mathit{a}}_{i,0}\}}_{i=1}^n)=g({\mathit{a}}_{i,0}^T{\mathit{R}}_{k,0}{\mathit{a}}_{j,0}),$$ (1)

for some strictly monotone link function g and s₀ × s₀ matrices ${\mathit{R}}_{1,0},\dots ,{\mathit{R}}_{K,0}$. In the above model, a_i,0 corresponds to the latent representation of the ith entity and R_k,0 specifies how these a_i,0’s interact for the k-th relation. To account for asymmetric relations, we do not restrict R_k,0’s to symmetric matrices. When the relations are symmetric, i.e.,

$$\mathrm{Pr}(Y_{ijk}=1|{\{{\mathit{a}}_{i,0}\}}_{i=1}^n)=\mathrm{Pr}(Y_{jik}=1|{\{{\mathit{a}}_{i,0}\}}_{i=1}^n),\forall i,j,k,$$

one can impose the symmetry constraints and obtain a similar derivation.

For continuous Y_ijk, a related tensor factorization model is the TUCKER-2 decomposition, which decomposes the tensor into

$$Y_{ijk}={\mathit{a}}_{i,0}^T{\mathit{R}}_{k,0}{\mathit{b}}_{j,0}+e_{ijk},\forall i,j,k,$$ (2)

for some ${\mathit{a}}_{1,0},\dots ,{\mathit{a}}_{n,0}\in {ℝ}^{s_1},{\mathit{b}}_{1,0},\dots ,{\mathit{b}}_{n,0}\in {ℝ}^{s_2},{\mathit{R}}_{1,0},\dots ,{\mathit{R}}_{K,0}\in {ℝ}^{s_1\times s_2}$ and some (random) errors ${\{e_{ijk}\}}_{ijk}$. By Equation 1, RESCAL can be interpreted as a “nonlinear” TUCKER-2 model with the additional constraints that s₁ = s₂ = s₀ and ${\mathit{a}}_{i,0}={\mathit{b}}_{i,0},\forall i$.

CP decomposition is another important tensor factorization method that decomposes a tensor into a sum of rank-1 tensors. It assumes that

$$Y_{ijk}=\sum _{s=1}^{s_0}{a}_{i,s}{b}_{j,s}{r}_{k,s}+e_{ijk},$$

for some ${\{a_{i,s}\}}_{i,s},{\{b_{j,s}\}}_{j,s},{\{r_{k,s}\}}_{k,s}$ and ${\{e_{ijk}\}}_{ijk}$. Define ${\mathit{a}}_{i,0}={(a_{i,1},\dots ,a_{i,s_0})}^T$ and ${\mathit{b}}_{i,0}={(b_{i,1},\dots ,b_{i,s_0})}^T$. In view of Equation 2, CP is a special TUCKER-2 model with the constraints that s₁ = s₂ = s₀ and ${\mathit{R}}_{k,0}=\mathrm{diag}(r_{k,1},\dots ,r_{k,s_0})$ where $\text{diag}(r_{k,1},\dots ,r_{k,s_0})$ is a diagonal matrix with the sth diagonal elements being r_k,s.

In this paper, the proposed information criteria are designed in particular for the RESCAL model. However, they can be extended to estimate s₀ in a more general tensor factorization framework including CP and TUCKER-2 models. We discuss this further in Section 5.

2.2. Identifiability

The parameterization in Equation 1 is not identifiable. To see this, for any nonsingular matrix $\mathit{C}\in {ℝ}^{s_0\times s_0}$, we define ${\mathit{a}}_i={\mathit{C}}^{-1}{\mathit{a}}_{i,0},{\mathit{R}}_k={\mathit{C}}^T{\mathit{R}}_{k,0}\mathit{C},\forall i,k$. Observe that

$${\mathit{a}}_{i,0}^T{\mathit{R}}_{k,0}{\mathit{a}}_{j,0}={\mathit{a}}_i^T{\mathit{R}}_k{\mathit{a}}_j,\forall i,j,k$$

and hence we have

$$\mathrm{Pr}(Y_{ijk}=1)=\text{g}({\mathit{a}}_i^T{\mathit{R}}_k{\mathit{a}}_j).$$

Let ${\mathit{A}}_0={[{\mathit{a}}_{1,0},\dots ,{\mathit{a}}_{n,0}]}^T$. We impose the following condition.

(A0) (i) Assume A₀ has full column rank. (ii) Assume the matrix $[{\mathit{R}}_{1,0}^T,\dots ,{\mathit{R}}_{K,0}^T]$ has full row rank.

(A0)(i) requires the latent factors to be linearly independent. (A0)(ii) holds when at least one of the R_k,0’s has full rank. Under Condition (A0), the following lemma states that the RESCAL model is identifiable up to a nonsingular linear transformation. In Section B.1 of the Appendix, we show (A0) is also necessary to guarantee such identifiability when ${\mathit{R}}_{1,0},\dots ,{\mathit{R}}_{K,0}$ are symmetric.

Lemma 1 (Identifiability).

Assume (A0) holds. Assume there exist some ${\{{\mathit{a}}_i\}}_i,{\{{\mathit{R}}_k\}}_k$ such that ${\mathit{a}}_i\in {ℝ}^{s_0}$, ${\mathit{R}}_k\in {ℝ}^{s_0\times s_0}$ and

$$g({\mathit{a}}_{i,0}^T{\mathit{R}}_{k,0}{\mathit{a}}_{j,0})=g({\mathit{a}}_i^T{\mathit{R}}_k{\mathit{a}}_j),\forall i,j,k.$$

Then, there exists some invertible matrix $\mathit{C}\in {ℝ}^{s_0\times s_0}$ such that

$${\mathit{a}}_i={\mathit{C}}^{-1}{\mathit{a}}_{i,0}\mathit{\text{and}}{\mathit{R}}_k={\mathit{C}}^T{\mathit{R}}_{k,0}\mathit{C}.$$

To fix the nonsingular transformation indeterminacy, we adopt a specific constrained parameterization and focus on estimating ${\{{\mathit{a}}_i^{*}\mathit{\}}}_i$ and ${\{R_k^{*}\}}_k$ where

$${\mathit{a}}_i^{*}={({\mathit{A}}_{s_0,0}^{-1})}^T{\mathit{a}}_{i,0}\text{and}{\mathit{R}}_k^{*}={\mathit{A}}_{s_0,0}{\mathit{R}}_{k,0}{\mathit{A}}_{s_0,0}^T,$$

where ${\mathit{A}}_{s_0,0}={[{\mathit{a}}_{1,0},\dots ,{\mathit{a}}_{s_0,0}]}^T$. Observe that

$$[{\mathit{a}}_1^{*},\dots ,{\mathit{a}}_{s_0}^{*}]={({\mathit{A}}_{s_0,0}^{-1})}^T[{\mathit{a}}_{1,0},\dots ,{\mathit{a}}_{s_0,0}]={({\mathit{A}}_{s_0,0}^{-1})}^T{\mathit{A}}_{s_0,0}^T={\mathit{I}}_{s_0},$$

where I_s0 stands for an s₀ × s₀ identity matrix. Therefore, the first $s_0{\mathit{a}}_i^{*}$’s are fixed as long as ${\mathit{A}}_{s_0,0}$ is nonsingular. By Lemma 1, the parameters ${\{{\mathit{a}}_i^{*}\}}_i$ and ${\{{\mathit{R}}_k^{*}\}}_k$ are estimable.

From now on, we only consider the logistic link function for simplicity, i.e, g(x) = 1/{1 + exp(−x)}. Results for other link functions can be similarly discussed.

3. Model Selection

Parameters ${\{{\mathit{a}}_i^{*}\}}_{i=1}^n$ and ${\{{\mathit{R}}_k^{*}\}}_{k=1}^K$ can be estimated by maximizing the (conditional) log-likelihood function. Since we use the logistic link function, the log-likelihood is equal to

$$\begin{gathered} {\mathcal{l}}_n(\mathit{Y};{\{{\mathit{a}}_i\}}_i,{\{{\mathit{R}}_k\}}_k)=\log \left(\prod _{ijk}\text{g}{({\mathit{a}}_i^T{\mathit{R}}_k{\mathit{a}}_j)}^{Y_{ijk}}{\{1-\text{g}({\mathit{a}}_i^T{\mathit{R}}_k{\mathit{a}}_j)\}}^{1-Y_{ijk}}\right) \\ =\sum _{ijk}(Y_{ijk}\log \{\text{g}({\mathit{a}}_i^T{\mathit{R}}_k{\mathit{a}}_j)\}+(1-Y_{ijk})\log \{1-\text{g}({\mathit{a}}_i^T{\mathit{R}}_k{\mathit{a}}_j)\}) \\ =\sum _{ijk}(Y_{ijk}{\mathit{a}}_i^T{\mathit{R}}_k{\mathit{a}}_j-\log \{1+\exp ({\mathit{a}}_i^T{\mathit{R}}_k{\mathit{a}}_j)\}), \end{gathered}$$

where the first equality is due to the conditional independence assumption.

We assume the number of latent factors s₀ is fixed. For any $s\in \{1,\dots ,s_{\max }\}$ here s_max is allowed to diverge with n and satisfies s_max ≥ s₀, we define the following constrained maximum likelihood estimator

$$({\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k)=\underset{\underset{\mathrm{vec}({\mathit{R}}_1^{(s)}),\dots ,\mathrm{vec}({\mathit{R}}_K^{(s)})\in {\mathrm{\Theta }}_r^{(s)}}{{\mathit{a}}_1^{(s)},\dots ,{\mathit{a}}_n^{(s)}\in {\mathrm{\Theta }}_a^{(s)}}}{\text{arg max}}{\mathcal{l}}_n(\mathit{Y};{\{{\mathit{a}}_i^{(s)}\}}_i,{\{{\mathit{R}}_k^{(s)}\}}_k),$$ (3)

$$\text{subject to}\left[{\mathit{a}}_1^{(s)},\dots ,{\mathit{a}}_s^{(s)}\right]={\mathit{I}}_s,$$ (4)

for some ${\mathrm{\Theta }}_a^{(s)}\subseteq {ℝ}^s,{\mathrm{\Theta }}_r^{(s)}\subseteq {ℝ}^{s^2}$, where the vec(·) operator stacks the entries of a matrix into a column vector. To estimate the number of latent factors, we define the following likelihood-based information criteria

$$\text{IC}(s)=2{\mathcal{l}}_n(\mathit{Y};{\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k)-s\kappa (n,K),$$

for some penalty functions κ(·,·). The estimated number of latent factors is given by

$$\widehat{s}=\underset{s\in \{1,\dots ,s_{\max }\}}{\text{arg max}}\mathrm{IC}(s).$$ (5)

In addition to the constraint in Equation 4, there exist many other constraints that would make the estimators identifiable. The choice of the identifiability constraints might affect the value of IC. However, it wouldn’t affect the value of $\widehat{s}$. Detailed discussions can be found in Section A of the Appendix.

A major technical difficulty in establishing the consistency of IC is due to the nonconcavity of the objective function given in Equation 3. For any ${\{{\mathit{a}}_j\}}_{j\in \{1,\dots ,n\}},{\left\{{\mathit{R}}_k\right\}}_{k\in \{1,\dots ,K\}}$, let

$$\mathit{\beta }={({\mathit{a}}_1^T,\dots ,{\mathit{a}}_n^T,\mathrm{vec}{({\mathit{R}}_1)}^T,\dots ,\mathrm{vec}{({\mathit{R}}_K)}^T)}^T,$$

be the set of parameters.

For any ${\mathit{b}}_1,\dots ,{\mathit{b}}_n\in {ℝ}^s,{\mathit{T}}_1,\dots ,{\mathit{T}}_K\in {ℝ}^{s\times s}$, we define

$$\mathit{\zeta }={({\mathit{b}}_1^T,\dots ,{\mathit{b}}_n^T,\mathrm{vec}{({\mathit{T}}_1)}^T,\dots ,\mathrm{vec}{({\mathit{T}}_K)}^T)}^T.$$

With some calculations, we can show that

$$\begin{gathered} -{\mathit{\zeta }}^T\frac{{\partial }^2{\mathcal{l}}_n}{\partial \mathit{\beta }\partial {\mathit{\beta }}^T}\mathit{\zeta }=\underset{I_1}{\underbrace{\sum _{ijk}{\pi }_{ijk}(1-{\pi }_{ijk}){({\mathit{b}}_i^T{\mathit{R}}_k{\mathit{a}}_j+{\mathit{a}}_i^T{\mathit{R}}_k{\mathit{b}}_j+{\mathit{a}}_i^T{\mathit{T}}_k{\mathit{a}}_j)}^2}} \\ +\underset{I_2}{\underbrace{\sum _{ijk}({\pi }_{ijk}-Y_{ijk})(2{\mathit{b}}_i^T{\mathit{R}}_k{\mathit{b}}_j+{\mathit{b}}_i^T{\mathit{T}}_k{\mathit{a}}_j+{\mathit{a}}_i^T{\mathit{T}}_k{\mathit{b}}_j)}}, \end{gathered}$$

where ${\pi }_{ijk}=\exp ({\mathit{a}}_i^T{\mathit{R}}_k{\mathit{a}}_j)/\{1+\exp ({\mathit{a}}_i^T{\mathit{R}}_k{\mathit{a}}_j)\}$. Here, I₁ is nonnegative. However, I₂ can be negative for some β and ζ. Therefore, the negative Hessian matrix is not positive semidefinite and the likelihood function is not concave. As a result, ${\hat{\mathit{a}}}_i^{s_0}$ and ${\hat{\mathit{R}}}_k^{s_0}$ may not be consistent to ${\hat{\mathit{a}}}_i^{*}$ and ${\hat{\mathit{R}}}_k^{*}$, even with the identifiability constraints in Equation 4. Here, the presence of I₂ is due to the bilinear formulation of the RESCAL model.

Let ${\theta }_{ijk}={\mathit{a}}_i^T{\mathit{R}}_k{\mathit{a}}_j$. Notice that ${\mathcal{l}}_n$ is concave in θ_ijk, ∀i, j, k. This motivates us to consider the following pseudometric:

$$\begin{gathered} d\left({\{{\mathit{a}}_{i,1}^{(s_1)}\}}_i,{\{{\mathit{R}}_{k,1}^{(s_1)}\}}_{k_1};{\{{\mathit{a}}_{i,2}^{(s_2)}\}}_i,{\{{\mathit{R}}_{k,2}^{(s_2)}\}}_{k_2}\right) \\ ={\left\{\frac{1}{n^{2}K}\sum _{ijk}{\left({({\mathit{a}}_{i,1}^{(s_1)})}^T{({\mathit{R}}_{k,1}^{(s_1)})}^T{\mathit{a}}_{j,1}^{(s_1)}-{({\mathit{a}}_{i,2}^{(s_2)})}^T{({\mathit{R}}_{k,2}^{(s_2)})}^T{\mathit{a}}_{j,2}^{(s_2)}\right)}^2\right\}}^{1/2}, \end{gathered}$$

for any integers s₁, s₂ > 0 and ${\mathit{a}}_{i,1}^{(s_1)}\in {ℝ}^{s_1}$, ${\mathit{R}}_{k,1}^{(s_1)}\in {ℝ}^{s_1\times s_1}$, ${\mathit{a}}_{i,2}^{(s_2)}\in {ℝ}^{s_2}$, ${\mathit{R}}_{k,2}^{(s_2)}\in {ℝ}^{s_2\times s_2}$. Apparently, d(·,·) is nonnegative, symmetric and satisfies the triangle inequality. Below, we establish the convergence rate of

$$d\left({\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k;{\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k\right).$$

We first introduce some notation. For any s > s₀, we define

$${\mathit{a}}_{i,0}^{(s)}=\{\begin{array}{ll}{({({\mathit{a}}_{i,0})}^T,{\mathbf{0}}_{s-s_0}^T)}^T,\hfill & i\notin \{s_0+1,\dots ,s\},\hfill \\ ({({\mathit{a}}_{i,0})}^T,\underset{i-s_0-1}{\underbrace{0,\dots ,0,}}1,\underset{s-i}{\underbrace{0,\dots ,0{)}^T}},\hfill & i\in \{s_0+1,\dots ,s\},\hfill \end{array}$$

and

$${\mathit{R}}_{k,0}^{(s)}=\left(\begin{array}{cc}{\mathit{R}}_{k,0}& {\mathit{O}}_{r,s-r}\\ {\mathit{O}}_{s-r,r}& {\mathit{O}}_{s-r,s-r}\end{array}\right),$$

where 0_q denotes a q-dimensional zero vector and O_p,q is an p × q zero matrix. With a slight abuse of notation, we write ${\mathit{a}}_{i,0}^{(s_0)}={\mathit{a}}_{i,0}$ and ${\mathit{R}}_{k,0}^{(s_0)}={\mathit{R}}_{k,0}$. Clearly, for any s ≥ s₀, we have

$${({\mathit{a}}_{i,0}^{(s)})}^T{\mathit{R}}_{k,0}^{(s)}{\mathit{a}}_{j,0}^{(s)}={\mathit{a}}_{i,0}^T{\mathit{R}}_{k,0}{\mathit{a}}_{j,0},\forall i,j,k,$$

and hence

$$({\{{\mathit{a}}_{i,0}^{(s)}\}}_i,{\{{\mathit{R}}_{k,0}^{(s)}\}}_k)=\underset{{\{{\mathit{a}}_i^{(s)}\}}_i,{\{{\mathit{R}}_k^{(s)}\}}_k}{\arg \max }\mathrm{E}{\mathcal{l}}_n(\mathit{Y};{\{{\mathit{a}}_i^{(s)}\}}_i,{\{{\mathit{R}}_k^{(s)}\}}_k),$$

Let

$${\mathit{a}}_i^{(s)*}={({\mathit{A}}_{s,0}^{-1})}^T{\mathit{a}}_{i,0}^{(s)}\text{and}{\mathit{R}}_k^{(s)*}={\mathit{A}}_{s,0}{\mathit{R}}_{k,0}^{(s)}{\mathit{A}}_{s,0}^T,$$

where ${\mathit{A}}_{s,0}={[{\mathit{a}}_{1,0}^{(s)},\dots ,{\mathit{a}}_{s,0}^{(s)}]}^T$. When ${\mathit{A}}_{s_0,0}$ is invertible, A_s,0’s are invertible for all s > s₀. The defined ${\{{\mathit{a}}_i^{(s)*}\}}^{\prime }$’s satisfy the identifiability constraints in Equation 4 for all s ≥ s₀. We make the following assumption.

(A1) Assume ${\mathit{a}}_i^{(s)*}\in {\mathrm{\Theta }}_a^{(s)}$ and $\mathrm{vec}({\mathit{R}}_k^{(s)*})\in {\mathrm{\Theta }}_r^{(s)}$, $\forall i=1,\dots ,n,k=1,\dots ,K$ and s₀ ≤ s ≤ s_max. In addition, assume ${\sup }_{\mathit{x}\in {\mathrm{\Theta }}_a^{(s)}}\Vert \mathit{x}{\Vert }_2\le {\omega }_a$, ${\sup }_{\mathit{y}\in {\mathrm{\Theta }}_r^{(s)}}\Vert \mathit{y}{\Vert }_2\le {\omega }_r$ for some ${\omega }_a,{\omega }_r>0$.

Lemma 2.

Assume (A1) holds, $s_{\max }=o\{\sqrt{n/\log (nK)}\}$. Then there exists some constant C₀ > 0 such that the following event occurs with probability tending to 1,

$$\underset{s\in \{s_0,\dots ,s_{\max }\}}{\max }\frac{1}{s^2}{d}^2\left({\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k;{\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k\right)\le \frac{\exp (2{\omega }_a^2{\omega }_r)(n+K)(\log n+\log K)}{n^{2}K}.$$

Under the condition $s_{\max }=o\{\sqrt{n/\log (nK)}\}$, we have that

$$\frac{s_{\max }^2(n+K)(\log n+\log K)}{n^{2}K}\le \frac{s_{\max }^2(2n\log n+2K\log K)}{n^{2}K}\le \frac{2s_{\max }^2\log n}{n}+\frac{2s_{\max }^2\log K}{n^2}=o(1).$$

When ω_a and ω_r are bounded, it follows that

$$\underset{s\in \{s_0,\dots ,s_{\max }\}}{\max }{d}^2\left({\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k;{\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k\right)\le \frac{s_{\max }^2\exp (2{\omega }_a^2{\omega }_r)(n+K)(\log n+\log K)}{n^{2}K}=o(1).$$

Hence, ${\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i$ and ${\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k$ are consistent under the pseudometric d for all overfitted models. On the contrary, for underfitted models, we require the following conditions.

(A2) Assume there exists some constant $\overline{c}>0$ such that ${\lambda }_{\min }({\mathit{A}}_0^T{\mathit{A}}_0)\ge n\overline{c}$.

(A3) Let $\overline{K}={\lambda }_{\min }(\sum _{k=1}^K{\mathit{R}}_{k,0}^T{\mathit{R}}_{k,0})$. Assume ${\lim \inf }_n\overline{K}>0$.

Lemma 3.

Assume (A2) and (A3) hold. The for any $s\in \{1,2,\dots ,s_0-1\}$, we have

$$d^2\left({\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k;{\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k\right)\ge \frac{{\overline{c}}^2\overline{K}}{K},$$

where $\overline{c}$ and $\overline{K}$ are defined in (A2) and (A3), respectively.

Assumption (A3) holds if there exists some $k_0\in \{1,\dots ,K\}$ such that

$${\lim \inf }_n{\lambda }_{\min }({\mathit{R}}_{k_0,0}{\mathit{R}}_{k_0,0}^T)>0.$$

When $\overline{K}\ge c^{\prime }K$ for some constant c^′ > 0, it follows from Lemma 3 that

$$\underset{n}{\lim \inf }d\left({\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k;{\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k\right)>0.$$

Based on these results, we establish the consistency of $\widehat{s}$ defined in Equation 5 below. For any sequences {a_n} and {b_n}, we write a_n ~ b_n if there exist some universal constants c₁, c₂ > 0 such that c₁a_n ≤ b_n ≤ c₂a_n.

Theorem 1.

Assume (A1)-(A3) hold, $K~n^{l_0}$ or some $0\le l_0\le 1,s_{\max }=o\{\sqrt{n/\log (nK)}\},\lim {\inf }_n{n}^{(1-l_0)/2}\overline{K}\ge \exp (2{\omega }_a^2{\omega }_r)\sqrt{\log n}$. Assume κ(n, K) satisfies

$$s_{\max }\exp ({\omega }_a^2{\omega }_r)(n+K)(\log n+\log K)\ll \kappa (n,K)\ll \frac{n^2\overline{K}}{\exp ({\omega }_a^2{\omega }_r)}.$$ (6)

Then, we have $\mathit{Pr}(\widehat{s}=s_0)\to 1$ where $\widehat{s}$ is defined in Equation 5.

Let $c(n,K)=\kappa (n,K){(n+K)}^{-1}{(\log n+\log K)}^{-1}$. When $s_{\max },{\omega }_a,{\omega }_r$ are bounded, it follows from Theorem 1 that IC is consistent provided that $c(n,K)\to \infty$ and $c(n,K)=o(n\overline{K}/\log n)$. Define

$${\tau }_{\alpha }(n,K)=\frac{{(n+K)}^{\alpha }}{{\max }^{\alpha }(n,K)},$$

for some α ≥ 0. Note that

$$1\le {\tau }_{\alpha }(n,K)\le 2^{\alpha }.$$ (7)

Consider the following criteria:

$${\text{IC}}_{\alpha }(s)=2{\mathcal{l}}_n(\mathit{Y};{\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k)-s{\tau }_{\alpha }(n,K)(n+K)(\log n+\log K)\log \{\log (nK)\}.$$ (8)

Note that the term $\log \{\log (nK)\}$ satisfies that $\log \{\log (nK)\}\to \infty$ and $\log \{\log (nK)\}=o(n/\log n)$. It follows from Equation 7 and Theorem 1 that IC_α is consistent for all α ≥ 0. When α > 0, the term ${\tau }_{\alpha }(n,K)$ adjust the model complexity penalty upwards. We notice that Bai and Ng (2002) used a similar finite sample correction term in their proposed information criteria for approximate factor models. Our simulation studies show that such adjustment is essential to achieve selection consistency for large K.

Conditions (A1) and (A2) are directly imposed on the realizations of ${\mathit{a}}_{1,0},\dots ,{\mathit{a}}_{n,0}$. In Section B.2 and B.3, we consider an asymptotic framework where ${\mathit{a}}_{1,0},\dots ,{\mathit{a}}_{n,0}$ are i.i.d according to some distribution function and show (A1) and (A2) hold with probability tending to 1. Therefore, under this framework, we still have $\mathrm{Pr}(\widehat{s}=s_0)\to 1$. The consistency of our information criterion remains unchanged.

Observe that we have a total of n × n × K = n²K observations. Consider the following BIC-type criterion:

$$\text{BIC}(s)=2{\mathcal{l}}_n(\mathit{Y};{\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k)-s\log (n^{2}K).$$ (9)

The model complexity penalty in BIC satisfies

$$\log (n^{2}K)=2\log n+\log K\ll (n+K)(\log n+\log K).$$

Hence, it does not meet Condition (6) in Theorem 1. As a result, BIC may fail to identity the true model. As shown in our simulation studies, BIC will choose overfitted models and is not selection consistent.

4. Numerical Experiments

This section is organized as follows. In Section 4.1, we introduce our algorithm for computing the maximum likelihood estimators of a logistic RESCAL model. Simulation studies are presented in Section 4.2. In Section 4.3, we apply the proposed information criteria to a real dataset.

4.1. Implementation

In this section, we propose an algorithm for computing ${\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i$ and ${\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k$. The algorithm is based upon a 3-block alternating direction method of multipliers (ADMM). Set ${\mathrm{\Theta }}_a^{(s)}={ℝ}^s$, ${\mathrm{\Theta }}_r^{(s)}={ℝ}^{s\times s}$, $[{\mathit{a}}_1^{(s)},\dots ,{\mathit{a}}_s^{(s)}]={\mathit{I}}_s$, ${\widehat{\mathit{a}}}_i^{(s)}$’s and ${\widehat{\mathit{R}}}_k^{(s)}$’s are defined by

$$({\{{\widehat{\mathit{a}}}_i^{(s)}\}}_{i=(s+1)}^n,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k)=\underset{{\{{\mathit{a}}_i^{(s)}\}}_{i=s+1}^n,{\{{\mathit{R}}_k^{(s)}\}}_k}{\arg \max }{\mathcal{l}}_n(\mathit{Y};{\{{\mathit{a}}_i^{(s)}\}}_i,{\{{\mathit{R}}_k^{(s)}\}}_k),$$ (10)

where

$${\mathcal{l}}_n(\mathit{Y};{\{{\mathit{a}}_i^{(s)}\}}_i,{\{{\mathit{R}}_k^{(s)}\}}_k)=\sum _{ijk}\left(Y_{ijk}{({\mathit{a}}_i^{(s)})}^T{\mathit{R}}_k^{(s)}{\mathit{a}}_j^{(s)}-\log [1+\exp \{{({\mathit{a}}_i^{(s)})}^T{\mathit{R}}_k^{(s)}{\mathit{a}}_j^{(s)}\}]\right).$$

For any ${\mathit{b}}_1,\dots ,{\mathit{b}}_n\in {ℝ}^s$, define

$${\overline{\mathcal{l}}}_n(\mathit{Y};{\{{\mathit{a}}_i^{(s)}\}}_i,{\{{\mathit{R}}_k^{(s)}\}}_k,{\{{\mathit{b}}_i^{(s)}\}}_i)=\sum _{ijk}\left(Y_{ijk}{({\mathit{a}}_i^{(s)})}^T{\mathit{R}}_k^{(s)}{\mathit{b}}_j^{(s)}-\log [1+\exp \{{({\mathit{a}}_i^{(s)})}^T{\mathit{R}}_k^{(s)}{\mathit{b}}_j^{(s)}\}]\right).$$

Fix $[{\mathit{b}}_1^{(s)},\dots ,{\mathit{b}}_s^{(s)}]={\mathit{I}}_s$, the optimization problem in Equation 10 is equivalent to

$$({\{{\widehat{\mathit{a}}}_i^{(s)}\}}_{i=s+1}^n,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k,{\{{\widehat{\mathit{b}}}_i^{(s)}\}}_{i=s+1}^n)=\underset{\begin{array}{c}{\{{\mathit{a}}_i^{(s)}\}}_{i=s+1}^n,{\{{\mathit{R}}_k^{(s)}\}}_k\\ {\{{\mathit{b}}_i^{(s)}\}}_{i=s+1}^n\end{array}}{\arg \max }{\overline{\mathcal{l}}}_n(\mathit{Y};{\{{\mathit{a}}_i^{(s)}\}}_i,{\{{\mathit{R}}_k^{(s)}\}}_k,{\{{\mathit{b}}_i^{(s)}\}}_i)$$

$$\text{subject to}{\mathit{a}}_i^{(s)}={\mathit{b}}_i^{(s)},\forall i=s+1,\dots ,n.$$

We then derive its augmented Lagrangian, which gives us

$$\begin{gathered} L_{\rho }({\{{\mathit{a}}_i^{(s)}\}}_{i=s+1}^n,{\{{\mathit{R}}_k^{(s)}\}}_k,{\{{\mathit{b}}_i^{(s)}\}}_{i=s+1}^n,{\{{\mathit{v}}_i^{(s)}\}}_{i=s+1}^n) \\ =-{\overline{\mathcal{l}}}_n(\mathit{Y};{\{{\mathit{a}}_i^{(s)}\}}_i,{\{{\mathit{R}}_k^{(s)}\}}_k,{\{{\mathit{b}}_i^{(s)}\}}_i)+\sum _{i=s+1}^n\rho {({\mathit{a}}_i^{(s)}-{\mathit{b}}_i^{(s)})}^T{\mathit{v}}_i^{(s)}+\sum _{i=s+1}^n\frac{\rho }{2}{\Vert {\mathit{a}}_i^{(s)}-{\mathit{b}}_i^{(s)}\Vert }_2^2, \end{gathered}$$

where ρ > 0 is a penalty parameter and, ${\mathit{v}}_{s+1}^{(s)},\dots ,{\mathit{v}}_n^{(s)}\in {ℝ}^s$.

Applying dual descent method yields the following steps, with l denotes the iteration number:

$${\{{\mathit{a}}_{i,l+1}^{(s)}\}}_{i=s+1}^n=\underset{{\{{\mathit{a}}_i^{(s)}\}}_{i=s+1}^n}{\arg \min }{L}_{\rho }({\{{\mathit{a}}_i^{(s)}\}}_{i=s+1}^n,{\{{\mathit{R}}_{k,l}^{(s)}\}}_k,{\{{\mathit{b}}_{i,l}^{(s)}\}}_{i=s+1}^n,{\{{\mathit{v}}_{i,l}^{(s)}\}}_{i=s+1}^n),$$ (11)

$${\{{\mathit{R}}_{k,l+1}^{(s)}\}}_{k=1}^K=\underset{{\{{\mathit{R}}_k^{(s)}\}}_{k=1}^K}{\arg \min }{L}_{\rho }({\{{\mathit{a}}_{i,l+1}^{(s)}\}}_{i=s+1}^n,{\{{\mathit{R}}_k^{(s)}\}}_k,{\{{\mathit{b}}_{i,l}^{(s)}\}}_{i=s+1}^n,{\{{\mathit{v}}_{i,l}^{(s)}\}}_{i=s+1}^n),$$ (12)

$${\{{\mathit{b}}_{i,l+1}^{(s)}\}}_{i=s+1}^n=\underset{{\{{\mathit{b}}_i^{(s)}\}}_{i=s+1}^n+1}{\arg \min }{L}_{\rho }({\{{\mathit{a}}_{i,l+1}^{(s)}\}}_{i=s+1}^n,{\{{\mathit{R}}_{k,l+1}^{(s)}\}}_k,{\{{\mathit{b}}_i^{(s)}\}}_{i=s+1}^n,{\{{\mathit{v}}_{i,l}^{(s)}\}}_{i=s+1}^n),$$ (13)

$${\mathit{v}}_{i,l+1}^{(s)}={\mathit{v}}_{i,l}^{(s)}+{\mathit{a}}_{i,l}^{(s)}-{\mathit{b}}_{i,l}^{(s)},\forall i=s+1,\dots ,n.$$

Let us examine Equation 11–13 in more details. In Equation 11, we rewrite the objective function as

$$\begin{gathered} L_{\rho }({\{{\mathit{a}}_i^{(s)}\}}_{i=s+1}^n,{\{{\mathit{R}}_{k,l}^{(s)}\}}_k,{\{{\mathit{b}}_{i,l}^{(s)}\}}_{i=s+1}^n,{\{{\mathit{v}}_{i,l}^{(s)}\}}_{i=s+1}^n) \\ =\sum _{i=s+1}^n\{\sum _{j,k}\left(\log [1+\exp \{{({\mathit{a}}_i^{(s)})}^T{\mathit{R}}_{k,l}^{(s)}{\mathit{b}}_{j,l}^{(s)}\}]-Y_{ijk}{({\mathit{a}}_i^{(s)})}^T{\mathit{R}}_{k,l}^{(s)}{\mathit{b}}_{j,l}^{(s)}\right) \\ +\rho {({\mathit{a}}_i^{(s)}-{\mathit{b}}_{i,l}^{(s)})}^T{\mathit{v}}_{i,l}^{(s)}+\frac{\rho }{2}{\Vert {\mathit{a}}_i^{(s)}-{\mathit{b}}_{i,l}^{(s)}\Vert }_2^2\}. \end{gathered}$$

Note that L_ρ can be represented as a separable sum of functions. As a result, ${\mathit{a}}_{i,l+1}^{(s)}$‘s can be solved in parallel. More specifically, we have

$$\begin{gathered} {\mathit{a}}_{i,l+1}^{(s)}=\underset{{\mathit{a}}^{(s)}}{\arg \min }\{\sum _{j,k}\left(\log [1+\exp \{{({\mathit{a}}_i^{(s)})}^T{\mathit{R}}_{k,l}^{(s)}{\mathit{b}}_{j,l}^{(s)}\}]-Y_{ijk}{({\mathit{a}}_i^{(s)})}^T{\mathit{R}}_{k,l}^{(s)}{\mathit{b}}_{j,l}^{(s)}\right) \\ +\rho {({\mathit{a}}_i^{(s)}-{\mathit{b}}_{i,l}^{(s)})}^T{\mathit{v}}_{i,l}^{(s)}+\frac{\rho }{2}{\Vert {\mathit{a}}_i^{(s)}-{\mathit{b}}_{i,l}^{(s)}\Vert }_2^2\}. \end{gathered}$$

Hence, each ${\mathit{a}}_{i,l+1}^{(s)}$ can be computed by solving a ridge type logistic regression with responses ${\{Y_{ijk}\}}_{j,k}$ and covariates ${\{{\mathit{R}}_{k,l}^{(s)}{\mathit{b}}_{j,l}^{(s)}\}}_{j,k}$.

In Equation 12, each ${\mathit{R}}_{k,l+1}^{(s)}$ can be independently updated by solving a logistic regression with responses ${\{Y_{ijk}\}}_{i,j}$ and covariates ${\mathit{b}}_{j,l}^{(s)}\otimes {\mathit{a}}_{i,l+1}^{(s)}$, i.e,

$$\mathrm{vec}({\mathit{R}}_{k,l+1}^{(s)})=\underset{{\mathit{r}}_k^{(s)}\in {ℝ}^{s^2}}{\arg \min }\sum _{ij}\left\{\log \left(1+\exp [\{{({\mathit{b}}_{j,l}^{(s)})}^T\otimes {({\mathit{a}}_{i,l+1}^{(s)})}^T\}{\mathit{r}}_k^{(s)}]\right)-Y_{ijk}\{{({\mathit{b}}_{j,l}^{(s)})}^T\otimes {({\mathit{a}}_{i,l+1}^{(s)})}^T\}{\mathit{r}}_k^{(s)}\right\},$$

where ⊗ denotes the Kronecker product.

Similar to Equation 11, each ${\mathit{b}}_{i,l+1}^{(s)}$ in Equation 13 can be independently computed by solving a ridge type regression with responses ${\{Y_{ijk}\}}_{j,k}$ and covariates ${\{{({\mathit{R}}_{k,l+1}^{(s)})}^T{\mathit{a}}_{j,l+1}^{(s)}\}}_{j,k}$.

Using similar arguments in Theorem 2 in Wang et al. (2017), we can show that the proposed 3-block ADMM algorithm converges for any sufficiently large ρ. In our implementation, we set ρ = nK/2. To guarantee global convergence, we randomly generate multiple initial estimators and solve the optimization problem multiple times based on these initial values.

4.2. Simulations

We simulate ${\{Y_{ijk}\}}_{ijk}$ from the following model:

$$\mathrm{Pr}\left(Y_{ijk}=1|{\left\{{\mathit{a}}_i\right\}}_i,{\left\{{\mathit{R}}_k\right\}}_k\right)=\frac{\exp \left({\mathit{a}}_i^T{\mathit{R}}_k{\mathit{a}}_j\right)}{1+\exp \left({\mathit{a}}_i^T{\mathit{R}}_k{\mathit{a}}_j\right)},$$

$${\mathit{a}}_1,{\mathit{a}}_2,\dots ,{\mathit{a}}_n\stackrel{iid}{~}N(0,1),$$

$${\mathit{R}}_1={\mathit{R}}_2=\cdots ={\mathit{R}}_K=\mathrm{diag}(\underset{s_0}{\underbrace{1,-1,1,-1,\dots ,1,-1}})$$

where N(0, 1) stands for a standard normal random variable and $\mathrm{diag}(v_1,\dots ,v_q)$ denotes a q × q diagonal matrix with the jth element equal to v_j.

We consider six simulation settings. In the first three settings, we fix K = 3 and set n = 100,150 and 200, respectively. In the last three settings, we increase K to 10, 20, 50, and set n = 50. In each setting, we further consider three scenarios, by setting s₀ = 2, 4 and 8. Let s_max = 12. The ADMM algorithm proposed in Section 4.1 is implemented in R. Some subroutines of the algorithm are written in C with the GNU Scientific Library (GSL, Galassi et al., 2015) to facilitate the computation. We compare the proposed IC_α (see Equation 8) with the BIC-type criterion (see Equation 9). In IC_α, we set α = 0, 0.5 and 1. Note that when α = 0, we have

$${\tau }_{\alpha }(n,K)=\frac{{(n+K)}^{\alpha }}{{\max }^{\alpha }(n,K)}=1.$$

Reported in Table 1 and 2 are the percentage of selecting the true models (TP) and the average of $\widehat{s}$ selected by IC₀, IC_0.5, IC₁ and BIC over 100 replications.

Table 1:

Simulation results for Setting I, II and III (standard errors in parenthesis)

	s0 = 2		s0 = 4		s0 = 8
n = 100, K = 3	TP	$\widehat{s}$	TP	$\widehat{s}$	TP	$\widehat{s}$
IC0	0.97 (0.02)	2.03 (0.02)	0.97 (0.02)	4.03 (0.02)	0.90(0.03)	7.90 (0.03)
IC0.5	0.97 (0.02)	2.03 (0.02)	0.98 (0.01)	4.02 (0.01)	0.90(0.03)	7.90 (0.03)
IC1	0.97 (0.02)	2.03 (0.02)	0.98 (0.01)	4.02 (0.01)	0.89(0.03)	7.89 (0.03)
BIC	0.00 (0.00)	11.99 (0.01)	0.00 (0.00)	12.00 (0.00)	0.00 (0.00)	11.99 (0.01)
n = 150, K = 3	TP	$\widehat{s}$	TP	$\widehat{s}$	TP	$\widehat{s}$
IC0	0.99 (0.01)	2.01 (0.01)	0.97 (0.02)	4.03 (0.02)	0.96(0.02)	8.04 (0.02)
IC0.5	0.99 (0.01)	2.01 (0.01)	0.97 (0.02)	4.03 (0.02)	0.96(0.02)	8.04 (0.02)
IC1	0.99 (0.01)	2.01 (0.01)	0.97 (0.02)	4.03 (0.02)	0.96(0.02)	8.04 (0.02)
BIC	0.00 (0.00)	12.00 (0.00)	0.00 (0.00)	12.00 (0.00)	0.00 (0.00)	11.98 (0.01)
n = 200, K = 3	TP	$\widehat{s}$	TP	$\widehat{s}$	TP	$\widehat{s}$
IC0	0.99 (0.01)	2.01 (0.01)	0.95 (0.02)	4.05 (0.02)	0.95(0.02)	8.05 (0.02)
IC0.5	0.99 (0.01)	2.01 (0.01)	0.95 (0.02)	4.05 (0.02)	0.95(0.02)	8.05 (0.02)
IC1	0.99 (0.01)	2.01 (0.01)	0.95 (0.02)	4.05 (0.02)	0.95(0.02)	8.05 (0.02)
BIC	0.00 (0.00)	12.00 (0.00)	0.00 (0.00)	11.99 (0.01)	0.00 (0.00)	11.98 (0.01)

Table 2:

Simulation results for Setting IV, V and VI (standard errors in parenthesis)

	s0 = 2		s0 = 4		s0 = 8
n = 50, K = 10	TP	$\widehat{s}$	TP	$\widehat{s}$	TP	$\widehat{s}$
IC0	1.00 (0.00)	2.00 (0.00)	0.97 (0.02)	4.03 (0.02)	0.69(0.05)	7.91 (0.06)
IC0.5	1.00 (0.00)	2.00 (0.00)	0.97 (0.02)	4.03 (0.02)	0.66(0.05)	7.75 (0.06)
IC1	1.00 (0.00)	2.00 (0.00)	0.98 (0.01)	4.02 (0.01)	0.60(0.05)	7.62 (0.06)
BIC	0.00 (0.00)	11.81 (0.06)	0.00 (0.00)	11.60 (0.06)	0.01 (0.01)	11.67 (0.07)
n = 50, K = 20	TP	$\widehat{s}$	TP	$\widehat{s}$	TP	$\widehat{s}$
IC0	0.97 (0.02)	2.03 (0.02)	0.95 (0.02)	4.05 (0.02)	0.73(0.04)	8.46 (0.10)
IC0.5	0.97 (0.02)	2.03 (0.02)	0.98 (0.01)	4.02 (0.01)	0.87(0.03)	8.09 (0.03)
IC1	0.98 (0.01)	2.02 (0.02)	1.00 (0.00)	4.00 (0.00)	0.79(0.04)	7.99 (0.05)
BIC	0.00 (0.00)	12.00 (0.00)	0.00 (0.00)	11.92 (0.03)	0.00 (0.00)	11.99 (0.01)
n = 50, K = 50	TP	$\widehat{s}$	TP	$\widehat{s}$	TP	$\widehat{s}$
IC0	0.98 (0.01)	2.02 (0.01)	0.93 (0.03)	4.07 (0.03)	0.17(0.04)	11.24 (0.15)
IC0.5	0.99 (0.01)	2.01 (0.01)	0.97 (0.02)	4.03 (0.02)	0.76(0.04)	8.24 (0.05)
IC1	1.00 (0.00)	2.00 (0.00)	0.98 (0.01)	4.02 (0.01)	0.79(0.04)	7.99 (0.05)
BIC	0.00 (0.00)	12.00 (0.00)	0.00 (0.00)	12.00 (0.00)	0.00 (0.00)	11.99 (0.01)

It can be seen from Table 1 and 2 that BIC fails in all settings. It always selects overfitted models. On the contrary, the proposed information criteria are consistent for most of the settings. For example, under settings where s₀ = 2 or 4, TPs of IC₀, IC_0.5 and IC₁ are larger than or equal to 93%. When s₀ = 8, expect for the last setting, TPs of the proposed information criteria are no less than 60% for all cases.

IC₀, IC_0.5 and IC₁ perform very similarly for small K. In the first three settings, TPs of these three information criteria are nearly the same for all cases. However, IC_0.5 and IC₁ are more robust than IC₀ for large K. This can be seen in the last scenario of Setting 6, where the TP of IC₀ is no more than 20%. Besides, in the last two settings, TP of IC₀ is smaller than IC_0.5 and IC₁ for all cases. These differences are due to the finite sample correction term ${\tau }_{\alpha }(n,K)$. As commented before, ${\tau }_{0.5}(n,K)$ and ${\tau }_1(n,K)$ increase the model complexity penalty term in IC_0.5 and IC₁ to avoid overfitting for large K.

In Section D of the Appendix, we examine the performance of our proposed information criteria under the scenario where ${\mathit{a}}_1,{\mathit{a}}_2,\dots ,{\mathit{a}}_n\stackrel{iid}{~}N(0,{\{{0.5}^{|i-j|}\}}_{i,j=1,\dots ,s_0})$. Results are similar to those presented at Table 1 and 2.

4.3. Real Data Experiments

In this section, we apply the proposed information criteria to the “Social Evolution” dataset (Madan et al., 2012). This dataset comes from MIT’s Human Dynamics Laboratory. It tracks everyday life of a whole undergraduate MIT dormitory from October 2008 to May 2009. We use the survey data, resulting in n = 84 participants and K = 5 binary relations. The five relations are: close relationship, political discussion, social interaction and two social media interaction.

We compute ${\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i\text{and}{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k$ for $s=\{1,\dots ,12\}$ and select the number of latent factors using the proposed information criteria and BIC. It turns out that IC₀, IC_0.5 and IC₁ all suggest the presence of 9 factors. In contrast, BIC selects 12 factors. To further evaluate the number of latent factors selected by the proposed information criteria, we consider the following cross-validation procedure. For any $s\in [1,\dots ,12]$, we randomly select 80% of the observations and estimate ${\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i$ and $\{{\widehat{\mathit{R}}}_k^{(s)}\}$ by maximizing the observed likelihood function based on these training samples. Then we compute

$${\widehat{\pi }}_{ijk}=\frac{\exp \{{({\widehat{\mathit{a}}}_i^{(s)})}^T{\widehat{\mathit{R}}}_k^{(s)}{\widehat{\mathit{a}}}_j^{(s)}\}}{1+\exp \{{({\widehat{\mathit{a}}}_i^{(s)})}^T{\widehat{\mathit{R}}}_k^{(s)}{\widehat{\mathit{a}}}_j^{(s)}\}}.$$

Based on these predicted probabilities, we calculate the area under the precision-recall curve (AUC) on the remaining 20% testing samples.

Reported in Table 3 are the AUC scores averaged over 100 replications. For any $s\in \{1,\dots ,12\}$, we denoted by AUC_s the corresponding AUC score. It can be seen from Table 3 that AUC_s first increases and then decreases as s increases. The maximum AUC score is achieved at s = 10. Observe that AUC₉ is very close to AUC₁₀, and it is larger than the remaining AUC scores. This demonstrates that the proposed information criteria select less latent factors while achieve better or similar link prediction results when compared to BIC.

Table 3:

AUC scores

s	1	2	3	4	5	6	7	8	9	10	11	12
AUC	0.7201	0.8341	0.8952	0.9095	0.9257	0.9364	0.9444	0.9486	0.9513	0.9518	0.9485	0.9467

5. Discussion

In this paper, we propose information criteria for selecting the number of latent factors in the RESCAL tensor factorization model and prove their model selection consistency. Although we focus on the logistic RESCAL model, the proposed information criteria can be applied to general tensor factorization models. More specifically, consider the following class of models:

$$Y_{ijk}=\text{g}({\mathit{a}}_{i,0}^T{\mathit{R}}_{k,0}{\mathit{b}}_{j,0})+e_{ijk},\forall i,j\in \{1,\dots ,n\},k\in \{1,\dots ,K\},$$ (14)

with any of (or without) the following constraints:

(C1) ${\mathit{R}}_{k,0}$ is diagonal;

(C2) ${\mathit{a}}_{i,0}={\mathit{b}}_{i,0}$ for $i\in \{1,\dots ,n\}$,

for some strictly increasing function g, ${\mathit{a}}_{i,0},{\mathit{b}}_{i,0}\in {ℝ}^{s_0},{\mathit{R}}_{k,0}\in {ℝ}^{s_0\times s_0}$ and some mean zero random errors ${\{e_{ijk}\}}_{ijk}$.

As commented in Section 2.1, such representation includes the RESCAL, CP and TUCKER-2 models. Specifically, it reduces to the TUCKER-2 model by setting g to be the identity function. If further (C1) holds, then the model in Equation 14 reduces to the CP model. When (C2) holds, it corresponds to the RESCAL model. Consider the following information criteria,

$$\text{IC}(s)={\mathcal{l}}_n(\mathit{Y};{\{{\widehat{\mathit{a}}}_i\}}_i,{\{{\widehat{\mathit{R}}}_k\}}_k,{\{{\widehat{\mathit{b}}}_i\}}_i)-s\kappa (n,K),$$

where ${\mathcal{l}}_n$ stands for the likelihood function and ${\widehat{\mathit{a}}}_i,{\widehat{\mathit{R}}}_k,{\widehat{\mathit{b}}}_i$ are the corresponding (constrained) MLEs. Similar to Theorem 1, we can show that with some properly chosen $\kappa (n,K)$, IC is consistent under this general setting.

Currently, we assume the tensor Y is completely observed. When some of the Y_ijk’s are missing, we can calculate ${\widehat{\mathit{a}}}_i^{(s)}$’s and ${\widehat{\mathit{R}}}_k^{(s)}$’s by maximizing the following observed likelihood function

$$\underset{\begin{array}{c}{\mathit{a}}_1^{(s)},\dots ,{\mathit{a}}_n^{(\mathit{a})}\in {\mathrm{\Theta }}_{\mathit{a}}^{(s)}\\ \mathrm{vec}({\mathit{R}}_1^{(s)}),\dots ,\mathrm{vec}({\mathit{R}}_K^{(s)})\in {\mathrm{\Theta }}_{\mathit{R}}^{(s)}\end{array}}{\arg \max }{\sum }_{(i,j,k)\in N_o{}_{bs}}\left(Y_{ijk}{({\mathit{a}}_i^{(s)})}^T{\mathit{R}}_k^{(s)}{\mathit{a}}_j^{(s)}-\log [1+\exp \{{({\mathit{a}}_i^{(s)})}^T{\mathit{R}}_k^{(s)}{\mathit{a}}_j^{(s)}\}]\right),$$

$$\text{subject to}[{\mathit{a}}_1^{(s)},\dots ,{\mathit{a}}_s^{(s)}]={\mathit{I}}_s,$$

where N_obs denotes the set of the observed responses. The above optimization problem can also be solved by a 3-block ADMM algorithm. Define the following class of information criteria,

$${\text{IC}}_{obs}(s)=\sum _{(i,j,k)\in N_{obs}}\left(Y_{ijk}{({\widehat{\mathit{a}}}_i^{(s)})}^T{\widehat{\mathit{R}}}_k^{(s)}{\widehat{\mathit{a}}}_j^{(s)}-\log [1+\exp \{{({\widehat{\mathit{a}}}_i^{(s)})}^T{\widehat{\mathit{R}}}_k^{(s)}{\widehat{\mathit{a}}}_j^{(s)}\}]\right)-\widehat{p}s\kappa (n,K),$$

where $\widehat{p}=\left|N_{obs}\right|/(n^{2}K)$ denotes the percentage of observed responses. Consistency of IC_obs can be similarly studied.

Appendix A. More on the Identifiability Constraint

Let π(·) be a permutation function of $\{1,\dots ,n\}$. Alternative to our estimator defined in Equation 3, one may consider

$$({\{{\widehat{\mathit{a}}}_{i,\pi }^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_{k,\pi }^{(s)}\}}_k)=\underset{\underset{\mathrm{Vec}({\mathit{R}}_1^{(s)}),\dots ,\mathrm{Vec}({\mathit{R}}_K^{(s)})\in {\mathrm{\Theta }}_r^{(s)}}{{\mathit{a}}_1^{(s)},\dots ,{\mathit{a}}_n^{(s)}\in {\mathrm{\Theta }}_{\mathit{a}}^{(s)}}}{\arg \max }{\mathcal{l}}_n(\mathit{Y};{\{{\mathit{a}}_i^{(s)}\}}_i,{\{{\mathit{R}}_k^{(s)}\}}_k),$$

$$\text{subject to}[{\mathit{a}}_{\pi (1)}^{(s)},\dots ,{\mathit{a}}_{\pi (s)}^{(s)}]={\mathit{I}}_s,$$

and the corresponding information criteria

$${\text{IC}}_{\pi }(s)=2{\mathcal{l}}_n(\mathit{Y};{\{{\widehat{\mathit{a}}}_{i,\pi }^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_{k,\pi }^{(s)}\}}_k)-s\kappa (n,K).$$

Since ${\mathcal{l}}_n(\mathit{Y};{\{{\mathit{a}}_i^{(s)}\}}_i,{\mathit{R}}_k^{(s)}{}_k)={\mathcal{l}}_n(\mathit{Y};{\{{\mathit{C}}^{-1}{\mathit{a}}_i^{(s)}\}}_i,{\{{\mathit{C}}^T{\mathit{R}}_k^{(s)}\mathit{C}\}}_k)$ for any invertible matrix $\mathit{C}\in {ℝ}^{s\times s},({\{{\widehat{\mathit{a}}}_{i,\pi }^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_{k,\pi }^{(s)}\}}_k)$ is also the maximizer of ${\mathcal{l}}_n(\mathit{Y};{\{{\mathit{a}}_i^{(s)}\}}_i,{\{{\mathit{R}}_k^{(s)}\}}_k)$ subject to the constraint that $[{\mathit{a}}_{\pi (1)}^{(s)},\dots ,{\mathit{a}}_{\pi (s)}^{(s)}]$ is invertible. Similarly, the estimator $({\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k)$ is the maximizer of ${\mathcal{l}}_n(\mathit{Y};{\{{\mathit{a}}_i^{(s)}\}}_i,{\{{\mathit{R}}_k^{(s)}\}}_k)$ subject to the constraint that $[{\mathit{a}}_1^{(s)},\dots ,{\mathit{a}}_s^{(s)}]$ is invertible. As a result, we have IC(s) = IC_π(s) as long as

$$[{\widehat{\mathit{a}}}_{\pi (1)}^{(s)},\dots ,{\widehat{\mathit{a}}}_{\pi (s)}^{(s)}]\text{is invertible and}[{\widehat{\mathit{a}}}_{1,\pi }^{(s)},\dots ,{\widehat{\mathit{a}}}_{s,\pi }^{(s)}]\text{is invertible}.$$ (15)

However, it remains unknown whether Equation 15 holds or not. Hence, there’s no guarantee that IC(s) = IC_π(s). This means the choice of the identifiability constraint might affect the value of our proposed information criterion.

In the following, we prove

$$\mathrm{Pr}\left(\underset{\begin{array}{c}\pi \{1,\dots ,n\}\to \{1,\dots ,n\}\\ \left|\right\{\pi (1),\dots ,\pi (n)\left\}\right|=n\end{array}}{\cap }\{\underset{s=1,\dots ,s_{\max }}{\arg \max }\{{\mathrm{IC}}_{\pi (s)}\}=s_0\}\right)\to 1.$$ (16)

This means with probability tending to 1, all the information criteria with different identifiability constraints will select the true model. Therefore, the choice of the identifiability constraint will not affect the performance of our method. For any $s\in \{s_0,\dots ,s_{\max }\}$, let ${\mathit{A}}_{s,0,\pi }={[{\mathit{a}}_{\pi (1),0}^{(s)},\dots ,{\mathit{a}}_{\pi (s),0}^{(s)}]}^T$,

$${\mathit{a}}_{i,\pi }^{(s)*}={({\mathit{A}}_{s,0,\pi }^{-1})}^T{\mathit{a}}_{i,0}^{(s)}\text{and}{\mathit{R}}_{k,\pi }^{(s)*}={\mathit{A}}_{s,0,\pi }{\mathit{R}}_{k,0}^{(s)}{\mathit{A}}_{s,0,\pi }^T.$$

We need the following condition.

(A4) Assume ${\mathit{a}}_{i,\pi }^{(s)*}\in {\mathrm{\Theta }}_a^{(s)}$ and $\mathrm{vec}\left({\mathit{R}}_{k,\pi }^{(s)*}\right)\in {\mathrm{\Theta }}_r^{(s)},\forall i=1,\dots ,n,k=1,\dots ,K,s_0\le s\le s_{\max }$ and any permutation function π(·). In addition, assume ${\sup }_{\mathit{x}\in {\mathrm{\Theta }}_a^{(s)}}\Vert \mathit{x}{\Vert }_2\le {\omega }_a$, ${\sup }_{\mathit{y}\in {\mathrm{\Theta }}_r^{(s)}}\Vert \mathit{y}{\Vert }_2\le {\omega }_r$ for some ${\omega }_a,{\omega }_r>0$.

Corollary 1.

Assume (A2)-(A4) hold, $K~n^{l_0}$ for some $0\le l_0\le 1,s_{\max }=o\{\sqrt{n/\log (nK)}\}\lim {\inf }_n{n}^{\left(1-l_0\right)/2}\overline{K}\ge \exp \left(2{\omega }_a^2{\omega }_r\right)\sqrt{\log n}$. Assume $\kappa (n,K)$ satisfies Condition (6). Then, (16) is satisfied.

Appendix B. More on the Technical Conditions

B.1. Discussion of Condition (A0)

In this section, we show the necessity of (A0) when the matrices ${\mathit{R}}_{1,0},\dots ,{\mathit{R}}_{K,0}$ are symmetric. More specifically, when (A0) doesn’t hold, we show there exist some $0\le s<s_0,{\overline{\mathit{a}}}_{1,0},\dots ,{\overline{\mathit{a}}}_{n,0}\in {ℝ}^s$ and ${\overline{\mathit{R}}}_{1,0},\dots ,{\overline{\mathit{R}}}_{K,0}\in {ℝ}^{s\times s}$ such that

$${\overline{\mathit{a}}}_{i,0}^T{\overline{\mathit{R}}}_{k,0}{\overline{\mathit{a}}}_{j,0}={\mathit{a}}_{i,0}^T{\mathit{R}}_{k,0}{\mathit{a}}_{j,0},\forall 1\le i,j\le n,1\le k\le K.$$ (17)

Let’s first consider the case where rank(A₀) = s for some s < s₀. Thus, it follows that

$${\mathit{A}}_0={\overline{\mathit{A}}}_0\mathit{C},$$

for some ${\overline{\mathit{A}}}_0\in {ℝ}^{n\times s}$ and $\mathit{C}\in {ℝ}^{s\times s_0}$. Set ${\overline{\mathit{a}}}_{i,0}$ to be the ith row of ${\overline{\mathit{A}}}_0$ and ${\overline{\mathit{R}}}_{k,0}=\mathit{C}{\mathit{R}}_{k,0}{\mathit{C}}^T$, Equation 17 is thus satisfied. In addition, the new matrix ${\overline{\mathit{A}}}_0$ shall have full column rank.

Let ${\mathit{R}}_0={({\mathit{R}}_{1,0}^T,\dots ,{\mathit{R}}_{K,0}^T)}^T,$. Consider the case where rank(R₀) = s for some s < s₀. It follows from the singular value decomposition that

$${\mathit{R}}_0={\mathit{U}}_0{\mathbf{\Lambda }}_0{\mathit{V}}_0^T,$$ (18)

for some diagonal matrix ${\mathbf{\Lambda }}_0\in {ℝ}^{s\times s}$ and some matrices ${\mathit{U}}_0\in {ℝ}^{K{s}_0\times s}$, ${\mathit{V}}_0\in {ℝ}^{s_0\times s}$ that satisfy ${\mathit{U}}_0^T{\mathit{U}}_0={\mathit{V}}_0^T{\mathit{V}}_0={\mathit{I}}_s$. Denoted by U_{k, 0} the submatrix of U₀ formed by rows in $\{(k-1)s_0+1,(k-1)s_0+2,\cdots ,k{s}_0\}$ and columns in {1, 2,...,s}. It follows from Equation 18 that

$${\mathit{R}}_{k,0}={\mathit{U}}_{k,0}{\mathbf{\Lambda }}_0{\mathit{V}}_0^T,\forall 1\le k\le K.$$

Since R_k,0 is symmetric, we have ${\mathit{U}}_{k,0}{\mathbf{\Lambda }}_0{\mathit{V}}_0^T={\mathit{V}}_0{\mathbf{\Lambda }}_0{\mathit{U}}_{k,0}^T={\mathit{R}}_{k,0}$. Notice that ${\mathit{V}}_0^T{\mathit{V}}_0={\mathit{I}}_s$. It follows that ${\mathit{U}}_{k,0}{\mathbf{\Lambda }}_0={\mathit{V}}_0{\mathbf{\Lambda }}_0{\mathit{U}}_{k,0}^T{\mathit{V}}_0$. Therefore, we have

$${\mathit{R}}_{k,0}={\mathit{V}}_0{\mathbf{\Lambda }}_0{\mathit{U}}_{k,0}^T{\mathit{V}}_0{\mathit{V}}_0^T.$$ (19)

Define ${\overline{\mathit{a}}}_{i,0}={\mathit{V}}_0^T{\mathit{a}}_{i,0},\forall 1\le i\le n$ and ${\overline{\mathit{R}}}_{k,0}={\mathbf{\Lambda }}_0{\mathit{U}}_{k,0}^T{\mathit{V}}_0$. In view of Equation 19, it is immediate to see that Equation 17 holds. Since ${\mathit{V}}_0^T{\mathit{V}}_0={\mathit{I}}_s$, we have ${\overline{\mathit{R}}}_{k,0}={\mathit{V}}_0^T{\mathit{V}}_0{\mathbf{\Lambda }}_0{\mathit{U}}_{k,0}^T{\mathit{V}}_0={\mathit{V}}_0^T{\mathit{R}}_{k,0}{\mathit{V}}_0$. As a result, ${\overline{\mathit{R}}}_{1,0},\dots ,{\overline{\mathit{R}}}_{K,0}$ are also symmetric. Suppose ${\overline{\mathit{R}}}_0={({\overline{\mathit{R}}}_{1,0}^T,\cdots ,{\overline{\mathit{R}}}_{K,0}^T)}^T$ doesn’t have full column rank. Using the same arguments, we can find some $s^{\prime }<s,{\tilde{\mathit{a}}}_{1,0},\cdots ,{\tilde{\mathit{a}}}_{n,0}\in {ℝ}^{s^{\prime }},{\tilde{\mathit{R}}}_{1,0},\cdots ,{\tilde{\mathit{R}}}_{K,0}\in {ℝ}^{s^{\prime }\times s^{\prime }}$ such that

$${\tilde{\mathit{a}}}_{i,0}^T{\tilde{\mathit{R}}}_{k,0}{\tilde{\mathit{a}}}_{j,0}={\mathit{a}}_{i,0}^T{\mathit{R}}_{k,0}{\mathit{a}}_{j,0},\forall 1\le i,j\le n,1\le k\le K.$$ (20)

We may repeat this procedure until we find some ${\tilde{\mathit{a}}}_{i,0}$, ${\tilde{\mathit{R}}}_{k,0}$’s satisfy Equation 20 and that the matrix ${({\tilde{\mathit{R}}}_{1,0}^T,\cdots ,{\tilde{\mathit{R}}}_{K,0}^T)}^T$ has full column rank.

B.2. Discussion of Condition (A1)

In this section, we consider an asymptotic framework where ${\mathit{a}}_{1,0},\dots ,{\mathit{a}}_{n,0}$ are i.i.d according to some distribution function and show (A1) holds with probability tending to 1. For any q-dimensional vector $\mathit{q}$, let $\Vert \mathit{q}{\Vert }_2$ denote its Euclidean norm. For any m × q matrix Q, $\Vert \mathit{Q}{\Vert }_2$ stands for the spectral norm of Q while $\Vert \mathit{Q}{\Vert }_F$ denotes its Frobenius norm. For simplicity, we assume ${\mathrm{\Theta }}_a=\{\mathit{x}\in {ℝ}^{s_0}:\Vert \mathit{x}{\Vert }_2\le {\omega }_a\}$ and ${\mathrm{\Theta }}_r=\{\mathit{y}\in {ℝ}^{s_0^2}:\Vert \mathit{y}{\Vert }_2\le {\omega }_r\}$. Assume ${\max }_{k=1,\dots ,K}{\Vert {\mathit{R}}_{k,0}\Vert }_F=O(1),{\Vert {\mathit{a}}_{1,0}\Vert }_2$ is bounded with probability 1. In addition, assume there exist some constants $c_0,t_0>0$ such that

$$\mathrm{Pr}({\Vert {[{\mathit{a}}_{1,0},\dots ,{\mathit{a}}_{s_0,0}]}^{-1}\Vert }_F>t)\le c_0{t}^{-1},\forall t\le t_0.$$ (21)

When s₀ = 1, the above condition is closely related to the margin assumption (Tsybakov, 2004; Audibert and Tsybakov, 2007) in the classification literature. It automatically holds when a_1,0 has a bounded probability density function.

In the following, we show with proper choice of ω_a and ω_r, (A1) holds with probability tending to 1. By the identifiability constraints, ${\Vert {\mathit{a}}_i^{(s)*}\Vert }_2=1,\forall i=1,\dots ,s$ and $s=s_0,\dots ,s_{\max }$. When ${\omega }_a,{\omega }_r\to \infty$, we have for sufficiently large n,

$$\mathrm{Pr}\left({\mathit{a}}_i^{(s)*}\in {\mathrm{\Omega }}_a\right)=1,\forall 1\le i\le s,s_0\le s\le s_{\max }.$$ (22)

By the definition of A_s,0, we have

$${\mathit{A}}_{s,0}^{-1}=\left(\begin{array}{cc}{\mathit{A}}_{s_0,0}^{-1}& {\mathit{O}}_{s_0\times \left(s-s_0\right)}\\ -{\mathit{A}}_{\left(s_0+1\right):s,0}{\mathit{A}}_{s_0,0}^{-1}& {\mathit{I}}_{s-s_0}\end{array}\right),$$ (23)

where ${\mathit{A}}_{\left(s_0+1\right):s,0}={[{\mathit{a}}_{s_0+1,0},\dots ,{\mathit{a}}_{s,0}]}^T$ .It follows that

$${\mathit{a}}_i^{(s)*}=\left(\begin{array}{c}{\mathit{A}}_{s_0,0}^{-1}{\mathit{a}}_{i,0}\\ {(\underset{i-s_0-1}{\underbrace{0,\dots ,0}},1,\underset{s-i}{\underbrace{0,\dots ,0}})}^T-{\mathit{A}}_{\left(s_0+1\right):s,0}{\mathit{A}}_{s_0,0}^{-1}{\mathit{a}}_{i,0}\end{array}\right),\forall s<i\le n,s_0\le s\le s_{\max }.$$

Under the given conditions, we have ${\Vert {\mathit{a}}_{i,0}\Vert }_2\le {\omega }_0$ with probability 1 for some constant ω₀ > 0. Therefore, we have

$$\underset{\underset{s<i\le n}{s_0\le s\le s\le s_{\max }}}{\max }{\Vert {\mathit{a}}_i^{{(s)}^{*}}\Vert }_2\le {\omega }_0{\Vert {\mathit{A}}_{s_0,0}^{-1}\Vert }_2+1+\sqrt{s_{\max }}{\omega }_0^2{\Vert {\mathit{A}}_{s_0,0}^{-1}\Vert }_2\le 1+(1+\sqrt{s_{\max }}{\omega }_0){\omega }_0{\Vert {\mathit{A}}_{s_0,0}^{-1}\Vert }_F.$$

By (21), it is immediate to see that

$$\mathrm{Pr}\left(\underset{\underset{1\le i\le n}{s_0\le s\le s_{\max }}}{\max }{\Vert {\mathit{a}}_i^{{(s)}^{*}}\Vert }_2>{\omega }_a\right)\to 0,$$

as long as ${\omega }_a⩾\sqrt{s_{\text{max}}}$. This together with Equation 22 yields

$$\mathrm{Pr}\left(\underset{\underset{1\le i\le n}{{}_{s_0\le s\le s_{\max }}}}{\max }{\Vert {\mathit{a}}_i^{{(s)}^{*}}\Vert }_2>{\omega }_a\right)\to 0.$$

Combining Equation 23 with the definition of ${\mathit{R}}_k^{(s)*}$ yields

$${\mathit{R}}_k^{(s)*}=\left(\begin{array}{cc}{\mathit{A}}_{s_0,0}^{-1}{\mathit{R}}_{k,0}{({\mathit{A}}_{s_0,0}^{-1})}^T& -{\mathit{A}}_{s_0,0}^{-1}{\mathit{R}}_{k,0}{({\mathit{A}}_{s_0,0}^{-1})}^T{\mathit{A}}_{(s_0+1):s,0}^T\\ -{\mathit{A}}_{(s_0+1):s,0}{\mathit{A}}_{s_0,0}^{-1}{\mathit{R}}_{k,0}{({\mathit{A}}_{s_0,0})}^T& {\mathit{A}}_{(s_0+1):s,0}{\mathit{A}}_{s_0,0}^{-1}{\mathit{R}}_{k,0}{({\mathit{A}}_{s_0,0}^{-1})}^T{\mathit{A}}_{(s_0+1);s,0}^T\end{array}\right)$$

Since ${\max }_{1\le k\le K}{\Vert {\mathit{R}}_{k,0}\Vert }_F=O(1),{\max }_{1\le i\le n}{\Vert {\mathit{a}}_{i,0}\Vert }_2=O(1)$ we have

$$\underset{1\le k\le K}{\max }{\Vert {\mathit{A}}_{s_0,0}^{-1}{\mathit{R}}_{k,0}{({\mathit{A}}_{s_0,0}^{-1})}^T\Vert }_F\le \underset{1\le k\le K}{\max }{\Vert {\mathit{A}}_{s_0,0}^{-1}\Vert }_F^2{\Vert {\mathit{R}}_{k,0}\Vert }_F=O({\Vert {\mathit{A}}_{s_0,0}^{-1}\Vert }_F^2),$$

and

$$\begin{gathered} \underset{\underset{s_0\le s\le s_{\max }}{1\le k\le K}}{\max }{\Vert {\mathit{A}}_{\left(s_0+1\right):s,0}{\mathit{A}}_{s_0,0}^{-1}{\mathit{R}}_{k,0}{\left({\mathit{A}}_{s_0,0}^{-1}\right)}^T\Vert }_F \\ \le \underset{\underset{s_0\le s\le \sin nx}{1\le k\le K}}{\max }{\Vert {\mathit{A}}_{s_0,0}^{-1}\Vert }_F^2{\Vert {\mathit{A}}_{\left(s_0+1\right):s,0}\Vert }_F{\Vert {\mathit{R}}_{k,0}\Vert }_F=O\left(\sqrt{s_{\max }}{\Vert {\mathit{A}}_{s_0,0}^{-1}\Vert }_F^2\right), \\ \underset{\underset{s_0\le s\le s_{\max }}{1\le k\le K}}{\max }{\Vert {\mathit{A}}_{\left(s_0+1\right):s,0}{\mathit{A}}_{s_0,0}^{-1}{\mathit{R}}_{k,0}{\left({\mathit{A}}_{s_0,0}^{-1}\right)}^T{\mathit{A}}_{\left(s_0+1\right):s,0}^T\Vert }_F \\ \le \underset{\underset{s_0\le s\le s_{\max }}{1\le k\le K}}{\max }{\Vert {\mathit{A}}_{s_0,0}^{-1}\Vert }_F^2{\Vert {\mathit{A}}_{\left(s_0+1\right);s,0}\Vert }_F^2{\Vert {\mathit{R}}_{k,0}\Vert }_F=O\left(s_{\max }{\Vert {\mathit{A}}_{s_0,0}^{-1}\Vert }_F^2\right). \end{gathered}$$

By (21), we have ${\max }_{1\le k\le K,s_0\le s\le s_{\max }}{\Vert {\mathit{R}}_k^{(s)*}\Vert }_F\le {\omega }_r$ with probability tending to 1, for any ω_r such that ${\omega }_r/s_{\max }\to \infty$.

B.3. Discussion of Condition (A2)

Assume the matrix $\text{E}{\mathit{a}}_{1,0}{\mathit{a}}_{1,0}^T$ is positive definite. Since s₀ is fixed, it follows from the law of large numbers that

$$\frac{1}{n}\sum _{i=1}^n{\mathit{a}}_{i,0}{\mathit{a}}_{i,0}^T\stackrel{P}{\to }{\text{E}\mathit{a}}_{1,0}{\mathit{a}}_{1,0}^T.$$

Therefore, (A2) holds with probability tending to 1.

Appendix C. Proofs

In the following, we provide proofs of Lemma 1, Lemma 2 and Theorem 1. We define

$${\mathcal{l}}_0({\{{\mathit{a}}_i\}}_i,{\{{\mathit{R}}_k\}}_k)=\text{E}{\mathcal{l}}_n(\mathit{Y};{\{{\mathit{a}}_i\}}_i,{\{{\mathit{R}}_k\}}_k),$$

for any ${\mathit{a}}_1,\dots ,{\mathit{a}}_n\in {ℝ}^s$, ${\mathit{R}}_1,\dots ,{\mathit{R}}_K\in {ℝ}^{s\times s}$ and any integer s ≥ 1. Define ${\theta }_{ijk}={({\mathit{a}}_i^{*})}^T{\mathit{R}}_k{\mathit{a}}_j^{*}$ and ${\widehat{\theta }}_{ijk}={\widehat{\mathit{a}}}_i^T{\widehat{\mathit{R}}}_k{\widehat{\mathit{a}}}_j$

C.1. Proof of Lemma 1

Assume there exist some ${\{{\mathit{a}}_i\}}_i,{\{{\mathit{R}}_k\}}_k$ such that

$$\text{g}({\mathit{a}}_{i,0}^T{\mathit{R}}_{k,0}{\mathit{a}}_{j,0})=\text{g}({\mathit{a}}_i^T{\mathit{R}}_k{\mathit{a}}_j),\forall i,j,k.$$

Since g(·) is strictly monotone, we have

$${\mathit{a}}_{i,0}^T{\mathit{R}}_{k,0}{\mathit{a}}_{j,0}={\mathit{a}}_i^T{\mathit{R}}_k{\mathit{a}}_j,\forall i,j,k,$$

or equivalently,

$$\left(\begin{array}{c}\mathit{A}{\mathit{R}}_1\\ ⋮\\ \mathit{A}{\mathit{R}}_K\end{array}\right){\mathit{A}}^T=\left(\begin{array}{c}{\mathit{A}}_0{\mathit{R}}_{1,0}\\ ⋮\\ {\mathit{A}}_0{\mathit{R}}_{K,0}\end{array}\right){\mathit{A}}_0^T.$$

where $\mathit{A}={[{\mathit{a}}_1,{\mathit{a}}_2,\dots ,{\mathit{a}}_n]}^T$. Thus, it follows that

$$\left(\begin{array}{c}{\mathit{A}}_0^T\mathit{A}{\mathit{R}}_1\\ ⋮\\ {\mathit{A}}_0^T\mathit{A}{\mathit{R}}_K\end{array}\right){\mathit{A}}^T=\left(\begin{array}{c}{\mathit{A}}_0^T{\mathit{A}}_0{\mathit{R}}_{1,0}\\ ⋮\\ {\mathit{A}}_0^T{\mathit{A}}_0{\mathit{R}}_{K,0}\end{array}\right){\mathit{A}}_0^T.$$

By (A0), the matrix ${\mathit{A}}_0^T{\mathit{A}}_0$ is invertible. As a result, we have

$$\left(\begin{array}{c}{({\mathit{A}}_0^T{\mathit{A}}_0)}^{-1}{\mathit{A}}_0^T\mathit{A}{\mathit{R}}_1\\ ⋮\\ {({\mathit{A}}_0^T{\mathit{A}}_0)}^{-1}{\mathit{A}}_0^T\mathit{A}{\mathit{R}}_K\end{array}\right){\mathit{A}}^T=\left(\begin{array}{c}{\mathit{R}}_{1,0}\\ ⋮\\ {\mathit{R}}_{K,0}\end{array}\right){\mathit{A}}_0^T.$$

Therefore,

$$\left(\sum _{k=1}^K{\mathit{R}}_{k,0}^T{({\mathit{A}}_0^T{\mathit{A}}_0)}^{-1}{\mathit{A}}_0^T\mathit{A}{\mathit{R}}_1\right){\mathit{A}}^T=\left(\sum _{k=1}^K{\mathit{R}}_{k,0}^T{\mathit{R}}_{k,0}\right){\mathit{A}}_0^T.$$

Notice that the matrix $\sum _{k=1}^K{\mathit{R}}_{k,0}^T{\mathit{R}}_{k,0}$ is invertible under Condition (A0). It follows that

$$\underset{\mathit{C}}{\underbrace{{\left(\sum _{k=1}^K{\mathit{R}}_{k,0}^T{\mathit{R}}_{k,0}\right)}^{-1}\left(\sum _{k=1}^K{\mathit{R}}_{k,0}^T{({\mathit{A}}_0^T{\mathit{A}}_0)}^{-1}{\mathit{A}}_0^T\mathit{A}{\mathit{R}}_k\right)}}{\mathit{A}}^T={\mathit{A}}_0^T.$$

By Lemma 5.1 in Banerjee and Roy (2014), we have rank(C) ≥ rank(A₀) = s₀. Therefore, C is invertible. It follows that

$$\mathit{A}={\mathit{A}}_0{({\mathit{C}}^T)}^{-1},$$ (24)

or equivalently,

$${\mathit{a}}_i={\mathit{C}}^{-1}{\mathit{a}}_{i,0},\forall i=1,\dots ,n.$$

By Equation 24, we obtain ${\mathit{A}}_0{({\mathit{C}}^T)}^{-1}{\mathit{R}}_k{\mathit{C}}^{-1}{\mathit{A}}_0^T={\mathit{A}}_0{\mathit{R}}_{k,0}{\mathit{A}}_0^T,\forall k$ and hence

$${\mathit{A}}_0^T{\mathit{A}}_0{({\mathit{C}}^T)}^{-1}{\mathit{R}}_k{\mathit{C}}^{-1}{\mathit{A}}_0^T{\mathit{A}}_0={\mathit{A}}_0^T{\mathit{A}}_0{\mathit{R}}_{k,0}{\mathit{A}}_0^T{\mathit{A}}_0,\forall k=1,\dots ,K.$$

Since ${\mathit{A}}_0^T{\mathit{A}}_0$ is invertible, this further implies ${({\mathit{C}}^T)}^{-1}{\mathit{R}}_k{\mathit{C}}^{-1}={\mathit{R}}_{k,0},\forall k,$ or equivalently,

$${\mathit{R}}_k={\mathit{C}}^T{\mathit{R}}_{k,0}\mathit{C},\forall k=1,\dots ,K.$$

The proof is hence completed.

C.2. Proof of Lemma 2

To prove Lemma 2, we need the following lemma.

Lemma 4 (Mendelson et al. (2008), Lemma 2.3).

Given d ≥ 1, and ε > 0, we have

$$N(\epsilon ,B_2^d,\Vert \cdot {\Vert }_2)\le {\left(1+\frac{2}{\epsilon }\right)}^d,$$

where $B_2^d$ is the unit ball in ${ℝ}^d$, and $N(\epsilon ,\cdot ,\Vert \cdot {\Vert }_2)$ the covering number with respect to the Euclidean metric (see Definition 2.2.3 in van der Vaart and Wellner (1996) for details).

Under Condition (Al), we have ${\mathcal{l}}_n(\mathit{Y};{\{{\mathit{a}}_i^{(s)*}\}}_i,{\{{\mathit{R}}_k^{(s)*}\}}_k)\le {\mathcal{l}}_n(\mathit{Y};{\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k).$ and hence

$${\mathcal{l}}_n(\mathit{Y};{\{{\mathit{a}}_i^{(s)*}\}}_i,{\{{\mathit{R}}_k^{(s)*}\}}_k)\le {\mathcal{l}}_n(\mathit{Y};{\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k).$$ (25)

Besides, we have

$$\underset{i}{\max }{\Vert {\widehat{\mathit{a}}}_i^{(s)}\Vert }_2\le {\omega }_a,\underset{k}{\max }{\Vert \mathrm{vec}({\widehat{\mathit{R}}}_k^{(s)})\Vert }_2\le {\omega }_r,\forall s\in \{s_0,\dots ,s_{\max }\},$$ (26)

and

$$\underset{i}{\max }{\Vert {\mathit{a}}_i^{*}\Vert }_2\le {\omega }_a,\underset{k}{\max }{\Vert \mathrm{vec}({\mathit{R}}_k^{*})\Vert }_2\le {\omega }_r.$$ (27)

Therefore,

$$\begin{gathered} \underset{i,j,k}{\max }\left|{({\widehat{\mathit{a}}}_i^{(s)})}^T{\widehat{\mathit{R}}}_k^{(s)}{\widehat{\mathit{a}}}_j^{(s)}\right|\le \left(\underset{i}{\max }{\Vert {\widehat{\mathit{a}}}_i^{(s)}\Vert }_2^2\underset{k}{\max }{\Vert {\widehat{\mathit{R}}}_k^{(s)}\Vert }_2\right)\le \left(\underset{i}{\max }{\Vert {\widehat{\mathit{a}}}_i^{(s)}\Vert }_2^2\underset{k}{\max }{\Vert {\widehat{\mathit{R}}}_k^{(s)}\Vert }_F\right) \\ \le \left(\underset{{}_i}{\max }{\Vert {\widehat{\mathit{a}}}_i^{(s)}\Vert }_2^2\underset{k}{\max }{\Vert \mathrm{vec}({\widehat{\mathit{R}}}_k^{(s)})\Vert }_2\right)\le {\omega }_a^2{\omega }_r,\forall s\in \{s_0,\dots ,s_{\max }\}. \end{gathered}$$ (28)

Similarly, we can show

$$\underset{i,j,k}{\max }{\Vert {({\mathit{a}}_i^{*})}^T{\mathit{R}}_k^{*}{\mathit{a}}_j^{*}\Vert }_2\le {\omega }_a^2{\omega }_r.$$ (29)

We define ${\theta }_{ijk}^{*}={({\mathit{a}}_i^{*})}^T{\mathit{R}}_k^{*}{\mathit{a}}_j^{*}\text{and}{\widehat{\theta }}_{ijk}^{(s)}={({\widehat{\mathit{a}}}_i^{(s)})}^T{\widehat{\mathit{R}}}_k^{(s)}{\widehat{\mathit{a}}}_j^{(s)}$. It follows from a second-order Taylor expansion that

$$\text{g}({\theta }_{ijk}^{*})({\theta }_{ijk}^{*}-{\widehat{\theta }}_{ijk}^{(s)})-\log \{1+\exp ({\theta }_{ijk}^{*})\}+\log \{1+\exp ({\widehat{\theta }}_{ijk}^{(s)})\}=\frac{\exp ({\tilde{\theta }}_{ijk}^{(s)}){({\theta }_{ijk}^{*}-{\widehat{\theta }}_{ijk}^{(s)})}^2}{2{\{1+\exp ({\tilde{\theta }}_{ijk}^{(s)})\}}^2},$$ (30)

for some ${\tilde{\theta }}_{ijk}^{(s)}$ lying on the line segment joining ${\theta }_{ijk}^{*}$ and ${\widehat{\theta }}_{ijk}^{(s)}$. By (28) and (29), we have for any i, j, k and $s\in \{s_0,\dots ,s_{\max }\},\left|{\tilde{\theta }}_{ijk}^{(s)}\right|\le {\omega }_a^2{\omega }_r$. This together with Equation 30 gives that

$$\begin{gathered} \text{g}({\theta }_{ijk}^{*}){\theta }_{ijk}^{*}-\log \{1+\exp ({\theta }_{ijk}^{*})\}-\text{g}({\theta }_{ijk}^{*}){\widehat{\theta }}_{ijk}^{(s)}+\log \{1+\exp ({\widehat{\theta }}_{ijk}^{(s)})\} \\ \ge \frac{{({\theta }_{ijk}^{*}-{\widehat{\theta }}_{ijk}^{(s)})}^2\exp ({\omega }_a^2{\omega }_r)}{2{\{1+\exp ({\omega }_a^2{\omega }_r)\}}^2}\ge \frac{{({\theta }_{ijk}^{*}-{\widehat{\theta }}_{ijk}^{(s)})}^2\exp ({\omega }_a^2{\omega }_r)}{8\exp (2{\omega }_a^2{\omega }_r)}=\frac{{({\theta }_{ijk}^{*}-{\widehat{\theta }}_{ijk}^{(s)})}^2}{8\exp ({\omega }_a^2{\omega }_r)}, \end{gathered}$$

and hence

$$\begin{array}{l}{\mathcal{l}}_0({\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k)-{\mathcal{l}}_0({\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k)\\ =\sum _{ijk}\left(\text{g}({\theta }_{ijk}^{*})({\theta }_{ijk}^{*}-{\widehat{\theta }}_{ijk}^{(s)})-\log \frac{\{1+\exp ({\theta }_{ijk}^{*})\}}{\{1+\exp ({\widehat{\theta }}_{ijk}^{(s)})\}}\right)\ge \frac{\sum _{ijk}{({\theta }_{ijk}^{*}-{\widehat{\theta }}_{ijk}^{(s)})}^2}{8\exp ({\omega }_a^2{\omega }_r)}.\end{array}$$ (31)

In the following, we provide an upper bound for

$$\begin{array}{l}\underset{{}_{s\in \{1,\dots ,s_{\max }\}}}{\max }\underset{\underset{\underset{{\max }_k{\Vert \mathrm{vec}\left({\mathit{R}}_k\right)\Vert }_2\le {\omega }_r}{{\max }_i{\Vert {\mathit{a}}_i\Vert }_2\le {\omega }_a}}{{\mathit{a}}_i\in {ℝ}^a,{\mathit{R}}_k\in {ℝ}^{s\times s}}}{\sup }\left|{\mathcal{l}}_0\left({\{{\mathit{a}}_i\}}_i,{\{{\mathit{R}}_k\}}_k\right)-{\mathcal{l}}_n\left(\mathit{Y};{\{{\mathit{a}}_i\}}_i,{\{{\mathit{R}}_k\}}_k\right)\right|\\ \underset{{}_{s\in \{1,\dots ,s_{\max }\}}}{\max }\underset{\underset{\underset{{\max }_k{\Vert \mathrm{vec}\left({\mathit{R}}_k\right)\Vert }_2\le {\omega }_r}{{\max }_i{\Vert {\mathit{a}}_i\Vert }_2\le {\omega }_a}}{{\mathit{a}}_i\in {ℝ}^a,{\mathit{R}}_k\in {ℝ}^{s\times s}}}{\sup }\left|\sum _{ijk}\left(Y_{ijk}-{\pi }_{ijk}^{*}\right){\mathit{a}}_i^T{\mathit{R}}_k{\mathit{a}}_j\right|,\end{array}$$

where ${\pi }_{ijk}^{*}=\exp ({\theta }_{ijk}^{*})/\{1+\exp ({\theta }_{ijk}^{*})\},\forall i,j,k$.

Let ${\epsilon }_a={\omega }_a/{(nK)}^2$ and $\{{\overline{\mathit{a}}}_1^{(s)},\dots ,{\overline{\mathit{a}}}_{N_{s,{\epsilon }_a}}^{(s)}\}$ be a minimal ε_a-net of the vector space $(\{\mathit{a}\in {ℝ}^s:\Vert \mathit{a}{\Vert }_2\le {\omega }_a\},\Vert \cdot {\Vert }_2)$. It follows from Lemma 4 that

$$\begin{gathered} N_{s,{\epsilon }_a}=N\left({\epsilon }_a,\{\mathit{a}\in {ℝ}^s:\Vert \mathit{a}{\Vert }_2\le {\omega }_a\},\Vert \cdot {\Vert }_2\right)=N\left(1/{(nK)}^2,B_2^s,\Vert \cdot {\Vert }_2\right) \\ \le {\left(1+2n^2{K}^2\right)}^s\le {(3nK)}^{2s}. \end{gathered}$$ (32)

Let ${\epsilon }_r={\omega }_r/{(nK)}^2$, and $\{{\overline{\mathit{R}}}_1^{(s)},\dots ,{\overline{\mathit{R}}}_{N_{s^2,{\epsilon }_r}}^{(s)}\}$ be a minimal ε_r-net of the vector space $(\{\mathit{R}\in {ℝ}^{s\times s}:\Vert \mathrm{vec}(\mathit{R}){\Vert }_2\le \omega \},\Vert \cdot {\Vert }_F)$. For any s × s matrices Q, we have $\Vert \mathit{Q}{\Vert }_F=\Vert \mathrm{vec}(\mathit{Q}){\Vert }_2$. Similar to (32), we can show that

$$N_{s^2,{\epsilon }_r}\le {(3nK)}^{2s^2}.$$ (33)

Hence, for any ${\mathit{a}}_i,{\mathit{a}}_j\in {ℝ}^s\text{and}{\mathit{R}}_k\in {ℝ}^{s\times s}$ satisfying ${\Vert {\mathit{a}}_i\Vert }_2,{\Vert {\mathit{a}}_j\Vert }_2\le {\omega }_a,{\Vert \mathrm{vec}({\mathit{R}}_k)\Vert }_2\le {\omega }_r$ there exist some ${\overline{\mathit{a}}}_{l_j}^{(s)},{\overline{\mathit{a}}}_{l_j}^{(s)},{\overline{\mathit{R}}}_{t_k}^{(s)}$, such that

$${\Vert {\mathit{a}}_i-{\overline{\mathit{a}}}_{l_i}^{(s)}\Vert }_2\le \frac{{\omega }_a}{{(nK)}^2},{\Vert {\mathit{a}}_j-{\overline{\mathit{a}}}_{l_j}^{(s)}\Vert }_2\le \frac{{\omega }_a}{{(nK)}^2},{\Vert {\mathit{R}}_k-{\overline{\mathit{R}}}_{t_k}^{(s)}\Vert }_F\le \frac{{\omega }_r}{{(nK)}^2}.$$ (34)

This further implies

$$\begin{gathered} |{\left({\mathit{a}}_i\right)}^T{\mathit{R}}_k{\mathit{a}}_j-{\left({\overline{\mathit{a}}}_{l_i}^{(s)}\right)}^T{\overline{\mathit{R}}}_{t_k}^{(s)}{\overline{\mathit{a}}}_{l_j}^{(s)}|\le |{\left({\mathit{a}}_i\right)}^T{\mathit{R}}_k{\mathit{a}}_j-{\left({\overline{\mathit{a}}}_{l_i}^{(s)}\right)}^T{\mathit{R}}_k{\mathit{a}}_j| \\ +|{\left({\overline{\mathit{a}}}_{l_i}^{(s)}\right)}^T{\mathit{R}}_k{\mathit{a}}_j-{\left({\overline{\mathit{a}}}_{l_i}^{(s)}\right)}^T{\overline{\mathit{R}}}_{t_k}^{(s)}{\mathit{a}}_j|+|{\left({\overline{\mathit{a}}}_{l_i}^{(s)}\right)}^T{\overline{\mathit{R}}}_{t_k}^{(s)}{\mathit{a}}_j-{\left({\overline{\mathit{a}}}_{l_i}^{(s)}\right)}^T{\overline{\mathit{R}}}_{t_k}^{(s)}{\overline{\mathit{a}}}_{l_j}^{(s)}| \\ \le {\Vert {\mathit{a}}_i-{\overline{\mathit{a}}}_{l_i}^{(s)}\Vert }_2{\Vert {\mathit{R}}_k\Vert }_2{\Vert {\mathit{a}}_j\Vert }_2+{\Vert {\overline{\mathit{a}}}_{l_i}^{(s)}\Vert }_2{\Vert {\mathit{R}}_k-{\overline{\mathit{R}}}_{t_k}^{(s)}\Vert }_2{\Vert {\mathit{a}}_j\Vert }_2+{\Vert {\overline{\mathit{a}}}_i^{(s)}\Vert }_2{\Vert {\overline{\mathit{R}}}_{t_k}^{(s)}\Vert }_2{\Vert {\mathit{a}}_j-{\overline{\mathit{a}}}_{l_j}^{(s)}\Vert }_2 \\ \le {\Vert {\mathit{a}}_i-{\overline{\mathit{a}}}_{l_i}^{(s)}\Vert }_2{\Vert {\mathit{R}}_k\Vert }_F{\Vert {\mathit{a}}_j\Vert }_2+{\Vert {\overline{\mathit{a}}}_{l_i}^{(s)}\Vert }_2{\Vert {\mathit{R}}_k-{\overline{\mathit{R}}}_{t_k}^{(s)}\Vert }_F\Vert {\mathit{a}}_j\Vert +{\Vert {\overline{\mathit{a}}}_i^{(s)}\Vert }_2{\Vert {\overline{\mathit{R}}}_{t_k}^{(s)}\Vert }_F{\Vert {\mathit{a}}_j-{\overline{\mathit{a}}}_{l_j}^{(s)}\Vert }_2 \\ \le 2\frac{{\omega }_a}{{(nK)}^2}{\omega }_a{\omega }_r+\frac{{\omega }_r}{{(nK)}^2}{\omega }_a^2=\frac{3{\omega }_a^2{\omega }_r}{{(nK)}^2}. \end{gathered}$$ (35)

Therefore, we have

$$\begin{gathered} \underset{s\in \{1,\dots ,s_{\max }\}}{\max }\frac{1}{s}\underset{\begin{array}{c}{\mathit{a}}_i\in {ℝ}^s,{\mathit{R}}_k\in {ℝ}^{s\times s}\\ {\max }_i{\Vert {\mathit{a}}_i\Vert }_2\le {\omega }_a\\ {\max }_k{\Vert \mathrm{vec}({\mathit{R}}_k)\Vert }_2\le {\omega }_r\end{array}}{\sup }\left|\sum _{ijk}\left(Y_{ijk}-{\pi }_{ijk}^{*}\right){\mathit{a}}_i^T{\mathit{R}}_k{\mathit{a}}_j\right| \\ \le \underset{s\in \{1,\dots ,s_{\max }\}}{\max }\frac{1}{s}\underset{\underset{t_1,\dots ,t_K\in \{1,\dots ,N_{s^2,{\epsilon }_r}\}}{l_1,\dots ,l_n\in \{1,\dots ,N_{s,{\epsilon }_a}\}}}{\max }\left|\sum _{ijk}\left(Y_{ijk}-{\pi }_{ijk}^{*}\right){\left({\overline{\mathit{a}}}_{l_i}^{(s)}\right)}^T{\overline{\mathit{R}}}_{t_k}^{(s)}{\overline{\mathit{a}}}_{l_j}^{(s)}\right|+\sum _{ijk}\left|Y_{ijk}-{\pi }_{ijk}^{*}\right|\frac{3{\omega }_a^2{\omega }_r}{n^2{K}^2} \\ \le \underset{s\in \{1,\dots ,s_{\max }\}}{\max }\frac{1}{s}\underset{\underset{t_1,\dots ,t_K\in \{1,\dots ,N_{s^2,{\epsilon }_r}\}}{l_1,\dots ,l_n\in \{1,\dots ,N_{s,{\epsilon }_a}\}}}{\max }\left|\sum _{ijk}\left(Y_{ijk}-{\pi }_{ijk}^{*}\right){\left({\overline{\mathit{a}}}_{l_i}^{(s)}\right)}^T{\overline{\mathit{R}}}_{t_k}^{(s)}{\overline{\mathit{a}}}_{l_j}^{(s)}\right|+\frac{3{\omega }_a^2{\omega }_r}{K}. \end{gathered}$$ (36)

By Bernstein’s inequality (van der Vaart and Wellner, 1996, Lemma 2.2.9), we obtain for any t > 0,

$$\begin{array}{cc}\underset{\underset{\underset{t_1,\dots ,t_K\in \{1,\dots ,N_{s^2,{\epsilon }_r}\}}{{}_{l_1,\dots ,l_n\in \{1,\dots ,N_{s,{\epsilon }_a}\}}}}{s\in \{1,\dots ,s_{\max }\}}}{\text{max}}& \mathrm{Pr}\left(\left|\sum _{ijk}\left(Y_{ijk}-{\pi }_{ijk}^{*}\right){\left({\overline{\mathit{a}}}_{l_i}^{(s)}\right)}^T{\overline{\mathit{R}}}_{t_k}^{(s)}{\overline{\mathit{a}}}_{l_j}^{(s)}\right|>t\right)\le 2\exp \left(-\frac{t^2}{2{\sigma }_0^2+2M_{0}t/3}\right)\end{array},$$ (37)

where

$$M_0=\underset{s\in \{1,\dots ,s_{\max }\}}{\max }\underset{t_1,\dots ,t_K\in \{1,\dots ,N_{s^2,{\epsilon }_r}\}}{\underset{l_1,\dots ,l_n\in \{1,\dots ,N_{s,{\epsilon }_a}\}}{\max }}\underset{ijk}{\max }\left|Y_{ijk}-{\pi }_{ijk}^{*}\Vert {\left({\overline{\mathit{a}}}_{l_i}^{(s)}\right)}^T{\overline{\mathit{R}}}_{t_k}^{(s)}{\overline{\mathit{a}}}_{l_j}^{(s)}\right|,$$

$${\sigma }_0^2=\underset{s\in \{1,\dots ,s_{\max }\}}{\max }\underset{t_1,\dots ,t_K\in \{1,\dots ,N_{s^2,{\epsilon }_r}\}}{\underset{l_1,\dots ,l_n\in \{1,\dots ,N_{s,{\epsilon }_a}\}}{\max }}\sum _{ijk}\text{E}{\left|Y_{ijk}-{\pi }_{ijk}^{*}\right|}^2{\left|{\left({\overline{\mathit{a}}}_{l_i}^{(s)}\right)}^T{\overline{\mathit{R}}}_{t_k}^{(s)}{\overline{\mathit{a}}}_{l_j}^{(s)}\right|}^2.$$

With some calculations, we can show that

$$\begin{gathered} \begin{array}{cccc}{M}_0\le & \underset{\underset{t_1,\dots ,t_K\in \{1,\dots ,N_{s^2,{\epsilon }_r}\}}{l_1,\dots ,l_n\in \{1,\dots ,N_{s,{\epsilon }_a}\}}}{\max }& {\Vert {\overline{\mathit{a}}}_{l_i}^{(s)}\Vert }_2{\Vert {\overline{\mathit{R}}}_{t_k}^{(s)}\Vert }_2{\Vert {\overline{\mathit{a}}}_{l_j}^{(s)}\Vert }_2\le & \underset{\underset{t_1,\dots ,t_K\in \{1,\dots ,N_{s^2,{\epsilon }_r}\}}{l_1,\dots ,l_n\in \{1,\dots ,N_{s,{\epsilon }_a}\}}}{\max }{\Vert {\overline{\mathit{a}}}_{l_i}^{(s)}\Vert }_2\Vert {\overline{\mathit{R}}}_{t_k}^{(s)}\Vert {}_F{\Vert {\overline{\mathit{a}}}_{l_j}^{(s)}\Vert }_2\end{array} \\ \begin{array}{ccc}\le & \underset{\underset{t_1,\dots ,t_K\in \{1,\dots ,N_{s^2,{\epsilon }_r}\}}{l_1,\dots ,l_n\in \{1,\dots ,N_{s,{\epsilon }_a}\}}}{\max }& {\Vert {\overline{\mathit{a}}}_{l_i}^{(s)}\Vert }_2{\Vert \mathrm{vec}\left({\overline{\mathit{R}}}_{t_k}^{(s)}\right)\Vert }_2{\Vert {\overline{\mathit{a}}}_{l_j}^{(s)}\Vert }_2\le {\omega }_a^2{\omega }_r\end{array}, \end{gathered}$$ (38)

and

$$\begin{gathered} {\sigma }_0^2\le \underset{{}_{s\in \{1,\dots ,s_{\max }\}}}{\max }\underset{\underset{t_1,\dots ,t_K\in \{1,\dots ,N_{s^2,{\epsilon }_r}\}}{l_1,\dots ,l_n\in \{1,\dots ,N_{s,{\epsilon }_a}\}}}{\max }\sum _{ijk}\text{E}{\left|Y_{ijk}-{\pi }_{ijk}^{*}\right|}^2{\Vert {\overline{\mathit{a}}}_{l_i}^{(s)}\Vert }_2^2{\Vert \mathrm{vec}\left({\overline{\mathit{R}}}_{t_k}^{(s)}\right)\Vert }_2^2{\Vert {\overline{\mathit{a}}}_{l_j}^{(s)}\Vert }_2^2 \\ \le {\omega }_a^4{\omega }_r^2\sum _{ijk}\text{E}{\left|Y_{ijk}-{\pi }_{ijk}^{*}\right|}^2\le {\omega }_a^4{\omega }_r^2{n}^{2}K. \end{gathered}$$ (39)

Let $t_s=5{\omega }_a^2{\omega }_{r}sn\sqrt{K\max (n,K)\log (nK)}$, we have

$$\underset{{}_{s\in \{1,\dots ,s_{\max }\}}}{\max }\mathrm{Pr}\left(\underset{\underset{t_1,\dots ,t_K\in \{1,\dots ,N_{s^2,{\epsilon }_r}\}}{l_1,\dots ,l_n\in \{1,\dots ,N_{s,{\epsilon }_a}\}}}{\max }\left|\sum _{ijk}\left(Y_{ijk}-{\pi }_{ijk}^{*}\right){\left({\overline{\mathit{a}}}_{l_i}^{(s)}\right)}^T{\overline{\mathit{R}}}_{t_k}^{(s)}{\overline{\mathit{a}}}_{l_j}^{(s)}\right|>t_s\right)$$

$$\le \underset{\underset{t_1,\dots ,t_K\in \{1,\dots ,N_{s^2,{\epsilon }_r}\}}{l_1,\dots ,l_n\in \{1,\dots ,N_{s,{\epsilon }_a}\}}}{\underset{s\in \{1,\dots ,s_{\max }\}}{\text{max}}}{N}_{s,{\epsilon }_a}^n{N}_{s^2,{\epsilon }_r}^K\mathrm{Pr}\left(\left|\sum _{ijk}\left(Y_{ijk}-{\pi }_{ijk}^{*}\right){\left({\overline{\mathit{a}}}_{l_i}^{(s)}\right)}^T{\overline{\mathit{R}}}_{t_k}^{(s)}{\overline{\mathit{a}}}_{l_j}^{(s)}\right|>t_s\right)$$

$$\le \underset{\underset{t_1,\dots ,t_K\in \{1,\dots ,N_{s^2,{\epsilon }_r}\}}{l_1,\dots ,l_n\in \{1,\dots ,N_{s,{\epsilon }_a}\}}}{\underset{s\in \{1,\dots ,s_{\max }\}}{\text{max}}}{(3nK)}^{2sn+2s^{2}K}\mathrm{Pr}\left(\left|\sum _{ijk}\left(Y_{ijk}-{\pi }_{ijk}^{*}\right){\left({\overline{\mathit{a}}}_{l_i}^{(s)}\right)}^T{\overline{\mathit{R}}}_{t_k}^{(s)}{\overline{\mathit{a}}}_{l_j}^{(s)}\right|>t_s\right)$$

$$\begin{array}{l}\le \underset{s\in \{1,\dots ,s_{\max }\}}{\max }2\exp \left(-\frac{25{\omega }_a^4{\omega }_r^2{s}^2{n}^{2}K\max (n,K)\log (nK)}{2{\omega }_a^4{\omega }_r^2\{n^{2}K+sn\sqrt{K\max (n,K)\log (nK)}/3\}}+\left(2sn+2s^{2}K\right)\log (3nK)\right)\\ =\underset{s\in \{1,\dots ,s_{\max }\}}{\max }2\exp \left(-\frac{25s^2\max (n,K)\log (nK)}{2+2s\sqrt{\max (n,K)\log (nK)}/(3n\sqrt{K})}+\left(2sn+2s^{2}K\right)\log (3nK)\right),\end{array}$$ (40)

where the first inequality is due to Bonferroni’s inequality, the second inequality follows by (32) and (33), the third inequality is due to (37)-(39).

Under the given conditions, we have $s\le s_{\max }=o\{n/\log (nK)\}$ and hence

$$\frac{s\sqrt{\max (n,K)\log (nK)}}{n\sqrt{K}}\le \frac{s\sqrt{n\log (nK)}}{n\sqrt{K}}+\frac{s\sqrt{K\log (nK)}}{n\sqrt{K}}\le \frac{s_{\max }\sqrt{\log (nK)}}{\sqrt{n}}+\frac{s_{\max }\sqrt{\log (nK)}}{n}\ll 1.$$

It follows that for any $s\in \{1,\dots ,s_{\max }\}$,

$$\begin{array}{r}\mathrm{Pr}\left(\underset{\underset{t_1,\dots ,t_K\in \{1,\dots ,N_{s^2,{\epsilon }_r}\}}{l_1,\dots ,l_n\in \{1,\dots ,N_{s,{\epsilon }_a}\}}}{\max }\left|\sum _{ijk}{\left(Y_{ijk}-{\pi }_{ijk}^{(s)}\right)}^T{({\overline{\mathit{a}}}_{l_j}^{(s)})}^T{\overline{\mathit{R}}}_{l_k}^{(s)}{\overline{\mathit{a}}}_{l_j}^{(s)}\right|>5{\omega }_a^2{\omega }_{r}sn\sqrt{K\max (n,K)\log (nK)}\right)\\ \le 2\exp \left(-\frac{25s^2\max (n,K)\log (nK)}{2+1}+4s^2\max (n,K)\log (3nK)\right)\\ \le 2\exp \{-4s^2\max (n,K)\log (nK)\}\le 2\exp \{-4s^{2}n\log (nK)\}.\end{array}$$

By Bonferroni’s inequality, we have

$$\begin{array}{l}\mathrm{Pr}\left(\underset{s\in \{1,\dots ,s_{\max }\}}{\cup }\{\underset{\underset{t_1,\dots ,t_K\in \{1,\dots ,N_{s^2,{\epsilon }_r}\}}{l_1,\dots ,l_n\in \{1,\dots ,N_{s,{\epsilon }_a}\}}}{\max }\left|\sum _{ijk}\left(Y_{ijk}-{\pi }_{ijk}^{*}\right){\left({\overline{\mathit{a}}}_{l_i}^{(s)}\right)}^T{\overline{\mathit{R}}}_{t_k}^{(s)}{\overline{\mathit{a}}}_{l_j}^{(s)}\right|>t_s\}\right)\\ \le \sum _{s=1}^{s_{\max }}2\exp \{-4s^{2}n\log (nK)\}\le \sum _{s=1}^{+\infty }2\exp \{-4sn\log (nK)\}\le \frac{2\exp \{-4n\log (nK)\}}{1-\exp \{-4n\log (nK)\}}\to 0.\end{array}$$

This together with (36) implies that

$$\underset{s\in \{1,\dots ,s_{\max }\}}{\max }\frac{1}{s}\underset{\begin{array}{c}{\mathit{a}}_i\in {ℝ}^s,{\mathit{R}}_k\in {ℝ}^{s\times s}\\ {\max }_i{\Vert {\mathit{a}}_i\Vert }_2\le {\omega }_a\\ {\max }_k{\Vert \mathrm{vec}({\mathit{R}}_k)\Vert }_2\le {\omega }_r\end{array}}{\sup }\left|\sum _{ijk}\left(Y_{ijk}-{\pi }_{ijk}^{*}\right){\mathit{a}}_i^T{\mathit{R}}_k{\mathit{a}}_j\right|\le 6{\omega }_a^2{\omega }_{r}n\sqrt{K\max (n,K)\log (nK)},$$ (41)

with probability tending to 1. Combining this with (26) and (27), we obtain with probability tending to 1,

$$\max ({\max }_s\left|{\mathcal{l}}_0\left({\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k\right)-{\mathcal{l}}_n\left(\mathit{Y};{\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k\right)\right|/s,$$ (42)

$$\left|{\mathcal{l}}_0\left({\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k\right)-{\mathcal{l}}_n\left(\mathit{Y};{\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k\right)\right|/s_0)\le 6{\omega }_a^2{\omega }_{r}n\sqrt{K\max (n,K)\log (nK)}.$$

Therefore, it follows from (31) that

$$\underset{s\in \{s_0,\dots ,s_{\max }\}}{\max }\frac{1}{s}\sum _{ijk}{({\theta }_{ijk}^{*}-{\widehat{\theta }}_{ijk}^{(s)}\mathit{)}}^2$$ (43)

$$\le \underset{s\in \{s_0,\dots ,s_{\max }\}}{\max }\frac{8\exp \left({\omega }_a^2{\omega }_r\right)}{s}\left({\mathcal{l}}_0\left({\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k\right)-{\mathcal{l}}_0\left({\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k\right)\right)$$

$$\le \underset{s\in \{s_0,\dots ,s_{\max }\}}{\max }\frac{8\exp \left({\omega }_a^2{\omega }_r\right)}{s}\left|{\mathcal{l}}_0\left({\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k\right)-{\mathcal{l}}_n\left(\mathit{Y};{\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k\right)\right|$$

$$+\underset{s\in \{s_0,\dots ,s_{\max }\}}{\max }\frac{8\exp \left({\omega }_a^2{\omega }_r\right)}{s}\left|{\mathcal{l}}_0\left({\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k\right)-{\mathcal{l}}_n\left(\mathit{Y};{\{{\widehat{\mathit{a}}}_i^{*}\}}_i,{\{{\widehat{\mathit{R}}}_k^{*}\}}_k\right)\right|$$

$$+\underset{s\in \{s_0,\dots ,s_{\max }\}}{\max }\frac{8\exp \left({\omega }_a^2{\omega }_r\right)}{s}\left({\mathcal{l}}_n\left(\mathit{Y};{\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k\right)-{\mathcal{l}}_n\left(\mathit{Y};{\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k\right)\right)$$

$$\le \underset{s\in \{s_0,\dots ,s_{\max }\}}{\max }\frac{8\exp \left({\omega }_a^2{\omega }_r\right)}{s}\left|{\mathcal{l}}_0\left({\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k\right)-{\mathcal{l}}_n\left(\mathit{Y};{\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k\right)\right|$$

$$+\underset{s\in \{s_0,\dots ,s_{\max }\}}{\max }\frac{8\exp \left({\omega }_a^2{\omega }_r\right)}{s}\left|{\mathcal{l}}_0\left({\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k\right)-{\mathcal{l}}_n\left(\mathit{Y};{\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k\right)\right|$$

$$\le 96{\omega }_a^2{\omega }_r\exp \left({\omega }_a^2{\omega }_r\right)n\sqrt{K\max (n,K)\log (nK)},$$

where the third inequality is due to that ${\mathcal{l}}_n\left(\mathit{Y};{\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k\right)\ge {\mathcal{l}}_n\left(\mathit{Y};{\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k\right)$, for all $s\in \{s_0,\dots ,s_{\max }\}$.

Let $r_{n,K}={(n+K)}^{-1/2}{(\log n+\log K)}^{-1/2}$. As $n\to \infty$, we have

$$r_{n,K}^{2}n\sqrt{K\max (n,K)\log (nK)}\le r_{n,K}^{2}n\sqrt{K(n+K)\log (nK)}=\frac{n\sqrt{K}}{\sqrt{(n+K)\log (nK)}}\ll \sqrt{nK}.$$

Since ${\omega }_a^2{\omega }_r\le \exp ({\omega }_a^2{\omega }_r)$, it follows from (43) that

$$\mathrm{Pr}\left(\underset{s\in \{s_0,\dots ,s_{\max }\}}{\max }\frac{r_{n,K}^2}{s^2\exp \left(2{\omega }_a^2{\omega }_r\right)}\sum _{ijk}{\left({\theta }_{ijk}^{*}-{\widehat{\theta }}_{ijk}^{(s)}\right)}^2\ge \sqrt{nK}\right)\to 0.$$ (44)

For any integer m ≥ 1, define

$$\begin{gathered} {\mathbb{S}}_m^{(s)}=\{\left({\{{\mathit{a}}_i^{(s)}\}}_i,{\{{\mathit{R}}_k^{(s)}\}}_k\right):m-1<\frac{r_{n,K}}{s\exp \left({\omega }_a^2{\omega }_r\right)}\sqrt{\sum _{ijk}{\{{\theta }_{ijk}^{*}-{\left({\mathit{a}}_i^{(s)}\right)}^T{\mathit{R}}_k^{(s)}{\mathit{a}}_j^{(s)}\}}^2}\le m, \\ {\mathit{a}}_i^{(s)}\in {ℝ}^s,\forall i,{\mathit{R}}_k^{(s)}\in {ℝ}^{s\times s},\forall k,\underset{i}{\max }{\Vert {\mathit{a}}_i^{(s)}\Vert }_2\le {\omega }_a,\underset{k}{\max }{\Vert \mathrm{vec}\left({\mathit{R}}_k^{(s)}\right)\Vert }_2\le {\omega }_r\}. \end{gathered}$$

For any $\left({\{{\mathit{a}}_i^{(s)}\}}_i,{\{{\mathit{R}}_k^{(s)}\}}_k\right)\in {\mathbb{S}}_m^{(s)}$, similar to (31), we can show that

$${\mathcal{l}}_0\left({\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k\right)-{\mathcal{l}}_0\left({\{{\mathit{a}}_i^{(s)}\}}_i,{\{{\mathit{R}}_k^{(s)}\}}_k\right)\ge \frac{\sum _{ijk}{\{{\theta }_{ijk}^{*}-{\left({\mathit{a}}_i^{(s)}\right)}^T{\mathit{R}}_k^{(s)}{\mathit{a}}_j^{(s)}\}}^2}{8\exp \left({\omega }_a^2{\omega }_r\right)}\ge \frac{{(m-1)}^2{s}^2{r}_{n,K}^{-2}}{8\exp \left(-{\omega }_a^2{\omega }_r\right)}.$$ (45)

The event $\left({\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k\right)\in {\mathbb{S}}_m^{(s)}$ implies that

$$\underset{\left({\{{\mathit{a}}_i^{(s)}\}}_i,{\{{\mathit{R}}_k^{(s)}\}}_k\right)\in {\mathbb{S}}_m^{(s)}}{\sup }{\mathcal{l}}_n\left(\mathit{Y};{\{{\mathit{a}}_i^{(s)}\}}_i,{\{{\mathit{R}}_k^{(s)}\}}_k\right)\ge {\mathcal{l}}_n\left(\mathit{Y};{\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k\right).$$

It follows from (45) that

$$\underset{\left({\{{\mathit{a}}_i^{(s)}\}}_i,{\{{\mathit{R}}_k^{(s)}\}}_k\right)\in {\mathbb{S}}_m^{(s)}}{\sup }\left|\sum _{ijk}\left(Y_{ijk}-{\pi }_{ijk}^{*}\right)\{{\theta }_{ijk}^{*}-{\left({\mathit{a}}_i^{(s)}\right)}^T{\mathit{R}}_k^{(s)}{\mathit{a}}_j^{(s)}\}\right|\ge \frac{{(m-1)}^2{s}^2\exp \left({\omega }_a^2{\omega }_r\right)}{8r_{n,K}^2}.$$ (46)

For any {l_i}_i and {t_k}_k satisfying (34), it follows from (35) that

$$\sum _{ijk}{\{{\theta }_{ijk}^{*}-{\left({\overline{\mathit{a}}}_{l_i}^{(s)}\right)}^T{\overline{\mathit{R}}}_{t_k}^{(s)}{\overline{\mathit{a}}}_{l_j}^{(s)}\}}^2$$

$$\le \sum _{ijk}{\{{\theta }_{ijk}^{*}-{({\mathit{a}}_i^{(s)})}^T{\mathit{R}}_k^{(s)}{\mathit{a}}_j^{(s)}+{({\mathit{a}}_i^{(s)})}^T{\mathit{R}}_k^{(s)}{\mathit{a}}_j^{(s)}-{({\overline{\mathit{a}}}_{l_i}^{(s)})}^T{\overline{\mathit{R}}}_{t_k}^{(s)}{\overline{\mathit{a}}}_{l_j}^{(s)}\}}^2$$

$$\begin{array}{l}\le 2\sum _{ijk}\left({\{{\theta }_{ijk}^{*}-{\left({\mathit{a}}_i^{(s)}\right)}^T{\mathit{R}}_k^{(s)}{\mathit{a}}_j^{(s)}\}}^2+{\{{\left({\mathit{a}}_i^{(s)}\right)}^T{\mathit{R}}_k^{(s)}{\mathit{a}}_j^{(s)}-{\left({\overline{\mathit{a}}}_{l_i}^{(s)}\right)}^T{\overline{\mathit{R}}}_{t_k}^{(s)}{\overline{\mathit{a}}}_{l_j}^{(s)}\}}^2\right)\\ \le \frac{2m^2{s}^2\exp \left(2{\omega }_a^2{\omega }_r\right)}{r_{n,K}^2}+\frac{6{\omega }_a^2{\omega }_r}{K}\le \frac{3m^2{s}^2\exp \left(2{\omega }_a^2{\omega }_r\right)}{r_{n,K}^2}.\end{array}$$ (47)

Let ${\mathrm{\Lambda }}_m^{(s)}=\{\left({\{l_i\}}_i,{\{t_k\}}_k\right):\sum _{ijk}{\{{\theta }_{ijk}^{*}-{\left({\overline{\mathit{a}}}_{l_i}^{(s)}\right)}^T{\overline{\mathit{R}}}_{t_k}^{(s)}{\overline{\mathit{a}}}_{l_j}^{(s)}\}}^2\le 3m^2{s}^2\exp \left(2{\omega }_a^2{\omega }_r\right)/r_{n,K}^2\}$. Similar to (36), we can show

$$\begin{gathered} \underset{s\in \{s_0,\dots ,s_{\max }\}}{\max }\frac{1}{s^2}\underset{\left({\{{\mathit{a}}_i^{(s)}\}}_i,{\{{\mathit{R}}_k^{(s)}\}}_k\right)\in {\mathbb{S}}_m^{(s)}}{\sup }\left|\sum _{ijk}\left(Y_{ijk}-{\pi }_{ijk}^{*}\right)\{{\theta }_{ijk}^{*}-{\left({\mathit{a}}_i^{(s)}\right)}^T{\mathit{R}}_k^{(s)}{\mathit{a}}_j^{(s)}\}\right| \\ \le \underset{s\in \{s_0,\dots ,s_{\max }\}}{\max }\frac{1}{s^2}\underset{\begin{array}{c}\begin{array}{l}{l}_1,\dots ,l_n\in \{1,\dots ,N_s,{\epsilon }_a\}\\ t_1,\dots ,t_K\in \{1,\dots ,N_{s^2,{\epsilon }_r}\}\end{array}\\ \left({\{1_i\}}_i,{\{t_k\}}_k\right)\in {\mathrm{\Lambda }}_m^{(s)}\end{array}}{\max }\left|\sum _{ijk}\left(Y_{ijk}-{\pi }_{ijk}^{*}\right)\{{\left({\overline{\mathit{a}}}_{l_i}^{(s)}\right)}^T{\overline{\mathit{R}}}_{t_k}^{(s)}{\overline{\mathit{a}}}_{l_j}^{(s)}-{\theta }_{ijk}^{*}\}\right|+\frac{3{\omega }_a^2{\omega }_r}{K}. \end{gathered}$$

Under the event defined in (46), for any m ≥ 9, we have

$$\underset{s\in \{s_0,\dots ,s_{\max }\}}{\max }\frac{1}{s^2}\underset{\begin{array}{c}\begin{array}{l}{l}_1,\dots ,l_n\in \{1,\dots ,N_s,{\epsilon }_a\}\\ t_1,\dots ,t_K\in \{1,\dots ,N_{s^2,{\epsilon }_r}\}\end{array}\\ \left({\{1_i\}}_i,{\{t_k\}}_k\right)\in {\mathrm{\Lambda }}_m^{(s)}\end{array}}{\max }\left|\sum _{ijk}\left(Y_{ijk}-{\pi }_{ijk}^{*}\right)\{{\left({\overline{\mathit{a}}}_{l_i}^{(s)}\right)}^T{\overline{\mathit{R}}}_{t_k}^{(s)}{\overline{\mathit{a}}}_{l_j}^{(s)}-{\theta }_{ijk}^{*}\}\right|$$

$$\ge \frac{{(m-1)}^2\exp ({\omega }_a^2{\omega }_r)}{8r_{n,K}^2}-\frac{3{\omega }_a^2{\omega }_r}{K}\ge \frac{{(m-1)}^2\exp ({\omega }_a^2{\omega }_r)}{16r_{n,K}^2}+\frac{4\exp ({\omega }_a^2{\omega }_r)}{r_{n,K}^2}-\frac{3{\omega }_a^2{\omega }_r}{K}\ge \frac{{(m-1)}^2\exp ({\omega }_a^2{\omega }_r)}{16r_{n,K}^2}.$$

Define

$${\left({\sigma }_m^{(s)}\right)}^2=\underset{\begin{array}{c}\begin{array}{l}{l}_1,\dots ,l_n\in \{1,\dots ,N_{s,{\epsilon }_{n,K}}\}\\ t_1,\dots ,t_K\in \{1,\dots ,N_{s^2,{\epsilon }_{n,K}}\}\end{array}\\ \left({\{1_i\}}_i,{\{t_k\}}_k\right)\in {\mathrm{\Lambda }}_m^{(s)}\end{array}}{\max }\sum _{ijk}\text{E}{\left|Y_{ijk}-{\pi }_{ijk}^{*}\right|}^2{\{{\left({\overline{\mathit{a}}}_{l_i}^{(s)}\right)}^T{\overline{\mathit{R}}}_{t_k}^{(s)}{\overline{\mathit{a}}}_{l_j}^{(s)}-{\theta }_{ijk}^{*}\}}^2.$$

By (47), it is immediate to see that

$${\left({\sigma }_m^{(s)}\right)}^2\le 3m^2{s}^2\exp \left(2{\omega }_a^2{\omega }_r\right)/r_{n,K}^2.$$ (48)

Similar to (37) and (40), we can show there exist some constants J₀ > 0, K₀ > 0 such that for any m ≥ J₀ and any s such that s₀ ≤ s ≤ s_max,

$$\mathrm{Pr}\left(\frac{1}{s^2}\underset{\begin{array}{c}\begin{array}{l}{l}_1,\dots ,l_n\in \{1,\dots ,N_{s,{\epsilon }_a}\}\\ t_1,\dots ,t_K\in \{1,\dots ,N_{s^2,{\epsilon }_r}\}\end{array}\\ \left({\{1_i\}}_i,{\{t_k\}}_k\right)\in {\mathrm{\Lambda }}_m^{(s)}\end{array}}{\max }\left|\sum _{ijk}\left(Y_{ijk}-{\pi }_{ijk}^{*}\right)\{{\left({\overline{\mathit{a}}}_{l_i}^{(s)}\right)}^T{\overline{\mathit{R}}}_{t_k}^{(s)}{\overline{\mathit{a}}}_{l_j}^{(s)}-{\theta }_{ijk}^{*}\}\right|\ge \frac{{(m-1)}^2\exp \left({\omega }_a^2{\omega }_r\right)}{16r_{n,K}^2}\right)$$

$$\le 2\exp \left(-\frac{{(m-1)}^4{s}^4\exp \left(2{\omega }_a^2{\omega }_r\right)/\left(256r_{n,K}^4\right)}{2{\left({\sigma }_m^{(s)}\right)}^2+{(m-1)}^2{s}^2{\omega }_a^2{\omega }_r\exp \left({\omega }_a{\omega }_r\right)M_0/\left(24r_{n,K}^2\right)}+\left(2sn+2s^{2}K\right)\log (3nK)\right)$$

$$\le 2\exp \left(-\frac{{(m-1)}^4{s}^4/\left(256r_{n,K}^4\right)}{6m^2{s}^2/r_{n,K}^2+{(m-1)}^2{s}^2/\left(24r_{n,K}^2\right)}+\left(2sn+2s^{2}K\right)\log (3nK)\right)\le 2\exp \left(-K_0{m}^2{s}^2{r}_{n,K}^{-2}\right),$$

where the second inequality is due to (38), (48) and the fact that ${\omega }_a^2{\omega }_r\le \exp ({\omega }_a^2{\omega }_r).$ This yields

$$\mathrm{Pr}\left(\left({\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k\right)\in {\mathbb{S}}_m^{(s)}\right)\le 2\exp \left(-K_0{m}^2{s}^2{r}_{n,K}^{-2}\right).$$ (49)

Let J_n,K be the integer such that $J_{n,K}-1\le {(nK)}^{1/4}\le J_{n,K}.$ In view of (44), we have for any m ≥ J₀,

$$\begin{array}{l}\mathrm{Pr}\left(\underset{s\in \{s_0,\dots ,s_{\max }\}}{\max }\frac{r_{n,K}}{s\exp \left({\omega }_a^2{\omega }_r\right)}\sqrt{\sum _{ijk}{\left({\theta }_{ijk}^{*}-{\widehat{\theta }}_{ijk}^{(s)}\right)}^2}>m\right)\le \sum _{s=s_0}^{s_{\max }}\sum _{l=m}^{J_{n,K}}2\exp \left(-K_0{l}^2{s}^2{r}_{n,K}^{-2}\right)+o(1)\\ \le \sum _{s=1}^{+\infty }\sum _{l=1}^{+\infty }2\exp \left(-K_{0}l{\mathit{mss}}_0{r}_{n,K}^{-2}\right)+o(1)\le \frac{2\exp \left(-s_{0}m{K}_0{r}_{n,K}^{-2}\right)}{1-2\exp \left(-s_{0}m{K}_0{r}_{n,K}^{-2}\right)}+o(1)=o(1).\end{array}$$

This implies there exists some constant C₀ > 0 such that the following event occurs with probability tending to 1,

$$\underset{s\in \{s_0,\dots ,s_{\max }\}}{\max }\frac{1}{s^2}\sum _{ijk}{\left({\widehat{\theta }}_{ijk}^{(s)}-{\theta }_{ijk}^{*}\right)}^2\le C_0\exp \left({\omega }_a^2{\omega }_r\right)r_{n,K}^{-2},$$ (50)

The proof is hence completed.

C.3. Proof of Lemma 3

For any s < s₀, we have

$$\begin{gathered} \sum _{ijk}{\left({\left({\widehat{\mathit{a}}}_i^{(s)}\right)}^T{\widehat{\mathit{R}}}_k^{(s)}{\widehat{\mathit{a}}}_j^{(s)}-{\left({\mathit{a}}_i^{*}\right)}^T{\mathit{R}}_k^{*}{\mathit{a}}_j^{*}\right)}^2 \\ \ge \underset{\begin{array}{l}{\mathit{a}}_1^{(s)},\dots ,{\mathit{a}}_n^{(s)}\in {ℝ}^s\\ {\mathit{R}}_1^{(s)},\dots ,{\mathit{R}}_K^{(s)}\in {ℝ}^{s\times s}\end{array}}{\inf }\sum _{ijk}{\left({\left({\mathit{a}}_i^{(s)}\right)}^T{\mathit{R}}_k^{(s)}{\mathit{a}}_j^{(s)}-{\left({\mathit{a}}_i^{*}\right)}^T{\mathit{R}}_k^{*}{\mathit{a}}_j^{*}\right)}^2 \end{gathered}$$ (51)

$$=\underset{\begin{array}{c}{\mathit{A}}^{(s)}\in {ℝ}^{n\times s}\\ {\mathit{R}}_1^{(s)},\dots ,{\mathit{R}}_K^{(s)}\in {ℝ}^{s\times s}\end{array}}{\inf }\sum _{k=1}^K{\Vert {\mathit{A}}^{(s)}{\mathit{R}}_k^{(s)}{\left({\mathit{A}}^{(s)}\right)}^T-{\mathit{A}}_0{\mathit{R}}_{k,0}{\mathit{A}}_0^T\Vert }_F^2$$

$$=\underset{\begin{array}{c}{\mathit{A}}^{(s)}\in {ℝ}^{n\times s}\\ {\mathit{R}}_1^{(s)},\dots ,{\mathit{R}}_K^{(s)}\in {ℝ}^{s\times s}\end{array}}{\inf }{\Vert \left(\begin{array}{c}{\mathit{A}}^{(s)}{\mathit{R}}_1^{(s)}\\ ⋮\\ {\mathit{A}}^{(s)}{\mathit{R}}_K^{(s)}\end{array}\right){\left({\mathit{A}}^{(s)}\right)}^T-\underset{{\mathit{B}}_0}{\underbrace{\left(\begin{array}{c}{\mathit{A}}_0{\mathit{R}}_{1,0}\\ ⋮\\ {\mathit{A}}_0{\mathit{R}}_{K,0}\end{array}\right)}}{\mathit{A}}_0^T\Vert }_F^2$$

$$\ge \underset{{\mathit{A}}^{(s)}\in {ℝ}^{n\times s},{\mathit{B}}^{(s)}\in {ℝ}^{nK\times s}}{\inf }{\Vert {\mathit{B}}^{(s)}{\left({\mathit{A}}^{(s)}\right)}^T-{\mathit{B}}_0{\mathit{A}}_0^T\Vert }_F^2.$$

Define

$$\left({\widehat{\mathit{A}}}^{(s)},{\widehat{\mathit{B}}}^{(s)}\right)=\arg \underset{\begin{array}{c}{\mathit{A}}^{(s)}\in {ℝ}^{n\times s}\\ {\mathit{B}}^{(s)}\in {ℝ}^{nK\times s}\end{array}}{\min }{\Vert {\mathit{B}}^{(s)}{\left({\mathit{A}}^{(s)}\right)}^T-{\mathit{B}}_0{\mathit{A}}_0^T\Vert }_F^2.$$

The above minimizers are not unique. Notice that rank $\left({\mathit{B}}_0{\mathit{A}}_0^T\right)\le \mathrm{rank}\left({\mathit{A}}_0^T\right)\le s_0.$. Assume ${\mathit{B}}_0{\mathit{A}}_0^T$ has the following singular value decomposition,

$${\mathit{B}}_0{\mathit{A}}_0^T=n{\mathit{U}}_n{\mathbf{\Lambda }}_n{\mathit{V}}_n^T,$$

for some ${\mathit{U}}_n\in {ℝ}^{nK\times s_0},{\mathit{V}}_n\in {ℝ}^{n\times s_0}\text{such that}{\mathit{U}}_n^T{\mathit{U}}_n={\mathit{V}}_n^T{\mathit{V}}_n={\mathit{I}}_{s_0},$ and some diagonal matrix

$${\mathbf{\Lambda }}_n=\mathrm{diag}\left({\text{λ}}_n^{(1)},{\text{λ}}_n^{(2)},\dots ,{\text{λ}}_n^{\left(s_0\right)}\right)$$

such that $\left|{\text{λ}}_n^{(1)}\right|\ge \left|{\text{λ}}_n^{(2)}\right|\ge \cdots \ge \left|{\text{λ}}_n^{\left(s_0\right)}\right|.$ Then one solution is given by

$${\widehat{\mathit{A}}}^{(s)}=n{\mathit{V}}_n{\mathbf{\Lambda }}_n{\mathit{U}}_n^T{\mathit{U}}_n^{(s)},{\widehat{\mathit{B}}}^{(s)}={\mathit{U}}_n^{(s)},$$

where ${\mathit{U}}_n^{(s)}$ is the submatrix of U_n formed by its first s columns.

Since ${\mathit{U}}_n^T{\mathit{U}}_n={\mathit{I}}_{s_0},$ we have

$$\begin{gathered} {\widehat{\mathit{B}}}^{(s)}{\left({\widehat{\mathit{A}}}^{(s)}\right)}^T=n{\mathit{U}}_n^{(s)}{\left({\mathit{U}}_n^{(s)}\right)}^T{\mathit{U}}_n{\mathbf{\Lambda }}_n{\mathit{V}}_n^T=n{\mathit{U}}_n^{(s)}\left({\mathit{I}}_s,{\mathit{O}}_{s,s_0-s}\right){\mathbf{\Lambda }}_n{\mathit{V}}_n^T \\ =n{\mathit{U}}_n\left(\begin{array}{cc}{\mathit{I}}_s& {\mathit{O}}_{s,s_0-s}\\ {\mathit{O}}_{s_0-s,s}& {\mathit{O}}_{s_0-s,s_0-s}\end{array}\right){\mathbf{\Lambda }}_n{\mathit{V}}_n^T, \end{gathered}$$

and hence

$$\begin{gathered} {\widehat{\mathit{B}}}^{(s)}{\left({\widehat{\mathit{A}}}^{(s)}\right)}^T-{\mathit{B}}_0{\mathit{A}}_0^T=n{\mathit{U}}_n\left(\begin{array}{cc}{\mathit{I}}_s& {\mathit{O}}_{s,s_0-s}\\ {\mathit{O}}_{s_0-s,s}& {\mathit{O}}_{s_0-s,s_0-s}\end{array}\right){\mathbf{\Lambda }}_n{\mathit{V}}_n^T-n{\mathit{U}}_n{I}_{s_0}{\mathbf{\Lambda }}_n{\mathit{V}}_n^T \\ =-n{\mathit{U}}_n\left(\begin{array}{cc}{\mathit{O}}_s& {\mathit{O}}_{s,s0-s}\\ {\mathit{O}}_{s_0-s,s}& {\mathit{I}}_{s_0-s,s_0-s}\end{array}\right){\mathbf{\Lambda }}_n{\mathit{V}}_n^T. \end{gathered}$$

This together with (51) implies that

$$\begin{array}{l}\sum _{ijk}{\left({\left({\widehat{\mathit{a}}}_i^{(s)}\right)}^T{\widehat{\mathit{R}}}_k^{(s)}{\widehat{\mathit{a}}}_j^{(s)}-{\left({\mathit{a}}_i^{*}\right)}^T{\mathit{R}}_k^{*}{\mathit{a}}_j^{*}\right)}^2\ge {\Vert {\widehat{\mathit{B}}}^{(s)}{\left({\widehat{\mathit{A}}}^{(s)}\right)}^T-{\mathit{B}}_0{\mathit{A}}_0^T\Vert }_F^2\\ \ge n^2\mathrm{trace}\left({\mathit{U}}_n\left(\begin{array}{cc}{\mathit{O}}_s& {\mathit{O}}_{s,s_0-s}\\ {\mathit{O}}_{s_0-s,s}& {\mathit{I}}_{s_0-s,s_0-s}\end{array}\right){\mathbf{\Lambda }}_n{\mathit{V}}_n^T{\mathit{V}}_n{\mathbf{\Lambda }}_n\left(\begin{array}{cc}{\mathit{O}}_s& {\mathit{O}}_{s,s_0-s}\\ {\mathit{O}}_{s_0-s,s}& {\mathit{I}}_{s_0-s,s_0-s}\end{array}\right){\mathit{U}}_n^T\right)\\ =n^2\mathrm{trace}\left({\mathit{U}}_n\left(\begin{array}{cc}{\mathit{O}}_s& {\mathit{O}}_{s,s_0-s}\\ {\mathit{O}}_{s_0-s,s}& {\mathit{I}}_{s_0-s,s_0-s}\end{array}\right){\mathbf{\Lambda }}_n^2\left(\begin{array}{cc}{\mathit{O}}_s& {\mathit{O}}_{s,s_0-s}\\ {\mathit{O}}_{s_0-s,s}& {\mathit{I}}_{s_0-s,s_0-s}\end{array}\right){\mathit{U}}_n^T\right)\end{array}$$ (52)

$$\begin{array}{l}=n^2\mathrm{trace}\left({\mathbf{\Lambda }}_n\left(\begin{array}{cc}{\mathit{O}}_s& {\mathit{O}}_{s,s_0-s}\\ {\mathit{O}}_{s_0-s,s}& {\mathit{I}}_{s_0-s,s_0-s}\end{array}\right){\mathit{U}}_n^T{\mathit{U}}_n\left(\begin{array}{cc}{\mathit{O}}_s& {\mathit{O}}_{s,s_0-s}\\ {\mathit{O}}_{s_0-s,s}& {\mathit{I}}_{s_0-s,s_0-s}\end{array}\right){\mathbf{\Lambda }}_n\right)\\ =n^2\mathrm{trace}\left({\mathbf{\Lambda }}_n\left(\begin{array}{cc}{\mathit{O}}_s& {\mathit{O}}_{s,s_0-s}\\ {\mathit{O}}_{s_0-s,s}& {\mathit{I}}_{s_0-s,s_0-s}\end{array}\right){\mathbf{\Lambda }}_n\right)\ge n^2{\left({\text{λ}}_n^{\left(s_0\right)}\right)}^2,\end{array}$$ (53)

where (52) is due to that ${\mathit{V}}_n^T{\mathit{V}}_n={\mathit{I}}_{s_0}$ and the equality in (53) is due to that ${\mathit{U}}_n^T{\mathit{U}}_n={\mathit{I}}_{s_0}.$

To summarize, we’ve shown

$$\sum _{ijk}{\left({\left({\widehat{\mathit{a}}}_i^{(s)}\right)}^T{\widehat{\mathit{R}}}_k^{(s)}{\widehat{\mathit{a}}}_j^{(s)}-{\left({\mathit{a}}_i^{*}\right)}^T{\mathit{R}}_k^{*}{\mathit{a}}_j^{*}\right)}^2\ge n^2{\left({\text{λ}}_n^{\left(s_0\right)}\right)}^2.$$ (54)

In the following, we provide a lower bound for ${\left({\text{λ}}_n^{\left(s_0\right)}\right)}^2.$ By definition, ${\left({\text{λ}}_{\text{n}}^{\left({\text{s}}_{\text{0}}\right)}\right)}^{\text{2}}$ is the s₀-th largest eigenvalue of

$$\frac{1}{n^2}{\mathit{A}}_0{\mathit{B}}_0^T{\mathit{B}}_0{\mathit{A}}_0^T=\frac{1}{n^2}\sum _{k=1}^K{\mathit{A}}_0{\mathit{R}}_{k,0}^T{\mathit{A}}_0^T{\mathit{A}}_0{\mathit{R}}_{k,0}{\mathit{A}}_0^T.$$

We first provide an lower bound for ${\lambda }_{\min }({\sum }_{k=1}^K{\mathit{R}}_{k,0}^T{\mathit{A}}_0^T{\mathit{A}}_0{\mathit{R}}_{k,0}/n).$ Let ${\mathrm{\Sigma }}_A={\mathit{A}}_0^T{\mathit{A}}_0/n.$ Consider the following eigenvalue decomposition:

$${\mathrm{\Sigma }}_A={\mathit{U}}_{\mathit{A}}{\mathbf{\Lambda }}_{\mathit{A}}{\mathit{U}}_{\mathit{A}}^T,$$

for some orthogonal matrix U_A and some diagonal matrix ${\mathbf{\Lambda }}_A.$ Under Assumption (A2), the matrix

$${\mathrm{\Sigma }}_A-\overline{c}{\mathit{I}}_{s_0}$$

is positive semidefinite. As a result, the matrix

$$\sum _{k=1}^K{\mathit{R}}_{k,0}^T\left({\mathrm{\Sigma }}_A-\overline{c}{\mathit{I}}_{s_0}\right){\mathit{R}}_{k,0}$$

is positive semidefinite. Therefore, we have

$${\text{λ}}_{\min }\left(\sum _{k=1}^K{\mathit{R}}_{k,0}^T{\mathrm{\Sigma }}_A{\mathit{R}}_{k,0}\right)=\underset{{\mathit{a}}_0\in {ℝ}^{s_0},{\Vert {\mathit{a}}_0\Vert }_2=1}{\inf }{\mathit{a}}_0^T\left(\sum _{k=1}^K{\mathit{R}}_{k,0}^T{\mathrm{\Sigma }}_A{\mathit{R}}_{k,0}\right){\mathit{a}}_0$$ (55)

$$\begin{array}{l}=\underset{{\mathit{a}}_0\in {ℝ}^{s_0},{\Vert {\mathit{a}}_0\Vert }_2=1}{\inf }\{{\mathit{a}}_0^T\left(\sum _{k=1}^K{\mathit{R}}_{k,0}^T\left({\mathrm{\Sigma }}_A-\overline{c}{\mathit{I}}_{s_0}\right){\mathit{R}}_{k,0}\right){\mathit{a}}_0+\overline{c}{\mathit{a}}_0^T\left(\sum _{k=1}^K{\mathit{R}}_{k,0}^T{\mathit{R}}_{k,0}\right){\mathit{a}}_0\}\\ \ge \overline{c}\underset{{\mathit{a}}_0\in {ℝ}^{s_0},{\Vert {\mathit{a}}_0\Vert }_2=1}{\inf }{\mathit{a}}_0^T\left(\sum _{k=1}^K{\mathit{R}}_{k,0}^T{\mathit{R}}_{k,0}\right){\mathit{a}}_0=\overline{c}{\lambda }_{\min }\left(\sum _{k=1}^K{\mathit{R}}_{k,0}^T{\mathit{R}}_{k,0}\right)=\overline{c}\overline{K}.\end{array}$$

By the eigenvalue decomposition, we have

$$\sum _{k=1}^K{\mathit{R}}_{k,0}^T{\mathrm{\Sigma }}_A{\mathit{R}}_{k,0}={\mathit{U}}_{RA}^T{\mathbf{\Lambda }}_{RA}{\mathit{U}}_{RA},$$

for some orthogonal matrix ${\mathbf{\Lambda }}_{RA}\in {ℝ}^{s_0\times s_0}$ and some diagonal matrix ${\mathit{U}}_{RA}\in {ℝ}^{s_0\times s_0}$. It follows from (55) that all the diagonal elements in ${\mathbf{\Lambda }}_{RA}$ are positive. Let ${\mathbf{\Lambda }}_{RA}^{1/2}$ be the diagonal matrix such that ${\mathbf{\Lambda }}_{RA}^{1/2}{\mathbf{\Lambda }}_{RA}^{1/2}={\mathbf{\Lambda }}_{RA}.$ Apparently, the diagonal elements in ${\mathbf{\Lambda }}_{RA}^{1/2}$ are nonzero. Notice that

$$\frac{1}{n^2}\sum _{k=1}^K{\mathit{A}}_0{\mathit{R}}_{k,0}^T{\mathit{A}}_0^T{\mathit{A}}_0{\mathit{R}}_{k,0}{\mathit{A}}_0^T=\frac{1}{n}{\mathit{A}}_0{\mathit{U}}_{RA}^T{\mathbf{\Lambda }}_{RA}^{1/2}{\mathit{U}}_{RA}{\mathit{U}}_{RA}^T{\mathbf{\Lambda }}_{RA}^{1/2}{\mathit{U}}_{RA}{\mathit{A}}_0^T.$$

The s₀ largest eigenvalues in $\frac{1}{n^2}\sum _{k=1}^K{\mathit{A}}_0{\mathit{R}}_{k,0}^T{\mathit{A}}_0^T{\mathit{A}}_0{\mathit{R}}_{k,0}{\mathit{A}}_0^T$ corresponds to the smallest eigenvalue in

$$\frac{1}{n}{\mathit{U}}_{RA}^T{\mathbf{\Lambda }}_{RA}^{1/2}{\mathit{U}}_{RA}{\mathit{A}}_0^T{\mathit{A}}_0{\mathit{U}}_{RA}^T{\mathbf{\Lambda }}_{RA}^{1/2}{\mathit{U}}_{RA}.$$

Similar to (55), we can show that

$$\begin{gathered} {\text{λ}}_{\min }\left(\frac{1}{n}{\mathit{U}}_{RA}^T{\mathbf{\Lambda }}_{RA}^{1/2}{\mathit{U}}_{RA}{\mathit{A}}_0^T{\mathit{A}}_0{\mathit{U}}_{RA}^T{\mathbf{\Lambda }}_{RA}^{1/2}{\mathit{U}}_{RA}\right)\ge \overline{c}{\lambda }_{\min }\left({\mathit{U}}_{RA}^T{\mathbf{\Lambda }}_{RA}^{1/2}{\mathit{U}}_{RA}{\mathit{U}}_{RA}^T{\mathbf{\Lambda }}_{RA}^{1/2}{\mathit{U}}_{RA}\right) \\ =\overline{c}{\text{λ}}_{\min }\left({\mathit{U}}_{RA}^T{\mathbf{\Lambda }}_{RA}{\mathit{U}}_{RA}\right)\ge {\overline{c}}^2{\lambda }_{\min }\left(\sum _{k=1}^K{\mathit{R}}_{k,0}^T{\mathrm{\Sigma }}_A{\mathit{R}}_{k,0}\right). \end{gathered}$$

Combining this together with (55), we obtain that

$${\left({\text{λ}}_{n,k}^{\left(s_0\right)}\right)}^2\ge {\overline{c}}^2\overline{K}.$$

It follows from (54) that

$$\sum _{ijk}{\left({\left({\widehat{\mathit{a}}}_i^{(s)}\right)}^T{\widehat{\mathit{R}}}_k^{(s)}{\widehat{\mathit{a}}}_j^{(s)}-{\left({\mathit{a}}_i^{*}\right)}^T{\mathit{R}}_k^{*}{\mathit{a}}_j^{*}\right)}^2\ge n^2{\overline{c}}^2\overline{K}.$$

This completes the proof.

C.4. Proof of Theorem 1

It suffices to show

$$\mathrm{Pr}\left(\text{IC}\left(s_0\right)>\underset{1\le s<s_0}{\max }\text{IC}(s)\right)\to 1,$$ (56)

and

$$\mathrm{Pr}\left(\text{IC}\left(s_0\right)>\underset{s_0\le s\le s_{\max }}{\max }\text{IC}(s)\right)\to 1.$$ (57)

We first show (56). Combining Lemma 3 with (31), we obtain that

$$2{\mathcal{l}}_0\left({\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k\right)-2{\mathcal{l}}_0\left({\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k\right)\ge \frac{\sum _{ijk}{\left({\theta }_{ijk}^{*}-{\widehat{\theta }}_{ijk}^{(s)}\right)}^2}{4\exp \left({\omega }_a^2{\omega }_r\right)}\ge \frac{n^2{\overline{c}}^2\overline{K}}{4\exp \left({\omega }_a^2{\omega }_r\right)},$$

for any $s\in \{1,\dots ,s_0-1\}.$ Combining this with (42), we have that

$$\begin{gathered} 2{\mathcal{l}}_n\left(\mathit{Y};{\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k\right)-2{\mathcal{l}}_n\left(\mathit{Y};{\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k\right)\ge 2{\mathcal{l}}_0\left({\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k\right)-2{\mathcal{l}}_0\left({\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k\right) \\ -2\left|{\mathcal{l}}_n\left(\mathit{Y};{\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k\right)-{\mathcal{l}}_0\left({\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k\right)\right|-2\left|{\mathcal{l}}_n\left(\mathit{Y};{\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k\right)-{\mathcal{l}}_0\left({\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k\right)\right| \\ \ge \frac{{\overline{c}}^2{n}^2\overline{K}}{4\exp \left({\omega }_a^2{\omega }_r\right)}-24{\omega }_a^2{\omega }_{r}n\sqrt{K\max (n,K)\log (nK)}, \end{gathered}$$ (58)

with probability tending to 1.

Under the given conditions, we have $K~n^{l_0}\text{for some}0\le l_0\le 1$. This together with the condition $n^{\left(1-l_0\right)/2}\overline{K}\gg \exp \left(2{\omega }_a^2{\omega }_r\right)\sqrt{\log n}$ yields

$${\omega }_a^2{\omega }_{r}n\sqrt{K\max (n,K)\log (nK)}=O\left(\exp \left({\omega }_a^2{\omega }_r\right)n^{3/2+l_0/2}\sqrt{\log n}\right)\ll \frac{n^2\overline{K}}{\exp \left({\omega }_a^2{\omega }_r\right)}.$$ (59)

By (58), we have with probability tending to 1 that

$$2{\mathcal{l}}_n\left(\mathit{Y};{\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k\right)-2{\mathcal{l}}_n\left(\mathit{Y};{\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k\right)\ge \frac{{\overline{c}}^2{n}^2\overline{K}}{8\exp \left({\omega }_a^2{\omega }_r\right)},$$ (60)

for all 1 ≤ s < s₀. By definition, we have

$${\mathcal{l}}_n\left(\mathit{Y};{\{{\widehat{\mathit{a}}}_i^{\left(s_0\right)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{\left(s_0\right)}\}}_k\right)\ge {\mathcal{l}}_n\left(\mathit{Y};{\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k\right).$$

This together with (60) gives that for all 1 ≥ s < s₀,

$$2{\mathcal{l}}_n\left(\mathit{Y};{\{{\widehat{\mathit{a}}}_i^{\left(s_0\right)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{\left(s_0\right)}\}}_k\right)-2{\mathcal{l}}_n\left(\mathit{Y};{\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k\right)\ge \frac{{\overline{c}}^2{n}^2\overline{K}}{8\exp \left({\omega }_a^2{\omega }_r\right)},$$ (61)

with probability tending to 1.

Under the given conditions, we have $\kappa (n,K)\ll n^2\overline{K}/\exp \left({\omega }_a^2{\omega }_r\right).$ Under the event defined in (61), we have that

$$\begin{gathered} \text{IC}\left(s_0\right)-\text{IC}(s)=2{\mathcal{l}}_n\left(\mathit{Y};{\{{\widehat{\mathit{a}}}_i^{\left(s_0\right)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{\left(s_0\right)}\}}_k\right)-2{\mathcal{l}}_n\left(\mathit{Y};{\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k\right)-\left(s_0-s\right)\kappa (n,K) \\ \ge \frac{{\overline{c}}^2{n}^2\overline{K}}{8\exp \left({\omega }_a^2{\omega }_r\right)}-s_0\kappa (n,K)\gg 0, \end{gathered}$$

since s₀ is fixed. This proves (56).

Now we show (57). Similar to (37)-(42), we can show the following event occurs with probability tending to 1,

$$\begin{gathered} \underset{s\in \{s_0,\dots ,s_{\max }\} \\ {\mathit{a}}_1^{(s)},\dots ,{\mathit{a}}_n^{(s)}\in {\mathrm{\Omega }}_a,{\mathit{R}}_1^{(s)},\dots ,{\mathit{R}}_K^{(s)}\in {\mathrm{\Omega }}_r \\ \frac{d\{{\{{\mathit{a}}_i^{(s)}\}}_i,{\{{\mathit{R}}_k^{(s)}\}}_k;{\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k)}{s\exp \left({\omega }_a^2{\omega }_r\right)}=O\left(\frac{r_{n,K}^{-1}}{n\sqrt{K}}\right)}{\sup }\frac{1}{s^2}\left|\sum _{ijk}\left(Y_{ijk}-{\pi }_{ijk}^{*}\right)\{{\theta }_{ijk}^{*}-{\left({\mathit{a}}_i^{(s)}\right)}^T{\mathit{R}}_k^{(s)}{\mathit{a}}_j^{(s)}\}\right|=O\left(\frac{\exp \left({\omega }_a^2{\omega }_r\right)}{r_{n,K}^2}\right). \end{gathered}$$ (62)

By Lemma 2, we obtain with probability tending to 1,

$$\underset{s\in \{s_0,\dots ,s_{\max }\}}{\max }\frac{1}{s^2}\left|\sum _{ijk}\left(Y_{ijk}-{\pi }_{ijk}^{*}\right)\{{\theta }_{ijk}^{*}-{\left({\widehat{\mathit{a}}}_i^{(s)}\right)}^T{\widehat{\mathit{R}}}_k^{(s)}{\widehat{\mathit{a}}}_j^{(s)}\}\right|=O\left(\frac{\exp \left({\omega }_a^2{\omega }_r\right)}{r_{n,K}^2}\right),$$

and hence

$$\underset{s\in \{s_0,\dots ,s_{\max }\}}{\max }\frac{1}{s}\left|\sum _{ijk}\left(Y_{ijk}-{\pi }_{ijk}^{*}\right)\{{\theta }_{ijk}^{*}-{\left({\widehat{\mathit{a}}}_i^{(s)}\right)}^T{\widehat{\mathit{R}}}_k^{(s)}{\widehat{\mathit{a}}}_j^{(s)}\}\right|=O\left(\frac{s_{\max }\exp \left({\omega }_a^2{\omega }_r\right)}{r_{n,K}^2}\right).$$ (63)

Since ${\mathcal{l}}_0\left({\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k\right)\ge {\mathcal{l}}_0\left({\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k\right),$ under the event defined in (63), we have

$$\begin{gathered} 2{\mathcal{l}}_n\left(\mathit{Y};{\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k\right)-2{\mathcal{l}}_n\left(\mathit{Y};{\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k\right)\le 2{\mathcal{l}}_0\left({\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k\right)-2{\mathcal{l}}_0\left({\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k\right) \\ -2\left|\sum _{ijk}\left(Y_{ijk}-{\pi }_{ijk}^{*}\right)\{{\theta }_{ijk}^{*}-{\left({\widehat{\mathit{a}}}_i^{(s)}\right)}^T{\widehat{\mathit{R}}}_k^{(s)}{\widehat{\mathit{a}}}_j^{(s)}\}\right|=O\left(\frac{s{s}_{\max }\exp \left({\omega }_a^2{\omega }_r\right)}{r_{n,K}^2}\right). \end{gathered}$$

Notice that

$${\mathcal{l}}_0\left({\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k\right)\le {\mathcal{l}}_0\left({\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k\right),$$

we have with probability tending to 1 that

$$2{\mathcal{l}}_n\left(\mathit{Y};{\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k\right)-2{\mathcal{l}}_n\left(\mathit{Y};{\{{\widehat{\mathit{a}}}_i^{\left(s_0\right)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{\left(s_0\right)}\}}_k\right)\le O\left(\frac{s{s}_{\max }\exp \left({\omega }_a^2{\omega }_r\right)}{r_{n,K}^2}\right),$$ (64)

for all $s_0<s\le s_{\text{max}}.$ Under the event defined in (64), we have

$$\begin{gathered} \text{IC}\left(s_0\right)-\text{IC}(s)\ge 2{\mathcal{l}}_n\left(\mathit{Y};{\{{\widehat{\mathit{a}}}_i^{(s_0)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{\left(s_0\right)}\}}_k\right)-2{\mathcal{l}}_n\left(\mathit{Y};{\{{\widehat{\mathit{a}}}_i^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_k^{(s)}\}}_k\right)+\left(s-s_0\right)\kappa (n,K) \\ \ge \left(s-s_0\right)\kappa (n,K)-O\left(s{s}_{\max }\exp \left({\omega }_a^2{\omega }_r\right)r_{n,K}^{-2}\right). \end{gathered}$$

Under the condition $\kappa (n,K)\gg s_{\max }\exp \left({\omega }_a^2{\omega }_r\right)(n+K)(\log n+\log K),$ we have that

$$\mathrm{Pr}\left(\text{IC}\left(s_0\right)>\underset{s_0<s\le s_{\max }}{\max }\mathrm{IC}(s)\right)\to 1,$$

This proves (57). The proof is hence completed.

C.5. Proof of Corollary 1

Using similar arguments in Lemma 3, we can show for any permutation function π : $\{1,\dots ,n\}\to \{1,\dots ,n\},$

$$\sum _{ijk}{\left({\left({\widehat{\mathit{a}}}_{i,\pi }^{(s)}\right)}^T{\widehat{\mathit{R}}}_{k,\pi }^{(s)}{\widehat{\mathit{a}}}_{j,\pi }^{(s)}-{\left({\mathit{a}}_{i,\pi }^{*}\right)}^T{\mathit{R}}_{k,\pi }^{*}{\mathit{a}}_{j,\pi }^{*}\right)}^2\ge n^2{\overline{c}}^2\overline{K}.$$ (65)

In addition, it follows from (41) and the condition $K=O\left(n^{l_0}\right)$ that

$$\mathrm{Pr}\left(\underset{\begin{array}{c}s\in \{1,\dots ,{\mathrm{s}}_{\max }\}\\ \pi :\{1,\dots ,n\}\to \{1,\dots ,n\}\end{array}}{\max }\frac{1}{s}\left|\sum _{ijk}\left(Y_{ijk}-{\pi }_{ijk}^{*}\right){\left({\widehat{\mathit{a}}}_{i,\pi }^{(s)}\right)}^T{\widehat{\mathit{R}}}_{k,\pi }^{(s)}{\widehat{\mathit{a}}}_{j,\pi }^{(s)}\right|\le C_0{\omega }_a^2{\omega }_r{n}^{3/2+l_0/2}\sqrt{\log n}\right)\to 1,$$

and

$$\mathrm{Pr}\left(\frac{1}{s_0}\left|\sum _{ijk}\left(Y_{ijk}-{\pi }_{ijk}^{*}\right){\left({\mathit{a}}_i^{*}\right)}^T{\mathit{R}}_k^{*}{\mathit{a}}_j^{*}\right|\le C_0{\omega }_a^2{\omega }_r{n}^{3/2+l_0/2}\sqrt{\log n}\right)\to 1,$$

for some constant C₀ > 0. Hence, the following event occurs with probability tending to 1,

$$\max (\underset{s,\pi }{\max }\left|{\mathcal{l}}_0\left({\{{\widehat{\mathit{a}}}_{i,\pi }^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_{k,\pi }^{(s)}\}}_k\right)-{\mathcal{l}}_n\left(\mathit{Y};{\{{\widehat{\mathit{a}}}_{i,\pi }^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_{k,\pi }^{(s)}\}}_k\right)\right|/s,$$ (66)

$$\left|{\mathcal{l}}_0\left({\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k\right)-{\mathcal{l}}_n\left(\mathit{Y};{\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k\right)\right|/s_0)\le C_0{\omega }_a^2{\omega }_{r}n\sqrt{K\max (n,K)\log (nK)}.$$

Combining (65) with (31) yields

$$2{\mathcal{l}}_0\left({\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k\right)-2\underset{\begin{array}{c}s\in \{1,\dots ,s_0-1\}\\ \pi :\{1,\dots ,n\}\to \{1,\dots ,n\}\end{array}}{\max }{\mathcal{l}}_0\left({\{{\widehat{\mathit{a}}}_{i,\pi }^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_{k,\pi }^{(s)}\}}_k\right)\ge \frac{n^2{\overline{c}}^2\overline{K}}{4\exp \left({\omega }_a^2{\omega }_r\right)}.$$ (67)

Under the given conditions, we have

$$\frac{n^2{\overline{c}}^2\overline{K}}{4\exp \left({\omega }_a^2{\omega }_r\right)}\gg {\omega }_a^2{\omega }_{r}n\sqrt{K\max (n,K)\log (nK)}.$$

This together with (66) and (67) gives

$$2{\mathcal{l}}_n\left(\mathit{Y};{\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k\right)-2\underset{\begin{array}{c}s\in \{1,\dots ,s_0-1\}\\ \pi :\{1,\dots ,n\}\to \{1,\dots ,n\}\end{array}}{\max }{\mathcal{l}}_n\left(\mathit{Y};{\{{\widehat{\mathit{a}}}_{i,\pi }^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_{k,\pi }^{(s)}\}}_k\right)\ge \frac{n^2{\overline{c}}^2\overline{K}}{8\exp \left({\omega }_a^2{\omega }_r\right)},$$

with probability tending to 1.

Under Condition (A4), we have ${\mathcal{l}}_n\left(\mathit{Y};{\{{\widehat{\mathit{a}}}_{i,\pi }^{\left(s_0\right)}\}}_i,{\{{\widehat{\mathit{R}}}_{k,\pi }^{\left(s_0\right)}\}}_k\right)\ge {\mathcal{l}}_n\left(\mathit{Y};{\{{\mathit{a}}_{i,\pi }^{\left(s_0\right)*}\}}_i,\{{\mathit{R}}_{k,\pi }^{\left(s_0\right)*}\}\right)={\mathcal{l}}_n\left(\mathit{Y};{\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k\right).$ Therefore, the following event occurs with probability tending to 1,

$$\underset{\pi :\{1,\dots ,n\}\to \{1,\dots ,n\}}{\min }2\left({\mathcal{l}}_n\left(\mathit{Y};{\{{\widehat{\mathit{a}}}_{i,\pi }^{\left(s_0\right)}\}}_i,{\{{\widehat{\mathit{R}}}_{k,\pi }^{\left(s_0\right)}\}}_k\right)-\underset{s\in \{1,\dots ,s_0-1\}}{\max }{\mathcal{l}}_n\left(\mathit{Y};{\{{\widehat{\mathit{a}}}_{i,\pi }^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_{k,\pi }^{(s)}\}}_k\right)\right)\ge \frac{n^2{\overline{c}}^2\overline{K}}{\exp \left({\omega }_a^2{\omega }_r\right)}.$$

Under the given conditions in Corollary 1, we obtain

$$\mathrm{Pr}\left(\underset{\pi :\{1,\dots ,n\}\to \{1,\dots ,n\}}{\cap }\left\{{\text{IC}}_{\pi }\left(s_0\right)>\underset{1\le s\le s_0-1}{\max }{\text{IC}}_{\pi }(s)\right\}\right)\to 1.$$ (68)

Similar to (46) and (49), we can show

$$\begin{gathered} \mathrm{Pr}\left(\underset{\pi :\{1,\dots ,n\}\to \{1,\dots ,n\}}{\cup }\{\left(\{{\widehat{\mathit{a}}}_{i,\pi }^{(s)}\},{\widehat{\mathit{R}}}_{k,\pi }^{(s)}\right)\in {\mathbb{S}}_m^{(s)}\}\right) \\ \le \mathrm{Pr}\left(\underset{\left({\{{\mathit{a}}_i^{(s)}\}}_i,{\{{\mathit{R}}_k^{(s)}\}}_k\right)\in {\mathbb{S}}_m^{(s)}}{\sup }\left|\sum _{ijk}\left(Y_{ijk}-{\pi }_{ijk}^{*}\right)\{{\theta }_{ijk}^{*}-{\left({\mathit{a}}_i^{(s)}\right)}^T{\mathit{R}}_k^{(s)}{\mathit{a}}_j^{(s)}\}\right|\ge \frac{{(m-1)}^2{s}^2\exp \left({\omega }_a^2{\omega }_r\right)}{8r_{n,K}^2}\right) \\ \le 2\exp \left(-K_0{m}^2{s}^2{r}_{n,K}^{-2}\right), \end{gathered}$$

for some constant K₀ > 0. Using similar arguments in the proof of Lemma 2, this yields with probability tending to 1,

$$\underset{\begin{array}{c}s\in \{s_0,\dots ,s_{\max }\}\\ \pi :\{1,\dots ,n\}\to \{1,\dots ,n\}\end{array}}{\max }\frac{1}{s^2}{d}^2\left({\{{\widehat{\mathit{a}}}_{i,\pi }^{(s)}\}}_i,{\{{\widehat{\mathit{R}}}_{k,\pi }^{(s)}\}}_k;{\{{\mathit{a}}_i^{*}\}}_i,{\{{\mathit{R}}_k^{*}\}}_k\right)\le \frac{\exp \left(2{\omega }_a^2{\omega }_r\right)(n+K)(\log n+\log K)}{n^{2}K}.$$

This together with (62) gives

$$\underset{s\in \{s_0,\dots ,s_{\max }\}}{\max }\frac{1}{s}\left|\sum _{ijk}\left(Y_{ijk}-{\pi }_{ijk}^{*}\right)\{{\theta }_{ijk}^{*}-{\left({\widehat{\mathit{a}}}_{i,\pi }^{(s)}\right)}^T{\widehat{\mathit{R}}}_{k,\pi }^{(s)}{\widehat{\mathit{a}}}_{j,\pi }^{(s)}\}\right|=O\left(\frac{s_{\max }\exp \left({\omega }_a^2{\omega }_r\right)}{r_{n,K}^2}\right),$$

with probability tending to 1. Using similar arguments in the proof of Theorem 1, we can show

$$\mathrm{Pr}\left(\underset{\pi :\{1,\dots ,n\}\to \{1,\dots ,n\}}{\cap }\left\{{\text{IC}}_{\pi }\left(s_0\right)>\underset{s_0<s\le s_{\max }}{\max }{\mathrm{IC}}_{\pi }(s)\right\}\right)\to 1.$$

This together with (68) yields (16). The proof is hence completed.

Appendix D. Additional Simulation Results

We simulate the response ${\{Y_{ijk}\}}_{ijk}$ from the following model:

$$\begin{gathered} \mathrm{Pr}\left(Y_{ijk}=1|{\{{\mathit{a}}_i\}}_i,{\{{\mathit{R}}_k\}}_k\right)=\frac{\exp \left({\mathit{a}}_i^T{\mathit{R}}_k{\mathit{a}}_j\right)}{1+\exp \left({\mathit{a}}_i^T{\mathit{R}}_k{\mathit{a}}_j\right)}, \\ {\mathit{a}}_1,{\mathit{a}}_2,\dots ,{\mathit{a}}_n\stackrel{iid}{~}N\left(0,{\{{0.5}^{|i-j|}\}}_{i,j=1,\dots ,s_0}\right), \\ {\mathit{R}}_1={\mathit{R}}_2=\cdots ={\mathit{R}}_K=\text{diag}(\underset{s_0}{\underbrace{1,-1,1,-1,\dots ,1,-1}}). \end{gathered}$$

We use the same six settings as described in Section 4.2. In each setting, we further consider three scenarios, by setting s₀ = 2, 4 and 6. Reported in Table 3 and 4 are the percentage of selecting the true models (TP) and the average of $\widehat{s}$ selected by IC₀, IC_0.5, IC₁ and BIC over 100 replications.

Table 4:

Simulation results for Setting I, II and III (standard errors in parenthesis)

	s0 = 2		s0 = 4		s0 = 6
n = 100, K = 3	TP	$\widehat{s}$	TP	$\widehat{s}$	TP	$\widehat{s}$
IC0	1.00(0.00)	2.00(0.00)	0.96(0.02)	3.98(0.02)	0.88(0.03)	5.87(0.04)
IC0.5	1.00(0.00)	2.00(0.00)	0.96(0.02)	3.98(0.02)	0.88(0.03)	5.87(0.04)
IC1	1.00(0.00)	2.00(0.00)	0.96(0.02)	3.98(0.02)	0.85(0.04)	5.81(0.05)
BIC	0.00(0.00)	11.98(0.01)	0.00(0.00)	11.99(0.01)	0.00(0.00)	12.00(0.00)
n = 150, K = 3	TP	$\widehat{s}$	TP	$\widehat{s}$	TP	$\widehat{s}$
IC0	1.00(0.00)	2.00(0.00)	0.97(0.02)	4.03(0.02)	0.94(0.02)	6.04(0.02)
IC0.5	1.00(0.00)	2.00(0.00)	0.97(0.02)	4.03(0.02)	0.94(0.02)	6.04(0.02)
IC1	1.00(0.00)	2.00(0.00)	0.97(0.02)	4.03(0.02)	0.94(0.02)	6.04(0.02)
BIC	0.00(0.00)	12.00(0.00)	0.00(0.00)	12.00(0.00)	0.00(0.00)	11.99(0.01)
n = 200, K = 3	TP	$\widehat{s}$	TP	$\widehat{s}$	TP	$\widehat{s}$
IC0	1.00(0.00)	2.00(0.00)	0.97(0.02)	4.03(0.02)	0.98(0.01)	6.02(0.01)
IC0.5	1.00(0.00)	2.00(0.00)	0.97(0.02)	4.03(0.02)	0.98(0.01)	6.02(0.01)
IC1	1.00(0.00)	2.00(0.00)	0.97(0.02)	4.03(0.02)	0.98(0.01)	6.02(0.01)
BIC	0.00(0.00)	12.00(0.00)	0.00(0.00)	12.00(0.00)	0.00(0.00)	11.99(0.01)

It can be seen from Table 4 and 5 that our proposed information criteria are consistent. In contrast, BIC fails in all settings. In addition, in the last two settings, TPs of IC_0.5 and IC₁ are larger than IC₀ for most of the cases. As commented before, these differences are due to the finite sample correction term τ_α(n, K).

Table 5:

Simulation results for Setting IV, V and VI (standard errors in parenthesis)

	s0 = 2		s0 = 4		s0 = 6
n = 50, K = 10	TP	$\widehat{s}$	TP	$\widehat{s}$	TP	$\widehat{s}$
IC0	1.00(0.00)	2.00(0.00)	0.96(0.02)	3.98(0.02)	0.73(0.04)	5.83(0.06)
IC0.5	1.00(0.00)	2.00(0.00)	0.95(0.02)	3.97(0.02)	0.69(0.05)	5.77(0.06)
IC1	1.00(0.00)	2.00(0.00)	0.93(0.03)	3.93(0.03)	0.63(0.05)	5.57(0.07)
BIC	0.00(0.00)	11.83(0.05)	0.00(0.00)	11.82(0.04)	0.00(0.00)	11.86(0.04)
n = 50, K = 20	TP	$\widehat{s}$	TP	$\widehat{s}$	TP	$\widehat{s}$
IC0	0.98(0.01)	2.02(0.01)	0.90(0.03)	4.10(0.03)	0.76(0.04)	6.06(0.05)
IC0.5	0.98(0.01)	2.02(0.01)	0.94(0.02)	3.98(0.02)	0.81(0.04)	5.99(0.04)
IC1	0.98(0.01)	2.02(0.01)	0.94(0.02)	3.94(0.02)	0.74(0.04)	5.81(0.05)
BIC	0.00(0.00)	12.00(0.00)	0.00(0.00)	12.00(0.00)	0.00(0.00)	11.99(0.01)
n = 50, K = 50	TP	$\widehat{s}$	TP	$\widehat{s}$	TP	$\widehat{s}$
IC0	0.96(0.02)	2.04(0.02)	0.88(0.03)	4.12(0.03)	0.68(0.05)	6.57(0.13)
IC0.5	0.98(0.01)	2.02(0.01)	0.94(0.02)	4.04(0.02)	0.82(0.04)	6.06(0.04)
IC1	0.98(0.01)	2.02(0.01)	0.94(0.02)	4.02(0.02)	0.74(0.04)	5.75(0.05)
BIC	0.00(0.00)	12.00(0.00)	0.00(0.00)	12.00(0.00)	0.00(0.00)	12.00(0.00)