Fast Moment Estimation for Generalized Latent Dirichlet Models

Abstract

We develop a generalized method of moments (GMM) approach for fast parameter estimation in a new class of Dirichlet latent variable models with mixed data types. Parameter estimation via GMM has computational and statistical advantages over alternative methods, such as expectation maximization, variational inference, and Markov chain Monte Carlo. A key computational advantage of our method, Moment Estimation for latent Dirichlet models (MELD), is that parameter estimation does not require instantiation of the latent variables. Moreover, performance is agnostic to distributional assumptions of the observations. We derive population moment conditions after marginalizing out the sample-specific Dirichlet latent variables. The moment conditions only depend on component mean parameters. We illustrate the utility of our approach on simulated data, comparing results from MELD to alternative methods, and we show the promise of our approach through the application to several datasets. Supplementary materials for this article are available online.

1. Introduction

Many modern statistical applications require the analysis of large-scale, heterogeneous data types including continuous, categorical, and count variables. For example, in social science, data often consist of collections of different data types (e.g., height, gender, and age); in population genetics, researchers are interested in analyzing genotype (integer-valued) and heterogeneous traits (e.g., blood pressure, BMI, alcoholic drinks per day). Often data take the form of an n × p matrix Y = (y₁, …, y_n)^T, with y_i (y_i1, …, y_ip)^T a p dimensional vector of measurements of varying types for subject i, for i =1, …, n.

This article focuses on (1) developing a new class of generalized latent variable models for mixed data types, and (2) performing fast and robust parameter estimation using the generalized method of moments (GMM). Many models have been proposed to address the first goal (reviewed below), but often these models lack robustness both statistically and computationally. In addition, parameter estimation in these models is often both inefficient and unstable, which necessitates our second goal. Hence, there is a clear need for new classes of models and corresponding robust and efficient approaches for routine inference under these models in general applications.

For modeling mixed scale data, there are two general strategies that have been most commonly used in the literature. The first is to assume an underlying Gaussian model, and then to characterize the feature dependence through a structured model of the Gaussian covariance (Muthén 1984). Such models are routinely used in the social science literature, focusing almost entirely on the case in which data are categorical or continuous, with the categorical variables arising via thresholding of Gaussian variables. These models are related to Gaussian copula models, which have been developed for broad classes of mixed scale data also including counts (Murray et al. 2013). The second approach is to define an exponential family distribution for each of the p variables, inducing dependence between variables through generalized linear models containing shared latent variables (Sammel, Ryan, and Legler 1997; Moustaki and Knott 2000; Dunson 2003).

There are a number of disadvantages to the above approaches. The underlying Gaussian framework is restrictive in forcing continuous variables to be Gaussian and non-continuous variables to be categorical. The exponential family latent variable models are potentially more flexible, but in practice suffer in several respects. Most notably, there is a fundamental lack of robustness due to the model structure, which arises because of the dual role of the latent variables in controlling sample dependence and the shape of the marginal distributions. For example, if there are two count variables, y_i1 and y_i2, that are highly correlated, then both counts must load strongly on a similar set of latent variables, which will induce over-dispersion in the marginal distributions with the variance much larger than the mean. There is no possibility of having highly correlated counts that do not have high over-dispersion. Copula models solve this problem by imposing a restrictive Gaussian copula covariance structure. Another fundamental issue is computation, which tends to rely on expectation-maximization (EM) or Markov chain Monte Carlo (MCMC) algorithms that alternate between updating latent variables and population parameters, intrinsically leading to slow convergence, inefficiency, and instability.

One promising approach to address the above issues is to rely on a generalized method of moments (GMM) estimator, which relates to the second goal in this article. The GMM estimator is robust to misspecification of the higher-order moments, while also leading to optimal efficiency among all estimators using only information on the initial moments (Hansen 1982). Applications of GMM to structural models with latent variables have a long history, with early examples including Bentler (1983) and Anderson and Gerbing (1988), among others. More recently, Gallant, Giacomini, and Ragusa (2013) applied GMM to a specific class of latent variable models by defining moment conditions based on the complete data, including the latent variables. In contrast, Bollen, Kolenikov, and Bauldry (2014) relied on a model-implied instrumental variable GMM estimator. Generally, current GMM methods focus on latent variable models that satisfy restrictive assumptions or require the instantiation of latent variables in a computationally intensive estimation algorithm.

The focus of our work is on a broad new class of latent variable models for mixed scale data. This eliminates some of the problems of current exponential family latent variable models, while also enabling derivation of an efficient GMM implementation that marginalizes out latent variables. With the first goal in mind, we focus on generalized mixed membership models, which incorporate Dirichlet latent variables. Mixed membership models have a rich literature with applications ranging from inference of ancestral populations in genomic data (Pritchard, Stephens, and Donnelly 2000a; Pritchard et al. 2000b) to topic modeling of documents (Blei, Ng, and Jordan 2003). Using Dirichlet latent variables to define cluster membership allows samples to partially belong to each of k latent components. Our model class includes latent Dirichlet allocation (Blei, Ng, and Jordan 2003) and simplex factor models (Bhattacharya and Dunson 2012) as special cases; however, we go beyond current approaches by allowing mixed scale data.

For the second goal, we develop an efficient moment tensor approach for parameter estimation with the Dirichlet latent variables marginalized out. This objective is related to recent moment tensor methods developed for latent variable models including mixtures of Gaussians, hidden Markov models, mixed membership models, and stochastic block models (Arora, Ge, and Moitra 2012; Anandkumar, Hsu, and Kakade 2012a; Anandkumar et al. 2012b; Hsu and Kakade 2013; Anandkumar et al. 2014a, 2014b). This recent work finds moment tensors of these latent variable models that have a symmetric PARAFAC (parallel factors) tensor decomposition (Kiers 2000). Parameter estimation using a moment tensor approach is performed as follows: (1) third-order empirical moment tensors are transformed to an orthogonal decomposable form; (2) orthogonal decompositions are performed using tensor power methods or singular value decompositions; (3) parameter estimates are recovered from the decomposition of the moment tensors. The moment tensor approach offers substantial computational advantages over other approximation methods, as illustrated in various applications (Tung and Smola 2014; Anandkumar et al. 2014a; Colombo and Vlassis 2015).

In this work, we develop a moment tensor approach for generalized mixed membership models. Our approach, moment estimation for latent Dirichlet models (MELD), differs from previous approaches in that the moment functions are defined for variables with different generative distributions. This goes beyond previous moment tensor approaches that construct moment tensors for homogeneous data distributions. Our approach constructs second- and third-order moments for variables with different data types. Parameter estimates are performed in the GMM framework by minimizing quadratic forms of the moment functions using a coordinate descent algorithm. Our algorithm scales as either O(p²) or O(p³), depending on whether third moments are used in the GMM procedure. In the setting where n is large and p is moderate, our algorithm shows clear computational advantages over alternative methods such as MCMC or EM. We also provide results regarding the asymptotic efficiency of our estimators.

In Section 2, we characterize the class of generalized Dirichlet latent variable models in MELD. In Section 3, we briefly review GMM and introduce the estimation procedure used in MELD. Asymptotic properties of the GMM estimator are also discussed. In Section 4, we evaluate the performance in a simulation study. We apply our method to three examples in Section 5 and conclude with a discussion in Section 6.

2. Generalized Dirichlet Latent Variable Models

In this section, we specify a generalized Dirichlet latent variable model. Let y_i = (y_i1, …, y_ip)^T denote a vector of p measurements having heterogeneous data types (e.g., continuous, categorical, and counts) over subjects i = 1, …, n. We are interested in flexible models for joint distributions of the data. A simple way to achieve this is to assume the elements of y_i are conditionally independent given a component index $s_i\in \left\{1,\dots ,k\right\}$, with the densities for each variable in each component having a parametric form. This is inflexible due to the assumption that each subject i belongs to exactly one component. We instead allow subjects to partially belong to each of the k components. We assume that the elements of y_i are conditionally independent given a latent Dirichlet vector ${\mathit{x}}_i={\left(x_{i1},\dots ,x_{ik}\right)}^T\in {\mathrm{\Delta }}^{k-1}$, with ∆^k−1 denoting the (k – 1) probability simplex. In particular, we let

$$y_{ij}\sim \sum _{h=1}^k{x}_{ih}{g}_j\left({\mathit{\varphi }}_{jh}\right),$$ (1)

where g_j (ϕ_jh) is the density of the jth variable specific to component h. The elements of x_i are interpretable as probability weights for subject i on each of k components; in this setting, pure subjects have weight vectors with all zeros except for a single one.

The corresponding density g_j (ϕ_jh) is absolutely continuous with respect to a dominating measure $\left(\mathrm{\Omega },\mathcal{H},\mu \right)$. In this article, we focus on exponential family models for the density, but the procedures we propose will hold for general parametric forms. Our model in Equation (1) generalizes current mixed membership models to allow mixed data types. The parameters $\mathbf{\Phi }={\left\{{\mathit{\varphi }}_{jh}\right\}}_{1\le j\le p,1\le h\le k}$ are mixture component parameters shared by all subjects. A full likelihood specification is completed by choosing a population distribution for the latent variable vector, x_i ∼ P, with P a distribution on the simplex ∆^k−1. We place a Dirichlet distribution on this latent variable vector x_i ∼ Dir(α) with parameter α = (α₁, …, α_k)^T.

Let m_i = (m_i1, … m_ip)^T denote a membership vector for subject i, where $m_{ij}\in \left\{1,\dots ,k\right\}$ indicates the component that feature j in subject i is generated from. We specify the following generative model:

$$y_{ij}\left|m_{ij}=h\sim g_j\left({\mathit{\varphi }}_{jh}\right),\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{m}_{ij}\right|{\mathit{x}}_i\sim \text{Multi}\left({\mathit{x}}_i\right),\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\mathit{x}}_i\sim \text{Dir}\left(\mathit{\alpha }\right).$$ (2)

When y_i is a multivariate categorical vector and g _j represents a multinomial distribution, Bhattacharya and Dunson (2012) proposed a simplex factor model. Their model reduces to latent Dirichlet allocation (Blei, Ng, and Jordan 2003) in the special case in which y_ij is the number of occurrences of word j in document i, and g_j (ϕ_jh) is a Poisson distribution, with scalar ϕ _jh indicating the rate of occurrence of word j in topic h. We also assume x_i ~ Dir(α), but allow a general form for the component specific densities. This leads to the following likelihood after marginalizing out the latent variables:

$$p\left(\mathit{Y}\left|\mathit{\alpha },\mathbf{\Phi }\right.\right)=\prod _{i=1}^n\left[\int \prod _{j=1}^p\left(\sum _{h=1}^k{x}_{ih}{g}_j\left(y_{ij}\left|{\mathit{\varphi }}_{jh}\right.\right)\right)\text{d}P\left({\mathit{x}}_i\right)\right].$$

Categorical data.

Let $y_{ij}\in \left\{1,\dots ,d_j\right\}$, for j = 1, …, p, so that we have multivariate categorical data that can be organized as a p-way contingency table. Let c = (c₁, …, c_p)^T with $c_j\in \left\{1,\dots ,d_j\right\}$ and ${\pi }_{\mathit{c}}=\mathrm{Pr}\left({\mathit{y}}_i=\mathit{c}\right)=\mathrm{Pr}\left(y_{i1}=c_1,\dots ,y_{ip}=c_p\right)$. Then $\mathit{\pi }=\left\{{\pi }_{\mathit{c}}\right\}$ is a probability tensor (Dunson and Xing 2009) satisfying ${\pi }_c\ge 0$ and ${\sum }_{\mathit{c}}{\pi }_{\mathit{c}}=1$. In this case, $g_j\left({\mathit{\varphi }}_{jh}\right)\stackrel{d}{=}\text{Multi}\left({\mathit{\varphi }}_{jh}\right)$ is a multinomial distribution, with ${\mathit{\varphi }}_{jh}={\left({\mathit{\varphi }}_{jh1},\mathrm{\dots },{\mathit{\varphi }}_{jh{d}_j}\right)}^T\hspace{0.17em}\hspace{0.17em}\in {\mathrm{\Delta }}^{d_j-1}$ a probability vector. As shown by Bhattacharya and Dunson (2012), marginalizing out the mixture proportion x_i leads to a Tucker decomposition of π

$${\pi }_{\mathit{c}}=\sum _{h_1=1}^k\cdot \cdot \cdot \sum _{h_p=1}^k{g}_{h_1,\mathrm{\dots },h_p}\prod _{j=1}^p{\mathit{\varphi }}_{j{h}_j{c}_j}.$$

Mixed data types.

When y_i includes variables of mixed data types, g_j (ϕ_jh) belongs to the exponential family of distributions with mean parameter ϕ_jh. We are interested in estimating mean parameters $\mathbf{\Phi }=\left({\mathbf{\Phi }}_1;\mathrm{\dots };{\mathbf{\Phi }}_p\right)$ with ${\mathbf{\Phi }}_j=\left({\mathit{\varphi }}_{j1},\mathrm{\dots },{\mathit{\varphi }}_{jk}\right)$. For categorical variables y_ij, ϕ_jh is a probability vector indicating the distribution of the categories in component h; for noncategorical variables y_ij, ϕ_jh is a scalar term indicating the mean parameter of that feature for component h.

3. Generalized Methods of Moments Estimation

In this section, we provide a procedure for parameter estimation in generalized Dirichlet latent variable models based on a GMM framework. We first state the moment functions and then propose a two-stage procedure for parameter estimation.

3.1. Moment Functions Used in MELD

The idea behind method of moments (MM) algorithms is to derive a list of moment equations that have expected value zero at the true parameter values. Given observations ${\left\{{\mathit{y}}_i\right\}}_{1\le i\le n}$, MM algorithms specify ℓ moment functions and define a moment vector $f\left(y_i,\mathit{\theta }\right)={\left(f_1\left(y_i,\mathit{\theta }\right),\mathrm{\dots },f_{\ell }\left(y_i,\mathit{\theta }\right)\right)}^T$ such that $\mathbb{E}\left[f\left(y_i,\mathit{\theta }\right)\right]=0$ at the true parameter $\mathit{\theta }={\mathit{\theta }}_0\in \mathbf{\Theta }\subseteq {ℝ}^p$. The moment vector is then used to specify an estimator by solving the following ℓ sample equations with p unknowns:

$$f_n\left(\mathit{\theta }\right)=\mathbf{0},$$ (3)

where $f_n\left(\mathit{\theta }\right)=\frac{1}{n}{\sum }_{i=1}^{n}f\left({\mathit{y}}_i,\mathit{\theta }\right)$ is the sample estimate of $\mathbb{E}\left[f\left({\mathit{y}}_i,\mathit{\theta }\right)\right]$. When f (y_i, θ) is linear in θ (e.g., linear regression with instrumental variables) with independent moment functions and ℓ = p, then θ can be uniquely determined. When ℓ> p, the system in Equation (3) might be over-determined, in which case standard MM cannot be applied.

Generalized method of moments (GMM) minimizes the following quadratic form rather than solving a linear system:

$$Q_n\left(\mathit{\theta };{\mathit{A}}_n\right)=f_n{\left(\mathit{\theta }\right)}^T{\mathit{A}}_n{f}_n\left(\mathit{\theta }\right),$$ (4)

where A_n is the asymptotically optimal positive semidefinite matrix:

$${\mathit{A}}_n^{-1}={\mathit{S}}_n=\text{Var}\left[n^{1/2}{f}_n\left(\mathit{\theta }\right)\right].$$

In the over-determined setting, minimizing the quadratic form still well defined. Hansen’s two stage procedure (Hansen 1982) obtains an initial estimate of $\widehat{\mathit{\theta }}$ using a suboptimal weight matrix, such as the identity. This initial estimate is used to calculate A_n, which is then plugged into Equation (4) to obtain the final estimate (Hall 2005), which is consistent and asymptotically normal (Hansen 1982).

Applying GMM to all of the parameters in the generalized Dirichlet latent variable model is not feasible due to the dimensionality of the parameter space. Higher-order moment functions are complex and involve large numbers of unknowns. We will make use of a tensor decomposition and define moment functions that only depend on lower order moments. This extends recent moment tensor approaches to our proposed broad class of latent variable models (Arora, Ge, and Moitra 2012; Anandkumar, Hsu, and Kakade 2012a; Anandkumar et al. 2012b; Hsu and Kakade 2013; Anandkumar et al. 2014a, 2014b). One key innovation in our procedure is that we allow for lower order interaction moments across variables, where each variable can have a different distribution. The moment functions will have rank one decompositions of the component mean parameters over the heterogeneous variables. These heterogeneous low-order moment functions lead to fast and robust parameter estimation in MELD, as we show in the simulation and application sections.

We now define the moment functions. Recall that y_ij is the jth variable for subject i, which may include various data types, including continuous, categorical, or count data. Each categorical variable y_ij is encoded as a vector b_ij of length d_j containing all zeros except for a one in position y_ij. If y_ij is a non-categorical variable, then b_ij = y_ij is a scalar value. In the following, we assume that there are k latent components and that the latent variable x_i follows a Dirichlet distribution with parameters α = (α₁, …, α_k)^T and ${\alpha }_0={\sum }_{h=1}^k{\alpha }_h$. Given an observation b_ij and hyperparameter α, we define moment functions based on second and third moments:

$$\begin{gathered} {\mathit{F}}_{jt}^{\left(2\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)={\mathit{b}}_{ij}\circ {\mathit{b}}_{it}-\frac{{\alpha }_0}{{\alpha }_0+1}{\mathit{\mu }}_j\circ {\mathit{\mu }}_t \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}-{\mathbf{\Phi }}_j{\text{Λ}}^{\left(2\right)}{{\mathbf{\Phi }}_t}^T,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}1\le j,t\le p,j\ne t \end{gathered}$$ (5)

$$\begin{array}{l}{\mathit{F}}_{jst}^{\left(3\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)={\mathit{b}}_{ij}\circ {\mathit{b}}_{is}\circ {\mathit{b}}_{it}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}-\frac{{\alpha }_0}{{\alpha }_0+2}\left({\mathit{b}}_{ij}\circ {\mathit{b}}_{is}\circ {\mathit{\mu }}_t+{\mathit{\mu }}_j\circ {\mathit{b}}_{is}\circ {\mathit{b}}_{it}\right.\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\left.+{\mathit{b}}_{ij}\circ {\mathit{\mu }}_s\circ {\mathit{b}}_{it}\right)+\frac{2{\alpha }_0^2}{\left({\alpha }_0+1\right)\left({\alpha }_0+2\right)}{\mathit{\mu }}_j\circ {\mathit{\mu }}_s\circ {\mathit{\mu }}_t\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}-{\Lambda }^{\left(3\right)}{\times }_1{\mathbf{\Phi }}_j{\times }_2{\mathbf{\Phi }}_s{\times }_3{\mathbf{\Phi }}_t,\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}1\le j,s,t\le p,\hspace{0.33em}\hspace{0.17em}j\ne s\ne t.\end{array}$$ (6)

We use ◦ to denote an outer product, and ×_s indicates multiplication of a tensor with a matrix for mode s. The means μ _j take the values ${\mathit{\mu }}_j=\mathbb{E}\left({\mathit{b}}_{ij}\left|\mathbf{\Phi }\right.\right)=\frac{1}{{\alpha }_0}{\mathbf{\Phi }}_j\mathit{\alpha }$ for j = 1, …, p. The second moment function ${\mathit{F}}_{jt}^{\left(2\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)$ is a d_j × d_t matrix, and the third moment function ${\mathit{F}}_{jt}^{\left(3\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)$ is a d_j × d_s × d_t tensor. We set d_j = 1 when variable j is noncategorical. The matrix ${\text{Λ}}^{\left(2\right)}$ is a k × k matrix with $\mathit{\alpha }/\left[{\alpha }_0\left({\alpha }_0+1\right)\right]$ along the diagonals and zero for all other entries. ${\text{Λ}}^{\left(3\right)}$ is a three-way tensor with the diagonal entry ${\lambda }_h^{\left(3\right)}=2{\alpha }_h/\left[{\alpha }_0\left({\alpha }_0+1\right)\left({\alpha }_0+2\right)\right]$ for h = 1, …, k and all the other entries are zero.

The following theorem states that, at the true parameter value, the expectations of the moment functions are zero. The proof can be found in Appendix A.

Theorem 1 (Moment conditions in MELD). The expectations of the second moment matrix ${\mathit{F}}_{jt}^{\left(2\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)$ and third moment tensor ${\mathit{F}}_{jt}^{\left(3\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)$ defined in Equations (5) and (6) are zero at true model parameter values ${\mathbf{\Phi }}_0$

$$\mathbb{E}\left[{\mathit{F}}_{jt}^{\left(2\right)}\left({\mathit{y}}_i,{\mathbf{\Phi }}_0\right)\right]=\mathbf{0},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\mathbb{E}\left[{\mathit{F}}_{jst}^{\left(3\right)}\left({\mathit{y}}_i,{\mathbf{\Phi }}_0\right)\right]=\mathbf{0},$$

with the expectations taken with respect to y_i.

3.2. Two Stage Optimal Estimation

To fix notation, we first restate Hansen’s two stage GMM procedure as:

1. estimate $\widehat{\mathit{\theta }}=\arg {\min }_{\mathit{\theta }}{Q}_n\left(\mathit{\theta };\mathit{I}\right)$;

2. given $\widehat{\mathit{\theta }}$ calculate S_n and set ${\mathit{A}}_n={\mathit{S}}_n^{-1}$;

3. compute $\widehat{\mathit{\theta }}=\arg {\min }_{\mathit{\theta }}{Q}_n\left(\mathit{\theta };{\mathit{A}}_n\right)$ as the final parameter estimate.

We specify two versions of MELD, which we denote as MELD-2 and MELD-3, depending on whether we use only the second moment matrix or both the second and third moment matrices for estimation.

We now define the second and third moment vectors. We denote as $f^{\left(2\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)$ the second moment vector, which is constructed by stacking the second moment matrices in Equation (5) as follows:

$$\begin{array}{l}{f}^{\left(2\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)=\left(\text{vec}{\left[{\mathit{F}}_{12}^{\left(2\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)\right]}^T,\mathrm{\dots },\text{vec}{\left[{\mathit{F}}_{1p}^{\left(2\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)\right]}^T,\right.\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\text{vec}{\left[{\mathit{F}}_{23}^{\left(2\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)\right]}^T,\mathrm{\dots },\text{vec}{\left[{\mathit{F}}_{2p}^{\left(2\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)\right]}^T,\mathrm{\dots },\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{\left.\text{vec}{\left[{\mathit{F}}_{p-1,p}^{\left(2\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)\right]}^T\right)}^T.\end{array}$$ (7)

As the moment matrix is symmetric in the sense that ${\mathit{F}}_{jt}^{\left(2\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)={\left[{\mathit{F}}_{tj}^{\left(2\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)\right]}^T$ we only consider the case where j < t. Assuming d levels for each element of y_i results in a moment vector of dimension p(p – 1)d²/2. The third moment vector $f^{\left(3\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)$ is constructed by stacking the second moment matrices and third moment tensors in Equations (5) and (6)

$$\begin{array}{l}{\mathit{f}}^{\left(3\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)=\left(\text{vec}{\left[{\mathit{F}}_{12}^{\left(2\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)\right]}^T,\mathrm{\dots },\text{vec}{\left[{\mathit{F}}_{p-1,p}^{\left(2\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)\right]}^T,\right.\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.17em}\hspace{0.17em}\text{vec}{\left[{\mathit{F}}_{123}^{\left(3\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)\right]}^T,\mathrm{\dots },\text{vec}{\left[{\mathit{F}}_{12p}^{\left(3\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)\right]}^T,\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.17em}\hspace{0.17em}\text{vec}{\left[{\mathit{F}}_{134}^{\left(3\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)\right]}^T,\mathrm{\dots },\text{vec}{\left[{\mathit{F}}_{13p}^{\left(3\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)\right]}^T\mathrm{\dots },\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.17em}\hspace{0.17em}{\left.\text{vec}{\left[{\mathit{F}}_{p-2,p-1,p}^{\left(3\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)\right]}^T\right)}^T.\end{array}$$ (8)

The moment matrices ${\mathit{F}}_{jt}^{\left(2\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)$ follow the same order as in ${\mathit{f}}^{\left(2\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)$. The third moment tensors ${\mathit{F}}_{jst}^{\left(3\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)$ are ordered such that the index on the right runs faster than the index on the left in the subscript. The third moment tensor is symmetric with respect to any permutations of its indices, so we include only the moment tensors with j < s < t. The dimension of this moment vector is $p\left(p-1\right)d^2/2+\left[p^3-3p\left(p-1\right)-p\right]d^3/6$ with the assumption of d levels for each element of y_i.

Based on the two moment vector versions, we define the following two quadratic functions used for our GMM estimation:

$$Q_n^{\left(2\right)}\left(\mathbf{\Phi };{\mathit{A}}_n^{\left(2\right)}\right)={\mathit{f}}_n^{\left(2\right)}{\left(\mathbf{\Phi }\right)}^T{\mathit{A}}_n^{\left(2\right)}{\mathit{f}}_n^{\left(2\right)}\left(\mathbf{\Phi }\right),$$ (9)

$$Q_n^{\left(3\right)}\left(\mathbf{\Phi };{\mathit{A}}_n^{\left(3\right)}\right)={\mathit{f}}_n^{\left(3\right)}{\left(\mathbf{\Phi }\right)}^T{\mathit{A}}_n^{\left(3\right)}{\mathit{f}}_n^{\left(3\right)}\left(\mathbf{\Phi }\right),$$ (10)

where ${\mathit{f}}_n^{\left(2\right)}\left(\mathbf{\Phi }\right)$ and ${\mathit{f}}_n^{\left(3\right)}\left(\mathbf{\Phi }\right)$ are sample estimates of the expectations of the moment vectors

$${\mathit{f}}_n^{\left(2\right)}\left(\mathbf{\Phi }\right)=\frac{1}{n}\sum _{i=1}^n{\mathit{f}}^{\left(2\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right),\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\mathit{f}}_n^{\left(3\right)}\left(\mathbf{\Phi }\right)=\frac{1}{n}\sum _{i=1}^n{\mathit{f}}^{\left(3\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right),$$

and ${\mathit{A}}_n^{\left(2\right)}$ and ${\mathit{A}}_n^{\left(3\right)}$ are two positive semidefinite matrices. We emphasize that when calculating ${\mathit{f}}_n^{\left(2\right)}\left(\mathbf{\Phi }\right)$, the μ _j and μ_t in ${\mathit{f}}^{\left(2\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)$ are replaced by their sample estimates ${\widehat{\mathit{\mu }}}_j=\frac{1}{n}{\sum }_{i=1}^n{\mathit{b}}_{ij}$ and ${\widehat{\mathit{\mu }}}_t=\frac{1}{n}{\sum }_{i=1}^n{\mathit{b}}_{it}$, instead of their parametric counterparts $\frac{1}{{\alpha }_0}{\mathbf{\Phi }}_j\mathit{\alpha }$ and $\frac{1}{{\alpha }_0}{\mathbf{\Phi }}_t\mathit{\alpha }$. In doing this the second moment matrix ${\mathit{F}}_{jt}^{\left(2\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)$ becomes a function of Φ only through ${\mathbf{\Phi }}_j{\Lambda }^{\left(2\right)}{\mathbf{\Phi }}_t^T$, which is a rank one summation of ϕ_jh and ϕ_th. This method facilitates development of a fast coordinate descent algorithm for parameter estimation. Similar methods are used for calculating ${\mathit{f}}_n^{\left(3\right)}\left(\mathbf{\Phi }\right)$.

In the first stage, we set the weight matrix to an identity matrix. Then, the quadratic functions in Equations (9) and (10) can be rewritten as follows:

$$Q_n^{\left(2\right)}\left(\mathbf{\Phi },\mathit{I}\right)=\sum _{j=1}^{p-1}\sum _{t=j+1}^p{‖{\mathit{F}}_{n,jt}^{\left(2\right)}\left(\mathbf{\Phi }\right)‖}_F^2,$$

$$\begin{array}{l}{Q}_n^{\left(3\right)}\left(\mathbf{\Phi },\mathit{I}\right)=\sum _{j=1}^{p-1}\sum _{t=j+1}^p{‖{\mathit{F}}_{n,jt}^{\left(2\right)}\left(\mathbf{\Phi }\right)‖}_F^2\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+\sum _{j=1}^{p-2}\sum _{s=j+1}^{p-1}\sum _{t=s+1}^p{‖{\mathit{F}}_{n,jst}^{\left(3\right)}\left(\mathbf{\Phi }\right)‖}_F^2,\end{array}$$

where ${\mathit{F}}_{n,jt}^{\left(2\right)}\left(\mathbf{\Phi }\right)$ and ${\mathit{F}}_{n,jst}^{\left(3\right)}\left(\mathbf{\Phi }\right)$ are defined as

$${\mathit{F}}_{n,jt}^{\left(2\right)}\left(\mathbf{\Phi }\right)=\frac{1}{n}\sum _{i=1}^n{\mathit{F}}_{jt}^{\left(2\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right),\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\mathit{F}}_{n,jst}^{\left(3\right)}\left(\mathbf{\Phi }\right)=\frac{1}{n}\sum _{i=1}^n{\mathit{F}}_{jst}^{\left(3\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right),$$

with μ_j, μ_s and μ_t replaced by ${\widehat{\mathit{\mu }}}_j$, ${\widehat{\mathit{\mu }}}_s$ and ${\widehat{\mathit{\mu }}}_t$ respectively in ${\mathit{F}}_{jt}^{\left(2\right)}\left({\text{y}}_i,\mathbf{\Phi }\right)$ and ${\mathit{F}}_{jst}^{\left(3\right)}\left({\text{y}}_i,\mathbf{\Phi }\right)$. ${‖\cdot ‖}_F^2$ indicates the Frobenius norm, the element-wise sum of squares.

We obtain a first-stage estimator of Φ by minimizing the quadratic forms using Newton-Raphson. Note that, after we substitute μ _j by its sample estimate, only the last term of ${\mathit{F}}_{n,jt}^{\left(2\right)}\left(\mathbf{\Phi }\right)$ and ${\mathit{F}}_{n,jst}^{\left(3\right)}\left(\mathbf{\Phi }\right)$ involves unknown parameter Φ. For simplicity, we denote

$${\mathit{E}}_{n,jt}^{\left(2\right)}={\mathit{F}}_{n,jt}^{\left(2\right)}\left(\mathbf{\Phi }\right)+{\mathbf{\Phi }}_j{\Lambda }^{\left(2\right)}{\mathbf{\Phi }}_t^T,$$

$${\mathit{E}}_{n,jst}^{\left(3\right)}={\mathit{F}}_{n,jst}^{\left(3\right)}\left(\mathbf{\Phi }\right)+{\Lambda }^{\left(3\right)}{\times }_1{\mathbf{\Phi }}_j{\times }_2{\mathbf{\Phi }}_s{\times }_3{\mathbf{\Phi }}_t.$$

The two quantities ${\mathit{E}}_{n,jt}^{\left(2\right)}$ and ${\mathit{E}}_{n,jst}^{\left(3\right)}$ can be computed directly from the samples. We optimize ϕ_jh with other mean parameters fixed. After calculating the gradient and Hessian of $Q_n^{\left(2\right)}\left({\mathit{\varphi }}_{jh},\mathit{I}\right)$, the update rule simply becomes

$${\mathit{\varphi }}_{jh}^s=\frac{{{\sum }_{t=1,t\ne j}^p\left({\overline{\mathit{E}}}_{n,jt}^{\left(2\right)}{\mathit{\varphi }}_{th}\right)}^T}{\left({\lambda }_h^{\left(2\right)}\right){\sum }_{t=1,t\ne j}^p{\mathit{\varphi }}_{th}^T{\mathit{\varphi }}_{th}},$$ (11)

where ${\overline{\mathit{E}}}_{n,jt}^{\left(2\right)}={\mathit{E}}_{n,jt}^{\left(2\right)}-{\sum }_{h^{\prime }\ne h}{\lambda }_{h^{\prime }}^{\left(2\right)}{\mathit{\varphi }}_{j{h}^{\prime }}\circ {\varphi }_{t{h}^{\prime }}$ and ${\lambda }_h^{\left(2\right)}$ is the hth diagonal entry of ${\text{Λ}}^{\left(2\right)}$.

The update rule for ϕ_jh with $Q_n^{\left(3\right)}\left({\mathit{\varphi }}_{jh},\mathit{I}\right)$ can be calculated as

$${\mathit{\varphi }}_{jh}^s=\frac{{\lambda }_h^{\left(2\right)}{\sum }_{t=1,t\ne j}^p{\left({\overline{\mathit{E}}}_{n,jt}^{\left(2\right)}{\mathit{\varphi }}_{th}\right)}^T+{\lambda }_h^{\left(3\right)}{\sum }_{s=1,s\ne j}^p\left[{\sum }_{t=1,t\ne s,t\ne j}^p{\left({\overline{\mathit{E}}}_{n,jst}^{\left(3\right)}{\times }_2{\varphi }_{sh}{\times }_3{\mathit{\varphi }}_{th}\right)}^T\right]}{{\left({\lambda }_h^{\left(2\right)}\right)}^2{\sum }_{t=1,t\ne j}^p{\mathit{\varphi }}_{th}^T{\mathit{\varphi }}_{th}+\hspace{0.17em}\hspace{0.17em}{\left({\lambda }_h^{\left(3\right)}\right)}^2{\sum }_{s=1,s\ne j}^p\left[{\sum }_{t=1,t\ne s,t\ne j}^p\left({\varphi }_{sh}^T{\varphi }_{sh}\right)\left({\mathit{\varphi }}_{th}^T{\mathit{\varphi }}_{th}\right)\right]},$$ (12)

where ${\overline{\mathit{E}}}_{n,jst}^{\left(3\right)}={\mathit{E}}_{n,jst}^{\left(3\right)}-{\sum }_{h^{\prime }\ne h}{\lambda }_{h^{\prime }}^{\left(3\right)}{\mathit{\varphi }}_{j{h}^{\prime }}\circ {\varphi }_{s{h}^{\prime }}\circ {\varphi }_{t{h}^{\prime }}$ and ${\lambda }_h^{\left(3\right)}$ is the hth diagonal entry of ${\text{Λ}}^{\left(3\right)}$. The derivations can be found in Appendix E. After updating ϕ_jh using the above equations, we retract ϕ_jh to its probability simplex when y_ij is a categorical variable.

We find the optimization procedure to be robust to parameter initialization. Our default initialization sets the parameters to the observed frequencies for categorical variables and to the empirical means for noncategorical variables. We also initialized the parameters of categorical variables to equal weights or a draw from the uniform distribution over the simplex with almost identical results. Although the optimization problem in Equations (9) and (10) is nonconvex, our algorithm tended to find a local minimum after a small number of iterations. Our convergence criteria corresponded to a difference less than 1 × 10⁻⁵ in the objective between two iterations.

MELD-2 minimizes the quadratic form Q⁽²⁾(Φ, A⁽²⁾), while MELD-3 minimizes Q⁽³⁾(Φ, A⁽³⁾). After an initial consistent estimator of Φ is found, we calculate the asymptotic covariance matrix of the moment vectors S_n and define a new weight matrix A_n = (S_n)⁻¹ for second stage GMM estimation. The form of S_n can be derived analytically for both MELD-2 and MELD-3, and we provide the results in the supplement materials. The calculation of A_n requires the inversion of a full-rank dense matrix S_n, with computation scaling as O(p²d² ) for MELD-2 and O(p³d³ ) for MELD-3. When including the off-diagonal terms in the weight matrix, the updating rules become intrinsically complicated. In practice, we only extract the diagonal elements of S_n and let A_n = 1/diag[(S_n)] in the second stage. This approximation has been used in previous GMM implementations (Jöreskog and Sörbom 1987). The gradient descent update equations can be found by slightly modifying Equations (11) and (12) to include weights.

Note that the moment functions do not solve the identifiability problems with respect to Φ. When α is a constant vector, any permutation τ of 1, …, k with ${\mathbf{\Phi }}_j\left(\tau \right)=\left({\mathit{\varphi }}_{j\tau \left(1\right)},\dots ,{\mathit{\varphi }}_{j\tau \left(k\right)}\right)$ for all j = 1, …, p satisfies the moment condition. This problem is inherited from the label switching problem in mixture models. A similar situation occurs when there are ties in α and the permutation is restricted to each tie. In real applications, we typically do not find ties, and our parameter estimates tend to be close to the true parameter values.

3.2.1. Algorithmic Guidance

In deciding which version of MELD to apply to a particular problem, the main considerations are how to balance robustness to misspecification, statistical efficiency, and computational time. MELD-3 will have better statistical efficiency (when the model is correctly specified), greater computational cost as a function of p as it scales as O(p³ ), and is less robust to model misspecification. In practice, this tradeoff is somewhat complex, as it is difficult to test to what extent the data comply with the modeling assumptions on the third moment. As a general rule of thumb, we recommend using MELD-2 when the additional computational cost of MELD-3 is prohibitive. If p is modest (as in the political-economic data below), and thus the additional computational burden is small, we would generally recommend MELD-3.

The performance of the one-stage versus two-stage estimator relies on whether the matrix A_n is estimated accurately. If the variance matrix is accurately estimated, the two-stage method will not perform worse than the one-stage method. However, we find that, when the sample size is small or the model is misspecified, the variance can be poorly estimated and the one-stage method can outperform the two stage method. In general, if the sample size is small in comparison to the number of variables, we suggest using the one-stage estimator.

3.3. Properties of Parameter Estimates

We use GMM asymptotic theory to show that parameter estimates in MELD are consistent. We assume the following regularity conditions on the moment vector $\mathit{f}\left({\mathit{y}}_i,\mathbf{\Phi }\right)$ and the parameter space Θ.

Assumption 1 (Regularity conditions (Assumptions 3.2, 3.9, and 3.10 (Hall 2005))).

(1) $\mathit{f}\left({\mathit{y}}_i,\mathbf{\Phi }\right)$ is continuous on Θ for all ${\mathit{y}}_i\in \mathcal{Y};$; (2) $\mathbb{E}\left[\mathit{f}\left({\mathit{y}}_i,\mathbf{\Phi }\right)\right]<\infty$ and continuous for $\mathbf{\Phi }\in \mathbf{\Theta }$; 3) Θ is compact and $\mathbb{E}\left[{\sup }_{\mathbf{\Phi }\in \mathbf{\Theta }}‖\mathit{f}\left({\mathit{y}}_i,\mathbf{\Phi }\right)‖\right]<\infty$.

Remark 1. Conditions (1) and (2) are satisfied in MELD. Condition (3) is satisfied automatically for categorical variables, noting that ${\mathit{\varphi }}_{jh}\in {\mathrm{\Delta }}^{d_j-1}\subset {ℝ}^{d_j}$ is compact. For noncategorical variables, we can restrict support to a large compact domain such as closed intervals in $ℝ$ without sacrificing practical performance.

With these conditions, we further assume that the weight matrix A_n converges to a positive definite matrix A in probability. We define the population analogs of the quadratic functions as

$$Q_0\left(\mathbf{\Phi };\mathit{A}\right)=\mathbb{E}{\left[\mathit{f}\left({\mathit{y}}_i,\mathbf{\Phi }\right)\right]}^T\mathit{A}\mathbb{E}\left[\mathit{f}\left({\mathit{y}}_{i,}\mathbf{\Phi }\right)\right],$$ (13)

We have the following lemma showing the uniform convergence of $Q_n\left(\mathbf{\Phi };{\mathit{A}}_n\right)$.

Lemma 1 (Uniform convergence (Lemma 3.1 (Hall 2005))). Under the regularity conditions in Assumption 1

$$\underset{\mathbf{\Phi }\in \mathbf{\Theta }}{\sup }\left|Q_n\left(\mathbf{\Phi };{\mathit{A}}_n\right)-Q_0\left(\mathbf{\Phi };\mathit{A}\right)\right|\stackrel{p}{\to }0.$$

Theorem 2 (Consistency). Under the same conditions in Lemma 1, the estimator $\widehat{\mathbf{\Phi }}$ that minimizes Q_n(Φ;A_n) in MELD-2 and MELD-3 converges to the true parameter Φ₀ in probability.

The proof can be found in Appendix B.

The asymptotic normality of $\widehat{\mathbf{\Phi }}$ can also be established by assuming the following conditions on $\partial \mathit{f}\left({\mathit{y}}_i,\mathbf{\Phi }\right)/\partial \mathbf{\Phi }$.

Assumption 2 (Conditions on $\partial \mathit{f}\left({\mathit{y}}_i,\mathbf{\Phi }\right)/\partial \mathbf{\Phi }$ (Assumptions 3.5, 3.12, and 3.13. (Hall 2005))).

(1) $\partial \mathit{f}\left({\mathit{y}}_i,\mathbf{\Phi }\right)/\partial \mathbf{\Phi }$ exists and is continuous on Θ for all ${\mathit{y}}_i\in \mathcal{Y};$; (2) Φ₀ is an interior point of Θ; (3) $\mathbb{E}\left[\partial \mathit{f}\left({\mathit{y}}_i,\mathbf{\Phi }\right)/\partial \mathbf{\Phi }\right]={\mathit{G}}_0\left(\mathbf{\Phi }\right)<\infty$; (4) G₀(Φ) is continuous on some neighborhood $N_{ϵ}$ of Φ₀; (5) the sample estimate G_n(Φ) uniformly converges to G₀(Φ).

Remark 2. We derive $\partial \mathit{f}\left({\mathit{y}}_i,\mathbf{\Phi }\right)/\partial \mathbf{\Phi }$ for both MELD-2 and MELD-3 in Appendix D. Conditions 1, 3, and 4 are satisfied in MELD. Condition 5 can be shown with continuousness of the derivative and the compactness of Θ.

Theorem 3 (Asymptotic normality). With Assumptions 1 and 2, we have

$$n^{1/2}\left(\text{vec}\left(\widehat{\mathbf{\Phi }}\right)-\text{vec}\left({\mathbf{\Phi }}_0\right)\right)\stackrel{p}{\to }\mathcal{N}\left(\mathbf{0}\hspace{0.17em}\text{,}\mathit{M}\mathit{S}{\mathit{M}}^T\right)$$

with

$$\mathit{M}={\left[{\left({\mathit{G}}_0\right)}^T\mathit{A}{\mathit{G}}_0\right]}^{-1}{\left({\mathit{G}}_0\right)}^T\mathit{A},$$

where ${\mathit{G}}_0=\mathbb{E}\left[\partial \mathit{f}\left({\mathit{y}}_i,\mathbf{\Phi }\right)/\partial \mathbf{\Phi }\right]\left|{ }_{\mathbf{\Phi }={\mathbf{\Phi }}_0}\right.$ and $\mathit{S}={\lim }_{n\to \infty }Var\left[n^{1/2}{\mathit{f}}_n\left({\mathbf{\Phi }}_0\right)\right]$.

The proof can be found in Appendix C. The optimal estimator can be obtained so that the weight matrix A_{n → A} = (S)⁻¹ (Hansen 1982).

3.4. Model Selection Using Goodness of Fit Tests

We use goodness-of-fit tests to choose the number of latent components k. With the optimal weight matrix A_n → (S)⁻¹, Wald-, Lagrange multiplier- (score), and likelihood ratio-type tests can be constructed. We could construct a sequence of test statistics with increasing k to quantify the goodness-of-fit in MELD. However, this approach requires the calculation of the optimal weight matrix and large matrix inversions. Instead, we quantify goodness of fit using a fitness index (FI) (Bentler 1983). Practically, we show in an extensive simulation study that the FI has low error in estimating k in MELD.

The FI is defined based on the value of the objective function evaluated at the parameter estimate (Bentler 1983)

$$\text{FI}=1-\frac{Q_n\left(\widehat{\mathbf{\Phi }},{\mathit{A}}_n\right)}{{\left({\mathit{e}}_n\right)}^T{\mathit{A}}_n{\mathit{e}}_n},$$ (14)

where e_n is the concatenation of $\text{vec}\left({\mathit{E}}_{n,jt}^{\left(2\right)}\right)$ with j < t for MELD-2 and the concatenation of both $\text{vec}\left({\mathit{E}}_{n,jt}^{\left(2\right)}\right)$ with j < t and $\text{vec}\left({\mathit{E}}_{n,jst}^{\left(3\right)}\right)$ with j < s < t for MELD-3. This FI may be constructed for any weight matrix A_n. It can be viewed as a normalized objective function; thus, the FI has values less than one. Larger values of the FI indicate a better fit.

4. Simulation Study

In this section, we evaluate the accuracy and run time of MELD-2 and MELD-3 in simulations with categorical and mixed data types. In the first stage, an identity weight matrix is used. In the second stage, we set A_n = 1/diag[(S_n)]. We denote MELD-2-j and MELD-3- j as the procedure using up to the jth stage of MELD-2 and MELD-3, respectively, for j = 1, 2.

4.1. Categorical Data

For a categorical data simulation, we considered a low-dimensional setting (p = 20) so that both second- and third-order moment functions may be efficiently calculated. A simulation with a moderate dimension (p = 100) is summarized in the supplementary materials.

We assumed each of the 20 categorical variables has d = 4 levels. We set the number of components to k = 3 and generated ϕ_jh from Dir(0.5, 0.5, 0.5, 0.5) with h = 1, …, k; α was set to (0.1, 0.1, 0.1)^T. We drew n = {50, 100, 200, 500, 1000, 10000} samples from the model in Equation (1) with g_j (ϕ_jh) a multinomial distribution. For each value of n, we generated ten independent datasets. We ran MELD for different values of k = {1, …, 5}. The FI criterion consistently chose the correct number of latent components k in first stage estimation (Table 1). For second-stage estimation, FI did not perform as well. The trajectories of the objective functions under different values of k are shown in Figure S1. Convergence of parameter estimates for the ten simulated datasets under n = 1000 and k = 3 is shown in Figure S2. MELD-2 converged in about 25 iterations and MELD-3 converged in about 10 iterations.

Table 1.

Goodness-of-fit tests using the fitness index (FI) in categorical simulation. Larger values of FI indicate better fit, with the maximum at one. Results shown are based on ten simulated datasets for each value of n. Standard deviations of FI are provided in parentheses. Bold type indicates the best FI across k.

n	k	MELD-2–1	MELD-2–2	MELD-3–1	MELD-3–2
50	1	0.824(0.020)	− 1.865(4.950)	0.585(0.031)	− 27.987(40.128)
	2	0.908(0.011)	− 1.285(2.630)	0.745(0.032)	− 27.240(54.415)
	3	0.930(0.004)	0.570(0.044)	0.775(0.022)	− 27.713(42.607)
	4	0.901(0.010)	0.588(0.032)	0.735(0.023)	0.254(0.090)
	5	0.795(0.033)	0.686(0.021)	0.653(0.024)	0.305(0.091)
100	1	0.860(0.012)	− 0.521(2.142)	0.651(0.021)	− 0.031(0.703)
	2	0.930(0.011)	0.282(0.644)	0.795(0.030)	− 4.457(8.924)
	3	0.960(0.005)	0.677(0.044)	0.851(0.009)	− 4.868(11.889)
	4	0.942(0.012)	0.691(0.042)	0.822(0.010)	0.225(0.019)
	5	0.863(0.046)	0.782(0.022)	0.768(0.044)	0.232(0.080)
200	1	0.869(0.012)	0.679(0.060)	0.682(0.021)	0.298(0.070)
	2	0.940(0.007)	0.699(0.047)	0.838(0.014)	0.306(0.054)
	3	0.980(0.001)	0.761(0.061)	0.919(0.004)	0.278(0.049)
	4	0.967(0.006)	0.780(0.019)	0.891(0.008)	0.287(0.050)
	5	0.911(0.017)	0.824(0.008)	0.864(0.015)	0.286(0.042)
500	1	0.882(0.007)	0.783(0.022)	0.713(0.012)	0.414(0.080)
	2	0.948(0.006)	0.799(0.019)	0.870(0.013)	0.427(0.080)
	3	0.992( <0.001)	0.884(0.024)	0.966(0.001)	0.388(0.073)
	4	0.983(0.004)	0.874(0.026)	0.938(0.005)	0.365(0.065)
	5	0.937(0.014)	0.894(0.006)	0.921(0.007)	0.353(0.042)
1000	1	0.888(0.003)	0.828(0.006)	0.729(0.008)	0.571(0.017)
	2	0.951(0.003)	0.855(0.009)	0.881(0.008)	0.615(0.030)
	3	0.996( <0.001)	0.950(0.005)	0.982(0.001)	0.609(0.031)
	4	0.989(0.002)	0.938(0.004)	0.961(0.003)	0.579(0.030)
	5	0.953(0.010)	0.932(0.006)	0.951(0.006)	0.550(0.034)
10000	1	0.920(0.001)	0.853(0.003)	0.812(<0.001)	0.582(0.009)
	2	0.980(<0.001)	0.914(0.002)	0.940(<0.001)	0.725(0.005)
	3	0.999( <0.001)	0.992(0.001)	0.998( <0.001)	0.900(0.008)
	4	0.990(0.003)	0.977(0.004)	0.981(<0.001)	0.861(0.008)
	5	0.885(0.005)	10.929(0.002)	0.981(<0.001)	0.856(0.008)

We compared MELD with the simplex factor model (SFM) (Bhattacharya and Dunson 2012) and latent Dirichlet allocation (LDA) (Blei, Ng, and Jordan 2003). For the LDA model, we used the topicmodels package in R (Hornik and Grün 2011) with both collapsed Gibbs sampling and variational EM. In our notation, y_ij is a word and y_i is a document. Our generative model is more flexible than LDA in that each variable was allowed to have its own k component distributions with parameter ϕ_j1, …, ϕ_jk. For SFM and LDA, we ran 10,000 steps of MCMC with fixed k and a burn-in of 5000 iterations. We collected every 50th sample to thin the chain. For LDA variational EM, we used the package defaults. Dirichlet parameters were set to α = 0.1 and β = 0.5 for mixture proportions and topic distributions, respectively. Mean squared errors (MSEs) were calculated for each variable y_ij by recovering the associated membership variable m_ij from the estimated parameters, calculating the estimate of y_ij using the estimated Φ_j parameter for membership variable m_ij, and then computing the MSE of that estimated y_ij with the true value (see supplementary materials). MSEs for different values of k are shown in Figure 1, and the running times of different methods are reported in Table S1. The LDA implementation in topicmodels does not allow k = 1, so we did not consider this case. The results show that MELD-2–1 was the most accurate and the fastest in most cases. MELD-3–2 did not perform well when n was small (i.e., n = 50), but estimation accuracy improved for n larger. SFM had comparable MSEs when n was small. However, when n = 500, 1000, or 10,000, MELD outperformed SFM. The model we simulated from is more flexible than the standard LDA model, so it was not surprising that LDA did not perform as well as MELD and SFM.

Figure 1.

Comparison of mean squared error (MSE) of estimated parameters in categorical simulations. For SFM and LDA with Gibbs sampling, posterior means of parameters were calculated using 100 posterior draws on each of the ten simulated datasets. MSEs were calculated for each variable y_ij by recovering the associated membership variable m_ij from the estimated parameters, calculating the estimate of y_ij using the estimated Φ_j parameter for membership variable m_ij, and then computing the MSE of that estimated y_ij with the true value. Details of MSEs and their standard deviations are in Supplementary Table S2– S4.

To provide a comparison more favorable to LDA, we generated samples from the LDA model. This is equivalent to setting ${\mathit{\varphi }}_{1h}=\cdots ={\mathit{\varphi }}_{ph}$ for h = 1, …, k. We set the same values for α and k, and we draw ϕ_jh from Dir(0.5, 0.5, 0.5, 0.5). The MSEs with different contamination levels (see below) are shown in Tables S5–S7. When there was no contamination and n was small (i.e., n = 50 or 100), LDA had the highest accuracy in parameter estimation. However, when the sample size was large or there was contamination, MELD outperformed LDA.

We also found that MELD using two-stage estimation had larger MSEs compared to MELD using one-stage estimation. This was especially true when the number of observations was small or the number of latent components was misspecified. FIs for two-stage estimation were less accurate. The loss in accuracy appeared to be a result of poorly estimating the diagonal weight matrix. We confirmed this hypothesis by calculating mean squared distances between true and estimated diagonal weight matrices for different n and k (Table S8; Figure S3).

We further evaluated performance in the presence of contamination. For each simulated dataset, we randomly replaced a proportion of observations (4% and 10%) with draws from a discrete uniform distribution. MSEs under different values of k are shown in Figure 1. With 4% contamination, MELD had the most accurate parameter estimation in almost all cases. MELD-2–1 performed the best, followed by MELD-3–1. When we increased contamination to 10%, MELD had the most accurate MSE in all cases. MELD-2–1 performed best, followed by MELD-2-MELD-2 consistently performed better than MELD-3, as expected given the robustness of using only low-order moments.

To evaluate the performance of MELD in terms of selecting the correct number of components, we varied the number of components k = {6, 7, 8, 9, 10} and generated 10,000 observations using the same simulation model. We found that MELD-3 was able to estimate the correct number of components for k ≤ 8, MELD-2–2 was accurate for k ≤ 7 and MELD-2–2 for k = 6 (see Tables S9– S11). For k ≥ 9, none of the variants of MELD were able to correctly estimate the number of components (see Tables S12 and S13).

4.2. Mixed Data Types

We considered a simulation setting mimicking applications in which DNA sequence variations influence a quantitative trait. An additional simulation combining categorical, Gaussian, and Poisson variables is summarized in the Supplementary Materials. We generated a sequence of nucleotides {A, C, G, T} at 50 genetic loci along with a continuous or integer-valued trait, resulting in p = 51 variables. We set k = 2 latent components and simulated n = 1000 individuals, with the first 500 from the first component and the last 500 from the second component. The continuous traits were drawn from $\mathcal{N}\left(-3,1\right)$ and $\mathcal{N}\left(3,1\right)$, while count traits were drawn from Poisson (5) and Poisson(10), respectively, for the two components. We chose eight loci J = {2, 4, 12, 14, 32, 34, 42, 44} to be associated with the trait. The multinomial parameters for each of the two components were randomly drawn from Dir(0.5, 0.5, 0.5, 0.5). The association between the trait and the eight loci is given by marginalizing the component variables. The distribution for the remaining 42 loci was fixed across individuals independently of the trait as multinomial with parameters generated from Dir(0.5, 0.5, 0.5, 0.5). Ten datasets were simulated from the model in Equation (1). To assess robustness, we added contamination (e.g., genotyping errors) by randomly replacing 4%, 10%, and 20% of the nucleotides with values uniformly generated from {A, C, G, T}.

We ran MELD-2–1 and chose the number of components k = {1, …, 5}. The fitness test FI chose the correct value of k on all ten datasets (Table S14). For each genomic locus, we calculated its marginal frequency according to the simulated data, and then computed the averaged Kullback–Leibler (KL) distance between the estimated component distributions and the marginal frequency as follows:

$$\begin{array}{l}\text{aveKL}\left(y_{ij}\right)=\frac{1}{k}\sum _{h=1}^k\sum _{c_j=1}^{d_j}\mathrm{Pr}\left(y_{ij}=c_j\left|m_{ij}=h\right.\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.33em}\hspace{0.33em}\times \log \left(\frac{\mathrm{Pr}\left(y_{ij}=c_j\left|m_{ij}=h\right.\right)}{\mathrm{Pr}\left(y_{ij}=c_j\right)}\right).\end{array}$$ (15)

A smaller averaged KL distance suggests that the component distributions were closer to the marginal distribution, implying that the locus frequency was not differentiated across components. The set J corresponded exactly to the eight loci with largest averaged KL distance in the MELD results in most cases (Table 2).

Table 2.

Quantitative trait association simulation with 50 nucleotides and one response. Nucleotides not in J = {2, 4, 12, 14, 32, 34, 42, 44} are labeled by an underline and missing nucleotides are crossed out. Results shown are for one of the ten simulated datasets. The complete results can be found in Tables S15 and S16.

Response	Contamination	Q(2)(Φ) 1st stage	Bayesian copula factor model
Gaussian	0%	{2, 4, 12, 14, 32, 34, 42, 44}	{1, 2, $\cancel{4}$, 12, 14, 32, 34, 42, 44}
	4%	{2, 4, 12, 14, 32, 34, 42, 44}	{2, $\cancel{4}$, 5, 12, 14, 32, 35, $\cancel{3}4$, 42, 44}
	10%	{2, 4, 12, 14, 32, 34, 42, 44}	{2, $\cancel{4}$, 5, 12, 14, 25, 32, $\cancel{3}4$, 42, 44}
	20%	{2, 4, 12, 14, 32, 34, 42, 44}	{1, 2, 4, $\cancel{1}2$, $\cancel{1}4$, 25, 32, 34, 42, 44}
Poisson	0%	{2, 4, 12, 14, 32, 34, 42, 44}	{$\cancel{2}$, 3, 4, 12, $\cancel{1}4$, 32, 34, 37, 42, 44}
	4%	{2, 4, 12, 14, 32, 34, 42, 44}	{$\cancel{2}$, 3, 4, 12, 14, 32, 34, 42, 44}
	10%	{2, 4, 12, 14, 32, 34, 42, 44}	{$\cancel{2}$, 3, 4, 12, 14, 32, 34, 42, 44}
	20%	{2, 4, 7, 12, 14, 32, 34, $\cancel{4}2$, 44}	{$\cancel{2}$, 3, 4, 12, 14, 30, 32, 34, $\cancel{4}2$, 44}

We compared results from MELD with results from the Bayesian copula factor model (Murray et al. 2013), which estimates a correlation matrix C between variables. From this estimate, we computed partial correlations between the response variable (trait) and each genetic locus (Hoff 2007). We ran MCMC for the Bayesian copula factor model for 10,000 iterations with the correct value for k, a burn-in of 5000 iterations, and saving every 50th iteration. We selected genomic locations for which the 95% posterior intervals of the partial correlation did not include zero. Table 2 shows the eight loci having the largest absolute values of the posterior mean partial correlation. The Bayesian copula factor model had much higher error rates than MELD.

5. Applications of MELD

Promoter sequence analysis.

We applied MELD to gene transcription promoter data available in the UCI machine learning repository (Lichman 2013). The data include n = 106 nucleotide sequences {A, C, G, T} of length 57. Of those, 53 sequences are located in promoter regions, and 53 sequences are located in nonpromoter regions. The goal is binary classification: using nucleotide sequence alone, we would like to predict whether or not the sequence is in a promoter or a non-promoter region.

We included the promoter or nonpromoter status of the sequences as an additional binary variable, giving us p = 57 + 1 categorical variables. We applied MELD Q⁽²⁾(Φ) with first-stage estimation on the full data and also separately on the subsets of promoter and non-promoter sequences. We set k = {1, …, 8}. For k = 2, MELD converged in 2.13 seconds, compared with SFM, which took 41.6 seconds to perform 10,000 MCMC iterations. The FI test selected k = 2 for the full data, k = 2 for the promoter data, and k = 1 for the nonpromoter data (Table S17).

We set k = 2 in the following analysis. For each nucleotide position, we calculated the averaged KL distance between the estimated component distributions and its marginal distribution using Equation (15). An interpretation of the averaged KL distance is that it quantifies stratification of each nucleotide distribution across components: a larger value indicates greater stratification, suggesting that the nucleotide is important in defining and differentiating the components. For the full dataset, we observed approximately two peaks of the averaged KL distance, one around the 15th nucleotide and one around the 42nd (Figure S5). The first peak corresponds to the start of the biologically conserved region for promoter sequences (Harley and Reynolds 1987). For MELD applied only to promoter sequences, this peak was reduced, suggesting that, at approximately the 15th nucleotide, the components all included similarly well conserved distributions of this nucleotide. The estimated component membership variables also showed the importance of nucleotides around the 15th locus (Figures S6 and S7). For the peak around the 42nd nucleotide, this phenomenon was reversed: the increased averaged KL distance remained in promoter sequences but diminished in non-promoter sequences. One possible explanation is that this region was conserved uniformly in the non-promoter region, but differentially conserved in the promoter region. Applying MELD to the entire data and separately within each group of sequences (promoter, nonpromoter) provides insight into the sequence differences across the two groups.

Political-economic risk data.

In a second application, we applied MELD to political-economic risk data (Quinn 2004), which include five proxy variables of mixed types measured for 62 countries. The dataset has been analyzed using a mixed Gaussian and probit factor model (Quinn 2004) and a Bayesian copula factor model (Murray et al. 2013). The data are available in the MCMCpack package. There are three categorical variables and two real valued variables (Table 3).

Table 3.

Variables in the political-economic risk data

Variable	Type	Explanation
ind.jud	binary	An indicator variable that measures the independence of the national judiciary. This variable is equal to one if the judiciary is judged to be independent and equal to zero otherwise.
blk.mkt	real	Black-market premium measurement. Original values are measured as the black-market exchange rate (local currency per dollar) divided by the official exchange rate minus one. Quinn (2004) transformed the original data to log scale.
lack.exp.risk	ordinal	Lack of appropriation risk measurement. Six levels with coding 0 < 1 < 2 < 3 < 4 < 5.
lack.corrup	ordinal	Lack of corruption measurement. Six levels with coding 0 < 1 < 2 < 3 < 4 < 5.
gdp.worker	real	Real gross domestic product (GDP) per worker in 1985 international prices. Recorded data are log transformed.

We applied MELD with k = {1, …, 5} using both MELD-2–1 and MELD-3–1. The FI criterion selected k = 4 for MELD-2–1 and k = 3 for MELD-3–1 (Table S18). We chose results from MELD-3–1 with k = 3 for further analysis.

The estimated component parameters for the five variables showed distinct interpretations of the three components (Figure S8). We might interpret the three components as low-risk, intermediate-risk, and high-risk political–economic status, respectively. The first component had a high probability of independence of the national judiciary (ind.jud being one) and a low measurement of black-market premium. The first component also had a high probability of observing 4th and 5th levels in lack.exp.risk and 3rd, 4th, and 5th levels in lack.corrup. The mean of the GDP per worker was highest among the three components. The second component had a relatively high probability of being zero in ind.jud and a large mean value of blk.mkt. Both of lack of appropriation risk measurement and lack of corruption measurement put higher weights on lower category numbers (0, 1, and 2), indicating more risk and higher levels of corruption; the GDP per worker was still high. We might interpret this component as a society being relatively unstable while still having a good economic forecast, meaning that GDP per worker is high, possibly through the black market. The last component had the least judicial independence as quantified by the probability of ind.jud being zero. The black-market premium is also low, as is the lack of risk level and lack of corruption level. The GDP per worker is by far the lowest among the three components. We might interpret this component as societies being the most unstable with the greatest economic risk. We found that, although the three components had distinct stratification, each country was a mixture of the three components (Figure S9).

Human single nucleotide polymorphism data.

As a third application, we applied MELD to Human HapMap phase 3 (HM3) genotype data (Stranger et al. 2012). The HM3 data consist of 608 individuals from diverse ancestral populations. We focused on single nucleotide polymorphisms (SNPs) on chromosome 21 for which the frequency of the less frequent variant is greater than 5%. SNPs were represented by the number of copies of the less frequent variant at each genetic locus {1,2,3}. After selecting every 10th locus to avoid including well-correlated SNPs, we were left with 1672 SNPs for this application.

We applied MELD-2–1 to estimate genotype distributions in different populations. We considered different numbers of ancestral populations, including k = {1, 2, 3, 4, 5}. On this analysis, FI chose k = 2 (Table S19). The bands in the estimated membership variables shown in Figure S10 suggest a clear population structure among the 608 individuals. We calculated average KL distance for the 1,672 SNPs. The five SNPs with highest average KL distance are shown in Table S20, along with their genotypes in the samples. As with the previous applications, these SNPs define and differentiate the two inferred ancestral populations.

6. Discussion

In this article, we developed a new class of latent variable models with Dirichlet-distributed latent variables for mixed data types. These generalized latent Dirichlet variable models extend previous mixed membership models, such as LDA (Blei, Ng, and Jordan 2003) and simplex factor models (Bhattacharya and Dunson 2012), to allow mixed data types. For this class of models, we developed a fast parameter estimation procedure using the generalized method of moments. Our procedure extends recent moment tensor methods (Anandkumar et al. 2014b) to models with mixed data types and does not require instantiation of latent variables. We derive population moment conditions after marginalizing out the Dirichlet latent variables. We compared the two stages of our estimation procedure in different simulation settings and found that, for small-sample sizes and misspecified models, one-stage estimation had better performance. One possibility in terms of improving two-stage estimation is to apply a penalization procedure to better estimate the weight matrix.

Our results show that MELD is a promising alternative to MCMC or EM methods for producing fast and robust parameter estimates. An online method to update moment statistics when new samples arrive would allow its extension to streaming data. One limitation of our method is that the Newton-Raphson method is of order O(p² ) using second moment functions and order O(p³ ) using third moment functions. One possible approach to speed up computation when p is large is to use stochastic gradient methods to calculate an approximate gradient in each step. Another possible extension is to adapt our model to include ordinal variables; for example, by thresholding latent variables.

Appendix

A. Proof of Theorem 1

Proof. We start with the case where y_ij is categorical with d_j levels. The probability vector $x_{i=}{\left(x_{i1},\dots ,x_{ik}\right)}^T\in {\mathrm{\Delta }}^{k-1}$ defines the mixture proportion of individual i. We assume x_i ∼ Dir(α₁, …, α_k). Define normalizer ${\alpha }_0={\sum }_{h=1}^k{\alpha }_h$ and α = (α₁, …, α_k)^T.

We use the standard basis for encoding. We encode y_ij = c_j as ${\mathit{b}}_{ij}\in {ℝ}^{d_j}$, a binary (0/1) vector with the c_jth coordinate being 1 and others being 0. Similarly, we encode the membership variable m_ij as a k dimensional binary vector ${\mathit{m}}_{ij}\in {ℝ}^k$. Consider the first moment of b_ij,

$$\begin{array}{l}{\mathit{\mu }}_j=\mathbb{E}\left({\mathit{b}}_{ij}\right)=\mathbb{E}\left[\mathbb{E}\left({\mathit{b}}_{ij}|{\mathit{m}}_{ij}\right)\right]=\mathbb{E}\left({\mathbf{\Phi }}_j{\mathit{m}}_{ij}\right)\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em} =\mathbb{E}\left[\mathbb{E}\left({\mathbf{\Phi }}_j{\mathit{m}}_{ij}|{\mathit{x}}_i\right)\right]=\mathbb{E}\left({\mathbf{\Phi }}_j{\mathit{x}}_i\right)={\mathbf{\Phi }}_j\frac{\mathit{\alpha }}{{\alpha }_0},\end{array}$$

where Φ_j = (ϕ_j1, … ϕ_jk).

We consider second-order moment conditions. There are four types of second-order moments: same-variable, same-subject (type SS), same-variable, cross-subject (type SC), cross-variable, same-subject (type CS), and cross-variable, cross-subject (type CC). Of the four types, only the CS type is required to prove the theorem. The CS type second moment for b_ij and b_it ( j ≠ t) can be written as

$$\mathbb{E}\left({\mathit{b}}_{ij}\circ {\mathit{b}}_{it}\right)={\mathbf{\Phi }}_j\mathbb{E}\left({\mathit{m}}_{ij}\circ {\mathit{m}}_{it}\right){\mathbf{\Phi }}_t^T={\mathbf{\Phi }}_j\mathbb{E}\left({\mathit{x}}_i\circ {\mathit{x}}_i\right){\mathbf{\Phi }}_t^T.$$

For a Dirichlet distributed variable,

$$\begin{array}{l}\mathbb{E}\left({\mathit{x}}_i\circ {\mathit{x}}_i\right)=\mathrm{cov}\left({\mathit{x}}_i\right)+\mathbb{E}\left({\mathit{x}}_i\right)\circ \mathbb{E}\left({\mathit{x}}_i\right)\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em} =\frac{1}{{\alpha }_0\left({\alpha }_0+1\right)}\text{diag}\left(\mathit{\alpha }\right)+\frac{{\alpha }_0}{{\alpha }_0^2\left({\alpha }_0+1\right)}\mathit{\alpha }\circ \mathit{\alpha }.\end{array}$$

Then, we have

$$\begin{array}{l}\mathbb{E}\left({\mathit{b}}_{ij}\circ {\mathit{b}}_{it}\right)={\mathbf{\Phi }}_j\mathbb{E}\left({\mathit{x}}_i\circ {\mathit{x}}_i\right){\mathbf{\Phi }}_t^T\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\frac{1}{{\alpha }_0\left({\alpha }_0+1\right)}\sum _{h=1}^k{\alpha }_h{\mathit{\varphi }}_{jh}\circ {\mathit{\varphi }}_{th}+\frac{{\alpha }_0}{{\alpha }_0+1}{\mathit{\mu }}_j\circ {\mathit{\mu }}_t.\end{array}$$ (A.1)

We next consider third-order moment conditions. There are eight different types of third-order moments for b_ij. Only the moments with different variables for the same subject are required to prove the theorem.

We consider the third cross moment for b_ij, b_it, and b_is with j ≠ t ≠ s for the same subject. First, we calculate $\mathbb{E}\left({\mathit{m}}_{ij}\circ {\mathit{m}}_{is}\circ {\mathit{m}}_{it}\right)$,

$$\begin{array}{l}\mathbb{E}\left({\mathit{m}}_{ij}\circ {\mathit{m}}_{is}\circ {\mathit{m}}_{it}\right)=\mathbb{E}\left[\mathbb{E}\left({\mathit{m}}_{ij}\circ {\mathit{m}}_{is}\circ {\mathit{m}}_{it}|{\mathit{x}}_i\right)\right]\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em} =\mathbb{E}\left({\mathit{x}}_i\circ {\mathit{x}}_i\circ {\mathit{x}}_i\right)=\frac{1}{{\alpha }_0\left({\alpha }_0+1\right)\left({\alpha }_0+2\right)}\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em} \hspace{1em}\times \left(\left(\mathit{\alpha }\circ \mathit{\alpha }\circ \mathit{\alpha }\right)+\sum _{h=1}^k{\alpha }_h\left({\mathit{e}}_h\circ {\mathit{e}}_h\circ \mathit{\alpha }\right)\right.\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em} \hspace{1em}+\sum _{h=1}^k{\alpha }_h\left({\mathit{e}}_h\circ \mathit{\alpha }\circ {\mathit{e}}_h\right)+\sum _{h=1}^k{\alpha }_h\left(\mathit{\alpha }\circ {\mathit{e}}_h\circ {\mathit{e}}_h\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em} \hspace{1em}\left.+2\sum _{h=1}^k{\alpha }_h\left({\mathit{e}}_h\circ {\mathit{e}}_h\circ {\mathit{e}}_h\right)\right).\end{array}$$

Here, e_h is a standard basis vector of length k with the hth coordinate equal to one. The third order moment tensor of b_ij ◦ b_is ◦ b_it can be derived as

$$\begin{array}{l}\mathbb{E}\left({\mathit{b}}_{ij}\circ {\mathit{b}}_{is}\circ {\mathit{b}}_{it}\right)=\mathbb{E}\left({\mathit{m}}_{ij}\circ {\mathit{m}}_{is}\circ {\mathit{m}}_{it}\right)\times \left\{{\mathbf{\Phi }}_j,{\mathbf{\Phi }}_s,{\mathbf{\Phi }}_t\right\}\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=\frac{1}{{\alpha }_0\left({\alpha }_0+1\right)\left({\alpha }_0+2\right)}\left(\left({\mathbf{\Phi }}_j\mathit{\alpha }\right)\circ \left({\mathbf{\Phi }}_s\mathit{\alpha }\right)\circ \left({\mathbf{\Phi }}_t\mathit{\alpha }\right)\right.\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+\sum _{h=1}^k{\alpha }_h\left[{\mathit{\varphi }}_{jh}\circ {\varphi }_{sh}\circ \left({\mathbf{\Phi }}_t\mathit{\alpha }\right)\right]\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+\sum _{h=1}^k{\alpha }_h\left[{\mathit{\varphi }}_{jh}\circ \left({\mathbf{\Phi }}_s\mathit{\alpha }\right)\circ {\mathit{\varphi }}_{th}\right]\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+\sum _{h=1}^k{\alpha }_h\left[\left({\mathbf{\Phi }}_j\mathit{\alpha }\right)\circ {\varphi }_{sh}\circ {\mathit{\varphi }}_{th}\right]\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\left.+2\sum _{h=1}^k{\alpha }_h{\mathit{\varphi }}_{jh}\circ {\varphi }_{sh}\circ {\mathit{\varphi }}_{th}\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=\frac{1}{{\alpha }_0\left({\alpha }_0+1\right)\left({\alpha }_0+2\right)}\left({\alpha }_0^3{\mathit{\mu }}_j\circ {\mathit{\mu }}_s\circ {\mathit{\mu }}_t\right.\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+{\alpha }_0^2\left({\alpha }_0+1\right)\mathbb{E}\left({\mathit{b}}_{ij}\circ {\mathit{b}}_{is}\circ {\mathit{\mu }}_t\right)-{\alpha }_0^3{\mathit{\mu }}_j\circ {\mathit{\mu }}_s\circ {\mathit{\mu }}_t\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+{\alpha }_0^2\left({\alpha }_0+1\right)\mathbb{E}\left({\mathit{\mu }}_j\circ {\mathit{b}}_{is}\circ {\mathit{b}}_{it}\right)-{\alpha }_0^3{\mathit{\mu }}_j\circ {\mathit{\mu }}_s\circ {\mathit{\mu }}_t\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+{\alpha }_0^2\left({\alpha }_0+1\right)\mathbb{E}\left({\mathit{b}}_{ij}\circ {\mathit{\mu }}_s\circ {\mathit{b}}_{it}\right)-{\alpha }_0^3{\mathit{\mu }}_j\circ {\mathit{\mu }}_s\circ {\mathit{\mu }}_t\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\left.+2\sum _{h=1}^k{\alpha }_h{\mathit{\varphi }}_{jh}\circ {\varphi }_{sh}\circ {\mathit{\varphi }}_{th}\right).\end{array}$$ (A.2)

The theorem follows from Equations (A.1) and (A.2). For noncategorical data, we let b_ij = y_ij and ϕ_jh is a scalar mean parameter for y_ij. Equations (A.1) and (A.2) still hold. □

B. Proof of Theorem 2

Proof. For notational simplicity, we suppress A_n in Q_n(Φ; A_n) and A in Q₀(Φ; A). Lemma 1 implies that ${\lim }_{n\to \infty }\mathrm{Pr}\left[\left|Q_n\left(\widehat{\mathbf{\Phi }}\right)-Q_0\left(\widehat{\mathbf{\Phi }}\right)\right|<ϵ/3\right]=1$ and ${\lim }_{n\to \infty }\mathrm{Pr}\left[\left|Q_n\left({\mathbf{\Phi }}_0\right)-Q_0\left({\mathbf{\Phi }}_0\right)\right|<ϵ/3\right]=1$ for $ϵ>0$. This result also implies that

$$\underset{n\to \infty }{\lim }\mathrm{Pr}\left[Q_0\left(\widehat{\mathbf{\Phi }}\right)<Q_n\left(\widehat{\mathbf{\Phi }}\right)+ϵ/3\right]=1.$$ (B.1)

$$\underset{n\to \infty }{\lim }\mathrm{Pr}\left[Q_n\left({\mathbf{\Phi }}_0\right)<Q_0\left({\mathbf{\Phi }}_0\right)+ϵ/3\right]=1.$$ (B.2)

On the other hand, $\widehat{\mathbf{\Phi }}$ minimizes Q_n(Φ), therefore

$$\underset{n\to \infty }{\lim }\mathrm{Pr}\left[Q_n\left(\widehat{\mathbf{\Phi }}\right)<Q_n\left({\mathbf{\Phi }}_0\right)+ϵ/3\right]=1.$$ (B.3)

Equations (B.1) and (B.3) imply that

$$\underset{n\to \infty }{\lim }\mathrm{Pr}\left[Q_0\left(\widehat{\mathbf{\Phi }}\right)<Q_n\left({\mathbf{\Phi }}_0\right)+2ϵ/3\right]=1.$$

Together with Equation (B.2), we get

$$\underset{n\to \infty }{\lim }\mathrm{Pr}\left[Q_0\left(\widehat{\mathbf{\Phi }}\right)<Q_0\left({\mathbf{\Phi }}_0\right)+ϵ\right]=1.$$

Therefore,

$$\underset{n\to \infty }{\lim }\mathrm{Pr}\left[0\le Q_0\left(\widehat{\mathbf{\Phi }}\right)<ϵ\right]=1$$ (B.4)

follows with Q₀(Φ₀) = 0 and $Q_0\left(\widehat{\mathbf{\Phi }}\right)\ge 0$. Next, we choose a neighborhood N, which contains Φ₀ in Θ. Due to the compactness of Θ, the neighborhood N^C is also compact. The continuity of Q₀(Φ) implies the existence of ${\inf }_{\mathbf{\Phi }\in N^C}{Q}_0\left(\mathbf{\Phi }\right)$ and that it is positive. Let $ϵ={\inf }_{\mathbf{\Phi }ϵN^C}{Q}_0\left(\mathbf{\Phi }\right)$, then we get

$$\underset{n\to \infty }{\lim }\mathrm{Pr}\left[0\le Q_0\left(\widehat{\mathbf{\Phi }}\right)<\underset{\mathbf{\Phi }ϵN^C}{\inf }{Q}_0\left(\mathbf{\Phi }\right)\right]=1.$$ (B.5)

Therefore, ${\lim }_{n\to \infty }\mathrm{Pr}\left(\widehat{\mathbf{\Phi }}\notin N^C\right)=1$, which suggests ${\lim }_{n\to \infty }\mathrm{Pr}\left(\widehat{\mathbf{\Phi }}\in N\right)=1$. Shrinking the neighborhood size of N, we get

$$\underset{n\to \infty }{\lim }\mathrm{Pr}\left(\widehat{\mathbf{\Phi }}={\mathbf{\Phi }}_0\right)=1.$$

□

C. Proof of Theorem 3

Proof. We approximate $f_n\left(\widehat{\mathbf{\Phi }}\right)$ using a first-order Taylor expansion

$$\begin{gathered} f_n\left(\widehat{\mathbf{\Phi }}\right)=f_n\left({\mathbf{\Phi }}_0\right)+G_n\left({\mathbf{\Phi }}_0\right)\left[\text{vec}\left(\widehat{\mathbf{\Phi }}\right)-\text{vec}\left({\mathbf{\Phi }}_0\right)\right] \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+O\left\{{\left[\text{vec}\left(\widehat{\mathbf{\Phi }}\right)-\text{vec}\left({\mathbf{\Phi }}_0\right)\right]}^2\right\} \end{gathered}$$

Ignoring the high-order term, we left multiply both sides by $\left[{\mathit{G}}_n{\left(\widehat{\mathbf{\Phi }}\right)}^T{A}_n\right]$. Then, we get

$$\begin{array}{l}\left[{\mathit{G}}_n{\left(\widehat{\mathbf{\Phi }}\right)}^T\right]{\mathit{A}}_n{\mathit{f}}_n\left(\widehat{\mathbf{\Phi }}\right)\approx {\left[{\mathit{G}}_n\left(\widehat{\mathbf{\Phi }}\right)\right]}^T{\mathit{A}}_n{\mathit{f}}_n\left({\mathbf{\Phi }}_0\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+{\left[{\mathit{G}}_n\left(\widehat{\mathbf{\Phi }}\right)\right]}^T{\mathit{A}}_n{\mathit{G}}_n\left({\mathbf{\Phi }}_0\right)\left[\text{vec}\left(\widehat{\mathbf{\Phi }}\right)-\text{vec}\left({\mathbf{\Phi }}_0\right)\right].\end{array}$$

The fact that estimator $\widehat{\mathbf{\Phi }}$ minimizes Q_n(Φ,A_n ) implies that the left-hand side equals zero. Therefore,

$$\begin{array}{l}{n}^{1/2}\left[\text{vec}\left(\widehat{\mathbf{\Phi }}\right)-\text{vec}\left({\mathbf{\Phi }}_0\right)\right]\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\approx -{\left\{{\left[{\mathit{G}}_n\left(\widehat{\mathbf{\Phi }}\right)\right]}^T{\mathit{A}}_n{\mathit{G}}_n\left({\mathbf{\Phi }}_0\right)\right\}}^{-1}{\left[{\mathit{G}}_n\left(\widehat{\mathbf{\Phi }}\right)\right]}^T{\mathit{A}}_n{n}^{1/2}{\mathit{f}}_n\left({\mathbf{\Phi }}_0\right).\end{array}$$

The theorem follows with $n^{1/2}{\mathit{f}}_n\left({\mathbf{\Phi }}_0\right)\stackrel{p}{\to }\mathcal{N}\left(\mathbf{0},\mathit{S}\right)$ and Assumptions 1 and 2. □

D. Derivatives of Moment Functions

D.1. Second Moment Matrix

The second moment matrix ${\mathit{F}}_{jt}^{\left(2\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)$ may be written as ${\mathit{b}}_{ij}\circ {\mathit{b}}_{it}-{\mathbf{\Phi }}_j\mathbb{E}\left({\mathit{x}}_i\circ {\mathit{x}}_i\right){\mathbf{\Phi }}_t^T$. The derivatives of ${\mathit{F}}_{jt}^{\left(2\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)$ with respect to Φ_j and Φ_t can be written as

$$\begin{array}{l}\frac{\partial \text{vec}\left[{\mathit{F}}_{jt}^{\left(2\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)\right]}{\partial \text{vec}\left({\mathbf{\Phi }}_j\right)}=-\frac{\partial \left\{\left[{\mathbf{\Phi }}_t\mathbb{E}\left({\mathit{x}}_i\circ {\mathit{x}}_i\right)\right]\otimes {\mathit{I}}_{d_j}\right\}\text{vec}\left({\mathbf{\Phi }}_j\right)}{\partial \text{vec}\left({\mathbf{\Phi }}_j\right)}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=-\left[{\mathbf{\Phi }}_t\mathbb{E}\left({\mathit{x}}_i\circ {\mathit{x}}_i\right)\right]\otimes {\mathit{I}}_{d_j},\end{array}$$

$$\begin{array}{l}\frac{\partial \text{vec}\left[{\mathit{F}}_{jt}^{\left(2\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)\right]}{\partial \text{vec}\left({\mathbf{\Phi }}_t\right)}=-\mathit{T}\frac{\partial \text{vec}\left[{\mathbf{\Phi }}_t\mathbb{E}\left({\mathit{x}}_i\circ {\mathit{x}}_i\right){\mathbf{\Phi }}_j^T\right]}{\partial \text{vec}\left({\mathbf{\Phi }}_t\right)}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em} \hspace{0.17em}\hspace{0.17em}=-\mathit{T}\frac{\partial \left\{\left[{\mathbf{\Phi }}_j\mathbb{E}\left({\mathit{x}}_i\circ {\mathit{x}}_i\right)\right]\otimes {\mathit{I}}_{d_t}\text{vec}\left({\mathbf{\Phi }}_t\right)\right\}}{\partial \text{vec}\left({\mathbf{\Phi }}_t\right)}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em} \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=-\mathit{T}\left\{\left[{\mathbf{\Phi }}_j\mathbb{E}\left({\mathit{x}}_i\circ {\mathit{x}}_i\right)\right]\otimes {\mathit{I}}_{d_t}\right\},\end{array}$$

where ⊗ indicates a Kronecker product and T is a d_tk × d_tk 0/1matrix that satisfies

$$\text{vec}\left[{\mathbf{\Phi }}_j\mathbb{E}\left({\mathit{x}}_i\circ {\mathit{x}}_i\right){\mathbf{\Phi }}_t^T\right]=\mathit{T}\text{vec}\left[{\mathbf{\Phi }}_t\mathbb{E}\left({\mathit{x}}_i\circ {\mathit{x}}_i\right){\mathbf{\Phi }}_j^T\right].$$

Therefore, $\mathbb{E}\left[\partial {\mathit{f}}^{\left(2\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)/\partial \mathbf{\Phi }\right]$ is a block matrix with blocks $-{\mathbf{\Phi }}_t^T\mathbb{E}\left({\mathit{x}}_i\circ {\mathit{x}}_i\right)\otimes {\mathit{I}}_{d_j}$ in columns corresponding to vec(Φ_t ) and rows corresponding to $\text{vec}\left[{\mathit{F}}_{jt}^{\left(2\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)\right]$.

D.2. Third Moment Tensor

We next consider the third moment tensor. Write ${\mathit{F}}_{jst}^{\left(3\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)$ as ${\mathit{b}}_{ij}\circ {\mathit{b}}_{ij}\circ {\mathit{b}}_{ij}-\mathbb{E}\left({\mathit{x}}_i\circ {\mathit{x}}_i\circ {\mathit{x}}_i\right){\times }_1{\mathbf{\Phi }}_j{\times }_2{\mathbf{\Phi }}_s{\times }_3{\mathbf{\Phi }}_t$. Then, only the second term involves Φ.

The derivatives of ${\mathit{F}}_{jst}^{\left(3\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)$ with respect to Φ_j can be written as

$$\begin{array}{l}\frac{\partial \text{vec}\left[{\mathit{F}}_{jst}^{\left(3\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)\right]}{\partial \text{vec}\left({\mathbf{\Phi }}_j\right)}=\frac{\partial \text{vec}\left\{{\left[\mathbb{E}\left({\mathit{x}}_i\circ {\mathit{x}}_i\circ {\mathit{x}}_i\right){\times }_1{\mathbf{\Phi }}_j{\times }_2{\mathbf{\Phi }}_s{\times }_3{\mathbf{\Phi }}_t\right]}_{\left(1\right)}\right\}}{\partial \text{vec}\left({\mathbf{\Phi }}_j\right)}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em} \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\frac{\partial \text{vec}\left\{\left[{\mathbf{\Phi }}_j\mathbb{E}{\left({\mathit{x}}_i\circ {\mathit{x}}_i\circ {\mathit{x}}_i\right)}_{\left(1\right)}{\left({\mathbf{\Phi }}_t\otimes {\mathbf{\Phi }}_s\right)}^T\right.\right\}}{\partial \text{vec}\left({\mathbf{\Phi }}_j\right)}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em} \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=-\left({\mathbf{\Phi }}_t\otimes {\mathbf{\Phi }}_s\right)\text{vec}{\left[\mathbb{E}{\left({\mathit{x}}_i\circ {\mathit{x}}_i\circ {\mathit{x}}_i\right)}_{\left(1\right)}\right]}^T\otimes {\mathit{I}}_{d_j},\end{array}$$

where subscript (1) indicates model-1 unfolding of a three-way tensor.

The derivatives of ${\mathit{F}}_{jst}^{\left(3\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)$ with respect to Φ_s and Φ_t can be calculated accordingly by introducing binary transformation matrices T ₍₂₎₍₁₎ and T ₍₃₎₍₁₎, both with size d_jd_sd_t × d_jd_sd_t, that satisfy

$$\begin{array}{l}\text{vec}\left\{{\left[{\mathit{F}}_{jst}^{\left(3\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)\right]}_{\left(1\right)}\right\}={\mathit{T}}_{\left(2\right)\left(1\right)}\text{vec}\left\{{\left[{\mathit{F}}_{jst}^{\left(3\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)\right]}_{\left(2\right)}\right\}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em} \hspace{0.17em}\hspace{0.17em}={\mathit{T}}_{\left(3\right)\left(1\right)}\text{vec}\left\{{\left[{\mathit{F}}_{jst}^{\left(3\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)\right]}_{\left(3\right)}\right\}.\\ \end{array}$$

Then,

$$\begin{array}{l}\frac{\partial \text{vec}\left[{\mathit{F}}_{jst}^{\left(3\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)\right]}{\partial \text{vec}\left({\mathbf{\Phi }}_s\right)}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=-{\mathit{T}}_{\left(2\right)\left(1\right)}\frac{\partial \text{vec}\left\{{\left[\mathbb{E}\left({\mathit{x}}_i\circ {\mathit{x}}_i\circ {\mathit{x}}_i\right){\times }_1{\mathbf{\Phi }}_j{\times }_2{\mathbf{\Phi }}_s{\times }_3{\mathbf{\Phi }}_t\right]}_{\left(2\right)}\right\}}{\partial \text{vec}\left({\mathbf{\Phi }}_s\right)}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=-{\mathit{T}}_{\left(2\right)\left(1\right)}\frac{\partial \text{vec}\left\{\left[{\mathbf{\Phi }}_s\mathbb{E}{\left({\mathit{x}}_i\circ {\mathit{x}}_i\circ {\mathit{x}}_i\right)}_{\left(2\right)}{\left({\mathbf{\Phi }}_t\otimes {\mathbf{\Phi }}_j\right)}^T\right.\right\}}{\partial \text{vec}\left({\mathbf{\Phi }}_s\right)}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=-{\mathit{T}}_{\left(2\right)\left(1\right)}\left\{\left({\mathbf{\Phi }}_t\otimes {\mathbf{\Phi }}_j\right){\left[\mathbb{E}{\left({\mathit{x}}_i\circ {\mathit{x}}_i\circ {\mathit{x}}_i\right)}_{\left(2\right)}\right]}^T\otimes {\mathit{I}}_{d_s}\right\},\end{array}$$

$$\begin{array}{l}\frac{\partial \text{vec}\left[{\mathit{F}}_{jst}^{\left(3\right)}\left({\mathit{y}}_i,\mathbf{\Phi }\right)\right]}{\partial \text{vec}\left({\mathbf{\Phi }}_t\right)}\\ \hspace{0.33em}\hspace{0.17em}=-{\mathit{T}}_{\left(3\right)\left(1\right)}\frac{\partial \text{vec}\left\{{\left[\mathbb{E}\left({\mathit{x}}_i\circ {\mathit{x}}_i\circ {\mathit{x}}_i\right){\times }_1{\mathbf{\Phi }}_j{\times }_2{\mathbf{\Phi }}_s{\times }_3{\mathbf{\Phi }}_t\right]}_{\left(3\right)}\right\}}{\partial \text{vec}\left({\mathbf{\Phi }}_t\right)}\\ \hspace{0.33em}\hspace{0.17em}=-{\mathit{T}}_{\left(3\right)\left(1\right)}\frac{\partial \text{vec}\left\{\left[{\mathbf{\Phi }}_t\mathbb{E}{\left({\mathit{x}}_i\circ {\mathit{x}}_i\circ {\mathit{x}}_i\right)}_{\left(3\right)}{\left({\mathbf{\Phi }}_s\otimes {\mathbf{\Phi }}_j\right)}^T\right.\right\}}{\partial \text{vec}\left({\mathbf{\Phi }}_t\right)}\\ \hspace{0.33em}\hspace{0.17em}=-{\mathit{T}}_{\left(3\right)\left(1\right)}\left\{\left({\mathbf{\Phi }}_s\otimes {\mathbf{\Phi }}_j\right){\left[\mathbb{E}{\left({\mathit{x}}_i\circ {\mathit{x}}_i\circ {\mathit{x}}_i\right)}_{\left(3\right)}\right]}^T\otimes {\mathit{I}}_{d_t}\right\}.\end{array}$$

Conditions (1), (3), and (4) in Assumption 2 in the main text follow after we calculate the derivatives of the moment functions.

E. Derivation of Newton-Raphson Update

We denote

$${\mathit{E}}_{n,jt}^{\left(2\right)}={\mathit{F}}_{n,jt}^{\left(2\right)}\left(\mathbf{\Phi }\right)+{\mathbf{\Phi }}_j{\text{Λ}}^{\left(2\right)}{\mathbf{\Phi }}_t^T,$$

$${\mathit{E}}_{n,jst}^{\left(3\right)}={\mathit{F}}_{n,jst}^{\left(3\right)}\left(\mathbf{\Phi }\right)+{\text{Λ}}^{\left(3\right)}{\times }_1{\mathbf{\Phi }}_j{\times }_2{\mathbf{\Phi }}_s{\times }_3{\mathbf{\Phi }}_t.$$

With the identity matrix, the two objective functions can be written as

$$Q_n^{\left(2\right)}\left(\mathbf{\Phi }\right)=\sum _{j=1}^{p-1}\sum _{t=j+1}^p{‖{\mathit{E}}_{n,jt}^{\left(2\right)}-{\mathbf{\Phi }}_j{\text{Λ}}^{\left(2\right)}{\mathbf{\Phi }}_t^T‖}_F^2,$$

$$\begin{array}{l}{Q}_n^{\left(3\right)}\left(\mathbf{\Phi }\right)=\sum _{j=1}^{p-1}\sum _{t=j+1}^p{‖{\mathit{E}}_{n,jt}^{\left(2\right)}-{\mathbf{\Phi }}_j{\text{Λ}}^{\left(2\right)}{\mathbf{\Phi }}_t^T‖}_F^2\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+\sum _{j=1}^{p-2}\sum _{s=j+1}^{p-1}\sum _{t=s+1}^p{‖{\mathit{E}}_{n,jst}^{\left(3\right)}-{\text{Λ}}^{\left(3\right)}{\times }_1{\mathbf{\Phi }}_j{\times }_2{\mathbf{\Phi }}_s{\times }_3{\mathbf{\Phi }}_t‖}_F^2.\end{array}$$

Here, we suppress the weight matrix in the objective functions. We first consider $Q_n^{\left(2\right)}\left(\mathbf{\Phi }\right)$. The terms that involve ϕ_jh are

$$\sum _{t=1,t\ne j}^p\left[-2{\lambda }_h^{\left(2\right)}{\left({\overline{\mathit{E}}}_{n,jt}^{\left(2\right)}{\mathit{\varphi }}_{th}\right)}^T{\mathit{\varphi }}_{jh}+{\left({\lambda }_h^{\left(2\right)}\right)}^2\left({\mathit{\varphi }}_{th}^T{\mathit{\varphi }}_{th}\right){\mathit{\varphi }}_{jh}^T{\mathit{\varphi }}_{jh}\right],$$

where ${\overline{\mathit{E}}}_{n,jt}^{\left(2\right)}={\mathit{E}}_{n,jt}^{\left(2\right)}-{\sum }_{h^{\prime }\ne h}{\lambda }_{h^{\prime }}^{\left(2\right)}{\mathit{\varphi }}_{j{h}^{\prime }}\circ {\varphi }_{t{h}^{\prime }}$ and ${\lambda }_h^{\left(2\right)}$ is the hth diagonal element of ${\text{Λ}}^{\left(2\right)}$. Here, we use the fact that ${‖{\mathit{E}}_{n,jt}^{\left(2\right)}-{\mathbf{\Phi }}_j{\text{Λ}}^{\left(2\right)}{\mathbf{\Phi }}_t^T‖}_F^2={‖{\mathit{E}}_{n,t\hspace{0.17em}j}^{\left(2\right)}-{\mathbf{\Phi }}_t{\text{Λ}}^{\left(2\right)}{\mathbf{\Phi }}_j^T‖}_F^2$. By letting

$${\mathit{\xi }}^{\left(2\right)}=-2{\lambda }_h^{\left(2\right)}\sum _{t=1,t\ne j}^p\left({\overline{\mathit{E}}}_{n,jt}^{\left(2\right)}{\mathit{\varphi }}_{th}\right),$$

$${\gamma }^{\left(2\right)}={\left({\lambda }_h^{\left(2\right)}\right)}^2\sum _{t=1,t\ne j}^p{\mathit{\varphi }}_{th}^T{\mathit{\varphi }}_{th},$$

the gradient $\nabla Q_n^{\left(2\right)}\left({\mathit{\varphi }}_{jh}\right)$ and Hessian ${\nabla }^2{Q}_n^{\left(2\right)}\left({\mathit{\varphi }}_{jh}\right)$ can be written as

$$\nabla Q_n^{\left(2\right)}\left({\mathit{\varphi }}_{jh}\right)={\mathit{\xi }}^{\left(2\right)}+2{\gamma }^{\left(2\right)}{\mathit{\varphi }}_{jh},$$

$${\nabla }^2{Q}_n^{\left(2\right)}\left({\mathit{\varphi }}_{jh}\right)=2{\gamma }^{\left(2\right)}\mathit{I}.$$

The update rule in (11) can be derived accordingly.

Then, we consider $Q_n^{\left(3\right)}\left(\mathbf{\Phi }\right)$. The terms that involve ϕ_jh are

$$\begin{array}{l}\sum _{t=1,t\ne j}^p\left[-2{\lambda }_h^{\left(2\right)}{\left({\overline{\mathit{E}}}_{n,jt}^{\left(2\right)}{\mathit{\varphi }}_{th}\right)}^T{\mathit{\varphi }}_{jh}+{\left({\lambda }_h^{\left(2\right)}\right)}^2\left({\mathit{\varphi }}_{th}^T{\mathit{\varphi }}_{th}\right){\mathit{\varphi }}_{jh}^T{\mathit{\varphi }}_{jh}\right]\\ \hspace{1em}+\sum _{s=1,s\ne j}^p\sum _{t=1,t\ne s,t\ne j}^p\left[-2〈{\overline{\mathit{E}}}_{n,jst}^{\left(3\right)},{\lambda }_h^{\left(3\right)}{\mathit{\varphi }}_{jh}\circ {\varphi }_{sh}\circ {\mathit{\varphi }}_{th}〉\right.\\ \hspace{1em}+\left.{‖{\lambda }_h^{\left(3\right)}{\mathit{\varphi }}_{jh}\circ {\varphi }_{sh}\circ {\mathit{\varphi }}_{th}‖}_F^2\right],\end{array}$$

where ${\overline{\mathit{E}}}_{n,jt}^{\left(2\right)}={\mathit{E}}_{n,jt}^{\left(2\right)}-{\sum }_{h^{\prime }\ne h}{\lambda }_{h^{\prime }}^{\left(2\right)}{\mathit{\varphi }}_{j{h}^{\prime }}\circ {\varphi }_{t{h}^{\prime }}$ and ${\overline{\mathit{E}}}_{n,jst}^{\left(3\right)}={\mathit{E}}_{n,jst}^{\left(3\right)}-{\sum }_{h^{\prime }\ne h}{\lambda }_{h^{\prime }}^{\left(3\right)}{\mathit{\varphi }}_{j{h}^{\prime }}\circ {\varphi }_{s{h}^{\prime }}\circ {\varphi }_{t{h}^{\prime }}$. Again we use the symmetric property of ${‖{\mathit{E}}_{n,jt}^{\left(2\right)}-{\mathbf{\Phi }}_j{\text{Λ}}^{\left(2\right)}{\mathbf{\Phi }}_t^T‖}_F^2$ and ${‖{\mathit{E}}_{n,jst}^{\left(3\right)}-\text{Λ}{\times }_1{\mathbf{\Phi }}_j{\times }_2{\mathbf{\Phi }}_s{\times }_3{\mathbf{\Phi }}_t‖}_F^2$. By organizing the terms, we get

$$\begin{array}{l}\sum _{t=1,t\ne j}^p\left[-2{\lambda }_h^{\left(2\right)}{\left({\overline{\mathit{E}}}_{n,jt}^{\left(2\right)}{\mathit{\varphi }}_{th}\right)}^T{\mathit{\varphi }}_{jh}+{\left({\lambda }_h^{\left(2\right)}\right)}^2\left({\mathit{\varphi }}_{th}^T{\mathit{\varphi }}_{th}\right){\mathit{\varphi }}_{jh}^T{\mathit{\varphi }}_{jh}\right]\\ \hspace{1em}+\sum _{s=1,s\ne j}^p\sum _{t=1,t\ne s,t\ne j}^p\left[-2{\lambda }_h^{\left(3\right)}{\left({\overline{\mathit{E}}}_{n,jst}^{\left(3\right)}{\times }_2{\varphi }_{sh}{\times }_3{\mathit{\varphi }}_{th}\right)}^T{\mathit{\varphi }}_{jh}\right.\\ \hspace{1em}+\left.{\left({\lambda }_h^{\left(3\right)}\right)}^2\left({\varphi }_{sh}^T{\varphi }_{sh}\right)\left({\mathit{\varphi }}_{th}^T{\mathit{\varphi }}_{th}\right)\left({\mathit{\varphi }}_{jh}^T{\mathit{\varphi }}_{jh}\right)\right],\end{array}$$

By letting

$$\begin{array}{l}{\mathit{\xi }}^{\left(3\right)}=-2{\lambda }_h^{\left(2\right)}\sum _{t=1,t\ne j}^p\left({\overline{\mathit{E}}}_{n,jt}^{\left(2\right)}{\mathit{\varphi }}_{th}\right)-2{\lambda }_h^{\left(3\right)}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\times \sum _{s=1,s\ne j}^p\left[\sum _{t=1,t\ne s,t\ne j}^p\left({\overline{\mathit{E}}}_{n,jst}^{\left(3\right)}{\times }_2{\varphi }_{sh}{\times }_3{\mathit{\varphi }}_{th}\right)\right],\end{array}$$

$$\begin{array}{l}{\gamma }^{\left(3\right)}={\left({\lambda }_h^{\left(2\right)}\right)}^2\sum _{t=1,t\ne j}^p{\mathit{\varphi }}_{th}^T{\mathit{\varphi }}_{th}+{\left({\lambda }_h^{\left(3\right)}\right)}^2\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\times \sum _{s=1,s\ne j}^p\left[\sum _{t=1,t\ne s,t\ne j}^p\left({\varphi }_{sh}^T{\mathit{\varphi }}_{th}\right)\left({\mathit{\varphi }}_{th}^T{\mathit{\varphi }}_{th}\right)\right],\end{array}$$

the gradient $\nabla Q_n^{\left(3\right)}\left({\mathit{\varphi }}_{jh}\right)$ and Hessian ${\nabla }^2{Q}_n^{\left(3\right)}\left({\mathit{\varphi }}_{jh}\right)$ can be written as

$$\nabla Q_n^{\left(3\right)}\left({\mathit{\varphi }}_{jh}\right)={\mathit{\xi }}^{\left(3\right)}+2{\gamma }^{\left(3\right)}{\mathit{\varphi }}_{jh},$$

$${\nabla }^2{Q}_n^{\left(3\right)}\left({\mathit{\varphi }}_{jh}\right)=2{\gamma }^{\left(3\right)}\mathit{I}.$$

The update rule in (12) follows directly.