Principal regression for high dimensional covariance matrices

Abstract

This manuscript presents an approach to perform generalized linear regression with multiple high dimensional covariance matrices as the outcome. In many areas of study, such as resting-state functional magnetic resonance imaging (fMRI) studies, this type of regression can be utilized to characterize variation in the covariance matrices across units. Model parameters are estimated by maximizing a likelihood formulation of a generalized linear model, conditioning on a well-conditioned linear shrinkage estimator for multiple covariance matrices, where the shrinkage coefficients are proposed to be shared across matrices. Theoretical studies demonstrate that the proposed covariance matrix estimator is optimal achieving the uniformly minimum quadratic loss asymptotically among all linear combinations of the identity matrix and the sample covariance matrix. Under certain regularity conditions, the proposed estimator of the model parameters is consistent. The superior performance of the proposed approach over existing methods is illustrated through simulation studies. Implemented to a resting-state fMRI study acquired from the Alzheimer’s Disease Neuroimaging Initiative, the proposed approach identified a brain network within which functional connectivity is significantly associated with Apolipoprotein E ε4, a strong genetic marker for Alzheimer’s disease.

1. Introduction

In this manuscript, we study a regression problem with covariance matrices as the outcome under a high dimensional setting. Suppose $y_{it}\in {ℝ}^p$ is a p-dimensional random vector, which is the tth acquisition from subject i, for t = 1, …, T_i and i = 1, …, n, where T_i is the number of observations of subject i and n is the number of subjects. Let T_max = max_i T_i. The term “high dimensionality” refers to the scenario when T_max ≪ p and p increases to infinity. The data, y_it, are assumed to follow a normal distribution with covariance matrix Σ_i. Here, without loss of generality, it is assumed that the distribution mean is zero as the study interest focuses on the covariance matrices. Let $x_i\in {ℝ}^q$ denote the q-dimensional covariates of interest acquired from subject i. For the covariance matrices, we assume the following regression model. For i = 1, …, n, the data heteroscedasticity satisfies the following generalized linear regression model with a logarithmic link function,

$$\text{log}\left({\mathit{\gamma }}^{\top }{\Sigma }_i\mathit{\gamma }\right)=x_i^{\top }\beta ,$$ (1.1)

where $\mathit{\gamma }\in {ℝ}^p$ is a linear projection and $\beta \in {ℝ}^q$ is the model coefficient. In x_i, the first element is set to one to include the intercept term. Using a logarithmic link function, it is guaranteed that Σ_i’s are positive semi-definite. The goal is to estimate γ and β using the observed data ${\left\{\left(y_{i1},\dots ,y_{i{T}_i}\right),x_i\right\}}_{i=1}^n$. In Model (1.1), γ is an unknown linear projection to be estimated such that the characteristic of the covariance matrices can be best captured by the covariates of interest.

One application of such a regression problem is to analyze covariate associated variations in brain coactivation in a functional magnetic resonance imaging (fMRI) study, where covariance/correlation matrices of the fMRI signals are generally utilized to reveal the coactivation patterns. Characterizing these patterns with population/individual covariates is of great interest in neuroimaging studies [34, 41]. Another example is the study of financial equities data. Considering a pool of stock values, covariance matrices over a period of time capture the comovement or synchronicity of the stocks. Firm and market-level information, such as industry type, firm’s cash flow, stock size, and book-to-market ratio, plays an essential role in determining the synchronicity. Quantifying such association is an important topic in financial theory [43].

Assuming T_min = min_i T_i > p and p is fixed, Zhao et al. [41] first studied Model (1.1) and proposed to estimate γ and β through a likelihood-based approach minimizing the negative log-likelihood function in the projection space. One sufficient condition to solve the likelihood-based criterion is that the sample covariance matrices are positive definite. Thus, the likelihood estimator is ill-posed when T_max < p as the sample covariance matrices are rank-deficient. Additionally, it has been shown that when p increases, the sample covariance matrix performs poorly and can lead to invalid conclusions. For example, the largest eigenvalue of the sample covariance matrix is not a consistent estimator, and the eigenvectors can be nearly orthogonal to the truth [22]. To circumvent difficulties raised by the high dimensionality, one solution is to impose structural assumptions, such as bandable covariance matrices, sparse covariance matrices, spiked covariance matrices, covariances with a tensor product structure, and latent graphical models [see a review of 6, and references therein]. Based on structural assumptions, many regularization-based methods have been developed. However, most of these methods produce covariance estimates that may not always be positive definite (numerically), and this creates subsequent numerical convergence issues when the quadratic product with Σ_i is negative in (1.1). Moreover, most regularization methods can be computationally expensive on finding the solution and may require searching over different regularization parameters, not to mention the computational costs increase multiplicatively when computing over multiple covariance matrices. Further research is also needed to evaluate what structural assumptions are most appropriate for fMRI data. Another class of high-dimensional covariance matrix estimator is the shrinkage estimator. Daniels and Kass [11] considered two shrinkage estimators of the covariance matrix, a correlation shrinkage and a rotation shrinkage, offering a compromise between completely unstructured and structured estimators to improve the robustness. Ledoit and Wolf [24] introduced a well-conditioned estimator of the covariance matrix, which is an optimal linear combination of the identity matrix and the sample covariance matrix under squared error loss. This estimator is guaranteed to be positive definite and is easy to compute based on a simple and explicit formula. These advantages make it desirable for formulating the proposed estimator. Instead of a linear combination, Ledoit and Wolf [25] extended this work to nonlinear transformations of the sample eigenvalues and presented a way of finding the transformation that is asymptotically equivalent to the oracle linear combination. Based on Tyler’s robust M-estimator [37] and the linear shrinkage estimator [24], Chen et al. [8] and Pascal et al. [28], in parallel, introduced robust estimators of covariance matrices for elliptical distributed samples.

To model multiple covariance matrices, procedures include regression-type approaches [1, 9, 21, 14, 43]; (common) principal component analysis related methods [13, 5, 20, 15]; and methods based on other types of matrix decomposition, such as the Cholesky decomposition [30]. Among these, Fox and Dunson [14] introduced a scalable nonparametric covariance regression model applying low-rank approximation. Franks and Hoff [15] generalized a Bayesian hierarchical model studying the heterogeneity in the covariance matrices to high dimensional settings. Assuming that the ideal covariance structure exists in the eigenspace of the data covariance matrix, Chen et al. [7] introduced a regression-based approach to remove the scanner effects in covariance achieving the goal of harmonization. Compared to the above-mentioned approaches, Model (1.1) offers higher flexibility in modeling the relationship with the covariates. For example, x can be either continuous or categorical, and one can easily include interactions and/or polynomials of the covariates.

In the high dimensional setting considered in this study, γ and β, as well as n covariance matrices, will be estimated under Model (1.1). It is well known that the eigenvalues of the sample covariance matrix are more dispersed than the truth [24]. The class of linear combinations of the identity and sample covariance matrix corrects this dispersion issue by shrinking towards the identity matrix. The choice of the identity matrix can also be interpreted as a prior without strong structural assumptions or prior knowledge. Interestingly, it will be shown that estimating each covariance matrix separately, such as using the shrinkage estimator proposed in Ledoit and Wolf [24], leads to suboptimal estimation accuracy for γ, β, and Σ_i’s. Thus, we propose a linear shrinkage estimator of all the covariance matrices jointly, of which the shrinkage coefficients are shared across matrices. In addition, it is shown that the proposed shrinkage estimator leads to a consistent estimator of model coefficients. We first replace the sample covariance formulation with the proposed shrinkage estimator, and then estimate (γ, β) through maximizing a plug-in likelihood evaluated at the shrinkage estimator. In fMRI studies, shrinkage is also a popular technique to improve the reliability of subject-level functional connectivity captured by the covariance matrix. In the technique, when estimating individual covariance matrix, population-level information is borrowed as prior knowledge [38, 31, 27, 32, 29].

The framework proposed in this manuscript has three major contributions.

1. This paper first studies a joint shrinkage estimator for multiple high dimensional covariance matrices, generalizing the linear shrinkage estimator for a single covariance matrix [24]. We show that the latter approach is suboptimal compared to the proposed joint covariance shrinkage estimator, where the shrinkage coefficients are shared across multiple matrices. Within this class of shrinkage estimators, we believe that this is among the first attempts to analyze the variations of a large number of covariance matrices associated with covariates in a regression setting under certain model assumptions.

2. The proposed shrinkage estimator of the covariance matrices is well conditioned and has uniformly minimum quadratic risk asymptotically among all linear combinations (Theorem 3.3).

3. Under certain regularity conditions, the proposed approach achieves consistent estimators of the parameters in Model (1.1) (Proposition 3.1).

The rest of the paper is organized as the following. Section 2 introduces the proposed shrinkage estimator of the covariance matrices and the pseudo-likelihood based method of estimating γ and β. Section 3 studies the asymptotic properties. In Section 4, the superior performance of the proposed approach over existing methods is demonstrated through simulation studies. Section 5 articulates an application to a resting-state fMRI data set acquired from the Alzheimer’s Disease Neuroimaging Initiative (ADNI). Section 6 concludes this paper with discussions. Technical proofs are collected in the appendix.

2. Methods

Considering the regression model (1.1), it is proposed to estimate the parameters by solving the following optimization problem.

$$\begin{gathered} \underset{(\beta ,\mathit{\gamma })}{\text{minimize}}\ell (\beta ,\mathit{\gamma })=\frac{1}{2}\sum _{i=1}^n{T}_i\left\{x_i^{\top }\beta +{\mathit{\gamma }}^{\top }{\widehat{\Sigma }}_i\mathit{\gamma }\cdot \text{exp}\left(-x_i^{\top }\beta \right)\right\}, \\ \text{such that \hspace{1em}}{\mathit{\gamma }}^{\top }H\mathit{\gamma }=1, \end{gathered}$$ (2.1)

where ${\widehat{\Sigma }}_i$ is an estimator of the covariance matrix Σ_i to be discussed later, which is positive definite, for i = 1, …, n; and H is a positive definite matrix in ${ℝ}^{p\times p}$, which is set to be the average of ${\widehat{\Sigma }}_i$’s, that is $H={\sum }_{i=1}^n{T}_i{\widehat{\Sigma }}_i/{\sum }_{i=1}^n{T}_i$. It is essential to impose a constraint on γ, otherwise the objective function of (2.1) is minimized at γ = 0 with fixed β. When ${\widehat{\Sigma }}_i=S_i={\sum }_{t=1}^{T_i}{y}_{it}{y}_{it}^{\top }/T_i$ (i.e., the sample covariance matrix), which is the proposal in Zhao et al. [41], it is equivalent to minimize the negative log-likelihood function of {γ^⊤ y_it}_i,t assuming the data are normally distributed. However, when T_max = max_i T_i < p, problem (2.1) is ill-posed as S_i’s are rank-deficient. Thus, the goal of this manuscript is to propose a well-conditioned estimator of Σ_i that yields optimal properties. To achieve this, a covariate-dependent linear shrinkage estimator, denoted as ${\Sigma }_i^{*}$, is proposed, which yields the minimum expected squared loss under regression model (1.1), where the expectation is taken over the sample covariance matrix S_i.

$$\begin{gathered} \underset{(\mu ,\rho )}{\text{minimize}}\frac{1}{n}\sum _{i=1}^n\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{\Sigma }_i^{*}\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2, \\ \text{such that \hspace{1em}}{\Sigma }_i^{*}=\rho \mu I+(1-\rho )S_i,\text{\hspace{1em} for }i=1,\dots ,n. \end{gathered}$$ (2.2)

The following theorem gives the solution to (2.2).

Theorem 2.1. For given (γ, β), the solution to optimization problem (2.2) is

$${\Sigma }_i^{*}=\frac{{\psi }^2}{{\delta }^2}\mu I+\frac{{\varphi }^2}{{\delta }^2}{S}_i,\text{\hspace{1em} for }i=1,\dots ,n,$$ (2.3)

and the minimum value is

$$\frac{1}{n}\sum _{i=1}^n\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{\Sigma }_i^{*}\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2=\frac{{\varphi }^2{\psi }^2}{{\delta }^2},$$ (2.4)

where

$$\mu =\frac{1}{n\left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)}\sum _{i=1}^n\text{exp}\left(x_i^{\top }\beta \right),{\varphi }^2=\frac{1}{n}\sum _{i=1}^n{\varphi }_i^2,{\psi }^2=\frac{1}{n}\sum _{i=1}^n{\psi }_i^2,{\delta }^2=\frac{1}{n}\sum _{i=1}^n{\delta }_i^2,{\varphi }_i^2={\left\{\mu \left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2,{\psi }_i^2=\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2,{\delta }_i^2=\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }-\mu \left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)\right\}}^2;$$

and Lemma 2.1 shows that ψ²/δ² + ϕ²/δ² = 1.

Lemma 2.1. For ∀ i ∈ {1, …, n}, ${\delta }_i^2={\varphi }_i^2+{\psi }_i^2$, and thus δ² = ϕ² + ψ².

According to Theorem 2.1, parameters ${\varphi }_i^2$, ${\psi }_i^2$ and ${\delta }_i^2$ are expected values as the objective is to minimize the expected squared loss. Thus, one cannot replace ${\widehat{\Sigma }}_i$ with ${\sum }_i^{*}$ in (2.1) and solve for solution using the data. For implementation in practice, the following sample counterparts are used to compute (2.3) and thus ${\widehat{\Sigma }}_i$ in (2.1). Let

$${\widehat{\delta }}_i^2={\left\{{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }-\mu \left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)\right\}}^2,{\widehat{\psi }}_i^2=\frac{1}{T_i}{\left\{{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2,{\widehat{\varphi }}_i^2={\widehat{\delta }}_i^2-{\widehat{\psi }}_i^2,{\widehat{\delta }}^2=\frac{1}{n}\sum _{i=1}^n{\widehat{\delta }}_i^2,{\widehat{\psi }}^2=\frac{1}{n}\sum _{i=1}^n\text{min}\left({\widehat{\psi }}_i^2,{\widehat{\delta }}_i^2\right),{\widehat{\varphi }}^2=\frac{1}{n}\sum _{i=1}^n{\widehat{\varphi }}_i^2,$$

and

$$S_i^{*}=\frac{{\widehat{\psi }}^2}{{\widehat{\delta }}^2}\mu I+\frac{{\widehat{\varphi }}^2}{{\widehat{\delta }}^2}{S}_i,\text{\hspace{1em} for }i=1,\dots ,n.$$ (2.5)

In Section 3, we show that $S_i^{*}$ is a consistent estimator of ${\sum }_i^{*}$ and is uniformly optimal asymptotically among all the linear combinations of the sample covariance matrices and the identity matrix regarding the quadratic risk. The objective function ℓ(β, γ) is an approximation of the negative log-likelihood function if replacing ${\widehat{\Sigma }}_i$ with the proposed shrinkage estimator $S_i^{*}$. Thus, optimizing (2.1) can be considered as a pseudo-likelihood approach under the normality assumption.

The proof of Theorem 2.1 and Lemma 2.1 is presented in Appendix Section A.1. Formulation (2.2) introduces a shrinkage estimator of the covariance matrix, where the shrinkage is shared across subjects and is optimal under the squared error loss. For each subject, ${\sum }_i^{*}$ is a linear combination of the sample covariance matrix S_i and the identity matrix. The weighting parameters, ρ and μ, are population level parameters that are shared across subjects. This is equivalent to imposing a linear shrinkage on the sample eigenvalues. Assuming γ is a common eigenvector of all the covariance matrices, μ is the average eigenvalue corresponding to γ. The level of shrinkage is determined by the leverage between the accuracy of S_i’s and the variation in the eigenvalues. If S_i’s are accurate or the errors are small relative to the variation in the eigenvalues, less shrinkage will be imposed; otherwise, if S_i’s are inaccurate and the errors are comparable or even higher than the eigenvalue variability, the sample covariance matrices will be shrank more.

Algorithm 1 summarizes the optimization procedure. As problem (2.1) is nonconvex, a series of random initializations of (γ, β) is considered and the one that achieves the minimum value of the objective function is the estimate. The initial values of γ can be set as the eigenvectors of the average sample covariance matrices, $\overline{S}={\sum }_{i=1}^n{T}_i{S}_i/{\sum }_{i=1}^n{T}_i$; and the initial values of β is the corresponding solution to (2.1) by replacing ${\widehat{\Sigma }}_i$ with a well-conditioned estimator, such as the estimator proposed in Ledoit and Wolf [24]. When $p<{\sum }_{i=1}^n{T}_i$, $\overline{S}$ is of full rank, and the sample eigenvectors are consistent estimators assuming all the covariance matrices have the same eigendecomposition. Step 3 in the algorithm updates the covariance matrix estimators with a global shrinkage parameter. In Section 4, through simulation studies, we show that it improves the performance in estimating the covariance matrices and β with lower bias and higher stability. The details of updating γ and β in Step 4 can be found in Algorithm 1 in Zhao et al. [41].

For higher-order components, one can first remove the identified components and use the new data to estimate the next with an additional orthogonality constraint, that is, the new component is orthogonal to the identified ones. Different from Algorithm 2 in Zhao et al. [41], there is no need to include a rank-completion step as $S_i^{*}$ is introduced to render the rank-deficiency issue. To determine the number of components, the metric of average deviation from diagonality is adopted [41]. Let ${\Gamma }^{(k)}\in {ℝ}^{p\times k}$ denote the first k estimated components, the average deviation from diagonality is defined as

$$\text{DfD}\left({\Gamma }^{(k)}\right)=\prod _{i=1}^n{\left(\frac{\text{det}\left\{\text{diag}\left({\Gamma }^{(k)\top }{S}_i^{*}{\Gamma }^{(k)}\right)\right\}}{\text{det}\left({\Gamma }^{(k)\top }{S}_i^{*}{\Gamma }^{(k)}\right)}\right)}^{T_i/{\sum }_{i}T},$$ (2.6)

where diag(A) is a diagonal matrix of the diagonal elements in a square matrix A, and det(A) is the determinant of A. If Γ^(k) is a common diagonalization of $S_i^{*}$’s, that is, ${\Gamma }^{(k)\top }{S}_i^{*}{\Gamma }^{(k)}$ is a diagonal matrix, for ∀ i = 1, …, n, then DfD(Γ^(k)) = 1. In practice, k can be chosen before DfD increases far away from one or before a sudden jump occurs.

3. Asymptotic Properties

In this section, we study the asymptotic properties of the proposed estimators. For i = 1, …, n, it is assumed that Σ_i has the eigendecomposition of ${\Sigma }_i={\Pi }_i{\Lambda }_i{\Pi }_i^{\top }$, where Λ_i = diag{λ_i1, …, λ_ip} is a diagonal matrix and Π_i = (π_i1, …, π_ip) is an orthonormal rotation matrix; {λ_i1, …, λ_ip} are the eigenvalues and the columns of Π_i are the corresponding eigenvectors. Let Z_i = Y_iΠ_i, where $Y_i={\left(y_{i1},\dots ,y_{i{T}_i}\right)}^{\top }\in {ℝ}^{T_i\times p}$ is the data matrix of subject i. Under the normality assumption, the columns of Z_i = (z_itj)_t,j are uncorrelated, and the rows, $z_{it}=\left(z_{i1},\dots ,z_{ip}\right)\in {ℝ}^p$ for t = 1, …, T_i, are normally distributed with mean zero and covariance matrix Λ_i. The following assumptions are imposed.

Assumption A1 There exists a constant C₁ independent of T_max such that p/T_max ≤ C₁, where T_max = max_i T_i.

Assumption A2 Let $N={\sum }_{i=1}^n{T}_i$, p/N → ∞ as n, T_min → ∞, where T_min = min_i T_i.

Assumption A3 There exists a constant C₂ independent of T_min and T_max such that ${\sum }_{j=1}^p\mathbb{E}\left(z_{i1j}^8\right)/p\le C_2$, for ∀ i ∈ {1, …, n}.

Assumption A4 Let $\mathcal{Q}$ denote the set of all the quadruples that are made of four distinct integers between 1 and p, for ∀ i ∈ {1, …, n},

$$\underset{T_i\to \infty }{\text{lim}}\frac{p^2}{T_i^2}\frac{{\sum }_{(j,k,l,m)\in \mathcal{Q}}{\left\{\text{Cov}\left(z_{i1j}{z}_{i1k},z_{i1l}{z}_{i1m}\right)\right\}}^2}{\left|\mathcal{Q}\right|}=0,$$ (3.1)

where $\left|\mathcal{Q}\right|$ is the cardinality of set $\mathcal{Q}$.

Assumption A5 All the covariance matrices share the same set of eigenvectors, i.e., Π_i = Π, for i = 1, …, n. For each Σ_i, there exists (at least) a column, indexed by j_i, such that γ = π_iji and Model (1.1) is satisfied.

Assumption A1 allows the data dimension, p, to be greater than the (maximum) number of observations, T_max, and to grow at the same rate as T_max does. This is a common regularity condition for shrinkage estimators [24]. Assumption A2 guarantees that the average sample covariance matrix $\overline{S}={\sum }_{i=1}^n{T}_i{S}_i/N$ utilized in the initial step of Algorithm 1 is positive definite. Together with Assumption A5, the eigenvectors of $\overline{S}$ are consistent estimators of Π [2]. Assumptions A3 and A4 regulate z_it on higher-order moments, which is equivalent to imposing restrictions on the higher-order moments of y_it. When the data are assumed to be normally distributed, both A3 and A4 are satisfied. Assumption A5 assumes that all the covariance matrices share the same eigenspace, though the ordering of the eigenvectors may differ. When p/T_min → 0, Zhao et al. [41] relaxed this assumption to partial common diagonalization and demonstrated the method robustness through numerical examples. Studying the asymptotic properties under the relaxation is difficult and not available in existing literature, especially when p > T_max.

Taking the eigenvectors of $\overline{S}$ as the initial values of γ, the following proposition demonstrates the consistency of the proposed estimator.

Proposition 3.1. Under Assumptions A1–A5, as n,T_min → ∞, the estimator of γ and β obtained by Algorithm 1 are asymptotically consistent.

To prove Proposition 3.1, we first study the asymptotic properties of $S_i^{*}$ and show that $S_i^{*}$ is the optimal linear shrinkage estimator of the covariance matrix under the squared loss. This is accomplished under the assumption that γ is given. As the initialization of γ is already a consistent estimator, the consistency of the solution after iteration follows. For β, it is firstly shown that the association between the shrinkage estimator, ${\Sigma }_i^{*}$, and the covariates is the same as the covariance matrix, Σ_i, does (Lemma 3.3). Thus, it is equivalent to optimize problems (2.1) and (2.2) to solve for β, and the solution is a consistent estimator of β based on the pseudo-likelihood theory [16]. In the iteration step of Algorithm 1, $S_i^{*}$ improves the estimation of the covariance matrices with lower squared loss, and in consequence, improves the estimation of γ and β. In Section 4, the improvement is demonstrated through simulation studies.

In Section 2, the optimization problem (2.2) introduces a linear combination of the sample covariance matrix and the identity matrix, ${\Sigma }_i^{*}$, that achieves the minimum expected squared error. From Theorem 2.1, the solution has population-level parameters. Thus, the sample counterpart, $S_i^{*}$, is introduced. The following Lemma 3.1 first shows that asymptotically, the weighting parameters in ${\Sigma }_i^{*}$ are well-behaved. Lemma 3.2 demonstrates that the corresponding sample counterpart of the weighting parameters are consistent estimators. Theorem 3.1 demonstrates that $S_i^{*}$ performs as well as ${\Sigma }_i^{*}$ does asymptotically.

Lemma 3.1. For given (γ, β), let T_min = min_i T_i, as T_min → ∞, μ, ϕ², ψ² and δ² are bounded.

Lemma 3.2. For given (γ, β), as T_min → ∞,

1. $\mathbb{E}{\left({\widehat{\delta }}_i^2-{\delta }_i^2\right)}^2\to 0$, for i = 1, …, n, and thus $\mathbb{E}{\left({\widehat{\delta }}^2-{\delta }^2\right)}^2\to 0$;

2. $\mathbb{E}{\left({\widehat{\psi }}_i^2-{\psi }_i^2\right)}^2\to 0$, for i = 1, …, n, and thus $\mathbb{E}{\left({\widehat{\psi }}^2-{\psi }^2\right)}^2\to 0$;

3. $\mathbb{E}{\left({\widehat{\varphi }}_i^2-{\varphi }_i^2\right)}^2\to 0$, for i = 1, …, n, and thus $\mathbb{E}{\left({\widehat{\varphi }}^2-{\varphi }^2\right)}^2\to 0$.

Theorem 3.1. For ∀ i ∈ {1, …, n}, $S_i^{*}$ is a consistent estimator of ${\Sigma }_i^{*}$, that is, as T_min = min_i T_i → ∞,

$$\mathbb{E}{\Vert S_i^{*}-{\Sigma }_i^{*}\Vert }^2\to 0.$$ (3.2)

Thus, the asymptotic expected loss of $S_i^{*}$ and ${\Sigma }_i^{*}$ are identical, that is,

$$\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{S}_i^{*}\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2-\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{\Sigma }_i^{*}\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2\to 0.$$ (3.3)

Next, we show that $S_i^{*}$ uniformly achieves the minimum quadratic risk asymptotically over all linear combinations of the sample covariance matrix and the identity matrix. For given (γ, β), let ${\Sigma }_i^{**}$ denote the solution to the following optimization problem,

$$\begin{gathered} \underset{{\rho }_1,{\rho }_2}{\text{minimize}}\frac{1}{n}\sum _{i=1}^n{\left\{{\mathit{\gamma }}^{\top }{\Sigma }_i^{**}\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2, \\ \text{ such that \hspace{1em}}{\Sigma }_i^{**}={\rho }_{1}I+{\rho }_2{S}_i,\text{\hspace{1em} for }i=1,\dots ,n\text{. } \end{gathered}$$ (3.4)

Theorem 3.2. $S_i^{*}$ is a consistent estimator of ${\Sigma }_i^{**}$, that is, as T_min = min_i T_i → ∞, for i = 1, …, n,

$$\mathbb{E}{\Vert S_i^{*}-{\Sigma }_i^{**}\Vert }^2\to 0.$$ (3.5)

Then, $S_i^{*}$ has the same asymptotic expected loss as ${\Sigma }_i^{**}$ does, that is,

$$\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{S}_i^{*}\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2-\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{\Sigma }_i^{**}\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2\to 0.$$ (3.6)

Theorem 3.3. Assume (γ, β) is given. With a fixed $n\in {\mathbb{N}}^{+}$, for any sequence of linear combinations ${\left\{{\widehat{\Sigma }}_i\right\}}_{i=1}^n$ of the identity matrix and the sample covariance matrix, where the combination coefficients are constant over i ∈ {1, …, n}, the estimator $S_i^{*}$ verifies:

$$\underset{T\to \infty }{\text{lim}}\underset{T_i\ge T}{\text{inf}}\left[\frac{1}{n}\sum _{i=1}^n\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{\widehat{\Sigma }}_i\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2-\frac{1}{n}\sum _{i=1}^n\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{S}_i^{*}\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2\right]\ge 0.$$ (3.7)

In addition, every sequence of ${\left\{{\widehat{\Sigma }}_i\right\}}_{i=1}^n$. that performs as well as ${\left\{S_i^{*}\right\}}_{i=1}^n$ identical to ${\left\{S_i^{*}\right\}}_{i=1}^n$ in the limit:

$$\underset{T\to \infty }{\text{lim}}\left[\frac{1}{n}\sum _{i=1}^n\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{\widehat{\Sigma }}_i\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2-\frac{1}{n}\sum _{i=1}^n\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{S}_i^{*}\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2\right]=0$$ (3.8)

$$\iff \mathbb{E}{\Vert {\widehat{\Sigma }}_i-S_i^{*}\Vert }^2\to 0,\mathit{\text{ for }}i=1,\dots ,n.$$ (3.9)

The difference between ${\Sigma }_i^{**}$ and ${\Sigma }_i^{*}$ is that ${\Sigma }_i^{**}$ minimizes the squared loss instead of the expected loss, while asymptotically they are equivalent (Theorems 3.1 and 3.2). Theorem 3.3 presents the main result that, with a fixed sample size n, the proposed shrinkage estimator ${\left\{S_i^{*}\right\}}_{i=1}^n$ achieves the uniformly minimum (average) quadratic risk asymptotically among all linear combinations of the identity matrix and the sample covariance matrix. Here, “average” implies an average over the subjects, and “asymptotically” refers to that the number of observations within each subject increases to infinity. Therefore, $S_i^{*}$ is asymptotically optimal. In addition, it is guaranteed that $S_i^{*}$ is positive definite (see a discussion in Appendix Section A.8). Thus, there exits unique solution to the optimization problem (2.1).

Next, we study the asymptotic properties of the model coefficient estimator. Let $\widehat{\beta }$ denote the solution to the optimization problem (2.1).

Lemma 3.3. For given γ, assume the linear shrinkage estimator, ${\Sigma }_i^{*}$, satisfies

$$\mathbb{E}\left({\mathit{\gamma }}^{\top }{\Sigma }_i^{*}\mathit{\gamma }\right)=\text{exp}\left(x_i^{\top }{\beta }^{*}\right),\mathit{\text{ for }}i=1,\dots ,n,$$ (3.10)

then

$${\mathit{\beta }}^{*}=\mathit{\beta }.$$ (3.11)

Theorem 3.4. For given γ, assume Assumptions A1–A5 are satisfied, $\widehat{\beta }$ is a consistent estimator of β as n, T_min → ∞, where T_min = min_i T_i.

Lemma 3.3 implies that under the rotation γ, the expectation of the shrinkage estimator, ${\Sigma }_i^{*}$, has the same association with the covariates as the true covariance matrix, Σ_i, does. $S_i^{*}$ is a consistent estimator of ${\Sigma }_i^{*}$ and is positive definite. This substantiates the choice of $S_i^{*}$ replacing the sample covariance matrix S_i in the optimization problem. Theorem 3.4 states the consistency of $\widehat{\beta }$.

4. Simulation Study

4.1. γ is known

In this section, we focus on examining the performance of the proposed method in estimating the covariance matrices and model coefficients by assuming the projection γ is known. Three methods are compared. (i) Estimate each individual covariance matrix using the estimator proposed in Ledoit and Wolf [24] and replace ${\widehat{\Sigma }}_i$ with it in the optimization problem (2.1). We denote this approach as LW-CAP (Ledoit and Wolf based Covariate Assisted Principal regression), where the shrinkage is estimated on each individual covariance matrix. (ii) Estimate the covariance matrices using the proposed shrinkage estimator $S_i^{*}$ in (2.5). We denote this approach as CS-CAP (Covariate dependent Shrinkage CAP), where the shrinkage parameters are assumed to be shared across subjects. (iii) Estimate each individual covariance matrix using the sample covariance matrix and plug into the optimization problem (2.1). This is the CAP approach proposed in Zhao et al. [41], which is only applicable when T_min = min_i T_i > p.

The covariance matrices are generated using the eigendecomposition Σ_i = ΠΛ_iΠ^⊤, where Π = (π₁, …, π_p) is an orthonormal matrix in ${ℝ}^{p\times p}$ and Λ_i = diag{λ_i1, …, λ_ip} is a diagonal matrix with the diagonal elements to be the eigenvalues, for i = 1, …, n. In Λ_i, the diagonal elements are exponentially decaying, where eigenvalues of the second and the fourth dimension (D2 and D4) satisfy the log-linear model in (1.1). We consider a case with a single predictor X (thus q = 2), which is generated from a Bernoulli distribution with probability 0.5 to be one. For D2, the coefficient β₁ = −1; and for D4, β₁ = 1. For the rest dimensions, λ_ij, for i = 1, …, n, is generated from a log-normal distribution, where the mean of the corresponding normal distribution decreases from 5 to −1 over j. Cases when p = 20, 50, 100 are considered.

We first compare the three approaches, LW-CAP, CS-CAP and CAP, under sample sizes n = 50 and T_i = T = 50 for all i and present the result in Table 1. In the estimation, for dimension j, γ is set to be π_j. In Table 1, we present the bias and the mean squared error (MSE) in estimating the eigenvalues and the model coefficient in D2 and D4. From the table, for both the eigenvalues and β₁, CS-CAP yields lower estimation bias and MSE than LW-CAP does. When p < T, CS-CAP achieves a similar estimation bias as the CAP approach does in estimating the covariance matrices, while the MSE is slightly lower. For the estimation of β₁, CS-CAP yields slightly lower bias. As the dimension p increases, the bias and MSE of eigenvalue estimates from LW-CAP increase; while the bias and MSE of the estimates from CS-CAP are similar at all p settings. This demonstrates the superiority of the proposed estimator in estimating the covariance matrices. Figure 1 presents the estimation bias and MSE of CS-CAP estimator at various levels of T when fixing n = 50 when p = 20. From the figure, as the number of observations within each subject increases, the estimates converge to the truth.

Table 1.

Bias and mean squared error (MSE) in estimating the eigenvalues of the covariance matrices and bias, MSE, and coverage probability (CP) in estimating β₁ coefficient with sample sizes n = 50 and T_i = T = 50, for i = 1, …, n, when γ is known.

		Method	${\widehat{\lambda }}_{ij}$		${\widehat{\beta }}_1$
			Bias	MSE	Bias	MSE	CP
p = 20	D2	LW-CAP	−6.520	225.360	0.053	0.006	0.795
		CS-CAP	−1.175	204.686	0.001	0.004	0.935
		CAP	−1.175	206.117	−0.003	0.004	0.935
	D4	LW-CAP	−7.422	277.888	−0.040	0.005	0.860
		CS-CAP	−1.223	249.881	0.001	0.004	0.905
		CAP	−1.223	251.595	0.005	0.004	0.910
p = 50	D2	LW-CAP	−7.975	244.326	0.028	0.004	0.915
		CS-CAP	−1.428	202.141	0.008	0.003	0.935
		CAP	-	-	-	-	-
	D4	LW-CAP	−8.641	295.221	−0.012	0.004	0.915
		CS-CAP	−1.242	248.254	0.001	0.004	0.925
		CAP	-	-	-	-	-
p = 100	D2	LW-CAP	−8.924	260.268	0.010	0.004	0.915
		CS-CAP	−0.973	203.151	−0.001	0.003	0.930
		CAP	-	-	-	-	-
	D4	LW-CAP	−10.487	331.864	−0.011	0.003	0.940
		CS-CAP	−1.705	245.754	−0.007	0.003	0.940
		CAP	-	-	-	-	-

Fig 1.

Bias and mean squared error (MSE) in estimating the eigenvalues of the covariance matrices and bias, MSE, and coverage probability in estimating β₁ coefficient using CS-CAP with the number of subjects n = 50 at various numbers of observations from each subject with p = 20 when γ is known.

4.2. γ is unknown

In this section, we evaluate the performance of the CS-CAP approach when γ is unknown and estimated by solving optimization problem (2.1) using Algorithm 1. The data are generated following the same procedure as in Section 4.1. To evaluate the performance in estimating the projection γ, we consider a similarity metric measured by $\left|\langle \widehat{\mathit{\gamma }},\mathit{\gamma }\rangle \right|$, where 〈·, ·〉 is the inner product of two vectors and $\widehat{\mathit{\gamma }}$ denotes the estimate of γ. When this metric is one, the two vectors are identical (up to sign flipping); and when this metric is zero, the two vectors are orthogonal. Case where p = 100 is studied. The performance of the CS-CAP approach is firstly compared to the LW-CAP approach with sample sizes n = 100 and T_i = T = 100. The results are presented in Table 2. From the table, the CS-CAP approach improves the performance with much lower MSE in estimating the eigenvalues, and lower MSE and higher coverage probability (CP) in estimating the β coefficient. After iterations, the CS-CAP approach yields an estimate of the projection with much higher similarity to the truth. To further examine the performance of the CS-CAP approach under finite sample size, combinations of sample sizes n = 50, 100, 500, 1000 and T_i = T = 50, 100, 500, 1000 are considered. Figure 2 presents the performance in estimating the second dimension (D2), including the bias, the MSE and the CP of ${\widehat{\beta }}_1$, the MSE of ${\widehat{\lambda }}_{ij}$, and the similarity of $\widehat{\mathit{\gamma }}$ to the eigenvector of D2 (Appendix Section B.1 presents the results of the fourth dimension, D4). From the figure, as n, T → ∞, all estimates converge to the truth.

Table 2.

Bias, mean squared error (MSE), and coverage probability (CP) from 500 bootstrap samples in estimating the β₁ coefficient, the similarity of $\widehat{\mathit{\gamma }}$ to π_j and the standard error (SE), and the MSE in estimating the eigenvalues ${\widehat{\lambda }}_{ij}$, for j = 2, 4. Data dimension p = 100, sample size n = 100 and T_i = T = 100.

	Method	${\widehat{\beta }}_1$			$\widehat{\lambda }$	${\widehat{\lambda }}_{ij}$
		Bias	MSE	CP	$\left|\langle \widehat{\gamma },{\pi }_j\rangle \right|$ (SE)	MSE
D2	LW-CAP	−0.027	0.002	0.782	0.653 (0.033)	1812.091
	CS-CAP	−0.023	0.001	0.855	0.931 (0.012)	173.225
D4	LW-CAP	0.018	0.002	0.770	0.666 (0.027)	2186.265
	CS-CAP	0.019	0.001	0.845	0.926 (0.011)	231.856

Fig 2.

Estimation performance of CS-CAP in estimating the second dimension (D2) when γ is unknown. For ${\widehat{\beta }}_1$, (a) bias, (b) mean squared error (MSE) and (c) coverage probability (CP) are presented, where CP is obtained from 500 bootstrap samples. For the eigenvalues ${\widehat{\lambda }}_{ij}$, (d) MSE is presented. For $\widehat{\mathit{\gamma }}$, (e) similarity to π₂ is presented. Data dimension p = 100. Sample sizes vary from n = 50, 100, 500, 100 and T_i = T = 50, 100, 500, 1000.

In Appendix Section B.2, the robustness of the proposed method to model misspecification is examined. Two types of model misspecification are considered, model misspecification in β and model misspecification in γ. When the log-linear model is misspecified, the proposed approach can correctly identify the linear projections under certain scenarios, while the estimate of model coefficients is biased. The proposed approach is robust to the setting that the eigenvectors of the covariance matrices are partially common, while it will not work when the eigenvectors are completely unique to each covariance matrix.

5. The Alzheimer’s Disease Neuroimaging Initiative Study

Data used in this study are obtained from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database (adni.loni.usc.edu). The ADNI was launched in 2003 as a public-private partnership, led by Principal Investigator Michael W. Weiner, MD. The primary goal of ADNI has been to test whether serial magnetic resonance imaging (MRI), positron emission tomography (PET), other biological markers, and clinical and neuropsychological assessment can be combined to measure the progression of mild cognitive impairment (MCI) and early Alzheimer’s disease (AD).

We apply the proposed approach to ADNI resting-state functional MRI (fMRI) data acquired at the baseline screening. AD is an irreversible neurodegenerative disease that destroys memory and related brain functions causing problems in cognition and behavior. Apolipoprotein E ε4 (APOE-ε4) has been consistently identified as a strong genetic risk factor for AD. With an increasing number of APOE-ε4 alleles, the lifetime risk of developing AD increases, and the age of onset decreases [10]. Thus, APOE-ε4 is generally treated as a potential therapeutic target [33]. In AD studies, resting-state fMRI is another emerging biomarker for diagnosis [23]. It is important to articulate the genetic impact on brain functional architecture. In this study, n = 194 subjects diagnosed with either MCI or AD are analyzed. Resting-state fMRI data collected at the initial screening are preprocessed. Time courses are extracted from p = 75 brain regions, including 60 cortical and 15 subcortical regions grouped into 10 functional modules, using the Harvard-Oxford Atlas in FSL [35]. For each time course, a subsample is taken with an effective sample size of T = 67 to remove the temporal dependence. The resulting data, denoted as y_it (for i = 1, …, n and t = 1, …, T), are assumed to follow a multivariate normal distribution with mean zero and covariance Σ_i. The off-diagonal elements in Σ_i represent the pairwise functional connectivity between brain regions and Σ_i represents the brain functional architecture of subject i. In the regression model, APOE-ε4, sex, and age are entered as the covariates (x_i’s). The validity of model assumptions is discussed in Appendix Section C.1.

The CS-CAP approach is applied to identify brain subnetworks within which the functional connectivity demonstrates a significant association with APOE-ε4. Using the deviation from diagonality criterion, CS-CAP identifies three components denoted as C1, C2, and C3. The model coefficients and 95% bootstrap confidence interval from 500 bootstrap samples are presented in Table 3. From the table, C3 is significantly associated with APOE-ε4 and age; C1 and C2 are significantly associated with sex and age. To better interpret C3, a fused lasso regression [36] is employed to sparsify the loading profile, similarly as in the sparse principal component analysis proposed in Zou et al. [42]. The fused lasso regularization is defined based on the modular information to impose local smoothness and consistency [19, 40]. Figure 3(a) presents the sparse loading profile colored by the corresponding functional module, and Figure 3(b) is the river plot illustrating the loading configuration. In C3, all regions with negative loadings are subcortical regions. Contributions to positive loadings are from regions in the default mode network (DMN), the ventral- and dorsal-attention networks, and the somato-motor network. Figure 3(c) presents these regions on a brain map. C3 is negatively associated with APOE-ε4 indicating that functional connectivity between regions in the same sign among APOE-ε4 carriers is lower, while connectivity between regions in the opposite signs among APOE-ε4 carriers is higher. The findings are in line with existing knowledge about AD. Compared to APOE-ε4 non-carriers, more functional connectivity between the left hippocampus and the insular/prefrontal cortex while more functional disconnection of the hippocampus has been observed in APOE-ε4 carriers [12]. Alterations in DMN connectivity in cognitively normal APOE-ε4 carriers have been reported across all age groups [3]. Increased connectivity in the limbic system, including the hippocampus, the amygdala, and the thalamus, has been detected in individuals with memory impairment [18, 17], though the effect of APOE-ε4 carriage lacks consensus [3]. It was shown that the limbic hyperconnectivity is positively associated with the memory performance, suggesting the preservation of brain function due to increased connectivity in the medial temporal lobe pathology [17].

Table 3.

Model coefficient estimate and 95% bootstrap confidence interval using the PS-CAP approach. The intervals are obtained over 500 bootstrap samples.

	APOE-ε4	Sex	Age
C1	0.012 (−0.031, 0.263)	−0.431 (−0.636, −0.230)	−0.227 (−0.319, −0.129)
C2	0.049 (−0.191, 0.309)	−0.544 (−0.867, −0.186)	−0.232 (−0.383, −0.066)
03	−0.156 (−0.270, −0.045)	−0.061 (−0.201, 0.075)	−0.241 (−0.328,−0.172)

Fig 3.

(a)The sparsified loading profile, (b) the module river plot, and (c) regions with nonzero loadings in a brain map of C3. In (a) and (b), the figure and the legend are colored by brain functional modules. In (c), the brain maps are colored by the loading weights.

6. Discussion

In this study, we introduce an approach to perform linear regression with multiple high dimensional covariance matrices as the outcome. A linear shrinkage estimator of the covariance matrix is firstly introduced, where the shrinkage coefficients are shared parameters across subjects. It is shown that the proposed estimator is optimal achieving the uniformly minimum quadratic loss asymptotically among all linear combinations of the identity matrix and the sample covariance matrix. Replacing the sample covariance matrices with the proposed well-conditioned estimator in the likelihood function, the linear projection parameter and the model coefficient are shown to be consistently estimated. Through simulation studies, the proposed approach demonstrates superior performance in estimating the covariance matrices and the model coefficients with lower estimation bias and variation over the existing methods. Applying to a resting-state fMRI data set acquired from the ADNI study, the findings are consistent with existing knowledge about AD.

The proposed framework extends the proposal in Zhao et al. [41] to high dimensional scenario. When p is small, the proposed shrinkage estimator demonstrates lower squared loss than the sample covariance matrix as suggested in both theoretical results and simulation studies. Different from the linear shrinkage estimator introduced in Ledoit and Wolf [24], which was proposed for a single covariance matrix estimation, the shrinkage coefficients considered in this study are population level parameters shared across subjects. This is superior than the individual shrinkage as the proposed one leverages the accuracy of the sample covariance matrix and the variability in the eigenvalues across subjects.

In this study, the asymptotic properties are studied under the assumption that the covariance matrices have the same eigendecomposition. We leave the study of the consistency relaxing this assumption to future research. The proposed shrinkage estimator is optimal with respect to a squared risk. However, this may overshrink the small eigenvalues [11]. Other types of loss function, such as the Stein’s loss, will be considered in the future. In the ADNI application, we included an ad hoc procedure to select important brain regions for interpretation. A next-step research is to include the regularization on γ into the optimization or to introduce an efficient approach to draw inference on the loadings (such as a bootstrap sampling procedure).

Appendix A: Theory and Proof

A.1. Proof of Theorem 2.1 and Lemma 2.1

Proof. Given (γ, β), $\mathbb{E}\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)={\mathit{\gamma }}^{\top }{\Sigma }_i\mathit{\gamma }=\text{exp}\left(x_i^{\top }\beta \right)$. For the objective function in (2.2), under the constraint that ${\Sigma }_i^{*}=\rho \mu I+(1-\rho )S_i$, we have

$$\begin{gathered} f(\mu ,\rho ) \\ =\frac{1}{n}\sum _{i=1}^n\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{\Sigma }_i^{*}\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2 \\ =\frac{1}{n}\sum _{i=1}^n\left[{\rho }^2{\left\{\mu \left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2+{(1-\rho )}^2\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2\right]. \end{gathered}$$

In order to minimize the objective function, as the objective function is convex, derivatives are firstly taken over μ and ρ.

For μ,

$$\begin{gathered} \frac{\partial f}{\partial \mu }={\rho }^2\frac{1}{n}\sum _{i=1}^{n}2\left\{\mu \left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)-\text{exp}\left(x_i^{\top }\beta \right)\right\}\left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)=0, \\ \Rightarrow \mu =\frac{1}{n\left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)}\sum _{i=1}^n\text{exp}\left(x_i^{\top }\beta \right). \end{gathered}$$

For ρ, let ${\varphi }_i^2={\left\{\mu \left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2$ and ${\psi }_i^2=\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2$,

$$\begin{gathered} \frac{\partial f}{\partial \rho }=2\rho \left(\frac{1}{n}\sum _{i=1}^n{\varphi }_i^2\right)-2(1-\rho )\left(\frac{1}{n}\sum _{i=1}^n{\psi }_i^2\right)=0, \\ \Rightarrow \rho =\frac{{\sum }_{i=1}^n{\psi }_i^2}{{\sum }_{k=1}^n{\varphi }_i^2+{\sum }_{i=1}^n{\psi }_i^2}. \end{gathered}$$

Let ${\delta }_i^2=\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }-\mu \left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)\right\}}^2$, then ${\delta }_i^2={\varphi }_i^2+{\psi }_i^2$. Let ${\varphi }^2={\sum }_{i=1}^n{\varphi }_i^2/n$, ${\psi }^2={\sum }_{i=1}^n{\psi }_i^2/n$, and ${\delta }^2={\sum }_{i=1}^n{\delta }_i^2/n$ (thus, δ² = ϕ² + ψ²), the optimizer of problem (2.2) is

$${\Sigma }_i^{*}=\frac{{\psi }^2}{{\delta }^2}\mu I+\frac{{\varphi }^2}{{\delta }^2}{S}_i,i=1,\dots ,n.$$

The minimum value of the function is

$$\begin{gathered} \frac{1}{n}\sum _{i=1}^n\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{\Sigma }_i^{*}\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2 \\ =\frac{1}{n}\sum _{i=1}^n\mathbb{E}{\left\{\frac{{\psi }^2}{{\delta }^2}\mu {\mathit{\gamma }}^{\top }\mathit{\gamma }+\frac{{\varphi }^2}{{\delta }^2}{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }-\frac{{\psi }^2+{\varphi }^2}{{\delta }^2}\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2 \\ =\frac{1}{n}\sum _{i=1}^n\left[\mathbb{E}{\left\{\frac{{\psi }^2}{{\delta }^2}\mu {\mathit{\gamma }}^{\top }\mathit{\gamma }-\frac{{\psi }^2}{{\delta }^2}\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2+\mathbb{E}{\left\{\frac{{\varphi }^2}{{\delta }^2}{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }-\frac{{\varphi }^2}{{\delta }^2}\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2\right] \\ =\frac{1}{n}\sum _{i=1}^n\left(\frac{{\psi }^4}{{\delta }^4}{\varphi }_i^2+\frac{{\varphi }^4}{{\delta }^4}{\psi }_i^2\right) \\ =\frac{{\psi }^4{\varphi }^2+{\varphi }^4{\psi }^2}{{\delta }^4} \\ =\frac{{\varphi }^2{\psi }^2}{{\delta }^2}. \end{gathered}$$

□

A.2. Proof of Proposition 3.1

Proof. Under Assumptions A2 and A5, the eigenvectors of $\overline{S}$ are consistent estimators of Π. Replace γ with its estimate in Theorems 3.1–3.3 and Theorem 3.4, the consistency of β follows. □

A.3. Proof of Lemma 3.1

Proof. (1) For μ,

$$\mu =\frac{1}{n\left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)}\sum _{i=1}^n\text{exp}\left(x_i^{\top }\beta \right)=\frac{1}{n}\sum _{i=1}^n\frac{{\mathit{\gamma }}^{\top }{\Sigma }_i\mathit{\gamma }}{{\mathit{\gamma }}^{\top }\mathit{\gamma }}\le \frac{1}{n}\sum _{i=1}^n{\Vert {\Sigma }_i\Vert }_2^2.$$

Under Assumption A2,

$$\begin{gathered} \frac{1}{n}\sum _{i=1}^n{\Vert {\Sigma }_i\Vert }_2^2=\frac{1}{n}\sum _{i=1}^n{\Vert {\Lambda }_i\Vert }_2^2 \\ \le \frac{1}{n}\sum _{i=1}^n{\Vert {\Lambda }_i\Vert }_F^2 \\ =\frac{1}{n}\sum _{i=1}^n\left\{\frac{1}{p}\sum _{j=1}^p\mathbb{E}{\left(z_{i1j}^2\right)}^2\right\} \\ =\frac{1}{n}\sum _{i=1}^n\left\{\frac{1}{p}\sum _{j=1}^p\mathbb{E}\left(z_{i1j}^4\right)\right\} \\ \le \frac{1}{n}\sum _{i=1}^n\sqrt{\frac{1}{p}\sum _{j=1}^p\mathbb{E}{\left(z_{i1j}\right)}^8} \\ \le \frac{1}{n}\sum _{i=1}^n\sqrt{C_2} \\ =\sqrt{C_2}, \end{gathered}$$

where ∥·∥_F is the Frobenius norm of a matrix.

(2) For ϕ², upper limits of ${\varphi }_i^2$ is derived first.

$$\begin{gathered} {\varphi }_i^2={\left\{\mu \left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2 \\ \le {\mu }^2{\left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)}^2+{\left\{\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2 \\ ={\mu }^2{\left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)}^2+{\left({\mathit{\gamma }}^{\top }{\Sigma }_i\mathit{\gamma }\right)}^2 \\ \le \left({\mu }^2+{\Vert {\Sigma }_i\Vert }_2^4\right){\left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)}^2. \end{gathered}$$

From the above derivation, we have

$${\mu }^2\le C_2,\text{ and }{\Vert {\Sigma }_i\Vert }_2^2={\Vert {\Lambda }_i\Vert }_2^2\le {\Vert {\Lambda }_i\Vert }_F^2\le \sqrt{C_2}.$$

Since γ is given, without loss of generality, assume that ∥γ∥₂ = 1, i.e., γ^⊤γ = 1. Then,

$${\varphi }_i^2\le 2C_2\left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)=2C_2.$$

Thus,

$${\varphi }^2=\frac{1}{n}\sum _{i=1}^n{\varphi }_i^2\le 2C_2.$$

(3) For ψ², analogously, ${\psi }_i^2$ is considered first.

$$\begin{gathered} {\psi }_i^2=\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2=\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }\left(S_i-{\Sigma }_i\right)\mathit{\gamma }\right\}}^2\le {\left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)}^2\mathbb{E}{\Vert S_i-{\Sigma }_i\Vert }_2^2 \\ \mathbb{E}{\Vert S_i-{\Sigma }_i\Vert }_F^2=\frac{1}{p}\sum _{j=1}^p\sum _{k=1}^p\mathbb{E}\left\{{\left(\frac{1}{T_i}\sum _{t=1}^{T_i}{y}_{itj}{y}_{itk}-{\sigma }_{ijk}\right)}^2\right\} \\ =\frac{1}{p}\sum _{j=1}^p\sum _{k=1}^p\mathbb{E}\left\{{\left(\frac{1}{T_i}\sum _{t=1}^{T_i}{z}_{itj}{z}_{itk}-{\lambda }_{ijk}\right)}^2\right\} \\ =\frac{1}{p}\sum _{j=1}^p\sum _{k=1}^p\text{Var}\left(\frac{1}{T_i}\sum _{t=1}^{T_i}{z}_{itj}{z}_{itk}\right) \\ =\frac{1}{p}\sum _{j=1}^p\sum _{k=1}^p\frac{1}{T_i}\text{Var}\left(z_{i1j}{z}_{i1k}\right) \\ \le \frac{1}{p{T}_i}\sum _{j=1}^p\sum _{k=1}^p\mathbb{E}\left(z_{i1j}^2{z}_{i1k}^2\right) \\ \le \frac{1}{p{T}_i}\sum _{j=1}^p\sum _{k=1}^p\sqrt{\mathbb{E}{z}_{i1j}^4}\sqrt{\mathbb{E}{z}_{i1k}^4} \\ \le \frac{p}{T_i}{\left(\frac{1}{p}\sum _{j=1}^p\sqrt{\mathbb{E}{z}_{i1j}^4}\right)}^2 \\ \le \frac{p}{T_i}\left(\frac{1}{p}\sum _{j=1}^p\mathbb{E}{z}_{i1j}^4\right) \\ \le \frac{p}{T_i}\sqrt{\frac{1}{p}\sum _{j=1}^p\mathbb{E}{z}_{i1j}^8} \\ \le C_1\sqrt{C_2} \end{gathered}$$

Thus, for ψ²,

$${\psi }^2=\frac{1}{n}\sum _{i=1}^n{\psi }_i^2\le \frac{1}{n}\sum _{i=1}^n{\left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)}^2{C}_1\sqrt{C_2}=C_1\sqrt{C_2}.$$

(4) Finally, for δ²,

$${\delta }^2={\varphi }^2+{\psi }^2\le 2C_2+C_1\sqrt{C_2}.$$

□

A.4. Proof of Lemma 3.2

Proof. In the proof of Lemma 3.2, here, it is assumed that γ is a column of Π_i indexed by j_i, for i = 1, …, n (Assumption A4).

(i) First, we prove the consistency of ${\widehat{\delta }}_i^2$.

$$\begin{gathered} {\widehat{\delta }}_i^2-{\delta }_i^2 \\ ={\left\{{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }-\mu \left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)\right\}}^2-\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }-\mu \left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)\right\}}^2 \\ =\left\{{\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)}^2-\mathbb{E}{\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)}^2\right\}-2\mu \left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)\left\{\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)-\mathbb{E}\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)\right\} \end{gathered}$$

Under Assumption A4,

$${\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }=\frac{1}{T_i}\sum _{t=1}^{T_i}{\mathit{\gamma }}^{\top }{y}_{it}{y}_{it}^{\top }\mathit{\gamma }=\frac{1}{T_i}\sum _{t=1}^{T_i}{z}_{it{j}_i}^2.$$

$${\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)}^2=\frac{1}{T_i^2}{\left(\sum _{t=1}^{T_i}{z}_{it{j}_i}^2\right)}^2=\frac{1}{T_i^2}\sum _{t=1}^{T_i}{z}_{it{j}_i}^4+\frac{1}{T_i^2}\sum _{t\ne s}{z}_{it{j}_i}^2{z}_{is{j}_i}^2.$$

$$\begin{gathered} \mathbb{E}{\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)}^2=\frac{1}{T_i^2}{T}_i\mathbb{E}{z}_{i1j_i}^4+\frac{1}{T_i}{T}_i\left(T_i-1\right){\left(\mathbb{E}{z}_{it{j}_i}^2\right)}^2 \\ =\frac{1}{T_i}\mathbb{E}{z}_{i1j_i}^4+\frac{T_i\left(T_i-1\right)}{T_i^2}{\left({\mathit{\gamma }}^{\top }{\Sigma }_i\mathit{\gamma }\right)}^2. \end{gathered}$$

For ∀ ϵ > 0,

$$\begin{gathered} \mathbb{P}\left\{\left|\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)-\mathbb{E}\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)\right|\ge ϵ\right\} \\ \le \frac{1}{{ϵ}^2}\text{Var}\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right) \\ =\frac{1}{{ϵ}^2}\left[\mathbb{E}{\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)}^2-{\left\{\mathbb{E}\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)\right\}}^2\right] \\ =\frac{1}{{ϵ}^2}\left\{\frac{1}{T_i^2}\mathbb{E}{z}_{i1j_i}^4+\frac{T_i\left(T_i-1\right)}{T_i^2}{\left({\mathit{\gamma }}^{\top }{\Sigma }_i\mathit{\gamma }\right)}^2-{\left({\mathit{\gamma }}^{\top }{\Sigma }_i\mathit{\gamma }\right)}^2\right\} \\ \stackrel{T_i\to \infty }{\to }0. \end{gathered}$$

$$\begin{gathered} \mathbb{E}{\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)}^4 \\ =\frac{1}{T_i^4}\mathbb{E}{\left(\sum _{t=1}^{T_i}{z}_{it{j}_i}^2\right)}^4 \\ =\frac{1}{T_i^4}\left\{\sum _t\mathbb{E}{z}_{it{j}_i}^8+2\sum _{t\ne s}\mathbb{E}{z}_{it{j}_i}^4{z}_{is{j}_i}^4+2\sum _u\mathbb{E}\left(z_{iu{j}_i}^4\sum _{t\ne s}{z}_{it{j}_i}^2{z}_{is{j}_i}^2\right)+\sum _{u\ne v}\sum _{t\ne s}\mathbb{E}\left(z_{it{j}_i}^2{z}_{is{j}_i}^2{z}_{iu{j}_i}^2{z}_{iv{j}_i}^2\right)\right\} \\ =\frac{1}{T_i^4}\left\{T_i\mathbb{E}{z}_{i1j_i}^8+2T_i\left(T_i-1\right){\left(\mathbb{E}{z}_{i1j_i}^4\right)}^2+2T_i^2\left(T_i-1\right)\mathbb{E}{z}_{i1j_i}^4{\left(\mathbb{E}{z}_{i1j_i}^2\right)}^2+T_i^2{\left(T_i-1\right)}^2{\left(\mathbb{E}{z}_{i1j_i}^2\right)}^4\right\}. \end{gathered}$$

$$\begin{gathered} {\left\{\mathbb{E}{\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)}^2\right\}}^2 \\ =\frac{1}{T_i^2}{\left(\mathbb{E}{z}_{i1j_i}^4\right)}^2+\frac{2T_i\left(T_i-1\right)}{T_i^3}\mathbb{E}{z}_{i1j_i}^4{\left(\mathit{\gamma }{\Sigma }_i\mathit{\gamma }\right)}^2+\frac{T_i^2{\left(T_i-1\right)}^2}{T_i^4}{\left(\mathit{\gamma }{\Sigma }_i\mathit{\gamma }\right)}^4. \end{gathered}$$

For ∀ ϵ > 0,

$$\begin{gathered} \mathbb{P}\left\{\left|{\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)}^2-\mathbb{E}{\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)}^2\right|\ge ϵ\right\} \\ \le \frac{1}{{ϵ}^2}\text{Var}{\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)}^2 \\ =\frac{1}{{ϵ}^2}\left[\mathbb{E}{\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)}^4-{\left\{\mathbb{E}{\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)}^2\right\}}^2\right] \\ =\frac{1}{{ϵ}^2}\left\{\frac{1}{T_i^3}\mathbb{E}{z}_{i1j_i}^8+\frac{T_i-2}{T_i^3}{\left(\mathbb{E}{z}_{i1j_i}^4\right)}^2\right\} \\ \stackrel{T_i\to \infty }{\to }0. \end{gathered}$$

Therefore, as T_min = min_i T_i → ∞,

$$\mathbb{E}{\left({\widehat{\delta }}_i^2-{\delta }_i^2\right)}^2\to 0,\text{ for }i=1,\dots ,n,\text{ and }\mathbb{E}{\left({\widehat{\delta }}^2-{\delta }^2\right)}^2\to 0.$$

(ii) Secondly, prove the consistency of ${\widehat{\psi }}_i^2$, for i = 1, …, n.

$${\widehat{\psi }}_i^2-{\psi }_i^2=\frac{1}{T_i}{\left\{{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2-\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2.$$

$$\begin{gathered} \mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2=\mathbb{E}{\left\{\frac{1}{T_i}\sum _t{z}_{it{j}_i}^2-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2 \\ =\frac{1}{T_i^2}\sum _t\text{Var}\left(z_{it{j}_i}^2\right) \\ =\frac{1}{T_i}\text{Var}\left(z_{i1j_i}^2\right). \end{gathered}$$

$$\begin{gathered} {\widehat{\psi }}_i^2-{\psi }_i^2 \\ =\frac{1}{T_i}\left[{\left\{{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2-\text{Var}\left(z_{i1j_i}^2\right)\right] \\ =\frac{1}{T_i}\left[{\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)}^2-\mathbb{E}{z}_{i1j_i}^4-2\text{exp}\left(x_i^{\top }\beta \right)\left\{{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}\right]. \end{gathered}$$

From above derivation and the fact that $\mathbb{E}\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)={\mathit{\gamma }}^{\top }{\Sigma }_i\mathit{\gamma }=\text{exp}\left(x_i^{\top }\beta \right)$, as T_i → ∞, for ∀ ϵ > 0,

$$\mathbb{P}\left\{\left|\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)-\mathbb{E}\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)\right|\ge ϵ\right\}\to 0.$$

As both (γ^⊤ S_iγ)² and $\mathbb{E}{z}_{i1j_i}^4$ are bounded, then, as T_min = min_i T_i → ∞,

$$\mathbb{E}{\left({\widehat{\psi }}_i^2-{\psi }_i^2\right)}^2\to 0,\text{ for }i=1,\dots ,n.$$

Let ${\tilde{\psi }}_i^2=\text{min}\left({\widehat{\psi }}_i^2,{\widehat{\delta }}_i^2\right)$.

$$\begin{gathered} {\tilde{\psi }}_i^2-{\psi }_i^2=\text{min}\left({\widehat{\psi }}_i^2,{\widehat{\delta }}_i^2\right)-{\psi }_i^2 \\ \le {\widehat{\psi }}_i^2-{\psi }_i^2 \\ \le \left|{\widehat{\psi }}_i^2-{\psi }_i^2\right| \\ \le \text{max}\left(\left|{\widehat{\psi }}_i^2-{\psi }_i^2\right|,\left|{\widehat{\delta }}_i^2-{\delta }_i^2\right|\right). \end{gathered}$$

${\delta }_i^2={\varphi }_i^2+{\psi }_i^2\ge {\psi }_i^2$, then

$$\begin{gathered} {\tilde{\psi }}_i^2-{\psi }_i^2=\text{min}\left({\widehat{\psi }}_i^2,{\widehat{\delta }}_i^2\right)-{\psi }_i^2 \\ =\text{min}\left({\widehat{\psi }}_i^2-{\psi }_i^2,{\widehat{\delta }}_i^2-{\psi }_i^2\right) \\ \ge \text{min}\left({\widehat{\psi }}_i^2-{\psi }_i^2,{\widehat{\delta }}_i^2-{\delta }_i^2\right) \\ \ge \text{min}\left(-\left|{\widehat{\psi }}_i-{\psi }_i^2\right|,-\left|{\widehat{\delta }}_i^2-{\delta }_i^2\right|\right) \\ \ge -\text{max}\left(\left|{\widehat{\psi }}_i-{\psi }_i^2\right|,\left|{\widehat{\delta }}_i^2-{\delta }_i^2\right|\right). \end{gathered}$$

$$\mathbb{E}{\left({\tilde{\psi }}_i^2-{\psi }_i^2\right)}^2\le \mathbb{E}\left\{\text{max}{\left(\left|{\widehat{\psi }}_i-{\psi }_i^2\right|,\left|{\widehat{\delta }}_i^2-{\delta }_i^2\right|\right)}^2\right\}\le \mathbb{E}{\left({\widehat{\psi }}_i^2-{\psi }_i^2\right)}^2+\mathbb{E}{\left({\widehat{\delta }}_i^2-{\delta }_i^2\right)}^2.$$

Therefore, as T_min = min_i T_i → ∞,

$$\mathbb{E}{\left({\tilde{\psi }}_i^2-{\psi }_i^2\right)}^2\to 0,\text{ for }i=1,\dots ,n,\text{ and }\mathbb{E}{\left({\widehat{\psi }}^2-{\psi }^2\right)}^2\to 0.$$

(iii) Lastly, ${\widehat{\varphi }}_i^2={\widehat{\delta }}_i^2-{\widehat{\psi }}_i^2$. The consistency of ${\widehat{\varphi }}_i^2$ (for i = 1, …, n) and ${\widehat{\varphi }}^2$ are straightforward. □

A.5. Proof of Theorem 3.1

In order to prove Theorem 3.1, the following lemma is firstly introduced. This lemma is also used to prove Lemma A.2 in the next section.

Lemma A.1. If $a_i^2$ is a sequence of nonnegative random variables (implicitly indexed by T_i) whose expectations converge to zero, for i = 1, …, n, and κ₁, κ₂ are two nonrandom scalars, and

$$\frac{a_i^2}{{\widehat{\delta }}_i^{k_1}{\delta }_i^{{\kappa }_2}}\le 2\left({\widehat{\delta }}_i^2+{\delta }_i^2\right)a.s.,$$

then, as T_min = min_i T_i → ∞,

$$\mathbb{E}\left(\frac{a_i^2}{{\widehat{\delta }}_i^{{\kappa }_1}{\delta }_i^{{\kappa }_2}}\right)\to 0.$$

Analogously, if a² is a sequence of nonnegative random variables (implicitly indexed by T_min = min_i T_i) whose expectations converge to zero, and κ₁, κ₂ are two nonrandom scalars, and

$$\frac{a^2}{\widehat{\delta }{\kappa }_1{\delta }^{{\kappa }_2}}\le 2\left({\widehat{\delta }}^2+{\delta }^2\right)a\text{.}s\text{., }$$

then, as T_min = min_i T_i → ∞,

$$\mathbb{E}\left(\frac{a^2}{\widehat{\delta }{\kappa }_1\delta {\kappa }_2}\right)\to 0.$$

Proof. For a fixed ϵ > 0, let ${\mathcal{T}}_i$ denote the set of indices T_i such that ${\delta }_i^2\le ϵ/8$. In Lemma 3.2, it is proved that $\mathbb{E}{\left({\widehat{\delta }}_i^2-{\delta }_i^2\right)}^2\to 0$. Thus, there exists an integer T_i1 such that ∀ T_i ≥ T_i1,

$$\mathbb{E}\left|{\widehat{\delta }}_i^2-{\delta }_i^2\right|\le ϵ/4.$$

For ∀ T_i ≥ T_i1 in the set ${\mathcal{T}}_i$,

$$\mathbb{E}\left(\frac{a_i^2}{{\widehat{\delta }}_i^{{\kappa }_1}{\delta }_i^{{\kappa }_2}}\right)\le 2\left(\mathbb{E}{\widehat{\delta }}_i^2+{\delta }_i^2\right)\le 2\left(\mathbb{E}\left|{\widehat{\delta }}_i^2-{\delta }_i^2\right|+2{\delta }_i^2\right)\le 2\left(\frac{ϵ}{4}+2\times \frac{ϵ}{8}\right)=ϵ.$$

Consider the complementary of set ${\mathcal{T}}_i$, since $\mathbb{E}{a}_i^2\to 0$, there exists an integer T_i2 such that, ∀ T_i ≥ T_i2,

$$\mathbb{E}{a}^2\le \frac{{ϵ}^{{\kappa }_1+{\kappa }_2+1}}{2^{4{\kappa }_1+3{\kappa }_2+1}}.$$

${\delta }_i^2$ is bounded by $2C_2+C_1\sqrt{C_2}$. Then, there exists an integer T_i3 such that, for ∀ Ti ≥ Ti3

$$\mathbb{P}\left(\left|{\widehat{\delta }}_i^2-{\delta }_i^2\right|\ge \frac{ϵ}{16}\right)\le \frac{4ϵ}{16\left(2C_2+C_1\sqrt{C_2}\right)+ϵ}.$$

Let 1_{·} denote the indicator function. For ∀ T_i ≥ max(T_i2, T_i3) outside the set ${\mathcal{T}}_i$, then

$$\begin{gathered} \begin{array}{l}\mathbb{E}\left(\frac{a_i^2}{{\widehat{\delta }}_i^{{\kappa }_1}{\delta }_i^{{\kappa }_2}}\right) \\ =\mathbb{E}\left(\frac{a_i^2}{{\widehat{\delta }}_i^{{\kappa }_1}{\delta }_i^{{\kappa }_2}}{1}_{\left\{{\widehat{\delta }}_i^2\le ϵ/16\right\}}\right)+\mathbb{E}\left(\frac{a_i^2}{{\widehat{\delta }}_i^{{\kappa }_1}{\delta }_i^{{\kappa }_2}}{1}_{\left\{{\widehat{\delta }}_i^2>ϵ/16\right\}}\right) \\ \le \mathbb{E}\left\{2\left({\widehat{\delta }}_i^2+{\delta }_i^2\right)1_{\left\{{\widehat{\delta }}_i^2\le ϵ/16\right\}}\right\}+{\left(\frac{16}{ϵ}\right)}^{{\kappa }_1}{\left(\frac{8}{ϵ}\right)}^{{\kappa }_2}\mathbb{E}\left(a_i^2{1}_{\left\{{\widehat{\delta }}_i^2>ϵ/16\right\}}\right) \\ \le 2\left\{\left(2C_2+C_1\sqrt{C_2}\right)+\frac{ϵ}{16}\right\}\mathbb{P}\left(\left|{\widehat{\delta }}_i^2-{\delta }_i^2\right|\ge \frac{ϵ}{16}\right)+{\left(\frac{16}{ϵ}\right)}^{{\kappa }_1}{\left(\frac{8}{ϵ}\right)}^{{\kappa }_2}\mathbb{E}\left(a_i^2\right) \\ \le 2\left\{\left(2C_2+C_1\sqrt{C_2}\right)+\frac{ϵ}{16}\right\}\frac{4ϵ}{16\left(2C_2+C_1\sqrt{C_2}\right)+ϵ}+{\left(\frac{16}{ϵ}\right)}^{{\kappa }_1}{\left(\frac{8}{ϵ}\right)}^{{\kappa }_2}\frac{{ϵ}^{{\kappa }_1+{\kappa }_2+1}}{2^{4{\kappa }_1+3{\kappa }_2+1}}\\ \le ϵ.\end{array} \end{gathered}$$

Bringing together the results inside and outside the set ${\mathcal{T}}_i$, for ∀ T_i ≥ max(T_i1, T_i2, T_i3),

$$\mathbb{E}\left(\frac{a_i^2}{{\widehat{\delta }}_i^{{\kappa }_1}{\delta }_i^{{\kappa }_2}}\right)\le ϵ.$$

The proof of the second part follows the same strategy. □

Now, we prove Theorem 3.1.

Proof. We first prove that $S_i^{*}$ is a consistent estimator of ${\Sigma }_i^{*}$.

$$\begin{gathered} {\Vert S_i^{*}-{\Sigma }_i^{*}\Vert }^2=\underset{\mathit{\gamma }\ne 0}{\text{max}}\frac{{\Vert {\mathit{\gamma }}^{\top }\left(S_i^{*}-{\Sigma }_i^{*}\right)\mathit{\gamma }\Vert }^2}{{\mathit{\gamma }}^{\top }\mathit{\gamma }} \\ =\underset{\mathit{\gamma }\ne 0}{\text{max}}\frac{1}{{\mathit{\gamma }}^{\top }\mathit{\gamma }}{\Vert \left(\frac{{\widehat{\varphi }}^2}{{\widehat{\delta }}^2}-\frac{{\varphi }^2}{{\delta }^2}\right)\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }-\mu {\mathit{\gamma }}^{\top }\mathit{\gamma }\right)\Vert }^2 \\ =\underset{\mathit{\gamma }\ne 0}{\text{max}}\frac{1}{{\mathit{\gamma }}^{\top }\mathit{\gamma }}{\left(\frac{{\widehat{\varphi }}^2}{{\widehat{\delta }}^2}-\frac{{\varphi }^2}{{\delta }^2}\right)}^2{\widehat{\delta }}_i^2. \end{gathered}$$

$$\begin{gathered} \frac{1}{n}\sum _{i=1}^n{\Vert S_i^{*}-{\Sigma }_i^{*}\Vert }^2=\underset{\mathit{\gamma }\ne 0}{\text{max}}\frac{1}{{\mathit{\gamma }}^{\top }\mathit{\gamma }}\frac{{\left({\widehat{\varphi }}^2{\delta }^2-{\varphi }^2{\widehat{\delta }}^2\right)}^2}{{\widehat{\delta }}^4{\delta }^4}\frac{1}{n}\sum _{i=1}^n{\widehat{\delta }}_i^2 \\ =\underset{\mathit{\gamma }\ne 0}{\text{max}}\frac{1}{{\mathit{\gamma }}^{\top }\mathit{\gamma }}\frac{{\left({\widehat{\varphi }}^2{\delta }^2-{\varphi }^2{\widehat{\delta }}^2\right)}^2}{{\widehat{\delta }}^2{\delta }^4}. \end{gathered}$$

Using the fact that ϕ² ≤ δ² and ${\widehat{\varphi }}^2\le {\widehat{\delta }}^2$,

$$\frac{{\left({\widehat{\varphi }}^2{\delta }^2-{\varphi }^2{\widehat{\delta }}^2\right)}^2}{{\widehat{\delta }}^2{\delta }^4}\le {\widehat{\delta }}^2\le 2\left({\widehat{\delta }}^2+{\delta }^2\right).$$

In Lemma 3.2, it is shown that $\mathbb{E}{\left({\widehat{\varphi }}^2-{\varphi }^2\right)}^2$ and $\mathbb{E}{\left({\widehat{\delta }}^2-{\delta }^2\right)}^2$ converge to zero. In addition, Lemma 3.1 shows that ϕ² and δ² are bounded. Thus,

$$\begin{gathered} \mathbb{E}{\left({\widehat{\varphi }}^2{\delta }^2-{\varphi }^2{\widehat{\delta }}^2\right)}^2=\mathbb{E}{\left\{\left({\widehat{\varphi }}^2-{\varphi }^2\right){\delta }^2-{\varphi }^2\left({\widehat{\delta }}^2-{\delta }^2\right)\right\}}^2 \\ \le {\delta }^4\mathbb{E}{\left({\widehat{\varphi }}^2-{\varphi }^2\right)}^2+{\varphi }^4\mathbb{E}{\left({\widehat{\delta }}^2-{\delta }^2\right)}^2 \\ \to 0. \end{gathered}$$

Let $a^2={\left({\widehat{\varphi }}^2{\delta }^2-{\varphi }^2{\widehat{\delta }}^2\right)}^2$, κ₁ = 2 and κ₂ = 4, then $\mathbb{E}{a}^2\to 0$, and using Lemma A.1,

$$\mathbb{E}\frac{{\left({\widehat{\varphi }}^2{\delta }^2-{\varphi }^2{\widehat{\delta }}^2\right)}^2}{{\widehat{\delta }}^2{\delta }^4}\to 0.$$

Thus,

$$\frac{1}{n}\sum _{i=1}^n\mathbb{E}{\Vert S_i^{*}-{\Sigma }_i^{*}\Vert }^2\to 0.$$

And therefore, for ∀ i,

$$\mathbb{E}{\Vert S_i^{*}-{\Sigma }_i^{*}\Vert }^2\to 0.$$

For the second statement,

$$\begin{gathered} \mathbb{E}\left|{\Vert S_i^{*}-{\Sigma }_i\Vert }^2-{\Vert {\Sigma }_i^{*}-{\Sigma }_i\Vert }^2\right|=\mathbb{E}\left|\langle S_i^{*}-{\Sigma }_i^{*},S_i^{*}+{\Sigma }_i^{*}-2{\Sigma }_i\rangle \right| \\ \le \sqrt{\mathbb{E}{\Vert S_i^{*}-{\Sigma }_i^{*}\Vert }^2}\sqrt{\mathbb{E}{\Vert S_i^{*}+{\Sigma }_i^{*}-2{\Sigma }_i\Vert }^2} \\ \to 0. \end{gathered}$$

Therefor,

$$\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{S}_i^{*}\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2-\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{\Sigma }_i^{*}\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2\to 0.$$

□

A.6. Proof of Theorem 3.2

Before proving Theorem 3.2, we first provide the solution to the optimization problem (3.4). Let

$$f\left({\rho }_1,{\rho }_2\right)=\frac{1}{n}\sum _{i=1}^n{\left\{{\mathit{\gamma }}^{\top }\left({\rho }_{1}I+{\rho }_2{S}_i\right)\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2.$$

$$\frac{\partial f}{\partial {\rho }_1}=\frac{1}{n}\sum _{i=1}^{n}2\left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)\left\{{\rho }_1{\mathit{\gamma }}^{\top }\mathit{\gamma }+{\rho }_2{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}=0$$

$$\frac{\partial f}{\partial {\rho }_2}=\frac{1}{n}\sum _{i=1}^{n}2\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)\left\{{\rho }_1\left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)+{\rho }_2\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)-\text{exp}\left(x_i^{\top }\beta \right)\right\}=0.$$

$$\Rightarrow {\rho }_2=\frac{{\sum }_i\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)\text{exp}\left(x_i^{\top }\beta \right)/n-\left(\sum _i{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }/n\right)\left(\sum _i\text{exp}\left(x_i^{\top }\beta \right)/n\right)}{\sum _i{\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)}^2/n-{\left(\sum _i{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }/n\right)}^2}$$

$$\begin{gathered} {\rho }_1=\frac{1}{{\mathit{\gamma }}^{\top }\mathit{\gamma }}\left\{\frac{1}{n}\sum _{i=1}^n\text{exp}\left(x_i^{\top }\beta \right)-\frac{1}{n}\sum _{i=1}^n{\rho }_2\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)\right\} \\ =\frac{1}{{\mathit{\gamma }}^{\top }\mathit{\gamma }}\left\{\frac{\left({\sum }_i{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }/n\right)\left({\sum }_i\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)\text{exp}\left(x_i^{\top }\beta \right)/n\right)}{{\sum }_i{\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)}^2/n-{\left({\sum }_i{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }/n\right)}^2}-\frac{\left({\sum }_i\text{exp}\left(x_i^{\top }\beta \right)/n\right)\left({\sum }_i{\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)}^2/n\right)}{{\sum }_i{\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)}^2/n-{\left({\sum }_i{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }/n\right)}^2}\right\}. \end{gathered}$$

In order to prove Theorem 3.2, the following lemma is introduced.

Lemma A.2. For given (γ, β), let T_min = min_i T_i, as T_min → ∞, for ∀ i ∈ {1, …, n},

$$\mathbb{E}\left(\left|\frac{{\widehat{\varphi }}_i^2{\widehat{\psi }}_i^2}{{\widehat{\delta }}_i^2}-\frac{{\varphi }_i^2{\psi }_i^2}{{\delta }_i^2}\right|\right)\to 0.$$

Then, as n, T_min → ∞,

$$\mathbb{E}\left(\left|\frac{{\widehat{\varphi }}^2{\widehat{\psi }}^2}{{\widehat{\delta }}^2}-\frac{{\varphi }^2{\psi }^2}{{\delta }^2}\right|\right)\to 0.$$

Proof.

$$\frac{{\widehat{\varphi }}_i^2{\widehat{\psi }}_i^2}{{\widehat{\delta }}_i^2}-\frac{{\varphi }_i^2{\psi }_i^2}{{\delta }_i^2}=\frac{{\widehat{\varphi }}_i^2{\widehat{\psi }}_i^2{\delta }_i^2-{\varphi }_i^2{\psi }_i^2{\widehat{\delta }}_i^2}{{\widehat{\delta }}_i^2{\delta }_i^2}.$$

Let $a_i^2=\left|{\widehat{\varphi }}_i^2{\widehat{\psi }}_i^2{\delta }_i^2-{\varphi }_i^2{\psi }_i^2{\widehat{\delta }}_i^2\right|$, κ₁ = 2 and κ₂ = 2. First need to verify the assumptions in Lemma A.1.

$$\left|\frac{{\widehat{\varphi }}_i^2{\widehat{\psi }}_i^2}{{\widehat{\delta }}_i^2}-\frac{{\varphi }_i^2{\psi }_i^2}{{\delta }_i^2}\right|\le \frac{{\widehat{\varphi }}_i^2{\widehat{\psi }}_i^2}{{\widehat{\delta }}_i^2}+\frac{{\varphi }_i^2{\psi }_i^2}{{\delta }_i^2}\le {\widehat{\varphi }}_i^2+{\varphi }_i^2\le {\widehat{\delta }}_i^2+{\delta }_i^2\le 2\left({\widehat{\delta }}_i^2+{\delta }_i^2\right),\text{\hspace{1em} a.s}\mathrm{..}$$

Furthermore,

$$\begin{gathered} \mathbb{E}\left(\left|{\widehat{\varphi }}_i^2{\widehat{\psi }}_i^2{\delta }_i^2-{\varphi }_i^2{\psi }_i^2{\widehat{\delta }}_i^2\right|\right) \\ =\mathbb{E}\left\{\left|\left({\widehat{\varphi }}_i^2{\widehat{\psi }}_i^2-{\varphi }_i^2{\psi }_i^2\right){\delta }_i^2-{\varphi }_i^2{\psi }_i^2\left({\widehat{\delta }}_i^2-{\delta }_i^2\right)\right|\right\} \\ =\mathbb{E}\left\{\left|\left({\widehat{\varphi }}_i^2-{\varphi }_i^2\right)\left({\widehat{\psi }}_i^2-{\psi }_i^2\right){\delta }_i^2+{\varphi }_i^2\left({\widehat{\psi }}_i^2-{\psi }_i^2\right){\delta }_i^2+\left({\widehat{\varphi }}_i^2-{\varphi }_i^2\right){\psi }_i^2{\delta }_i^2-{\varphi }_i^2{\psi }_i^2\left({\widehat{\delta }}_i^2-{\delta }_i^2\right)\right|\right\} \\ \le \sqrt{\mathbb{E}{\left({\widehat{\varphi }}_i^2-{\varphi }_i^2\right)}^2}\sqrt{\mathbb{E}{\left({\widehat{\psi }}_i^2-{\psi }_i^2\right)}^2}{\delta }_i^2+{\varphi }_i^2\mathbb{E}\left|{\widehat{\psi }}_i^2-{\psi }_i^2\right|{\delta }_i^2+\mathbb{E}\left|{\widehat{\varphi }}_i^2-{\varphi }_i^2\right|{\psi }_i^2{\delta }_i^2-{\varphi }_i^2{\psi }_i^2\mathbb{E}\left|{\widehat{\delta }}_i^2-{\delta }_i^2\right|. \end{gathered}$$

The right-hand side converges to zero. Therefore, $\mathbb{E}{a}_i^2\to 0$, conditions in Lemma A.1 are satisfied. Therefore,

$$\mathbb{E}\left|\frac{{\widehat{\varphi }}_i^2{\widehat{\psi }}_i^2}{{\widehat{\delta }}_i^2}-\frac{{\varphi }_i^2{\psi }_i^2}{{\delta }_i^2}\right|\to 0.$$

Analogously, it can be shown that

$$\mathbb{E}\left|\frac{{\widehat{\varphi }}^2{\widehat{\psi }}^2}{{\widehat{\delta }}^2}-\frac{{\varphi }^2{\psi }^2}{{\delta }^2}\right|\to 0.$$

□

Next, we prove Theorem

Proof. Let α_i = (γ^⊤Σ_iγ)(γ^⊤S_iγ) − {μ(γ^⊤γ)}² and $\alpha ={\sum }_{i=1}^n{\alpha }_i/n$. $\mathbb{E}\left({\alpha }_i\right)={\text{exp}}^2\left(x_i^{\top }\beta \right)-{\mu }^2{\left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)}^2$, then

$$\mathbb{E}\alpha =\frac{1}{n}\sum _{i=1}^n{\text{exp}}^2\left(x_i^{\top }\beta \right)-{\mu }^2\left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)={\varphi }^2.$$

First, need to prove that α − ϕ² converges to zero in quadratic mean.

$$\begin{gathered} \text{Var}\left({\alpha }_i\right) \\ =\text{Var}\left\{\left({\mathit{\gamma }}^{\top }{\Sigma }_i\mathit{\gamma }\right)\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)-{\mu }^2{\left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)}^2\right\} \\ =\text{Var}\left\{\left({\mathit{\gamma }}^{\top }{\Sigma }_i\mathit{\gamma }\right)\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)\right\}+\text{Var}\left\{{\mu }^2{\left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)}^2\right\}-2\text{Cov}\left\{\left({\mathit{\gamma }}^{\top }{\Sigma }_i\mathit{\gamma }\right)\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right),{\mu }^2{\left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)}^2\right\} \\ =\text{Var}\left\{\left({\mathit{\gamma }}^{\top }{\Sigma }_i\mathit{\gamma }\right)\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)\right\}. \end{gathered}$$

$$\left({\mathit{\gamma }}^{\top }{\Sigma }_i\mathit{\gamma }\right)\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)={\lambda }_{i{j}_i}\left(\frac{1}{T_i}\sum _{t=1}^{T_i}{z}_{it{j}_i}^2\right).$$

$$\begin{gathered} \text{Var}\left\{\left({\mathit{\gamma }}^{\top }{\Sigma }_i\mathit{\gamma }\right)\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)\right\}=\text{Var}\left\{\frac{1}{T_i}\sum _{t=1}^{T_i}{\lambda }_{i{j}_i}{z}_{it{j}_i}^2\right\} \\ =\frac{1}{T_i}\text{Var}\left({\lambda }_{i{j}_i}{z}_{i1j_i}^2\right) \\ \le \frac{1}{T_i}\mathbb{E}{\left({\lambda }_{i{j}_i}{z}_{i1j_i}^2\right)}^2 \\ \le \frac{1}{T_i}\mathbb{E}{\lambda }_{i{j}_i}^2{z}_{i1j_i}^4 \\ \le \frac{1}{T_i}{\left(\mathbb{E}{z}_{i1j_i}^2\right)}^2\mathbb{E}{z}_{i1j_i}^4 \\ \le \frac{1}{T_i}{\left(\mathbb{E}{z}_{i1j_i}^4\right)}^2 \\ \le \frac{1}{T_i}\mathbb{E}{z}_{i1j_i}^8 \\ \le \frac{C_2}{T_i}. \end{gathered}$$

$$\text{Var}(\alpha )=\frac{1}{n^2}\sum _{i=1}^n\text{Var}\left({\alpha }_i\right)\le \frac{C_2}{n^2}\sum _{i=1}^n\frac{1}{T_i}\to 0,\text{ as }{T}_{\text{min}}=\underset{i}{\text{min }}{T}_i\to \infty .$$

This proves that α − ϕ² converges to 0 in quadratic mean. In the following, we prove that $S_i^{*}$ is a consistent estimator of ${\Sigma }_i^{**}$.

$$\begin{gathered} S_i^{*}=\frac{{\widehat{\psi }}^2}{{\widehat{\delta }}^2}\mu I+\frac{{\widehat{\varphi }}^2}{{\widehat{\delta }}^2}{S}_i=\frac{{\widehat{\delta }}^2-{\widehat{\psi }}^2}{{\widehat{\delta }}^2}\mu I+\frac{{\widehat{\varphi }}^2}{{\widehat{\delta }}^2}{S}_i=\mu I+\frac{{\widehat{\varphi }}^2}{{\widehat{\delta }}^2}\left(S_i-\mu I\right). \\ {\Sigma }_i^{**}={\rho }_{1}I+{\rho }_2{S}_i=\left({\rho }_1+{\rho }_2\mu \right)I+{\rho }_2\left(S_i-\mu I\right). \end{gathered}$$

$$\begin{gathered} \begin{array}{l}\frac{1}{n}\sum _{i=1}^n{\Vert S_i^{*}-{\Sigma }_i^{**}\Vert }^2 \\ =\frac{1}{n}\sum _{i=1}^n{\Vert \left(\mu -{\rho }_1-{\rho }_2\mu \right)I+\left(\frac{{\widehat{\varphi }}^2}{{\widehat{\delta }}^2}-{\rho }_2\right)\left(S_i-\mu I\right)\Vert }^2 \\ =\frac{1}{n}\sum _{i=1}^n\left\{\underset{\mathit{\gamma }\ne 0}{\text{max}}\frac{1}{{\mathit{\gamma }}^{\top }\mathit{\gamma }}{\Vert \left(\mu -{\rho }_1-{\rho }_2\mu \right)\left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)+\left(\frac{{\widehat{\varphi }}^2}{{\widehat{\delta }}^2}-{\rho }_2\right)\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }-\mu \left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)\right)\Vert }^2\right\} \\ \underset{\mathit{\gamma }\ne 0}{\text{max}}\left\{{\left(\mu -{\rho }_1-{\rho }_2\mu \right)}^2\left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)+\frac{1}{{\mathit{\gamma }}^{\top }\mathit{\gamma }}{\left(\frac{{\widehat{\varphi }}^2}{{\widehat{\delta }}^2}-{\rho }_2\right)}^2{\widehat{\delta }}_i^2+2\left(\mu -{\rho }_1-{\rho }_2\mu \right)\left(\frac{{\widehat{\varphi }}^2}{{\widehat{\delta }}^2}-{\rho }_2\right)\left(\frac{1}{n}\sum _{i=1}^n{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }-\mu \left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)\right)\right\}.\\ {\left(\mu -{\rho }_1-{\rho }_2\mu \right)}^2 \\ =\frac{{\left({\sum }_i{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }/n-{\sum }_i\text{exp}\left(x_i^{\top }\beta \right)/n\right)}^2}{{\left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)}^2{\left\{{\left({\sum }_i{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }/n\right)}^2-{\sum }_i{\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)}^2/n\right\}}^2}.\frac{{\left\{\left({\sum }_i{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }/n\right)\left({\sum }_i\text{exp}\left(x_i^{\top }\beta \right)/n\right)-{\sum }_i\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)\text{exp}\left(x_i^{\top }\beta \right)/n\right\}}^2}{{\left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)}^2{\left\{{\left({\sum }_i{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }/n\right)}^2-{\sum }_i{\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)}^2/n\right\}}^2}\end{array} \end{gathered}$$

$$\begin{gathered} \mathbb{E}{\left\{\frac{1}{n}\sum _i{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }-\frac{1}{n}\sum _i\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2 \\ =\frac{1}{n^2}\sum _i\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2+\frac{1}{n^2}\sum _{i\ne i^{\prime }}\mathbb{E}\left\{{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}\left\{{\mathit{\gamma }}^{\top }{S}_{i^{\prime }}\mathit{\gamma }-\text{exp}\left(x_{i^{\prime }}^{\top }\beta \right)\right\}. \end{gathered}$$

$$\begin{gathered} \mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2=\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }-\mathbb{E}\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)\right\}}^2 \\ =\text{Var}\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right) \\ =\frac{1}{T_i}\mathbb{E}{z}_{i1j_i}^4+\frac{T_i\left(T_i-1\right)}{T_i^2}{\left({\mathit{\gamma }}^{\top }{\Sigma }_i\mathit{\gamma }\right)}^2-{\left({\mathit{\gamma }}^{\top }{\Sigma }_i\mathit{\gamma }\right)}^2 \\ \stackrel{T_i\to \infty }{\to }0. \end{gathered}$$

It is assumed that the samples/subjects are independent, therefore,

$$\mathbb{E}\left\{{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}\left\{{\mathit{\gamma }}^{\top }{S}_{i^{\prime }}\mathit{\gamma }-\text{exp}\left(x_{i^{\prime }}^{\top }\beta \right)\right\}=0.$$

Thus,

$$\mathbb{E}{\left\{\frac{1}{n}\sum _i{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }-\frac{1}{n}\sum _i\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2\to 0,\text{ as }{T}_{\text{min}}\to \infty .$$

$$\begin{gathered} \mathbb{E}{\left\{\left(\frac{1}{n}\sum _i{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)\left(\frac{1}{n}\sum _i\text{exp}\left(x_i^{\top }\beta \right)\right)-\frac{1}{n}\sum _i\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2 \\ \le \mathbb{E}{\left(\frac{1}{n}\sum _i{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)}^2{\left(\frac{1}{n}\sum _i\text{exp}\left(x_i^{\top }\beta \right)\right)}^2+\mathbb{E}{\left\{\frac{1}{n}\sum _i\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2. \end{gathered}$$

$$\begin{gathered} \mathbb{E}{\left(\frac{1}{n}\sum _i{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)}^2 \\ =\frac{1}{n^2}\sum _i\mathbb{E}{\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)}^2+\frac{1}{n^2}\sum _{i\ne i^{\prime }}\mathbb{E}\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)\left({\mathit{\gamma }}^{\top }{S}_{i^{\prime }}\mathit{\gamma }\right) \\ =\frac{1}{n^2}\sum _i\left\{\frac{1}{T_i^2}\mathbb{E}{z}_{it{j}_i}^4+\frac{1}{T_i^2}\sum _{t\ne s}\mathbb{E}{z}_{it{j}_i}^2{z}_{is{j}_i}^2\right\}+\frac{1}{n^2}\sum _{i\ne i^{\prime }}\left(\frac{1}{T_i^2}\sum _{t=1}^{T_i}\mathbb{E}{z}_{it{j}_i}^2\right)\left(\frac{1}{T_{i^{\prime }}^2}\sum _{t=1}^{T_{i^{\prime }}}\mathbb{E}{z}_{i^{\prime }t{j}_{i^{\prime }}}^2\right) \\ =\frac{1}{n^2}\sum _i\left\{\frac{1}{T_i}\mathbb{E}{z}_{i1j_i}^4+\frac{T_i\left(T_i-1\right)}{T_i^2}{\left({\mathit{\gamma }}^{\top }{\Sigma }_i\mathit{\gamma }\right)}^2\right\}+\frac{1}{n^2}\sum _{i\ne i^{\prime }}\left(\frac{1}{T_i}\left({\mathit{\gamma }}^{\top }{\Sigma }_i\mathit{\gamma }\right)\right)\left(\frac{1}{T_{i^{\prime }}}\left({\mathit{\gamma }}^{\top }{\Sigma }_{i^{\prime }}\mathit{\gamma }\right)\right) \\ \stackrel{T_{\text{min}}\to \infty }{\to }\frac{1}{n^2}\sum _i{\left({\mathit{\gamma }}^{\top }{\Sigma }_i\mathit{\gamma }\right)}^2. \end{gathered}$$

$$\begin{gathered} \mathbb{E}{\left\{\frac{1}{n}\sum _i\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2 \\ =\frac{1}{n^2}\sum _i\sum _i\mathbb{E}{\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)}^2{\text{exp}}^2\left(x_i^{\top }\beta \right)+\frac{1}{n^2}\sum _{i\ne i^{\prime }}\mathbb{E}\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)\text{exp}\left(x_i^{\top }\beta \right)\mathbb{E}\left({\mathit{\gamma }}^{\top }{S}_{i^{\prime }}\mathit{\gamma }\right)\text{exp}\left(x_{i^{\prime }}^{\top }\beta \right) \\ =\frac{1}{n^2}\sum _i\left\{\frac{1}{T_i}\mathbb{E}{z}_{it{j}_i}^4+\frac{T_i\left(T_i-1\right)}{T_i^2}{\left({\mathit{\gamma }}^{\top }{\Sigma }_i\mathit{\gamma }\right)}^2\right\}{\left({\mathit{\gamma }}^{\top }{\Sigma }_i\mathit{\gamma }\right)}^2+\frac{1}{n^2}\sum _{i\ne i^{\prime }}{\left({\mathit{\gamma }}^{\top }{\Sigma }_i\mathit{\gamma }\right)}^2{\left({\mathit{\gamma }}^{\top }{\Sigma }_{i^{\prime }}\mathit{\gamma }\right)}^2 \\ \stackrel{T_{\text{min}}\to \infty }{\to }\frac{1}{n^2}\sum _i{\left({\mathit{\gamma }}^{\top }{\Sigma }_i\mathit{\gamma }\right)}^4+\frac{1}{n^2}\sum _{i\ne i^{\prime }}{\left({\mathit{\gamma }}^{\top }{\Sigma }_i\mathit{\gamma }\right)}^2{\left({\mathit{\gamma }}^{\top }{\Sigma }_{i^{\prime }}\mathit{\gamma }\right)}^2. \end{gathered}$$

$$\begin{gathered} \mathbb{E}{\left\{\left(\frac{1}{n}\sum _i{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)\left(\frac{1}{n}\sum _i\text{exp}\left(x_i^{\top }\beta \right)\right)-\frac{1}{n}\sum _i\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2 \\ \le \mathbb{E}{\left(\frac{1}{n}\sum _i{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)}^2{\left(\frac{1}{n}\sum _i\text{exp}\left({\text{x}}_i^{\top }\beta \right)\right)}^2+\mathbb{E}{\left\{\frac{1}{n}\sum _i\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2 \\ \stackrel{T_{\text{min}}\to \infty }{\to }\frac{1}{n^2}\sum _i{\left({\mathit{\gamma }}^{\top }{\Sigma }_i\mathit{\gamma }\right)}^2{\left(\frac{1}{n}\sum _i\left({\mathit{\gamma }}^{\top }{\Sigma }_i\mathit{\gamma }\right)\right)}^2+\frac{1}{n^2}\sum _i{\left({\mathit{\gamma }}^{\top }{\Sigma }_i\mathit{\gamma }\right)}^4+\frac{1}{n^2}\sum _{i\ne i^{\prime }}{\left({\mathit{\gamma }}^{\top }{\Sigma }_i\mathit{\gamma }\right)}^2{\left({\mathit{\gamma }}^{\top }{\Sigma }_{i^{\prime }}\mathit{\gamma }\right)}^2. \end{gathered}$$

The above quantity on the right is bounded by a constant from above. Therefore, as T_min → ∞,

$${\left(\mu -{\rho }_1-{\rho }_2\mu \right)}^2\to 0.$$

$${\left(\frac{{\widehat{\varphi }}^2}{{\widehat{\delta }}^2}-{\rho }_2\right)}^2={\left(\frac{{\widehat{\varphi }}^2}{{\widehat{\delta }}^2}-\frac{{\varphi }^2}{{\widehat{\delta }}^2}\right)}^2+{\left(\frac{{\varphi }^2}{{\widehat{\delta }}^2}-\frac{\alpha }{{\widehat{\delta }}^2}\right)}^2+{\left(\frac{\alpha }{{\widehat{\delta }}^2}-{\rho }_2\right)}^2.$$

Since ${\widehat{\delta }}^4$ is bounded,

$$\mathbb{E}{\left({\widehat{\varphi }}^2-{\varphi }^2\right)}^2\to 0\Rightarrow \mathbb{E}{\left(\frac{{\widehat{\varphi }}^2}{{\widehat{\delta }}^2}-\frac{{\varphi }^2}{{\widehat{\delta }}^2}\right)}^2\to 0.$$

$$\mathbb{E}{\left({\varphi }^2-\alpha \right)}^2\to 0\Rightarrow \mathbb{E}{\left(\frac{{\varphi }^2}{{\widehat{\delta }}^2}-\frac{\alpha }{{\widehat{\delta }}^2}\right)}^2\to 0.$$

Let ${\rho }_2={\rho }_2^{(1)}/{\rho }_2^{(2)}$, where

$$\begin{gathered} {\rho }_2^{(1)}=\frac{1}{n}\sum _i\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)\text{exp}\left({\text{x}}_i^{\top }\beta \right)-\left(\frac{1}{n}\sum _i{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)\left(\frac{1}{n}\sum _i\text{exp}\left({\text{x}}_i^{\top }\beta \right)\right), \\ {\rho }_2^{(2)}=\frac{1}{n}\sum _i{\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)}^2-{\left(\frac{1}{n}\sum _i{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)}^2. \end{gathered}$$

$$\begin{gathered} \mathbb{E}{\left(\alpha -{\rho }_2^{(1)}\right)}^2 \\ ={\left(\frac{1}{n}\sum _i\text{exp}\left(x_i^{\top }\beta \right)\right)}^2\mathbb{E}{\left\{\frac{1}{n}\sum _i\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)-\frac{1}{n}\sum _i\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2 \\ \to 0. \end{gathered}$$

$$\begin{gathered} {\widehat{\delta }}^2 \\ =\frac{1}{n}\sum _{i=1}^n{\left\{{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }-\mu \left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)\right\}}^2 \\ =\frac{1}{n}\sum _{i=1}^n{\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)}^2-2\left(\frac{1}{n}\sum _{i=1}^n{\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)\left(\frac{1}{n}\sum _{i=1}^n\text{exp}\left(x_i^{\top }\beta \right)\right)+{\left(\frac{1}{n}\sum _{i=1}^n\text{exp}\left(x_i^{\top }\beta \right)\right)}^2. \end{gathered}$$

It can be concluded that as T_min → ∞,

$$\mathbb{E}{\left(\widehat{\delta }-{\rho }_2^2\right)}^2=\mathbb{E}{\left\{\frac{1}{n}\sum _{i=1}^n\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right)-\frac{1}{n}\sum _{i=1}^n\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2\to 0,$$

and

$$\begin{gathered} \mathbb{E}{\left(\frac{{\widehat{\varphi }}^2}{{\widehat{\delta }}^2}-{\rho }_2\right)}^2\to 0. \\ \mathbb{E}\left\{\frac{1}{n}\sum _{i=1}^n{\Vert S_i^{*}-{\Sigma }_i^{**}\Vert }^2\right\}\to 0,\Rightarrow \mathbb{E}{\Vert S_i^{*}-{\Sigma }_i^{**}\Vert }^2\to 0. \end{gathered}$$

This implies that

$$\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{S}_i^{*}\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2-\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{\Sigma }_i^{**}\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2\to 0.$$

A.7. Proof of Theorem 3.3

Proof. For the first statement,

$$\begin{gathered} \underset{T_{\text{min}}\to \infty }{\text{lim}}\underset{T_i\ge T_{\text{min}}}{\text{inf}}\left[\frac{1}{n}\sum _{i=1}^n\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{\widehat{\Sigma }}_i\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2-\frac{1}{n}\sum _{i=1}^n\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{S}_i^{*}\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2\right] \\ \ge \text{inf}\left[\frac{1}{n}\sum _{i=1}^n\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{\widehat{\Sigma }}_i\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2-\frac{1}{n}\sum _{i=1}^n\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{\Sigma }_i^{**}\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2\right] \\ +\text{lim}\left[\frac{1}{n}\sum _{i=1}^n\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{\Sigma }_i^{**}\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2-\frac{1}{n}\sum _{i=1}^n\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{S}_i^{*}\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2\right]. \end{gathered}$$

By Theorem 3.2, the second term on the right converges to zero, and the first term is ≥ 0 by the definition of ${\Sigma }_i^{**}$.

For the second statement,

$$\begin{gathered} \underset{T_{\text{min}}\to \infty }{\text{lim}}\left[\frac{1}{n}\sum _{i=1}^n\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{\widehat{\Sigma }}_i\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2-\frac{1}{n}\sum _{i=1}^n\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{S}_i^{*}\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2\right]=0 \\ \iff \underset{T_{\text{min}}\to \infty }{\text{lim}}\left[\frac{1}{n}\sum _{i=1}^n\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{\widehat{\Sigma }}_i\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2-\frac{1}{n}\sum _{i=1}^n\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{\Sigma }_i^{**}\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2\right]=0 \\ \iff \underset{T_{\text{min}}\to \infty }{\text{lim}}\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{\widehat{\Sigma }}_i\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2-\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }{\Sigma }_i^{**}\mathit{\gamma }-\text{exp}\left(x_i^{\top }\beta \right)\right\}}^2=0 \\ \iff \underset{T_{\text{min}}\to \infty }{\text{lim}}\mathbb{E}{\Vert {\mathit{\gamma }}^{\top }{\widehat{\Sigma }}_i\mathit{\gamma }-{\mathit{\gamma }}^{\top }{\Sigma }_i^{**}\mathit{\gamma }\Vert }^2=0 \\ \iff \underset{T_{\text{min}}\to \infty }{\text{lim}}\mathbb{E}{\Vert {\mathit{\gamma }}^{\top }{\widehat{\Sigma }}_i\mathit{\gamma }-{\mathit{\gamma }}^{\top }{S}_i^{*}\mathit{\gamma }\Vert }^2=0 \\ \iff \underset{T_{\text{min}}\to \infty }{\text{lim}}\mathbb{E}{\Vert {\widehat{\Sigma }}_i-S_i^{*}\Vert }^2=0. \end{gathered}$$

This finishes the proof of this theorem.□

A.8. $S_i^{*}$ is well-conditioned

In this section, we show that the proposed estimator $S_i^{*}$ is well-conditioned and thus, invertible. This is achieved by two steps: for i = 1, …, n, (1) prove that the largest eigenvalue of $S_i^{*}$ is bounded in probability; (2) prove that the smallest eigenvalue of $S_i^{*}$ is bounded away from zero in probability. The proof follows the same strategy as in Ledoit and Wolf [24], but considers the case with multiple covariance matrices.

The covariance matrix Σ_i has the eigendecomposition as ${\Sigma }_i={\Pi }_i{\Lambda }_i{\Pi }_i^{\top }$. Let $U_i={\Lambda }_i^{-1/2}{Y}_i$. Denote λ_max(A) and λ_min(A) as the maximum and minimum eigenvalue of a matrix A, respectively.

$$\begin{gathered} {\lambda }_{\text{max}}\left(S_i^{*}\right)={\lambda }_{\text{max}}\left(\frac{{\widehat{\psi }}^2}{{\widehat{\delta }}^2}\mu I+\frac{{\widehat{\varphi }}^2}{{\widehat{\delta }}^2}{S}_i\right) \\ =\frac{{\widehat{\psi }}^2}{{\widehat{\delta }}^2}\mu +\frac{{\widehat{\varphi }}^2}{{\widehat{\delta }}^2}{\lambda }_{\text{max}}\left(S_i\right). \end{gathered}$$

$$\mu =\frac{1}{n}\sum _{i=1}^n\text{exp}\left(x_i^{\top }\beta \right)=\frac{1}{n}\sum _{i=1}^n{\lambda }_{i{j}_i}\le \underset{i}{\text{max }}{\lambda }_{\text{max}}\left({\Lambda }_i\right).$$

$$\begin{gathered} {\lambda }_{\text{max}}\left(S_i\right)={\lambda }_{\text{max}}\left(\frac{1}{T_i}{\Lambda }^{1/2}{U}_i{U}_i^{\top }{\Lambda }_i^{1/2}\right) \\ \le {\lambda }_{\text{max}}\left(\frac{1}{T_i}{U}_i{U}_i^{\top }\right){\lambda }_{\text{max}}\left({\Lambda }_i\right) \\ \le {\lambda }_{\text{max}}\left(\frac{1}{T_i}{U}_i{U}_i^{\top }\right)\underset{i}{\text{max }}{\lambda }_{\text{max}}\left({\Lambda }_i\right). \end{gathered}$$

Assume that p/T_max converges to a limit, denoted as c. Based on Assumption A1, c ≤ C₁. Based on the results in Yin et al. [39], as T_min = min_i T_i → ∞, for i = 1, …, n,

$$\text{lim\hspace{0.17em}\hspace{0.17em}}{\lambda }_{\text{max}}\left(\frac{1}{T_i}{U}_i{U}_i^{\top }\right)={(1+\sqrt{c})}^2,\text{\hspace{0.17em}\hspace{0.17em}a}.\text{s}.$$

This implies that

$$\mathbb{P}\left\{{\lambda }_{\text{max}}\left(S_i^{*}\right)\le {(1+\sqrt{c})}^2\underset{i}{\text{max}}{\lambda }_{\text{max}}\left({\Lambda }_i\right)\right\}\to 1,$$

and

$$\mathbb{P}\left\{{\lambda }_{\text{max}}\left(S_i^{*}\right)\le {\left(1+\sqrt{C_1}\right)}^2\underset{i}{\text{max}}{\lambda }_{\text{max}}\left({\Lambda }_i\right)\right\}\to 1.$$

Therefore, if p/T_max converges to a constant, the largest eigenvalue of $S_i^{*}$ is bounded in probability. If p/T_max has no limit, under Assumption A1, there exists a subsequence such that p/T_max converges. Along this sequence, the largest eigenvalue of $S_i^{*}$ is bounded in probability. This is true for any converging sequence, and in addition, the upper bound is independent of the particular subsequence. As a result, it holds for the whole sequence.

Next, we show that the smallest eigenvalue of $S_i^{*}$ is bounded away from zero in probability. Analogously, we have

$$\begin{gathered} {\lambda }_{\text{min}}\left(S_i\right)={\lambda }_{\text{min}}\left(\frac{1}{T_i}{\Lambda }^{1/2}{U}_i{U}_i^{\top }{\Lambda }_i^{1/2}\right) \\ \ge {\lambda }_{\text{min}}\left(\frac{1}{T_i}{U}_i{U}_i^{\top }\right){\lambda }_{\text{min}}\left({\Lambda }_i\right) \\ \ge {\lambda }_{\text{min}}\left(\frac{1}{T_i}{U}_i{U}_i^{\top }\right)\underset{i}{\text{min}}{\lambda }_{\text{min}}\left({\Lambda }_i\right). \end{gathered}$$

First, assume p/T_max converges to a constant c. If c ∈ (0, 1), based on the results in Bai and Yin [4],

$$\text{lim\hspace{0.17em}\hspace{0.17em}}{\lambda }_{\text{min}}\left(\frac{1}{T_i}{U}_i{U}_i^{\top }\right)={(1-\sqrt{c})}^2,\text{\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}a}.\text{s}.$$

Assume c ≤ 1 − κ for some κ ∈ (0, 1). One can conclude that

$$\mathbb{P}\left\{{\lambda }_{\text{min}}\left(S_i^{*}\right)\ge {(1-\sqrt{1-\kappa })}^2\underset{i}{\text{min}}{\lambda }_{\text{min}}\left({\Lambda }_i\right)\right\}\to 1.$$

When c > 1 − κ, we propose to identify a lower bound from the following

$${\lambda }_{\text{min}}\left(S_i^{*}\right)={\lambda }_{\text{min}}\left(\frac{{\widehat{\psi }}^2}{{\widehat{\delta }}^2}\mu I+\frac{{\widehat{\varphi }}^2}{{\widehat{\delta }}^2}{S}_i\right)\ge \frac{{\widehat{\psi }}^2}{{\widehat{\delta }}^2}\mu .$$

Compare the right-hand side in the above to it population counterpart,

$$\frac{{\widehat{\psi }}^2}{{\widehat{\delta }}^2}\mu -\frac{{\psi }^2}{{\delta }^2}\mu =\mu \left\{\frac{{\widehat{\psi }}^2-{\psi }^2}{{\delta }^2}+{\widehat{\psi }}^2\left(\frac{1}{{\widehat{\delta }}^2}-\frac{1}{{\delta }^2}\right)\right\}.$$

From Lemmas 3.1 and 3.2, we can show that the above converges to zero in probability. First, consider ${\psi }^2={\sum }_{i=1}^n{\psi }_i^2/n$, where ${\psi }_i^2=\mathbb{E}{\left\{{\mathit{\gamma }}^{\top }\left(S_i-{\Sigma }_i\right)\mathit{\gamma }\right\}}^2$. From the proof of Lemma 3.1,

$$\begin{gathered} \mathbb{E}{\Vert S_i-{\Sigma }_i\Vert }^2=\frac{1}{p{T}_i}\sum _{j=1}^p\sum _{k=1}^p\mathbb{E}\left(z_{i1j}^2{z}_{i1k}^2\right)-\frac{1}{p{T}_i}\sum _{j=1}^p\sum _{k=1}^p{\lambda }_{ijk}^2 \\ =\frac{p}{T_i}\left\{\frac{1}{p^2}\sum _{j=1}^p\sum _{k=1}^p\mathbb{E}\left(z_{i1j}^2{z}_{i1k}^2\right)\right\}-\frac{1}{p{T}_i}\sum _{j=1}^p{\lambda }_{ijj}^2. \end{gathered}$$

As T_min → ∞, the second term on the right-hand side converges to zero. For ϵ > 0, there exists a constant M > 0 such that when T_min > M, ${\sum }_{j=1}^p{\lambda }_{ijj}^2/\left(p{T}_i\right)<ϵ$. Thus, ${\psi }_i^2\ge (1-\kappa )-ϵ$ and ψ² ≥ (1 − κ) − ϵ.

$$\begin{gathered} {\lambda }_{\text{min}}\left(S_i^{*}\right)\ge \frac{{\widehat{\psi }}^2}{{\widehat{\delta }}^2}\mu \\ =\frac{{\psi }^2}{{\delta }^2}\mu +\left(\frac{{\widehat{\psi }}^2}{{\widehat{\delta }}^2}\mu -\frac{{\psi }^2}{{\delta }^2}\mu \right) \\ \ge \frac{{\psi }^2}{{\delta }^2}\mu -ϵ \\ \ge \frac{{\psi }^2}{2C_2+C_1\sqrt{C_2}}-ϵ \\ \ge \frac{(1-\kappa )-ϵ}{2C_2+C_1\sqrt{C_2}}-ϵ. \end{gathered}$$

For a choice of ϵ, we have

$$\mathbb{P}\left\{{\lambda }_{\text{min}}\left(S_i^{*}\right)\ge \frac{1-\kappa }{2\left(2C_2+C_1\sqrt{C_2}\right)}\right\}\to 1.$$

Therefore, for both c ≤ 1 − κ and c > 1 − κ, the smallest eigenvalue of $S_i^{*}$ is bounded away from zero. Analogous to the proof of the largest eigenvalue, for the case that p/T_max does not have a limit, we can also have the conclusion for the whole sequence. Since both the largest and the smallest eigenvalues are bounded, $S_i^{*}$ is well-conditioned and invertible.

A.9. Proof of Lemma 3.3 and Theorem 3.4

We first proof Lemma 3.3.

Proof.

$$\begin{gathered} \mathbb{E}\left({\mathit{\gamma }}^{\top }{\Sigma }_i^{*}\mathit{\gamma }\right)=\frac{{\psi }^2}{{\delta }^2}\mu \left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)+\frac{{\varphi }^2}{{\delta }^2}\mathbb{E}\left({\mathit{\gamma }}^{\top }{S}_i\mathit{\gamma }\right) \\ =\frac{{\psi }^2}{{\delta }^2}\mu \left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)+\frac{{\varphi }^2}{{\delta }^2}\text{exp}\left(x_i^{\top }\beta \right) \\ =\text{exp}\left(x_i^{\top }{\beta }^{*}\right). \end{gathered}$$

$$\frac{{\sum }_i\text{exp}\left(x_i^{\top }{\beta }^{*}\right)/n}{{\sum }_i\text{exp}\left(x_i^{\top }\beta \right)/n}=\frac{{\psi }^2}{{\delta }^2}\frac{\mu \left({\mathit{\gamma }}^{\top }\mathit{\gamma }\right)}{{\sum }_i\text{exp}\left(x_i^{\top }\beta \right)/n}+\frac{{\varphi }^2}{{\delta }^2}=\frac{{\psi }^2}{{\delta }^2}+\frac{{\varphi }^2}{{\delta }^2}=1.$$

$$\Rightarrow \frac{1}{n}\sum _{i=1}^n\text{exp}\left({\text{x}}_i^{\top }{\beta }^{*}\right)=\frac{1}{n}\sum _{i=1}^n\text{exp}\left({\text{x}}_i^{\top }\beta \right).$$

Therefore,

$${\mathit{\beta }}^{*}=\mathit{\beta }.$$

Next, we prove that the proposed estimator β is a consistent estimator (Theorem 3.4).

Proof. Using the consistency of pseudo-likelihood estimator [16] and the conclusion in Lemma 3.3, $\widehat{\beta }$ is a consistent estimator of β. □

Appendix B: Additional Simulation Results

B.1. γ unknown

Here, we present the performance of estimating the fourth dimension (D4) when γ is unknown (Figure B.1). From the figures, as n and T increase, the estimate of the covariance matrices, the projection and the model coefficient converge to the truth.

B.2. Model misspecification

B.2.1. Model misspecification in β

In this section, we examine the performance of the proposed approach when the log-linear model (1.1) is misspecified. We consider the case when the data dimension is p = 20 and sample size n = 100 and T_i = T = 100 for illustration. Two scenarios are considered. In the first scenario, the true model has two correlated covariates generated from a bivariate normal distribution with mean zero, standard deviation 0.5, and correlation 0.2:

$$\text{log}\left({\lambda }_{ij}\right)={\beta }_0+{\beta }_1{x}_{i1}+{\beta }_2{x}_{i2}.$$ (B.1)

In D2, |β₁| = |β₂| and in D4, |β₁| = 2|β₂|. Under the misspecified case, the second covariate, x_i2, is ignored. Table B.1 presents the results using the proposed approach.

Fig B.1.

Estimation performance of PS-CAP in estimating the fourth dimension (D4) when γ is unknown. For ${\widehat{\beta }}_1$, (a) bias, (b) mean squared error (MSE) and (c) coverage probability (CP) are presented, where CP is obtained from 500 bootstrap samples. For the eigenvalues ${\widehat{\lambda }}_{ij}$, (d) MSE is presented. For $\widehat{\lambda }$, (e) similarity to π₄ is presented. Data dimension p = 100. Sample sizes vary from n = 50, 100, 500, 100 and T_i = T = 50, 100, 500, 1000.

The second scenario considers the following log-linear model for the eigenvalues is considered, which includes an interaction between the covariates:

$$\text{log}\left({\lambda }_{ij}\right)={\beta }_0+{\beta }_1{x}_{i1}+{\beta }_2{x}_{i2}+{\beta }_3\left(x_{i1}\times x_{i2}\right),$$ (B.2)

where x_i1 is generated from a Bernoulli distribution with probability 0.5 to be one and x_i2 is generated from a normal distribution with mean zero and standard deviation 0.5. Table B.2 presents the estimation results using the proposed method. Under the misspecified case, the interaction between the two covariates is ignored. Thus, in the table, it has no estimate of β₃.

From both tables, under either correctly specified or misspecified model, the proposed approach correctly identifies the components related to the covariates. Under the misspecified model, the estimate of β is biased.

B.2.2. Model misspecification in γ

In this section, we discuss the robustness of the proposed approach to the violation of the assumption that all the covariance matrices share the same eigenspace. One advantage of the proposed shrinkage estimator of the covariance matrix is that it will not change the eigenvectors compared to the sample covariance matrix. In Section E.6 of the supplementary materials of Zhao et al. [41], the performance under the partial common diagonalization assumption was examined through a simulation study. In the setting, two eigencomponents are set to be the same across subjects and the rest are unique to each subject. The method can correctly identify the common component across subjects that is related to the covariates. As the proposed approach in this study has the property of preserving the eigenstructure, under the setting of partial common diagonalization, it will also correctly identify the common component that is related to the covariates.

Here, we also consider a case that each covariance matrix has a unique eigenspace, that is, the covariance matrices are generated using the eigendecomposition ${\Sigma }_i={\Pi }_i{\Lambda }_i{\Pi }_i^{\top }$, where Π_i is an orthonormal matrix in ${ℝ}^{p\times p}$, for i = 1, …, n. The rest simulation settings are the same as in Section 4.2. Table B.3 presents the results when p = 20, n = 100, and T_i = T = 100. For the estimation of γ, we compare with an average of the eigenvectors (after scaling to unit ℓ₂-norm). For D2, through the correlation between the estimated γ and the average if 0.915, the estimation of β₁ is off. For D4, both the estimate of γ and β₁ are away from the truth. Therefore, an assumption of partial common diagonalization is essential for the proposed framework.

Appendix C: Additional Analysis of the ADNI Study

C.1. Validity of model assumptions

In resting-state fMRI studies, the output data are generally considered normally distributed. For each time course, data are temporally correlated of at most lag two [26]. Thus, we subsample the data to remove the temporal correlation. Figure C.1 presents the normal Q-Q plot and the histogram of the data extracted from one brain region of one subject. From the figure, the marginal distribution is close to normal. Thus, the normality assumption is satisfied.

In Section 3, five assumptions are imposed to achieve estimation consistency of the parameters. By setting C₁ = 2, Assumption A1 is satisfied. In the fMRI dataset, the increase of the total number of observations across subjects (i.e. $N={\sum }_{i=1}^n{T}_i$) can be faster than the number of variables (p). Thus, Assumption A2 can be satisfied. Under the normality assumption, the eighth order moment exits, and Assumptions A3 and A4 are valid. Assumption A5 concerns the population eigenvalues. We cannot easily assess this assumption using sample covariance matrices due to the large bias under the high-dimensional setting [22]. We thus can only provide some empirical examination while noting that the empirical results should be evaluated with caution due to this bias issue. To empirically assess the validity of Assumption A5, we first calculate the average sample covariance matrix and then compare the eigenvectors of the average covariance matrix with the eigenvectors of each individual’s sample covariance matrix. When the correlation of two eigenvectors is greater than 0.5, we say there is a high similarity, allowing variability and bias in sample eigenvectors. About 67% of the eigenvectors have a high similarity across multiple subjects. Since the individual sample covariance matrix is rank-deficient, the eigenvectors are not unique. With about 67% overlapping, the assumption of common eigenstructure is partially satisfied. In addition, as discussed in Section B.2.2, when the eigenstructure is partially common across subjects, it will not impact the identification of the common components that are related to the covariates. The proposed approach identifies three components based on the metric of average deviation from diagonality suggesting that these three components commonly diagonalize the covariance matrices. Here the assumption that the log-linear model is correctly specified is challenging to validate using data alone. The current model is considered based on the domain knowledge and the study interest of AD research.

Table B.1.

Bias and mean squared error (MSE) in estimating β, and the similarity of $\widehat{\lambda }$ to π_j and the standard error (SE), for j = 2, 4, under the misspecified and correctly specified models for model (B.1). Data dimension p = 20, sample size n = 100 and T_i = T = 100.

		${\widehat{\beta }}_1$		${\widehat{\beta }}_2$		$\widehat{\lambda }$
		Bias	MSE	Bias	MSE	$\left|\langle \widehat{\gamma },{\pi }_j\rangle \right|$ (SE)
D2	Misspecified	0.105	0.014	-	-	0.994 (0.003)
	Correctly specified	0.002	0.001	< 0.001	0.001	0.993 (0.003)
D4	Misspecified	−0.081	0.008	-	-	0.991 (0.004)
	Correctly specified	−0.015	0.001	0.008	0.001	0.983 (0.009)

Table B.2.

Bias and mean squared error (MSE) in estimating β, and the similarity of $\widehat{\lambda }$ to π_j and the standard error (SE), for j = 2, 4, under the misspecified and correctly specified models for model (B.2). Data dimension p = 20, sample size n = 100 and T_i = T = 100.

		${\widehat{\beta }}_1$		${\widehat{\beta }}_2$		${\widehat{\beta }}_3$		$\widehat{\lambda }$
		Bias	MSE	Bias	MSE	Bias	MSE	$\left|\langle \widehat{\gamma },{\pi }_j\rangle \right|$ (SE)
D2	Misspecified	< 0.001	0.002	−0.252	0.066	-	-	0.993 (0.003)
	Correctly specified	0.002	0.001	−0.002	0.002	−0.001	0.003	0.993 (0.003)
D4	Misspecified	−0.010	0.001	−0.113	0.014	-	-	0.987 (0.006)
	Correctly specified	−0.010	0.001	0.015	0.002	−0.005	0.003	0.988 (0.006)

Table B.3.

Bias and mean squared error (MSE) in estimating β₁ and the similarity of $\widehat{\lambda }$ to the average of π_ij (denoted as ${\overline{\pi }}_j$) and the standard error (SE), for j = 2, 4, when each covariance matrix has a unique eigenspace. Data dimension p = 20, sample size n = 100 and T_i = T = 100.

	${\widehat{\beta }}_1$			$\widehat{\lambda }$
	Truth	Bias	MSE	$\left|\langle \widehat{\gamma },{\overline{\mathit{\pi }}}_j\rangle \right|$ (SE)
D2	−1	0.980	0.971	0.915 (0.031)
D4	1	−0.523	0.293	0.571 (0.060)

Fig C.1.

Normal Q-Q plot and histogram of the data extracted from one brain region of one subject.