Expandable factor analysis

Summary

Bayesian sparse factor models have proven useful for characterizing dependence in multivariate data, but scaling computation to large numbers of samples and dimensions is problematic. We propose expandable factor analysis for scalable inference in factor models when the number of factors is unknown. The method relies on a continuous shrinkage prior for efficient maximum a posteriori estimation of a low-rank and sparse loadings matrix. The structure of the prior leads to an estimation algorithm that accommodates uncertainty in the number of factors. We propose an information criterion to select the hyperparameters of the prior. Expandable factor analysis has better false discovery rates and true positive rates than its competitors across diverse simulation settings. We apply the proposed approach to a gene expression study of ageing in mice, demonstrating superior results relative to four competing methods.

1. Introduction

Factor analysis is a popular approach to modelling covariance matrices. Letting , and denote the true number of factors, number of dimensions and covariance matrix, respectively, factor models set , where is the loadings matrix and is a diagonal matrix of positive residual variances. To allow computation to scale to large , is commonly assumed to be of low rank and sparse. These assumptions imply that and the number of nonzero loadings is small. A practical problem is that and the locations of zeros in are unknown. A number of Bayesian approaches exist to model this uncertainty in and sparsity (Carvalho et al., 2008; Knowles & Ghahramani, 2011), but conventional approaches that rely on posterior sampling are intractable for large sample sizes and dimensions . Continuous shrinkage priors have been proposed that lead to computationally efficient sampling algorithms (Bhattacharya & Dunson, 2011), but the focus is on estimating , with treated as a nonidentifiable nuisance parameter. Our goal is to develop a computationally tractable approach for inference on that models the uncertainty in and the locations of zeros in . To do this, we propose a novel shrinkage prior and a corresponding class of efficient inference algorithms for factor analysis.

Penalized likelihood methods provide computationally efficient approaches for point estimation of and . If is known, then many such methods exist (Kneip & Sarda, 2011; Bai & Li, 2012). Sparse principal components analysis estimates a sparse assuming , where is the identity matrix (Jolliffe et al., 2003; Zou et al., 2006; Shen & Huang, 2008; Witten et al., 2009). The assumptions of spherical residual covariance and known are restrictive in practice. There are several approaches to estimating . In econometrics, it is popular to rely on test statistics based on the eigenvalues of the empirical covariance matrix (Onatski, 2009; Ahn & Horenstein, 2013). It is also common to fit the model for different choices of , and choose the best value based on an information criterion (Bai & Ng, 2002). Recent approaches instead use the trace norm or the sum of column norms of as a penalty in the objective function to estimate (Caner & Han, 2014). Alternatively, Ročková & George (2016) use a spike-and-slab prior to induce sparsity in with an Indian buffet process allowing uncertainty in ; a parameter-expanded expectation-maximization algorithm is then used for estimation.

We propose a Bayesian approach for estimation of a low-rank and sparse , allowing to be unknown. Our approach relies on a novel multi-scale generalized double Pareto prior, inspired by the generalized double Pareto prior for variable selection (Armagan et al., 2013) and by the multiplicative gamma process prior for loadings matrices (Bhattacharya & Dunson, 2011). The latter approach focuses on estimation of , but does not explicitly estimate or . The proposed prior leads to an efficient and scalable computational algorithm for obtaining a sparse estimate of with appealing practical and theoretical properties. We refer to our method as expandable factor analysis because it allows the number of factors to increase as more dimensions are added and as increases.

Expandable factor analysis combines the representational strengths of Bayesian approaches with the computational benefits of penalized likelihood methods. The multi-scale generalized double Pareto prior is concentrated near low-rank matrices; in particular, a high probability is placed around matrices with rank . Local linear approximation of the penalty imposed by the prior equals a sum of weighted penalties on the elements of . This facilitates maximum a posteriori estimation of a sparse using an extension of the coordinate descent algorithm for weighted -regularized regression (Zou & Li, 2008). The hyperparameters of our prior are selected using a version of the Bayesian information criterion for factor analysis. Under the theoretical set-up for high-dimensional factor analysis in Kneip & Sarda (2011), we show that the estimates of loadings are consistent and that the estimates of nonzero loadings are asymptotically normal.

2. Expandable factor analysis

2.1. Factor analysis model

Consider the usual factor model. Let , and be the mean-centred data matrix, latent factor matrix and residual error matrix, respectively, where and are unknown. We use index for samples, index for dimensions, and index for factors. If is the residual error variance matrix, then the factor model for is

$$\begin{array}{c}\hfill y_{id}=\sum _{j=1}^{k^{\ast }}{z}_{ij}{\lambda }_{dj}+e_{id},z_{ij}\sim N(0,1),e_{id}\mid {\sigma }_d^2\sim N(0,{\sigma }_d^2),\end{array}$$ (1)

where and are independent . Equivalently,

$$\begin{array}{c}\hfill y_i=\mathrm{\Lambda }{z}_i+e_i,y_i=(y_{i1},\dots ,y_{ip}{)}^{\mathrm{T}},z_i=(z_{i1},\dots ,z_{i{k}^{\ast }}{)}^{\mathrm{T}},e_i=(e_{i1},\dots ,e_{ip}{)}^{\mathrm{T}}\end{array}$$ (2)

for sample and . Similarly, model (1) reduces to regression in the space of latent factors

$$\begin{array}{c}\hfill y_d=Z{\lambda }_d+e_d,y_d=(y_{1d},\dots ,y_{nd}{)}^{\mathrm{T}},{\lambda }_d=({\lambda }_{d1},\dots ,{\lambda }_{d{k}^{\ast }}{)}^{\mathrm{T}},e_d=(e_{1d},\dots ,e_{nd}{)}^{\mathrm{T}}\end{array}$$ (3)

for dimension . Unlike usual regression, the design matrix in (3) is unknown.

Penalized estimation of is typically based on (2) or (3). The loss is estimated as the regression-type squared error after imputing using the eigendecomposition of the empirical covariance matrix or an expectation-maximization algorithm. The choice of penalty on presents a variety of options. If the goal is to select factors that affect any of the variables, then the sum of column norms of can be used as a penalty; a recent example is the group bridge penalty, , where and is an upper bound on . The selected factors correspond to the nonzero columns of the estimated (Caner & Han, 2014). To further obtain elementwise sparsity, a nonconcave variable selection penalty can be applied to the elements in . The estimate of depends on the choice of criterion for selecting the tuning parameters (Hirose & Yamamoto, 2015).

Our expandable factor analysis differs from this typical approach in several important ways. We start from a Bayesian perspective, and place a prior on that is structured to allow uncertainty in while shrinking towards loadings matrices with many zeros and . If is an upper bound on , then the prior is designed to automatically allow a slow rate of growth in as the number of dimensions increases by concentrating in neighbourhoods of matrices with rank bounded above by . To our knowledge, this is a unique feature of our approach, justifying its name. Expandability is an appealing characteristic, as more factors should be needed to accurately model the dependence structure as the dimension of the data increases.

2.2. Multi-scale generalized double Pareto prior

We would like to design a prior on such that maximum a posteriori estimates of have the following four characteristics:

• (a) the estimate of a loading with large magnitude should be nearly unbiased;

• (b) a thresholding rule, such as soft-thresholding, is used to estimate the loadings so that loadings estimates with small magnitudes are automatically set to zero;

• (c) the estimator of any loading is continuous in the data to limit instability; and

• (d) the -norm of the th column of the estimated does not increase as increases.

The first three properties are related to nonconcave variable selection (Fan & Li, 2001). Properties (b) and (d) together ensure existence of a column index after which all estimated loadings are identically zero. Automatic relevance determination and multiplicative gamma process priors satisfy (d) but fail to satisfy (b). No existing prior for loadings matrices satisfies properties (a)–(d) simultaneously (Carvalho et al., 2008; Bhattacharya & Dunson, 2011; Knowles & Ghahramani, 2011).

In order to satisfy these four properties and obtain a computationally efficient inference procedure, it is convenient to start with a prior for a loadings matrix having infinitely many columns; in practice, all of the elements will be estimated to be zero after a finite column index that corresponds to the estimated number of factors. Bhattacharya & Dunson (2011) showed that the set of loadings matrices that leads to well-defined covariance matrices is

$$\begin{array}{c}\hfill \mathcal{C}=\{\mathrm{\Lambda }:\underset{1\le d\le p}{max}\sum _{j=1}^{\mathrm{\infty }}{\lambda }_{dj}^2<\mathrm{\infty }\}\text{.}\end{array}$$

We propose a multi-scale generalized double Pareto prior for having support on . This prior is constructed to concentrate near low-rank matrices, placing high probability around matrices with rank at most .

The multi-scale generalized double Pareto prior on specifies independent generalized double Pareto priors on (; ) so that the density of is

$$\begin{array}{c}\hfill p_{\mathrm{m}\mathrm{g}\mathrm{d}\mathrm{P}}(\mathrm{\Lambda })=\prod _{d=1}^p\prod _{j=1}^{\mathrm{\infty }}{p}_{\mathrm{g}\mathrm{d}\mathrm{P}}({\lambda }_{dj}\mid {\alpha }_j,{\eta }_j),p_{\mathrm{g}\mathrm{d}\mathrm{P}}({\lambda }_{dj}\mid {\alpha }_j,{\eta }_j)=\frac{{\alpha }_j}{2{\eta }_j}(1+\frac{|{\lambda }_{dj}|}{{\eta }_j}{)}^{-({\alpha }_j+1)},\end{array}$$ (4)

where is the generalized double Pareto density with parameters and (Armagan et al., 2013). This prior on ensures that properties (a)–(c) are satisfied. Property (d) is satisfied by choosing parameter sequences and () such that two conditions hold: the prior measure on has density in (4), and has as its support. These conditions hold for the form of and () specified by the following lemma.

Lemma 1

If , () and , then . 

The proof is given in the Supplementary Material, along with the other proofs.

As in Bhattacharya & Dunson (2011), we truncate to a finite number of columns for tractable computation. This truncation is accomplished by mapping to , with retaining the first columns of . The choice of is such that is arbitrarily close to , where distance between and is measured using the -norm of their elementwise difference. In addition, for computational convenience, we assume that the hyperparameters and () are analytic functions of the parameters and , respectively, with these functions satisfying the conditions of Lemma 1.

The following lemma defines the forms of and () in terms of and .

Lemma 2

If , , and for , then , where has density in (4) with hyperparameters and (). Furthermore, given , there exists a positive integer for every such that for all , , () and , we have that where . 

The penalty imposed on the loadings by the prior grows exponentially with as the column index increases. This property of the prior ensures that all the loadings are estimated to be zero after a finite column index, which corresponds to the estimated number of factors.

3. Estimation algorithm

3.1. Expectation-maximization algorithm

We rely on an adaptation of the expectation-maximization algorithm to estimate and . Choose a positive integer of order as the upper bound on ; the estimate of the number of factors will be less than or equal to . The results are not sensitive to the choice of due to the properties of the multi-scale generalized double Pareto prior, provided is sufficiently large. If is too small, then the estimated number of factors will be equal to the upper bound, suggesting that this bound should be increased. Given , define and () as in Lemma 2, with and being prespecified constants.

We present the objective function as a starting point for developing the coordinate descent algorithm and provide derivations in the Supplementary Material. Let and , where the superscript denotes an estimate at iteration and denotes the conditional expectation given , and based on (1). The objective function for parameter updates in iteration is

$$\begin{array}{cc}\hfill \underset{\begin{array}{c}\hfill {\lambda }_d,{\sigma }_d^2\\ \hfill d=1,\dots ,p\end{array}}{\arg min}\sum _{d=1}^p(& \frac{n+2}{2npk}\log {\sigma }_d^2+\frac{\Vert w_d^{(t)}-X^{(t)}{\lambda }_d{\Vert }^2-{w_d^{(t)}}^{\mathrm{T}}{w}_d^{(t)}+(Y^{\mathrm{T}}Y/n{)}_{dd}}{2pk{\sigma }_d^2}\hfill \\ & \hspace{0.17em}+[\sum _{j=1}^k\frac{{\alpha }_j(\delta )+1}{npk}\log \left\{1+\frac{|{\lambda }_{dj}|}{{\eta }_j(\rho )}\right\}]),\hfill \end{array}$$ (5)

where and ().

3.2. Estimating parameters using a convex objective function

The objective (5) is written as a sum of terms. The th term corresponds to the objective function for the regularized estimation of the th row of the loadings matrix, , with a specific form of log penalty on (Zou & Li, 2008). Local linear approximation at of the log penalty on in (5) implies that each row of is estimated separately at iteration :

$$\begin{array}{c}\hfill {\lambda }_d^{\mathrm{l}\mathrm{l}\mathrm{a}(t+1)}=\underset{{\lambda }_d}{\arg min}\frac{\Vert w_d^{(t)}-X^{(t)}{\lambda }_d{\Vert }^2}{2pk{\sigma }_d^{2(t)}}+\sum _{j=1}^k\frac{{\alpha }_j(\delta )+1}{npk\{{\eta }_j(\rho )+|{\lambda }_{dj}^{(t)}|\}}|{\lambda }_{dj}|(d=1,\dots ,p)\text{.}\end{array}$$ (6)

This problem corresponds to regularized estimation of regression coefficients with as the response, as the design matrix, as the error variance, and a weighted penalty on .

The solution to (6) is found using block coordinate descent. Let column of and row of without the th element be written as and . Then the update to estimate is

$$\begin{array}{c}\hfill {\lambda }_{dj}^{\mathrm{l}\mathrm{l}\mathrm{a}(t+1)}=\frac{\text{sign}({\tilde{\lambda }}_{dj}^{(t)})}{f_{jj}^{(t)}}(|{\tilde{\lambda }}_{dj}^{(t)}|-c_{dj}^{(t)}{)}_{+},c_{dj}^{(t)}=\frac{{\sigma }_d^{2(t)}\{{\alpha }_j(\delta )+1\}}{n\{{\eta }_j(\rho )+|{\lambda }_{dj}^{(t)}|\}}(j=1,\dots ,k),\end{array}$$ (7)

where and . Fix at in (5) to update in iteration as

$$\begin{array}{cc}\hfill {\sigma }_d^{2(t+1)}& \hspace{0.17em}=\frac{n}{n+2}\{(Y^{\mathrm{T}}Y/n{)}_{dd}+{\lambda }^{\mathrm{l}\mathrm{l}\mathrm{a}(t+1)\mathrm{T}}{F}^{(t)}{\lambda }^{\mathrm{l}\mathrm{l}\mathrm{a}(t+1)}-2l_d^{(t)\mathrm{T}}{\lambda }^{\mathrm{l}\mathrm{l}\mathrm{a}(t+1)}\}\text{.}\hfill \end{array}$$ (8)

If any root--consistent estimate of is used instead of in (6), then it acts as a warm starting point for the estimation algorithm. This leads to a consistent estimate of in one step of coordinate descent (Zou & Li, 2008). An implementation of this approach for known values of and is summarized in steps (i)–(iv) of Algorithm 1 using the R (R Development Core Team, 2017) package glmnet (Friedman et al., 2010).

Algorithm 1

Estimation algorithm for expandable factor analysis.

• Notation :

  • 1. is the diagonal matrix containing diagonal elements of a symmetric matrix .

  • 2. Chol() is the upper triangular Cholesky factorization of a symmetric positive-definite matrix .

  • 3. is a block-diagonal matrix with forming the diagonal blocks.

  • 4. , where .

• Input :

  • 1. Data and upper bound on the rank of the loadings matrix.

  • 2. The - grid with grid indices (; ).

• Do :

  • 1. Centre data about their mean (; ).

  • 2. Let . Then estimate eigenvalues and eigenvectors of : and .

  • 3. Define to be the matrix .

  • 4. Begin estimation of , and across the - grid:

  • For

  •    For

  •     (i) Define , if , and if ().

  •     (ii) Initialize the following statistics required in (7):

    $$\begin{array}{cc}& {\mathrm{\Sigma }}^0=\text{diag}(S_{\hat{y}\hat{y}}-{\mathrm{\Lambda }}^0{{\mathrm{\Lambda }}^0}^{\mathrm{T}}),{\mathrm{\Omega }}^0={\mathrm{\Lambda }}^0{{\mathrm{\Lambda }}^0}^{\mathrm{T}}+{\mathrm{\Sigma }}^0,G^0=({\mathrm{\Omega }}^0{)}^{-1}{\mathrm{\Lambda }}^0,L^0=S_{\hat{y}\hat{y}}{G}^0,\hfill \\ & {\mathrm{\Delta }}^0=I_k-{{\mathrm{\Lambda }}^0}^{\mathrm{T}}{G}^0,F^0={\mathrm{\Delta }}^0+{G^0}^{\mathrm{T}}{S}_{\hat{y}\hat{y}}{G}^0,R^0=\text{Chol}(F^0).\hfill \end{array}$$
  •     (iii) Define , , and required to solve (6):

    $$\begin{array}{cc}\hfill X& \hspace{0.17em}=\mathrm{b}\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}(\underset{p\text{\hspace{0.5em}times}}{\underbrace{R^0,\dots ,R^0}}),w=\{\underset{k\text{\hspace{0.5em}times}}{\underbrace{({\mathrm{\Sigma }}^0{)}_{11}^{-1},\dots ,({\mathrm{\Sigma }}^0{)}_{11}^{-1}}},\dots ,\underset{k\text{\hspace{0.5em}times}}{\underbrace{({\mathrm{\Sigma }}^0{)}_{pp}^{-1},\dots ,({\mathrm{\Sigma }}^0{)}_{pp}^{-1}}}\},\hfill \\ \hfill y& \hspace{0.17em}=\text{vec}\{(R^0{)}^{-1}{L^0}^{\mathrm{T}}\},v=\frac{1}{npk}(\frac{{\alpha }_1+1}{{\eta }_1+|{\lambda }_{11}^0|},\dots ,\frac{{\alpha }_k+1}{{\eta }_k+|{\lambda }_{1k}^0|},\dots ,\frac{{\alpha }_1+1}{{\eta }_1+|{\lambda }_{p1}^0|},\dots ,\frac{{\alpha }_k+1}{{\eta }_k+|{\lambda }_{pk}^0|}).\hfill \end{array}$$

  •     (iv) Estimate in (7) and in (8) using the R package glmnet in three steps:

  •    result glmnetx = , y = , weights = , intercept = FALSE,

          standardize = FALSE, penalty.factor = .

  •    coefresult, s = , exact = TRUE [-1, ].

  •    ().

  • (v) Set , , , and estimate the posterior weight in (10).

  • End for.

  • Set .

  • End for.

  • 5. Obtain grid index for the estimate of , where

• Return :

  • , and .

The estimate of obtained using (7) satisfies properties (a)–(d) described earlier. The adaptive threshold in (7) ensures that property (a) is satisfied. The soft-thresholding rule to estimate ensures that property (b) is satisfied. The local linear approximation (6) has continuous first derivatives in the parameter space excluding zero, so property (c) is also satisfied (Zou & Li, 2008). The estimate satisfies property (d) due to the structured penalty imposed by the prior.

We comment briefly on the choice of prior and uncertainty quantification. We build on the generalized double Pareto prior instead of other shrinkage priors not only because the estimate of satisfies properties (a)–(d), but also because local linear approximation of the resulting penalty has a weighted form. We exploit this for efficient computations and use a warm starting point to estimate a sparse in one step using Algorithm 1. Uncertainty estimates of the nonzero loadings are obtained from Laplace approximation, and the remaining loadings are estimated as zero without uncertainty quantification.

3.3. Root--consistent estimate of

The root--consistent estimate of exists under Assumptions A0–A4 given in the Appendix. If and () are the eigenvalues and eigenvectors of the empirical covariance matrix , then is the eigendecomposition of . It is known that is a root--consistent estimator of if is fixed and . If , and , then is a root--consistent estimator of ; see the Supplementary Material for a proof. Scaling by is required because the largest eigenvalue of tends to infinity as (Kneip & Sarda, 2011). This scaling does not change our estimation algorithm for in (7), except that is changed to ().

3.4. Bayesian information criterion to select and

The parameter estimates in (7) and (8) depend on the hyperparameters through and , both of which are unknown. To estimate and , we use a grid search. Let and form a - grid. If is the value of (, ) at grid index , then and are the hyperparameters of our prior defined using Lemma 2, and and are the parameter estimates based on this prior. Algorithm 1 first estimates and for every by choosing warm starting points and then estimates (, ) using all the estimated and . These two steps in the estimation of (, ) are described next.

The structured penalty imposed by our prior implies that has the maximum number of nonzero loadings. Algorithm 1 exploits this structure by first estimating and then other loadings matrices along the - grid by successively thresholding nonzero loadings in to 0. Let be the set that contains the locations of nonzero loadings in . The estimation path of Algorithm 1 across the - grid is such that () and .

After the estimation of and , is set to if has the maximum posterior probability. Let be the cardinality of set . Given , there are loadings matrices that have nonzero loadings but differ in the locations of the nonzero loadings. Assuming that each of these matrices is equally likely to represent the locations of nonzero loadings in the true loadings matrix, the prior for is

$$\begin{array}{c}\hfill \text{pr}({\mathcal{M}}^{(r,s)}\mid {\delta }_r,{\rho }_s)\propto {(\genfrac{}{}{0ex}{}{pk}{|{\mathcal{M}}^{(r,s)}|})}^{-1}(r=1,\dots ,R;s=1,\dots ,S)\text{.}\end{array}$$ (9)

Let be the posterior probability of . Then an asymptotic approximation to is

$$\begin{array}{c}\hfill -2\log f(Y,{\mathrm{\Lambda }}^{(r,s)}\mid {\delta }_r,{\rho }_s)+|{\mathcal{M}}^{(r,s)}|\log n+2|{\mathcal{M}}^{(r,s)}|\log (pk)\end{array}$$ (10)

if terms of order smaller than are ignored, where is the joint density of and based on (1). The first term in (10) measures the goodness-of-fit, and the last two terms penalize complexity of a factor model with samples and loadings with the locations of nonzero loadings in . Theorem 3 in the next section shows that and have the same asymptotic order under certain regularity assumptions, where is the extended Bayesian information criteria of Chen & Chen (2008) and is an unknown constant. The analytic forms of and are the same when and terms of order smaller than are ignored, so we use for estimating in our numerical experiments.

4. Theoretical properties

Let and be the fixed points of and The updates (7) and (8) define the map , where . The following theorem shows that our estimation algorithm retains the convergence properties of the expectation-maximization algorithm.

Theorem 1

If represents the objective (5), then does not decrease at every iteration. Let be the local linear approximation of (5). Assume that only for stationary points of ; then the sequence converges to its stationary point . 

Let be the true loadings matrix and the residual variance matrix. We define (; ) and express as having columns. The locations of true nonzero loadings are in the set . Let and be the estimates of and obtained using our estimation algorithm for a specific choice of and (); then is an estimator of . If and , then and retain elements of and with indices in the set . The following theorem specifies the asymptotic properties of , and .

Theorem 2

Suppose that Assumptions A0–A6 in the Appendix hold and that , and . Then, for any and

• (i) , and are consistent estimators of , and , respectively 

• (ii) and in distribution, where is a symmetric positive-definite matrix and . 

Theorem 2 holds for any multi-scale generalized double Pareto prior with hyperparameters and () that satisfies Assumption 5. In practice, the estimate of depends on the choice of and . Restricting the search to the hyperparameters indexed along the - grid, Algorithm 1 sets the values of the hyperparameters to and (), where achieves its maximum at grid index . The following theorem justifies this method of selecting hyperparameters and shows the asymptotic relationship between and .

Theorem 3

Suppose that the generalized double Pareto prior with hyperparameters defined using leads to estimation of . Let be another set that contains the locations of nonzero loadings in an estimated for a given . Define and . If Assumptions A0–A7 in the Appendix hold, then for any such that

• (i) in probability as  

• (ii) as . 

Let be a point on the - grid that leads to estimation of . Then, Theorem 3 shows that Algorithm 1 selects with probability tending to 1 because will be larger than any where is such that .

5. Data analysis

5.1. Set-up and comparison metrics

We compared our method with those of Caner & Han (2014), Hirose & Yamamoto (2015), Ročková & George (2016) and Witten et al. (2009). The first competitor was developed to estimate the rank of , and the last three competitors were developed to estimate . We used two versions of Roǎková and George’s method. The first version uses the expectation-maximization algorithm developed in Ročková & George (2016), and the second version adds an extra step in every iteration of the algorithm that rotates the loadings matrix using the varimax criterion.

We evaluated the performance of the methods for estimating on simulated data using the root mean square error, proportion of true positives, and proportion of false discoveries:

$$\begin{array}{cc}& \text{mean square error}=\sum _{d=1}^p\sum _{j=1}^k(|{\lambda }_{dj}^{\ast }|-|{\hat{\lambda }}_{dj}|{)}^2/(pk),\text{true positive rate}=|\hat{\mathcal{M}}\cap {\mathcal{M}}^{\ast }|/|{\mathcal{M}}^{\ast }|,\hfill \\ & \text{false discovery rate}=|\hat{\mathcal{M}}\mathrm{\setminus }{\mathcal{M}}^{\ast }|/|\hat{\mathcal{M}}|,\hfill \end{array}$$

where and are the true and estimated loadings matrices and and are the true and estimated locations of nonzero loadings. We assume that for any and . Since and could differ in sign, mean square error compared their magnitudes.

5.2. Simulated data analysis

The simulation settings were based on examples in Kneip & Sarda (2011). The number of dimensions varied among . The rank of every simulated loadings matrix was fixed at . The magnitudes of nonzero loadings in a column were equal and decreased as , , , and from the first to the fifth column. The signs of the nonzero loadings were chosen such that the columns of any loadings matrix were orthogonal, with a small fraction of overlapping nonzero loadings between adjacent columns:

$$\begin{array}{c}\hfill {\lambda }_{dj}^{\ast }=\{\begin{array}{ll}\hfill 2(6-j),& 1+(j-1)\frac{p}{k^{\ast }}\le d\le j\frac{p}{k^{\ast }},1\le j\le k^{\ast },\hfill \\ \hfill -2(6-j),& 1+j\frac{p}{k^{\ast }}\le d\le (j+1)\frac{p}{k^{\ast }},1\le j\le k^{\ast }-1,\hfill \\ \hfill -2(6-j),& (j-1)\frac{p}{k^{\ast }}\le d\le j\frac{p}{k^{\ast }}-1,2\le j\le k^{\ast },\hfill \\ \hfill 0,& \text{otherwise.}\hfill \end{array}\hspace{0.17em}\end{array}$$

The error variances increased linearly from to for . With varying sample sizes , data were simulated using model (1) for all combinations of and . The simulation set-up was replicated ten times and all five methods were applied in every replication by fixing the upper bound on the number of factors at . The - grid had dimensions , and increased linearly from to while increased linearly from to when and from to when .

All five methods had the same computational complexity of for one iteration, but their runtimes differed depending on their implementations, with the method of Witten et al. (2009) being the fastest. Figure 1 shows that Hirose and Yamamoto’s method and both versions of Roǎková and George’s method significantly overestimated for large . The method of Witten et al. slightly overestimated across all settings. Caner and Han’s method showed excellent performance and accurately estimated across all simulation settings, except when and or . When was larger than 500, Assumption A4 was satisfied and our method accurately estimated as 5 in every setting, performing better than Caner and Han’s method when .

Fig. 1.

Rank estimate averaged across simulation replications for the methods of Caner & Han (2014) (crosses), Hirose & Yamamoto (2015) (squares), Ročková & George (2016) varimax-free version (empty circles), Ročková & George (2016) varimax version (filled circles) and Witten et al. (2009) (diamonds), as well as our estimation algorithm (triangles). In each panel the horizontal grey line represents the true number of factors; error bars represent Monte Carlo errors.

The four methods for estimating differed significantly in their root mean square errors, true positive rates, and false discovery rates; see Figs 2–4. Hirose and Yamamoto’s method had the highest false discovery rates and the lowest true positive rates across most settings. Both versions of Roǎková and George’s method estimated an overly dense across most settings, resulting in high true positive rates and high false discovery rates. The extra rotation step in the second version of Roǎková and George’s method resulted in excellent mean square error performance; however, varimax rotation is a post-processing step. A similar step to reduce the mean square error could be added to our method, for example by including a step to rotate the in step 3 of Algorithm 1 using the varimax criterion. When and were small, the method of Witten et al. achieved the lowest false discovery rates while our method achieved the highest true positive rates. When and were larger than 250 and 100, respectively, Assumption A4 was satisfied and our method simultaneously achieved the highest true positive rates and lowest false discovery rates while maintaining competitive mean square errors relative to the rotation-free methods.

Fig. 2.

Root mean square error averaged across simulation replications for the methods of Hirose & Yamamoto (2015) (squares), Ročková & George (2016) varimax-free version (empty circles), Ročková & George (2016) varimax version (filled circles) and Witten et al. (2009) (diamonds), as well as our estimation algorithm (triangles). Error bars represent Monte Carlo errors.

Fig. 4.

False discovery rate averaged across simulation replications for the methods of Hirose & Yamamoto (2015) (squares), Ročková & George (2016) original version (empty circles), Ročková & George (2016) varimax version (filled circles) and Witten et al. (2009) (diamonds), as well as our estimation algorithm (triangles). Error bars represent Monte Carlo errors.

Fig. 3.

True positive rate averaged across simulation replications for the methods of Hirose & Yamamoto (2015) (squares), Ročková & George (2016) varimax-free version (empty circles), Ročková & George (2016) varimax version (filled circles) and Witten et al. (2009) (diamonds), as well as our estimation algorithm (triangles). Error bars represent Monte Carlo errors.

5.3. Microarray data analysis

We used gene expression data on ageing in mice from the AGEMAP database (Zahn et al., 2007). There were 40 mice aged 1, 6, 16 and 24 months in this study. Each age group included five male and five female mice. Tissue samples were collected from 16 different tissues, including the cerebrum and cerebellum, for every mouse. Gene expression levels in every tissue sample were measured on a microarray platform. After normalization and removal of missing data, gene expression data were available for all 8932 probes across 618 microarrays. We used a factor model to estimate the effect of latent biological processes on gene expression variation.

AGEMAP data were centred before analysis following Perry & Owen (2010). Gene expression measurements were represented by , where and . Further, age represented the age of mouse and gender was 1 if mouse was female and 0 otherwise. Least-squares estimates of the intercept, age effect and gender effect in the linear model (), with idiosyncratic error , were represented as , and . Using these estimates for , the mean-centred data were defined as

$$\begin{array}{c}\hfill {\hat{y}}_{id}=y_{id}-{\hat{\beta }}_{0d}+{\hat{\beta }}_{1d}{\text{age}}_i+{\hat{\beta }}_{2d}{\text{gender}}_i(i=1,\dots ,n;d=1,\dots ,p)\text{.}\end{array}$$

Four mice were randomly held out, and all tissue samples for these mice in were used as test data. The remaining samples were used as training data. This set-up was replicated ten times. All four methods were applied to the training data in every replication by fixing the upper bound on the number of factors at 10. The - grid had dimensions , and increased linearly from to while increased linearly from to .

The results for all five methods were stable across all ten folds of crossvalidation. Caner and Han’s method, Hirose and Yamamoto’s method, both versions of Roǎková and George’s method, the method of Witten et al. and our method selected 10, 10, 10, 4 and 1, respectively, as the number of latent biological processes across all folds. Our result matched the result of Perry & Owen (2010), who confirmed the presence of one latent variable using rotation tests. Our simulation results and the findings in Perry & Owen (2010) strongly suggest that our method accurately estimated and the other methods overestimated .

We also estimated the factors for the test data. With denoting test datum and denoting the singular value decomposition of , the factor estimate of test datum was , where denotes the number of samples in the training data. Perry & Owen (2010) found that factor estimates for the tissue samples from cerebrum and cerebellum, respectively, had bimodal densities. We used the density function in R with default settings to obtain kernel density estimates of the factors. Hirose and Yamamoto’s method and both versions of Roǎková and George’s method estimated the number of factors as 10, which made the results challenging to interpret. The method of Witten et al. recovered bimodal densities in all four factors for both tissue samples, but it was unclear which of these four factors corresponded to the factor estimated by Perry & Owen (2010). Our method estimated the number of factors to be 1 and recovered the bimodal density in both tissue samples.

Supplementary material

Supplementary material available at Biometrika online includes derivation of the expectation-maximization algorithm, proofs of Lemmas 1 and 2 and Theorems 1–3, supporting figures for the results in § 5.3, and the R code used for data analysis.

Appendix

Assumptions

Assumptions A0–A4 follow from the theoretical set-up for high-dimensional factor models in Kneip & Sarda (2011). Assumption 5 is based on results in Zou & Li (2008) for variable selection.

Assumption A0.

Let , , , , , and ().

Assumption A1.

There exist finite positive constants , and such that , and (; ).

Assumption A2.

There exists a constant such that , , and are -sub-Gaussian for every . A random variable is -sub-Gaussian if for any .

Assumption A3.

Let be the eigenvalues of ; then there exists a such that , , , and .

Assumption A4.

The sample size and dimension are large enough that and .

Assumption A5.

Let be the upper bound on and let , , and () be defined as in Lemma 2. Then , , and () as , and .

Assumption A6.

The elements of the set are fixed and do not change as or increases to .

Model (2) is recovered upon substituting into Assumption A0. Assumption A1 ensures that is positive definite. Assumption A2 ensures that the empirical covariances are good approximations of the true covariances. Specifically, for any ,

$$\begin{array}{cc}\hfill \underset{1\le j,l\le p}{sup}\left|\frac{1}{n}\sum _{i=1}^n{w}_{ij}{w}_{il}-\text{cov}(w_{ij},w_{il})\right|& \hspace{0.17em}\le t,\underset{1\le j,l\le p}{sup}\left|\frac{1}{n}\sum _{i=1}^n{e}_{ij}{e}_{il}-\text{cov}(e_{ij},e_{il})\right|\le t,\hfill \\ \hfill \underset{1\le j,l\le p}{sup}\left|\frac{1}{n}\sum _{i=1}^n{w}_{ij}{e}_{il}\right|& \hspace{0.17em}\le t,\underset{1\le j,l\le p}{sup}\left|\frac{1}{n}\sum _{i=1}^n{y}_{ij}{y}_{il}-\text{cov}(y_{ij},y_{il})\right|\le t\hfill \end{array}$$

hold simultaneously with probability at least . If , then as and . Assumption A3 guarantees identifiability of when is large and . Assumption A4 is required to ensure that is a root--consistent estimator of as , and .

One additional assumption is required to relate and .

Assumption A7

Let for a fixed constant such that .

Assumption A7 and equation (4.6) in Theorem 3 of Kneip & Sarda (2011) imply that for any such that , because as .