A Likelihood-Free Estimator of Population Structure Bridging Admixture Models and Principal Components Analysis

Characterizing genetic variation in humans is an important task in statistical genetics, enabling disease-gene mapping in genome-wide association studies (GWAS) and informing studies of human evolutionary history. A common approach to quantifying genetic variation...

Abstract

We introduce a simple and computationally efficient method for fitting the admixture model of genetic population structure, called ALStructure. The strategy of ALStructure is to first estimate the low-dimensional linear subspace of the population admixture components, and then search for a model within this subspace that is consistent with the admixture model’s natural probabilistic constraints. Central to this strategy is the observation that all models belonging to this constrained space of solutions are risk-minimizing and have equal likelihood, rendering any additional optimization unnecessary. The low-dimensional linear subspace is estimated through a recently introduced principal components analysis method that is appropriate for genotype data, thereby providing a solution that has both principal components and probabilistic admixture interpretations. Our approach differs fundamentally from other existing methods for estimating admixture, which aim to fit the admixture model directly by searching for parameters that maximize the likelihood function or the posterior probability. We observe that ALStructure typically outperforms existing methods both in accuracy and computational speed under a wide array of simulated and real human genotype datasets. Throughout this work, we emphasize that the admixture model is a special case of a much broader class of models for which algorithms similar to ALStructure may be successfully employed.

UNDERSTANDING structured genetic variation in human populations remains a foundational problem in modern genetics. Such an understanding allows researchers to correct for population structure in GWAS studies, enabling accurate disease-gene mapping (Knowler et al. 1988; Marchini et al. 2004; Song et al. 2015). Additionally, characterizing genetic variation is important for the study of human evolutionary history (Cavalli-Sforza et al. 1988; Esteban et al. 1998; Li et al. 2008).

To this end, much work has been done to develop methods to estimate what Alexander et al. (2009) term global ancestry. In the global ancestry framework, the goal is to simultaneously estimate two quantities:

1. the allele frequencies of ancestral populations.

2. the admixture proportions of each modern individual.

Many popular global ancestry estimation methods have been developed within a probabilistic framework. In these methods, which we will refer to as likelihood-based approaches, the strategy is to fit a probabilistic model to the observed genome-wide genotype data by either maximizing the likelihood function (Tang et al. 2005; Alexander et al. 2009) or the posterior probability (Pritchard et al. 2000; Raj et al. 2014; Gopalan et al. 2016). The probabilistic model fit in each of these cases is the admixture model, described in detail in the Model and Theory section below, in which the global ancestry quantities (i) and (ii) are explicit parameters to be estimated.

A related line of work relies on principal components analysis (PCA) and other eigen-decomposition methods, rather than directly fitting probabilistic models; as such, we will refer to them collectively as PCA-based approaches. These methods find many of the same applications as global ancestry estimates while obviating a direct computation of global ancestry itself. For example, the EIGENSTRAT method of Patterson et al. (2006) and Price et al. (2006) uses the principal components of observed data to correct for population stratification in GWAS, avoiding altogether the estimation of admixture proportions or ancestral allele frequencies. Similarly, Hao et al. (2016) observe that many important applications of global ancestry really only require individual-specific allele frequencies. In a sense, individual-specific allele frequencies are simpler than global ancestry; while global ancestry specifies all of the individual-specific allele frequencies, the converse is not true. Therefore, Hao et al. (2016) introduce a simple truncated-PCA method that accurately and efficiently estimates individual-specific allele frequencies alone.

Both likelihood-based and PCA-based approaches have distinct merits and drawbacks. The PCA-based methods are computationally efficient and accurate in practice. It is shown, for instance, that the individual-specific allele frequencies obtained by truncated-PCA are empirically more accurate than those obtained by likelihood-based methods (Hao et al. 2016). Another attractive feature of PCA-based methods is that they make minimal assumptions about the underlying data-generative model. However, as mentioned before, PCA-based methods do not provide the full global ancestry estimates that their corresponding likelihood-based methods do. Most notably, they do not provide direct estimates of admixture proportions, which are often of primary interest in some applications. Additionally, PCA-based methods often have weaker statistical justifications, as they are typically not based on a probabilistic model. Although Tipping and Bishop (1999) introduced a probabilistic interpretation of PCA for multivariate Normal data, to our knowledge, no such interpretation of PCA exists when the data are Binomial, as is the case in the admixture model.

Recognizing the relative advantages of each approach, several researchers have attempted to bridge the gap between likelihood-based and PCA-based approaches. In spirit, this is also the approach that we take in the present work, and so we briefly review previous contributions to contextualize the advances made by our own method. Engelhardt and Stephens (2010) observed that fitting the admixture model was related to PCA in the sense that both could be posed as matrix factorization problems, which differ only in the constraints imposed on factors. They then introduced a third matrix factorization problem, called Sparse Factor Analysis (SFA), which encourages a sparsity through a particular prior. However, since SFA does not enforce the probabilistic constraints of the admixture model (nor the orthogonality constraints of PCA), its output cannot be directly interpreted as an estimate of global ancestry. Lawson et al. (2012) provided further insight into the mathematical relationship between admixture models and PCA and introduced a method for the analysis of phased haplotype data. This method, called fineSTRUCTURE, fits a version of the admixture model in which each observed individual belongs to a single (rather than admixed) population. Zheng and Weir (2016) introduced a method called EIGMIX that leverages PCA to infer admixture proportions from unphased genotype data. While EIGMIX allows individual genomes to be derived from a mixture of multiple ancestral populations (unlike fineSTRUCTURE), it requires a set of sampled individuals known to be derived from single ancestral populations. A related line of work uses PCA-based approaches to fit models of local ancestry, in which inferences about the ancestry of individual genetic loci are desired (for example, Brisbin et al. 2012).

While the aforementioned literature illustrates that PCA can be leveraged to provide information about population structure, each approach falls short of providing complete estimates of global ancestry under the general admixture model. The method which we introduce in the present work, called ALStructure, does precisely this. ALStructure requires no additional assumptions (such as the existence of unadmixed individuals in Zheng and Weir 2016), no specialized input (such as the unphased haplotypes of Lawson et al. 2012), and provides direct estimates of admixture proportions (unlike Engelhardt and Stephens 2010). As such, ALStructure is the only existing PCA-based method that can provide a direct substitute to the most popular likelihood-based approaches. As an additional important advantage, the underlying mathematical theory that justifies ALStructure is sufficiently general so as to apply to a class of models that subsumes the admixture model. As such, we believe that imitable algorithms to ALStructure could be useful beyond the present genetics application.

The basic strategy of ALStructure is to eliminate the primary shortcomings of PCA-based methods while retaining their important advantages over likelihood-based methods. In particular, we extend the approach taken in Hao et al. (2016) in two ways. First, we replace classical PCA with the closely related method of Latent Subspace Estimation (LSE) (Chen and Storey 2015). In so doing, we will make mathematically rigorous the empirically effective truncated-PCA method of Hao et al. (2016) for estimating individual-specific allele frequencies. Second, we use the method of alternating least squares (ALS) to transform the individual-specific allele frequencies obtained via LSE into estimates of global ancestry.

We perform a body of simulations and analyze several globally and locally sampled human studies to demonstrate the performance of the proposed method, showing that ALStructure typically outperforms existing methods both in terms of accuracy and speed. We also discuss its implementation and the trade-offs between theoretical guarantees and run-time. We find that ALStructure is a computationally efficient and statistically accurate method for modeling admixture and decomposing systematic variation due to population structure.

The remainder of this paper is organized as follows. Model and Theory introduces the admixture model and details the mathematical underpinnings of our approach. The ALStructure Algorithm summarizes the ALStructure algorithm. A reader primarily interested in a basic understanding of the operational procedure of ALStructure and its applications may proceed to The ALStructure Algorithm after reading The admixture model subsection, as the remainder of Model and Theory is more technical in nature. Results From Simulated Data and Applications to Global Human Studies assess the performance of ALStructure on a wide range of real simulated datasets.

Model and Theory

In this section and the following we present the ALStructure method and detail some of its mathematical underpinnings. In The admixture model, we define the admixture model: the underlying probabilistic model assumed by ALStructure. Optimal model constraints describes the overall strategy of ALStructure as an optimality search subject to constraints rather than navigating a complex likelihood surface. Leveraging constraints to estimate $\widehat{\mathit{F}}$ describes how the constraints can be used to estimate individual-specific allele frequencies. In Latent subspace estimation we present a mathematical result from Chen and Storey (2015) upon which the ALStructure algorithm heavily relies. Leveraging constraints to estimate $\mathit{P}$ and $\mathit{Q}$ describes why estimating global ancestry, given the individual-specific allele frequencies, is equivalent to a constrained matrix factorization problem. An efficient algorithm based on the method of ALS is also provided in this section for performing the constrained matrix factorization. The complete ALStructure algorithm is then presented in The ALStructure Algorithm.

Throughout this work, we adhere to the following notational convention: for a matrix $\mathit{A}$, we denote the i row vector of $\mathit{A}$ by ${\mathit{a}}_{i\cdot }$, the j column vector of $\mathit{A}$ by ${\mathit{a}}_{\cdot j}$, and the $\left(i,j\right)$ element of $\mathit{A}$ by $a_{ij}$.

The admixture model

The observed data $\mathit{X}$ is an $m\times n$ matrix in which m (the number of SNPs) is typically much larger than n (the number of individuals). An element $x_{ij}$ of $\mathit{X}$ takes values 0, 1, or 2 according to the number of reference alleles in the genotype at locus i for individual j.

ALStructure makes the assumption common to all likelihood-based methods that the data are generated from the admixture model. Under this model, the genotypes are generated independently according to $x_{ij}|f_{ij}\sim \text{Binomial}\left(2,f_{ij}\right)$, where $\mathit{F}$ is an $m\times n$ matrix encoding all of the Binomial parameters. Each element $f_{ij}$ is an individual-specific allele frequency: the frequency of allele i in individual j. $\mathit{F}$ is further assumed to be of rank d, where $d\ll n\ll m$. d may be thought of as the number of ancestral populations from which the observed population is derived. $\mathit{F}$ then admits a factorization $\mathit{F}=\mathit{PQ}\mathrm{,}$ in which $\mathit{P}$ and $\mathit{Q}$ have the following properties:

$$\mathit{P}\in {\mathrm{ℝ}}^{m\times d}\hspace{0.17em}\text{with}\hspace{0.17em}{p}_{ij}\in \left[0,1\right]\hspace{0.17em}\forall \left(i,j\right)$$

$$\mathit{Q}\in {\mathrm{ℝ}}^{d\times n}\hspace{0.17em}\text{with}\hspace{0.17em}{q}_{ij}\ge 0\forall \left(i,j\right)\hspace{0.17em}\text{and}\hspace{0.17em}{\displaystyle \sum _i{q}_{ij}}=1\forall j$$

The matrices $\mathit{P}$ and $\mathit{Q}$ have the following interpretations: (i) each row ${\mathit{p}}_{i\cdot }$ of $\mathit{P}$ represents the frequencies of a single SNP for each of the d ancestral populations, and (ii) each column ${\mathit{q}}_{\cdot j}$ of $\mathit{Q}$ represents the admixture proportions of a single individual. Together, $\mathit{P}$ and $\mathit{Q}$ encode the global ancestry parameters of the observed population; the goal of existing likelihood-based methods is to estimate these matrices. By contrast, the truncated-PCA method of Hao et al. (2016) is focused on estimating $\mathit{F}$ and not its factors. Equation 1 summarizes the admixture model.

$$\underset{m\times n}{\left(\begin{array}{ccccc}& & & & \\ & & & & \\ & & & & \\ & & & & \\ & & \mathit{F}& & \\ & & & & \\ & & & & \\ & & & & \end{array}\right)}=\underset{m\times d}{\left(\begin{array}{ccc}& & \\ & & \\ & & \\ & & \\ & \mathit{P}& \\ & & \\ & & \\ & & \end{array}\right)}\underset{d\times n}{\left(\begin{array}{ccccc}& & & & \\ & & \mathit{Q}& & \\ & & & & \end{array}\right)}$$ (1)

The model introduced in Pritchard et al. (2000), which we refer to as the PSD model, is an important special case of the admixture model. It additionally assumes the following prior distributions on $\mathit{P}$ and $\mathit{Q}$:

$$p_{ij}\sim \text{Balding-Nichols}\left(F_i,p_i\right)$$

$${\mathit{q}}_{\cdot j}\sim \text{Dirichlet}\left(\mathit{\alpha }\right)$$

The Balding-Nichols distribution (Balding and Nichols 1995) is a reparameterization of the Beta distribution, in which $F_i$ is the $F_{ST}$ (Weir and Cockerham 1984) at locus i, and $p_i$ is the population minor allele frequency at locus i. Specifically, $\text{Balding-Nichols}\left(F,p\right)=\text{Beta}\left(\frac{1-F}{F}p,\frac{1-F}{F}\left(1-p\right)\right)$. (Other prior distributions can be used for $\mathit{P}$ and $\mathit{Q}$ (Pritchard et al. 2000), but here we refer to the PSD model as that using the priors listed here.) Existing Bayesian methods (Pritchard et al. 2000; Raj et al. 2014; Gopalan et al. 2016) fit the PSD model specifically, while existing maximum likelihood (ML) methods (Tang et al. 2005; Alexander et al. 2009) and ALStructure require only the admixture model assumptions.

Although we focus on fitting the admixture model in the present work, it is important to note that the general strategy of the ALStructure algorithm is insensitive to the particular details of this model. The necessary features that the theoretical underpinnings of ALStructure require are: (i) higher moments of $x_{ij}$ are bounded, (ii) $\mathit{F}$ is low rank, and (iii) $m\gg n$. (For a precise statement of the theoretical assumptions of LSE, see Chen and Storey 2015.) For example, an imitable algorithm could be applied to high dimensional data $\mathit{X}$ where $x_{ij}|f_{ij}\sim \text{Poisson}\left(f_{ij}\right),$ and $\mathit{F}$ is a low rank matrix whose factors $\mathit{P}$ and $\mathit{Q}$ potentially have natural constraints. Because of its generality, we hope that the approach of ALStructure will find useful application beyond the analysis of admixture.

Optimal model constraints

Most existing methods for fitting the admixture model employ various optimization techniques to search for the ML parameters (Pritchard et al. 2000; Alexander et al. 2009) or the maximum a posteriori estimate (Raj et al. 2014; Gopalan et al. 2016). Our approach has a fundamentally different character: rather than searching through a rough likelihood landscape in pursuit of an optimal solution, ALStructure seeks a feasible solution to a set of optimal constraints. To be more specific, we begin with the observation that any solution satisfying a particular set of constraints is risk-minimizing among a class of unbiased estimators. Because any feasible solution is optimal, we can think of the constraints themselves as being optimal. Notably, the need to maximize likelihood is altogether obviated.

The challenge of this approach is twofold. First, the constraints themselves need to be estimated from the data: they are not directly observable. This is done through the method of LSE detailed in Latent subspace estimation. Second, feasible solutions to the estimated constraints will not typically exist. For this reason, we seek solutions that approximately satisfy the constraints, thereby converting a feasibility problem to a least squares (LS) optimization problem. This procedure is done through the method of ALS and is detailed in Leveraging constraints to estimate $\mathit{P}$ and $\mathit{Q}$. Throughout the remainder of the present subsection, we detail the constraints themselves.

There are several constraints that any reasonable estimate of the parameters of the admixture model must obey. The first is simply that the parameter estimates $\widehat{\mathit{F}}\mathrm{,}$ $\widehat{\mathit{P}},$ and $\widehat{\mathit{Q}}$ obey the relationship $\widehat{\mathit{F}}=\widehat{\mathit{P}}\widehat{\mathit{Q}}$. We will refer to this constraint as the Equality constraint. The second obvious requirement is that entries of matrices $\widehat{\mathit{P}}$ and $\widehat{\mathit{Q}}$ obey the probabilistic constraints of the admixture model:

$$p_{ij}\in \left[0,1\right]\forall \left(i,j\right)$$ (2)

$$q_{ij}\ge 0\forall \left(i,j\right)$$ (3)

$${\displaystyle \sum _i{q}_{ij}}=1\forall j$$ (4)

As we will encounter these constraints frequently, we refer to Equation 2 as the “$\square$” constraint, and Equation 3 and Equation 4 as the “$△$” constraint. This is simply because the constraints on $\mathit{P}$ demarcate the boundaries of a d-dimensional unit cube (the generalization of a square), whereas the constraints on $\mathit{Q}$ demarcate a d-dimensional simplex (the generalization of a triangle). Together we refer to the $\square$ and $△$ constraints as the Boundary constraints.

The final constraint we require is that the row vectors of $\widehat{\mathit{F}}$ lie in the linear subspace spanned by the rows vectors of $\mathit{Q}$. If we denote $\langle \mathit{A}\rangle$ to be the rowspace of a matrix $\mathit{A}$, we can summarize this condition as:

$$\langle \widehat{\mathit{F}}\rangle =\langle \mathit{Q}\rangle$$ (5)

We will refer to Equation 5 as the LS (linear subspace) constraint. The LS constraint is the only nontrivial constraint that ALStructure enforces. The fact that $\langle \mathit{F}\rangle =\langle \mathit{Q}\rangle$ is a simple consequence of the linearity of the admixture model; indeed, all rows of $\mathit{F}$ are linear combinations of rows of $\mathit{Q}$ since $\mathit{F}=\mathit{PQ}$. The LS constraint thus requires the same property for our estimate $\widehat{\mathit{F}}$. It is important to note that the LS constraint is not the same as requiring that $\langle \widehat{\mathit{F}}\rangle =\langle \widehat{\mathit{Q}}\rangle$: this is ensured by the Equality constraint. Rather, the LS constraint requires that the row vectors of $\widehat{\mathit{F}}$ belong to the rowspace of the true $\mathit{Q}$ matrix. The apparent challenge of enforcing the LS constraint is that, a priori, one does not have access to $\langle \mathit{Q}\rangle$. However, ALStructure takes advantage of a recent result from Chen and Storey (2015) that $\langle \mathit{Q}\rangle$ can be consistently estimated directly from the data matrix $\mathit{X}$ in the asymptotic regime of interest, when the number of SNPs m grows large. The result of Chen and Storey (2015) is, in fact, much more general than is needed in our setting, and, therefore, will likely be useful in many other problems. Because of its importance to this work, we further discuss this result in the context of the admixture model in Latent subspace estimation, and show that a modified PCA of $\mathit{X}$ consistently recovers $\langle \mathit{Q}\rangle$.

Leveraging constraints to estimate $\widehat{\mathit{F}}$

The key step in ALStructure is to note that enforcing the LS constraint provides us with an immediate estimate for $\mathit{F}$. To motivate our estimator, first observe that the simple estimate $\stackrel{\sim }{\mathit{F}}=\frac{1}{2}\mathit{X}$ is, in some sense, a reasonable approximation of $\mathit{F}$: it is unbiased since $f_{ij}=\frac{1}{2}\text{E}\left[x_{ij}\right]$ under the admixture model. However, this estimate leaves much to be desired—most importantly, the estimate $\stackrel{\sim }{\mathit{F}}$ will in general be of full rank (n) rather than of low rank (d) and it will have a large variance. Assuming, for now, that we are provided with the true rowspace $\langle \mathit{Q}\rangle$ of $\mathit{F}$, a natural thing to try is to project the rows of $\frac{1}{2}\mathit{X}$ onto this linear subspace. Below, we show that this estimator has some appealing properties.

Let us denote the operator ${\text{Proj}}_{\langle \mathit{S}\rangle }\left(\mathit{X}\right)$ such that the rows of the matrix $\mathit{X}$ are projected onto the linear subspace $\langle \mathit{S}\rangle$. (Note that the notation ${\text{Proj}}_{\langle \mathit{S}\rangle }\left(\mathit{X}\right)$ typically refers to projection of the columns of $\mathit{X}$ onto the linear subspace $\langle \mathit{S}\rangle$, but here we use this notation to denote projection of the rows of $\mathit{X}$ onto $\langle \mathit{S}\rangle$.) If we are given an orthonormal basis $\left\{{\mathit{s}}_i\right\}$ of the d-dimensional linear subspace $\langle \mathit{S}\rangle$, then:

$${\text{Proj}}_{\langle \mathit{S}\rangle }\left(\mathit{X}\right)\equiv \mathit{X}\left({\displaystyle \sum _{i=1}^d{\mathit{s}}_i{\mathit{s}}_i^T}\right)$$

Lemma 1 below provides us a simple condition under which estimators of $\mathit{F}$ formed by such projections are unbiased.

Lemma 1:

For a rank d matrix $\mathit{F}$ that admits a factorization $\mathit{F}=\mathit{PQ}\mathrm{,}$ and a random matrix $\mathit{X}$ such that $\frac{1}{2}\text{E}\left[\mathit{X}\right]=\mathit{F}\mathrm{,}$ any estimator of $\mathit{F}$ of the form ${\widehat{\mathit{F}}}_{\langle \mathit{S}\rangle }\equiv \frac{1}{2}{\mathrm{Proj}}_{\langle \mathit{S}\rangle }\left(\mathit{X}\right)$ is unbiased if, and only if, $\langle \mathit{Q}\rangle \subseteq \langle \mathit{S}\rangle$.

Lemma 1 is proved in Appendix A.1. In particular we note that

$$\widehat{\mathit{F}}\equiv {\widehat{\mathit{F}}}_{\langle \mathit{Q}\rangle }$$ (6)

is unbiased.

In addition to being unbiased, the estimator $\widehat{\mathit{F}}$ is optimal in the following sense. Among all unbiased estimators constructed by projecting $\mathit{X}$ onto a linear subspace, $\widehat{\mathit{F}}$ minimizes a matrix equivalent of the mean squared error.

Lemma 2:

For a rank d matrix $\mathit{F}$ that admits a factorization $\mathit{F}=\mathit{PQ}\mathrm{,}$ and a random matrix $X,$ such that $\frac{1}{2}\text{E}\left[\mathit{X}\right]=\mathit{F}$, the estimator $\widehat{F}\equiv \frac{1}{2}{\mathrm{Proj}}_{\langle Q\rangle }\left(\mathit{X}\right)$ is an unbiased estimator of $\mathit{F}$ and has the smallest risk among all unbiased estimators of the form $\stackrel{\sim }{\mathit{F}}\equiv \frac{1}{2}{\mathrm{Proj}}_{\langle \mathit{S}\rangle }\left(\mathit{X}\right)$. We define the risk to be the expectation of the squared Frobenius norm:

$$R\left(\widehat{\mathit{F}},\mathit{F}\right)=\text{E}\left[\Vert \widehat{\mathit{F}}-\mathit{F}\Vert {}^2\right]$$

$$=\text{E}\left[{\displaystyle \sum _{ij}{\left({\widehat{f}}_{ij}-f_{ij}\right)}^2}\right]$$

Lemma 2 is proved in detail in Appendix A.2; however, the basic intuition is straightforward. Projecting $\mathit{X}$ onto a linear space $\langle \mathit{S}\rangle \subset \langle \mathit{Q}\rangle$ is biased (by Lemma 1). While projecting $\mathit{X}$ onto a space $\langle \mathit{S}\rangle \supset \langle \mathit{Q}\rangle$ will result in an unbiased estimate of $\mathit{F}$ (again, by Lemma 1), dimensions orthogonal to $\langle \mathit{Q}\rangle$ fit noise, increasing the variance (and therefore the mean squared error) of the estimate.

We note that this strategy is related to the strategy taken in Hao et al. (2016), in which $\mathit{F}$ was estimated by projecting $\frac{1}{2}\mathit{X}$ onto the space spanned by the first d principal components. In that work, it was observed that this simple strategy of estimating $\mathit{F}$ typically outperformed existing methods. We will see in Latent subspace estimation that the space spanned by the first d principal components is a good estimator for $\langle \mathit{Q}\rangle$ itself, but it can be improved practically and with theoretical guarantees by performing a modified PCA. Therefore, Lemma 2 provides a theoretical justification for the empirically accurate method put forward in Hao et al. (2016).

Latent subspace estimation

We have shown that the linear subspace $\langle \mathit{Q}\rangle$ can be leveraged to provide a desirable estimate of $\mathit{F}$. However, as $\langle \mathit{Q}\rangle$ is a linear subspace spanned by latent variables, it is not directly observable and must be estimated. Here, we show how a general technique developed in Chen and Storey (2015), which we will refer to as Latent Subspace Estimation (LSE), can be used to compute a consistent estimate of $\langle \mathit{Q}\rangle$ from the observed data $\mathit{X}$.

LSE is closely related to PCA, a popular technique that identifies linear combinations of variables that sequentially maximize variance explained in the data (Jolliffe 2002). As PCA is commonly used to find low-dimensional structure in high-dimensional data, a natural approach to estimating $\langle \mathit{Q}\rangle$ would be to employ SNP-wise PCA. More specifically, we might consider the linear space spanned by the first few eigenvectors of the $n\times n$ matrix, $\frac{1}{m}{\mathit{X}}^T\mathit{X}$ as an estimate of $\langle \mathit{Q}\rangle$.

The LSE-based estimate of $\langle \mathit{Q}\rangle$ almost exactly matches this PCA-based intuition. The only difference is that LSE accounts for the heteroscedastic nature of the admixture model, as detailed in Chen and Storey (2015). LSE has the theoretical advantage of asymptotically capturing $\langle \mathit{Q}\rangle$ in the high-dimensional setting (i.e., as $m\to \mathrm{\infty }$). Theorem 1, as stated here, is a special case of a more general theorem from Chen and Storey (2015), rewritten here for the special case of the admixture model.

Theorem 1:

Let us define ${\widehat{\delta }}_j=\frac{1}{m}{\displaystyle {\sum }_{i}2x_{ij}}-x_{ij}^2,$ and let $\mathit{D}$ be the diagonal matrix with jth entry equal to ${\widehat{\delta }}_j$. The d eigenvectors $\left\{{\upsilon }_1,\dots ,{\upsilon }_d\right\},$ corresponding to the top d eigenvalues of the matrix $\mathit{G}=\frac{1}{m}{\mathit{X}}^T\mathit{X}-\mathit{D}\mathrm{,}$ span the latent subspace $\langle \mathit{Q}\rangle$ in the sense that

$$\underset{m\to \mathrm{\infty }}{\text{lim}}\langle \left\{{\mathit{v}}_1,\dots ,{\mathit{v}}_d\right\}\rangle △\langle \mathit{Q}\rangle =\mathit{\varnothing }$$

with probability 1, where $△$ denotes the symmetric set difference. Further, the smallest $n-d$ eigenvalues of $\mathit{G}$ converge to 0 with probability 1.

Theorem 1 provides us with a simple procedure for estimating $\langle \mathit{Q}\rangle$ directly from data. One first computes ${\widehat{\delta }}_j$ and constructs the $n\times n$ matrix $\mathit{D}$. Next, an eigen-decomposition of the adjusted matrix $\frac{1}{m}{\mathit{X}}^T\mathit{X}-\mathit{D}$ is computed. Finally, we estimate $\langle \mathit{Q}\rangle$ as

$$\widehat{\langle \mathit{Q}\rangle }=\langle {\mathit{V}}_{\left(1:d\right)}^T\rangle$$ (7)

where ${\mathit{V}}_{1:d}$ are the first d columns from the singular value decomposition of $\mathit{G}$.

We stress that the general form of Theorem 1 from Chen and Storey (2015) makes LSE applicable to a vast array of models beyond factor models and the admixture model discussed here. As a further benefit to the LSE methodology, it is both easy to implement and computationally appealing. The entire computation of $\widehat{\langle \mathit{Q}\rangle }$ requires a single eigen-decomposition of an $n\times n$ matrix, where the accuracy depends only on large m.

Leveraging constraints to estimate $\mathit{P}$ and $\mathit{Q}$

Now that we have a method for obtaining the estimate $\widehat{\mathit{F}}$ by leveraging the LS constraint, what remains is to find estimates for $\mathit{P}$ and $\mathit{Q}$. Since the estimate $\widehat{\mathit{F}}$ has several appealing properties, as outlined in Leveraging constraints to estimate $\widehat{\mathit{F}}$, the approach of ALStructure is simply to keep $\widehat{\mathit{F}}$ fixed and seek matrices $\widehat{\mathit{P}}$ and $\widehat{\mathit{Q}}$ that obey the Equality and Boundary constraints of the admixture model. Below we discuss some of the general properties of this approach: namely the question of existence and uniqueness of solutions. We will briefly discuss the general problem of nonidentifiability in the admixture model, and provide simple and interpretable conditions under which the admixture model is identifiable. Finally, we will provide simple algorithms for computing $\widehat{\mathit{P}}$ and $\widehat{\mathit{Q}}$ from $\widehat{\mathit{F}}$ based on the method of ALS.

Existence, uniqueness, and anchor conditions:

First, we develop some terminology. We will say that an $m\times n$ matrix $\mathit{A}$ admits an admixture-factorization if the following feasibility problem has a solution:

$$\text{find:}(\mathit{B},\mathit{C})$$ (8)

$$\text{subject to:}\mathit{A}=\mathit{BC}\text{\hspace{0.17em}and}\left(\square ,△\right)$$

In words, the feasibility problem in Equation 8 simply seeks a factorization of $\mathit{A}$ that obeys the Equality and Boundary constraints from Optimal model constraints imposed by the admixture model. The smallest integer d for which $\left(\mathit{B},\mathit{C}\right)$ is a solution to Equation 8 with $\mathit{B}$ an $m\times d$ matrix and $\mathit{C}$ a $d\times n$ matrix is the admixture-rank of $\mathit{A}$, which we denote ${\text{rank}}_{\text{ADM}}\left(\mathit{A}\right)$. By seeking a rank d admixture-factorization of $\widehat{\mathit{F}}$, ALStructure converts a problem of high-dimensional statistical inference to a matrix factorization problem.

This simple approach has two apparent shortcomings:

1. A rank d admixture-factorization of $\widehat{\mathit{F}}$ may not exist.

2. If a valid factorization exists, it will not be unique.

Item (i) is a technical problem; though $\mathit{F}$ admits a rank d admixture factorization by assumption, the same is not true for $\widehat{\mathit{F}}$ in general. Even though the rank of $\widehat{\mathit{F}}$ is d by construction, $\text{rank}\left(\widehat{\mathit{F}}\right)\ne {\text{rank}}_{\text{ADM}}\left(\widehat{\mathit{F}}\right)$ in general. ALStructure avoids this problem by changing the feasibility problem expressed in Equation 8 to the following optimization problem:

$$\underset{\left(\mathit{B},\mathit{C}\right)}{\text{minimize}}\Vert \mathit{A}-\mathit{BC}\Vert$$ (9)

$$\text{subject to:}(\square ,△)$$

It is important to note that (ii) is not a problem unique to ALStructure, but is a fundamental limitation for any ML method as well. This is because the likelihood function depends on $\widehat{\mathit{P}}$ and $\widehat{\mathit{Q}}$ only through their product $\widehat{\mathit{F}}$; more formally, the admixture model is nonidentifiable. One unavoidable source of nonidentifiability is that any solution $\left(\widehat{\mathit{P}},\widehat{\mathit{Q}}\right)$ to the matrix factorization problem in Equation 8 will remain a valid solution after applying a permutation to the columns of $\widehat{\mathit{P}}$ and the rows of $\widehat{\mathit{Q}}$. A natural question to ask is: “When is there a unique factorization $\widehat{\mathit{F}}=\widehat{\mathit{P}}\widehat{\mathit{Q}}$ up to permutations?”

Two important types of sufficient conditions under which unique factorizations exist up to permutations are anchor SNPs and anchor individuals. We note that the concept of anchors has been previously employed in the field of topic modeling, where anchor words are of interest (Arora et al. 2013). We define an anchor SNP as one that is fixed in all ancestral populations except one. The anchor SNPs condition is then satisfied if each of the d ancestral populations has at least one corresponding anchor SNP. Analogously, we define an anchor individual as one whose entire genome is inherited from a single ancestral population. The anchor individuals condition is then satisfied if each of the d ancestral populations has at least one corresponding anchor individual. The assumption of anchor individuals is equivalent to the assumption of “surrogate ancestral samples” required by the EIGMIX method of Zheng and Weir (2016). The fact that either a set of d anchor SNPs or d anchor individuals makes the admixture model identifiable up to permutations follows from a simple argument found in Appendix A.3. For the special case of $d=3$, Figure 1 graphically displays the anchor conditions. It is important to remember that ALStructure does not require anchors to function. Rather, anchors provide interpretable conditions under which solutions provided by ALStructure, or any likelihood-based method, can be meaningfully compared to the underlying truth.

Figure 1.

Summary of sufficient conditions for a factorization $\mathit{F}=\mathit{PQ}$ to be unique for $d=3$. Axes represent the components of the row vectors of $\mathit{P}$ and the column vectors of $\mathit{Q}$ respectively. (left) Anchor SNPs: there is at least one row of $\mathit{P}$ on each of the red lines. (right) Anchor genotypes: there is at least one column of $\mathit{Q}$ on each of the red dots.

The anchor SNP and anchor individual conditions are not necessarily the only sufficient conditions for ensuring identifiability of the admixture model, and indeed, to the best or our knowledge, there is not currently a complete characterization of conditions for which the admixture model is identifiable. We regard this as an important open problem. In practice, ALStructure is capable of retrieving solutions remarkably close to the underlying truth even in simulated scenarios far from satisfying the anchor conditions, including conditions that are challenging for existing methods.

Computation:

Here, we present two simple algorithms for solving the optimization problem:

$$\underset{\left(\mathit{P},\mathit{Q}\right)}{\text{minimize}}\Vert \widehat{\mathit{F}}-\mathit{PQ}\Vert$$ (10)

$$\text{subject to:}\left(\square ,△\right)$$

The first algorithm, which we call cALS (constrained ALS), has the advantage that it is guaranteed to converge to a stationary point of the nonconvex objective function in (10). While a stationary point will not generally correspond to a globally optimal solution, global optimization is seldom possible for nonconvex problems.

Although theoretically appealing, this algorithm relies on solving many constrained quadratic programming problems and is, consequently, potentially slow. To overcome this problem, we introduce a second algorithm called tALS (truncated ALS), which simply ignores the problematic quadratic constraints in cALS. Though lacking a theoretical guarantee of convergence, the increase in speed is significant and the outputs of the two algorithms are often practically indistinguishable.

We note that the general method of ALS is not novel. In particular, previous work has developed ALS methods for the problem of nonnegative matrix factorization (NNMF) (Paatero and Tapper 1994; Lee and Sebastian 1999). In NNMF, one seeks a low-rank factorization, $\mathit{A}=\mathit{BC}\mathrm{,}$ in which all elements of the factors $\mathit{B}$ and $\mathit{C}$ are non-negative. Algorithms analogous to cALS and tALS, but with non-negativity constraints rather than the $\square$ and $△$ constraints, have previously been considered (Berry et al. 2007; Cichocki et al. 2007; Gillis and Glineur 2012; Kim et al. 2014).

An algorithm with provable convergence:

While problem (10) is nonconvex as stated, the following two subproblems are convex:

$$\underset{\mathit{P}}{\text{minimize}}\Vert \widehat{\mathit{F}}-\mathit{PQ}\Vert$$ (11)

subject to:$\square$

$$\underset{\mathit{Q}}{\text{minimize}}\Vert \widehat{\mathit{F}}-\mathit{PQ}\Vert$$ (12)

subject to:$△$That (11) and (12) are convex problems is clear; norms are always convex functions, and $\square$ and $△$ are convex constraints. In particular, (11) and (12) are both members of the well-studied class of Quadratic Programs (QP), for which many efficient algorithms exist (Boyd and Vandenberghe 2009). We propose the following procedure for factoring $\widehat{\mathit{F}}$, which we call Constrained ALS.

Algorithm 1: Constrained ALS Algorithm.

• procedure cALS$\left(\widehat{\mathit{F}},d\right)$.

•  Initialize $\widehat{\mathit{P}}$ arbitrarily.

•  repeat.

•   Solve (12) with $\mathit{P}=\widehat{\mathit{P}}$ and return $\widehat{\mathit{Q}}$.

•   Solve (11) with $\mathit{Q}=\widehat{\mathit{Q}}$ and return $\widehat{\mathit{P}}$.

•  until Convergence of $\widehat{\mathit{P}}$ and $\widehat{\mathit{Q}}$.

• return $\left(\widehat{\mathit{P}},\widehat{\mathit{Q}}\right)$.

Despite the original problem being nonconvex, Algorithm 1 is guaranteed to converge to a stationary point of the objective function in (10) as a result of the following theorem from Grippo and Sciandrone (2000).

Theorem 2:

For the two block problem,

$$\underset{\mathit{P},\mathit{Q}}{\mathrm{minimize}}f\left(\mathit{P},\mathit{Q}\right)$$

if $\left\{{\mathit{P}}_i\right\}$ and $\left\{{\mathit{Q}}_i\right\}$ are sequences of optimal solutions to the two subproblems:

$$\underset{\mathit{P}}{\mathrm{minimize}}f\left(\mathit{P},{\mathit{Q}}_i\right)$$

$$\underset{\mathit{Q}}{\mathrm{minimize}}f\left({\mathit{P}}_i,\mathit{Q}\right)$$

then any limit point $\left(\mathit{P},\mathit{Q}\right)$ will be a stationary point of the original problem. (Note that the result from Grippo and Sciandrone (2000) is actually more general than this. We reproduce the special case above in order to make clear the connection to our problem.)

An efficient heuristic algorithm:

If we remove all constraints on $\mathit{P}$ and $\mathit{Q}$ from Equation 11 and Equation 12, the resulting optimization problems are simple linear LS.

$$\underset{\mathit{P}}{\text{minimize}}\Vert \mathit{F}-\mathit{PQ}\Vert$$ (13)

$$\underset{\mathit{Q}}{\text{minimize}}\Vert \mathit{F}-\mathit{PQ}\Vert$$ (14)

Our algorithm proceeds by alternating between solving the unconstrained LS problems (13) and (14). After each step, the optimal solution will not necessarily obey the constraints of problem (10). To keep our algorithm from converging on an infeasible point, we truncate the solution to force it into the feasible set. More precisely, each element of the solution ${\mathit{P}}^{\ast }$ to (13) is truncated to satisfy $\square$ and each column of the solution ${\mathit{Q}}^{\ast }$ to (14) is projected to the closest point on the simplex defined by the $△$ constraint. Simplex-projection is nontrivial; however, it is a well-studied optimization problem. Here we use a particularly simple and fast algorithm from Chen and Ye (2011). This algorithm, which we call the Truncated ALS Algorithm, is detailed in Algorithm 2.

Algorithm 2: Truncated ALS Algorithm.

• procedure tALS$\left(\widehat{\mathit{F}},d\right)$

•  Initialize $\widehat{\mathit{P}}$ arbitrarily.

•  repeat

•   Solve (14) with $\mathit{P}=\widehat{\mathit{P}}$, and return the simplex-projected solution $\widehat{\mathit{Q}}$.

•   Solve (13) with $\mathit{Q}=\widehat{\mathit{Q}}$ and return the truncated solution $\widehat{\mathit{P}}$.

•  until Convergence of $\widehat{\mathit{P}}$ and $\widehat{\mathit{Q}}$

• return $\left(\widehat{\mathit{P}},\widehat{\mathit{Q}}\right)$

An example dataset:

Figure 2 displays the output of cALS and tALS on a dataset from the PSD model with the parameters: $m=100,000$, $n=500$, $d=3$, $\mathit{\alpha }=\left(0.1,0.1,0.1\right)$. As can be seen, the output fits for $\mathit{Q}$ provided by cALS and tALS are practically indistinguishable to the eye, and are both excellent approximations of the ground truth. The cALS algorithm performed slightly better than the tALS algorithm ($8.5\times {10}^{-3}$ and $8.7\times {10}^{-3}$ RMSE, respectively). However, cALS took 3.5 hr to complete while tALS terminated in under 1.5 min. Because of the significant gains in efficiency, we use tALS exclusively throughout the remainder of this paper. The analyst who requires theoretical guarantees can, of course, use the cALS algorithm instead. Appendix B provides a more detailed comparison between the tALS and cALS algorithms on simulated data.

Figure 2.

Biplots of the first two rows of $\mathit{Q}$ (left), ${\widehat{\mathit{Q}}}_{\text{tALS}}$ (middle) and ${\widehat{\mathit{Q}}}_{\text{cALS}}$ (right). Blue points are provided as a visual aid and delineate a common subset of individuals.

The ALStructure algorithm

In this section, we briefly outline the entire ALStructure method whose components were motivated in depth in Model and Theory. In order to fit the admixture model, we obtain estimates $\widehat{\mathit{F}}$, $\widehat{\mathit{P}}$, and $\widehat{\mathit{Q}}$ from the SNP matrix $\mathit{X}$ through the following three-part procedure:

1. Estimate the linear subspace $\langle \mathit{Q}\rangle$ from the data $\mathit{X}$.

2. Project $\frac{1}{2}\mathit{X}$ onto the estimate $\widehat{\langle \mathit{Q}\rangle }$ to obtain an estimate of $\mathit{F}$.

3. Factor the estimate $\widehat{\mathit{F}}$ subject to the Equality and Boundary constraints to obtain $\widehat{\mathit{P}}$ and $\widehat{\mathit{Q}}$.

For convenience, we detail the entire ALStructure algorithm in Algorithm 3, and annotate each of the three steps described above. We note that we have decided to use the tALS function rather than the cALS function in our definition of the ALStructure algorithm, valuing the speed advantage of tALS over the theoretical guarantees of cALS. If desired, one could of course choose to use the cALS function instead without making any other alterations to the ALStructure.

Algorithm 3: ALStructure.

• procedure ALStructure$\left(X,d\right)$

•  for $j=0\hspace{0.17em}\text{to}\hspace{0.17em}n\hspace{0.17em}\mathbf{do}$                (i)

•   ${\widehat{\delta }}_j\leftarrow \frac{1}{m}{\displaystyle {\sum }_{i=1}^{m}2x_{ij}}-x_{ij}^2$

•  $\mathit{D}\leftarrow \text{diag}\left(\left\{{\widehat{\delta }}_1,\dots ,{\widehat{\delta }}_n\right\}\right)$

•  $\mathit{G}\leftarrow \frac{1}{m}{X}^{T}X-D$

•  Compute eigen-decomposition $\mathit{G}=\mathit{VW}{\mathit{V}}^T$

•  $\widehat{\mathit{F}}\leftarrow \frac{1}{2}{\text{Proj}}_{\widehat{\langle \mathit{Q}\rangle }}\left(\mathit{X}\right)=\frac{1}{2}\mathit{X}{\mathit{V}}_{\left(1:d\right)}{\mathit{V}}_{\left(1:d\right)}^T{\mathit{V}}_{\left(1:d\right)}^T$       (ii)

•  $(\widehat{\mathit{P}},\widehat{\mathit{Q}})\leftarrow \text{tALS}\left(\widehat{\mathit{F}},d\right)$               (iii)

• return $\left(\widehat{\mathit{F}},\widehat{\mathit{P}},\widehat{\mathit{Q}}\right)$

We emphasize here that ALStructure’s estimate $\widehat{\mathit{Q}}$ is ultimately derived from the LSE-based estimate of the latent subspace $\widehat{\langle \mathit{Q}\rangle }$. As the method of LSE is closely linked to PCA, we consider ALStructure to be a unification of PCA-based and likelihood-based approaches.

Perhaps the most striking feature of Algorithm 3 is its brevity. One advantage of this simplicity is its ease of implementation. Although Algorithm 3 has been implemented in the R package ALStructure, it can clearly be reimplemented in any language quite easily. Equally important is that all of the operations in Algorithm 3 are standard. The only two computationally expensive components are a single eigen-decomposition if n is large, and QR decompositions to find linear least squares (LLS) solutions in the tALS algorithm. Both of these problems have a rich history and consequently have many efficient algorithms. It is likely that the ALStructure implementation of Algorithm 3 can be significantly sped up by utilizing approximate or randomized algorithms for the eigen-decomposition and/or LLS computations. In its current form, ALStructure simply uses the base R functions eigen() and solve() for the eigen-decomposition and LLS computations, respectively. Despite this, the current implementation of ALStructure is typically faster than existing algorithms as can be seen in Results From Simulated Data and Applications to Global Human Studies below.

The ALStructure method is a nonparametric estimator in the following ways. It makes no assumptions about the probability distributions of $\mathit{P}$ or $\mathit{Q}$. Any random variable taking values in $\left\{0,1\right\}$ is by necessity Bernoulli. In this vein, the assumption that $x_{ij}\sim \text{Binomial}\left(2,f_{ij}\right)$ is not a parametric assumption per se, but rather an assumption about independence of alleles. Finally, the likelihood function is not utilized in estimating $\mathit{P}$ and $\mathit{Q}$, making ALStructure likelihood-free.

For choosing the dimensionality of the model d, we recommend utilizing the recently proposed structural Hardy-Weinberg equilibrium (sHWE) test (Hao and Storey 2017). This test can perform a genome-wide goodness of fit test to the assumptions made in the admixture model over a range of d. It then identifies the minimal value of d that obtains the optimal goodness of fit. There are other ways to choose d, by using the theory and methods in Chen and Storey (2015) or by using other recent proposals (Patterson et al. 2006; Hao et al. 2016).

Results from Simulated Data

Simulated data sets

In this section, we compare the performance of ALStructure to three existing methods for global ancestry estimation, Admixture, fastSTRUCTURE and terastructure. Admixture, developed by Alexander et al. (2009), is a popular algorithm that takes a ML approach to fit the admixture model. Both fastSTRUCTURE (Raj et al. 2014) and terastructure (Gopalan et al. 2016) are Bayesian methods that fit the PSD model using variational Bayes approaches. We abbreviate these methods as ADX, FS, and TS in the figures. A comparison among these three methods appears in Gopalan et al. (2016), so we will focus on how they compare to ALStructure.

To this end, we first tested all algorithms on a diverse array of simulated datasets. The bulk of our simulated data sets come from the classical PSD model (defined in The admixture model), in which columns of $\mathit{Q}$ are distributed according to the $\text{Dirichlet}\left(\mathit{\alpha }\right)$ distribution and the rows of $\mathit{P}$ are drawn from the Balding-Nichols distribution. We varied the following parameters in our simulated datasets: m, n, d, and $\mathit{\alpha }$. Of particular note is the variation of $\mathit{\alpha }$. For this we used four $\mathit{\alpha }$-prototypes: ${\mathit{\alpha }}_1=\left(10,10,10\right)$, ${\mathit{\alpha }}_2=\left(1,1,1\right)$, ${\mathit{\alpha }}_3=\left(0.1,0.1,0.1\right)$, and ${\mathit{\alpha }}_4=\left(10,1,0.1\right)$. These four prototypes were chosen because they represent four qualitatively different distributions on the Dirichlet simplex as shown in Figure 3: ${\mathit{\alpha }}_1$ corresponds to points distributed near the center of the simplex, ${\mathit{\alpha }}_2$ corresponds to points distributed evenly across the simplex, ${\mathit{\alpha }}_3$ corresponds to points distributed along the edges of the simplex, and ${\mathit{\alpha }}_4$ corresponds to an asymmetric distribution in which points are concentrated in one of the corners of the simplex.

Figure 3.

Examples of typical random samples from the four different $\mathit{\alpha }$-prototypes. As can be seen, only ${\mathit{\alpha }}_2$ and ${\mathit{\alpha }}_3$ approximately obey the “anchor-individuals” condition.

When we produced datasets with $d>3$, we extended the prototypes in the natural way; for example for $d=6$, the ${\mathit{\alpha }}_4$ is becomes $\left(10,10,1,1,0.1,0.1\right)$. Table 1 lists all of the parameters we used to generate data under the Dirichlet model, for a total of 96 distinct combinations.

Table 1. Parameters of all simulated datasets.

Parameters
m	${10}^5$, $5\times {10}^5$
n	$5\times {10}^2$, ${10}^3$, $5\times {10}^3$, ${10}^4$
d	3, 6, 9
$\mathit{\alpha }$-prototypes	(10, 10, 10)
	(1, 1, 1)
	(0.1, 0.1, 0.1)
	(10, 1, 0.1)

The parameters of the Balding-Nichols distributions from which rows of the $\mathit{P}$ matrix were drawn were taken from real data, following the same strategy taken in Gopalan et al. (2016). Specifically, $F_i$ and $p_i$ were estimated for each SNP in the Human Genome Diversity Project (HGDP) dataset (Cavalli-Sforza 2005). Then, for each simulated dataset, m random samples are taken (with replacement) from the HGDP parameter estimates.

In addition to simulating $\mathit{Q}$ matrices from the classical $\text{Dirichlet}\left(\mathit{\alpha }\right)$ distribution with many different parameters $\mathit{\alpha }$, we also simulated data from the spatial model of admixture developed in Ochoa and Storey (2016). We deliberately chose to study this model because it is ill-suited for ALStructure; while ALStructure relies on the estimation of the d-dimensional linear subspace $\langle \mathit{Q}\rangle$, the columns of $\mathit{Q}$ produced under the spatial model lie on a one-dimensional curve within $\langle \mathit{Q}\rangle$. Despite this fundamentally challenging scenario, we see that ALStructure is often capable of recovering an accurate approximation.

Results from the PSD model

In order to give a representative picture of the relative performance of ALStructure against existing algorithms, we first plot the fits of all of the algorithms for two particular data sets out of the total 96 model data sets: (i) the data set in which ALStructure performs the best and (ii) the data set in which ALStructure performs the worst, according to mean absolute error (defined below).

On the left side of Figure 4, we see that all four algorithms perform very well for the data set in which ALStructure performs best, which comes from the ${\mathit{\alpha }}_3$-prototype. On the right side of Figure 4, the dataset was generated from the ${\mathit{\alpha }}_4$-prototype. We see that, while ALStructure certainly deviates substantially from the truth, so does every algorithm. Both fastSTRUCTURE and terastructure provide results that are qualitatively very different from the truth; where fastSTRUCTURE compresses all columns of $\mathit{Q}$ onto a single edge of the simplex, terastructure spreads them out through the interior of the simplex. Both Admixture and ALStructure provide solutions qualitatively similar to the truth. While the points in the Admixture solution extend much further along the edge of the simplex than the true model, the ALStructure solution spreads into the interior of the simplex more than the true model.

Figure 4.

Model fits by ALStructure, Admixture, fastSTRUCTURE, and terastructure on two particular simulated datasets. The left panel shows the fits corresponding to the dataset on which ALStructure performed the best. The right panel shows the fits corresponding to the dataset on which ALStructure performed the worst. Each point represents a column of the $\mathit{Q}$ matrix and is plotted by the first and second coordinates. Blue points are plotted as a visual aid and delineate a common subset of individuals.

Figure 5 provides a comprehensive summary of the performance of ALStructure against the existing algorithms on all simulated datasets. The top panels of Figure 5 summarize the accuracy of each of the four algorithms, according to two metrics: root mean squared error (RMSE) and mean absolute error (MAE).

$$\text{RMSE}\equiv \sqrt{\frac{1}{dn}{\displaystyle \sum _{k=1}^d{\displaystyle \sum _{j=1}^n{\left({\widehat{q}}_{kj}-q_{kj}\right)}^2}}}$$

$$\text{MAE}\equiv \frac{1}{dn}{\displaystyle \sum _{k=1}^d{\displaystyle \sum _{j=1}^n\left|{\widehat{q}}_{kj}-q_{kj}\right|}}$$

The bottom left panel of Figure 5 shows mean per observation log-likelihood, $\frac{1}{mn}{\displaystyle {\sum }_{i=1}^m{\displaystyle {\sum }_{j=1}^n\text{log \hspace{0.17em}}P\left(x_{ij}{|\widehat{f}}_{ij}\right)}}$, on all simulated data sets. (To obtain full data log-likelihoods, multiple these numbers by $mn$.) It is interesting to note that ALStructure performs comparably to other methods from the likelihood perspective despite the fact that it is the only method that does not explicitly utilize the likelihood function. However, we emphasize that likelihood is an imperfect metric of model fit for two reasons. First, because of the highly nonidentifiable nature of the admixture model as discussed in Model and Theory, many models are equivalent from the likelihood perspective. Therefore, if one is primarily concerned about the accuracy of admixture estimates, the RMSE or MAE metrics may be more suitable. Second, in high-dimensional models, it has been demonstrated that high likelihood may yield far inferior estimates (Efron 2013). Starting with Stein’s Paradox (Stein 1956), it has been shown in many settings that the ML estimator for several parameters may be uniformly worse in accuracy than methods that leverage shared information in the data.

Figure 5.

Summary of performance of ALStructure and existing algorithms. The points are colored by $\mathit{\alpha }$-prototype.

The bottom right panel of Figure 5 shows the distributions of run times for each algorithm on all modeled datasets. Due to the size of the simulated datasets and our computational constraints, each algorithm did not terminate on each of the 96 datasets. In Figure 5, we plot only the datasets for which all four algorithms successfully terminated. See Appendix C for more details. It is clear that ALStructure is competitive with respect to both model fit and time. ALStructure outperforms all methods according to both RMSE and MAE. With respect to time, ALStructure is clearly favored (one should note that the y-axis is on the $\text{log}$ scale).

Results from the spatial model

As a challenge to ALStructure, we simulate data from a model developed in Ochoa and Storey (2016), which we will refer to as the spatial model. This model mimics an admixed population that was generated by a process of diffusion in a one-dimensional environment. There are d unmixed ancestral populations equally spaced at positions $\left\{x_0,x_0+1,\dots ,x_0+d-1\right\}$ on an infinite line. If all populations begin to diffuse at time $t=0$ at the same diffusive rate, then population i will be distributed as a Gaussian with mean ${\mu }_i=x_0+i-1$ and SD σ. Therefore, under the spatial model, an individual sampled from position x will have admixture proportions shown in Equation 15,

$$\left(q_1\left(x\right),q_2\left(x\right),\dots ,q_d\left(x\right)\right)=\left(\frac{f_{\left({\mu }_1,\sigma \right)}\left(x\right)}{{\displaystyle {\sum }_{i=1}^d{f}_{\left({\mu }_i,\sigma \right)}\left(x\right)}},\frac{f_{\left({\mu }_2,\sigma \right)}\left(x\right)}{{\displaystyle {\sum }_{i=1}^d{f}_{\left({\mu }_i,\sigma \right)}\left(x\right)}},\dots ,\frac{f_{\left({\mu }_d,\sigma \right)}\left(x\right)}{{\displaystyle {\sum }_{i=1}^d{f}_{\left({\mu }_i,\sigma \right)}\left(x\right)}}\right)$$ (15)

where $f_{\left(\mu ,\sigma \right)}$ denotes the Gaussian distribution with parameters $\left(\mu ,\sigma \right)$.

Although this is just a special case of the admixture model, one would expect the spatial model to be particularly challenging for ALStructure because the admixture proportions belong to a one-dimensional curve parameterized by x, and ALStructure necessitates the estimation of a d-dimensional linear subspace in ${\mathrm{ℝ}}^n$. The challenge is much more pronounced when the populations are highly admixed (large σ). Figure 6 shows the model fits provided by ALStructure.

Figure 6.

ALStructure fits of datasets from the Spatial model. (left) Stacked barplots of ALStructure fits. (right) Biplots of ALStructure fits. The parameter σ was set to 0.5, 1, and 2 for the top, middle and bottom rows, respectively. Blue points are plotted as a visual aid and delineate corresponding columns of $\mathit{Q}$ and $\widehat{\mathit{Q}}$.

Indeed, for large values of σ $\left(\sigma =2\right)$, ALStructure fails to correctly capture the admixture proportions. However, for smaller values of σ $\left(\sigma =\left\{1,0.5\right\}\right)$, it can be seen that the fits provided by ALStructure are excellent. In all simulations $m={10}^5$, $n={10}^3$, and $d=3$.

We note that Gopalan et al. (2016) tested Admixture, fastSTRUCTURE, and terastructure on data drawn from the spatial model (which they refer to as “Scenario B”). They showed this model posed a significant challenge for all three methods, but found that terastructure performed the best.

Applications to Global Human Studies

Here, we apply ALStructure and existing methods to three globally sampled human genotype datasets: the Thousand Genomes Project (TGP), HGDP, and Human Origins (HO) datasets (Cavalli-Sforza 2005; Lazaridis et al. 2014; The 1000 Genomes Project Consortium et al. 2015). Table 2 summarizes several basic parameters of each of the datasets and Appendix D details the procedures used for building each dataset. Although we recommend using sHWE from Hao and Storey (2017) for choosing d, here we take the number of ancestral populations d directly from Gopalan et al. (2016) so that our results are easily comparable to those of the latter study.

Table 2. Dataset parameters.

Dataset	m	n	d	$m\times n$
TGP	1,229,310	1815	8	$\sim 2.2\times {10}^9$
HGDP	550,303	940	10	$\sim 5.2\times {10}^8$
HO	372,446	2251	14	$\sim 8.4\times {10}^8$

Figure 7 shows scatterplots of the first two rows of $\widehat{\mathit{Q}}$ for each of the three datasets provided by each of the four fits. To disambiguate the inherent nonidentifiability (see Model and Theory), we ordered the rows of the fits $\widehat{\mathit{Q}}$ by decreasing variation explained: $s_i^2={\Vert \mathit{X}{\widehat{\mathit{q}}}_{i\cdot }^T\Vert }^2$. Perhaps the most striking aspect of Figure 7 is the difference between the fits produced by each method. With the notable exception that Admixture and ALStructure have similar fits for the TGP and HGDP datasets, every pair of comparable scatterplots (i.e., within a single row of Figure 7) are qualitatively different. Figure 11 in the Appendix displays the same data represented as stacked barplots of the admixture proportions. In this representation too, qualitative differences between the fits are also evident. Table 3 shows the mean per observation log-likelihood of the fits provided by each of the four methods. Figure 12 in the Appendix shows that the distributions of per observation likelihood are nearly indistinguishable across all methods.

Figure 7.

Biplots of the first two rows of $\mathit{Q}$ (ranked by variation explained) of the fits of the TGP (top), HGDP (middle), and HO (bottom) datasets for each algorithm. Individuals are colored by coarse subpopulation from which they are sampled.

Table 3. The mean per-observation log-likelihood of each dataset under each method’s fit.

Method	Admixture	fastSTRUCTURE	terastructure	ALStructure
TGP	−0.7097	−0.7136	−0.7130	−0.7100
HGDP	−0.7494	−0.7536	−0.7608	−0.7505
HO	−0.7467	−0.7515	−0.7534	−0.7477

Next we compare the performance of ALStructure to existing methods both in terms of efficiency and accuracy. Unlike in the case of simulated datasets where the ground truth is known, here we cannot directly compare the quality of model fits across methods. Instead, we assess the quality of each method by its performance on data simulated from real data fits.

For concreteness, we briefly outline the process below:

1. Fit each dataset with each of the four methods to obtain 12 model fits.

2. Simulate datasets from the admixture model using parameters obtained in the previous step.

3. Fit each of the 12 simulated datasets with each of the four datasets (48 fits) and compute error measures.

The process above treats each of the four methods symmetrically, evaluating each method based on its ability to fit data simulated from both its own model fits as well as every other methods’ model fits.

Figure 8 summarizes the performance of each method with respect to both model fit and efficiency on data simulated from the above described process. As with the results on simulated datasets from Results from the PSD model, it is clear that ALStructure is competitive with respect to both model fit and time. Both Admixture and ALStructure outperform fastSTRUCTURE and terastructure by all quality of fit metrics. ALStructure far outperforms all methods with respect to time (one should note that the y-axis is on the $\text{log}$ scale).

Figure 8.

Summary of performance of ALStructure and preexisting algorithms on data simulated from real model fits. The points are colored by dataset.

In Appendix E, we compare the performance of ALStructure to pre-existing methods on an additional nonglobal dataset from Basu et al. (2016). In this dataset, individuals are sampled from 18 modern Indian subpopulations. India’s genetic admixture is of particular interest because of its long history of sociocultural norms promoting endogamy. We find that each of the four methods produce admixture estimates qualitatively similar to each other for this dataset (see Figure 10 in the Appendix). One possible explanation for this observed similarity is that the genetic history of India more closely mimics the admixture model than does global genetic history, as suggested by Lawson et al. (2018).

Discussion

In this work, we introduced ALStructure, a new method to fit the admixture model from observed genotypes. Our method attempts to find common ground between two previously distinct approaches to understanding genetic variation: likelihood-based approaches and PCA-based approaches. ALStructure features important merits from both. Like the likelihood-based approaches, ALStructure is grounded in the probabilistic admixture model, and provides full estimates of global ancestry. However, operationally the ALStructure method closely resembles PCA-based approaches. In particular, ALStructure’s estimates of global ancestry are derived from a consistent PCA-derived estimate that captures the underlying low-dimensional latent subspace. In this way, ALStructure can be considered a unification of likelihood-based and PCA-based methods.

Because ALStructure is operationally similar to PCA-based methods, it is computationally efficient. Specifically, the only computationally expensive operations required by the ALStructure algorithm are singular value and QR decompositions. Both of these computations have been extensively studied and optimized. Although ALStructure already performs favorably compared to preexisting algorithms in computational efficiency, it is likely that, by applying more sophisticated matrix decomposition techniques, ALStructure may see significant improvements in speed. Although extremely simple, ALStructure typically outperforms preexisting algorithms both in terms of accuracy and time. This observation holds under a wide array of datasets, both simulated and real.

The usefulness of PCA-based approaches has been increasingly recognized in related settings, such as the mixed membership stochastic block model (Rubin-Delanchy et al. 2017) and topic models (Ke and Wang 2017). The basic approach we have presented is quite general. In particular, the set of models that satisfy the underlying assumptions of LSE is large, subsuming the admixture model as well as many other probabilistic models with low intrinsic dimensionality. Consequently, we expect that the ALStructure method can be trivially altered to apply to many similar problems beyond the estimation of global ancestry.

Appendix A: Additional Mathematical Details

A.1 Proof of Lemma 1

First we show that $\widehat{\mathit{F}}$ is unbiased. Note that:

$$\text{E}\left[\widehat{\mathit{F}}\right]=\frac{1}{2}\text{E}[{\text{Proj}}_{\langle \mathit{Q}\rangle }\left(\mathit{X}\right)]$$

$$=\frac{1}{2}{\text{\hspace{0.17em}Proj}}_{\langle \mathit{Q}\rangle }\left(\text{E}\left[\mathit{X}\right]\right)$$

$$={\text{Proj}}_{\langle \mathit{Q}\rangle }\left(\mathit{F}\right)$$

$$=\mathit{F}$$

Between the first and second line, we note that the projection operator is linear and take advantage of linearity of expectation. Between the second and third line, we used the observation that $\frac{1}{2}\text{E}\left[\mathit{X}\right]=\mathit{F}$. Finally, ${\text{Proj}}_{\langle \mathit{Q}\rangle }\mathit{F}=\mathit{F}$ since all rows of $\mathit{F}$ belong to $\langle \mathit{Q}\rangle$. From an identical argument one can see that for projection onto any other subspace $\langle \mathit{S}\rangle$, the corresponding estimator ${\widehat{\mathit{F}}}_{\langle \mathit{S}\rangle }\equiv \frac{1}{2}{\text{Proj}}_{\langle \mathit{S}\rangle }\left(\mathit{X}\right)$ will have the property that

$$\text{E}\left[{\widehat{\mathit{F}}}_{\langle \mathit{S}\rangle }\right]={\text{Proj}}_{\langle \mathit{S}\rangle }\left(\mathit{F}\right)$$

It is clear that, if $\langle \mathit{Q}\rangle \subseteq \langle \mathit{S}\rangle$, then $\text{E}{\left[\widehat{\mathit{F}}\right]}_{\langle \mathit{S}\rangle }]={\text{Proj}}_{\langle \mathit{S}\rangle }\left(\mathit{F}\right)=\mathit{F}\mathrm{,}$ since the projection operator acts as the identity operator for vectors belonging to the subspace $\langle \mathit{S}\rangle$.

Next we show that the converse is true: $\text{E}\left[{\text{Proj}}_{\langle \mathit{S}\rangle }\right]=\mathit{F}$ implies $\langle \mathit{Q}\rangle \subseteq \langle \mathit{S}\rangle$. To do this, we prove the contrapositive statement. If $\langle \mathit{Q}\rangle ⊈\langle \mathit{S}\rangle ,$ then $\text{E}\left[{\widehat{\mathit{F}}}_{\mathit{S}}\right]\ne \mathit{F}$. This can be seen by noting that each row in $\widehat{\mathit{F}}={\text{Proj}}_{\langle \mathit{Q}\rangle }\left(\mathit{F}\right)$ is a vector in the linear subspace $\langle \mathit{Q}\rangle$ projected into the linear subspace $\langle \mathit{S}\rangle$; rows of $\widehat{\mathit{F}}$ therefore belong to the linear subspace $\langle \mathit{Q}\rangle \cap \langle \mathit{S}\rangle$. Unless $\langle \mathit{Q}\rangle \subseteq \langle \mathit{S}\rangle$, then the dimension of $\langle \mathit{Q}\rangle \cap \langle \mathit{S}\rangle$ is strictly less than d, the dimension of $\langle \mathit{Q}\rangle$ and the rank of $\mathit{F}$. Therefore, if $\langle \mathit{Q}\rangle ⊈\langle \mathit{S}\rangle$, the rank of $\text{E}\left[{\widehat{\mathit{F}}}_{\mathit{S}}\right]$ will be less than the rank of $\mathit{F}$, implying $\text{E}\left[{\widehat{\mathit{F}}}_{\mathit{S}}\right]\ne \mathit{F}$.

A.2 Proof of Lemma 2

Note that we can write the squared Frobenius norm as follows:

$$L\left(\mathit{F},\widehat{\mathit{F}}\right)\equiv {\Vert \widehat{\mathit{F}}-\mathit{F}\Vert }^2$$

$$=\text{Tr}\left[{\left(\widehat{\mathit{F}}-\mathit{F}\right)}^T\left(\widehat{\mathit{F}}-\mathit{F}\right)\right]$$

$$=\text{Tr}\left[{\widehat{\mathit{F}}}^T\widehat{\mathit{F}}\right]-2\text{Tr}\left[{\widehat{\mathit{F}}}^T\mathit{F}\right]+\text{Tr}\left[{\mathit{F}}^T\mathit{F}\right]$$ (16)

First, let us compute the risk of our projection estimator $\widehat{\mathit{F}}$. Suppose we have an orthonormal basis $\left\{{\mathit{v}}_i\right\}$ of $\langle \mathit{Q}\rangle$. Using the definition of $\widehat{\mathit{F}}$ from Equation 6, and the fact that the rows of both $\mathit{F}$ belong to $\langle \mathit{Q}\rangle$, we note that we can write any row of either matrix in terms of the basis vectors $\left\{{\mathit{v}}_i\right\}$:

$${\mathit{f}}_{i\cdot }={\displaystyle \sum _j\langle {\mathit{f}}_{i\cdot }^T,{\mathit{v}}_j\rangle {\mathit{v}}_j^T}$$ (17)

$${\widehat{\mathit{f}}}_{i\cdot }=\frac{1}{2}{\displaystyle \sum _j\langle {\mathit{x}}_{i\cdot }^T,{\mathit{v}}_j\rangle {\mathit{v}}_j^T}$$ (18)

By rewriting the matrices $\widehat{\mathit{F}}$ and $\mathit{F}$ with respect to the basis $\left\{{\mathit{v}}_i\right\},$ and using Equation 17 and Equation 18, it is a straightforward calculation to show that

$$\text{Tr}\left[{\mathit{F}}^T\mathit{F}\right]={\displaystyle \sum _{i=1}^m{\displaystyle \sum _{j=1}^k{\langle {\mathit{f}}_{i\cdot }^T,{\mathit{v}}_j\rangle }^2}}$$

$$\text{Tr}\left[{\widehat{\mathit{F}}}^T\widehat{\mathit{F}}\right]={\displaystyle \sum _{i=1}^m{\displaystyle \sum _{j=1}^k{\langle \frac{1}{2}{\mathit{x}}_{i\cdot }^T,{\mathit{v}}_j\rangle }^2}}$$

Substituting this result into Equation 16 and taking expectations, we have the following expression for our loss function:

$$R\left(\mathit{F},\widehat{\mathit{F}}\right)=\text{E}\left[L\left(\mathit{F},\widehat{\mathit{F}}\right)\right]$$

$$=\text{E}\left[\text{Tr}\left[{\widehat{\mathit{F}}}^T\widehat{\mathit{F}}\right]-2\text{Tr}\left[{\widehat{\mathit{F}}}^T\mathit{F}\right]+\text{Tr}\left[{\mathit{F}}^T\mathit{F}\right]\right]$$

$$=\text{E}\left[\text{Tr}\left[{\widehat{\mathit{F}}}^T\widehat{\mathit{F}}\right]-\text{Tr}\left[{\mathit{F}}^T\mathit{F}\right]\right]$$

$$={\displaystyle \sum _{i=1}^m{\displaystyle \sum _{j=1}^k\text{E}\left[{\langle \frac{1}{2}{\mathit{x}}_{i\cdot }^T,{\mathit{v}}_j\rangle }^2\right]}}-{\langle {\mathit{f}}_{i\cdot }^T,{\mathit{v}}_j\rangle }^2$$

$$=\frac{1}{4}{\displaystyle \sum _{i=1}^m{\displaystyle \sum _{j=1}^k\text{Var}\left[\langle {\mathit{x}}_{i\cdot }^T,{\mathit{v}}_j\rangle \right]}}$$ (19)

By studying Equation 19, we can see the estimator $\widehat{\mathit{F}}$ has several favorable properties. First note that the risk is a sum of $m\times k$ nonnegative numbers since $\text{Var}\left[\mathit{Z}\right]\ge 0$ for any random variable $\mathit{Z}$. If we were to project onto a larger subspace 〈S〉, where $\langle \mathit{S}\rangle \supset \langle \mathit{Q}\rangle$, we would add terms to Equation 19 and consequently increase our risk. If we were to project onto a smaller subspace $\langle \mathit{S}\rangle \subset \langle \mathit{Q}\rangle$, then the risk may decrease; however, our new estimator will now be biased by Lemma 1. From these observations, we conclude that $\widehat{\mathit{F}}$ is optimal in the sense described in the Lemma 2.

A.3 Proof of sufficiency of anchors

Here, we show that either a set of anchor SNPs or a set of anchor individuals is sufficient to specify a unique factorization $\mathit{F}=\mathit{PQ}$ up to the nonidentifiability associated with row permutations.

Proposition 1.

For a rank d matrix $\mathit{F}$ with a factorization $\mathit{F}=\mathit{PQ}$, if there is a set S of d rows of $\mathit{P}$ such that for each $i\in \left\{1,2,\dots ,d\right\}$ there exists a row vector ${\mathit{p}}_{i\cdot }\in S$ such that ${\mathit{p}}_{i\cdot }={\delta }_i{e}_i$ for ${\delta }_i\ne 0$, where $e_i$ is a vector of length n in which element i is 1, and all other elements are 0, then the factorization is unique up to permutation. When such a set S exists, we say that we have “anchor SNPs.”

Proof.

Let us denote the matrix $\mathit{D}=\text{diag}\left({\delta }_1,{\delta }_2,\dots ,{\delta }_d\right)$. Without loss of generality, let us assume that S is the first d rows of P and are ordered such that

$$\mathit{P}=\left(\begin{array}{c}\mathit{D}\\ \mathit{A}\end{array}\right)$$

for some $\left(m-d\right)\times d$ matrix $\mathit{A}$. Then there is a unique $\mathit{Q}$ for this $\mathit{F}$ matrix (up to permutation) which is

$$\mathit{Q}={\mathit{D}}^{-1}{\mathit{F}}_{1:d}$$

The matrix $\mathit{A}$ is also uniquely determined by $\mathit{F}$ once $\mathit{Q}$ is fixed. To see this, note that

$${\mathit{f}}_{j\cdot }={\mathit{p}}_{j\cdot }\mathit{Q}$$

where ${\mathit{f}}_{j\cdot }$ and ${\mathit{p}}_{j\cdot }$ denote the j row of $\mathit{F}$ and $\mathit{P}$ respectively. Since ${\mathit{f}}_{j\cdot }$ is fixed and $\mathit{Q}$ is unique under the anchor SNP assumption, there is a unique solution for ${\mathit{p}}_{j\cdot }$ by the linear independence of the rows of $\mathit{Q}$.

The interpretation of the anchor SNPs assumption is that every ancestral population has at least one SNP that appears only in it. The presence of such an SNP is therefore a guarantee that the individual is a member of a particular population. Note that an identical argument could be made when we have a set S of d columns of $\mathit{Q}$ that have exactly one nonzero entry at unique locations. When such a set exists, we say that we have “anchor individuals.” Under the admixture model, the simplex constraint requires that the nonzero entry of each anchor genotype is exactly one. In this scenario, there exists at least one individual from each ancestral population whose entire genome was inherited by a single ancestral population. We summarize these results in the following corollary and visualize the anchor SNP and anchor genotype scenarios in Figure 1.

Corollary 1.

Whenever a rank d matrix $\mathit{F}$ admits a factorization $\mathit{F}=\mathit{PQ}$ such that there are either a set of anchor SNPs or a set of anchor genotypes, the factorization is unique up to permutation.

Appendix B: tALS and cALS Comparisons

Figure 9 displays the $\widehat{\mathit{Q}}$ estimates of the tALS and cALS algorithms on simulated data from each of the four α-prototypes described in Simulated data sets. For each of these datasets, $m={10}^5$, $n=500$, $d=3$. We see that estimates provided by the tALS and cALS algorithms agree very well with each other for all α-prototypes. However, the run times are substantially different between these two methods, as displayed in Table 4: tALS terminates in minutes while cALS terminates in hours. Notably, the run times of the tALS algorithm also appear to be less sensitive to the α-prototype than the cALS algorithm. Most notably, the cALS algorithm takes an order of magnitude longer to run on the ${\alpha }_4$ prototype than any of the other α-prototypes. These observations support our preference for the tALS algorithm over the cALS algorithm.

Figure 9.

Biplots of the first two rows of $\mathit{Q}$ (top), ${\widehat{\mathit{Q}}}_{\text{tALS}}$ (middle) and ${\widehat{\mathit{Q}}}_{\text{cALS}}$ (bottom) for each of the four α-prototypes.

Under ${\alpha }_2$ and ${\alpha }_3$, the estimates provided by tALS and cALS also agree very well with the true $\mathit{Q}$ matrices. This is not the case under ${\alpha }_1$ and ${\alpha }_4$, where both algorithms provide substantially different results than the ground truth. However, because both of these α-prototypes lack a complete set of anchor SNPs, the model may well be unidentifiable.

Appendix C: Simulation Details

Due to time and computational constraints, each algorithm did not terminate on each of the 96 datasets generated for the simulations. In all, 326 of the 384 total simulations terminated during the 1 week time limit with a budget of 300 GB. Admixture completed 81 simulations, fastSTRUCTURE completed 80 simulations terastructure completed 81 simulations, and ALStructure completed 84 simulations. For the sake of comparison, Figure 5 only shows the datasets for which all algorithms terminated.

Appendix D: HGDP, TGP, and HO Dataset Details

In Applications to Global Human Studies we analyze human genotype data from globally sampled individuals. These data come from three public sources: HGDP (Cavalli-Sforza 2005), TGP (The 1000 Genomes Project Consortium et al. 2015), and HO (Lazaridis et al. 2014). The various preprocessing steps are detailed below for each dataset.

TGP:

The 1000 Genomes Project dataset (TGP) samples globally from 26 populations and is available here: ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/release/20130502/supporting/hd_genotype_chip/. Related individuals and SNPs with minor allele frequency $<5\text{\%}$ are removed. The dimensions of this dataset are 1716 individuals and 520,036 SNPs.

HGDP:

The HGDP samples globally from 51 populations and is available here: http://www.hagsc.org/hgdp/files.html. Individuals with first- or second-degree relatives and SNPs with minor allele frequency $<5\text{\%}$ are removed. The dimensions of this dataset are 940 individuals and 550,303 SNPs.

HO:

The Affymetrix HO dataset samples globally from 147 populations and is available here: http://genetics.med.harvard.edu/reich/Reich_Lab/Datasets.html. Nonhuman or ancient samples and SNPs with $<5\text{\%}$ minor allele frequency are removed. The dimensions of this dataset are 2248 individuals and 372,446 SNPs.

Appendix E: Application to a Nonglobal Dataset

In this appendix we apply ALStructure and preexisting methods to a dataset from Basu et al. (2016). In this dataset, individuals from 18 mainland Indian subpopulations are sampled. Following Basu et al. (2016), we set $d=4$ for each method. Figure 10 plots the first two rows of $\mathit{Q}$ output from Admixture, fastSTRUCTURE, terastructure, and ALStructure, respectively. As in the results from Applications to Global Human Studies, rows of $\mathit{Q}$ are ordered according to variation explained.

Figure 10.

Bi-plots of the first two rows of $\mathit{Q}$ (ranked by variation explained) of the fits of the Basu et al. (2016) dataset for each algorithm. Individuals are colored by the subpopulation from which they are sampled.

Figure 11.

The admixture proportions of each globally sampled dataset as a stacked barplot. Because of the nonidentifiability of the model, the order of the rows of $\mathit{Q}$ are arbitrary. To disambiguate this, we order the rows of $\mathit{Q}$ in each dataset by decreasing average admixture. The coloring in Figure 11 is then done according to this ordering. As an aid to the eye, we also reorder the columns of $\mathit{Q}$ according to decreasing proportion of the first row of $\mathit{Q}$ of the ALStructure fit. The choice to order all fits according to ALStructure is arbitrary; however, all fits must be ordered consistently to make meaningful comparisons possible. As can be seen, each of the fits differ significantly from each other on every dataset.

Figure 12.

Left panels: The distribution of likelihoods for each element of $\mathit{X}$ for each method and dataset. Right panels: The same as the left panels, except on a log-scale.

As can be seen, the estimated admixture proportions produced by each method are all qualitatively similar. Table 5 shows the likelihood of the data from each method, with each method performing similarly. The methods ranked by decreasing mean log-likelihood are: Admixture, ALStructure, fastSTRUCTURE, terastructure.

Table 4. RMSE between true and estimated $\mathit{Q}$ matrices for each method and each α-prototype (rows 1 and 2); RMSE between two estimated $\mathit{Q}$ matrices (row 3); run-time (rows 4 and 5).

	${\alpha }_1$	${\alpha }_2$	${\alpha }_3$	${\alpha }_4$
$\text{RMSE}\left(Q,{\widehat{Q}}_{\text{tALS}}\right)$	$6.2\times {10}^{-2}$	$1.4\times {10}^{-2}$	$8.7\times {10}^{-3}$	$1.9\times {10}^{-1}$
$\text{RMSE}\left(Q,{\widehat{Q}}_{\text{cALS}}\right)$	$6.3\times {10}^{-2}$	$1.5\times {10}^{-2}$	$8.5\times {10}^{-3}$	$2.2\times {10}^{-1}$
$\text{RMSE}\left({\widehat{Q}}_{\text{tALS}},{\widehat{Q}}_{\text{tALS}}\right)$	$4.2\times {10}^{-3}$	$2.3\times {10}^{-3}$	$5.7\times {10}^{-4}$	$3.9\times {10}^{-2}$
tALS run time	2.6 min	1.4 min	1.5 min	2.8 min
cALS run time	4.1 hr	3.0 hr	3.5 hr	36.8 hr

Table 5. Mean log-likelihood from of each method applied to data set Basu et al. (2016).

Method	Admixture	fastSTRUCTURE	terastructure	ALStructure
Mean log-likelihood	−0.7360	−0.7369	−0.7373	−0.7365