Coancestry superposed on admixed populations yields measures of relatedness at individual-level resolution

Abstract

The admixture model is widely applied to estimate and interpret population structure among individuals. Here we consider a “standard admixture” model that assumes the admixed populations are unrelated and also a generalized model, where the admixed populations themselves are related via coancestry (or covariance) of allele frequencies. The generalized model yields a potentially more realistic and substantially more flexible model that we call “super admixture”. This super admixture model provides a one-to-one mapping in terms of probability moments with the population-level kinship model, the latter of which is a general model of genome-wide relatedness and structure based on identity-by-descent. We introduce a method to estimate the super admixture model that is based on method of moments, does not rely on likelihoods, is computationally efficient, and scales to massive sample sizes. We apply the method to several human data sets and show that the admixed populations are indeed substantially related, implying the proposed method captures a new and important component of evolutionary history and structure in the admixture model. We show that the fitted super admixture model estimates relatedness between all pairs of individuals at a resolution similar to the kinship model. The super admixture model therefore provides a tractable, forward generating probabilistic model of complex structure and relatedness that should be useful in a variety of scenarios.

1. Introduction

Populations are structured when genotype frequencies do not follow Hardy-Weinberg proportions. This may be due to several factors, including finite population sizes, migration, and genetic drift [1, 2]. Our goal here is to develop a framework and estimation method of a forward generating probability process that captures the observed genetic structure and relatedness among a set of individuals in a population-based study.

The framework is based on covarying allele frequencies among populations [3] and individuals [4], which we will refer to as coancestry [3–5]. The data underlying the proposed method are single nucleotide polymorphism (SNP) genotypes measured throughout the genome on a set of individuals. The aim is to formulate and estimate a model of the underlying process that leads to individual-specific allele frequencies (IAFs), which are parameters consisting of possibly distinct allele frequencies for every individual-SNP pair. IAFs have been formulated in previous work [6, 7] and they are the estimation target in several established admixture methods [8–10], a genome-wide association test for structured populations [11], and a test of structural Hardy-Weinberg equilibrium [12].

A joint probability distribution of the IAFs under a neutral model has been developed that yields covariances for all pairs of IAFs, parameterized by ancestral allele frequencies and coancestry parameters [4, 5]. This model produces a one-to-one mapping with the kinship parameters from the identity-by-descent model [13, 14], excluding close familial genetic relationships. This coancestry model therefore captures pairwise individual-level structure and relatedness equivalent to the kinship model. However, similarly to the kinship model, the coancestry parameterization is in terms of expected values, variances, and covariances of the IAFs and genotypes. It does not explicitly define a forward-generating probability model of IAFs.

Admixture models have been explored as a possible way to define such a forward-generating probability model [4, 5]. The products of an admixture model are individual-specific admixture proportions and population-specific allele frequencies. The IAFs are modeled as a weighted average of these antecedent population allele frequencies by the individual-specific admixture proportions. Several methods treat the admixture proportions and antecedent population allele frequencies as unknown parameters without explicitly making any assumptions about their random distributions [8–10]. Other methods place a prior probability distribution on them for Bayesian modeling fitting purposes [15–17]; however, these Bayesian methods do not include these prior distributions as an inference target.

In considering a model of random antecedent population allele frequencies, one could assume that the allele frequencies are independently generated among all antecedent populations based on a common set of parameters (e.g., independent draws from the Balding-Nichols distribution [18]). We will call this assumption the “standard admixture” model. However, this standard admixture model may be overly restrictive; rather, one could implement a coancestry model of the antecedent allele frequencies according to pairwise covariances [4, 5]. We will call this model the “super admixture” model, as coancestry (or covariance) is superposed on the admixed antecedent populations. Fig. 1 displays a schematic of these models.

Figure 1:

Graphical representations of the coancestry model, the standard admixture model, and the super admixture model. (A) In the coancestry model, individuals in the present-day population are connected by a complex genealogy. (B) In the standard admixture model, the arrows connecting $T$ with $S_1,\dots ,S_K$ reflect that the antecedent populations evolved independently from $T$. Arrows connecting $S_1,\dots ,S_K$ with individuals in the present-day population reflect that these individuals were admixed from independent antecedent populations. (C) In the super admixture model, dashed lines connecting all pairs of antecedent populations reflect that antecedent populations have coancestry parameterized by $\mathit{\Lambda }$. Arrows connecting $S_1,\dots ,S_K$ with individuals in the present-day population reflect that these individuals were admixed from covarying antecedent populations.

Here, we develop a method that estimates the parameters in the super admixture model, which includes the standard admixture model as a special case. The method is based on method of moments estimation and geometric considerations, so it does not make assumptions about the probability distributions of the parameters and it does not involve costly likelihood maximization computations. Likelihood maximization is the most common approach used in fitting the admixture models [8, 9, 15–17], but we build from a recently proposed distribution-free moment-based method, called ALStructure, that only uses linear projections and geometric constraints on parameters to estimate the model [10]. ALStructure performs favorably to likelihood based methods (even in achieving a high likelihood) and can be tractably scaled to massive data sets. Our proposed super admixture method complements this framework and has similar advantages.

We establish super admixture through computational studies and analyses of data sets, including the human genome diversity panel (HGDP) [19], the 1000 genomes project (TGP) [20], the Human Origins study (HO) [21], and a study on individuals with Inadian ancestry (IND) [22]. We show on all of these data sets that the super admixture method is capable of capturing the same relatedness and structure as a model-free individual-level coancestry estimator [4], whereas the standard admixture model does not. We demonstrate that the framework can generate bootstrap genotypes that retain the structure seen in the human studies. For example, Fig. 2 shows these results on the HO study. We show that the coancestry among antecedent populations estimated by super admixture yields new insights and visualizations of structure previously unavailable, for example, Fig. 3 on the HO study. We develop and perform a statistical test to demonstrate on the studies that coancestry among the admixed antecedent populations is statistically different from zero to an high degree of significance.

Figure 2:

Heatmaps of individual-level coancestry estimates in the HO data set.

Figure 3:

(A) Heatmap of antecedent population coancestry estimates in the HO data set. (B) Dendrogram representation of the antecedent population coancestry estimates. (C) Stacked bar plot of admixture proportions.

Our proposed framework makes several contributions: (i) a distribution-free framework that can account for arbitrarily complex relationships among the admixed antecedent populations in the admixture model; (ii) admixture-based estimation of individual-level pairwise coancestry at a resolution equivalent to general, model-free coancestry and kinship; (iii) a partitioning of the super admixture model into evolutionary, genealogical, and statistical sampling components; and (iv) a tractable algorithm to form bootstrap samples of genotypes from the estimated evolutionary process.

2. Super admixture framework

Here, we first introduce the data and models, and then we detail the proposed framework. We describe how the framework is used to estimate the super admixture model, generate parameters and data from the model, and perform a hypothesis test of the standard versus super admixture models.

2.1. Coancestry

We assume that $m$ SNPs are measured on $n$ individuals. The genotype measurements are denoted by $x_{ij}$ for $i=1,\dots ,m$ and $j=1,\dots ,n$. For each SNP, one of the alleles is counted as a 0 and the other as a 1, implying that the SNP genotypes are $x_{ij}\in \{0,1,2\}$ where $x_{ij}=0$ is homozygous for the 0 allele, $x_{ij}=1$ is a heterozygote, and $x_{ij}=2$ is homozygous for the 1 allele. We assume that $\mathbb{E}\left[x_{ij}\mid {\pi }_{ij}\right]=2{\pi }_{ij}$ for IAF ${\pi }_{ij}$. This IAF parameterization allows each individual-SNP pair to possibly have a distinct allele frequency. The classical scenario where there is one allele frequency per SNP is a special case where ${\pi }_{i1}={\pi }_{i2}=\cdots ={\pi }_{in}$. The conditional expected value $\mathbb{E}\left[x_{ij}\mid {\pi }_{ij}\right]=2{\pi }_{ij}$ also allows for the IAFs ${\pi }_{ij}$ to be random parameters, which we assume here.

We utilize an existing coancestry model where the IAFs are random parameters with respect to some ancestral population $T$ that is common to all $n$ individuals [4, 5]. This is a neutral model where

$$\mathbb{E}\left[{\pi }_{ij}\mid T\right]=a_i$$ (1)

$$\mathbb{C}\left[{\pi }_{ij},{\pi }_{ik}\mid T\right]=a_i\left(1-a_i\right){\theta }_{jk}$$ (2)

for $i=1,\dots ,m$ and $j,k=1,\dots ,n$. The parameter $a_i$ is the ancestral allele frequency in $T$ for SNP $i$ and $0\le {\theta }_{jk}\le 1$ is the coancestry for individuals $j$ and $k$ with respect to $T$. (Note that the $a_i$ and ${\theta }_{jk}$ parameters depend on $T$ and could be different if conditioning on a different common ancestral population.) The coancestry model we utilize also makes the assumption used in previous work [4, 5, 7–12] that

$$x_{ij}\mid {\pi }_{ij}~\text{Binomial}\left(2,{\pi }_{ij}\right)$$

where the $x_{ij}$ are jointly independent. Under this model, it follows that

$$\mathbb{C}\left[x_{ij},x_{ik}\mid T\right]=\left\{\begin{array}{ll}2a_i\left(1-a_i\right)\left(1+{\theta }_{jj}\right)& j=k,\\ 4a_i\left(1-a_i\right){\theta }_{jk}& j\ne k.\end{array}\right.$$

A one-to-one mapping exists with the identity-by-descent kinship model (often used in GWAS methods), denoted by ${\varphi }_{jk}$, by matching variances and covariances [4, 5]. The parameters map so that

$${\theta }_{jk}=\left\{\begin{array}{ll}2{\varphi }_{jk}-1& \text{if}j=k,\\ {\varphi }_{jk}& \text{if}j\ne k.\end{array}\right.$$ (3)

When ${\text{min}}_{jk}{\theta }_{jk}=0$, then $T$ is the most recent common ancestral population [4]. The full set of parameters is denoted by the $n\times n$ symmetric matrix $\mathit{\Theta }$ with $(j,k)$ entry ${\theta }_{jk}$.

2.2. Admixture models

General admixture

We first describe a general formulation of the admixture model, of which standard and super admixture are special cases. There are $K$ populations $S_1,S_2,\dots ,S_K$ descended from $T$ that precede the present day population, which we refer to as “antecedent populations”. While $T$ has allele frequencies $a_1,a_2,\dots ,a_m$, antecedent population $S_u$ has allele frequencies $p_{1u},p_{2u},\dots ,p_{mu}$ for $u=1,2,\dots ,K$. The allele frequencies $\left\{p_{iu}\right\}$ are random parameters from a distribution parameterized by $\left\{a_i\right\}$ plus other possible parameters that characterize the evolutionary process from $T$ to $S_u$.

For each individual $j$, there is a genealogical process from population $T$ to the present day population. This is captured by a random $K$-vector $q_{1j},q_{2j},\dots ,q_{Kj}$ of admixture proportions, where $0\le q_{uj}\le 1$ and $\sum _{u=1}^K{q}_{uj}=1$. The parameter $q_{uj}$ is the proportion of the individual $j$ randomly descended from $S_u$. Therefore, the IAFs are such that

$${\pi }_{ij}=\sum _{u=1}^K{p}_{iu}{q}_{uj}.$$ (4)

We collect the antecedent population allele frequencies into the $m\times K$ matrix $\mathit{P}$ and the admixture proportions into the $K\times n$ matrix $\mathit{Q}$, it follows that

$$\mathbf{\Pi }=\mathit{P}\mathit{Q},$$

where $\mathbf{\Pi }$ is an $m\times n$ matrix with $(i,j)$ entry ${\pi }_{ij}$.

Standard admixture

We define the standard admixture model to be the case where the antecedent allele frequencies are independently distributed. Specifically, in this model $p_{iu}$ is a random parameter with mean $a_i$ and variance $a_i\left(1-a_i\right)f_u$. The standard admixture model is defined as follows for $i=1,2,\dots ,m$ and $u=1,2,\dots ,K$.

$$\begin{gathered} \text{Standard}\text{Admixture}: \\ {p}_{i1},p_{i2},\dots ,p_{iK}\text{are}\text{jointly}\mathit{\text{independent}} \\ \mathbb{E}\left[p_{iu}\mid T\right]=a_i \\ \mathbb{V}\left[p_{iu}\mid T\right]=a_i\left(1-a_i\right)f_u \end{gathered}$$

Under this parameterization, $a_i$ is the ancestral allele frequency in $T$ and $f_u$ is the inbreeding coefficient or $F_{\text{ST}}$ of antecedent population $S_u$ with respect to $T$. Since the $\left\{p_{iu}\right\}$ are jointly independent, there is no coancestry among antecedent populations and there is no dependence among loci.

One well-known distribution that could be utilized here is the Balding-Nichols (BN) distribution [18] with parameters $a_i$ and $f_u$:

$$p_{iu}~\text{Beta}\left(\frac{1-f_u}{f_u}{a}_i,\frac{1-f_u}{f_u}\left(1-a_i\right)\right).$$ (5)

We will write this re-parameterized Beta distribution as $\text{BN}\left(a_i,f_u\right)$. This achieves the expected value and variance of the standard admixture definition. The Balding-Nichols distribution is often used to generate allele frequencies for a set of populations to achieve desired expected allele frequencies and $F_{\text{ST}}$ values. This distribution has been discussed as useful for generating antecedent allele frequencies in the standard admixture model [4–7, 23].

Super admixture

The super admixture model extends the standard admixture model in that it includes a covariance among antecedent population allele frequencies, which we refer to as population-level coancestry. While we denoted individual-level coancestry by ${\theta }_{jk}$, we will denote population-level coancestry by ${\lambda }_{uv}$ for $u,v=\mathrm{1,2},\dots ,K$ where $0\le {\lambda }_{uv}\le 1$. We collect these values into the $K\times K$ symmetric coancestry matrix $\mathit{\Lambda }$. The super admixture model is defined as follows for $i=1,2,\dots ,m$ and $u,v=1,2,\dots ,K$.

$$\begin{gathered} \text{Super}\text{Admixture}: \\ {p}_{i1},p_{i2},\dots ,p_{iK}\text{are}\text{jointly}\mathit{\text{dependent}} \\ \mathbb{E}\left[p_{iu}\mid T\right]=a_i \\ \mathbb{V}\left[p_{iu}\mid T\right]=a_i\left(1-a_i\right){\lambda }_{uu} \\ ℂ\left[p_{iu},p_{iv}\mid T\right]=a_i\left(1-a_i\right){\lambda }_{uv} \end{gathered}$$ (6)

In this model we assume that allele frequencies between loci are independent, so the random $K$-vectors $\left(p_{h1},p_{h2},\dots ,p_{hK}\right)$ and $\left(p_{i1},p_{i2},\dots ,p_{iK}\right)$ are independent for $h\ne i$. Thus, a potential generalization of the super admixture model is to include dependence among loci. Otherwise, the super admixture model is general in that it allows for the full range of coancestry values among antecedent populations.

Forward generating probability process

We now describe the super admixture model as a forward generating probability process. Suppose that the admixture proportions $\mathit{Q}$ are drawn from some probability distribution $\mathcal{Q}$. Then, for $i=1,2,\dots ,m$ and $j=1,2,\dots ,n$:

$$\begin{gathered} \left(p_{i1},p_{i2},\dots ,p_{iK}\right)~(\mathit{a},\mathit{\Lambda }) \\ \left(q_{1j},q_{2j},\dots ,q_{Kj}\right)~𝒬 \\ {\pi }_{ij}=\sum _{u=1}^K{p}_{iu}{q}_{uj} \\ {x}_{ij}\mid {\pi }_{ij}~\text{Binomial}\left(2,{\pi }_{ij}\right) \end{gathered}$$

The joint probability of all random quantities can be factored as follows:

$$\mathbb{P}\left(\mathit{X},\mathit{Q},\mathit{P}∣T,𝒬\right)=\mathbb{P}\left(\mathit{P}∣T\right)\mathbb{P}\left(\mathit{Q}∣𝒬\right)\mathbb{P}\left(\mathit{X}∣\mathit{P},\mathit{Q}\right).$$

One interpretation of this is that $\mathbb{P}(\mathit{P}\mid T)$ represents evolutionary sampling, $\mathbb{P}(\mathit{Q}\mid 𝒬)$ represents genealogical sampling, and $\mathbb{P}(\mathit{X}\mid \mathit{P},\mathit{Q})$ represents statistical sampling.

Individual-level coancestry in the admixture models

Recall that in the covariance model, the covariance of two IAFs for a given SNP is $\mathbb{C}\left[{\pi }_{ij},{\pi }_{ik}\mid T\right]=a_i\left(1-a_i\right){\theta }_{jk}$, shown in Eq. (2). Conditioning on the admixture proportions $\mathit{Q}$, which are ancillary to allele frequencies, this covariance under the super admixture model is, for $j,k=1,2,\dots ,n$,

$$\begin{gathered} \mathbb{C}\left[{\pi }_{ij},{\pi }_{ik}\mid \mathit{Q},T\right]=\mathbb{C}\left[\sum _{u=1}^K{p}_{iu}{q}_{uj},\sum _{v=1}^K{p}_{iv}{q}_{vk}\mid \mathit{Q},T\right] \\ =\sum _{u=1}^K\sum _{v=1}^K{q}_{uj}{q}_{vk}\mathbb{C}\left[p_{iu},p_{iv}\mid T\right] \\ =a_i\left(1-a_i\right)\sum _{u=1}^K\sum _{v=1}^K{q}_{uj}{q}_{vk}{\lambda }_{uv}. \end{gathered}$$ (7)

By setting the covariance from Eq. (2) equal to Eq. (7), it follows that under the super admixture model the individual-level coancestry is the following.

$$\begin{gathered} \text{Super}\text{Admixture}\text{Individual-level}\text{Coancestry}: \\ {\theta }_{jk}=\sum _{u=1}^K\sum _{v=1}^K{q}_{uj}{q}_{vk}{\lambda }_{uv} \end{gathered}$$ (8)

In the standard admixture model, $\mathbb{V}\left[p_{iu}\mid T\right]=a_i\left(1-a_i\right)f_u$, whereas in the super admixture model $\mathbb{V}\left[p_{iu}\mid T\right]=a_i\left(1-a_i\right){\lambda }_{uu}$. If we set $f_u={\lambda }_{uu}$, the difference between the standard and super admixture models is therefore that in the standard model, ${\lambda }_{uv}=0$ for $u\ne v$. To work with a single notation, we will therefore write ${\lambda }_{uu}$ in place of $f_u$ for the standard admixture model. The coancestry in this model is as follows.

$$\begin{gathered} \text{Standard}\text{Admixture}\text{Individual-level}\text{Coancestry}: \\ {\theta }_{jk}=\sum _{u=1}^K{q}_{uj}{q}_{uk}{\lambda }_{uu} \\ {\lambda }_{uv}=0\text{for}u\ne v \end{gathered}$$ (9)

Considering all pairs of individuals simultaneously, the individual-level coancestry matrix $\mathbf{\Theta }$ can be written in terms of the antecedent population-level coancestry $\mathit{\Lambda }$ and the admixture proportions $\mathit{Q}$ as

$$\mathbf{\Theta }={\mathit{Q}}^{\prime }\mathit{\Lambda }\mathit{Q},$$

which is an important relationship we utilize to estimate $\mathit{\Lambda }$.

2.3. Estimating coancestry among antecedent populations

Here, we propose a method to estimate the antecedent population-level coancestry $\mathit{\Lambda }$ under the super admixture model, with the standard admixture model estimate as a special case. The rationale is to leverage the relationship, $\mathbf{\Theta }={\mathit{Q}}^{\prime }\mathit{\Lambda }\mathit{Q}$. Given values for $\mathbf{\Theta }$ and $\mathit{Q}$, we identify values of $\mathit{\Lambda }$ that make ${\mathit{Q}}^{\prime }\mathit{\Lambda }\mathit{Q}$ close to $\mathbf{\Theta }$, while obeying the geometic constraints of $\mathit{\Lambda }$ (i.e., $0\le {\lambda }_{uv}\le 1$ and ${\lambda }_{uv}={\lambda }_{vu}$).

Given values for $\mathbf{\Theta }$ and $\mathit{Q}$, we formulate the problem of the estimating the antecedent population-level coancestry $\mathit{\Lambda }$ under the super admixture model as follows.

Problem 1.

$$\begin{gathered} \underset{\mathit{\Lambda }\in {\mathbb{R}}^{K\times K}}{\text{min}}{‖\mathbf{\Theta }-{\mathit{Q}}^{\prime }\mathit{\Lambda }\mathit{Q}‖}_F^2 \\ \text{subject}\text{to}:0\le {\lambda }_{uv}\le 1\text{and}{\lambda }_{uv}={\lambda }_{vu} \\ \text{for}u,v=1,\dots ,K \end{gathered}$$

where $\Vert \cdot {\Vert }_F$ represents the Frobenius norm defined in Appendix A.1. We utilize the proximal forward-backward (PFB) method [24] to solve this optimization problem, resulting in Algorithm 1 for solving Problem 1. Every sequence of ${\left({\mathit{\Lambda }}_t\right)}_{t\in \mathbb{N}}$ generated from this algorithm is guaranteed to converge to a solution of the corresponding problem. The PFB method and how to employ it to our setting are detailed in Appendix B.2. The performance of Algorithm 1 is demonstrated in Appendix C.

Algorithm 1:

Estimating $\mathit{\Lambda }$ for the super admixture model given $\mathbf{\Theta }$ and $\mathit{Q}$

	input: Coancestry matrix $\mathit{\Theta }$ and admixture proportions matrix $\mathit{Q}$
1	let $L={\sigma }_{\text{max}}^4(\mathit{Q})$
2	let ${\mathit{\Lambda }}_0\leftarrow {\left(\mathit{Q}{\mathit{Q}}^{\prime }\right)}^{-1}\mathit{\Theta }{\left(\mathit{Q}{\mathit{Q}}^{\prime }\right)}^{-1}$
3	for $t=1,2,\dots$ do
4	$\mathit{G}\leftarrow 2\mathit{Q}\left({\mathit{Q}}^{\prime }{\mathit{\Lambda }}_{t-1}\mathit{Q}-\mathit{\Theta }\right){\mathit{Q}}^{\prime }$
5	${\mathit{\Lambda }}^{*}\leftarrow {\mathit{\Lambda }}_{t-1}-\frac{1}{L}\mathit{G}$
6	${\mathit{\Lambda }}_t=\left\{{\lambda }_{uv,t}\right\}\text{where}{\lambda }_{uv,t}=\text{max}\left(0,\text{min}\left(1,{\lambda }_{uv}^{*}\right)\right)$
7	return ${\mathit{\Lambda }}_t$

${\sigma }_{\text{max}}(\cdot )$ denotes the maximum singular value (Appendix A.1).

To estimate all components of the super admixture model, one needs estimates of the $n\times n$ individual-level coancestry matrix $\mathbf{\Theta }$, the $K\times K$ antecedent population-level coancestry matrix $\mathit{\Lambda }$, the $m\times K$ matrix of antecedent population allele frequencies $\mathit{P}$, and the $K\times n$ matrix of admixture proportions $\mathit{Q}$. There exists a wide range of methods for estimating $\mathit{P}$ and $\mathit{Q}$ [9, 10, 15, 17, 25]. Here, we utilize the ALStructure method [10], which implements method of moments and geometric constraints to estimate $\mathit{Q}$ similarly to our approach here. In that method, a linear basis of $\mathit{Q}$ is determined from $\mathit{X}$ that has theoretical guarantees to span the true basis as the number of SNPs $m$ grows large. A projection-based estimate $\stackrel{\mathbf{ˆ}}{\mathbf{\Pi }}$ of the IAFs is also formed. The quantity $\Vert \stackrel{\mathbf{ˆ}}{\mathbf{\Pi }}-\mathit{Q}\mathit{P}{\Vert }_F$ is then algorithmically minimized through geometrically constrained alternating least squares to yield estimates $\stackrel{\mathit{ˆ}}{\mathit{Q}}$ and $\stackrel{\mathit{ˆ}}{\mathit{P}}$.

We utilize the structural Hardy-Weinberg (sHWE) test [12] for determining the number of antecedent populations $K$, as outlined in that work. The approach is to consider a range of $K$ values to test the assumption that $x_{ij}\mid {\pi }_{ij}~\text{Binomial}\left(2,{\pi }_{ij}\right)$ based on the estimates ${\hat{\pi }}_{ij}$ and a goodness-of-fit statistic with a parametric bootstrap null distribution; $K$ is then parsimoniously chosen to satisfy this modeling assumption from a genome-wide perspective. A method of moments estimator of $\mathbf{\Theta }$ was derived in ref. [4], where it was shown to have favorable properties and is consistent for the true values under certain assumptions. We denote this Ochoa-Storey (OS) estimate by ${\stackrel{\mathbf{ˆ}}{\mathbf{\Theta }}}^{\text{OS}}$ and review its details in Appendix B.1. If one has alternative ways to estimate $\mathbf{\Theta }$ and $\mathit{Q}$, and to determine $K$, then those can be used within our framework as well.

Algorithm 2:

Estimating $\mathit{\Lambda }$ for the super admixture model given $\mathit{X}$

	input: Genotype matrix $\mathit{X}$
1	calculate the OS estimate of individual-level coancestry ${\widehat{\mathit{\Theta }}}^{\text{OS}}$
2	choose $K$ from the structural Hardy-Weinberg (sHWE) goodness of fit procedure
3	calculate the estimate $\widehat{\mathit{Q}}$ for $K$ via the ALStructure method
4	calculate the estimate ${\widehat{\mathit{\Lambda }}}^{\text{sup}}$ by applying Algorithm 1 with inputs ${\widehat{\mathit{\Theta }}}^{\text{OS}}$ and $\widehat{\mathit{Q}}$
5	return ${\widehat{\mathit{\Lambda }}}^{\text{sup}}$

Note that one can further calculate a corresponding estimate for individual-level coancestry by

$${\stackrel{\mathbf{ˆ}}{\mathbf{\Theta }}}^{\text{sup}}={\stackrel{\mathit{ˆ}}{\mathit{Q}}}^{\prime }{\stackrel{\mathbf{ˆ}}{\mathit{\Lambda }}}^{\text{sup}}\stackrel{\mathit{ˆ}}{\mathit{Q}},$$

which can be compared to ${\stackrel{\mathbf{ˆ}}{\mathbf{\Theta }}}^{\text{OS}}$ in order to aid in model fit assessment.

We can estimate $\mathit{\Lambda }$ under the standard admixture model by modifying the constraints in Problem 1. This leads to Problem B. 1 and Algorithm B. 2 described in Appendix B. 2. Algorithm 2 can then be used to form the estimate ${\hat{\mathit{\Lambda }}}^{\text{std}}$ under the standard admixture model with Algorithm 1 in Line 4 replaced by Algorithm B.2. The corresponding estimate for individual-level coancestry can be calculated as ${\stackrel{\mathbf{ˆ}}{\mathbf{\Theta }}}^{\text{std}}={\stackrel{\mathit{ˆ}}{\mathit{Q}}}^{\prime }{\stackrel{\mathbf{ˆ}}{\mathit{\Lambda }}}^{\text{std}}\stackrel{\mathit{ˆ}}{\mathit{Q}}$. The performance of Algorithm B. 2 is also demonstrated in Appendix C.

2.4. Simulating antecedent population allele frequencies

We now introduce a method to generate antecedent population allele frequencies with given coancestry $\mathit{\Lambda }$. We noted above in Eq. (5) that for the standard admixture model, one way to generate allele frequencies $p_{i1},p_{i2},\dots ,p_{iK}$ is via independent realizations from the Balding-Nichols (BN) distribution: $p_{iu}~\text{BN}\left(a_i,{\lambda }_{uu}\right)$ for $u=1,2,\dots ,K$. As there is no default approach to extending this to the super admixture case, we propose a method here called “double-admixture”. The main idea of the method is that we form two layers of allele frequencies: the first layer is composed of independent draws from the BN distribution, and the second layer mixes these to create $p_{i1},p_{i2},\dots ,p_{iK}$ with coancestry $\mathit{\Lambda }$.

Let $S$ be the number of components that will be mixed, $\mathit{W}$ be the $S\times K$ matrix of mixture proportions, and $\mathit{\Gamma }$ an $S\times S$ diagonal matrix. The entries of $\mathit{W}$ are $w_{su}$ where $0\le w_{su}\le 1$ and $\sum _{s=1}^S{w}_{su}=1$ for $u=1,2,\dots ,K$. The diagonal values of $\mathit{\Gamma }$ are represented by ${\gamma }_s$ where $0\le {\gamma }_s\le 1$, and all other values are 0. Suppose that for $i=1,\dots ,m$ we generate

$$z_{is}~\text{BN}\left(a_i,{\gamma }_s\right)$$

independently for $s=1,\dots ,S$, and we then set

$$p_{iu}=\sum _{s=1}^S{z}_{is}{w}_{su}$$

for $u=1,\dots ,K$. It can be verified that

$$\begin{gathered} \mathbb{E}\left[p_{iu}\right]=a_{i}u=1,\dots ,K \\ ℂ\left[p_{iu},p_{iv}\right]=a_i\left(1-a_i\right)\sum _{s=1}^S{w}_{su}{w}_{sv}{\gamma }_s \end{gathered}$$

for $u,v=1,2,\dots ,K$. By matching these equations with Eq. (6), one can see that if

$${\lambda }_{uv}=\sum _{s=1}^S{w}_{su}{w}_{sv}{\gamma }_s$$ (10)

then $p_{i1},p_{i2},\dots ,p_{iK}$ has coancestry $\mathit{\Lambda }$ as desired. In matrix terms, Eq. (10) is equivalent to

$$\mathit{\Lambda }={\mathit{W}}^{\prime }\mathit{\Gamma }\mathit{W}.$$ (11)

Therefore, the double-admixture method is based on the following optimization problem.

Problem 2.

$$\begin{gathered} \underset{\mathit{W},\mathit{\Gamma }}{\text{min}}{‖\mathit{\Lambda }-{\mathit{W}}^{\prime }\mathit{\Gamma }\mathit{W}‖}_F^2 \\ \text{subject}\text{to}:0\le w_{su}\le 1,\sum _{s=1}^S{w}_{su}=1 \\ ϵ\le {\gamma }_s\le 1-ϵ\text{for}\text{small}ϵ>0 \\ \text{for}u=1,2,\dots ,K;s=1,2,\dots ,S \end{gathered}$$

Algorithm 3:

Calculating $\mathit{W}$ and $\mathit{\Gamma }$ in the double-admixture method

	input: Antecedent populations coancestry $\mathit{\Lambda }$, number of BN distributions $S$, step size parameters ${\tau }_1$ and ${\tau }_2$, and a small positive number $ϵ$
1	let ${\mathit{\Gamma }}_0$ be an $S\times S$ diagonal matrix with diagonal elements drawn independently from Uniform(0,1)
2	let ${\mathit{W}}_0$ be an $S\times K$ matrix whose columns ${\left(w_{1u},w_{2u},\dots ,w_{Su}\right)}^{\prime }$ are drawn independently from Dirichlet(l)
3	for $t=1,2,\dots$ do
4	$L_1\leftarrow 4\left(\Vert \mathit{\Lambda }{\Vert }_2{\Vert {\mathit{\Gamma }}_{t-1}\Vert }_2+3K{\Vert {\mathit{\Gamma }}_{t-1}\Vert }_2^2\right)$
5	${\mathit{G}}_1\leftarrow -4{\mathit{\Gamma }}_{t-1}{\mathit{W}}_{t-1}\left(\mathit{\Lambda }-{\mathit{W}}_{t-1}^{\prime }{\mathit{\Gamma }}_{t-1}{\mathit{W}}_{t-1}\right)$
6	${\mathit{W}}^{*}\leftarrow {\mathit{W}}_{t-1}-\frac{1}{{\tau }_1{L}_1}{\mathit{G}}_1$
7	for $u=1,\dots ,K$ do
8	${\mathit{w}}_{u,t}\leftarrow {𝒫}_{\Delta }\left({\mathit{w}}_u^{*}\right)$ wher ${\mathit{w}}_{u,t}$ and ${\mathit{w}}_u^{*}$ are the corresponding columns of ${\mathit{W}}_t$ and ${\mathit{W}}^{*}$
9	$L_2\leftarrow 2{\Vert {\mathit{W}}_t\Vert }_2^4$
10	${\mathit{G}}_2\leftarrow -2{\mathit{W}}_t\left(\mathit{\Lambda }-{\mathit{W}}_t^{\prime }{\mathit{\Gamma }}_{t-1}{\mathit{W}}_t\right){\mathit{W}}_t^{\prime }$
11	${\mathit{\Gamma }}^{*}\leftarrow {\mathit{\Gamma }}_{t-1}-\frac{1}{{\tau }_2{L}_2}{\mathit{G}}_2$
12	${\gamma }_{s,t}=\text{max}\left(ϵ,\text{min}\left(1-ϵ,{\gamma }_{ss}^{*}\right)\right)\text{for}s=1,2,\dots ,S$
13	${\mathit{\Gamma }}_t=\text{diag}\left({\gamma }_{1,t},{\gamma }_{2,t},\dots ,{\gamma }_{S,t}\right)$
14	return ${\mathit{\Gamma }}_t$ and ${\mathit{W}}_t$

Here, we set $S=2K,{\tau }_1={\tau }_2=1.1,ϵ=0.01$; user should investigate their choices. $\Vert \cdot {\Vert }_2$ denotes the spectral norm and ${𝒫}_{\Delta }$ denotes projection onto the unit simplex (Appendix A.1).

We adapted the proximal alternating linearized minimization (PALM) method [26] to solve Problem 2, resulting in Algorithm 3 for calculating the parameters in the double-admixture method. Every sequence ${\left({\mathit{W}}_t,{\mathit{\Gamma }}_t\right)}_{t\in \mathbb{N}}$ generated from Algorithm 3 is guaranteed to converge to a critical point. Integrating Algorithm 3 with the generative steps for $p_{iu}$ described above, Algorithm 4 simulates antecedent population allele frequencies with the desired coancestry. In Appendix B.3, the PALM method is briefly introduced and the convergence of Algorithm 3 is proved.

Algorithm 4:

The double-admixture algorithm for simulating $\mathit{P}$

	input: Ancestral allele frequencies $\mathit{a}$, coancestry among antecedent populations $\mathit{\Lambda }$, other input arguments for Algorithm 3
1	calculate $\widehat{\mathit{\Gamma }}$ and $\widehat{\mathit{W}}$ using Algorithm 3
2	for $i=1,\dots ,m$ do
3	generate $z_{is}~\text{BN}\left(a_i,{\widehat{\gamma }}_s\right)$ independently for $s=1,2,\dots ,S$
4	set $p_{iu}\leftarrow {\sum }_{s=1}^S{z}_{is}{\widehat{w}}_{su}\text{for}u=1,2,\dots ,K$
5	return $\mathit{P}$

One possible drawback of the double-admixture method is that the approach relies on the existence of $\mathit{W}$ and $\mathit{\Gamma }$ so that $\mathit{\Lambda }={\mathit{W}}^{\prime }\mathit{\Gamma }\mathit{W}$. We do not currently have a theoretical guarantee for such $\mathit{W}$ and $\mathit{\Gamma }$ (although one may exist since $S$ can be made large). Therefore, we provide a complementary method in Appendix B.4, the NORmal To Anything (NORTA) approach [27], serving as a tool for simulating $\mathit{P}$ when the double-admixture method is not applicable. It should be noted that the double-admixture method solves the optimization one time for the entire process so that its running time is independent of the number of loci $m$. In contrast, the NORTA method has to solve $K\times (K-1)/2$ root-finding problems per locus and therefore has a complexity of $𝒪\left(K^{2}m\right)$, rendering it significantly more time consuming. The performances of the double-admixture and NORTA methods are demonstrated in Appendix C.

Note that if we set $\mathit{\Gamma }=\mathit{\Lambda }$ for a diagonal standard admixture $\mathit{\Lambda }$ and $\mathit{W}={\mathit{I}}_K$ (where ${\mathit{I}}_K$ is the $K\times K$ identity matrix), then the double-admixture method reduces to the BN sampling from Eq. (5), which produces valid antecedent population frequencies for the standard admixture model. From this observation, the double-admixture method can be viewed as a generalization of BN sampling.

2.5. Generating bootstrap datasets from realistic population structures

By utilizing the double-admixture method, we implemented the following algorithm to simulate genotypes from the super admixture model, shown in Algorithm 5. We assessed whether Algorithm 5 generates genotypes that satisfy the moment constraints imposed by the super admixture model in Appendix C. Algorithm 5 is especially useful when inputs $\mathit{a},\mathit{\Lambda }$, and $\mathit{Q}$ reflect real populations. When these parameters are unavailable one can utilize an admixture method to estimate $\mathit{Q}$ and the method proposed here to estimate $\mathit{\Lambda }$. The ancestral allele frequencies can be estimated with simple sample means. We outline Algorithm 6, with the ALStructure algorithm for estimation of $\mathit{Q}$ and the super admixture algorithm for estimation of $\mathit{\Lambda }$.

Algorithm 5:

Generating genotypes $\mathit{X}$ from the super admixture model

	input: Ancestral allele frequencies $\mathit{a}$, antecedent populations coancestry $\mathit{\Lambda }$, and admixture proportions $\mathit{Q}$
1	generate $\mathit{P}$ using Algorithm 4
2	let $\mathit{\Pi }=\mathit{P}\mathit{Q}$
3	let $\mathit{X}=\left\{x_{ij}\right\}$ by generating $x_{ij}\mid {\pi }_{ij}~\text{Binom}\left(2,{\pi }_{ij}\right)$ independently for $i=1,2,\dots ,m$ and $j=1,2,\dots ,n$
4	return $\mathit{X}$

Line 1 can also be completed with the NORTA method, Algorithm B.4.

Algorithm 6:

Generating bootstrap genotypes ${\mathit{X}}^{*}$ from observed genotypes $\mathit{X}$

	input: Genotype matrix $\mathit{X}$
1	let $\widehat{\mathit{a}}=\left\{{\widehat{a}}_i\right\}\text{where}{\widehat{a}}_i=\frac{1}{2n}{\sum }_{j=1}^n{x}_{ij}\text{for}i=1,2,\dots ,m$
2	obtain ${\widehat{\mathit{\Lambda }}}^{\text{sup}}$ and $\widehat{\mathit{Q}}$ from Algorithm 2 with input $\mathit{X}$
3	generate ${\mathit{P}}^{*}$ using Algorithm 4 with inputs $\widehat{\mathit{a}}$ and ${\widehat{\mathit{\Lambda }}}^{\text{sup}}$
4	let ${\mathit{\Pi }}^{*}={\mathit{P}}^{*}\widehat{\mathit{Q}}$
5	let ${\mathit{X}}^{*}=\left\{x_{ij}^{*}\right\}$ by generating $x_{ij}^{*}\mid {\pi }_{ij}^{*}~\text{Binom}\left(2,{\pi }_{ij}^{*}\right)$ independently for $i=1,2,\dots ,m$ and $j=1,2,\dots ,n$
6	return ${\mathit{X}}^{*}$

${\stackrel{\mathbf{ˆ}}{\mathit{\Lambda }}}^{\text{sup}}$ can be replaced with ${\stackrel{\mathbf{ˆ}}{\mathit{\Lambda }}}^{\text{std}}$ in Line 2, in which case the BN sampling from Eq. (5) is used in Line 3. Line 3 can also be completed with the NORTA method, Algorithm B.4, if using ${\stackrel{\mathbf{ˆ}}{\mathit{\Lambda }}}^{\text{sup}}$.

We note that Algorithm 6 is a semi-parametric bootstrap simulation; Line 3 is semiparametric, ${\mathbf{\Pi }}^{*}$ is semi-parametric because $\stackrel{\mathit{ˆ}}{\mathit{Q}}$ is nonparametric, and Line 5 is parametric. The output ${\mathit{X}}^{*}$ can be interpreted as a bootstrap replication of $\mathit{X}$, where the population structure in ${\mathit{X}}^{*}$ recapitulates the structure in $\mathit{X}$. The process that the bootstrap method recapitulates is not just resampled genotypes for a fixed matrix of estimated IAFs. Rather, the antecedent population allele frequencies are resampled, also leading to resampled IAFs, so both evolutionary and statistical resampling occur.

2.6. Significance test of coancestry among antecedent populations

Here, we develop a hypothesis test of the standard admixture model (null) versus the super admixture model (alternative). We show below that on real data sets the test results are highly significant against the null in favor of the alternative. In terms of model parameters, the test is defined as follows:

$$H_0:\text{max}\left({\left\{{\lambda }_{uv}\right\}}_{u\ne v}\right)=0\left(\text{standard}\text{admixture}\text{model}\right)$$

$$H_1:\text{max}\left({\left\{{\lambda }_{uv}\right\}}_{u\ne v}\right)>0\left(\text{super}\text{admixture}\text{model}\right)$$

A straightforward test-statistic is $U={‖{\hat{\mathit{\Lambda }}}^{\text{sup}}-{\hat{\mathit{\Lambda }}}^{\text{std}}‖}_F$. The larger $U$ is, the more evidence there is against the null hypothesis in favor of the alternative hypothesis. In order to calculate a $p$-value for this test-statistic, we need to know the distribution of $U$ when the null hypothesis is true. To this end, we adapt the bootstrap method of Algorithm 6, leading to Algorithm 7.

Algorithm 7:

Hypothesis test of no coancestry among antecedent populations

	input: Genotype matrix $\mathit{X}$ and number of bootstrap replications $B$
1	calculate ${\widehat{a}}_i=\frac{1}{2n}{\sum }_{j=1}^n{x}_{ij}\text{for}i=1,2,\dots ,m$
2	calculate estimates ${\widehat{\mathit{\Lambda }}}^{\text{std}}$, ${\widehat{\mathit{\Lambda }}}^{\text{sup}}$, and $\widehat{\mathit{Q}}$ by Algorithm 2 with input $\mathit{X}$
3	calculate the observed test-statistic $U={\Vert {\widehat{\mathit{\Lambda }}}^{\text{sup}}-{\widehat{\mathit{\Lambda }}}^{\text{std}}\Vert }_F$
4	for $b=1,2,\dots ,B$ do
5	generate $p_{iu}^{*}~\text{BN}\left({\widehat{a}}_i,{\widehat{\lambda }}_{uu}^{\text{std}}\right)$ independently and let ${\mathit{P}}^{*}=\left\{p_{iu}^{*}\right\}\text{for}i=1,2,\dots ,m$ and $u=1,2,\dots ,K$
6	let ${\mathit{\Pi }}^{*}={\mathit{P}}^{*}\widehat{\mathit{Q}}$
7	let ${\mathit{X}}^{*}=\left\{x_{ij}^{*}\right\}$ by generating $x_{ij}^{*}\mid {\pi }_{ij}^{*}~\text{Binom}\left(2,{\pi }_{ij}^{*}\right)$ independently for $i=1,2,\dots ,m$ and $j=1,2,\dots ,n$
8	calculate estimates ${\widehat{\mathit{\Lambda }}}^{\text{std*}}$ and ${\widehat{\mathit{\Lambda }}}^{\text{sup*}}$ by Algorithm 2 with input ${\mathit{X}}^{*}$
9	calculate the bootstrap null test-statistic $U^{*(b)}={\Vert {\widehat{\mathit{\Lambda }}}^{\text{sup*}}-{\widehat{\mathit{\Lambda }}}^{\text{std*}}\Vert }_F$
10	return $p-\text{value}=\frac{1}{B}{\sum }_{b=1}^{B}1\left(U^{*(b)}\ge U\right)$

To evaluate the validity of the proposed test, we performed this hypothesis testing on various simulation designs (Appendix C). Our simulations show that the test produces valid $p$-values, which are conservative (Fig. C.4), meaning the test has a maximum type I error rate less than or equal to the nominal level of the test. On real data sets analyzed below, these $p$-values are small, so the conservative behavior that we observe in simulations does not appear to be relevant for populations with nontrivial levels of structure.

3. Analysis of human studies

We applied the super admixture framework to four published studies: the human genome diversity panel (HGDP) [19], the 1000 genomes project (TGP) [20], the Human Origins study (HO) [21], and a study on individuals with Indian ancestry (IND) [22]. Within the TGP study, we also analyzed a subset of admixed populations with American ancestry, denoted by AMR. While HGDP, TGP, and HO are sampled from ancestries throughout the world, the IND and AMR data sets are regionally sampled. This yielded five data sets that collectively represent a range of population structures and study designs. Discussions of the results on HO, AMR, and IND are in the main text, while HGDP and TGP are in Appendix D.

3.1. Calculations

We processed the data sets and performed quality control checks to produce a genotype matrix $\mathit{X}$ for each as the starting point of our analysis (Appendix D.1). We next applied Algorithm 2 to $\mathit{X}$ to obtain ${\stackrel{\mathbf{ˆ}}{\mathit{\Lambda }}}^{\text{sup}}$ and ${\stackrel{\mathbf{ˆ}}{\mathit{\Lambda }}}^{\text{std}}$, the estimates of antecedent population coancestry for the super admixture and standard admixture models, respectively. We also calculated their corresponding individual-level coancestry estimates ${\stackrel{\mathbf{ˆ}}{\mathbf{\Theta }}}^{\text{sup}}$ and ${\stackrel{\mathbf{ˆ}}{\mathbf{\Theta }}}^{\text{std}}$. As a part of Algorithm 2, we calculated the appropriate number of antecedent populations $K$ using the structural Hardy-Weinberg method [12] (detailed in Appendix D.6). The values of $K$ ranged from $K=11$ for HO to $K=3$ for AMR, which are consistent with earlier work [10, 12, 28]. Also, in Algorithm 2 we calculated estimates of the admixture proportion matrices $\stackrel{\mathit{ˆ}}{\mathit{Q}}$ using the ALStructure method [10].

To evaluate the accuracy of ${\stackrel{\mathbf{ˆ}}{\mathbf{\Theta }}}^{\text{sup}}$ and ${\stackrel{\mathbf{ˆ}}{\mathbf{\Theta }}}^{\text{std}}$, we computed the OS estimate [4] of individual-level coancestry ${\stackrel{\mathbf{ˆ}}{\mathbf{\Theta }}}^{\text{OS}}$ on each data set. The OS estimate of $\mathbf{\Theta }$ is based on general assumptions and is a consistent estimator for arbitrary population structures under the appropriate conditions. Since ${\stackrel{\mathbf{ˆ}}{\mathbf{\Theta }}}^{\text{OS}}$ makes no assumptions about the distributions of the IAFs or coancestry parameters, it serves as a benchmark for our methods¹, allowing us to observe if the super admixture or standard admixture models lose information about individual-level coancestry relative to OS. As shown in Table D.1, the Frobenius-based distances from ${\stackrel{\mathbf{ˆ}}{\mathbf{\Theta }}}^{\text{sup}}$ to ${\stackrel{\mathbf{ˆ}}{\mathbf{\Theta }}}^{\text{OS}}$ are about 10 to 40 times smaller than those from ${\stackrel{\mathbf{ˆ}}{\mathbf{\Theta }}}^{\text{std}}$ to ${\stackrel{\mathbf{ˆ}}{\mathbf{\Theta }}}^{\text{OS}}$. The distance from ${\stackrel{\mathbf{ˆ}}{\mathbf{\Theta }}}^{\text{sup}}$ to ${\stackrel{\mathbf{ˆ}}{\mathbf{\Theta }}}^{\text{OS}}$ is smaller than is arguably practically relevant, meaning that ${\stackrel{\mathbf{ˆ}}{\mathbf{\Theta }}}^{\text{sup}}$ achieves the resolution of ${\stackrel{\mathbf{ˆ}}{\mathbf{\Theta }}}^{\text{OS}}$ for practical purposes.

We carried out Algorithm 7 to perform a hypothesis test of the standard admixture model versus the super admixture model for all five datasets, with $B=1000$ bootstrap iterations. For all data sets, no bootstrap null test-statistic was equal to or greater than the observed test-statistic, so $p$-value < 0.001 for all data sets. The bootstrap null test-statistics and observed test-statistic for all data sets are shown in Fig. D.9.

We applied Algorithm 6 to generate bootstrap replications ${\mathit{X}}^{*}$ from each data set’s genotype matrix $\mathit{X}$. We applied the double-admixture method (Algorithm 4) and the NORTA method (Algorithm B.5) to include the performance of both methods. We computed the OS estimate ${\stackrel{\mathbf{ˆ}}{\mathbf{\Theta }}}^{\text{OS}*}$ of individual-level coancestry for each ${\mathit{X}}^{*}$.

3.2. Visualizing results

We firstly visualized the results by making heatmaps of individual-level coancestry estimates ${\stackrel{\mathbf{ˆ}}{\mathbf{\Theta }}}^{\text{OS}},{\stackrel{\mathbf{ˆ}}{\mathbf{\Theta }}}^{\text{sup}}$, and ${\stackrel{\mathbf{ˆ}}{\mathbf{\Theta }}}^{\text{std}}$. We also made heatmaps of ${\stackrel{\mathbf{ˆ}}{\mathbf{\Theta }}}^{\text{OS}*}$ from bootstrap resampled genotypes using both the double-admixture and NORTA methods for generating antecedent population allele frequencies. These are displayed as follows: HO − Fig. 2, AMR − Fig. 4, IND − Fig. 6, HGDP − Fig. D.5, and TGP − Fig. D.7. It can be seen that for all data sets, ${\stackrel{\mathbf{ˆ}}{\mathbf{\Theta }}}^{\text{OS}}$ and ${\stackrel{\mathbf{ˆ}}{\mathbf{\Theta }}}^{\text{sup}}$ are qualitatively equivalent, which is quantitatively supported by Table D.1 showing they are very close. The estimates ${\stackrel{\mathbf{ˆ}}{\mathbf{\Theta }}}^{\text{OS}*}$ from the two bootstrap methods are also qualitatively equivalent to ${\stackrel{\mathbf{ˆ}}{\mathbf{\Theta }}}^{\text{OS}}$ and ${\stackrel{\mathbf{ˆ}}{\mathbf{\Theta }}}^{\text{sup}}$. Finally, it can be seen that the standard admixture coancestry estimate ${\stackrel{\mathbf{ˆ}}{\mathbf{\Theta }}}^{\text{std}}$ is not close to the other estimates, further indicating the standard admixture model is not sufficient for these data sets.

Figure 4:

Heatmaps of individual-level coancestry estimates in the AMR data set.

Figure 6:

Heatmap of individual-level coancestry estimates in the merged data set of mainland Indians from IND, and Central/South Asians and East Asians from HGDP.

We secondly visualized the results by building on the standard colored stacked bar plots of $\stackrel{\mathit{ˆ}}{\mathit{Q}}$ displaying the admixture proportions of the $K$ antecedent populations for the individuals. In our case, we have additional information, which is the estimated antecedent population coancestry matrix ${\stackrel{\mathbf{ˆ}}{\mathit{\Lambda }}}^{\text{sup}}$ from the super admixture model. This matrix gives additional information about the relationship among the antecedent populations that we would like to visualize. The first way we visualized ${\stackrel{\mathbf{ˆ}}{\mathit{\Lambda }}}^{\text{sup}}$ was create a heatmap of its values. We then constructed a dendogram built from ${\stackrel{\mathbf{ˆ}}{\mathit{\Lambda }}}^{\text{sup}}$ that is displayed above the stacked bar plot. This gives the user insight into the relatedness of the antecedent populations and connect them to the stacked bar plots. To this end, we calculated a distance matrix $\mathit{D}$ from ${\stackrel{\mathbf{ˆ}}{\mathit{\Lambda }}}^{\text{sup}}$ according to:

$$d_{uv}=\left\{\begin{array}{ll}0& \text{if}u=v\\ \text{max}\left({\stackrel{\mathbf{ˆ}}{\mathit{\Lambda }}}^{\text{sup}}\right)-{\hat{\lambda }}_{uv}^{\text{sup}}& \text{if}u\ne v.\end{array}\right.$$

We then applied the standard agglomerative clustering method to $\mathit{D}$ using “weighted pair group method with arithmetic mean” (WPGMA) to obtain a dendrogram. These are displayed in the data sets as follows: HO − Fig. 3, AMR − Fig. 5, IND − Fig. 7, HGDP − Fig. D.6, and TGP − Fig. D.8.

Figure 5:

(A) Heatmap of antecedent population coancestry estimates in AMR. (B) Dendrogram representation of the antecedent population coancestry estimates. (C) Stacked bar plot of admixture proportions.

Figure 7:

Heatmap of antecedent population coancestry estimates in the merged data set of mainland Indians from IND, and Central/South Asians and East Asians from HGDP. (B) Dendrogram representation of the antecedent population coancestry estimates. (C) Stacked bar plot of admixture proportions.

3.3. Human Origins (HO) study

The Human Origins datasets (HO) consists of 2124 individuals from 170 sub-subpopulations grouped into 11 subpopulations. We observed the estimated individual-level coancestry agrees with current knowledge of early human migrations [29−32]. In Fig. 2, we observed the first major split between Sub-Saharan Africa and North Africa. This split reflects the divergence between Sub-Saharan Africans and the rest of human populations resulting from an out-of-Africa migration around 50–60 kya. Another split occurred between South Asia and East Asia, revealing the separation between West Eurasians and East Asians around 40–45 kya. Among the East Asia clade, we identified that the Oceanians have highest coancestry within and lowest coancestry between other subpopulations, consistent with the theory that Oceanians split earliest from the rest of East Asians.

The coancestry among antecedent populations is also compatible with early human dispersals (Fig. 3). Specifically, in the dendrogram plot of the antecedent population coancestry (Fig. 3B), we note that the first branch split individuals from Sub-Saharan Africa represented by the antecedent populations $S_1$ and $S_2$ from individuals outside of Sub-Saharan Africa represented by the other antecedent populations. Individuals outside of Sub-Saharan Africa further branched off into two lineages: the West Eurasians represented by antecedent populations $S_3,S_4$ and $S_5$, and the East Asians represented by antecedent populations $S_6-S_{11}$. Then the Oceanians represented by the antecedent population $S_9$ split off from the majority of East Asian ancestry, while the latter further diverged into present-day Asians (antecedent populations $S_6,S_7,S_8$) and present-day Americans (antecedent populations $S_{10}$ and $S_{11}$).

3.4. Admixed individuals (AMR) from the 1000 Genomes Project (TGP)

The AMR subset of TGP has 353 individuals from four regions (Mexican-American (MXL): 65, Puerto Rican (PUR): 104, Colombian (CLM): 97, Peruvian (PEL): 87). The individual-level coancestry plot (Fig. 4) revealed that this dataset does not have a discrete population structure. Instead, the coancestry changes smoothly over individuals, indicating wide-ranging historical admixture events. This is consistent with the AMR population descending from European, Native American, and Sub-Saharan African ancestries during the post-Columbian era [33, 34].

In the analysis of the coancestry among antecedent populations (Fig. 5), we identified three major sources of ancestry: Sub-Saharan African ancestry represented by the antecedent population $S_1$, West Eurasian ancestry represented by the antecedent population $S_2$, and Native American ancestry represented by the antecedent population $S_3$. The first split occurred between Sub-Saharan Africans $\left(S_1\right)$ and individuals outside of Sub-Saharan Africa ($S_2$ and $S_3$), and the second split between the West Eurasians $\left(S_2\right)$ and the Native Americans $\left(S_3\right)$. We also noted that the Puerto Ricans contain the highest amount of Sub-Saharan African ancestry; the Peruvians have the highest proportion of Native American ancestry; the Colombians and the Mexican-Americans display extensive variation in in their admixture proportions of European and Native American ancestry. Our observations were confirmed by previous analyses of AMR populations [28, 33, 34].

3.5. Indian (IND) study

We combined the mainland Indians from the IND study with the Central/South Asia and the East Asia populations from HGDP to study the relationship between present-day Indians and other populations in Asia. Our merged data set consists of 298 mainland Indians from fou linguistic groups (Indo-European (IE): 92, Dravidian: 53, Austro-Asiatic (AA): 79, Tibeto-Burman (TB): 74), together with 190 Central/South Asians and 210 East Asians from HGDP. Previous analyses of South Asian populations have shown that the Indo-European speakers show a considerable amount of the Western Eurasian relatedness and are ancestrally close to Central Asians. The Austro-Asiatic speakers and the Tibeton-Burman speakers were mixed from East Asian ancestry. The Tibeton-Burman speakers generally have significant genomic proportions derived from East Asian ancestry so that some Tibeton-Burman speakers can be difficult to distinguish from East Asian populations based on genome-wide measures of relatedness. Consistent with these findings [22, 35, 36], we observe a split between Indo-European speakers and the rest of mainland Indians in the heatmap of individual-level coancestry (Fig. 6). The Indo-European speakers and the Central/South Asians of HGDP have relatively similar levels of coancestry. The second split occurred between the Austro-Asiatic speakers and the Tibeto-Burman speakers. The Tibeto-Burman speakers and East Asians of HGDP have relatively similar levels of coancestry.

Our analysis reveals that there are three major branches of antecedent populations for this dataset (Fig. 7). The branch of antecedent populations $S_1$ and $S_2$ is most prevalent in Central/South Asians of HGDP and Indo-European speakers, suggesting this branch was at least partially derived from a West Eurasian source. The branch of the antecedent populations $S_3,S_4$ and $S_5$ is widespread in Dravidian speakers and Austro-Asiatic speakers, indicating it is relevant to South Indian ancestry and Austro-Asiatic speaker ancestry. The third branch of the antecedent populations $S_6$ and $S_7$ likely represents East Asian ancestry due to its high prevalence in the Tibeto-Burman speakers and East Asians of HGDP.

4. Discussion

The super admixture framework is an extension of the highly used admixture model. It superposes coancestry among the admixed antecedent populations. It provides a forward generating probability process that encompasses random evolutionary, genealogical, and statistical sampling processes. The antecedent populations are modeled to have an arbitrarily complex coancestry. This allows the generation of individual-specific allele frequencies (IAFs) that capture complex population structures and permit the estimation of individual-level coancestry that is at the resolution of general individual-level coancestry and kinship estimators for arbitrarily complex structures.

There are numerous parameters estimated from genome-wide genotype data that relate to structure, such as coancestry, inbreeding, and $F_{\text{ST}}$. When traits are included, one often estimates parameters in the context of genome-wide association studies [23, 37], genome-wide heritability [38–40] and polygenic risk scores [41, 42]. There does not exist a straightforward, general method for quantifying uncertainty among these various estimates. Within our framework, we have shown how to perform a bootstrap resampling method that randomly generates new genetic data that recapitulate population structure observed in real data. This bootstrap method may provide a way to formulate general methods for quantifying uncertainty in genome-wide genotype studies.

We developed a hypothesis test where one can test the standard versus super admixture model on real data. When we applied it to the five data sets analyzed here, all of them were highly significant in rejecting the standard admixture model in favor of the super admixture model. The individual-level coancestry estimates from the super admixture model also agreed with the general coancestry estimate, whereas the standard admixture individual-level coancestry estimates did not.

The stacked bar plot visualization of admixture proportions among individuals is ubiquitous in analyzing population structure. We showed here how the estimated antecedent population coancestry can be plotted with the stacked bar plot to visualize the relationship among the antecedent populations in conjunction with the bar plot. The admixture proportions among individuals are then interpretable in terms of the evolutionary history of the antecedent populations. We demonstrated this visualization on five data sets and showed how it agreed with known results on these human populations.

Understanding population structure in humans is one of the central problems in modern genetics. We demonstrated that the proposed super admixture framework is a powerful tool for learning admixed population coancestry, improving the analysis of genetic data from structured populations, bridging admixture with individual-level coancestry and kinship, and simulating new data reflecting a structured population. We anticipate that the super admixture framework will be useful in analyzing complex population structure in future applications.