Model selection in ADMIXTURE can be inconsistent: proof of the $K=2$ phenomenon

Abstract

STRUCTURE and ADMIXTURE are two popular methods for detecting population structure in genetic data. They model observed genotypes as mixtures of latent ancestral populations, and the inferred admixture proportions can be used to visualize and summarize population structure. A key parameter in these models is the number of ancestral populations, $K$. Selecting $K$ is a challenging problem. Perhaps the most widely used method is Evanno’s $\text{Δ}K$, which selects $K$ based on the second-order change in log-likelihood as $K$ increases. However, practitioners have often noted that $\text{Δ}K$ often favors overly small $K$, frequently returning $K=2$ even when more meaningful substructure is present. In this paper, we provide a theoretical explanation for this phenomenon: we prove that, under certain conditions, the $\text{Δ}K$ method can be inconsistent, meaning that it can fail to identify the true number of populations even with infinite data.

1. Introduction

A common first step when exploring genetic data is to group or cluster sampled genomes into a smaller number of interpretable units. Two workhorse methods for clustering genetic data predominate. Principal components analysis (PCA) provides a fast, model-free representation of major axes of genetic variation [Patterson et al., 2006, Price et al., 2006]. Alternatively, model-based methods such as STRUCTURE [Pritchard et al., 2000] and ADMIXTURE [Alexander et al., 2009] posit a fixed number $K$ of ancestral populations, each with specific allele frequencies, and represent sampled genomes as a mixture over these populations.

An appeal of model-based approaches is interpretability: the inferred clusters represent hypothesized ancestral populations, and individual mixing proportions correspond to ancestry fractions. But this interpretation depends on specifying the number of populations $K$, which is usually non-obvious. Setting $K$ too small (underfitting) can smooth out real structure, while setting $K$ too large (overfitting) can turn noise into apparent structure. Unlike PCA, where dimensionality can be explored post hoc, model-based approaches require this choice in advance, and different criteria can yield different conclusions.

Despite the widespread use of STRUCTURE and ADMIXTURE, the question of how best to select $K$ remains unsettled [Novembre, 2016]. Perhaps the most widely used approach is Evanno’s $\text{Δ}K$ [Evanno et al., 2005], which detects an “elbow” in a likelihood summary as $K$ increases. However, practitioners have noted that $\text{Δ}K$ often favors overly small $K$, especially $K=2$. Indeed, a recent survey of 1,264 studies that utilized STRUCTURE found that use of $\text{Δ}K$ increased over time and was associated with a substantially higher rate of reporting $K=2$ [Janes et al., 2017], with potentially serious implications for species conservation and management. A follow-up simulation study supported the tendency of $\text{Δ}K$ to select $K=2$, and found that the magnitude of $\text{Δ}K$ is related to the level of divergence between populations [Cullingham et al., 2020]. This phenomenon is also noted in the seminal paper [Evanno et al., 2005].

Although the tendency of $\text{Δ}K$ to underfit has been documented empirically, it lacks a rigorous mathematical explanation. Here we provide one, by proving that $\text{Δ}K$ can be inconsistent: in some regimes, even with infinite data, it selects $K=2$ despite the true value being larger. We give an explicit sufficient condition for this to occur in terms of the probabilistic divergence between population allele frequency distributions, and show how this condition can be satisfied under a realistic population genetic model when $F_{ST}$ is sufficiently low. To the best of our knowledge, this is the first theoretical explanation for the $K=2$ phenomenon.

2. Results

In this study, we focus on maximum likelihood estimation (MLE) of the STRUCTURE model, i.e. the ADMIXTURE method [Alexander et al., 2009]. We quickly review the model in order to be able to state our results.

ADMIXTURE assumes genotype matrix $\mathbf{\text{X}}={\left(X_{n\ell }\right)}_{n,\ell =1}^{N,L}\in {\{0,1\}}^{N\times L}$ of $N$ individuals at $L$ SNPs is generated by admixing between $K_0$ “true” populations¹. Each population $k\left(k\in \left[K_0\right]\right)$ is represented by its allele frequency vector ${\mathbf{\text{P}}}_k^0={\left(P_{k\ell }^0\right)}_{\ell =1}^L\in {[0,1]}^L$. Each individual $n\in \{1,\dots ,N\}$ is a mixture of those populations, with proportion $Q_{nk}^0$ belonging to population $k$, where $Q_{nk}^0\ge 0$, ${\sum }_{k=1}^{K_0}{Q}_{nk}^0=1$. Hence, it models the genotype data as

$$X_{n\ell }\stackrel{ind.}{~}\mathrm{Ber}\left(\sum _{k=1}^{K_0}{Q}_{nk}^0{P}_{k\ell }^0\right),n\in \{1,\dots ,N\},\ell \in \{1,\dots ,L\}.$$ (1)

The parameters ${\mathbf{\text{P}}}^0={\left(P_{k\ell }^0\right)}_{k,\ell =1}^{K_0,L}$ and ${\mathbf{\text{Q}}}^0={\left(Q_{nk}^0\right)}_{n,k=1}^{N,K_0}$ and the number of populations $K_0$ are unknown and have to be estimated from data.

2.1. Model selection using second-order changes of log-likelihood functions

Now we present an Evanno-style “elbow” criterion for selecting $K$ adapted to the MLE framework. For a given $K$, the average log-likelihood of the model with parameters ($\mathbf{\text{Q}}$, $\mathbf{\text{P}}$) is

$$\begin{gathered} 𝓛(\mathbf{\text{Q}},\mathbf{\text{P}})=\frac{1}{NL}\sum _{n,\ell =1}^{N,L}\log \mathrm{Ber}\left(X_{n\ell }\mid \sum _{k=1}^K{Q}_{nk}{P}_{k\ell }\right) \\ =\frac{1}{NL}\sum _{n,\ell =1}^{N,L}\left(X_{n\ell }\log \left(\sum _{k=1}^K{Q}_{nk}{P}_{k\ell }\right)+\left(1-X_{n\ell }\right)\log \left(1-\sum _{k=1}^K{Q}_{nk}{P}_{k\ell }\right)\right). \end{gathered}$$

For every $K$ ranging from 1 to some pre-specified upper bound $\overline{K}$, we seek the MLE

$${\widehat{\mathbf{\text{Q}}}}^{(K)},{\widehat{\mathbf{\text{P}}}}^{(K)}=\underset{\mathbf{\text{Q}},\mathbf{\text{P}}}{\arg \max }𝓛(\mathbf{\text{Q}},\mathbf{\text{P}}).$$ (2)

Let ${\widehat{𝓛}}^{(K)}=𝓛\left({\widehat{\mathbf{\text{Q}}}}^{(K)},{\widehat{\mathbf{\text{P}}}}^{(K)}\right)$. For $K=2,\dots ,\overline{K}-1$, we examine the second-order change of these log-likelihood values as

$$\widehat{\text{Δ}}(K):=\left|2{\widehat{𝓛}}^{(K)}-{\widehat{𝓛}}^{(K-1)}-{\widehat{𝓛}}^{(K+1)}\right|,$$ (3)

and decide the most plausible value of $K$ as

$$\widehat{K}=\underset{K=2,\dots ,\overline{K}-1}{\arg \max }\widehat{\text{Δ}}(K).$$ (4)

Note that our definition of $\widehat{\text{Δ}}(K)$ differs from the original $\text{Δ}K$ in [Evanno et al., 2005], which divides by a standard-deviation term estimated across runs. We analyze the un-normalized form because the normalized statistic is a ratio of dependent random variables, hence difficult to analyze theoretically. Also, whereas this normalization was originally intended to prevent overfitting (overfitted log-likelihoods have higher variance), at least in the scenarios considered here, the $\text{Δ}K$ method underfits, so this normalization only exacerbates the problem.

2.2. Inconsistency of model selection

We focus on the case of true $K_0=3$ populations and denote the true parameters by ${\mathbf{\text{Q}}}^0\in {[0,1]}^{N\times 3}$ and ${\mathbf{\text{P}}}^0\in {[0,1]}^{3\times L}$. We state informal asymptotic versions of the two main theorems here, and defer full non-asymptotic statements and proofs to the SI Appendix. Our results make the following assumptions about the data generating process.

Assumption 1

(Boundedness of the allele frequency ${\mathbf{\text{P}}}^0$). There exists a positive constant $\underset{\_}{c}$ so that true parameters ${\mathbf{\text{P}}}^0={\left(P_{k\ell }^0\right)}_{k,\ell =1}^{3,L}$ satisfies

$$e^{-\underset{\_}{c}}\le P_{k\ell }^0\le 1-e^{-\underset{\_}{c}}\forall k\in \{1,2,3\},\ell \in \{1,\dots ,L\}.$$

Assumption 1 ensures that the allele frequencies are bounded away from 0 and 1. This is necessary to keep the log-likelihood terms from diverging near the boundaries of the parameter space. It is not overly restrictive, since it is common practice to QC-filter rare variants (low minor allele frequency), which are more error-prone and often less informative for PCA or ADMIXTURE analyses.

Assumption 2

(Data generating assumption of ${\mathbf{\text{Q}}}^0$). $N$ samples are partitioned into three groups ${𝒩}_1\cup {𝒩}_2\cup {𝒩}_3$ so that $Q_{nk}^0=1$ for every $n\in {𝒩}_k$ and $k\in [3]$. We also assume $\left|{𝒩}_1\right|=\left|{𝒩}_2\right|=\left|{𝒩}_3\right|=N/3$.

Assumption 2 places each individual at a vertex of the simplex (i.e., $Q_{nk}^0\in \{0,1\}$). It is made for mathematical convenience, but it is also favorable for identifying population structure: every individual is purely a member of a single population, which maximizes the signal of population structure in the data.

2.2.1. Measures of population divergence

Our first theorem is stated in terms of the Kullback-Leibler (KL) divergence between allele frequencies across populations. KL divergence emerges naturally when studying the asymptotics of likelihood-based inference methods. In the sequel, we offer a second result phrased in terms of population genetic divergences (i.e., $F_{ST}$) that may be more familiar to practitioners.

For two populations with allele frequencies ${\mathbf{\text{P}}}_1$ and ${\mathbf{\text{P}}}_2\in {[0,1]}^L$, let

$$\mathrm{KL}\left(P_{1\ell }\Vert P_{2\ell }\right):=\mathrm{KL}\left(\mathrm{Ber}\left(P_{1\ell }\right)\Vert \mathrm{Ber}\left(P_{2\ell }\right)\right)=P_{1\ell }\log \left(\frac{P_{1\ell }}{P_{2\ell }}\right)+\left(1-P_{1\ell }\right)\log \left(\frac{1-P_{1\ell }}{1-P_{2\ell }}\right)$$

denote the Kullback-Leibler divergence between allele frequencies of SNP $\ell$ for all $\ell \in [L]$, and let

$$\text{KL}\left({\mathbf{\text{P}}}_1\Vert {\mathbf{\text{P}}}_2\right):=\sum _{\ell \in [L]}\text{KL}\left(P_{1\ell }\Vert P_{2\ell }\right),\overline{\text{KL}}\left({\mathbf{\text{P}}}_1\Vert {\mathbf{\text{P}}}_2\right)=\frac{1}{L}\text{KL}\left({\mathbf{\text{P}}}_1\Vert {\mathbf{\text{P}}}_2\right)$$

denote their sum and average over $\ell$, respectively. We suppose a tree-like structure for three populations so that ${\mathbf{\text{P}}}_2^0$ and ${\mathbf{\text{P}}}_3^0$ are closer to each other compared to ${\mathbf{\text{P}}}_1^0$, and this is quantified via

$${𝒟}_{31}^{(L)}:=\overline{\text{KL}}\left({\mathbf{\text{P}}}_1^0\Vert \frac{{\mathbf{\text{P}}}_1^0+{\mathbf{\text{P}}}_2^0+{\mathbf{\text{P}}}_3^0}{3}\right)+\overline{\text{KL}}\left({\mathbf{\text{P}}}_2^0\Vert \frac{{\mathbf{\text{P}}}_1^0+{\mathbf{\text{P}}}_2^0+{\mathbf{\text{P}}}_3^0}{3}\right)+\overline{\text{KL}}\left({\mathbf{\text{P}}}_3^0\Vert \frac{{\mathbf{\text{P}}}_1^0+{\mathbf{\text{P}}}_2^0+{\mathbf{\text{P}}}_3^0}{3}\right),$$

$${𝒟}_{32}^{(L)}:=\overline{\text{KL}}\left({\mathbf{\text{P}}}_2^0\Vert \frac{{\mathbf{\text{P}}}_2^0+{\mathbf{\text{P}}}_3^0}{2}\right)+\overline{\text{KL}}\left({\mathbf{\text{P}}}_3^0\Vert \frac{{\mathbf{\text{P}}}_2^0+{\mathbf{\text{P}}}_3^0}{2}\right).$$

The population-level analogs of ${𝒟}_{31}^{(L)}$ and ${𝒟}_{32}^{(L)}$ are obtained by averaging over the locus distribution (equivalently, by taking $L\to \infty$):

$${𝒟}_{31}:=\mathbb{E}\left[{𝒟}_{31}^{(L)}\right]=\underset{L\to \infty }{\lim }{𝒟}_{31}^{(L)},$$

$${𝒟}_{32}:=\mathbb{E}\left[{𝒟}_{32}^{(L)}\right]=\underset{L\to \infty }{\lim }{𝒟}_{32}^{(L)}.$$

2.2.2. First main result

Our first main result shows that Evanno’s second-order criterion can underfit in the idealized three-population setting described above.

Theorem 1.

Suppose that

$${𝒟}_{32}<\frac{1}{3}{𝒟}_{31}.$$ (5)

Then, as $N$, $L\to \infty$, the second-order method (4) selects $\widehat{K}=2$ with probability tending to 1 (despite the true $K_0=3$), i.e., it is inconsistent.

${𝒟}_{32}$ is the average information loss from merging populations 2 and 3, while ${𝒟}_{31}$ captures overall three-way heterogeneity. Condition (5) therefore says that merging 2 and 3 is cheap relative to total dispersion, so the elbow criterion can prefer $\widehat{K}=2$ even asymptotically. The SI Appendix formalizes this by relating fitted log-likelihood gaps to their population counterparts with error ${ϵ}_{NL}\to 0$ as $N$, $L\to \infty$.

Note that Theorem 1 conditions on ${\mathbf{\text{P}}}^0$ and ${\mathbf{\text{Q}}}^0$, and does not assume any particular mechanism for producing them. Thus any process that generates parameters satisfying the divergence inequality (5) can lead to inconsistency of $\text{Δ}K$. In the next section we provide an example of such a process based on a realistic population-genetic model.

2.2.3. Second main result

Our next theorem shows how the divergence inequality (5) can be satisfied when the data are generated under a hierarchical population model with realistic covariance [Balding and Nichols, 1995]. At each locus $\ell$, let $P_{\ell }^{\ast }$ denote the root allele frequency. Conditional on $P_{\ell }^{\ast }$, we draw an internal node $P_{23\ell }^0$ and three leaves $P_{1\ell }^0$, $P_{2\ell }^0$, $P_{3\ell }^0$ via

$$P_{1\ell }^0\left|P_{\ell }^{*}\sim \mathrm{Beta}\left(\frac{1-F_{\text{out}}}{F_{\text{out}}}{P}_{\ell }^{*},\frac{1-F_{\text{out}}}{F_{\text{out}}}\left(1-P_{\ell }^{*}\right)\right)\right.$$ (6)

$$P_{23\ell }^0\left|P_{\ell }^{\ast }\sim \text{Beta}\left(\frac{1-F_{\text{root}}}{F_{\text{root}}}{P}_{\ell }^{\ast },\frac{1-F_{\text{root}}}{F_{\text{root}}}\left(1-P_{\ell }^{\ast }\right)\right)\right.$$ (7)

$$P_{2\ell }^0,P_{3\ell }^0\left|P_{23\ell }^0\sim \mathrm{Beta}\left(\frac{1-F_{\text{sub}}}{F_{\text{sub}}}{P}_{23\ell }^0,\frac{1-F_{\text{sub}}}{F_{\text{sub}}}\left(1-P_{23\ell }^0\right)\right).\right.$$ (8)

Here $F_{\text{root}}$, $F_{\text{sub}}$, $F_{\text{out}}$ are drift parameters along the edges $P^{*}\to P_{23}^0$, $P_{23}^0\to \left\{P_2^0,P_3^0\right\}$, and $P^{*}\to P_1^0$, respectively (see Figure 1). To match the conditional variances of the leaves, so that $P_1^0$, $P_2^0$, $P_3^0$ are equally dispersed around $P^{*}$, we set

$$F_{\text{out}}=F_{\text{root}}+F_{\text{sub}}\left(1-F_{\text{root}}\right)$$ (9)

Figure 1:

Nested Balding–Nichols model with drift parameters on branches.

Theorem 2.

Suppose the population allele frequencies are generated according to the nested Balding-Nichols model (6)–(9), and that the ancestral allele frequencies $P_{\ell }^{*}$ are bounded away from 0 and 1. If $F_{root}$ and $F_{sub}$ are sufficiently small and

$$F_{root}/F_{sub}>3/4,$$ (10)

the $\text{Δ}K$ method selects $\widehat{K}=2$ with probability tending to 1 as $N$, $L\to \infty$.

The proof of Theorem 2 relies on a “small drift” argument by expanding the extant allele frequencies $P_1^0$, $P_2^0$, $P_3^0$ around $P^{\ast }$ to first-order in $F_{\text{root}}$ and $F_{\text{sub}}$.

We performed numerical simulations to check the predicted threshold (10). We considered $F_{\text{sub}}\in \{0.01,0.05,0.2\}$ and 33 values of $F_{\text{root}}$ for each $F_{\text{sub}}$ (99 settings total), including a dense set of ratios near the theoretical boundary $F_{\text{root}}/F_{\text{sub}}=3/4$. This range of $F_{ST}$ covers values typically seen in human populations [Bhatia et al., 2013]. As $F_{\text{root}}\to 0$ the demography approaches a star topology, while for larger values, the relationship becomes increasingly hierarchical. For each parameter setting, we simulated $L=2000$ SNPs and $N=150$ diploid individuals (50 from each of the three populations). For each simulated data set, we fit ADMIXTURE for $K=1,\dots ,5$ and retained the maximum log-likelihood at each $K$ across multiple random restarts. We then compute $\widehat{\text{Δ}}(K)$ for each $K$ and select $\widehat{K}=\arg \max \widehat{\text{Δ}}(K)$.

Figure 2A displays $\widehat{K}$ against ${\log }_{10}\left(F_{\text{root}}/F_{\text{sub}}\right)$, and Figure 2B shows the same outcomes in the $\left(F_{\text{sub}},F_{\text{root}}\right)$ plane. The simulation results in Figure 2 show a clear transition near the predicted boundary (10). It is interesting to note that when the distance $F_{\text{sub}}$ between ${\mathbf{\text{P}}}_2^0$ and ${\mathbf{\text{P}}}_3^0$ is fixed and $F_{\text{root}}$ is small, $\widehat{\text{Δ}}(K)$ can correctly choose $\widehat{K}=3$. But when $F_{\text{root}}$ increases, $\widehat{\text{Δ}}(K)$ favors merging ${\mathbf{\text{P}}}_2^0$ and ${\mathbf{\text{P}}}_3^0$ into one population.

Figure 2:

Threshold phenomenon predicted by Theorem 2. (A) Across simulated three-population settings under the nested Balding–Nichols model, the $\widehat{\text{Δ}}$ method selects $\widehat{K}\in \{2,3\}$ as a function of ${\log }_{10}\left(F_{\text{root}}/F_{\text{sub}}\right)$. (B) The same settings shown in the $\left(F_{\text{sub}},F_{\text{root}}\right)$ plane (log scale). The dashed line in each plot marks the theoretical phase boundary $F_{\text{root}}/F_{\text{sub}}=3/4$.

3. Discussion

We provide a theoretical explanation for the widely observed tendency of Evanno’s $\text{Δ}K$ to underfit in ADMIXTURE analyses. In particular, we identify regimes where $\text{Δ}K$ is inconsistent, selecting $\widehat{K}=2$ even when the true number of populations is $K_0=3$, and show how this could arise in a realistic three-population model with $F_{ST}$ values similar to those observed in modern humans.

Our results do not imply that $\text{Δ}K$ always fails. They identify a specific failure mode, most relevant when populations are closely related. In practice, $\text{Δ}K$ should be interpreted alongside other selection criteria and biological context, and results across a range of $K$ should be reported rather than relying on a single selected value, as advocated by prior work [Pritchard et al., 2000, Evanno et al., 2005, Janes et al., 2017].

Here we focused on maximum likelihood estimation and the $\text{Δ}K$ method. In practice, other model selection techniques and inference methods are often used to fit the ADMIXTURE model. Nevertheless, many if not all of them rely on comparing log-likelihoods across different $K$ to perform model selection. It is therefore plausible to conjecture that they are vulnerable to the same underfitting phenomenon when populations are closely related. We leave a rigorous analysis of these other methods to future work.