THE BEHAVIOR OF ADMIXED POPULATIONS IN NEIGHBOR-JOINING INFERENCE OF POPULATION TREES

Abstract

Neighbor-joining is one of the most widely used methods for constructing evolutionary trees. This approach from phylogenetics is often employed in population genetics, where distance matrices obtained from allele frequencies are used to produce a representation of population relationships in the form of a tree. In phylogenetics, the utility of neighbor-joining derives partly from a result that for a class of distance matrices including those that are additive or tree-like—generated by summing weights over the edges connecting pairs of taxa in a tree to obtain pairwise distances—application of neighbor-joining recovers exactly the underlying tree. For populations within a species, however, migration and admixture can produce distance matrices that reflect more complex processes than those obtained from the bifurcating trees typical in the multispecies context. Admixed populations—populations descended from recent mixture of groups that have long been separated—have been observed to be located centrally in inferred neighbor-joining trees, with short external branches incident to the path connecting their source populations. Here, using a simple model, we explore mathematically the behavior of an admixed population under neighbor-joining. We show that with an additive distance matrix, a population admixed among two source populations necessarily lies on the path between the sources. Relaxing the additivity requirement, we examine the smallest nontrivial case—four populations, one of which is admixed between two of the other three—showing that the two source populations never merge with each other before one of them merges with the admixed population. Furthermore, the distance on the constructed tree between the admixed population and either source population is always smaller than the distance between the source populations, and the external branch for the admixed population is always incident to the path connecting the sources. We define three properties that hold for four taxa and that we hypothesize are satisfied under more general conditions: antecedence of clustering, intermediacy of distances, and intermediacy of path lengths. Our findings can inform interpretations of neighbor-joining trees with admixed groups, and they provide an explanation for patterns observed in trees of human populations.

1. Introduction

Distance matrix methods in phylogenetics construct trees of taxa using algorithms applied to matrices that tabulate pairwise evolutionary distances between the taxa.^1,2 Among these methods, neighbor-joining^3,4 is one of the most popular.^5–7 One of its key features is its consistency: if the distance matrix is additive, such that a tree of taxa exists that generates the distances in the matrix, then neighbor-joining recovers this exact tree.^5,8,9 Further, neighbor-joining is robust in that theoretical and simulation-based studies have found it to infer sensible trees under a broad range of mathematical and biological conditions.^7,9–13

As trees have long been used in population genetics to describe relationships among populations,^14,15 the neighbor-joining algorithm has been applied extensively as a population clustering tool, using distance matrices calculated from population-level allele frequencies. In humans, neighbor-joining trees have been and continue to be a regular feature of studies of population relationships.^16–20 In population-genetic studies, because migration and admixture sometimes generate evolutionary histories that cannot easily be described by a bifurcating tree of populations, a neighbor-joining tree is treated as a type of population clustering diagram rather than a precise representation of the evolutionary history of the populations.

When neighbor-joining has been used with admixed populations—populations recently descended from two or more source groups that have long been separated—particular characteristics of the inferred trees have often been observed (Fig. 1). For example, one simulation study based on human data identified a reduction in the external branch length leading to an admixed population as the strength of gene flow with other populations was increased.²¹ It has also been suggested on the basis of observed human population trees that a short external branch for a population on a constructed neighbor-joining tree can imply recent admixture of the population, and that admixed populations often appear in the “middle” of a neighbor-joining tree, on branches incident to paths connecting possible source populations.^21–25 This pattern is evident in Fig. 2, in which admixed Mestizo populations from Latin America lie on branches incident to the path connecting Native American and European populations. Here, we seek to understand these results on the behavior of admixed populations in the application of the neighbor-joining algorithm. We therefore apply neighbor-joining to populations that satisfy a simple admixture model, first considering the case in which the distance matrix is additive. Next, for the case of n = 4 taxa, we use a mechanistic mathematical investigation to examine three specific properties of neighbor-joining trees involving an admixed population.

Fig. 1.

Properties observed for admixed taxa in neighbor-joining trees. Taxon t₈ represents an admixture of source populations t₁ and t₂. The admixed taxon appears on a short external branch incident to the path connecting the source populations. Denoting distances on the tree by d̂ and topological path lengths that count edges separating pairs of taxa by b̂, the tree illustrates the properties of intermediacy of distances (d̂_t1,t8 < d̂_t1,t2 and d̂_t2,t8 < d̂_t1,t2, or equivalently, d̂_u,t8 < d̂_u,t1 and d̂_u,t8 < d̂_u,t2, where u is the unique node that places t₁, t₂, and t₈ in different subtrees), and intermediacy of path lengths (b̂_t1,t8 ≤ b̂_t1,t2 and b̂_t2,t8 ≤ b̂_t1,t2).

Fig. 2.

Neighbor-joining tree of admixed Mestizo populations together with Native American and European populations that represent ancestral source regions for the admixed populations. The tree, obtained using Neighbor and Drawtree in the Phylip package,²⁶ uses data on 678 microsatellite loci in 13 Mestizo, 26 Native American, and 8 European populations.^27–29 Allele frequencies were computed from 872 individuals—249 Mestizo, 463 Native American, and 160 European—and distances were computed with Microsat³⁰ using one minus the proportion of shared alleles.³¹ Mestizo, Native American, and European branches appear in yellow, purple, and blue, respectively. Mestizos lie in the “middle” of the tree, connecting to the path that links the Native Americans and Europeans. External branches for Mestizo populations are shorter on average (0.102) than for Native American (0.146) and European populations (0.109); Mestizo populations have 9 of the 15 shortest external branches.

2. The neighbor-joining algorithm

We briefly review the neighbor-joining algorithm.^3,4 Consider a set of n taxa, together with a distance function d computed for each pair of taxa, such that the distance between taxa i and j is denoted d_ij. The algorithm takes as input the distance matrix D containing entries d_ij, with i and j ranging from 1 to n, and it outputs a bifurcating unrooted tree. D is symmetric (d_ij = d_ji), with zeroes on the diagonals (d_ii = 0) and nonnegative real entries (d_ij ≥ 0).

As in other agglomerative algorithms that construct bifurcating trees,^2,32 at each of a series of steps, the two nearest taxa according to a selection criterion are connected to a new interior node, becoming “neighbors” on the constructed tree. Branch lengths from the new node to the nodes it agglomerates, as well as the distances to all remaining nodes, are then calculated, and a new distance matrix is obtained. This procedure is repeated iteratively until the last three nodes remain, and these three nodes are then connected to a final interior node. Because the last three nodes are always joined, the number of taxa must exceed three for neighbor-joining to have a nontrivial decision at the first step.

At each step, the key decision is the choice of the pair of taxa that are agglomerated. Neighbor-joining uses an n × n matrix Q, containing entries q_ij for pairs of taxa (i, j):

$$q_{ij}=(n-2)d_{ij}-\sum _{k=1}^n{d}_{ik}-\sum _{k=1}^n{d}_{jk}.$$ (1)

The two taxa that are agglomerated are those with the minimal value of q_ij (choosing randomly in case of ties). If taxa i and j are agglomerated, then their distances to the new node u become

$$d_{iu}=\frac{1}{2}{d}_{ij}+\frac{1}{2(n-2)}\left(\sum _{k=1}^n{d}_{ik}-\sum _{k=1}^n{d}_{jk}\right)$$ (2)

$$d_{ju}=\frac{1}{2}{d}_{ij}+\frac{1}{2(n-2)}\left(\sum _{k=1}^n{d}_{jk}-\sum _{k=1}^n{d}_{ik}\right).$$ (3)

The distances of all remaining nodes k to node u are computed as

$$d_{ku}=(d_{ik}+d_{jk}-d_{ij})/2.$$ (4)

The next agglomeration then proceeds from an (n − 1) × (n − 1) distance matrix that replaces distances involving nodes i and j with those involving the single node u.

3. An admixture scenario

We examine a scenario in which one of the taxa is admixed among two of the others. This taxon can be viewed as having been formed from its two source taxa, such that individual members of the taxon have ancestors in both source groups. We label the taxa t₁, t₂, …, t_n. Without loss of generality, let taxon t_n be the admixed group, and suppose that it is an admixture of taxa t₁ and t₂. The relationships among the remaining n − 3 taxa (t₃, t₄, …, t_n−1) and between these taxa and t₁, t₂, and t_n are not specified; we do not consider any additional admixture relationships that might exist among these taxa. We assume n ≥ 4, so that at least one taxon is considered in addition to t₁, t₂, and t_n.

In a standard statistical model of admixture used in population genetics, allele frequencies in an admixed taxon are given by linear combinations of the allele frequencies of the source taxa.^33–37 We denote by λ the proportion of the ancestry of taxon t_n arising from t₁ and by 1 − λ the corresponding proportion arising from t₂, where 0 < λ < 1. For any allelic type, if p_ti denotes the frequency of the specified allele in taxon t_i, then

$$p_{t_n}=\lambda p_{t_1}+(1-\lambda )p_{t_2}.$$ (5)

It follows that if for each of the taxa in a pair, a distance function d is linear in each component of the allele frequency vector at a locus, then the distances between the admixed taxon and other taxa are obtained as linear combinations of corresponding distances involving taxa t₁ and t₂. Therefore, for 1 ≤ i ≤ n − 1,

$$d_{t_n,t_i}=\lambda d_{t_1,t_i}+(1-\lambda )d_{t_2,t_i}.$$ (6)

Eq. 6 continues to hold if for a series of loci, the distance function d is linear for each taxon in each component of the allele frequency vector at each locus, as would occur if the distance between a pair of taxa at a set of loci were computed as the mean of locus-wise distances that were each linear in the components of the allele frequency vector at the specified locus.

We assume that the distance function supplied to neighbor-joining satisfies eq. 6, and that it is symmetric, nonnegative, and zero if and only if it is computed between a taxon and itself; we otherwise do not concern ourselves with the form of the function. While typical population-genetic distance functions often involve nonlinear relationships with allele frequencies and do not necessarily follow eq. 6—consider the nonlinear graphs in Figure 3 of Boca & Rosenberg,³⁸ which illustrate that for the F_ST measure and an admixed population t_n whose frequencies are linear combinations of those of populations t₁ and t₂, F_ST (t_n, t₁) ≠ λF_ST (t₁, t₁) + (1 − λ)F_ST (t₂, t₁)—eq. 6 is a natural extension of the ubiquitous eq. 5 from allele frequencies to distance functions. For F_ST, it can be shown from eqs. 1 and 7 of Boca & Rosenberg³⁸ that for small λ, F_ST (t_n, t₁) ≈ λF_ST (t₁, t₁) + (1 − λ)F_ST (t₂, t₁). Thus, we view eq. 6 as a reasonable first approximation for examining properties of neighbor-joining in an admixture scenario.

4. The neighbor-joining algorithm in an admixture scenario

Our goal is to construct a distance matrix according to the admixture rule in eq. 6, mechanistically apply neighbor-joining to the matrix, and characterize the properties of the inference process and the resulting inferred tree. We examine two settings. In the first, arbitrarily many taxa are considered, and their distances produce an additive distance matrix (and therefore satisfy a tree metric³⁹). In the second, a general matrix is investigated, with distances that do not necessarily follow a tree metric, but the matrix includes only four taxa.

4.1. The additive case for n taxa

We first assume that the distance matrix is additive. In this case, by the consistency property of the neighbor-joining algorithm,^5,8,9 distances between taxa on the constructed neighbor-joining tree exactly equal those of the input matrix. Denote by d̂ the distance function computed for pairs of nodes in the inferred neighbor-joining tree, such that for taxa t_i and t_j, d̂_titj is the sum of the lengths of the branches on the path connecting t_i and t_j. Recalling that d represents distance in the input distance matrix, if the matrix is additive, then for all (t_i, t_j),

$${\widehat{d}}_{t_i,t_j}=d_{t_i,t_j}.$$ (7)

Because d_t1,tn = (1 − λ)d_t1,t2 and d_t2,tn = λd_t1,t2 by eq. 6,

$${\widehat{d}}_{t_1,t_n}=(1-\lambda ){\widehat{d}}_{t_1,t_2}$$ (8)

$${\widehat{d}}_{t_2,t_n}=\lambda {\widehat{d}}_{t_1,t_2}.$$ (9)

It follows that d̂_t1,tn + d̂_t2,tn = d̂_t1,t2, from which we can infer that taxa t₁, t₂, and t_n are collinear in the inferred neighbor-joining tree, with t_n in the interior of the path from t₁ to t₂.

We can obtain an even stronger result. Consider a case with at least four taxa: t₁, t₂, t_n, and, without loss of generality, t₃ (Fig. 3A). In the inferred neighbor-joining tree, a path of length c connects taxon t₃ to some point P on the path from t₁ to t₂ (including the endpoints). Without loss of generality, we can assume that P lies on the path from t₁ to t_n (including the endpoints). We denote the distances d̂_t1,P and d̂_P,tn by nonnegative values y and z, respectively. We denote d̂_t1,t2 = d_t1,t2 = x, for some nonnegative x.

Fig. 3.

The case of an additive distance matrix for n taxa. (A) Illustration of distances in the model. In the text it is shown that y = 0. (B) The structure required for a tree. Taxa t₁ and t₂ lie at multifurcating nodes, with t_n on the line connecting them. The leaves connected to the multifurcations have labels t₃, t₄, …, t_n−1.

By eq. 7, d̂_t1,tn = y + z = d_t1,tn = (1 − λ)x. By eqs. 6 and 7,

$$d_{t_3,t_n}=\lambda d_{t_3,t_1}+(1-\lambda )d_{t_3,t_2}$$ (10)

$${\widehat{d}}_{t_3,t_n}=\lambda {\widehat{d}}_{t_3,t_1}+(1-\lambda ){\widehat{d}}_{t_3,t_2}.$$ (11)

In other words, c + z = λ(c + y) + (1 − λ)(c + x − y). Together with the relationship y + z = (1 − λ)x and the assumption that λ > 0, eq. 11 implies that y = 0. It then follows that taxon t₃ lies on a line with taxa t₁, t₂, and t_n. Further, taxon t₃ lies on the side of taxon t₁ opposite to taxa t₂ and t_n; otherwise, by eq. 11, we would have (1 − λ)x − c = λc + (1 − λ)(x − c), which requires c = 0. In turn, c = 0 implies d̂_t3,t1 = 0, and hence, d_t3,t1 = 0, contradicting the assumption that all pairs of taxa are separated by positive distances in the distance matrix.

We have therefore shown that for an additive tree with taxon t_n admixed between t₁ and t₂, any additional taxon beyond t₁, t₂, and t_n must be collinear with t₁, t₂, and t_n, and must lie exterior to the path connecting t₁ and t₂. Thus, each additional taxon t₃, t₄, …, t_n−1 is connected to t₁ or t₂ by an external branch. The admixture model together with the assumption of an additive distance matrix imposes such a strong restriction on the set of allowed distance matrices that it forces all taxa onto a highly constrained tree (Fig. 3B). When we consider the placement of each taxon t₃, t₄, …, t_n−1, we find that this tree has two multifurcating nodes separated by a line that joins taxa t₁ and t₂, with t_n as the only intervening taxon.

The additive case can assist in explaining phenomena observed empirically with admixed populations in the application of neighbor-joining:^21–25 in the additive case, t_n has external branch length 0, a result compatible with the short external branches detected for admixed taxa. Further, t_n lies on the path connecting t₁ and t₂, compatible with the observation that admixed taxa lie in the “middle” of inferred neighbor-joining trees, with external branches incident to the paths connecting their source taxa. We can thus see that the empirical Fig. 2 resembles Fig. 3B, as the short internal branches among Native Americans and Europeans give rise to a shape with near multifurcations on each side of the admixed Mestizo groups.

4.2. The case of n = 4 taxa, not necessarily additive

The additive case is restrictive and atypical of the population-genetic context, in which migration and admixture generate non-tree-like evolution. We can then consider the more general setting of arbitrary genetic distance matrices with positive entries, examining the smallest nontrivial case, with n = 4 taxa. In this case, the admixed taxon is t₄, with source taxa t₁ and t₂. We set the distances among taxa t₁, t₂, and t₃ to be d_t1,t2 = x, d_t1,t3 = y and d_t2,t3 = z, for some positive x, y, and z. Employing eq. 6, the distance matrix D has the form:

$$D=\left(\begin{array}{cccc}0& x& y& (1-\lambda )x\\ x& 0& z& \lambda x\\ y& z& 0& \lambda y+(1-\lambda )z\\ (1-\lambda )x& \lambda x& \lambda y+(1-\lambda )z& 0\end{array}\right).$$ (12)

Using eq. 1 to calculate the matrix Q used in deciding which taxa will agglomerate, we obtain

$$Q=\left(\begin{array}{cccc}0& q_1& q_2& q_3\\ q_1& 0& q_3& q_2\\ q_2& q_3& 0& q_1\\ q_3& q_2& q_1& 0\end{array}\right),$$ (13)

where

$$q_1=-(x+y+z)$$ (14)

$$q_2=-(2-\lambda )x-\lambda y-(2-\lambda )z$$ (15)

$$q_3=-(1+\lambda )x-(1+\lambda )y-(1-\lambda )z.$$ (16)

Examining the relationships among q₁, q₂, and q₃, we have that

$$q_1<q_2\iff x+z<y$$ (17)

$$q_1<q_3\iff x+y<z$$ (18)

$$q_2<q_3\iff y<(1-2\lambda )x+z.$$ (19)

As in the work of Eickmeyer & Yoshida,⁴⁰ we partition the four-dimensional space of possible values of (λ, q₁, q₂, q₃) according to the tree topologies produced by neighbor-joining.

Three tree topologies are possible with the four taxa (Fig. 4). In Fig. 4A, taxon t₄ is separated by three edges from taxa t₁ and t₂, which themselves are separated by only two edges. In Fig. 4B, t₄ is separated by two edges from t₂ and by three edges from t₁; t₁ and t₂ are separated by three edges. Taxa t₁ and t₂ are also separated by three edges in Fig. 4C, but t₄ is instead separated by two edges from t₁ and by three edges from t₂.

Fig. 4.

The three possible topologies for n = 4 taxa. Node u is the unique node that places t₁, t₂, and t₄ in different subtrees.

Seven possibilities exist for the smallest entry of Q: (1) q₁, (2) q₂, (3) q₃, (4) q₁ and q₂ (tied), (5) q₁ and q₃ (tied), (6) q₂ and q₃ (tied), and (7) q₁, q₂, and q₃ (all tied). Each choice leads to a particular outcome among the three tree topologies in Fig. 4, with two or more topologies being possible outcomes in cases that involve ties. For each value among q₁, q₂, and q₃, two pairs of taxa produce the same value in the matrix Q. It can be shown that in each case, either choice of which pair is first to agglomerate leads to the same inferred tree. Without loss of generality, we choose the pair that does not include taxon t₃.

Four of the seven cases are not possible. In case 1, summing x + z < y and x + y < z in eqs. 17 and 18, we obtain x < 0. In case 4, setting q₁ = q₂ in eq. 17, x + z = y, from which we obtain x < 0 using x + y < z in eq. 18. Similarly, in case 5, q₁ = q₃ in eq. 18 produces x < 0 using x + z < y in eq. 17. In case 7, eqs. 17–19 become equalities, leading to x = 0 when x + z = y is substituted into eq. 19. All of these cases contradict the assumption that x > 0.

We consider the three allowable cases (cases 2, 3, and 6). For each of the possible inferred neighbor-joining trees, denote by u the unique interior node that places taxa t₁, t₂, and t₄ in distinct subtrees (Fig. 4). Denote by d̂ the distance between nodes on the inferred tree. In case 2, q₂ is smallest, taxa t₂ and t₄ agglomerate first, and using eqs. 2–4, we obtain

$${\widehat{d}}_{u,t_2}=(\lambda /4)(3x-y+z)$$ (20)

$${\widehat{d}}_{u,t_4}=(\lambda /4)(x+y-z)$$ (21)

$${\widehat{d}}_{u,t_1}=(1-\lambda )x.$$ (22)

We can show that d̂_u,t4 < d̂_u,t2 and d̂_u,t4 < d̂_u,t1. The first of these two inequalities is equivalent to λy < λ(x + z), which holds because λ > 0, and because y < x + z by eq. 17. For the second inequality, note first that y < (1 − 2λ)x + z by eq. 19. Substituting the right-hand side in place of y in eq. 21, d̂_u,t4 is less than [2λ(1 − λ)]x/4, which in turn is less than d̂_u,t1 because 0 < λ < 1.

In case 3, q₃ is smallest, taxa t₁ and t₄ agglomerate first, and using eqs. 2–4, we obtain

$${\widehat{d}}_{u,t_1}=[(1-\lambda )/4](3x+y-z)$$ (23)

$${\widehat{d}}_{u,t_4}=[(1-\lambda )/4](x-y+z)$$ (24)

$${\widehat{d}}_{u,t_2}=\lambda x.$$ (25)

Similarly to case 2, we show d̂_u,t4 < d̂_u,t1 and d̂_u,t4 < d̂_u,t2. The first inequality is equivalent to (1 − λ)z < (1 − λ)(x + y), which holds because λ < 1, and because z < x + y by eq. 18. For the second equality, (1 − 2λ)x + z < y by eq. 19. Substituting the left-hand side in place of y in eq. 24, d̂_u,t4 is less than [(2λ(1 − λ)]x/4, which in turn is smaller than d̂_u,t2 because 0 < λ < 1.

Finally, in case 6, q₂ and q₃ are tied with the smallest values, and either t₂ and t₄ agglomerate first as in case 2, or t₁ and t₄ agglomerate first as in case 3. Neighbor-joining produces the tree in Fig. 4C with probability 1/2, and the tree in Fig. 4B with probability 1/2. With either choice, the same arguments used to demonstrate d̂_u,t4 < d̂_u,t1 and d̂_u,t4 < d̂_u,t2 in cases 2 and 3 apply, except that y is equal to (instead of greater than or less than) (1 − 2λ)x + z.

This collection of results demonstrates three phenomena for four-taxon trees built from distance matrices formed according to our admixture model. (1) The admixed taxon agglomerates with one of its two source taxa before the sources agglomerate with each other. Cases 2, 3, and 6 are the only ones allowable, and in these cases, the first neighbor-joining step agglomerates the admixed taxon t₄ with one of the sources. (2) Denoting by u the unique node for which the admixed taxon and its source taxa all lie in different subtrees, the distance on the neighbor-joining tree of the admixed taxon to u is smaller than the distances to u of both source taxa. We demonstrated this result in each of the allowed cases, and it therefore holds in general. (3) The number of edges separating the source taxa on the inferred neighbor-joining tree, for each source taxon, is greater than or equal to the number of edges separating the admixed taxon from the source taxon. Only the trees in Figs. 4B and 4C are possible outcomes of neighbor-joining in our model, and the result holds for each of these trees.

5. Properties

Using the four-taxon results, we can formally define three properties of a distance matrix and its resulting neighbor-joining tree. The properties are well-defined for arbitrary n, and it is possible to evaluate whether a given n-taxon distance matrix satisfies them when neighbor-joining is applied. All three properties are possessed by all matrices generated by the four-taxon case of our admixture model.

Property 1: antecedence of clustering

The admixed taxon clusters with one of its source taxa before the source taxa cluster together. Stated precisely, some clade containing t₁ but not t₂ or t_n merges with some clade containing t_n but not t₁ or t₂, or, some clade containing t₂ but not t₁ or t_n merges with some clade containing t_n but not t₁ or t₂, before any clade containing t₁ but not t₂ or t_n merges with any clade containing t₂ but not t₁ or t_n.

Here we allow a clade to have any size, and potentially only a single taxon. In identifying the steps at which t₁, t₂, and t_n merge into the neighbor-joining tree, as in our four-taxon case, to ensure that these taxa do not all merge simultaneously at the final stage, we adopt the convention that if a four-taxon stage is reached in which t₁, t₂, and t_n lie in separate subtrees, we choose to agglomerate two among these three subtrees rather than agglomerating the third one with the unique available subtree that does not contain t₁, t₂, or t_n.

Property 2: intermediacy of distances

The distance on the constructed neighbor-joining tree between the admixed taxon and either of its source taxa is smaller than the corresponding distance between the two source taxa. That is, d̂_t1,tn < d̂_t1,t2 and d̂_t2,tn < d̂_t1,t2. Equivalently, if u is the unique node in the constructed neighbor-joining tree for which t₁, t₂, and t_n lie in different subtrees, then d̂_u,tn < d̂_u,t1 and d̂_u,tn < d̂_u,t2.

Property 3: intermediacy of path lengths

The number of edges separating the source taxa in the constructed neighbor-joining tree is greater than or equal to the number of edges separating the admixed taxon and either source taxon. If we define b̂_ij as the number of edges in the path separating nodes i and j in the inferred tree, then b̂_t1,t2 ≥ b̂_t1,tn and b̂_t1,t2 ≥ b̂_t2,tn.

We have already demonstrated that in our admixture model, Properties 2 and 3 hold for all distance matrices in the n-taxon additive case; for Property 2, using eqs. 8 and 9 and 0 < λ < 1, d̂_t1,tn < d̂_t1,t2 and d̂_t2,tn < d̂_t1,t2. For Property 3, we have shown that for an n-taxon additive distance matrix, taxon t_n lies on the interior of the path connecting t₁ and t₂, and it is the only taxon so located. Thus, b̂_t1,t2 = 2, while b̂_t1,tn = b̂_t2,tn = 1, and Property 3 holds.

6. Discussion

We have examined neighbor-joining in a model in which an admixed taxon is produced from two source taxa, finding that for a four-taxon scenario, distance matrices and their resulting trees possess three properties: antecedence of clustering, in which the admixed population clusters with one of the sources before the sources cluster with each other; intermediacy of distances, in which the distance on the constructed tree between the admixed taxon and either source taxon is less than the distance between the sources; and intermediacy of path lengths, in which the number of edges separating the admixed taxon and either source taxon is no larger than the number of edges separating the sources. We have further shown that for an arbitrary number of taxa, the latter two properties hold when the distance matrix is additive.

By a mechanistic examination, we have found that our model has features seen in empirical observations of neighbor-joining trees that involve admixed populations. In particular, the placement of admixed populations on short external branches incident to the paths connecting their source populations^21–25 matches the demonstration in the additive and four-taxon cases of the intermediacy of distances and intermediacy of path lengths properties. The theoretical approach validates the view that populations that are centrally located on neighbor-joining trees and that possess short external branches might be recently admixed.

Our results suggest a broader investigation of the extent to which the three properties hold with an arbitrary number of taxa. We have not reported a result regarding antecedence of clustering in the n-taxon additive case, nor have we commented on any of the properties for general n-taxon distance matrices that are not necessarily additive. However, we expect that Properties 1–3 will be satisfied by our admixture model considerably more often than in a model in which no special constraints are imposed on distances that involve the nth taxon. As our model also involves an nth taxon with special features, the general analysis of the model might benefit from the “rogue taxon” framework of Cueto & Matsen,⁴¹ in which the addition of an nth taxon alters the tree produced for an initial group of n − 1 taxa.

An additional direction is to study alternative admixture models. Distance methods are most sensible when a distance is nearly additive; however, eq. 6 severely restricts the distance matrix, as it forces a structure with two multifurcating nodes. This aspect of the model can be relaxed by assuming that the distance is additive for taxa t₁, t₂, …, t_n−1, and that only distances involving t_n satisfy eq. 6. For 1 ≤ i ≤ n − 1, we can then apply the distance

$$d_{t_n,t_i}=\{\begin{array}{ccc}{d}_{t_1,t_i}-(1-\lambda )d_{t_1,t_2}& \text{if}& (1-2\lambda )d_{t_1,t_2}\le d_{t_1,t_i}\\ d_{t_2,t_i}-\lambda d_{t_1,t_2}& \text{if}& (1-2\lambda )d_{t_1,t_2}\ge d_{t_1,t_i}.\end{array}$$ (26)

With this distance function, t_n is simply placed on the path from t₁ to t₂ in a preexisting tree relating taxa t₁, t₂, …, t_n−1 (Fig. 5). Properties 2 and 3 continue to hold.

Fig. 5.

Two possible placements of t_n with respect to t₁, t₂, and t_i, illustrating how t₁, t₂, …, t_n can have an additive distance matrix when t₁, t₂, …, t_n−1 have an additive distance matrix. For convenience, we illustrate only one representative taxon t_i from among {t₃, t₄, …, t_n−1}.

To obtain eq. 26, we first suppose that t₁, t₂, …, t_n−1 have an additive distance matrix. We wish to place taxon t_n on the tree that generates the matrix so that the matrix for t₁, t₂, …, t_n is additive. First, given λ, t_n is placed on the path from t₁ to t₂ such that eqs. 8 and 9 are satisfied. It remains to compute d̂_tn,ti for i = 3, 4, …, n − 1. Denote by u the unique node of the tree that places t₁, t₂, and t_i in distinct subtrees (Fig. 5). Then

$${\widehat{d}}_{t_1,u}=({\widehat{d}}_{t_1,t_2}+{\widehat{d}}_{t_1,t_i}-{\widehat{d}}_{t_2,t_i})/2$$ (27)

$${\widehat{d}}_{t_2,u}=({\widehat{d}}_{t_1,t_2}+{\widehat{d}}_{t_2,t_i}-{\widehat{d}}_{t_1,t_i})/2$$ (28)

$${\widehat{d}}_{t_i,u}=({\widehat{d}}_{t_1,t_i}+{\widehat{d}}_{t_2,t_i}-{\widehat{d}}_{t_1,t_2})/2.$$ (29)

If d̂_t1,tn ≤ d̂_t1,u, then t_n lies on the path from t₁ to u (Fig. 5A), and

$${\widehat{d}}_{t_n,t_i}={\widehat{d}}_{u,t_i}+{\widehat{d}}_{t_1,t_2}-{\widehat{d}}_{t_1,t_n}.$$ (30)

If, on the other hand, d̂_t1,tn ≥ d̂_t1,u, then t_n lies on the path from t₂ to u (Fig. 5B), and

$${\widehat{d}}_{t_n,t_i}={\widehat{d}}_{u,t_i}+{\widehat{d}}_{t_1,t_2}-{\widehat{d}}_{t_2,t_n}.$$ (31)

Applying eqs. 8, 9, and 27–29 together with the fact that d̂ = d for additive distance matrices, we produce the relationship in eq. 26.

Analysis of the three properties using this modified form for the admixture model, or more generally using specific distance functions commonly employed in population genetics, will further illuminate the features of neighbor-joining in admixed populations. Such analyses might also facilitate investigations of the behavior with admixed populations of other tree-building methods, or of phylogenetic network methods⁴² that are more directly designed to accommodate taxa with non-tree-like evolutionary histories.