Within a Sample from a Population, the Distribution of the Number of Descendants of a Subsample’s Most Recent Common Ancestor

Abstract

Sample n individuals uniformly at random from a population, and then sample m individuals uniformly at random from the sample. Consider the most recent common ancestor (MRCA) of the subsample of m individuals. Let the subsample MRCA have j descendants in the sample (m≤ j ≤ n). Under a Moran or coalescent model (and therefore under many other models), the probability that j = n is known. In this case, the subsample MRCA is an ancestor of every sampled individual, and the subsample and sample MRCAs are identical. The probability that j = m is also known. In this case, the subsample MRCA is an ancestor of no sampled individual outside the subsample. This article derives the complete distribution of j, enabling inferences from the corresponding p-value. The text presents hypothetical statistical applications pertinent to taxonomy (the gene flow between Neanderthals and anatomically modern humans) and medicine (the association of genetic markers with disease).

1 Introduction

Consider the following hypothetical situation. Within a sample of n individuals, a subsample of m individuals share a morphological character. Upon genetic analysis, the m individuals share some genetic characters with a further j–m≥0 individuals within the sample. One might desire a p-value to test whether j–m is “too small”, i.e., to test whether the concentration of the morphological character among individuals with the genetic characters is too excessive to reflect chance alone. This article derives a p-value by giving the sampling distribution of j. Depending on its context, a small p-value might suggest among other possibilities, e.g., that gene flow between the subpopulations represented by the subsample and its complement within the sample is not free (i.e., that the mathematical assumptions underlying the coalescent are violated), or that the genetic characters have a causal influence on the morphological character. The Discussion demonstrates how the p-value might be relevant to rejecting the hypothesis of free gene flow between Neanderthals and anatomically modern humans (Krings et al. 1997, Nordborg 1998, Krings et al. 2000) or to associating a genetic disease or phenotype to a set of DNA markers necessary but not sufficient for it.

To determine the distribution corresponding to the p-value, consider Kingman’s coalescent (Kingman 1982a, Kingman 1982b), where n individuals are sampled uniformly at random at time t₀ from a large population. Kingman examined a haploid population, but coalescent models can also apply to sexual populations (Nordborg 2004, Pollak 2004, Wakeley et al. 2012). A pure death process D_t (t ≥ 0) counts the ancestors of the sample at prior times t₀–t. The process D_t transitions through the states n→n–1→…→2→1, with the state D_t = k (k = 2, …, n) having a sojourn time τ_k exponentially distributed with parameter $d_k=\frac{1}{2}k(k-1)$ and with the state D_t= 1 absorbing.

The sample ancestry can be described using ℰ_n, the set of all equivalence relations on the n individuals. Consider the Markov chain ℛ_n→ℛ_n–1→…→ℛ₂→ ℛ₁, whose state-space is ℰ_n, where ℛ_k corresponds to having D_t = k ancestors (k = n, n – 1, …, 1). The variate ℛ_k partitions the n individuals into k equivalence classes, each equivalence class corresponding to an ancestor and containing the ancestor’s descendants at time t₀. Define the identity relation Δ = {(i, i): i = 1,2,…,n} and the trivial relation Θ={(i, j): i, j = 1,2, …,n}. Given ξ, η ∈ ℰ_n, let ξ ≺ η denote that η can be obtained from ξ by combining two equivalence classes in ξ, and in fact, Δ = ℛ_n ≺ ℛ_n–1 ≺ … ≺ ℛ₂ ≺ ℛ₁ = Θ. The transition probabilities of the Markov chain {ℛ_k} are

$$\mathbb{P}\{{ℛ}_{k-1}=\eta |{ℛ}_k=\xi \}=\{\begin{array}{cc}2/[k(k-1)]& \text{if}\xi \prec \eta \\ 0& \text{otherwise}\end{array}.$$ (1)

Kingman shows that if ξ contains k equivalence classes,

$$\mathbb{P}\{{ℛ}_k=\xi \}=\frac{(n-k)!k!(k-1)!}{n!(n-1)!}{\lambda }_1!{\lambda }_2!\dots {\lambda }_k!,$$ (2)

where λ₁, λ₂, …, λ_k, are the sizes of the equivalence classes of ξ. Eq (1) implies Eq (2), so Eq (2) holds for any model imposing Eq (1) on the ancestry of a sample, in particular the Moran model (Moran 1962, Kimura and Crow 1964, Watterson 1984, Donnelly and Tavare 1986a, Donnelly and Tavare 1986b) (without mutation), or indeed any model of ancestry approximating a coalescent process closely enough.

Now, draw a subsample of m individuals uniformly at random from the sample of n individuals. The subsample has a most recent common ancestor (MRCA). For 1 ≤ m ≤ j ≤ n , let p_{n,m; j} denote the probability that the subsample MRCA has j descendants within the sample. For j = n, e.g., the subsample has the same MRCA as the sample. From Theorem 2 in (Saunders et al. 1984) with l₁ = l₂ = 2 (or Example 1 in (Saunders et al. 1984)),

$$p_{n,m;n}=\frac{m-1}{m+1}\frac{n+1}{n-1}.$$ (3)

(See also p. 77 in (Hein et al. 2005))

In a standard notation (Graham et al. 1994), let n^m = n(n–1) … (n–m + 1) denote the falling factorial for 1≤m≤n, with n⁰ = 1. In addition to Eq (3), we have the trivial boundary cases p_n,1;1 = p_n,n;n = 1, so for m < n , consider the recursion

$$p_{n,m;m}=\frac{m(m-1)}{n(n-1)}{p}_{n-1,m-1;m-1}+\frac{(n-m)(n-m-1)}{n(n-1)}{p}_{n-1,m;m},$$ (4)

which conditions on ℛ₂ , the two terms corresponding to coalescences: (1) within the subsample (probability m(m–1)/[n(n–1)]); and (2) outside of the subsample (probability (n–m)(n–m–1)/[n(n–1)]) Eq (4) can provide an inductive proof of the formula

$$p_{n,m;m}=\frac{2(m-1)!}{(m+1){(n-1)}^{\underset{¯}{m-1}}}$$ (5)

from (Wiuf and Donnelly 1999). (See also, e.g., p. 84 in (Hein et al. 2005) and Eq (1) in (Rosenberg 2007.)

If j = m, Eq (5) provides a p-value p_n,m;m, to test whether under the assumptions underlying the coalescent, subsample ancestries are likely to coalesce before coalescing with the remainder of the sample (see, e.g., p. 86 in (Hein et al. 2005) for examples concerning Neanderthal ancestry (Nordborg 1998, Harris and Hey 1999)). If j > m, then the relevant (left-sided) p-value becomes a sum $p_{n,m;j,•}={\sum }_{i=m}^j{p}_{n,m;i}$. With the motivating applications mentioned in the Introduction, Theorem 1 in the Results section extends the analytic formula for p_{n,m; j} from j = m and j = n to 1 ≤ m ≤ j ≤ n.

2 Theory

Theorem 1: Let 1 ≤ m ≤ n, and consider a sample whose ancestry satisfies Eq (1). Under the set-up described above, for m = 1, definitions show that p_{n,m; j} equals 1 if j = 1 and 0 otherwise. For m > 1,

$$p_{n,m;j}=\{\begin{array}{cc}\frac{m-1}{m+1}\frac{2{(j-2)}^{\underset{¯}{m-2}}}{{(n-1)}^{\underset{¯}{m-1}}}& \text{for}2\le m\le j<n\\ \frac{m-1}{m+1}\frac{n+1}{n-1}& \text{for}2\le m\le j=n\end{array},$$ (6)

with p_n,m;j= 0 unless 2 ≤ m ≤ j ≤ n.

Remark: Eq (6) reduces to Eq (5) in the case j = m, as it should.

Proof: Note the following identity for 1≤ a ≤ b:

$$a\sum _{i=a}^b{(i-1)}^{\underset{¯}{a-1}}=\sum _{i=a}^b[i-(i-a)]{(i-1)}^{\underset{¯}{a-1}}=\sum _{i=a}^b[i^{\underset{¯}{a}}-{(i-1)}^{\underset{¯}{a}}]=b^{\underset{¯}{a}}$$ (7)

where the second equality follows because i(i–1)^a–1 = i^a and (i–1)^a–1(i–a) = (i–1)^a–1.

Thus, ${\sum }_{j=m}^n{p}_{n,m;j}=1$ for 2 ≤m ≤ n:

$$\frac{m-1}{m+1}[\frac{n+1}{n-1}+\sum _{j=m}^{n-1}\frac{2{(j-2)}^{\underset{¯}{m-2}}}{{(n-1)}^{\underset{¯}{m-1}}}]=\frac{m-1}{m+1}[\frac{n+1}{n-1}+\frac{2}{{(n-1)}^{\underset{¯}{m-1}}}\sum _{j=m}^{n-1}{(j-2)}^{\underset{¯}{m-2}}]=\frac{m-1}{m+1}[\frac{n+1}{n-1}+\frac{2}{{(n-1)}^{\underset{¯}{m-1}}}\frac{{(n-2)}^{\underset{¯}{m-1}}}{m-1}],=\frac{m-1}{m+1}[\frac{n+1}{n-1}+\frac{2}{n-1}\frac{n-m}{m-1}]=1$$ (8)

where the second equality follows from Eq (7).

For m = 1, definitions yield p_n,1;1 = 1, and for m > 1, p_n,m;j = 0 unless m ≤ j ≤ n. To set up an inductive proof of Theorem 1 for the cases in Eq (6), let 𝒫_i be the proposition that Theorem 1 holds for every 2 ≤ m ≤ j ≤ n ≤ i. To start the induction, 𝒫₂ is true, because by definition p_2,2;2 = 1, agreeing with Eq (6) for 2 ≤ m ≤ j = n ≤2 (the other case 2 ≤ m ≤ j < n ≤ 2 being vacuous).

For the inductive step, assume 𝒫_n–1 holds for some fixed n ≥3. From Eq (2) for k = 2, the probability that one of the two equivalence classes of ℛ₂ contains all m subsample individuals and has a total of i elements is

$$[\frac{(n-2)!2!1!}{n!(n-1)!}i!(n-i)!]\frac{(n-m)!}{(n-i)!(i-m)!}=\frac{2}{n-1}\frac{i^{\underset{¯}{m}}}{n^{\underset{¯}{m}}},$$ (9)

because there are (n–m)!/[(n–i)!(i–m)!] equally probable ways of forming the two equivalence classes of ℛ₂ by placing the m subsample individuals into an equivalence class of i elements.

As usual, let empty sums equal 0. To check that 𝒫_n follows from 𝒫_n–1, we check first that Eq (6) holds f or 2 ≤ m ≤ j < n, then conclude from ${\sum }_{j=m}^n{p}_{n,m;j}=1$ and Eq (8) that Eq (6) also holds for 2 ≤ m ≤ j = n. For 2 ≤ m ≤ j < n, then,

$$p_{n,m;j}=\frac{2}{n-1}\sum _{i=j}^{n-1}\frac{i^{\underset{¯}{m}}}{n^{\underset{¯}{m}}}{p}_{i,m;j}=\frac{2}{n-1}[\frac{j^{\underset{¯}{m}}}{n^{\underset{¯}{m}}}\frac{m-1}{m+1}\frac{j+1}{j-1}+\sum _{i=j+1}^{n-1}\frac{i^{\underset{¯}{m}}}{n^{\underset{¯}{m}}}\frac{m-1}{m+1}\frac{2{(j-2)}^{\underset{¯}{m-2}}}{{(i-1)}^{\underset{¯}{m-1}}}]=\frac{m-1}{m+1}\frac{2}{n-1}\frac{1}{n^{\underset{¯}{m}}}[j^{\underset{¯}{m}}\frac{j+1}{j-1}+2{(j-2)}^{\underset{¯}{m-2}}\sum _{i=j+1}^{n-1}i]=\frac{m-1}{m+1}\frac{2}{n-1}\frac{1}{n^{\underset{¯}{m}}}\{j^{\underset{¯}{m}}\frac{j+1}{j-1}+2{(j-2)}^{\underset{¯}{m-2}}\frac{1}{2}[n^{\underset{¯}{2}}-{(j+1)}^{\underset{¯}{2}}\left]\right\}=\frac{m-1}{m+1}\frac{2{(j-2)}^{\underset{¯}{m-2}}}{{(n-1)}^{\underset{¯}{m-1}}}$$ (10)

where the first equality is justified as follows. One of the two equivalence classes of ℛ₂ must contain all m individuals from the subsample. Let the equivalence class of ℛ₂ containing the subsample has size i, where i ∈ {j, j + 1, …, n – 1}. In each case, Eq (9) gives the probability of the size i, the weight for p_i,m;j in the right side of the first equality in Eq (10).

Now, for m ≥ 2, every j with p_n,m;j > 0 satisfies 2 ≤ m ≤ j ≤ n. Because Eq (10) applies to the cases 2 ≤ m ≤ j < n, Eq (8) shows that Eq (3) for p_n,m;n holds. Thus, Eq (6) also holds for 2 ≤ m ≤ j = n , so 𝒫_n follows from 𝒫_n–1, completing the induction and the proof.

Corollary 1: In the set-up of Theorem 1, the left-sided p-value $p_{n,m;j,•}={\sum }_{i=m}^j{p}_{n,m;i}$ is p_n,1;1,•= for n ≥1 and

$$p_{n,m;j,•}=\{\begin{array}{cc}\frac{2}{m+1}\frac{{(j-1)}^{\underset{¯}{m-1}}}{{(n-1)}^{\underset{¯}{m-1}}}& \text{for}2\le m\le j<n\\ 1& \text{for}2\le m\le j=n\end{array}.$$ (11)

Proof: For 2 ≤ m ≤ j < n,

$$p_{n,m;j,•}=\sum _{i=m}^j{p}_{n,m;i}=\frac{2}{(m+1){(n-1)}^{\underset{¯}{m-1}}}(m-1)\sum _{i=m}^j{(i-2)}^{\underset{¯}{m-2}}=\frac{2}{(m+1){(n-1)}^{\underset{¯}{m-1}}}(m-1)\sum _{i^{\prime }=m-1}^{j-1}{(i^{\prime }-1)}^{\underset{¯}{m-2}}=\frac{2{(j-1)}^{\underset{¯}{m-1}}}{(m+1){(n-1)}^{\underset{¯}{m-1}}},$$ (12)

where the third equality changes the index of summation from i to i′= i – 1, and the final equality follows from the identity in Eq (7) with a = m – 1 and b = j. Theorem 1 completes the proof, because for 2 ≤ m ≤ j = n, $p_{n,m;n,•}={\sum }_{i=m}^n{p}_{n,m;i}=1$.

Corollary 2: In the set-up of Theorem 1, and in the limit n → ∞ and jn⁻¹ → f with m fixed, the left-sided p-value $p_{n,m;j,•}={\sum }_{i=m}^j{p}_{n,m;i}$ satisfies

$$\text{lim}{p}_{n,m;j,•}=\{\begin{array}{cc}\frac{2}{m+1}{f}^{m-1}& \text{for}2\le m\le j<n\\ 1& \text{for}2\le m\le j=n\end{array}.$$ (13)

3 Numerical Results

Typically, as Eq (3) suggests, much of the probability mass of {p_n,m;j: j = m,m + 1,…,n} occurs at p_n,m;n. To explore the corresponding left-sided p-values $p_{n,m;j,•}={\sum }_{i=m}^j{p}_{n,m;i}$ numerically, p_n,m;n,• = 1, and

$$p_{n,m;j,•}=p_{n,m;j+1,•}\frac{j-m+1}{j}\text{for2}\le m\le j<n,$$ (14)

suggesting that typical left-sided p-values {p_{n,m;j, •}} (m ≤ j ≤ n) decrease rapidly as j decreases from j = n to j = m.

In Figure 1, e.g., p_16,8;12,• ≈ 0.01 , suggesting that inference on samples as small as n = 16 can be surprisingly strong, even for j > m. For large samples (n→ ∞), Corollary 2 confirms that small subsamples of m individuals can produce strong inferences, even if j is much larger than m.

Figure 1. A semi-logarithmic plot of the left-sided p-value p_n,m;j,• against j for m ≤ j ≤ n = 8 and m ≤ j ≤ n = 16.

The solid circles and lines correspond to n = 8; the open circles/triangles and dotted lines correspond to n = 16. Points on a curve share a common value of m, given in the appropriate color on the left of the curve. Each point on the curve corresponds to (j, p_n,m;j,•) plotted for the values of n and m common to the points on the curve.

4 Discussion

Theorem 1 is related to combinatorial results on unique event polymorphisms and the sub-trees corresponding to a mutation (Wiuf and Donnelly 1999). More specifically, a special case of Corollary 1 (Eq (5)) has been presented as a p-value for inferring monophyly (e.g., Eq (1) in (Rosenberg 2007), and as a p-value for inferring non-random mating (e.g., in the informal discussion of the phylogenetic relationship of Neanderthals and modern humans on p. 84 of (Hein et al. 2005)). Because Corollary 1 generalizes Eq (5), it has similar applications in taxonomy.

As a hypothetical example, consider the phylogenetic tree presented in (Krings et al. 2000) (the following description of the tree suffices for present purposes). The tree was consistent with reciprocal monophyly of Neanderthals and modern humans, but contained too few Neanderthals to conclude reciprocal monophyly at p ≤0.05 from tree topology alone (e.g., (Rosenberg 2007)). An alternative statistic, estimated times to most recent common ancestor (TMRCA) (Nordborg 1998), effectively excluded random mating, but not the possibility of some gene flow. The statistical conclusions based on TMRCAs, however, require assumptions about the entire history of human population sizes (e.g., (Tang et al. 2002)), whereas conclusions based on tree topology alone require less restrictive assumptions, ones about the size of the human population co-existing with Neanderthals. In general, less restrictive assumptions yield more robust statistical tests, so an inference based on tree topology alone is more robust than an inference based on estimated TMRCAs.

As Neanderthal sequences accumulate, the present state of knowledge does not exclude inferences on a future genetic sample of Neanderthals and modern humans that generates a hypothetical tree with n – m human ancestors co-existing with m Neanderthals. The hypothetical tree might contain j – m > 0 human ancestors sharing a most recent common ancestor with the Neanderthals, the remaining human ancestors lying on a second lineage. With the use of the p-value in Corollary 1, the tree topology on its own (despite displaying gene flow) could suffice to reject (and reject more strongly than the present data permit) random mating between Neanderthals and human ancestors.

As another hypothetical example of Corollary 1, consider a study of a human genetic disease sampling n individuals under restrictions justifying a statistical analysis with coalescent theory. Let m individuals within the sample display the disease, along with a genetic marker suspected as necessary for the disease. Because of incomplete genetic penetrance or the absence of additional but unknown genetic factors required for disease, a further j–m > 0 individuals might display the relevant marker without displaying the disease. As in the coalescent theory of unique event polymorphisms (R. C. Griffiths and Tavare 1998, Wiuf and Donnelly 1999, R.C. Griffiths and Tavare 2003, Tavare 2004), assume that the mutation generating the marker occurred only once. Corollary 1 could test if the association of the marker with the disease reaches statistical significance.

Any statistical test using Eq (6) or Corollary 1 is based solely on Eq (1), an assumption common to most coalescent models. Accordingly, such a test is more robust than tests based on more specific coalescent models. Although the test loses power through its sparse assumptions, the Numerical Results section suggests that nonetheless, tests based on Corollary 1 can be surprisingly powerful.