The Site Frequency Spectrum for General Coalescents

Abstract

General genealogical processes such as Λ- and Ξ-coalescents, which respectively model multiple and simultaneous mergers, have important applications in studying marine species, strong positive selection, recurrent selective sweeps, strong bottlenecks, large sample sizes, and so on. Recently, there has been significant progress in developing useful inference tools for such general models. In particular, inference methods based on the site frequency spectrum (SFS) have received noticeable attention. Here, we derive a new formula for the expected SFS for general Λ- and Ξ-coalescents, which leads to an efficient algorithm. For time-homogeneous coalescents, the runtime of our algorithm for computing the expected SFS is $O\left(n^2\right),$ where n is the sample size. This is a factor of $n^2$ faster than the state-of-the-art method. Furthermore, in contrast to existing methods, our method generalizes to time-inhomogeneous Λ- and Ξ-coalescents with measures that factorize as $\mathrm{\Lambda }\left(dx\right)/\zeta \left(t\right)$ and $\mathrm{\Xi }\left(dx\right)/\zeta \left(t\right),$ respectively, where ζ denotes a strictly positive function of time. The runtime of our algorithm in this setting is $O\left(n^3\right).$ We also obtain general theoretical results for the identifiability of the Λ measure when ζ is a constant function, as well as for the identifiability of the function ζ under a fixed Ξ measure.

WHEN summarizing sequence data from n individuals, a natural and often-used statistic is the site frequency spectrum (SFS), ${\widehat{\tau }}_n={\left({\widehat{\tau }}_{n,1},\dots ,{\widehat{\tau }}_{n,n-1}\right)}^T,$ where ${\widehat{\tau }}_{n,k}$ is simply the number of sites at which k of n individuals carry the mutant (or the derived) allele. Despite being only $n-1$ numbers, the SFS still contains a surprising amount of information about the history and structure of the population from which the individuals were sampled. Indeed, for neutrally evolving populations that are well modeled by Kingman’s coalescent (Kingman 1982), the expected value of the SFS was first computed for populations of constant size (Fu 1995), extended to populations of variable size (Griffiths and Tavaré 1998; Polanski et al. 2003; Polanski and Kimmel 2003), and has since been used as a statistic for demographic inference in numerous studies (e.g., Nielsen 2000; Gutenkunst et al. 2009; Coventry et al. 2010; Gravel et al. 2011; Excoffier et al. 2013; Bhaskar et al. 2015; Kamm et al. 2015; Gao and Keinan 2016).

Yet not all populations are well modeled by Kingman’s coalescent. In fact, Kingman’s coalescent can be viewed as a special case of a broader class of coalescent processes called Λ-coalescents (Pitman 1999; Sagitov 1999). While Kingman’s coalescent permits only pairwise mergers of lineages, Λ-coalescents allow two or more lineages to merge simultaneously in a single coalescence event. Such events arise when a single individual has many offspring (Möhle and Sagitov 2001; Eldon and Wakeley 2006), under models of recurrent selective sweeps (Durrett and Schweinsberg 2004, 2005), in populations undergoing continuous strong selection (Neher and Hallatschek 2013; Schweinsberg 2015), and in many other models. Λ-Coalescents can further be seen as special cases of a broader class of coalescents called Ξ-coalescents (Schweinsberg 2000). In Ξ-coalescents, more than one merger event can occur simultaneously, resulting in simultaneous multiple mergers. While Ξ-coalescents have received less attention than Λ-coalescents in the literature, they still arise in certain models of selection (Huillet 2014), models of selective sweeps (Durrett and Schweinsberg 2005), models with repeated strong bottlenecks (Birkner et al. 2009), and for certain diploid mating models (Möhle and Sagitov 2003). Also, since Ξ-coalecents generalize Λ-coalescents, any results presented about Ξ-coalescents immediately pertain to Λ-coalescents.

More formally, time-homogeneous Ξ-coalescents are governed by a measure $\mathrm{\Xi }\left(dx\right)$ on the set $\left\{\left(x_1,x_2,\dots \right):x_1\ge x_2\ge \cdots \ge 0,{\displaystyle {\sum }_{i=1}^{\mathrm{\infty }}{x}_i}\le 1\right\}.$ Furthermore, we consider time-inhomogeneous Ξ-coalescents with measures that decompose into a time-independent part $\mathrm{\Xi }\left(dx\right)$ and a strictly positive function $\zeta :{\mathrm{ℝ}}_{\ge 0}\to {\mathrm{ℝ}}_{+}$ of time, where $\zeta \left(t\right)$ represents (for historical reasons) the inverse intensity. That is, the coalescent is now governed by the measure $\mathrm{\Xi }\left(dx\right)/\zeta \left(t\right).$ For example, for Kingman’s coalescent, $\mathrm{\Xi }\left(dx\right)={\delta }_{\mathbf{0}}\left(dx\right),$ the point mass at zero, and $\zeta \left(t\right)$ corresponds to the scaled effective population size at time t. For other models, $\zeta \left(t\right)$ does not necessarily correspond to the population size, but has an interpretation specific to the model. For example, Neher and Hallatschek (2013) show empirically that the rate of coalescence events in a model of continuous strong selection is a nonlinear function of the population size and the first two moments of the distribution of mutational effects. For a review of the mechanics of Λ-coalescents, see Pitman (1999) and for a review of Ξ-coalescents, see Schweinsberg (2000). For an alternative perspective, see Donnelly and Kurtz (1999) and Birkner et al. (2009) for a lookdown construction of particle systems with general reproduction mechanisms.

As mentioned above, the expected SFS for Kingman’s coalescent is well understood and can, in fact, be computed for an arbitrary ζ in $O\left(n^2\right)$ time (Polanski and Kimmel 2003). For Λ- and Ξ-coalescents, however, the expected SFS can be computed only for constant ζ and the method for Λ-coalescents takes $O\left(n^4\right)$ time (Birkner et al. 2013a) and the method for Ξ-coalescents takes time exponential in n as a sum over partitions of the first n numbers must be performed (Blath et al. 2015). Here we present a method that can compute the expected SFS for time-inhomogeneous Λ- and Ξ-coalescents with arbitrary ζ in $O\left(n^3\right)$ time. In the case where ζ is a constant function, our method can compute the expected SFS in $O\left(n^2\right)$ time given the rate matrix Q of the ancestral process, which is defined more precisely below. We also prove some results about the sample size needed to make Λ identifiable for popular classes of Λ measures for constant ζ, as well as results about the sample size needed to make ζ identifiable for a fixed $\mathrm{\Xi }\left(dx\right).$

There has also been some related work on determining the asymptotic behavior of the expected SFS as $n\to \mathrm{\infty }.$ In this setting, Berestycki et al. (2007, 2014) derive some simple formulas for time-homogeneous Λ-coalescents that come down from infinity. For finite n, however, these asymptotic formulas can be rather inaccurate. Indeed, even for $n=10,000,$ Birkner et al. (2013a) show that for some Λ-coalescents, there is a sizable discrepancy between the asymptotic formulas and the SFS obtained by simulation, highlighting the need for finite-sample calculations. Nevertheless, such asymptotic results highlight some interesting properties of Λ-coalescents and are reviewed in Berestycki (2009).

The remainder of this article is organized as follows. We first present our main results about the computation of the SFS for time-inhomogeneous coalescents and discuss the practical runtime of our implementation. We also investigate the variation in the empirical SFS and study the ability to infer the underlying model, using the empirical SFS. Then, we prove some identifiability results about general coalescents. We conclude with a discussion on the implications of our results.

Main Theoretical Results on the Expected SFS

Here we present our theoretical results on the expected SFS for a general Ξ-coalescent with a measure of the form $\mathrm{\Xi }\left(dx\right)/\zeta \left(t\right).$ These results lead to an $O\left(n^3\right)$-time algorithm for computing the expected SFS and can be improved to $O\left(n^2\right)$ if ζ is a constant function. Briefly, we use subsampling arguments to show that the expected SFS ${\tau }_n=\mathbb{E}\left[{\widehat{\tau }}_n\right]$ can be computed from $a_n:={\left(\mathbb{E}{T}_2^{\mathrm{MRCA}},\dots ,\mathbb{E}{T}_n^{\mathrm{MRCA}}\right)}^T,$ where $\mathbb{E}{T}_k^{\mathrm{MRCA}}$ denotes the expected time to the most recent common ancestor for sample size $k\in \left\{2,\dots ,n\right\}.$ Then, we show how to compute $a_n,$ using a spectral decomposition of the rate matrix Q of the ancestral process (also known as the block-counting process) of the time-homogeneous coalescent corresponding to $\mathrm{\Xi }\left(dx\right).$ More specifically, Q is a lower triangular matrix, where ${\left(Q\right)}_{ij}$ is the instantaneous rate at which i unlabeled lineages merge to form j unlabeled lineages when $\zeta \equiv 1.$ For example, for Kingman’s coalescent,

$${\left(Q\right)}_{ij}=\{\begin{array}{ll}\left(\begin{array}{c}i\\ 2\end{array}\right),\hfill & j=i-1,\hfill \\ -\left(\begin{array}{c}i\\ 2\end{array}\right),\hfill & j=i,\hfill \\ 0,\hfill & \text{otherwise}.\hfill \end{array}$$

Using this notation, we are now ready to state our main result. The rest of this section provides lemmas that contain formulas for the matrices in Theorem 1, as well as a proof of those lemmas and Theorem 1.

Theorem 1. Consider an arbitrary time-inhomogeneous Ξ-coalescent governed by a measure $\mathrm{\Xi }\left(dx\right)/\zeta \left(t\right),$ such that the expected time $c_{k,k}$ to the first coalescence for a sample of size k is finite for $k\in \left\{2,\dots ,n\right\}.$ Let $c_n={\left(c_{2,2},\dots ,c_{n,n}\right)}^T.$ Then, there exists a universal matrix $A\in {\mathrm{ℝ}}^{n-1\times n-1}$ that does not depend on the measure and a matrix $L\in {\mathrm{ℝ}}^{n-1\times n-1}$ that depends on Ξ but not ζ, such that

$${\tau }_n=\frac{\theta }{2}A{a}_{n}and{a}_n=L{c}_n,$$

where $\theta /2$ is the population-scaled mutation rate. Furthermore, this allows ${\tau }_n$ to be computed in $O\left(n^3\right)$ time.

Computing the matrix L in Theorem 1 is costly. For time-homogeneous coalescents, it is possible to compute $a_n$ directly, resulting in the following corollary:

Corollary 1. In the same setting as Theorem 1, if ζ is a constant function, then ${\tau }_n$ can be computed in $O\left(n^2\right)$ time.

In what follows, Lemmas 1 and 2 provide formulas to compute the universal matrix A, while Lemmas 3 and 4 provide formulas to compute L, which is related to the spectral decomposition of the rate matrix Q. The expected first coalescence times $c_n={\left(c_{2,2},\dots ,c_{n,n}\right)}^T$ can be computed as (Polanski and Kimmel 2003; Bhaskar et al., 2015)

$$c_{k,k}={\displaystyle {\int }_0^{\mathrm{\infty }}\mathrm{\mathbb{P}}\left\{\text{time}\hspace{0.17em}\text{of}\hspace{0.17em}\text{first}\hspace{0.17em}\text{coalescence}\hspace{0.17em}\text{for}\hspace{0.17em}k\hspace{0.17em}\text{individuals}>t\right\}dt}={\displaystyle {\int }_0^{\mathrm{\infty }}{e}^{{\left(Q\right)}_{kk}{\displaystyle {\int }_0^t\left(1/\zeta \left(s\right)\right)ds}}dt}.$$

Note that since A and L do not depend on ζ, the SFS depends on time and the inhomogeneity of the coalescent process only through the first coalescence times $c_n.$

Lemma 1. Let ${\gamma }_n:={\left({\tau }_{2,1},{\tau }_{3,2},\dots ,{\tau }_{n,n-1}\right)}^T$ denote the antisingleton entries (i.e., entries where exactly one individual has the ancestral allele and all other individuals have the derived allele) of the SFS for samples of sizes $2,\dots ,n.$ Then,

$${\tau }_n=B{\gamma }_n,$$

where the entries of $B\in {\mathrm{ℝ}}^{n-1\times n-1}$ are given by

$${\left(B\right)}_{ij}=\{\begin{array}{ll}{\left(-1\right)}^{i-j}\frac{1}{j+1}\left(\begin{array}{c}n-i-1\\ j-i\end{array}\right)\left(\begin{array}{c}n\\ i\end{array}\right),\hfill & i\le j,\hfill \\ 0,\hfill & i>j.\hfill \end{array}$$

Proof. We use induction to show that

$${\tau }_{n,i}={\displaystyle \sum _{j=i}^{n-1}{\left(-1\right)}^{i-j}}\frac{1}{j+1}\left(\begin{array}{c}n-i-1\\ j-i\end{array}\right)\left(\begin{array}{c}n\\ i\end{array}\right){\tau }_{j+1,j}.$$ (1)

Using exchangeability and a subsampling argument similar to that of Kamm et al. (2015, lemma 2), we obtain, for $k>l+1,$

$${\tau }_{k-1,l}=\frac{l+1}{k}{\tau }_{k,l+1}+\frac{k-l}{k}{\tau }_{k,l},$$ (2)

which follows from removing an individual uniformly at random from a sample of size k. Now, define the level of ${\tau }_{n,i}$ as $n-i$ and note that (1) holds for level 1, i.e., for ${\tau }_{l,l-1}$ on the left-hand side. Assume that (1) holds for level $n-i-1.$ Then,

$$\begin{array}{c}{\tau }_{n,i}=\frac{n}{n-i}{\tau }_{n-1,i}-\frac{i+1}{n-i}{\tau }_{n,i+1}\\ =\frac{n}{n-i}\left[{\displaystyle \sum _{j=i}^{n-2}{\left(-1\right)}^{i-j}}\frac{1}{j+1}\left(\begin{array}{c}n-i-2\\ j-i\end{array}\right)\left(\begin{array}{c}n-1\\ i\end{array}\right){\tau }_{j+1,j}\right]\\ \hspace{0.17em}-\frac{i+1}{n-i}\left[{\displaystyle \sum _{j=i+1}^{n-1}{\left(-1\right)}^{i+1-j}}\frac{1}{j+1}\left(\begin{array}{c}n-i-2\\ j-i-1\end{array}\right)\left(\begin{array}{c}n\\ i+1\end{array}\right){\tau }_{j+1,j}\right]\\ =\left(\begin{array}{c}n\\ i\end{array}\right)\left\{\frac{1}{i+1}{\tau }_{i+1,i}+{\left(-1\right)}^{n-1-i}\frac{1}{n}{\tau }_{n,n-1}+{\displaystyle \sum _{j=i+1}^{n-2}{\left(-1\right)}^{i-j}}\frac{1}{j+1}\left[\left(\begin{array}{c}n-i-2\\ j-i\end{array}\right)+\left(\begin{array}{c}n-i-2\\ j-i-1\end{array}\right)\right]{\tau }_{j+1,j}\right\}\\ =\left(\begin{array}{c}n\\ i\end{array}\right){\displaystyle \sum _{j=i}^{n-1}{\left(-1\right)}^{j-i}}\frac{1}{j+1}\left(\begin{array}{c}n-i-1\\ j-i\end{array}\right){\tau }_{j+1,j},\end{array}$$

where the first equality holds by the recursion (2) and the second equality holds by the inductive hypothesis, by noting that ${\tau }_{n-1,i}$ and ${\tau }_{n,i+1}$ are both one level below ${\tau }_{n,i}.$□

The following lemma relates ${\mathit{\gamma }}_n$ to $a_n:$

Lemma 2. Let γ, $a_n,$ and θ be defined as above. Then,

$${\gamma }_n=\frac{\theta }{2}C{a}_n,$$

where $C\in {\mathrm{ℝ}}^{n-1\times n-1}$ is bidiagonal with ${\left(C\right)}_{k,k-1}=-\left(k+1\right)$ and ${\left(C\right)}_{kk}=k+1$ for $k\in \left\{2,\dots ,n-1\right\},$ and ${\left(C\right)}_{11}=2.$

Proof. As in the Proof of Lemma 1, we employ a subsampling argument. Consider a sample of size $k+1.$ The only way that a subsample of size k can have a different time to the most recent common ancestor is if the removed individual is a singleton after all of the other lineages have coalesced. The probability that we remove that singleton to form our subsample is $1/\left(k+1\right).$ Then, the expected amount of time during which there is one singleton and all of the other individuals have coalesced scaled by the mutation rate is exactly the antisingleton entry. Thus,

$$\frac{1}{k+1}{\tau }_{k+1,k}=\frac{\theta }{2}\left(\mathbb{E}{T}_{k+1}^{\mathrm{MRCA}}-\mathbb{E}{T}_k^{\mathrm{MRCA}}\right)$$

for $k>1.$ When $k=1,$ there are only two lineages, so the total branch length is the antisingleton entry. Thus, ${\tau }_{2,1}=\left(\theta /2\right)2\mathbb{E}{T}_2^{\mathrm{MRCA}}.$ Rewriting this as a matrix equation for $k\in \left\{1,\dots ,n-1\right\}$ completes the Proof. □

By combining Lemmas 1 and 2, we obtain the universal matrix $A=BC.$ We now show how to compute the Ξ-dependent matrix L. First, we establish the following result on the decomposition of the rate matrix Q; this result was also obtained by Möhle and Pitters (2014, equation 2.3) for the Bolthausen–Sznitman coalescent.

Lemma 3. Fix an arbitrary Ξ-coalescent with ${\lambda }_i\ne {\lambda }_j$ for $i\ne j,$ where ${\lambda }_i:={\displaystyle {\sum }_{k=1}^{i-1}{\left(Q\right)}_{ik}}=-{\left(Q\right)}_{ii}.$ Let $Q\in {\mathrm{ℝ}}^{n\times n}$ denote the rate matrix of the ancestral process corresponding to $\mathrm{\Xi }\left(dx\right)$ (that is, the process counting the number of extant lineages at time t). Then,

$$Q=UE{U}^{-1},$$

where ${\left(E\right)}_{ij}={\delta }_{ij}{\left(Q\right)}_{ii},$ with ${\delta }_{ij}$ being the Kronecker delta that equals 1 if $i=j$ and 0 otherwise, and

$${\left(U\right)}_{ij}=\{\begin{array}{ll}1,\hfill & i=j,\hfill \\ \frac{1}{{\lambda }_i-{\lambda }_j}{\displaystyle \sum _{k=j}^{i-1}{\left(Q\right)}_{ik}{\left(U\right)}_{kj}},\hfill & i>j,\hfill \\ 0,\hfill & otherwise.\hfill \end{array}$$

Proof. By the construction of U,

$${\left(U\right)}_{ij}{\left(Q\right)}_{jj}={\displaystyle \sum _{k=j}^i{\left(Q\right)}_{ik}{\left(U\right)}_{kj}},$$

which implies that $UE=QU.$ Then, since U is triangular and has strictly positive diagonal entries, it is invertible. Therefore, $Q=UE{U}^{-1}.$□

The following result relates $a_n:={\left(\mathbb{E}{T}_2^{\mathrm{MRCA}},\dots ,\mathbb{E}{T}_n^{\mathrm{MRCA}}\right)}^T$ and $c_n={\left(c_{2,2},\dots ,c_{n,n}\right)}^T:$

Lemma 4. Let $a_n$ and $c_n$ be defined as above. Fix an arbitrary Ξ measure and a strictly positive function ζ. Now consider a time-inhomogeneous coalescent governed by $\mathrm{\Xi }\left(dx\right)/\zeta \left(t\right).$ If $c_{k,k}<\mathrm{\infty },$ for $2\le k\le n,$ then

$$a_n=-{\left(UD\right)}_{2:n,2:n}{c}_n,$$

where $D\in {\mathrm{ℝ}}^{n\times n}$ is the diagonal matrix $\mathrm{diag}\left({\left[U^{-1}\right]}_{\cdot ,1}\right),$ with ${\left[U^{-1}\right]}_{\cdot ,1}$ denoting the first column of $U^{-1},$ and ${\left(UD\right)}_{2:n,2:n}$ denotes the submatrix of $UD$ in rows and columns 2 through n.

Proof. Note that $\mathbb{E}{T}_k^{\mathrm{MRCA}}={\displaystyle {\int }_0^{\mathrm{\infty }}\mathrm{\mathbb{P}}\left\{T_k^{\mathrm{MRCA}}>t\right\}dt}.$ Therefore,

$$\begin{array}{c}\mathbb{E}{T}_k^{\mathrm{MRCA}}={\displaystyle {\int }_0^{\mathrm{\infty }}\mathrm{\mathbb{P}}\left\{T_k^{\mathrm{MRCA}}>t\right\}dt}={\displaystyle {\int }_0^{\mathrm{\infty }}{\displaystyle \sum _{l=2}^k{\left[e^{Q{\displaystyle {\int }_0^t\left(1/\zeta \left(s\right)\right)ds}}\right]}_{kl}dt}}\\ ={\displaystyle {\int }_0^{\mathrm{\infty }}{\displaystyle \sum _{l=2}^n{\left[e^{Q{\displaystyle {\int }_0^t\left(1/\zeta \left(s\right)\right)ds}}\right]}_{kl}dt}}\\ ={\displaystyle {\int }_0^{\mathrm{\infty }}{\displaystyle \sum _{l=2}^n{\left[U{e}^{E{\displaystyle {\int }_0^t\left(1/\zeta \left(s\right)\right)ds}}{U}^{-1}\right]}_{kl}dt}},\end{array}$$

where the third equality follows from the fact that Q is lower triangular and hence so is its exponential. Now, since U is lower triangular, its inverse is as well. Therefore, we may ignore the value of ${\left[e^{E{\displaystyle {\int }_0^t\left(1/\zeta \left(s\right)\right)ds}}\right]}_{1,1}.$ Letting $F\left(t\right):=e^{E{\displaystyle {\int }_0^t\left(1/\zeta \left(s\right)\right)ds}}$ but with $F_{1,1}\left(t\right):=0,$ note that ${\displaystyle {\int }_0^{\mathrm{\infty }}F\left(t\right)dt}=\mathrm{diag}\left(0,c_n\right).$ Then we have

$$\mathbb{E}{T}_k^{\mathrm{MRCA}}={\displaystyle {\int }_0^{\mathrm{\infty }}{\displaystyle \sum _{l=2}^n{\left[UF\left(t\right)U^{-1}\right]}_{kl}dt}}={\displaystyle \sum _{l=2}^n{\left[U\hspace{0.17em}\mathrm{diag}\left(0,c_n\right)U^{-1}\right]}_{kl}}.$$

Now, note that ${\left(U\right)}_{i,1}=1$ for all i by Lemma 3 and induction. This implies ${\displaystyle {\sum }_{l=1}^n{\left[U^{-1}\right]}_{il}}={\delta }_{i1},$ or ${\displaystyle {\sum }_{l=2}^n{\left[U^{-1}\right]}_{il}}={\delta }_{i1}-{\left[U^{-1}\right]}_{i1}.$ Using this identity, we can rewrite the above expression for $\mathbb{E}{T}_k^{\mathrm{MRCA}}$ as

$$\mathbb{E}{T}_k^{\mathrm{MRCA}}=-{\displaystyle \sum _{j=2}^n{\left[{\left(UD\right)}_{2:n,2:n}\right]}_{k-1,j}{c}_{j,j}},$$

where $D=\mathrm{diag}\left({\left[U^{-1}\right]}_{\cdot ,1}\right).$ Collecting these equations over $k\in \left\{2,\dots ,n\right\}$ in matrix form leads to the desired result. □

Using Lemma 4, we now see that the matrix L from Theorem 1 is simply $-{\left(UD\right)}_{2:n,2:n}.$ Lemma 3 provides a recursion to compute U, and D may be computed by noting that ${\left(U^{-1}\right)}_{11}=1$ and then since $U{U}^{-1}=I,$ we have

$$U_{i1}^{-1}=-{\displaystyle \sum _{j=1}^{i-1}{\left(U\right)}_{ij}{\left(U^{-1}\right)}_{j1}}.$$

Proof of Theorem 1. Combining Lemmas 1, 2, and 4, we obtain the equations in Theorem 1. For the runtime, note that each of the $O\left(n^2\right)$ entries of U requires $O\left(n\right)$ computations, and so computing U is $O\left(n^3\right).$ The matrices composing A are known in closed form, however, and constructing D requires filling only $O\left(n\right)$ entries, each requiring $O\left(n\right)$ computations for a total of $O\left(n^2\right).$ To then obtain the SFS from $c_n$ simply requires iterated matrix vector products taking $O\left(n^2\right)$ time. The overall procedure thus requires $O\left(n^3\right).$

□

Lemma 5. For coalescents of the form $\mathrm{\Xi }\left(dx\right)/\zeta \left(t\right)$ where ζ is a constant function, $a_n$ can be computed recursively from $c_n$ and Q as follows:

$$\begin{array}{l}\mathbb{E}{T}_2^{\mathrm{TMRCA}}=c_{2,2}\\ \mathbb{E}{T}_k^{\mathrm{TMRCA}}=c_{k,k}+{\displaystyle \sum _{l=2}^{k-1}\frac{{\left(Q\right)}_{kl}}{{\lambda }_k}\mathbb{E}{T}_l^{\mathrm{TMRCA}}},for\hspace{0.17em}k>2.\end{array}$$

Proof. The formulas follow immediately from the homogeneity of the process, recursing on the number of individuals, and noting that the probability that the first coalescence event for a sample of size k results in k lineages merging down to l lineages is ${\left(Q\right)}_{kl}/{\lambda }_k.$

□

Proof of Corollary 1. Use Lemma 5 to compute $a_n$ in $O\left(n^2\right)$ time. Then, ${\tau }_n=A{a}_n$ by Theorem 1, which also takes $O\left(n^2\right)$ time to compute.

□

Remark 1. Other than computing U, the algorithm presented in Theorem 1 is $O\left(n^2\right).$ Thus, for the Bolthausen–Sznitman coalescent (Bolthausen and Sznitman 1998) or Kingman’s coalescent, where U is known in closed form (Möhle and Pitters 2014, theorem 1.1 and appendix), the SFS can be computed in $O\left(n^2\right)$ time even for nonconstant ζ.

Remark 2. The above results can easily be extended to a coalescent where both ζ and $\mathrm{\Xi }\left(dx\right)$ depend on t, as long as $\mathrm{\Xi }\left(dx\right)$ is piecewise constant. For example, in the recent past the population may evolve according to a β-coalescent, whereas for t greater than some $t_0$ the population may evolve according to Kingman’s coalescent. By setting ζ appropriately in Theorem 1, one may obtain a “truncated SFS” (Kamm et al. 2015) for each different $\mathrm{\Xi }\left(dx\right).$ Then, using the truncated SFS for each epoch and the same machinery as in Kamm et al. (2015), one may compute the full SFS. The same techniques also allow one to consider multiple populations, with each population perhaps evolving according to its own Ξ measure.

Numerical Results

We implemented Theorem 1 and Corollary 1 in Mathematica, and the notebook is available upon request. We can compute the SFS for an arbitrary coalescent for a sample of size $n=100$ in ∼1 sec and a sample of size $n=300$ in a matter of minutes on a laptop computer, which is orders of magnitude faster than the >1 hr reported for a sample size of $n=100$ using the current state-of-the-art method (Blath et al. 2015). Furthermore, Blath et al. (2015) consider only specific Ξ measures where the number of simultaneous multiple mergers is restricted. Our method has the same runtime for all Ξ measures (after computing the rate matrix and the vector of first coalescence times). See Figure 1 for runtime vs. sample size. Furthermore, as noted above, if the spectral decomposition of the rate matrix Q is known, then the algorithm is $O\left(n^2\right).$ We also present runtimes for the Bolthausen–Sznitman coalescent [which has a closed-form solution for the spectral decomposition (Möhle and Pitters 2014)] in Figure 1.

Figure 1.

Runtime result (in seconds). Theorem 1 was used to compute the SFS for the time-homogeneous Bolthausen–Sznitman coalescent. The solid line uses Lemma 3 to compute the spectral decomposition of Q resulting in a cubic runtime. The dashed line uses the closed-form representation of the spectral decomposition of the Bolthausen–Sznitman coalescent (Möhle and Pitters 2014, theorem 1.1) to compute the SFS in quadratic time. The dotted line uses Corollary 1, which is also quadratic.

As long as the rate matrix Q of the ancestral process can be found exactly, our method is numerically stable. This is the case for popular Λ-coalescents such as point-mass coalescents and β-coalescents, as well as point mass Ξ-coalescents. If the rate matrix must be evaluated numerically, however, high-precision computation may be needed to avoid potential numerical problems due to catastrophic cancellation.

Using simulations, we now investigate the variation in the empirical SFS across independent realizations of the coalescent process and study the ability to infer the underlying model, using the empirical SFS. We consider three different ζ’s, illustrated in Figure 2. Due to the association with population sizes in the case of Kingman’s coalescent, we refer to ζ as the history or population size history. However, we caution that depending on the finite population size model, ζ may not represent the population size, but some other biologically relevant parameter. We consider a constant size history, a bottleneck history that undergoes a temporary 10-fold size reduction, and a growth history with repeated population doublings. For each ζ, we consider β$\left(2-\alpha ,\alpha \right)$-coalescents with $\alpha \in \left\{1,1.5,2\right\}.$ Note that $\alpha =1$ corresponds to the Bolthausen–Sznitman coalescent, while $\alpha =2$ corresponds to the Kingman coalescent. For each of the nine distinct values of $\left(\zeta ,\alpha \right),$ we simulated $m=1000$ independent trees with $n=20$ leaves.

Figure 2.

$\zeta \left(t\right)$ for three demographic scenarios: a constant size history, a bottleneck history that undergoes a temporary 10-fold size reduction, and a growth history with repeated population doublings. Note that the y-axis is stretched by $y\mapsto \log \left(y\right).$

In Figure 3, we examine the observed variation in branch lengths across independent realizations of the coalescent process, from which we can deduce the variation in the observed SFS. Specifically, assume that each tree sampled from the coalescent process has the same mutation rate, and, without loss of generality, assume that time has been scaled such that the mutation rate is 1. Let ${\tilde{\tau }}_{n,k}$ be the sum of branch lengths with k leaves and recall that ${\widehat{\tau }}_{n,k}$ is the kth entry of the empirical SFS on the n observed individuals. Then, $\mathrm{\mathbb{P}}\left({\widehat{\tau }}_n|{\tilde{\tau }}_n\right)\sim \text{Poisson}\left({\tilde{\tau }}_n\right),$ and $\mathbb{E}\left[{\tilde{\tau }}_n\right]={\tau }_n.$ In Figure 3, we plot ${\tilde{\tau }}_{n,k}$ for each simulated tree, as well as its expected value ${\mu }_{n,k}.$ Defining ${\sigma }_{n,k}^2:=\mathrm{Var}\left({\tilde{\tau }}_{n,k}\right)$ for this case of $m=1,$ we also plot an estimate of the standard deviation ${\widehat{\sigma }}_{n,k}=\sqrt{\widehat{E}\left[{\tilde{\tau }}_{n,k}^2\right]-{\tau }_{n,k}^2},$ where $\widehat{E}$ is the empirical expectation. Now, if we sum the branch lengths and mutations over m independent trees [so then $\mathbb{E}\left[{\tilde{\tau }}_{n,k}\right]=m{\mu }_{n,k}$ and $\mathrm{Var}\left({\tilde{\tau }}_{n,k}\right)=m{\sigma }_{n,k}^2$], then ${\mu }_{n,k}$ and ${\sigma }_{n,k}^2$ describe the limiting behavior of both ${\tilde{\tau }}_{n,k}$ and ${\widehat{\tau }}_{n,k}$ as $m\to \mathrm{\infty }:$ by the central limit theorem, $\left(1/\sqrt{m}\right)\left({\tilde{\tau }}_{n,k}-{\tau }_{n,k}\right){\to }_d\mathcal{N}\left(0,{\sigma }_{n,k}^2\right)$ and $\left(1/\sqrt{m}\right)\left({\widehat{\tau }}_{n,k}-{\tau }_{n,k}\right){\to }_d\mathcal{N}\left(0,{\sigma }_{n,k}^2+{\mu }_{n,k}\right).$

Figure 3.

The distribution of the branch length subtending k leaves, for random trees under a β$\left(2-\alpha ,\alpha \right)$-coalescent and $n=20.$ The solid line is the expected value from Theorem 1. We simulated 1000 independent trees per scenario and their branch length results are shown as gray dots and box plots; the dashed line denotes the estimated standard deviation of the distribution. Note the y-axis is stretched by $y\mapsto \sqrt{y}.$ The mean and standard deviation give the limiting behavior of ${\widehat{\tau }}_{n,k}$ for many independent trees, under the central limit theorem. For most k (say, $k\ge 5$), the branch length is usually 0 and has high variance relative to the mean. Thus ${\widehat{\tau }}_{n,k}$ will tend to have higher relative accuracy for the smaller entries k.

A recent inconsistency result (Koskela et al. 2015, theorem 1) shows that a Λ measure cannot be inferred from a single tree ($m=1$), even as $n\to \mathrm{\infty }.$ Indeed, we see in Figure 3 that the branch lengths ${\tilde{\tau }}_{n,k}$ of a single tree can deviate substantially from ${\tau }_{n,k}.$ For most k (say, $k\ge 5$), typically ${\tilde{\tau }}_{n,k}=0$ or ${\tilde{\tau }}_{n,k}\gg {\mu }_{n,k},$ given a single tree. That is, for a single tree, branches subtending more than a few leaves are either not observed or much larger than the expected branch length. However, smaller k (especially the singletons, $k=1$) have smaller relative standard deviation ${\sigma }_{n,k}/{\mu }_{n,k}$ and thus will tend to have lower relative error $\left({\widehat{\tau }}_{n,k}-{\tau }_{n,k}\right)/{\tau }_{n,k}\approx \mathcal{N}\left(0,\left({\sigma }_{n,k}^2+{\mu }_{n,k}\right)/m{\mu }_{n,k}^2\right)$ as m increases.

In the case of Kingman’s coalescent, ζ is inferred by minimizing the Kullback-Leibler (KL) divergence between a normalized version of the empirical SFS and a normalized version of the expected SFS (e.g., Bhaskar et al. 2015, equation 10). Recall that the KL divergence, $D_{\mathrm{KL}}({\Phi }_1\parallel {\Phi }_2)$ between two discrete probability distributions, ${\Phi }_1\text{and}{\Phi }_2$ is ${\sum }_{\mathrm{i}}{\Phi }_{1\left(\mathrm{i}\right)}\log ({\Phi }_{1\left(\mathrm{i}\right)}{\Phi }_{2\left(\mathrm{i}\right)})$ where $0\log \left(0\right)$ is defined to be 0. We investigate how KL divergence behaves as a function of the number m of independent trees simulated in the case of Λ-coalescents. Let ${\tau }_n^{\left(\zeta ,\alpha \right)}$ be the expected SFS under model $\left(\zeta \left(t\right),\alpha \right)$ and ${\tilde{\tau }}_n^{\left(\zeta ,\alpha \right)}\left(m\right)$ the corresponding branch lengths summed over the first m simulated trees. Define $P^{\left(\zeta ,\alpha \right)}\left(k\right)\propto {\tau }_{n,k}^{\left(\zeta ,\alpha \right)}$ as the true probability distribution of derived alleles under scenario $\left(\zeta \left(t\right),\alpha \right)$ and ${\tilde{P}}_m^{\left(\zeta ,\alpha \right)}\left(k\right)\propto {\tilde{\tau }}_{n,k}^{\left(\zeta ,\alpha \right)}\left(m\right)$ as the conditional distribution of derived alleles, given the first m trees simulated under $\left(\zeta \left(t\right),\alpha \right).$ In Figure 4, we plot the KL divergence $D_{\mathrm{KL}}\left({\tilde{P}}_m^{\left({\zeta }_1,{\alpha }_1\right)}\parallel P^{\left({\zeta }_2,{\alpha }_2\right)}\right)$ as a function of m, for every $\left({\zeta }_1\left(t\right),{\zeta }_2\left(t\right),{\alpha }_1,{\alpha }_2\right)$ considered above (that is, ζ is constant, bottleneck, or growth, and α is 1, 1.5, or 2). In this case, we see that minimizing $D_{\mathrm{KL}}$ identifies the true scenario $\left({\zeta }_1\left(t\right),{\alpha }_1\left(t\right)\right)=\left({\zeta }_2\left(t\right),{\alpha }_2\left(t\right)\right)$ with access to only a moderate number of independent trees (between 10 and 100).

Figure 4.

The KL divergence $D_{\mathrm{KL}}\left({\tilde{P}}_m^{\left({\zeta }_1,{\alpha }_1\right)}\parallel P^{\left({\zeta }_2,{\alpha }_2\right)}\right),$ where $P^{\left({\zeta }_2,{\alpha }_2\right)}\left(k\right)\propto {\tau }_{n,k}^{\left({\zeta }_2,{\alpha }_2\right)}$ is the distribution of derived alleles under scenario $\left({\zeta }_2\left(t\right),{\alpha }_2\right),$ and ${\tilde{P}}_m^{\left({\zeta }_1,{\alpha }_1\right)}\left(k\right)\propto {\tilde{\tau }}_{n,k}^{\left({\zeta }_1,{\alpha }_1\right)}\left(m\right)$ is the conditional distribution of derived alleles, given the first m trees simulated under $\left({\zeta }_1\left(t\right),{\alpha }_1\right)$ and a mutation hitting one of those trees. For m large enough, ${\mathit{D}}_{\text{KL}}$ is minimized by the true parameters; i.e., $\left({\zeta }_1\left(t\right),{\alpha }_1\left(t\right)\right)=\left({\zeta }_2\left(t\right),{\alpha }_2\left(t\right)\right).$ $D_{\text{KL}}$ can typically discriminate the true scenario for $m=100$ trees. For $m=10$ trees, $D_{\text{KL}}$ is often, but not always, minimized by the true scenario.

Figure 4 is encouraging, as not too many independent trees are needed to distinguish between the different scenarios $\left(\zeta \left(t\right),\alpha \right).$ Unfortunately, in some cases it may be impossible to even sample two independent trees (J. Koskela, personal communication). For example, in the model of Birkner et al. (2013b), a multiple-merger event happens over a single “generation,” which can cause the multiple merger to affect unlinked sites, resulting in correlated coalescence times. However, in other models, multiple-merger events may affect the genome only locally, and thus trees from unlinked sites are independent. For example, in the selective sweep model of Durrett and Schweinsberg (2005), multiple mergers are caused by selective sweeps taking place over $O\left(\text{log}\left(N\right)\right)$ “generations,” and a site experiences a multiple merger if $r_N\log \left(2N\right)/s_N=O\left(1\right),$ where $N,s_N,r_N$ respectively parameterize the population size, selection strength, and recombination distance to the selected site. Thus, the independence of unlinked trees is not necessarily determined by the Λ or Ξ measure itself, but instead by the prelimiting model.

Identifiability Results

Before attempting to infer ζ or Ξ in practice, it is important to know whether such inference is possible using the SFS. For instance, when inferring ζ, if two different functions ${\zeta }_1$ and ${\zeta }_2$ produce the same SFS, then it is impossible to distinguish between the two using only the SFS. In such a case, we say that ζ is not identifiable. For Kingman’s coalescent if one allows ζ to be an arbitrary positive function that produces a finite SFS, then ζ is not identifiable (Myers et al. 2008). ζ is identifiable in the case of Kingman’s coalescent, however, if one restricts ζ to be from a set of biologically realistic functions (technically, a set of functions with only a finite number of oscillations) (Bhaskar and Song 2014, theorem 11). We show that a similar result holds for all coalescents of the form $\mathrm{\Xi }\left(dx\right)/\zeta \left(t\right)$ where $\mathrm{\Xi }\left(dx\right)$ is fixed.

In general it is impossible to infer Ξ from the SFS if Ξ is not restricted. There has been some interest, however, in the case of distinguishing between a subset of Λ-coalescents (Eldon et al. 2015). We prove some results about the identifiability of the measure for various subsets of Λ measures when ζ is a constant function. We also consider the question posed by Eldon et al. (2015) of whether the SFS can distinguish between exponential growth under Kingman’s coalescent and a class of Λ-coalescents with constant ζ, and we show that indeed it is possible to distinguish between these cases with a surprisingly small number of samples. We note that our identifiability results require knowledge of the exact expected SFS, whereas Eldon et al. (2015) focus on the case where the expected SFS is approximated using an empirical SFS, which is what occurs in practice.

Throughout this section we assume that one has the exact expected SFS (i.e., the object computed by Theorem 1).

Identifiability of ζ for fixed Ξ measure

Before proceeding to the results and proofs, we first introduce some notation. Let ${\mathrm{ℳ}}_K\left(\mathrm{ℱ}\right)$ denote the set of piecewise defined functions with at most K pieces made from some function family ℱ. Furthermore, let $\mathcal{S}\left(\mathrm{ℱ}\right)$ denote the sign-change complexity of ℱ. Informally, $\mathcal{S}\left(\mathrm{ℱ}\right)$ is the supremum of the number of times $f_1-f_2$ crosses 0 over functions $f_1,f_2\in \mathrm{ℱ},$ which is related to the number of oscillations each $f\in \mathrm{ℱ}$ is allowed to have [see Bhaskar and Song 2014, definition 4, for a formal definition of $\mathcal{S}\left(\mathrm{ℱ}\right)$]. We will also write ${\psi }_n^{\mathrm{\Xi }}$ for the number of 0 entries in ${\left[U^{-1}\right]}_{\cdot ,1}$ in the spectral decomposition of Q for a coalescent on n individuals governed by $\mathrm{\Xi }\left(dx\right).$ Furthermore, denote by χ the space of Ξ measures such that ${\left(Q\right)}_{k,k-1}>0$ for all k. That is, χ is the set of Ξ measures where for any sample size there is positive probability of a single pairwise merger. If we are considering only Λ-coalescents, then χ contains all Λ measures except for ${\delta }_1,$ the star coalescent. We now present our main identifiability results and a conjectured bound on ${\psi }_n^{\text{Ξ}}.$

Our main result on the identifiability of ζ is the following theorem.

Theorem 2. For an arbitrary Ξ-coalescent governed by the measure $\mathrm{\Xi }\left(dx\right)/\zeta \left(t\right)$ where $\mathrm{\Xi }\in \mathrm{\chi }$ is fixed, suppose $\mathcal{S}\left(\mathrm{ℱ}\right)<\mathrm{\infty }$ and $n\ge 2K+\left(2K-1\right)\mathcal{S}\left(\mathrm{ℱ}\right)+{\psi }_n^{\mathrm{\Xi }}.$ Then for each expected SFS ${\tau }_n$ there exists a unique $\zeta \in {\mathrm{ℳ}}_K\left(\mathrm{ℱ}\right)$ consistent with ${\tau }_n.$

First, note that in the case of Kingman’s coalescent, ${\psi }_n^{{\delta }_0}=0$ for all n, and so in some sense, Kingman’s coalescent is optimal in terms of the number of samples needed to ensure that a certain model space is identifiable. For the Bolthausen–Sznitman coalescent, ${\psi }_n^1=0$ for all n, which follows from the spectral decomposition (Möhle and Pitters 2014, theorem 1.1). For the point mass Λ-coalescent with mass at $1/2,$ for $n\ge 5,$ all odd entries of ${\left[U^{-1}\right]}_{\cdot ,1}$ are 0 and so ${\psi }_n^{{\delta }_{1/2}}>0$ for $n\ge 5,$ thus implying that larger samples (relative to Kingman’s coalescent or the Bolthausen–Sznitman coalescent) are needed for this coalescent to ensure that a given model space is identifiable. We suspect that $\mathrm{\Lambda }\left(dx\right)={\delta }_{1/2}$ is the worst case among all Ξ-coalescents in χ for identifiability, resulting in the following conjecture:

Conjecture 1. For all $\mathrm{\Xi }\in \mathrm{\chi }$ and $n\ge 3$, ${\psi }_n^{\mathrm{\Xi }}\le \left(n-1\right)/2-1.$

If this conjecture is true, then the bound on the sample size needed to have identifiability in Theorem 2 can be simplified to $n\ge 2\left[2K+\left(2K-1\right)\mathcal{S}\left(\mathrm{ℱ}\right)\right].$

Identifiability of the Λ measure for a constant ζ

We also have the following results for Λ-coalescents about the identifiability of the Λ measure.

Theorem 3. Consider the set of point-mass Λ-coalescents: $\left\{{\delta }_z:z\in \left[0,1\right]\right\}.$ If Λ is restricted to be in this set and $n\ge 3,$ then the expected SFS ${\tau }_n$ uniquely determines Λ.

Theorem 4. Consider the set of β-coalescents: $\left\{\mathrm{ℒ}\left(X\right):X\sim \beta \left(2-\alpha ,\alpha \right),\alpha \in \left[1,2\right)\right\}.$ If Λ is restricted to be in this set and $n\ge 3,$ then the expected SFS ${\tau }_n$ uniquely determines Λ.

Theorem 5. Consider the set of coalescents $\left\{{\delta }_0/a{e}^{-bt}:a,b>0\right\}\mathrm{\cup }\left\{{\delta }_z:z\in \left[0,1\right]\right\}\mathrm{\cup }\left\{\mathrm{ℒ}\left(X\right):X\sim \beta \left(2-\alpha ,\alpha \right),\alpha \in \left[1,2\right)\right\},$ that is, Kingman’s coalescent with exponential growth, point-mass coalescents, or β-coalescents. If Λ is restricted to be in this set and $n\ge 4,$ then the expected SFS ${\tau }_n$ uniquely determines Λ.

Theorem 5 gives a positive theoretical answer to the question of whether the SFS can distinguish between exponential growth and multiple-merger coalescents. Using the techniques presented below, it is straightforward to obtain similar results for other subsets of Λ-coalescents.

Proofs of the identifiability results

The following lemma is used in proving the theorems in this section and may be of independent interest, as it shows that given the SFS for n individuals one can compute the expected time to most recent common ancestor for sample sizes $2,\dots ,n$ or vice versa.

Lemma 6. For all Λ- and Ξ-coalescents, there is a bijection between the expected SFS ${\tau }_n$ and the expected times $a_n$ to the most recent common ancestor.

Proof. Combine Lemmas 1 and 2 to see that ${\tau }_n=BC{a}_n,$ with B and C being universal. Then, since B is upper triangular and all of its diagonal entries are nonzero, it is invertible. Furthermore, since C is bidiagonal and the diagonal entries are all nonzero, it is also invertible. Therefore, $BC$ is invertible and since ${\tau }_n$ and $a_n$ are related through an invertible matrix, the transformation is bijective.

□

To prove Theorem 2 we use the following lemma.

Lemma 7. Let ${\lambda }_k=-\left(Q_{kk}\right).$ For all $\mathrm{\Xi }\in \mathit{\chi }$ and all Λ other than $\mathrm{\Lambda }\left(dx\right)={\delta }_1\left(dx\right)$ (i.e., the star coalescent), the sequence ${\left({\lambda }_k\right)}_{k\ge 2}$ is strictly increasing.

Proof. Consider a sample of size $k+1$ and a subsample of size k. Without loss of generality, assume individual $k+1$ is removed to produce the subsample. The time to the first event is the same for both samples unless the first event involves only individual $k+1$ and one lineage from $\left\{1,\dots ,k\right\}$. That is, the total rate when there are $k+1$ lineages is equal to the total rate when there are k lineages plus k times the rate at which exactly a particular pair of individuals coalesce. Formally,

$${\lambda }_{k+1}={\lambda }_k+\frac{k}{\left(\begin{array}{c}k+1\\ 2\end{array}\right)}{\left(Q\right)}_{k+1,k}.$$

By assumption, ${\left(Q\right)}_{k+1,k}>0,$ and so the total rates must be strictly increasing.

□

We now prove Theorem 2. Our proof relies heavily on the proof of the corresponding result for Kingman’s coalescent (Bhaskar and Song 2014, theorem 11). We essentially show that this setting satisfies the same hypotheses as the Kingman’s coalescent case and then use that result to complete our proof.

Proof of Theorem 2. By Lemma 6, the SFS is uniquely determined by $a_n.$ Then, furthermore, note that from Lemma 3 the matrix U is invertible since it is triangular with all nonzero entries along the diagonal. Then, by the same argument as in Bhaskar and Song (2014, equation 12), we know that if the model space is not identifiable, then for each k not corresponding to a zero in D (contributing to ${\psi }_n^{\mathrm{\Xi }}$), ${\lambda }_k$ must be the root of the Laplace transform of two different functions in the model space. By Lemma 7, these are all distinct, resulting in $n-{\psi }_n^{\mathrm{\Xi }}$ roots. Then, by taking $n-{\psi }_n^{\mathrm{\Xi }}$ sufficiently large, we obtain a contradiction via the generalized version of Descarte’s rule of signs (Bhaskar and Song 2014, theorem 4) and the theorem is proved.

□

We now prove Theorems 3, 4, and 5. The idea is to explicitly calculate the $\mathbb{E}{T}_k^{\mathrm{MRCA}}$ for the first few k for each allowed Λ measure and then use Lemma 6 to show that if Λ is uniquely determined by the first few $\mathbb{E}{T}_k^{\mathrm{MRCA}},$ then it is uniquely determined by ${\tau }_n.$

Proof of Theorem 3. $\mathbb{E}{T}_2^{\mathrm{MRCA}}=1$ for all Λ in the set of possible Λ’s. Consider $\mathrm{\Lambda }={\delta }_z.$ Using Lemma 5 we see that

$$\mathbb{E}{T}_3^{\mathrm{MRCA}}=\frac{1}{3-2z}+\frac{3-3z}{3-2z}\cdot 1=\frac{4-3z}{3-2z}.$$ (3)

This is a monotonically decreasing function of $z\in \left[0,1\right],$ and so Λ is uniquely determined by $\mathbb{E}{T}_3^{\mathrm{MRCA}}.$ Then, appealing to Lemma 6, we see that Λ is uniquely determined by the SFS for $n\ge 3.$

□

Proof of Theorem 4. A calculation similar to (3) gives $\mathbb{E}{T}_3^{\mathrm{MRCA}}=\left(2+3\alpha \right)/\left(2+2\alpha \right)$ for $\mathrm{\Lambda }=\mathrm{ℒ}\left(X\right),X\sim \beta \left(2-\alpha ,\alpha \right),$ where $\alpha \in \left[1,2\right).$ This is a monotonically increasing function of $\alpha \in \left[1,2\right)$ and the claim follows from the same argument as in the proof of Theorem 3.

□

Proof of Theorem 5. Suppose that two distinct Λ-coalescents within the set of allowed models produce the same expected SFS for $n\ge 4.$ Then, by Lemma 6, they would have the same values of $\mathbb{E}{T}_2^{\mathrm{MRCA}},$ $\mathbb{E}{T}_3^{\mathrm{MRCA}},$ and $\mathbb{E}{T}_4^{\mathrm{MRCA}}.$ By Theorems 3 and 4, we know that the Λ measures cannot both be point-mass coalescents or β-coalescents. From Bhaskar and Song (2014, corollary 8), we also know that the Λ measures cannot both be Kingman’s coalescent with different exponential growth parameters. There are thus three cases. They are all straightforward, albeit tedious.

Case 1. One Λ measure is a point-mass coalescent and the other is a β-coalescent. Letting $\mathbb{E}{T}_2^{\mathrm{MRCA}}=1$ (without loss of generality), we can explicitly compute $\mathbb{E}{T}_4^{\mathrm{MRCA}}$ for the point-mass coalescent and the β-coalescent, using the same recursive idea as in the Proof of Theorem 3. Let $p_{m,k}$ denote the probability that when there are m lineages, exactly k of them are involved in the next coalescence event. Then, by Lemma 5 $\mathbb{E}{T}_4^{\mathrm{MRCA}}=c_{4,4}+p_{4,2}\mathbb{E}{T}_3^{\mathrm{MRCA}}+p_{4,3}\mathbb{E}{T}_2^{\mathrm{MRCA}}.$ In particular, for the point-mass coalescent ${\delta }_z,$ this implies $\mathbb{E}{T}_4^{\mathrm{MRCA}}=5/3+1/\left(2z-3\right)+\left(3-2z\right)/\left(18-24z+9z^2\right).$ Now, recalling the expression of $\mathbb{E}{T}_3^{\mathrm{MRCA}}$ in (3) and letting

$$\mathbb{E}{T}_3^{\mathrm{MRCA}}=t$$ (4)

implies $z=\left(3t-4\right)/\left(2t-3\right).$ Plugging this into $\mathbb{E}{T}_4^{\mathrm{MRCA}},$ we see that for the point-mass coalescent,

$$\mathbb{E}{T}_4^{\mathrm{MRCA}}=\frac{1}{3}\left[6t-4-\frac{2t-3}{6+t\left(3t-8\right)}\right].$$ (5)

A similar calculation for the β-coalescent shows that

$$\mathbb{E}{T}_4^{\mathrm{MRCA}}=\frac{1}{3}\left(6t-2+\frac{1}{t-2}\right),$$ (6)

with $t:=\mathbb{E}{T}_3^{\mathrm{MRCA}}$ under the β-coalescent. Equating (5) and (6) and solving for t results in the solution $t=1$ or $t=4/3.$ But, if $t=1,$ then we see that $z=1,$ the star coalescent, which corresponds to $\alpha =0$ for the β-coalescent, which is not in the set of allowed β-coalescents. If $t=4/3,$ we see that $z=0,$ which corresponds to Kingman’s coalescent, and $\alpha =2$ for the β-coalescent, which again is not in the set of allowed β-coalescents. Therefore, a point-mass coalescent and a β-coalescent with $\alpha \in \left[1,2\right)$ cannot have the same $\mathbb{E}{T}_2^{\mathrm{MRCA}},\mathbb{E}{T}_3^{\mathrm{MRCA}},$ and $\mathbb{E}{T}_4^{\mathrm{MRCA}}$ simultaneously.

Case 2. One Λ measure is a point-mass coalescent and the other is Kingman’s coalescent with exponential growth. Without loss of generality, assume that $\mathbb{E}{T}_2^{\mathrm{MRCA}}=1$ for the point-mass Λ-coalescent. The exponential-growth Kingman’s coalescent model considered here has $c_{m,m}=-\left(1/b\right)e^{\left(\begin{array}{c}m\\ 2\end{array}\right)/\left(ab\right)}\mathrm{Ei}\left[-\left(\begin{array}{c}m\\ 2\end{array}\right)/\left(ab\right)\right],$ where $\mathrm{Ei}\left(x\right):=-{\displaystyle {\int }_{-x}^{\mathrm{\infty }}\left(e^{-t}/t\right)dt}$ is the exponential integral (Bhaskar et al. 2015, supplemental material equation 5). Then, the constraint $\mathbb{E}{T}_2^{\mathrm{MRCA}}=c_{2,2}=1$ implies $b=-e^{1/d}\mathrm{Ei}\left(-1/d\right),$ where $d:=ab.$ Furthermore, assuming this constraint and applying Theorem 1 to Kingman’s coalescent, we obtain

$$\mathbb{E}{T}_3^{\mathrm{MRCA}}=\frac{3}{2}-\frac{c_{3,3}}{2}=\frac{3}{2}-\frac{e^{2/d}\mathrm{Ei}\left(-3/d\right)}{2\mathrm{Ei}\left(-1/d\right)},$$ (7)

$$\mathbb{E}{T}_4^{\mathrm{MRCA}}=\frac{9}{5}-c_{3,3}+\frac{c_{4,4}}{5}=\frac{9}{5}-\frac{e^{2/d}\mathrm{Ei}\left(-3/d\right)}{\mathrm{Ei}\left(-1/d\right)}+\frac{e^{5/d}\mathrm{Ei}\left(-6/d\right)}{5\mathrm{Ei}\left(-1/d\right)}.$$ (8)

Now, in addition to $\mathbb{E}{T}_2^{\mathrm{MRCA}},$ if the two coalescents have the same values of $\mathbb{E}{T}_3^{\mathrm{MRCA}}$ and $\mathbb{E}{T}_4^{\mathrm{MRCA}},$ then the right-hand sides of (4) and (7) must agree, while the right-hand sides of (5) and (8) must agree. This implies

$$\frac{f_1\left(d\right)+e^{5/d}\mathrm{Ei}\left(-\left(6/d\right)\right)f_2\left(d\right)}{\mathrm{Ei}\left(-\left(1/d\right)\right)f_2\left(d\right)}=0,$$ (9)

where

$$\begin{array}{c}{f}_1\left(d\right):=2\mathrm{Ei}\left(-\frac{1}{d}\right)\left\{e^{4/d}{\left[\mathrm{Ei}\left(-\frac{3}{d}\right)\right]}^2-4e^{2/d}\mathrm{Ei}\left(-\frac{3}{d}\right)\mathrm{Ei}\left(-\frac{1}{d}\right)+{\left[\mathrm{Ei}\left(-\frac{1}{d}\right)\right]}^2\right\},\\ f_2\left(d\right):=3e^{4/d}{\left[\mathrm{Ei}\left(-\frac{3}{d}\right)\right]}^2-2e^{2/d}\mathrm{Ei}\left(-\frac{3}{d}\right)\mathrm{Ei}\left(-\frac{1}{d}\right)+3{\left[\mathrm{Ei}\left(-\frac{1}{d}\right)\right]}^2.\end{array}$$

However, by Lemma 8 in the Appendix, there is no $d\in \left(0,\mathrm{\infty }\right)$ such that (9) holds.

Case 3. One Λ measure is a β-coalescent and the other is Kingman’s coalescent with exponential growth. If these two coalescents produce the same values of $\mathbb{E}{T}_2^{\mathrm{MRCA}},\mathbb{E}{T}_3^{\mathrm{MRCA}},$ and $\mathbb{E}{T}_4^{\mathrm{MRCA}},$ then we must have $t=3/2-e^{2/d}\mathrm{Ei}\left(-3/d\right)/2\mathrm{Ei}\left(-1/d\right)$ in (6), and equating (6) and (8) implies

$$\frac{g_1\left(d\right)+3e^{5/d}\mathrm{Ei}\left(-\left(6/d\right)\right)g_2\left(d\right)}{\mathrm{Ei}\left(-\left(1/d\right)\right)g_2\left(d\right)}=0,$$ (10)

where

$$\begin{array}{c}{g}_1\left(d\right):=2\mathrm{Ei}\left(-\frac{1}{d}\right)\left[-4e^{2/d}\mathrm{Ei}\left(-\frac{3}{d}\right)+\mathrm{Ei}\left(-\frac{1}{d}\right)\right],\\ g_2\left(d\right):=e^{2/d}\mathrm{Ei}\left(-\frac{3}{d}\right)+\mathrm{Ei}\left(-\frac{1}{d}\right).\end{array}$$

However, by Lemma 9 in the Appendix, there is no $d\in \left(0,\mathrm{\infty }\right)$ such that (10) holds.

Since each of the three cases results in a contradiction, we see that no such Λ measures exist, proving the identifiability claim.

□

Discussion

We have presented an efficient algorithm for computing the SFS for a very general class of coalescents. While Λ- and Ξ-coalescents seem to be primarily used in practice to model the genealogies of marine species (Árnason 2004; Hedgecock and Pudovkin 2011), these coalescents also model a wide range of other phenomena, including continuous strong positive selection (Neher and Hallatschek 2013), recurrent selective sweeps (Durrett and Schweinsberg 2004, 2005), strong bottlenecks (Birkner et al. 2009), and many others. Perhaps one of the reasons these coalescents are less widely used than Kingman’s coalescent is because efficient inference tools have not yet been developed to the same extent.

Multiple-merger coalescents have also attracted some interest recently in the context of extremely large sample sizes (Bhaskar et al. 2014). In such cases the sample size is too large for the assumption of only pairwise mergers of lineages imposed by Kingman’s coalescent to be biologically plausible, and indeed using Kingman’s coalescent to model such populations causes biases in inference (Bhaskar et al. 2014). It should be possible to extend the results presented in this article to discrete-time coalescents, such as the “exact coalescent” (Fu 2006) corresponding to the coalescent arising from the discrete-time Wright–Fisher process, or to any of the discrete-time random-mating models considered by Eldon and Wakeley (2006).

We also presented some encouraging identifiability results. While it is impossible in the general case to infer the inverse intensity function ζ or the measure of a Λ-coalescent from the SFS, for many biologically important cases identifiability does indeed hold. The method we presented for proving that the Λ measure is identifiable for constant ζ is powerful, but straightforward and should make it easy to prove whether the measure is identifiable for other sets of Λ- or Ξ-coalescents. While we considered the identifiability of Λ only for fixed, constant ζ and the identifiability of ζ for fixed Λ or Ξ, it would be interesting to see whether identifiability results can still be obtained for some model spaces while allowing both Λ and ζ to vary. It would also be interesting to extend our identifiability results for the Λ measure to some of the biologically relevant Ξ-coalescents.

Our identifiability results generally assumed access to the expected SFS. In practice, one observes a finite number of sites and so one has only a noisy estimate of the SFS. Our simulation study shows that, given a moderate number of independent trees, the empirical SFS is accurate enough to distinguish $\mathrm{\Lambda }\left(dx\right)/\eta \left(t\right)$ for some simple models. However, the effects of noisy data are still largely unknown, especially in cases where convergence to the expected SFS is not guaranteed. The accuracy of inferring ζ with the empirical SFS has been studied for Kingman’s coalescent (Terhorst and Song 2015), and it would be interesting to extend these results to general Λ-coalescents and to the inference of the Λ-measure itself; the results presented here should make such an analysis more tractable.

Appendix

Here we present two lemmas that are used in Theorem 5. Proofs are tedious but straightforward.

Lemma 8. For $d\in \left(0,\mathrm{\infty }\right),$

$$\frac{f_1\left(d\right)+e^{5/d}\mathrm{Ei}\left(-\left(6/d\right)\right)f_2\left(d\right)}{\mathrm{Ei}\left(-\left(1/d\right)\right)f_2\left(d\right)}\ne 0,$$

where

$$\begin{array}{c}{f}_1\left(d\right):=2\mathrm{Ei}\left(-\frac{1}{d}\right)\left\{e^{4/d}{\left[\mathrm{Ei}\left(-\frac{3}{d}\right)\right]}^2-4e^{2/d}\mathrm{Ei}\left(-\frac{3}{d}\right)\mathrm{Ei}\left(-\frac{1}{d}\right)+{\left[\mathrm{Ei}\left(-\frac{1}{d}\right)\right]}^2\right\},\\ f_2\left(d\right):=3e^{4/d}{\left[\mathrm{Ei}\left(-\frac{3}{d}\right)\right]}^2-2e^{2/d}\mathrm{Ei}\left(-\frac{3}{d}\right)\mathrm{Ei}\left(-\frac{1}{d}\right)+3{\left[\mathrm{Ei}\left(-\frac{1}{d}\right)\right]}^2.\end{array}$$

Lemma 9. For $d\in \left(0,\mathrm{\infty }\right),$

$$\frac{g_1\left(d\right)+3e^{5/d}\mathrm{Ei}\left(-\left(6/d\right)\right)g_2\left(d\right)}{\mathrm{Ei}\left(-\left(1/d\right)\right)g_2\left(d\right)}\ne 0,$$

where

$$\begin{array}{c}{g}_1\left(d\right):=2\mathrm{Ei}\left(-\frac{1}{d}\right)\left[-4e^{2/d}\mathrm{Ei}\left(-\frac{3}{d}\right)+\mathrm{Ei}\left(-\frac{1}{d}\right)\right],\\ g_2\left(d\right):=e^{2/d}\mathrm{Ei}\left(-\frac{3}{d}\right)+\mathrm{Ei}\left(-\frac{1}{d}\right).\end{array}$$

In what follows, let ${\mathrm{E}}_1\left(x\right):={\displaystyle {\int }_x^{\mathrm{\infty }}\left(e^{-t}/t\right)dt}=-\mathrm{Ei}\left(-x\right).$ It is clear that ${\mathrm{E}}_1\left(x\right)>0$ for all $x>0.$ Additionally,

$$e^{n/d}{\mathrm{E}}_1\left(\frac{n+1}{d}\right)={\displaystyle {\int }_{1/d}^{\mathrm{\infty }}\frac{e^{-t}}{t+n/d}dt},$$ (A1)

which follows from the definition of ${\mathrm{E}}_1$ and a change of variables.

Proof of Lemma 8. First, by noting that $f_2\left(d\right)=3{\left[e^{2/d}{\mathrm{E}}_1\left(3/d\right)-{\mathrm{E}}_1\left(1/d\right)\right]}^2+4e^{2/d}{\mathrm{E}}_1\left(3/d\right){\mathrm{E}}_1\left(1/d\right),$ it is easy to see that the denominator is strictly negative for $d\in \left(0,\mathrm{\infty }\right).$ We now show that the numerator is strictly positive for $d\in \left(0,\mathrm{\infty }\right).$ First, by rearranging terms we see that

$$\begin{array}{c}{f}_1\left(d\right)+e^{5/d}\mathrm{Ei}\left(-\frac{6}{d}\right)f_2\left(d\right)=4\left[{\mathrm{E}}_1\left(\frac{1}{d}\right)-e^{5/d}{\mathrm{E}}_1\left(\frac{6}{d}\right)\right]e^{2/d}{\mathrm{E}}_1\left(\frac{3}{d}\right){\mathrm{E}}_1\left(\frac{1}{d}\right)\\ \hspace{0.17em}-{\left[{\mathrm{E}}_1\left(\frac{1}{d}\right)-e^{2/d}{\mathrm{E}}_1\left(\frac{3}{d}\right)\right]}^2\left[2{\mathrm{E}}_1\left(\frac{1}{d}\right)+3e^{5/d}{\mathrm{E}}_1\left(\frac{6}{d}\right)\right].\end{array}$$ (A2)

Then, note

$${\mathrm{E}}_1\left(\frac{1}{d}\right)-e^{2/d}{\mathrm{E}}_1\left(\frac{3}{d}\right)={\displaystyle \frac{2}{d}{\int }_{1/d}^{\mathrm{\infty }}\frac{e^{-t}}{t\left(t+2/d\right)}dt}<\frac{4}{d}{\displaystyle {\int }_{1/d}^{\mathrm{\infty }}\frac{e^{-t}}{t\left(t+5/d\right)}}=\frac{4}{5}\left[{\mathrm{E}}_1\left(\frac{1}{d}\right)-e^{5/d}{\mathrm{E}}_1\left(\frac{6}{d}\right)\right].$$

Applying this inequality to the negative term on the right-hand side of (12), we see

$$\begin{array}{l}{f}_1\left(d\right)+e^{5/d}\mathrm{Ei}\left(-\frac{6}{d}\right)f_2\left(d\right)\\ \hspace{0.17em}>4\left[{\mathrm{E}}_1\left(\frac{1}{d}\right)-e^{5/d}{\mathrm{E}}_1\left(\frac{6}{d}\right)\right]\\ \hspace{0.17em}\hspace{0.17em}\times \left\{e^{2/d}{\mathrm{E}}_1\left(\frac{3}{d}\right){\mathrm{E}}_1\left(\frac{1}{d}\right)-\left[\frac{4}{5d}{\mathrm{E}}_1\left(\frac{1}{d}\right)+\frac{6}{5d}{e}^{5/d}{\mathrm{E}}_1\left(\frac{6}{d}\right)\right]\left({\displaystyle {\int }_{1/d}^{\mathrm{\infty }}\frac{e^{-t}}{t\left(t+2/d\right)}dt}\right)\right\}\\ \hspace{0.17em}>4\left[{\mathrm{E}}_1\left(\frac{1}{d}\right)-e^{5/d}{\mathrm{E}}_1\left(\frac{6}{d}\right)\right]{\mathrm{E}}_1\left(\frac{1}{d}\right)\left[e^{2/d}{\mathrm{E}}_1\left(\frac{3}{d}\right)-\frac{4}{15}{\mathrm{E}}_1\left(\frac{1}{d}\right)-\frac{2}{5}{e}^{5/d}{\mathrm{E}}_1\left(\frac{6}{d}\right)\right]\\ \hspace{0.17em}=4\left[{\mathrm{E}}_1\left(\frac{1}{d}\right)-e^{5/d}{\mathrm{E}}_1\left(\frac{6}{d}\right)\right]{\mathrm{E}}_1\left(\frac{1}{d}\right)\left[{\displaystyle {\int }_{1/d}^{\mathrm{\infty }}\frac{\left(\left(1/3\right)t^2+\left(7/3d\right)t-8/3d^2\right)e^{-t}}{t\left(t+2/d\right)\left(t+5/d\right)}}\right],\end{array}$$

which is >0 for any $d\in \left(0,\mathrm{\infty }\right)$ since ${\mathrm{E}}_1\left(1/d\right)>e^{5/d}{\mathrm{E}}_1\left(6/d\right)$ and $\left(1/3\right)t^2+\left(7/3d\right)t-8/3d^2>0$ for $t>1/d.$ □

Proof of Lemma 9. The denominator of (10) is equal to ${\mathrm{E}}_1\left(1/d\right)\left[e^{2/d}{\mathrm{E}}_1\left(3/d\right)+{\text{E}}_1\left(1/d\right)\right],$ which is strictly positive for $d\in \left(0,\mathrm{\infty }\right),$ by definition of ${\mathrm{E}}_1\left(x\right).$ Furthermore, the numerator is strictly negative for $d\in \left(0,\mathrm{\infty }\right)$ by noting the following:

$$\begin{array}{l}{g}_1\left(d\right)+3e^{5/d}\mathrm{Ei}\left(-\frac{6}{d}\right)g_2\left(d\right)\\ \hspace{0.17em}=\left({\displaystyle {\int }_{1/d}^{\mathrm{\infty }}\frac{e^{-t}}{t}dt}\right)\left[{\displaystyle {\int }_{1/d}^{\mathrm{\infty }}\frac{2e^{-t}}{t}+\frac{3e^{-t}}{t+5/d}-\frac{8e^{-t}}{t+2/d}dt}\right]+3\left({\displaystyle {\int }_{1/d}^{\mathrm{\infty }}\frac{e^{-t}}{t+5/d}dt}\right)\left({\displaystyle {\int }_{1/d}^{\mathrm{\infty }}\frac{e^{-t}}{t+2/d}dt}\right)\\ \hspace{0.17em}=\left({\displaystyle {\int }_{1/d}^{\mathrm{\infty }}\frac{e^{-t}}{t}dt}\right)\left[{\displaystyle {\int }_{1/d}^{\mathrm{\infty }}\frac{\left(-3t^2-\left(20/d\right)t+20/d^2\right)e^{-t}}{t\left(t+2/d\right)\left(t+5/d\right)}dt}\right]\\ \hspace{0.17em}\hspace{0.17em}+3\left({\displaystyle {\int }_{1/d}^{\mathrm{\infty }}\frac{e^{-t}}{t+5/d}dt}\right)\left[\left({\displaystyle {\int }_{1/d}^{\mathrm{\infty }}\frac{e^{-t}}{t}dt}\right)-\left({\displaystyle {\int }_{1/d}^{\mathrm{\infty }}\frac{\left(2/d\right)e^{-t}}{t\left(t+2/d\right)}dt}\right)\right]\\ \hspace{0.17em}=\left({\displaystyle {\int }_{1/d}^{\mathrm{\infty }}\frac{e^{-t}}{t}dt}\right)\left[{\displaystyle {\int }_{1/d}^{\mathrm{\infty }}\frac{\left(-\left(14/d\right)t+20/d^2\right)e^{-t}}{t\left(t+2/d\right)\left(t+5/d\right)}dt}\right]-\frac{6}{d}\left({\displaystyle {\int }_{1/d}^{\mathrm{\infty }}\frac{e^{-t}}{t+5/d}dt}\right)\left({\displaystyle {\int }_{1/d}^{\mathrm{\infty }}\frac{e^{-t}}{t\left(t+2/d\right)}dt}\right)\\ \hspace{0.17em}<\left({\displaystyle {\int }_{1/d}^{\mathrm{\infty }}\frac{e^{-t}}{t}dt}\right)\left[{\displaystyle {\int }_{1/d}^{\mathrm{\infty }}\frac{\left(-\left(14/d\right)t+20/d^2\right)e^{-t}}{t\left(t+2/d\right)\left(t+5/d\right)}d}t\right]-\frac{1}{d}\left({\displaystyle {\int }_{1/d}^{\mathrm{\infty }}\frac{e^{-t}}{t}dt}\right)\left({\displaystyle {\int }_{1/d}^{\mathrm{\infty }}\frac{e^{-t}}{t\left(t+2/d\right)}dt}\right)\\ \hspace{0.17em}=\left({\displaystyle {\int }_{1/d}^{\mathrm{\infty }}\frac{e^{-t}}{t}dt}\right)\left[{\displaystyle {\int }_{1/d}^{\mathrm{\infty }}\frac{\left(-\left(15/d\right)t+15/d^2\right)e^{-t}}{t\left(t+2/d\right)\left(t+5/d\right)}dt}\right]\\ \hspace{0.17em}<0.\end{array}$$

Therefore, (10) holds. □