Bayesian Nonparametric Inference of Population Size Changes from Sequential Genealogies

Abstract

Sophisticated inferential tools coupled with the coalescent model have recently emerged for estimating past population sizes from genomic data. Recent methods that model recombination require small sample sizes, make constraining assumptions about population size changes, and do not report measures of uncertainty for estimates. Here, we develop a Gaussian process-based Bayesian nonparametric method coupled with a sequentially Markov coalescent model that allows accurate inference of population sizes over time from a set of genealogies. In contrast to current methods, our approach considers a broad class of recombination events, including those that do not change local genealogies. We show that our method outperforms recent likelihood-based methods that rely on discretization of the parameter space. We illustrate the application of our method to multiple demographic histories, including population bottlenecks and exponential growth. In simulation, our Bayesian approach produces point estimates four times more accurate than maximum-likelihood estimation (based on the sum of absolute differences between the truth and the estimated values). Further, our method’s credible intervals for population size as a function of time cover 90% of true values across multiple demographic scenarios, enabling formal hypothesis testing about population size differences over time. Using genealogies estimated with ARGweaver, we apply our method to European and Yoruban samples from the 1000 Genomes Project and confirm key known aspects of population size history over the past 150,000 years.

FOR a single nonrecombining locus, neutral coalescent theory predicts the set of timed ancestral relationships among sampled individuals, known as a gene genealogy (Kingman 1982; Hudson 1983, 1990; Tajima 1983). In the coalescent model with variable population size, the rate at which two lineages have a common ancestor (or coalesce) is a function of the population size in the past. Here we denote the population size trajectory by $N\left(t\right),$ where t is time in the past, and use the term local genealogy to describe ancestral relationships at one nonrecombining locus. When analyzing multilocus sequences, a single local genealogy will not represent the full history of the sample. Instead, the set of ancestral relationships and recombination events among a sample of multilocus sequences can be represented by a graph, known as the ancestral recombination graph (ARG), which depicts the complex structure of neighboring local genealogies and results in a computationally expensive model for inferring $N\left(t\right)$ (Griffiths and Marjoram 1997; Wiuf and Hein 1999).

Recent studies have leveraged approximations for the coalescent with recombination—the sequentially Markov coalescent (SMC) (McVean and Cardin 2005) and its variant SMC′ (Marjoram and Wall 2006; Chen et al. 2009)—both of which model local genealogies as a continuous-time Markov process along sequences (Figure 1). The difference between the SMC and the SMC′ is that the SMC models only the class of recombination events that alter local genealogies of the sample; in general, the SMC′ is a better approximation to the ARG than the SMC (Chen et al. 2009; Wilton et al. 2015). Because of these features, in this work we rely on the SMC′ to model local genealogies with recombination.

Figure 1.

SMC′ hidden Markov model for inferring population size trajectories, drawn according to Rasmussen et al. (2014) to highlight notation specific to our study. (A) Observed sequence data in a segment of length L from five individuals. Three loci are shown delimited by recombination breakpoints $b_1$ and $b_2$. Only the derived mutations at polymorphic sites are shown. (B) Corresponding local genealogies $g_i$ for each locus i. The five sampled individuals are depicted as solid black circles. Local genealogies have a Markovian degree-1 dependency. Each intercoalescent time (the time interval between coalescent events denoted as open circles) provides information about past population size (number of solid gray circles at a given time point). Moving from left to right after recombination breakpoint $b_1,$ the pruning location $p_1$ is selected from genealogy $g_0$ and the pruned branch is regrafted back on the genealogy (solid blue circle). The coalescent event of $g_0$ depicted as a solid red circle in $g_1$ is deleted. This creates the next genealogy $g_1$. The process continues until L. At L, the population size trajectory $N\left(t\right)$ (depicted as a black curve superimposed on $g_2$) can be inferred.

Under the coalescent and SMC′ models, population size trajectories and sequence data are separated by two stochastic processes: (i) a state process that describes the relationship between the population size trajectory and the set of local genealogies and (ii) an observation process that describes how the hidden local genealogies are observed through patterns of nucleotide diversity in the sequence data. The observation process includes mutation and genotyping error while the state process models coalescence. Population size trajectories are then inferred from sequence data, using these coalescent-based hidden Markov models. In this study, we restrict attention to the state process and present a novel Bayesian approach for inferring population size trajectories from local genealogies. We solve a number of key modeling and inference problems and thus provide a basis for developing efficient algorithms to infer population parameters from sequence data directly.

Whole-genome inference of population size trajectories has been hampered by the enormous state space of local genealogies for large sample sizes. The pioneering pairwise sequentially Markov coalescent (PSMC) method of Li and Durbin (2011) employed the SMC to infer $N\left(t\right)$ from a sample of size 2 ($n=2$). In this method, time is discretized and the population size trajectory is piecewise constant. Subsequent methods for samples larger than 2 similarly rely on the discretization of time. The natural extension of the PSMC to $n>2$ is the multiple sequentially Markovian coalescent (MSMC) (Schiffels and Durbin 2014). However, the MSMC models only the most recent coalescent event of the sample; thus MSMC’s estimation of population sizes is limited to very recent times. Other recent methods propose efficient ways of exploring the state space of hidden genealogies for $n>2$ (Sheehan et al. 2013; Rasmussen et al. 2014), yet also rely on discretizing the state space of local genealogies and assume a piecewise constant trajectory of population sizes.

Gaussian process-based Bayesian inference of population size trajectories has proved to be a powerful and flexible nonparametric approach when applied to a single local genealogy (Palacios and Minin 2013; Lan et al. 2015). The two main advantages of the Gaussian process (GP)-based approach are (i) it does not require a specific functional form of the population size trajectory (such as constant or exponential growth) and (ii) it does not require an arbitrary specification of change points in a piecewise constant or linear framework.

In this article, we overcome the limitations of existing methods—discretizing time, assuming a piecewise constant trajectory, and reporting only point estimates for past population sizes—by introducing a Bayesian nonparametric approach with a GP to model the population size trajectory as a continuous function of time. More specifically, we model the logarithm of the population size trajectory a priori as a Gaussian process (the log ensures our estimates are positive). As mentioned above, we assume that local gene genealogies are known. For our Bayesian approach, we develop a Markov chain Monte Carlo (MCMC) method to sample from the posterior distribution of population sizes over time. Our MCMC algorithm uses the recently developed algorithm, split Hamiltonian Monte Carlo (splitHMC) (Shahbaba et al. 2014; Lan et al. 2015). To compare our Bayesian GP-based estimation of population size trajectories with a piecewise constant maximum-likelihood-based estimation (e.g., Li and Durbin 2011; Sheehan et al. 2013; Schiffels and Durbin 2014), we implement the expectation-maximization (EM) algorithm within our framework and compute the observed Fisher information to obtain confidence intervals of the maximum-likelihood estimates.

Finally, we address a key problem for inference of population size trajectories under sequentially Markov coalescent models: the efficient computation of transition densities needed in the calculation of likelihoods. Here, we express the transition densities of local genealogies in terms of local ranked tree shapes (Tajima 1983) and coalescent times and show that these quantities are statistically sufficient for inferring population size trajectories either from sequence data directly or from the set of local genealogies. The use of ranked tree shapes allows us to exploit the state process of local genealogies efficiently since the space of ranked tree shapes has a smaller cardinality than the space of labeled topologies (Sainudiin et al. 2014).

Methods: SMC′ Calculations

Following notation similar to that in Rasmussen et al. (2014) (Table 1), a realization of the embedded SMC′ chain consists of a set of m local genealogies $\left(g_0,g_1,\dots ,g_{m-1}\right),$ $m-1$ recombination breakpoints at chromosomal locations $\left(b_1,b_2,\dots ,b_{m-1}\right),$ and $m-1$ pruning locations $\left(p_1,p_2,\dots ,p_{m-1}\right),$ where $p_i=\left(u_i,w_i\right)$ indicates the time of the recombination event $u_i$ and the branch $w_i$ where recombination happened in genealogy $g_{i-1}$ (Figure 1). Genealogy $g_0$ corresponds to the genealogy of n sequences that contains the set of timed ancestral relationships among the n individuals for the chromosomal segment $(0,b_1]$. Genealogy $g_i$ corresponds to the genealogy of the same n sequences for the chromosomal segment $(b_i,b_{i+1}]$ for $i=1,2,\dots ,m-2$. Finally, $t_j^i$ denotes the time when two of j lineages coalesce in genealogy $g_i,$ measured in units of generations before present.

Table 1. Notation for the SMC′ model used in this work.

Symbol	Description
Parameters
ρ	Recombination rate per site per generation
$N\left(t\right)$	Effective population size trajectory with time measured in units of $N_0$ generations
τ	Hyperparameter that controls the smoothness of the log-Gaussian process prior on $N\left(t\right)$
Notation specific to SMC′ chain
L	Length of observed sequences
$b_i$	Chromosomal location of the ith recombination breakpoint
m	No. local genealogies corresponding to $m-1$ recombination events
$s_{i+1}=b_{i+1}-b_i$	Segment length for local genealogy i
$g_i$	Local genealogy for the segment $(b_{i-1},b_i]$
Notation specific to local genealogy
n	Sample size or no. sequences
$l_i$	Total tree length of local genealogy $g_i$
$A^i\left(t\right)$	Piecewise constant function of the number of ancestral lineages at time t in local genealogy $g_i$
$t_j^i$	Coalescent time in genealogy $g_i$ when two of j lineages coalesce. $A^i\left(t_j^i-\right)=j;$ $A^i\left(t_j^i+\right)=j-1.$
$t^i=\left(t_n^i,t_{n-1}^i,\dots ,t_2^i\right)$	Vector of coalescent times of genealogy $g_i$
$p_i=\left(u_i,w_i\right)$	Pruning location along local genealogy $g_i$
$u_i$	Time when the recombination event happened along the height of the genealogy $g_i$
$w_i$	Lineage on genealogy $g_{i-1}$ where the recombination event happened
${w^{\prime }}_i$	New lineage added on genealogy $g_i$ where the recombination event happened
$t_{\text{new}}^i$	Coalescent time in genealogy $g_i$ when the lineage $w_i$ coalesces
$t_{\mathrm{del}}^i$	Coalescent time in genealogy $g_{i-1}$ that no longer exists in genealogy $g_i$
$c_i$	Lineage on genealogy $g_i$ that coalesces with lineage ${w^{\prime }}_i$
$F_{j,k}^i$	No. free lineages in local genealogy $g_i$ that do not coalesce in the time interval $(t_{j+1}^i,$$t_k^i)$
$I^i\left(t\right)$	Piecewise constant function that takes values in $\left\{0,1,2\right\}$ indicating no. ancestral lineages at time t in genealogy $g_i$ where the pruning event would produce a visible transition to $g_{i+1}$
Discretization
d	No. change points at which $N\left(t\right)$ is estimated
$x=\left(x_1,\dots ,x_d\right)$	Times at which $N\left(t\right)$ is estimated

Using uppercase letters to denote random variables, the evolution of the SMC′ process along chromosomal segments is governed by a point process $B={\left\{B_i\right\}}_{i\in \mathrm{\mathbb{N}}}$ that represents the random locations of recombination breakpoints. We use $S_i=B_i-B_{i-1},$ for $i=1,2,\dots ,m,$ to denote the segment lengths for each local genealogy, with $S_0=B_0=0.$ Let $G={\left\{G_i\right\}}_{i\in \mathrm{\mathbb{N}}}$ be the chain that records the local genealogies, and let $P=\left(U,W\right)={\left\{\left(U_i,W_i\right)\right\}}_{i\in \mathrm{\mathbb{N}}}$ represent the chain that records the pruning locations (time and branch) on G. The sequence $\left(G_i,P_i=\left\{U_i,W_i\right\},B_i\right)$ has the following conditional independence relation:

$$\text{Pr}\left[G_i=g_i,U_i\le u_i,W_i=w_i,S_i\le s|{\left\{g_j,b_j\right\}}_{j=0}^{i-1},{\left\{u_j,w_j\right\}}_{j=1}^{i-1}\right]$$

$$=\mathrm{Pr}\left[S_i\le s_i|g_{i-1}\right]$$ (1)

$$\times \mathrm{Pr}\left[U_i\le u_i,W_i=w_i|g_{i-1}\right]$$ (2)

$$\times \mathrm{Pr}\left[G_i=g_i|U_i\le u_i,W_i=w_i,g_{i-1}\right].$$ (3)

Thus, given a chain of local genealogies, pruning locations, and recombination breakpoints, the joint transition probability to a new genealogy, pruning location, and locus length can be expressed as the product of the locus-length probability conditioned on the current genealogy (Expression 1, above), the pruning location probability conditioned on the current genealogy (Expression 2, above), and the transition probability of the new genealogy conditioned on the current genealogy and pruning location (Expression 3, above).

Complete data transition densities

Consider the chain of local genealogies $g=\left(g_0,g_1,\dots ,g_{m-1}\right)$ with recombination breakpoints at $b=\left(0,b_1,\dots ,b_{m-1}\right)$. According to the SMC′ process, the first local genealogy $g_0$ follows the standard coalescent density

$$\mathrm{Pr}\left[G_0=g_0|N\left(t\right)\right]={\displaystyle \prod _{j=2}^n\frac{1}{N\left(t_j^0\right)}\exp \left\{-{\displaystyle {\int }_{t_{j+1}^0}^{t_j^0}\frac{A^0\left(t\right)\left(A^0\left(t\right)-1\right)dt}{2N\left(t\right)}}\right\}},$$ (4)

where $t_{n+1}^0=0$ and $t_n^0<\dots <t_2^0$ are the set of coalescent times in local genealogy $g_0$. The piecewise constant function $A^i\left(t\right)$ denotes the number of ancestral lineages present at time t in genealogy $g_i;$ that is, $A^i\left(t\right)={\displaystyle {\sum }_{j=1}^{n}j{1}_{t\in \left(t_{j+1}^i,t_j^i\right)}},$ with $t_1^i=\mathrm{\infty }$.

Given a current local genealogy $g_{i-1},$ the distribution of the length $S_i=B_i-b_{i-1}$ of the current locus depends on the current state of the SMC′ chain through the local genealogy’s total tree length $l_{i-1}$ (the sum of all branch lengths in $g_{i-1}$) and the recombination rate per site per generation ρ:

$$f\left(s_i|g_{i-1},\rho \right)=\rho l_{i-1}\exp \left\{-\rho l_{i-1}{s}_i\right\}.$$ (5)

At recombination breakpoint $b_i,$ a new local genealogy $g_i$ is generated that depends on the previous local genealogy $g_{i-1}$ and the population size trajectory $N\left(t\right)$ (Figure 1). To generate $g_i$ we first randomly choose a pruning location $p_i$ (consisting of a pruning time $u_i$ and a lineage $w_i$) uniformly along $g_{i-1}$. At pruning location $p_i,$ we add a new lineage ${w^{\prime }}_i$ and coalesce it further in the past at time $t_{\text{new}}^i$ with some lineage, $c_i$ (Figure 2). We then delete the $w_i$ lineage’s segment from $u_i$ to $t_{\mathrm{del}}^i$ (the coalescent time of lineage $w_i$). The transition density to a new genealogy at recombination breakpoint $b_i$ is then

$$\mathrm{Pr}\left[p_i=\left(u_i,w_i\right),t_{\text{new}}^i,c_i|g_{i-1},N\left(t\right)\right]$$

$$=\mathrm{Pr}\left[p_i=\left(u_i,w_i\right)|g_{i-1}\right]\mathrm{Pr}\left[t_{\text{new}}^i,c_i|u_i,g_{i-1},N\left(t\right)\right]$$

$$=\left(\frac{1}{l_{i-1}}\right)\frac{1}{N\left(t_{\text{new}}^i\right)}\exp \left\{-{\displaystyle {\int }_{u_i}^{t_{\text{new}}^i}\frac{A^{i-1}\left(t\right)dt}{N\left(t\right)}}\right\},$$ (6)

where $l_{i-1}$ denotes the total tree length of $g_{i-1}$.

Figure 2.

Schematic representation of SMC′ transitions given a recombination breakpoint at location $b_i$ (indicated as an arrow in each panel). (A) Visible transition. We uniformly sample the pruning location $p_i$ from $g_{i-1}$ at time $u_i$ along some branch $w_i,$ and we add a new branch ${w^{\prime }}_i$ at $u_i$ and regraft it (dashed black line). The new branch ${w^{\prime }}_i$ coalesces with some branch $c_i$ at time $t_{\text{new}}^i$. We then delete branch $w_i$ and the coalescent time $t_{\mathrm{del}}^i$ to generate genealogy $g_i$. Any pruning time along the branch $w_i$ (shown in green) would have produced the same visible transition from $g_{i-1}$ to $g_i$. (B) Invisible transition. We uniformly sample the pruning location $p_i=\left(u_i,w_i\right),$ add a new branch ${w^{\prime }}_i$ at $u_i,$ and regraft it. The new branch ${w^{\prime }}_i$ coalesces with itself (dashed black line), that is, $C_i=w_i,$ and then the segment $\left(u_i,t_{\mathrm{del}}^i\right)$ of $w_i$ is deleted. If $C_i=w_i,$ any pruning location along the green branches would have produced the same invisible transition.

This generative process for local genealogies can result in a visible transition, where a genealogy $g_i$ is different from $g_{i-1}$ (Figure 2A), or an invisible transition, where $g_i$ is identical to $g_{i-1}$ (Figure 2B).

An invisible transition ($g_i=g_{i-1}$) occurs when $c_i=w_i$. Given the pruning location $p_i=\left(u_i,w_i\right),$ an invisible transition occurs when $T_{\text{new}}^i\in \left(u_i,t_{\mathrm{del}}^i\right)$ and $C_i,$ the random variable indicating the lineage that coalesces with lineage ${w^{\prime }}_i,$ takes the value $w_i$. The probability of an invisible transition is given by

$$\mathrm{Pr}\left[G_i=g_{i-1}|p_i=\left(u_i,w_i\right),g_{i-1},N\left(t\right)\right]$$

$$=\mathrm{Pr}\left[u_i\le T_{\text{new}}^i\le t_{\mathrm{del}}^i,C_i=w_i|p_i=\left(u_i,w_i\right),g_{i-1},N\left(t\right)\right]$$

$$={\displaystyle {\int }_{u_i}^{t_{\mathrm{del}}^i}\frac{1}{N\left(t\right)}\exp \left\{-{\displaystyle {\int }_{u_i}^t\frac{A^{i-1}\left(u\right)du}{N\left(u\right)}}\right\}dt}.$$

Thus, the joint transition probability to an invisible event with pruning location $\left(u_i,w_i\right),$ given $g_{i-1},$ is

$$\mathrm{Pr}\left[G_i=g_{i-1},p_i=\left(u_i,w_i\right)|g_{i-1},N\left(t\right)\right]$$

$$=\frac{1}{l_{i-1}}\mathrm{Pr}\left[G_i=g_{i-1}|p_i=\left(u_i,w_i\right),g_{i-1},N\left(t\right)\right].$$

Transition densities averaged over unknown pruning locations

Even though we assume that local genealogies are known, to build inferential frameworks for sequence data in the future, we do not wish to make the same assumption about pruning locations. Thus, we average over pruning locations to obtain marginal transition densities between genealogies for both visible and invisible transitions.

Visible transitions:

To compute the marginal visible transition density to a new genealogy $g_i=\left\{g_{i-1}\mathrm{\setminus }\left\{t_{\mathrm{del}}^i\right\}\mathrm{\cup }\left\{t_{\text{new}}^i,\left({w^{\prime }}_i,c_i\right)\right\}\right\},$ we need to average over all possible pruning locations $p_i=\left(u_i,w_i\right)$ along $g_{i-1}$. By comparing the two genealogies $g_{i-1}$ and $g_i$ in Figure 2A, we know that $p_i$ corresponds to the lineage $w_i$ some time along $\left(0,t_4^{i-1}\right)$ or, equivalently, along $\left(0,t_{\mathrm{del}}^i\right)$. In general, comparison of $g_{i-1}$ and $g_i$ may not provide complete information to identify the lineage that was pruned. When the children of the node corresponding to $t_{\mathrm{del}}$ and the children of the node corresponding to $t_{\text{new}}$ are the same, pruning different branches can lead to the same transition. We enumerate all cases of incomplete information for visible transitions in Supporting Information, File S1, and File S2.

We introduce a function $I^{i-1}\left(t\right),$ equal to the number of possible lineages at time t where the pruning location along $g_{i-1}$ would produce a visible transition to $g_i$. $I^{i-1}\left(t\right)$ is a piecewise constant function that takes the values in $\left\{0,1,2\right\}$ depending on whether the pruning location $p_i$ can happen in $0,1,$ or 2 branches at time t. In the example in Figure 2A,

$$I^{i-1}\left(t\right)=\{\begin{array}{ll}1,\hfill & \text{if}\hspace{0.17em}t\in \left(0,t_4^{i-1}\right),\hfill \\ 0,\hfill & \text{if}\hspace{0.17em}t\in \left(t_4^{i-1},\mathrm{\infty }\right).\hfill \end{array}$$ (7)

For a general piecewise constant function $I^{i-1}\left(t\right),$ the marginal visible transition density to a new genealogy is

$$\begin{array}{l}\mathrm{Pr}\left[G_i=g_i|g_{i-1},N\left(t\right)\right]\\ =\frac{1}{l_{i-1}}{\displaystyle {\int }_0^{\mathrm{\infty }}{I}^{i-1}\left(u\right)\mathrm{Pr}\left[t_{\text{new}}^i,c_i|u,w_i\right]du}\\ =\frac{1}{l_{i-1}}{\displaystyle {\int }_0^{\mathrm{\infty }}{I}^{i-1}\left(u\right)\frac{1}{N\left(t_{\text{new}}^i\right)}}\exp \left\{-{\displaystyle {\int }_u^{t_{\text{new}}^i}\frac{A^{i-1}\left(t\right)dt}{N\left(t\right)}}\right\}du.\end{array}$$ (8)

Invisible transitions:

To compute the marginal transition probabilities for invisible events, we must average over all possible pruning locations $p_i$. Consider the example in Figure 2B and choosing a pruning time ($u_i$) along $g_{i-1}$. To have an invisible transition, the coalescing branch $C_i$ must be the same pruning branch $W_i$. In Figure 2B the new coalescent time $T_{\text{new}}^i$ can happen along five lineages in the interval $\left(0,t_5^{i-1}\right),$ three lineages in the interval $\left(t_5^{i-1},t_4^{i-1}\right),$ and two lineages in the interval $\left(t_4^{i-1},t_3^{i-1}\right)$. To generalize this calculation, we introduce the quantity $F_{j,k}^i$ with $n\ge j\ge k\ge 2,$ which denotes the number of lineages in $g_i$ that are free (do not coalesce), in the time segment $\left(t_{j+1}^i,t_k^i\right),$ with $t_{n+1}^i=0.$ The time interval $\left(t_{j+1}^i,t_k^i\right)$ includes the interval of pruning $\left(t_{j+1}^i,t_j^i\right)$ up to the interval of self-coalescence $\left(t_{k+1}^i,t_k^i\right)$. Thus, if the pruning time happens at time $U_i\in \left(t_{j+1}^i,t_j^i\right),$ an invisible transition with new coalescent time $T_{\text{new}}^i\in \left(t_{k+1}^i,t_k^i\right)$ can happen along $F_{j,k}^i$ free lineages.

In Figure 2B, $u_i$ happened in the time interval $\left(0,t_5^{i-1}\right)$. If the new coalescent time $T_{\text{new}}^i$ happens in the interval $\left(u_i,t_5^{i-1}\right)$ along the same (unknown) pruning branch, then this invisible transition has probability

$$\begin{array}{l}\mathrm{Pr}\left[G_i=g_{i-1},T_{\text{new}}^i\in \left(t_6^{i-1},t_5^{i-1}\right)|u_i,g_{i-1},N\left(t\right)\right]\\ =F_{5,5}^{i-1}{\displaystyle {\int }_{u_i}^{t_5^{i-1}}\frac{1}{N\left(t\right)}\exp \left\{-{\displaystyle {\int }_{u_i}^t\frac{A^{i-1}\left(u\right)du}{N\left(u\right)}}\right\}dt},\end{array}$$

with $F_{5,5}=5.$

Now consider the same example of Figure 2B but with an unknown pruning time $u_i$. The joint event where recombination occurs at pruning time $U_i\in \left(t_6^{i-1},t_5^{i-1}\right)$ and coalescent time $T_{\text{new}}^i$ occurs in the interval $\left(t_6^{i-1},t_5^{i-1}\right)$ and this results in an invisible transition has probability

$$\mathrm{Pr}\left[G_i=g_{i-1},U_i\in \left(t_6^{i-1},t_5^{i-1}\right),T_{\text{new}}^i\in \left(t_6^{i-1},t_5^{i-1}\right)|g_{i-1},N\left(t\right)\right]$$

$$=\frac{F_{5,5}^{i-1}{\displaystyle {\int }_{t_6^{i-1}}^{t_5^{i-1}}{\displaystyle {\int }_{u_i}^{t_5^{i-1}}\left(1/N\left(t\right)\right)\exp \left\{-{\displaystyle {\int }_{u_i}^t\left(A^{i-1}\left(u\right)du/N\left(u\right)\right)}\right\}dt}d{u}_i}}{l_{i-1}}$$ (9)

$$=\frac{F_{5,5}^{i-1}{P}_{5,5}^{i-1}}{l_{i-1}},$$ (10)

where $P_{5,5}^{i-1}$ denotes the double integral expression in Equation 9 for ease of notation.

An invisible transition would also result if $U_i\in \left(t_6^{i-1},t_5^{i-1}\right)$ and $T_{\text{new}}^i\in \left(t_5^{i-1},t_4^{i-1}\right)$ along the same (unknown) pruning branch; in Figure 2B, this can happen along three lineages, so $F_{5,4}^{i-1}=3$ and this event has probability

$$\begin{array}{l}\mathrm{Pr}\left[G_i=g_{i-1},U_i\in \left(t_6^{i-1},t_5^{i-1}\right),T_{\text{new}}^i\in \left(t_5^{i-1},t_4^{i-1}\right)|g_{i-1},N\left(t\right)\right]\\ =\frac{F_{5,4}^{i-1}{\displaystyle {\int }_{t_6^{i-1}}^{t_5^{i-1}}\exp \left\{-{\displaystyle {\int }_{u_i}^{t_5^{i-1}}\left(A^{i-1}\left(u\right)du/N\left(u\right)\right)}\right\}}}{l_{i-1}}\\ \times {\displaystyle {\int }_{t_5^{i-1}}^{t_4^{i-1}}\frac{1}{N\left(t\right)}\exp \left\{-{\displaystyle {\int }_{t_5^{i-1}}^t\frac{A^{i-1}\left(u\right)du}{N\left(u\right)}}\right\}dt}d{u}_i\\ =\frac{F_{5,4}^{i-1}{P}_{5,4}^{i-1}}{l_{i-1}}.\end{array}$$

If we continue considering the cases where $U_i\in \left(t_6^{i-1},t_5^{i-1}\right)$ and $T_{\text{new}}^i\in \left(t_4^{i-1},t_3^{i-1}\right)$ or $T_{\text{new}}^i\in \left(t_3^{i-1},t_2^{i-1}\right),$ we have $F_{5,3}^{i-1}=2$ and $F_{5,2}^{i-1}=0.$ Then, the joint probability of an invisible event and $U_i\in \left(t_6^{i-1},t_5^{i-1}\right)$ is

$$\mathrm{Pr}\left[G_i=g_{i-1},U_i\in \left(t_6^i,t_5^i\right)|g_{i-1},N\left(t\right)\right]=\frac{{\displaystyle {\sum }_{k=2}^5{F}_{j,k}^{i-1}{P}_{j,k}^{i-1}}}{l_{i-1}}.$$

For the cases when $U_i\in \left(t_{j+1}^{i-1},t_j^{i-1}\right)$ and the new coalescent time $T_{\text{new}}^i$ falls in another coalescent interval $\left(t_{k+1}^{i-1},t_k^{i-1}\right),$ we need to compute the following: the joint probability of $U_i\in \left(t_{j+1}^{i-1},t_j^{i-1}\right)$ and no coalescence in the interval $\left(u_i,t_j^{i-1}\right),$

$$\frac{1}{l_{i-1}}{Q}_j^{i-1}=\frac{1}{l_{i-1}}{\displaystyle {\int }_{t_{j+1}^{i-1}}^{t_j^{i-1}}\exp \left\{-{\displaystyle {\int }_{u_i}^{t_j^{i-1}}\frac{A^{i-1}\left(u\right)du}{N\left(u\right)}}\right\}d{u}_i};$$

the probability of no coalescence in any of the intermediate coalescent intervals $\left(t_{l+1}^{i-1},t_l^{i-1}\right),$

$$q_l^{i-1}=\exp \left\{-{\displaystyle {\int }_{t_{l+1}^{i-1}}^{t_l^{i-1}}\frac{A^{i-1}\left(u\right)du}{N\left(u\right)}}\right\};$$

and the probability of coalescing at $T_{\text{new}}^i\in \left(t_{k+1}^{i-1},t_k^{i-1}\right),$

$$1-q_k^{i-1}.$$

Then,

$$\frac{1}{l_{i-1}}{P}_{j,k}^{i-1}=\frac{1}{l_{i-1}}{Q}_j^{i-1}{q}_{j-1}^{i-1}{q}_{j-2}^{i-1}\dots q_{k+1}^{i-1}\left(1-q_k^{i-1}\right)$$

represents the probability that the pruning location is $w_i$ at time $U_i\in \left(t_{j+1}^{i-1},t_j^{i-1}\right)$ and the new lineage ${w^{\prime }}_i$ coalesces at time $T_{\text{new}}^i\in \left(t_{k+1}^{i-1},t_k^{i-1}\right)$ with lineage $c_i=w_i$. Overall, the marginal transition probability to an invisible event is

$$\begin{array}{l}\mathrm{Pr}\left[G_i=g_{i-1}|g_{i-1},N\left(t\right)\right]\\ ={\displaystyle {\int }_0^{t_2^{i-1}}\mathrm{Pr}\left[G_i=g_{i-1},u_i|g_{i-1},N\left(t\right)\right]d{u}_i}\\ ={\displaystyle \sum _{j=2}^n\mathrm{Pr}\left[G_i=g_{i-1},U_i\in \left(t_{j+1}^{i-1},t_j^{i-1}\right)|g_{i-1},N\left(t\right)\right]}\\ =\frac{1}{l_{i-1}}{\displaystyle \sum _{j=2}^n{\displaystyle \sum _{k=2}^j{F}_{j,k}^{i-1}{P}_{j,k}^{i-1}}}.\end{array}$$ (11)

The likelihood of the embedded SMC′ chain

Instead of having a complete realization of the embedded SMC′ chain of m local genealogies $g_0,\dots ,g_{m-1}$ and pruning locations $p_1,\dots ,p_{m-1}$ at recombination breakpoints $b_1,\dots ,b_{m-1},$ we assume that our data (unless otherwise noted) consist only of m local genealogies at recombination breakpoints from a chromosomal segment of length L (including visible and invisible events). Note that our observed data are not sequence data. More specifically, our observed data are

$$Y=\left\{\left(g_0,0\right),\left(g_1,b_1\right)\dots ,\left(g_{m-1},b_{m-1}\right),s_m=L-b_{m-1}\right\}.$$ (12)

Then, the observed data likelihood is

$$\begin{array}{l}{\mathrm{ℒ}}_{\mathrm{obs}}\left(Y;N\left(t\right),\rho \right)=\stackrel{\text{factors}\hspace{0.17em}\text{that}\hspace{0.17em}\text{depend}\hspace{0.17em}\text{on}\hspace{0.17em}N\left(t\right)}{\overbrace{\mathrm{Pr}\left[g_0|N\left(t\right)\right]\left[{\displaystyle \prod _{i=0}^{m-2}\mathrm{Pr}\left[g_{i+1}|g_i,N\left(t\right)\right]}\right]}}\\ \times \underset{\text{factors}\hspace{0.17em}\text{that}\hspace{0.17em}\text{depend}\hspace{0.17em}\text{on}\hspace{0.17em}\rho }{\underbrace{h\left(L-b_{m-1}|g_{m-1},\rho \right)\left[{\displaystyle \prod _{i=0}^{m-2}f\left[s_{i+1}|g_i,\rho \right]}\right]}},\end{array}$$ (13)

where $h\left(L-b_{m-1}|g_{m-1},\rho \right)$ is the survival function in state $g_{m-1}$. Equation 13 is factored into terms that depend on $N\left(t\right)$ alone and ones that depend on ρ alone. The terms that depend on ρ, given by Equation 5, depend on the data only through total tree lengths $l_0,\dots ,l_{m-1}$ and locus lengths $s_1,\dots ,s_{m-1},L-b_{m-1}$. By the factorization theorem for sufficient statistics, local tree lengths $l_0,\dots ,l_{m-1}$ and locus lengths $s_1,\dots ,s_{m-1},L-b_{m-1}$ are sufficient for inferring ρ. Moreover, recombination locations $b_0,b_1,\dots ,b_{m-1}$ do not provide information about $N\left(t\right)$.

Methods: Inference

Current coalescent-based methods that infer a population size trajectory $N\left(t\right)$ from whole-genome data assume $N\left(t\right)$ is a piecewise constant function with change points $x_1=0<x_2<\dots <x_d$ (Li and Durbin 2011; Sheehan et al. 2013; Rasmussen et al. 2014; Schiffels and Durbin 2014). That is,

$$N\left(t\right)={\displaystyle \sum _{i=1}^d{N}_i{1}_{t\in (x_{i-1},x_i]}}.$$ (14)

Equation 14 presents two challenges. The first challenge lies in the specification of the change points: the narrower an interval is, the higher the probability that we do not observe coalescent times in that interval; further, the fewer observed coalescent times in an interval, the greater the uncertainty is of the estimate ${\widehat{N}}_i$ (if the estimate even exists). The second challenge lies in the specification of the time window $\left(0,x_d\right):$ if $x_d$ is set too far in the past, we might not have enough data to accurately estimate $N\left(t\right)$ for $x_d\le t<\mathrm{\infty }$.

To solve the first challenge, Li and Durbin (2011) and Rasmussen et al. (2014) distribute the d change points evenly on a logarithmic scale,

$$x_j=\frac{1}{\kappa }\left\{\exp \left[\frac{j}{d}\log \left(1+\kappa x_d\right)\right]-1\right\},$$ (15)

where κ is specified by the user. Schiffels and Durbin (2014) propose discretizing time according to the quantiles of the exponential distribution, $x_j=\left(-1/\lambda \right)\log \left[1-j/d\right],$ where λ is the rate of an exponential distribution. Schiffels and Durbin (2014) model the time to the most recent coalescent event and set $\lambda =\left(\begin{array}{c}n\\ 2\end{array}\right)$ However, this equation is not directly applicable here because we use all coalescent events for inference.

In the following sections, we first present our Bayesian nonparametric method and then develop a maximum-likelihood method under a piecewise constant trajectory so we can directly compare an EM-based method to our Bayesian nonparametric method.

Gaussian-process-based Bayesian nonparametric estimation of $N\left(t\right)$

For our Bayesian methodology, we assume the log-Gaussian process prior on the population size trajectory,

$$N\left(t\right)=\exp \left[f\left(t\right)\right],f\left(t\right)\sim \mathcal{G}\mathcal{P}\left(\mathbf{0},C\left(\tau \right)\right),$$ (16)

where $\mathcal{G}\mathcal{P}\left(\mathbf{0},C\left(\tau \right)\right)$ denotes a Gaussian process with mean function 0 and inverse covariance function $C^{-1}\left(\tau \right)=\tau C^{-1}$ with precision parameter τ. For computational convenience, we use Brownian motion as our prior for $f\left(t\right)$ since its inverse covariance matrix is sparse. We place a Gamma prior on the precision parameter τ, $\tau \sim \Gamma \left(\alpha ,\beta \right)$. Assuming that recombination rate ρ is known, the posterior distribution of model parameters (Figure 3) is then

$$\begin{array}{l}\mathrm{Pr}\left[N\left(t\right),\tau |g_0,\dots ,g_{m-1}\right]\propto \mathrm{Pr}\left[g_0|N\left(t\right)\right]\\ \times \left\{{\displaystyle \prod _{i=0}^{m-2}\mathrm{Pr}\left[g_{i+1}|g_i,N\left(t\right)\right]}\right\}\mathrm{Pr}\left[N\left(t\right)|\tau \right]\mathrm{Pr}\left(\tau \right).\end{array}$$ (17)

The first two factors on the right side of Equation 17, detailed in Equations 8 and 11, involve integration over $N\left(t\right),$ an infinite-dimensional random function (Equation 16). We approximate the integral

$${\displaystyle {\int }_a^b\frac{dt}{N\left(t\right)}}={\displaystyle {\int }_a^b\exp \left[-f\left(t\right)\right]dt},$$

by the Riemann sum over a partition of the integration interval. That is,

$${\displaystyle {\int }_a^b\exp \left[-f\left(t\right)\right]dt}\approx {\displaystyle \sum _{j=i}^k\exp \left[-f_j^{\text{*}}\right]{\Delta }_j},$$ (18)

for $x_i<a<x_{i+1}<\dots <x_{k-1}<b<x_k,$ ${\Delta }_i=x_{i+1}-a,$ ${\Delta }_k=b-x_{k-1},$ and ${\Delta }_j=x_{j+1}-x_j$ for $i<j<k-1.$ $f_j^{\text{*}}$ is a representative value of $f\left(t\right)$ in the interval $\left(x_j,x_{j+1}\right);$ in our implementation, we set $f_j^{\text{*}}=f\left(x_j^{\text{*}}\right)$ with $x_j^{\text{*}}=\left(x_j+x_{j+1}\right)/2$. This way, we discretize our time window in d evenly spaced segments $x_1=0<x_2<\dots <x_d,$ with $x_d=\max \left(t_1^0,\dots ,t_1^{m-1}\right),$ the maximum time to the most common ancestor observed in the sequence of local genealogies, and approximate $N\left(t\right)$ by a piecewise linear function evaluated at $\left(x_1^{\text{*}},x_2^{\text{*}},\dots ,x_d^{\text{*}}\right)$.

Figure 3.

Structure of our Bayesian model for inferring population size trajectories from a realization of the SMC′ process at recombination breakpoints. Hyperparameter τ controls the smoothness of the log-Gaussian process prior on $N\left(t\right)$. Local genealogies depend on $N\left(t\right)$ and form a Markov chain of degree 1. Given the current local genealogy $g_{i-1},$ we sample the location of the new recombination breakpoint $b_i$ and a pruning location $p_i$ on genealogy $g_{i-1}$. The new genealogy $g_i$ depends on $N\left(t\right),$ $p_i,$ and $g_{i-1}$.

We condition on the set of m local genealogies $g_0,\dots ,g_{m-1}$ (assuming pruning locations are not known) to generate posterior samples for the vector $f*=\left[\log N\left(x_1^{\text{*}}\right),\dots ,\log N\left(x_d^{\text{*}}\right)\right]$ and τ and use these posterior samples to infer $N\left(t\right)$ at $t\in \left(x_1^{\text{*}},\dots ,x_d^{\text{*}}\right),$ where $x_i^{\text{*}}=\left(x_i+x_{i+1}\right)/2$. Updating the vector $f$ and τ separately is not recommended because of their strong dependency (Lan et al. 2015). Therefore, we update ($f,\tau )$ jointly in an MCMC sampling algorithm, using splitHMC (Shahbaba et al. 2014; Lan et al. 2015). splitHMC updates all model parameters jointly and it can be extended to a full inferential framework that is directly applicable to sequence data. The splitHMC method relies on Hamiltonian dynamics to propose a new state of the model parameters jointly with a higher acceptance rate than simple methods such as random-walk Metropolis (Neal 2009). splitHMC relies on our ability to calculate the log-likelihood of the observed data and the gradient vector of the log-likelihood (i.e., the score function). The log-likelihoods of the observed data are approximated via sums of the form in Equation 18. We approximate the score function $\nabla {\mathrm{ℒ}}_{\mathrm{obs}}\left(Y;f*\right)$ with respect to $f*$ by applying Fisher’s identity,

$$\nabla {\mathrm{ℒ}}_{\mathrm{obs}}\left(Y;f*\right)=E_{f*}\left[\nabla {\mathrm{ℒ}}_{\mathrm{c}}\left(Y_{\mathrm{c}};f*\right)|Y\right],$$

where, at each iteration in the MCMC, expectation is calculated using the current value of $f*$ (see Appendix).

Alternatively, one can update $N\left(t\right)$ in the MCMC algorithm, using the elliptical slice sampler (Murray et al. 2010) with a fixed value of τ (perhaps estimated from previous studies or from a preliminary run from the split Hamiltonian Monte Carlo algorithm). The advantage of using the elliptical slice sampler over the split Hamiltonian Monte Carlo is purely computational (the elliptical slice sampler does not require calculation of the score function).

Maximum-likelihood estimation of $N\left(t\right)$ with measures of uncertainty

We assume that the population size trajectory $N\left(t\right)$ is defined as in Equation 14. The standard coalescent density (Equation 4) and the transition densities defined in Equations 8 and 11 are tractable, so calculation of the likelihood (Equation 13) is tractable. However, maximization of the likelihood function cannot be performed analytically because pruning locations are missing. We implement an EM algorithm (Dempster et al. 1977) to find the maximum-likelihood estimator of $N=\left(N_1,\dots ,N_d\right)$. The complete data $Y_{\mathrm{c}}$ for inferring $N\left(t\right)$ are then the set of local genealogies $g_0,\dots ,g_{m-1}$ and the set of pruning locations $p_1\dots ,p_{m-1}$. For the invisible transitions, we also need to know the new coalescent times ${\left\{t_{\text{new}}^i\right\}}_{i\mathrm{\in ℐ}},$ where $\mathrm{ℐ}\subset \left\{1,2,\dots ,m-1\right\}$ denotes the set of indexes of invisible transitions.

The complete data log-likelihood is then

$${\mathrm{ℒ}}_{\mathrm{c}}\left(Y_{\mathrm{c}};N\right):=\log \mathrm{Pr}\left[g_0|N\left(t\right)\right]$$ (19)

$$+{\displaystyle \sum _{i=1}^{m-1}\log \hspace{0.17em}Pr\left[p_i=\left(u_i,w_i\right),t_{\text{new}}^i,c_i|g_{i-1},N\left(t\right)\right]}.$$

The EM algorithm starts by initializing the population size trajectory to a piecewise constant function with change points $x_1,\dots ,x_d$ with arbitrarily chosen vector $N^0$. At the kth iteration of the algorithm we set

$$N^k=\underset{N}{\arg \hspace{0.17em}\max }{E}_{N^{k-1}}\left[{\mathrm{ℒ}}_{\mathrm{c}}\left(Y_{\mathrm{c}};N\right)|Y\right].$$ (20)

The conditional expectation in Equation 20 is conditional on the observed data $Y$ defined in Equation 12. Let $x^i=\left\{x_1^i,x_2^i,\dots ,x_{d+n-1}^i\right\}$ be the ordered set of time points corresponding to the change points $x_1,\dots ,x_d$ and the coalescent time points ${\mathbf{t}}^i$ of local genealogy i. If the transition from $g_i$ to $g_{i+1}$ is visible, we replace the jth time point $x_j^i$ by $t_{\text{new}}^{i+1},$ where j corresponds to the index such that $x_{j-1}^i<t_{\text{new}}^{i+1}\le x_j^i$. For ease of notation, we denote the number of time intervals $\left|x^i\right|$ by $D=d+n-2.$ Let

$$a_j^0=\{\begin{array}{ll}1,\hfill & \text{if}\hspace{0.17em}{x}_{j+1}^0=t_k^0,\hspace{1em}\text{for}\hspace{0.17em}k=2,\dots ,n,\hfill \\ 0,\hfill & \text{otherwise},\hfill \end{array}$$

be an indicator function that takes the value of 1 when the jth interval contains a coalescent time of the first genealogy $g_0$. Then, the log density of the first genealogy is

$$\begin{array}{l}\log \mathrm{Pr}\left[g_0|N\left(t\right)\right]\\ =-{\displaystyle \sum _{j=1}^D\left\{a_j^0\log N\left(x_{j+1}^0\right)\right\}}\\ -{\displaystyle \sum _{j=1}^D\left\{\frac{A^0\left(x_{j+1}^0\right)\left[A^0\left(x_{j+1}^0\right)-1\right]\left(x_{j+1}^0-x_j^0\right)\exp \left[-\log N\left(x_{j+1}^0\right)\right]}{2}\right\}}.\end{array}$$ (21)

Let

$$z_j^i=\{\begin{array}{ll}1,\hfill & \text{ if}\hspace{0.17em}{x}_j^i<t_{\text{new}}^{i+1}\le x_{j+1}^i,\hfill \\ 0,\hfill & \text{otherwise},\hfill \end{array}$$

be an indicator function that takes the value of 1 when the new coalescent time of genealogy i happens in the corresponding time interval $\left(x_j^i,x_{j+1}^i\right),$ and let the adjusted interval length be

$${\Delta }_j^i=\{\begin{array}{ll}{x}_{j+1}^i-x_j^i,\hfill & \text{if}\hspace{0.17em}{u}_{i+1}<x_j^i,\hspace{1em}\text{and}\hspace{0.17em}{x}_{j+1}^i<t_{\text{new}}^{i+1}\hfill \\ \hfill & \left(\text{after}\hspace{0.17em}\text{pruning}\hspace{0.17em}\text{and}\hspace{0.17em}\text{before}\hspace{0.17em}\text{coalescence}\right),\hfill \\ x_{j+1}^i-u_{i+1},\hfill & \text{if}\hspace{0.17em}{x}_j^i<u_{i+1}<x_{j+1}^i\le t_{\text{new}}^{i+1}\hfill \\ \hfill & \left(\text{before}\hspace{0.17em}\text{coalescence}\hspace{0.17em}\text{with}\hspace{0.17em}\text{pruning}\hspace{0.17em}\text{adjustment}\right),\hfill \\ t_{\text{new}}^{i+1}-u_{i+1},\hfill & \text{if}\hspace{0.17em}{x}_j^i<u_{i+1}<t_{\text{new}}^{i+1}<x_{j+1}\hfill \\ \hfill & \left(\text{adjustment}\hspace{0.17em}\text{for}\hspace{0.17em}\text{pruning}\hspace{0.17em}\text{and}\hspace{0.17em}\text{coalescence}\right),\hfill \\ t_{\text{new}}^{i+1}-x_j^i,\hfill & \text{if}\hspace{0.17em}{u}_{i+1}<x_j^i<t_{\text{new}}^{i+1}<x_{j+1}^i\hfill \\ \hfill & \left(\text{after}\hspace{0.17em}\text{pruning}\hspace{0.17em}\text{with}\hspace{0.17em}\text{coalescence}\hspace{0.17em}\text{adjustment}\right),\hfill \\ 0,\hfill & \text{otherwise}.\hfill \end{array}$$

Then, the augmented transition density can be expressed as

$$\begin{array}{l}\log \mathrm{Pr}\left[p_i=\left(u_i,w_i\right),t_{\text{new}}^i,c_i|g_{i-1},N\left(t\right)\right]\\ =\log \mathrm{Pr}\left[p_i=\left(u_i,w_i\right),t_{\text{new}}^i,c_i,z^i,{\mathbf{\Delta }}^i|g_{i-1},N\left(t\right)\right]\\ =-\log l_{i-1}-{\displaystyle \sum _{j=1}^D\left\{z_j^{i-1}\log N\left(x_{j+1}^{i-1}\right)\right\}}\\ -{\displaystyle \sum _{j=1}^D\left\{A^{i-1}\left(x_{j+1}^{i-1}\right){\mathrm{\Delta }}_j^{i-1}\exp \left[-\log N\left(x_{j+1}^{i-1}\right)\right]\right\}},\end{array}$$ (22)

where $z^i$ and ${\mathbf{\Delta }}^i$ are the vectors with $z_j^i$ and ${\Delta }_j^i$ elements. For the EM algorithm we need to compute the conditional expected vectors $E\left[z_j^i|Y\right]$ and $E\left[{\mathbf{\Delta }}_j^i|Y\right].$ The details of these calculations are in the Appendix.

We use the Fisher information matrix to compute approximate standard errors of $\log \widehat{N}$ and use these standard errors together with asymptotic normality of maximum-likelihood estimators to produce confidence intervals for log population size piecewise trajectories. We compute the observed Fisher information matrix following Louis (1982),

$${\widehat{I}}_Y\left[\widehat{N}\right]={\text{E}}_{\widehat{N}}\left[-H{\mathrm{ℒ}}_{\mathrm{c}}\left(Y_{\mathrm{c}};\widehat{N}\right)|Y\right]-{\text{E}}_{\widehat{N}}\left[\nabla {\mathrm{ℒ}}_{\mathrm{c}}\left(Y_{\mathrm{c}};\widehat{N}\right)\nabla {\mathrm{ℒ}}_{\mathrm{c}}{\left(Y_{\mathrm{c}};\widehat{N}\right)}^{\prime }|Y\right],$$

where $\nabla {\mathrm{ℒ}}_{\mathrm{c}}\left(Y_{\mathrm{c}};\widehat{N}\right)$ is the gradient and $H{\mathrm{ℒ}}_{\mathrm{c}}\left(Y_{\mathrm{c}};\widehat{N}\right)$ is the Hessian of the complete-data log-likelihood with respect to $\log N$. This requires the calculation of conditional cross-product means and conditional second moments described in File S7.

Data availability

The R code for all simulation studies and analysis of sequence data conducted in this article are publicly available at http://ramachandran-data.brown.edu/.

Results

We simulated 1000 local genealogies of 2, 20, and 100 individuals from each of the three different demographic models described in Table 2, using MaCS (Chen et al. 2009); see File S3 for details of these simulations. We assumed that all individuals were sampled at time $t=0.$

Table 2. Simulated demographic scenarios.

Demographic model	$N\left(t\right)$
Constant population size	$N\left(t\right)=1$
Exponential growth followed by constant size	$N\left(t\right)=\{\begin{array}{ll}1,\hfill & \text{for}\hspace{1em}t\in \left(0,0.1\right),\hfill \\ \exp \left[-10\left(t-0.1\right)\right],\hfill & \text{for}\hspace{1em}t\in \left(0.1,\mathrm{\infty }\right).\hfill \end{array}$
Population bottleneck	$N\left(t\right)=\{\begin{array}{ll}1,\hfill & \text{for}\hspace{1em}t\in \left(0,0.3\right),\hfill \\ 0.1,\hfill & \text{for}\hspace{1em}t\in \left(0.3,0.5\right),\hfill \\ 1,\hfill & \text{for}\hspace{1em}t\in \left(0.5,\mathrm{\infty }\right).\hfill \end{array}$

The argument t denotes time measured in units of $N_0$ generations.

We compared our point estimates with the truth for each demographic model, using the sum of relative errors (SRE),

$$\mathrm{SRE}={\displaystyle \sum _{i=1}^K\frac{\left|\widehat{N}\left(x_i\right)-N\left(x_i\right)\right|}{N\left(x_i\right)}},$$ (23)

where $\widehat{N}\left(x_i\right)$ is the estimated population size trajectory at time $x_i$. We compute SRE at equally spaced time points $x_1,\dots ,x_K$. Second, we compute the mean relative width (MRW) as

$$\mathrm{MRW}={\displaystyle \sum _{i=1}^K\frac{\left|{\widehat{N}}_{\text{up}}\left(x_i\right)-{\widehat{N}}_{\text{low}}\left(x_i\right)\right|}{KN\left(x_i\right)}},$$ (24)

where ${\widehat{N}}_{\text{up}}\left(x_i\right)$ corresponds to the $97.5\text{\%}$ upper limit and ${\widehat{N}}_{\text{low}}\left(x_i\right)$ corresponds to the $2.5\text{\%}$ lower limit of $\widehat{N}\left(x_i\right)$. For EM estimates, $\left[{\widehat{N}}_{\text{low}}\left(x_i\right),{\widehat{N}}_{\text{up}}\left(x_i\right)\right]$ corresponds to the 95% confidence interval estimated using the observed Fisher information; for Bayesian GP estimates, $\left[{\widehat{N}}_{\text{low}}\left(x_i\right),{\widehat{N}}_{\text{up}}\left(x_i\right)\right]$ corresponds to the 95% Bayesian credible interval (BCI) of $\widehat{N}\left(x_i\right)$. To measure how well these intervals cover the truth, we compute the envelope measure (ENV) in the following way:

$$\mathrm{ENV}=\frac{{\displaystyle {\sum }_{i=1}^{K}I\left({\widehat{N}}_{\text{up}}\left(x_i\right)\le N\left(x_i\right)\le {\widehat{N}}_{\text{low}}\left(x_i\right)\right)}}{K}.$$ (25)

We compute SRE, MRW, and ENV for $K=150$ at equally spaced time points.

For our Bayesian GP estimates, we estimate $N\left(x_i\right)$ at $d=100$ time points, unless stated otherwise.

The parameters of the Gamma prior on the GP precision parameter τ were set to $\alpha =\beta =0.001,$ reflecting our lack of prior information about the smoothness of the population size trajectory.

For our EM estimates, we used different discretizations based on Equation 15 and varying the number of change points d and κ over the fixed interval $\left(0,x_d\right)$ with $x_d$ set to be the maximum observed coalescent time. For the cases where we consider only one genealogy ($m=1$), the EM approach becomes standard maximum-likelihood estimation.

We summarize our posterior inference and compare our Bayesian GP method to the EM method in Figure 5, Figure 6, and Figure 7. The population size trajectory is log-transformed for ease of visualization and for direct comparison with other methods (Minin et al. 2008; Palacios and Minin 2013).

Figure 5.

Inference of population size trajectories $N\left(t\right)$ for a pair of individuals ($n=2$). Simulated data under constant population size (A), exponential and constant trajectory (B), and a bottleneck (C). We show estimates from $m=100,$ $m=500,$ and $m=1000$ local genealogies. Dashed lines show the true trajectories, blue lines and light blue areas represent EM point estimates and 95% confidence areas, and red lines and pink areas represent Bayesian GP posterior medians and 95% BCIs. Boxplots of the TMRCA are shown at the bottom of each plot.

Figure 6.

Inference of population size trajectories $N\left(t\right)$ for $n=20.$ Simulated data under constant population size (A), exponential and constant trajectory (B), and a bottleneck (C). We show estimates from $m=1$ genealogy, $m=100$ local genealogies, and $m=1000$ local genealogies. Dashed lines show the true trajectories, blue lines and light blue areas represent EM point estimates and 95% confidence areas, and red lines and pink areas represent Bayesian GP posterior medians and 95% BCIs. Boxplots of the TMRCA are shown at the bottom of each plot.

Figure 7.

Inference of population size trajectories $N\left(t\right)$ for $n=100.$ Simulated data under constant population size (A), exponential and constant trajectory (B), and bottleneck (C). We show estimates from $m=1$ genealogy, $m=100$ local genealogies, and $m=1000$ local genealogies. Dashed lines show the true trajectories, blue lines and light blue areas represent EM point estimates and 95% confidence areas, and red lines and pink areas represent Bayesian GP posterior medians and 95% BCIs. Boxplots of the TMRCA are shown at the bottom of each plot.

Sensitivity of EM estimates of $N\left(t\right)$ to discretization

In Figure 4, we show our Bayesian GP and EM estimates of a constant population size trajectory from a single genealogy of 100 individuals with different discretizations. We find that our Bayesian GP point estimates depicted in Figure 4A recover the truth (dashed line) almost perfectly with less uncertainty than the EM (Figure 4, B and C). Comparing our Bayesian GP estimates with different discretizations [50, 100, and 200 equally spaced time points (Figure 4A)], we find that increasing the number of time points improves inference (Table 3) but that the differences between estimates among the three discretizations are marginal (Figure 4A). In contrast, we show that different grid definitions alter the EM estimates (Figure 4B). It is not clear how to define a good strategy for the definition of the grid for the EM method, even for the simple model of constant population size. For example, increasing κ from 100 to 500 with 5 change points (Figure 4B) does not improve estimation. Increasing the number of change points does not necessarily improve the estimates either, for example, increasing the number of change points from 5 to 10 for $\kappa =10$ (Figure 4, B and C). EM grid sensitivity is persistent even when the number of genealogies increases; Figure S2 in File S4 shows that the best definition of change points when our data consist of 1000 local genealogies of 100 individuals is 10 evenly distributed change points.

Figure 4.

Sensitivity to parameter discretization. Population size trajectories estimated from one simulated genealogy ($m=1$) of 100 individuals with a constant population size are compared. We show true trajectories as dashed lines. (A) Bayesian GP estimates at $d=50,100,$ and 200 equally spaced time points. (B) EM estimates of a piecewise constant trajectory with $d=5$ change points and $\kappa =1,10,$ and 100 (Equation 15). (C) EM estimates of a piecewise constant trajectory with $d=10$ change points and $\kappa =10,100,$ and 500 (Equation 15). Point estimates are shown as solid black lines. 95% percent credible intervals and 95% confidence intervals are shown by gray areas.

Table 3. Summary statistics for simulation results depicted in Figure 4.

Simulation of a single genealogy with $n=100$	SRE	MRW	ENV (%)
MLE $d=5,$ $\kappa =1$	41.80	14.76	100.0
MLE $d=5,$ $\kappa =10$	41.05	2.98	100.0
MLE $d=5,$ $\kappa =100$	57.12	1.72	100.0
MLE $d=10,$ $\kappa =10$	47.93	16.08	100.0
MLE $d=10,$ $\kappa =100$	61.77	3.91	100.0
MLE $d=10,$ $\kappa =500$	31.52	3.60	100.0
Bayesian GP $d=50$	6.98	1.88	100.0
Bayesian GP $d=100$	5.52	2.15	100.0
Bayesian GP $d=200$	4.96	1.70	100.0

SRE is the sum of relative errors (Equation 23), MRW is the mean relative width of the 95% BCI (Equation 24), and ENV is the envelope measure (Equation 25). Values in boldface type indicate best performance.

Comparing methods for estimating $N\left(t\right)$

Figure 5 shows the estimated population size trajectories when the number of samples is two for the three different demographic scenarios and varying the number of local genealogies (100, 500, and 1000 local genealogies). For constant and exponential growth, our EM method assumes a piecewise constant trajectory of 10 change points ($d=10$) and $\kappa =1,$ using Equation 15 (similar to Li and Durbin 2011 and Rasmussen et al. 2014). For the bottleneck scenario, some of the intervals did not have coalescent events; hence, for this case we assumed a piecewise constant trajectory of 5 change points ($d=5$) and $\kappa =1$ for constructing our EM estimates. We show the boxplots of the time to the most recent common ancestor (TMRCA) at the bottom of each plot in Figure 5, which indicate the uncertainty expected in our estimates.

Both approaches, EM and Bayesian GP, show narrower confidence and credible intervals at the center of the distribution of the TMRCA, particularly during the bottleneck in Figure 5C.

For the constant population size model in Figure 5A, our Bayesian GP considerably outperforms our EM estimates. This is not surprising since a priori $\log N\left(t\right)$ has mean 0 in our Bayesian approach (Equation 16). Moreover, EM confidence intervals cover the truth only ∼30% of the time, while the GP method covers 100% of the truth (Table 4A). Despite placing a mean-0 prior on $\log N\left(t\right),$ the Bayesian GP method accurately recovers sudden changes as shown in the bottleneck scenario. Although our Bayesian GP prior on $\log N\left(t\right)$ is Brownian motion (which is not differentiable at any point), our Bayesian GP recovers smooth curves (Figure 5B).

Table 4. Summary of simulation results depicted in Figure 5.

A. Simulations with n = 2
	SRE			MRW			ENV
Simulation and method	m = 100	m = 500	m = 1000	m = 100	m = 500	m = 1000	m = 100 (%)	m = 500 (%)	m = 1000 (%)
Const. EM	39.80	41.78	38.60	0.98	0.26	0.08	31.3	28.0	19.3
Const. GP	30.60	4.25	3.04	0.49	0.33	0.22	100.0	100.0	100.0
Exp. EM	64.68	25.70	33.70	0.91	0.16	0.12	42.0	26.0	6.6
Exp. GP	28.38	32.70	26.76	2.04	0.45	0.33	100.0	56.0	50.6
Bottle. EM	48.48	46.51	127.70	0.43	0.45	1.37	40.6	30.0	34.0
Bottle. GP	33.76	45.14	223.58	3.44	6.84	17.13	98.0	94.6	94.6
B. Simulations with n = 20
	SRE			MRW			ENV
	m = 1	m = 100	m = 1000	m = 1	m = 100	m = 1000	m = 1 (%)	m = 100 (%)	m = 1000 (%)
Const. EM	60.87	121.30	25.60	2.28	2.16	0.23	100.0	37.7	39.3
Const. GP	31.74	3.94	13.22	1.06	0.70	0.36	100.0	100.0	100.0
Exp. EM	40.97	40.66	40.22	3.11	0.37	0.19	100.0	38.6	19.3
Exp. GP	25.35	27.03	65.61	3.53	1.56	0.42	100.0	100.0	39.3
Bottle. EM	147.93	78.40	78.20	6.98	0.81	68.4	66.0	78.6	49.33
Bottle. GP	68.93	78.2	50.92	2.74	2.47	1.47	92.0	79.3	78.6
C. Simulations with n = 100
	SRE			MRW			ENV
	m = 1	m = 100	m = 1000	m = 1	m = 100	m = 1000	m = 1 (%)	m = 100 (%)	m = 1000 (%)
Const. EM	41.05	220.85	43.41	2.98	4.93	0.99	100.0	35.3	48.0
Const. GP	5.52	34.78	12.17	2.15	1.49	0.47	100.0	100.0	89.3
Exp. EM	76.86	40.22	27.63	3.23	0.81	0.13	87.3	42.0	14.0
Exp. GP	114.53	25.82	26.42	3.57	1.55	0.83	100.0	100.0	100.0
Bottle. EM	194.77	59.54	127.68	3.95	1.08	0.85	84.0	51.3	45.3
Bottle. GP	90.27	44.14	42.68	6.98	2.62	1.74	100.0	94.7	96.0

SRE is the sum of relative errors calculated as in (23), MRW is the mean relative width of the 95% BCI as Const: Constant population simulation scenario. Exp: Exponential growth followed by constant population size simulation scenario and Bottle: Population bottleneck. defined in (24), and ENV is the envelope measure calculated as in (25). Values in boldface type indicate best performance for each demographic model and sample size.

Table 4A shows the performance statistics for the estimates of $N\left(t\right)$ in Figure 5. In general, our Bayesian GP has wider credible intervals than the EM confidence intervals but these credible intervals cover the true trajectory better than the EM confidence intervals in all cases (MRW and ENV in Table 4). Our Bayesian GP estimates also generally have smaller sums of relative errors (SRE in Table 4). Under the bottleneck scenario, our Bayesian GP produces greater sums of relative errors than does the EM, but our Bayesian GP estimates recover the truth more accurately than the EM during the bottleneck.

Figure 6 and Figure 7 show our estimates when $n=20$ and $n=100$ (Table 4, B and C, gives performance statistics). In general, our GP-based estimates have smaller SRE and larger ENV than the EM-based estimates and hence, the MRW is usually wider in the GP-based estimates, accurately reflecting the uncertainty of the estimates. As expected, increasing the number of loci (m) generally decreases the width of the confidence and credible intervals of our estimates (MRW). Although this is generally true for EM estimates as well, EM estimates have very low coverage of the truth (MRE in Table 4) when the number of loci increases.

Sampling more individuals vs. sequencing more loci

Figure 5, Figure 6, and Figure 7 show our estimates for $n=2,20,$ and 100 sampled individuals across varying numbers of loci. Performance of EM estimates depends strongly on the definition of the grid, so we focus here on the Bayesian GP estimates. We find that increasing the number of loci decreases uncertainty of our estimates and allows us to infer $N\left(t\right)$ farther back in time. Increasing the number of samples does not necessarily increase the performance of our GP estimates (File S6). For example, under the bottleneck scenario, we are able to detect the bottleneck fairly accurately even for two samples with $m=1000$ local genealogies. This is because most TMRCAs observed under the bottleneck scenario occur during the bottleneck (Figure 5, Figure 6, and Figure 7), regardless of the sample size. In contrast, in our exponential growth scenario, increasing the number of samples from $n=2$ to $n=100$ improves accuracy: point estimates are closer to the truth (SRE in Table 4, A–C) and credible intervals cover the truth completely (ENV of $100\text{\%}$).

Sequential Tajima’s genealogies are sufficient statistics under the SMC′

Under the SMC′, marginally at each locus along the chromosome, a local genealogy is a realization of Kingman’s n-coalescent (Kingman 1982), a continuous-time Markov chain taking its values in the set ${\mathcal{K}}_n$ of sequences of partitions of the label set $\left\{1,2,\dots ,n\right\}$. A local genealogy g of n individuals includes labeled topology $K_n$ and coalescent times $t=\left(t_n,\dots ,t_2\right)$. The state space of a local genealogy is then $\mathcal{G}={\mathcal{K}}_n\otimes {\mathrm{ℝ}}_{}^{+n-1},$ and the cardinality of the set ${\mathcal{K}}_n$ is $n!\left(n-1\right)!/2^{n-1}$. However, only the set of ordered coalescent times carries information about $N\left(t\right)$. For a single locus, the set of coalescent times provides sufficient statistics for inferring $N\left(t\right)$ (see Proof in the Appendix). A natural question that follows is whether the coalescent times corresponding to the set of local genealogies are sufficient statistics for inferring $N\left(t\right)$ under the SMC′ model. We find that the sufficient statistics for inferring $N\left(t\right)$ under the SMC′ model are the coalescent times, when taken together with local ranked tree shapes (tree with no labels but ranked coalescent events). For a single locus, the set of coalescent times together with the ranked tree shape corresponds to a realization of Tajima’s n-coalescent. Tajima’s n-coalescent (Tajima 1983) is a continuous-time Markov chain taking its values in the set ${\mathcal{G}}_n$ of ranked tree shapes [also called histories, evolutionary relationships, or vintaged and sized coalescent (Sainudiin et al. 2014)]. The state space of Tajima’s local genealogy is then ${\mathcal{G}}^T={\mathcal{G}}_n\otimes {\mathrm{ℝ}}^{+n-1},$ and the cardinality of the set ${\mathcal{G}}_n$ corresponds to the sequence of Euler zigzag numbers whose first 10 elements are $1,1,1,2,5,16,61,272,1385,7936$ (Disanto and Wiehe 2013). The probability of getting a particular type of ranked tree shape $H_n$ of n samples (Tajima 1983) is given by

$$P\left(H_n\right)=\frac{2^{n-c-1}}{\left(n-1\right)!},$$ (26)

where c is the number of cherries, defined as branching events that lead to exactly two leaves.

We defined transition densities in terms of coalescent times and $F_{i,j}$ quantities (see Methods: SMC′ calculations). The set of all $F_{i,j}$ quantities from a local genealogy forms a triangular matrix: an F matrix. We show that (i) F matrices are in bijection with ranked tree shapes and (ii) the set of local Tajima’s genealogies has sufficient statistics for inferring $N\left(t\right)$ under the SMC′ model (see Appendix). These observations are crucial for inferring $N\left(t\right)$ from sequence data directly. Coalescent-based inference from sequence data relies on marginalization over the hidden state space of genealogies. In the Appendix, we show that the state space needed is the space of local Tajima’s genealogies, as opposed to the space of local Kingman’s genealogies. For $n=10$ sequences, there are $2,571,912,000$ possible labeled topologies while only $7936$ possible ranked tree shapes.

Application to human data

We applied our method to a 2-Mb region on chromosome 1 (187,500,000–189,500,000) with no genes from five Yorubans from Ibadan, Nigeria (YRI) and five Utah residents of central European descent (CEU) from the 1000 Genomes pilot project (1000 Genomes Project Consortium 2012) and previously analyzed for the same purpose (Sheehan et al. 2013). We used ARGweaver (Rasmussen et al. 2014) to obtain a sample path of local genealogies for the two populations (YRI and CEU). The parameters used were 200 change points, a mutation rate of $\mu =1.26\times {10}^{-8},$ and a recombination rate of $\rho =1.6\times {10}^{-8}$ (Rasmussen et al. 2014) (File S5). We note that ARGweaver assumes the SMC process and our method assumes the SMC′ process. Moreover, our inference is based on a single sample of the SMC process with known pruning times. Our ARGweaver set of local genealogies is discretized at 200 time points and our GP-based inference is influenced by this discretization. In Figure 8 we show our estimates of past Yoruban (in blue) and European population sizes (in green). The two population size trajectories experience a series of bottlenecks and overlap until ∼100 KYA, assuming a diploid reference population size of $N_0$ = 10,000 and a generation time of 25 years. In Figure 8 we recover an out-of-Africa bottleneck that starts ∼100 KYA and ends ∼30 KYA in the European population. These results are consistent with previously published results (Gronau et al. 2011; Li and Durbin 2011; Rasmussen et al. 2011; Sheehan et al. 2013; Schiffels and Durbin 2014). In File S5, Figure S4A, we show the estimates of $\log N\left(t\right)$ instead of $N\left(t\right)$ and time measured in units of $N_0$ generations (as in Figure 5, Figure 6, and Figure 7). We note that this two-step procedure of inferring local genealogies with ARGweaver and then using our method introduces biases and ignores genealogical uncertainty. In File S5, we correct for some of the bias caused by using this two-step procedure and show that our inferred population size trajectory remains valid for the recent past.

Figure 8.

Inference of human population size trajectories $N\left(t\right)$ for $n=10.$ Green solid line and green areas represent the posterior median and 95% BCI for the European population (CEU) and blue solid line and blue areas represent the posterior median and 95% BCI for the Yoruban population (YRI). Time is measured in years in the past, assuming a generation length of 25 years and a reference diploid population of 10,000 individuals. The x-axis is log transformed.

Assessing the effect of using genealogies inferred with ARGweaver

We simulated sequence data, used ARGweaver for inferring a set of local genealogies, and used our method on those genealogies to obtain estimates of $\log N\left(t\right)$. To this end, we took the sequences of the first 1000 local genealogies of $n=20$ individuals simulated with MaCS as described in section 3 of File S5. We then generated the sequence lengths ($s_i,$ for the locus corresponding to $g_{i-1}$) as in Equation 5,

$$s_i\sim \text{Exponential}\left(\rho \times l_{i-1}\times N_0\right),$$

where $l_{i-1}$ is the tree length of $g_{i-1}$ in units of $N_0$ generations and $N_0$ is the current population size. In our simulations, we set $N_0=20,000,$ $\rho =1.8\times {10}^{-8}$. To simulate sequence data of length $s_i$ over genealogy $g_{i-1},$ we used Seq-Gen (Rambaut and Grassly 1997) implemented in the R package phyclust (Chen 2011) from the Jukes–Cantor mutation model (Jukes and Cantor 1969) with mutation rate $\mu =2\times 1.8\times {10}^{-8}$. We then used ARGweaver to infer a sample of local genealogies with the same corresponding parameters and with 200 change points for discretization of time. Figure 9 shows three estimations of effective population size trajectories for our three simulation scenarios. Figure 9, A–C, left, shows our GP-based estimates from 1000 simulated genealogies from MaCS; Figure 9, A–C, center, shows our GP-based estimates from a realization of local genealogies obtained from ARGweaver; and Figure 9, A–C, right, shows our GP-based estimates correcting the number of lineages used in our calculations, replacing $A^i\left(t\right)$ by $A^i\left(t\right)-1$ in our likelihood calculations. We find that our estimates are not only noisier using this approach but also biased.

Figure 9.

Assessing the effect of a two-step inferential approach. (A–C) Left, our GP-based estimates when 1000 local genealogies are known; center, our GP-based estimates when ARGweaver estimated local genealogies are used for inference; right, our corrected GP-based estimates when ARGweaver estimated local genealogies are used for inference.

Discussion

In this article, we propose a Gaussian-process-based Bayesian nonparametric method for estimating effective population size trajectories $N\left(t\right)$ from a sequence of local genealogies, accounting for recombination. Under a variety of simulated demographic scenarios and sampling designs, our method recovers the truth with better precision and accuracy than a maximum-likelihood approach (Figure 5, Figure 6, and Figure 7). We apply our method to genealogies estimated using ARGweaver (Rasmussen et al. 2014) for European and African samples in the 1000 Genomes; this application to real data recovers the known features of the out-of-Africa bottleneck (Figure 8).

Several recent approaches have emerged for inferring population size trajectories from multiple whole-genome sequences using the SMC (Li and Durbin 2011; Sheehan et al. 2013; Schiffels and Durbin 2014). However, current SMC-based methods rely on maximum-likelihood inference (EM) of both a discretized parameter space and a discretized state space to gain computational tractability, and incur the costs of reduced accuracy and biased estimates. Although in principle the EM approach and the Bayesian nonparametric approach approximate $N\left(t\right)$ similarly—by either a piecewise constant or a piecewise linear function—the Bayesian nonparametric approach is not affected by increasing the number of parameters (or change points) in the estimation of $N\left(t\right)$. For comparison with existing methods, we implemented an EM approach to infer population size trajectories from a sequence of local genealogies and we note that increasing the number of loci may actually increase the bias of the EM estimates (Figure 5, Figure 6, and Figure 7). For example, in simulation, our EM approach incorrectly detects the initial period of the simulated bottleneck ($\sim 0.8N_0$ instead of $0.5N_0$ generations ago) with narrow confidence intervals (Figure 7C).

Using Bayesian GP for inferring population size trajectories offers many advantages over the EM approach. Similar to Palacios and Minin’s (2013) approach to inference from a single genealogy, we a priori assume that $N\left(t\right)$ follows a log Brownian motion process. This allows us to model $N\left(t\right)$ as a continuous positive function. The main advantage of using a Brownian motion process is that its inverse covariance function is a sparse matrix that allows for fast computations. Since the likelihood function involves integration over $N\left(t\right),$ this integral is approximated by the Riemann sum over a regular grid of points. The finer the grid is, the better the approximation. We find that our method performs well for inferring $N\left(t\right)$ at 100 change points in all our examples and, more importantly, results are not sensitive to the number of change points used in the analysis (Figure 4). Our Bayesian approach relies on MCMC for inference from the posterior distribution of model parameters. Because population sizes at different grid points are correlated, we adapt the recently developed MCMC technique splitHMC for jointly sampling all model parameters (Shahbaba et al. 2014; Lan et al. 2015). splitHMC is a Metropolis sampling algorithm that efficiently proposes states that are distant from current states with high acceptance rates. It has been shown to be more efficient in inferring $N\left(t\right)$ from a single genealogy than elliptical slice sampling or regular Hamiltonian Monte Carlo sampling (Lan et al. 2015). However, splitHMC relies on calculating the score function at every single iteration. Because the pruning time in each local genealogy is unknown, we calculate the score function via Fisher’s formula.

In simulations, we find that our algorithm scales well with hundreds of individuals; our computational bottleneck is in the number of local genealogies. We envision that extending the current methodology to inference from sequence data directly will require a strategy for sampling shorter genomic segments. This would be a probabilistic alternative to arbitrarily choosing segment lengths (Sheehan et al. 2013; Rasmussen et al. 2014).

Under the SMC model, every recombination event along the genome translates to a new coalescent event for the sample under study, so increasing the number of loci results in more realizations of the coalescent process. The longer the segments are and the larger the number of samples taken, the greater the chance of observing variation due to recombination. This fact makes it hard to define a sampling strategy: Longer genomes or larger sample sizes? We show that increasing the number of local genealogies improves precision of our Bayesian GP estimates (Figure 5, Figure 6, and Figure 7). However, resolution into the past from contemporaneous sequences highly depends on the actual population size trajectory $N\left(t\right)$.

We used ARGweaver (Rasmussen et al. 2014) to generate two samples of contiguous local genealogies corresponding to a 2-Mb region of chromosome 1 for five Europeans (CEU) and five Africans (YRI) from the 1000 Genomes Project; this genomic region is free of genes and was also analyzed in Sheehan et al. (2013). Taking these two samples of local genealogies as our data (4186 local genealogies for CEU and 6247 local genealogies for YRI), we were able to use our Bayesian GP method to infer Yoruban and European effective population size trajectories (Figure 8). We find an out-of-Africa bottleneck that began ∼100 KYA and ended ∼30 KYA in the European population, consistent with Li and Durbin (2011); Rasmussen et al. (2011); Gronau et al. (2011); Sheehan et al. (2013), and Schiffels and Durbin (2014). We note that our estimates are based on a single sample of local genealogies and thus ignore genealogical uncertainty. Moreover, we generated our data from the posterior distribution of local genealogies, using ARGweaver at 200 time intervals, so our GP-based approach cannot fully detect sudden changes that may occur between the discretized times. In addition, ARGweaver assumes an SMC prior model on local genealogies and our GP-based method assumes the SMC′ process; the lack of invisible recombination events in ARGweaver’s genealogies will bias inference (as shown in simulated data in Figure 9).

The natural next extension for our method presented in this study is to infer $N\left(t\right)$ from sequence data directly and not from the set of local genealogies. Our MCMC approach allows us to extend the current methodology in a Bayesian hierarchical framework where the SMC′ process would be used as a prior distribution over local genealogies. The work we present here suggests a combination of ARGweaver accommodating SMC′ and GP priors would result in an efficient method for inferring population size trajectories from sequence data directly. In addition, our model can be easily modified to model a variable recombination rate along chromosomal segments and to jointly infer variable recombination rates and $N\left(t\right)$.

Finally, we show that, under the SMC′ model, local ranked tree shapes and coalescent times correspond to a set of local Tajima’s genealogies; these Tajima’s genealogies are sufficient statistics for inferring $N\left(t\right)$. Under the SMC′ model, the state space needed for inferring population size trajectories from sequence data is that of a sequence of local Tajima’s genealogies. This lumping, or reduction of the original SMC′ process, will allow more efficient inference from sequence data directly.

Current methods for inferring population size trajectories make trade-offs to analyze whole genomes that limit both biological understanding of sudden population size changes and the ability to test hypotheses regarding population size changes. This work represents a critical set of theoretical results that lay the groundwork for efficient estimation of detailed histories from sequence data with measures of uncertainty.

Appendix

Discretization

For both our Bayesian method and our EM method, we assume that $N\left(t\right)$ is a piecewise linear (or piecewise constant) function with d change points. Let $x^i=\left\{x_1^i,x_2^i,\dots ,x_{d+n-1}^i\right\}$ be the ordered set of time points corresponding to the change points $x_1,\dots ,x_d$ and the coalescent time points $t^i$ of local genealogy i. Then, we calculate all the factors needed for the observed data likelihood (Equation 13) and the complete data likelihood (Equation 19).

Let ${\tilde{F}}_{k,j}^i$ denote the discretized version of $F^i$ that represents the number of branches in $g_i$ that do not coalesce with any other branch in the time interval $\left(x_k^i,x_{j+1}^i\right)$. Note that the indexes here are in increasing order, $k\le j$. Similarly, let $\left(1/l_i\right){\tilde{P}}_{k,j}^i$ denote the probability that $U_i$ (the pruning time along genealogy i) occurs in $\left(x_k^i,x_{k+1}^i\right)$ and the self-coalescing event occurs at time $t_{\text{new}}^i$ in $\left(x_j^i,x_{j+1}^i\right)$. That is,

$${\tilde{P}}_{k,j}^i=\{\begin{array}{ll}\frac{1}{A^i\left(x_{j+1}^i\right)}\left({\mathrm{\Delta }}_j^i-{\tilde{Q}}_j^i\right)\hfill & k=j\hfill \\ {\tilde{Q}}_k^i{\tilde{q}}_{k+1}^i{\tilde{q}}_{k+2}^i\dots {\tilde{q}}_{j+1}^i\left(1-{\tilde{q}}_j^i\right)\hfill & k<j,\hfill \end{array}$$ (A1)

where

$$\frac{1}{l_i}{\tilde{Q}}_k^i=\frac{1}{l_i}\frac{N\left(x_{k+1}^i\right)}{A^i\left(x_{k+1}^i\right)}\left[1-{\tilde{q}}_k^i\right]$$ (A2)

is the joint probability of pruning time $U_i\in \left(x_k^i,x_{k+1}^i\right)$ and not coalescing back to the same branch in the time interval $\left(x_k^i,x_{k+1}^i\right),$ and

$${\tilde{q}}_k^i=\exp \left\{-\frac{A^i\left(x_{k+1}^i\right){\mathrm{\Delta }}_k^i}{N\left(x_{k+1}^i\right)}\right\}.$$

Expectation-Maximization Algorithm

E step

Equations 21 and 22 show that for the E-step, the only expectations we need are $E\left[z_j^i|Y\right]$ and $E\left[{\Delta }_j^i|Y\right]$. We compute these expressions as follows:

$$E\left[z_j^i|Y\right]=\{\begin{array}{ll}\frac{{\displaystyle {\sum }_{k=1}^j{\tilde{F}}_{k,j}^i{\tilde{P}}_{k,j}^i}}{{\displaystyle {\sum }_{j=1}^D{\displaystyle {\sum }_{k=j}^D{\tilde{F}}_{k,j}^i{\tilde{P}}_{k,j}^i}}},\hfill & \text{for}\hspace{0.17em}i\in \mathrm{ℐ}\left(\text{invisible}\right).\hfill \\ z_j^i,\hfill & \text{for}\hspace{0.17em}i\in {\mathrm{ℐ}}^c\left(\text{visible}\right).\hfill \end{array}$$

For $i\in \mathrm{ℐ}$

$$E\left[{\mathrm{\Delta }}_j^i|Y\right]=\left(x_{j+1}^i-x_j^i\right)\frac{{\displaystyle {\sum }_{k=1}^{j-1}{\displaystyle {\sum }_{l=j+1}^D{\tilde{F}}_{k,l}^i{\tilde{P}}_{k,l}^i}}}{{\displaystyle {\sum }_{j=1}^D{\displaystyle {\sum }_{k=j}^D{\tilde{F}}_{k,j}^i{\tilde{P}}_{k,j}^i}}}$$

$$+{\displaystyle {\int }_{x_j^i}^{x_{j+1}^i}\left(x_{j+1}^i-u\right)\exp \left\{-\frac{\left(x_{j+1}^i-u\right)A^i\left(x_{j+1}^i\right)}{N\left(x_{j+1}^i\right)}\right\}du}\frac{{\displaystyle {\sum }_{k=j+1}^D{\tilde{F}}_{j,k}^i\left({\tilde{P}}_{j,k}^i/{\tilde{Q}}_j^i\right)}}{{\displaystyle {\sum }_{j=1}^D{\displaystyle {\sum }_{k=j}^D{\tilde{F}}_{k,j}^i{\tilde{P}}_{k,j}^i}}}$$

$$+{\displaystyle {\int }_{x_j^i}^{x_{j+1}^i}{\displaystyle {\int }_u^{x_{j+1}^i}\left(t-u\right)\frac{1}{N\left(x_{j+1}^i\right)}\exp \left\{-\frac{\left(t-u\right)A^i\left(x_{j+1}^i\right)}{N\left(x_{j+1}^i\right)}\right\}dt}du}\frac{{\tilde{F}}_{j,j}^i}{{\displaystyle {\sum }_{j=1}^D{\displaystyle {\sum }_{k=j}^D{\tilde{F}}_{k,j}^i{\tilde{P}}_{k,j}^i}}}$$

$$+{\displaystyle {\int }_{x_j^i}^{x_{j+1}^i}\left(t-x_j^i\right)\exp \left\{-\frac{\left(t-x_j^i\right)A^i\left(x_{j+1}^i\right)}{N\left(x_{j+1}^i\right)}\right\}dt}\frac{{\displaystyle {\sum }_{k=1}^{j-1}{\tilde{F}}_{k,j}^i\left({\tilde{P}}_{k,j}^i/\left(1-{\widehat{q}}_j^i\right)\right)}}{{\displaystyle {\sum }_{j=1}^D{\displaystyle {\sum }_{k=j}^D{\tilde{F}}_{k,j}^i{\tilde{P}}_{k,j}^i}}},$$

and for $i\in {\mathrm{ℐ}}^c,$ let

$$y_j^i=\{\begin{array}{ll}1,\hfill & \text{if}\hspace{0.17em}{t}_{\text{new}}^i\ge x_{k+1}^i,\hfill \\ 0,\hfill & \text{otherwise}.\hfill \end{array}$$

Then,

$$E\left[{\Delta }_j^i|Y\right]=\{\begin{array}{ll}0,\hfill & \text{if }\hspace{0.17em}{\displaystyle {\sum }_{k=1}^j{I}^i\left(x_{k+1}^i\right)}=0\hspace{1em}\text{or}\hspace{1em}{y}_j^i=0,\hfill \\ x_{j+1}^i-x_j^i,\hfill & \text{if}\hspace{0.17em}{I}^i\left(x_{j+1}^i\right)=0,\hspace{1em}\text{and }\hspace{1em}{\displaystyle {\sum }_{k=1}^j{I}^i\left(x_{k+1}^i\right)>0},\hspace{1em}\text{and}\hspace{1em}{y}_j^i=1,\hfill \\ {\delta }_j^i,\hfill & \text{otherwise,}\hfill \end{array}$$

where

$${\delta }_j^i=\left(x_{j+1}^i-x_j^i\right)\left[\frac{{\displaystyle {\sum }_{k=1}^{j-1}{I}^i\left(x_{k+1}^i\right){\tilde{Q}}_k^i}{\displaystyle {\prod }_{l=k+1}^{D_i-1}{\left[{\widehat{q}}_l^i\right]}^{y_l^i}}}{{\displaystyle {\sum }_{k=1}^D{I}^i\left(x_{k+1}^i\right){\tilde{Q}}_k^i}{\displaystyle {\prod }_{l=k+1}^D{\left[{\widehat{q}}_l^i\right]}^{y_l^i}}}\right]$$ (A3)

$$+{\displaystyle {\int }_{x_j^i}^{x_{j+1}^i}\left(x_{j+1}^i-u\right)}\exp \left\{-\frac{\left(x_{j+1}^i-u\right)A^i\left(x_{j+1}\right)}{N\left(x_{j+1}^i\right)}\right\}du\left[\frac{I^i\left(x_{j+1}^i\right){\displaystyle {\prod }_{l=j+1}^D{\left[{\widehat{q}}_l^i\right]}^{y_l^i}}}{{\displaystyle {\sum }_{k=1}^D{I}^i\left(x_{k+1}^i\right){\tilde{Q}}_k^i}{\displaystyle {\prod }_{l=k+1}^D{\left[{\widehat{q}}_l^i\right]}^{y_l^i}}}\right].$$

M step

Now, for the kth iteration of the algorithm and maximizing the complete data log-likelihood (Equation 19), we have

$$N_l^k=\frac{{\displaystyle {\sum }_{j=1}^{D}0.5A^0\left(x_{j+1}^0\right)}\left[A^0\left(x_{j+1}^0\right)-1\right]\left(x_{j+1}^0-x_j^0\right)1_{l,j}^0+{\displaystyle {\sum }_{i=0}^{m-2}{\displaystyle {\sum }_{j=1}^D{A}^i\left(x_{j+1}^i\right)E_{N^{k-1}}\left[{\Delta }_j^i|Y\right]1_{l,j}^i}}}{{\displaystyle {\sum }_{j=1}^D{a}_j^0{1}_{l,j}^0}+{\displaystyle {\sum }_{i=0}^{m-2}{\displaystyle {\sum }_{j=1}^D{E}_{N^{k-1}}\left[z_j^i\right]1_{l,j}^i}}},$$

where

$$1_{l,j}^i=\{\begin{array}{ll}1,\hfill & \text{if}\hspace{0.17em}{x}_l<x_{j+1}^i\le x_{l+1},\hfill \\ 0,\hfill & \text{otherwise}\hfill \end{array}$$

is an indicator function that takes the value of 1 when $\left(x_l,x_{l+1}\right)$ covers the interval $\left(x_j^i,x_{j+1}^i\right)$.

Observed Score Function for Split Hamiltonian Monte Carlo

Our Bayesian approach relies on splitHMC sampling from the posterior distribution of model parameters. This method requires the calculation of the observed score function. We use Fisher’s identity and calculate the observed score function as the conditional expected complete score function. The lth element of $\nabla {\mathrm{ℒ}}_{\mathrm{obs}}$ is

$$\begin{array}{l}{\left(\nabla {\mathrm{ℒ}}_{\mathrm{obs}}\right)}_l=-{\displaystyle \sum _{j=1}^{D-1}{a}_j^0{1}_{l,j}^0}+\frac{1}{2}{\displaystyle \sum _{j=1}^{D-1}{A}^0\left(x_{j+1}^0\right)\left[A^0\left(x_{j+1}^0\right)-1\right]\left(x_{j+1}^0-x_j^0\right)1_{l,j}^0}\exp \left[-\log N_l\right]\\ -\hspace{0.17em}{\displaystyle \sum _{i=0}^{m-2}{\displaystyle \sum _{j=1}^{D-1}E\left[z_j^i|Y\right]1_{l,j}^0}}+{\displaystyle \sum _{i=0}^{m-2}{\displaystyle \sum _{j=1}^{D-1}{A}^i\left(x_{j+1}^i\right)E\left[{\Delta }_j^i|Y\right]1_{l,j}^i}}\exp \left[-\log N_l\right].\end{array}$$ (A4)

Sufficient Statistics Under SMC′

Here, we first formally show that under the standard coalescent and for a single locus, the set of coalescent times has sufficient statistics for inferring $N\left(t\right);$ that is, information about the topology is irrelevant for inference of $N\left(t\right)$. We then investigate the properties of our F quantities (see Methods: SMC′ Calculations) needed for the calculation of the transition densities (Equation 11) and show through a series of propositions that the F quantities and coalescent times are the sufficient statistics for inferring $N\left(t\right)$ under the SMC′ process.

Proposition 1. For a single locus, the set of coalescent times are sufficient statistics for inferring $N\left(t\right)$.

Proof. This can be proved using the factorization theorem. The marginal density of a local genealogy (Equation 3) has a unique factor that depends on $N\left(t\right)$ and g only through $t_n,\dots ,t_2$. The values of $A\left(t\right)$ are induced by the natural order of the coalescent times.

▪

Let F denote a lower triangular matrix of size $n\times n$ with the $F_{i,j}$ entry the number of lineages that do not coalesce in the time interval $\left(t_{i+1},t_j\right),$ as defined in Methods: SMC′ Calculations and with the following properties:

1. $F_{i,1}=0$ for all $i=1,\dots ,n$ (The first column contains 0’s for completion).

2. $F_{i,j}=0$ for all $j>i$ (lower triangular matrix).

3. $F_{i,i}=i$ for all $i\ge 2$ (the diagonal corresponds to the number of lineages at each intercoalescent interval).

4. $F_{i,i-1}=i-2$ for all $i\ge 2$ (at each intercoalescent interval, we lose two free lineages, so the second diagonal correspond to the number of lineages minus two).

5. For $j<n-1,$ the last row of F is defined according to

   $$F_{n,j-1}=\{\begin{array}{ll}{F}_{n,j}-2,\hfill & \text{with}\hspace{0.17em}\text{probability}\hspace{0.17em}p=\frac{\left(\begin{array}{c}{F}_{n,j}\\ 2\end{array}\right)}{\left(\begin{array}{c}j\\ 2\end{array}\right)},\hfill \\ F_{n,j}-1,\hfill & \text{with}\hspace{0.17em}\text{probability}\hspace{0.17em}p=\frac{F_{n,j}\left(j-F_{n,j}\right)}{\left(\begin{array}{c}j\\ 2\end{array}\right)},\hfill \\ F_{n,j},\hfill & \text{with}\hspace{0.17em}\text{probability}\hspace{0.17em}p=\frac{\left(\begin{array}{c}j-F_{n,j}\\ 2\end{array}\right)}{\left(\begin{array}{c}j\\ 2\end{array}\right)}.\hfill \end{array}$$
6. Let c denote the number of cherries; then

   $$c={\displaystyle \sum _{j=2}^n{1}_{\left\{F_{n,j}-F_{n,j-1}=2\right\}}}.$$

7. For $i<n$ and $j<i-1,$ if $F_{n,j-1}=F_{n,j}-2,$ then $F_{i,j-1}=F_{i,j}-2.$

8. Let $v_i$ denote the set of lineages in the intercoalescent interval $\left(t_i,t_{i-1}\right)$ with direct descendant internal nodes. The lineage labels correspond to the label of the coalescent time, when the direct descendant internal node was created. That is, the lineage created at $t_n$ has label n: $v_n=\left\{n\right\};$ the lineage created at $t_i$ has label i. Let $\left|v_i\right|$ denote the size of the set $v_i$. Note that $1\le \left|v_i\right|\le c$ and

   $$\left|v_i\right|={\displaystyle \sum _{j=i}^n{1}_{\left\{F_{n,j}-F_{n,j-1}=2\right\}}}-{\displaystyle \sum _{j=i}^n{1}_{\left\{F_{n,j}-F_{n,j-1}=0\right\}}}.$$
9. For $i<n$ and $j<i-1,$ if $F_{n,j-1}=F_{n,j}-1,$ then at time $t_j,$ there is a coalescence between a singleton and a lineage in the set $v_j$. Let $a_j$ be the lineage selected uniformly at random from $v_j;$ then

   $$F_{i,j-1}=\{\begin{array}{ll}{F}_{i,j}-1\hfill & \text{if}\hspace{0.17em}i>a_j\hfill \\ F_{i,j}-2\hfill & \text{if}\hspace{0.17em}j<i\le a_j.\hfill \end{array}$$
10. For $i<n$ and $j<i-1,$ if $F_{n,j-1}=F_{n,j},$ then at time $t_j,$ there is a coalescence between two lineages $a_j^1$ and $a_j^2$ from the set $v_j$. Let $a_j^1$ denote the minimum and $a_j^2$ the maximum of the two lineages selected; then

    $$F_{i,j-1}=\{\begin{array}{ll}{F}_{i,j}\hfill & \text{if}\hspace{0.17em}i>a_j^2\hfill \\ F_{i,j}-1\hfill & \text{if}\hspace{0.17em}{a}_j^1<i\le a_j^2\hfill \\ F_{i,j}-2\hfill & \text{if}\hspace{0.17em}j<i\le a_j^1.\hfill \end{array}$$

We show the correspondence between a ranked tree shape and the F matrix in the example of Figure A1. The first row and the first column are set to 0, and the first two diagonals are known with probability 1: $F_{i,i}=i$ and $F_{i,i-1}=F_{i,i}-2$ for $i>1.$ In our example, $n=5$ and so the first diagonal corresponds to $\left(0,2,3,4,5\right)$ and the second diagonal corresponds to $\left(0,1,2,3\right)$. The last row, $F_5,$ contains 0, followed by the number of branches that do not coalesce in the time intervals $\left(t_6,t_2\right),$ $\left(t_6,t_3\right),$ $\left(t_6,t_4\right),$ and $\left(t_6,t_5\right)$ corresponding to $\left(0,0,2,3,5\right)$.

Figure A1.

Ranked tree shape for $n=5.$

Proposition 2. There is a bijection between the set of ranked tree shapes ${\mathcal{G}}_n$ and $\mathrm{ℱ},$ the set of F matrices.

Proof. The probability of the F matrix can be expressed as the product of the conditional probabilities of the columns of the F matrix; that is,

$$\begin{array}{c}\mathrm{Pr}\left(F\right)=\mathrm{Pr}\left(F_{\cdot ,n}\right){\displaystyle \prod _{j=1}^{n-1}\mathrm{Pr}\left(F_{\cdot ,n-j}|F_{\cdot ,n-j+1}\right)}\\ ={\displaystyle \prod _{j=2}^{n-2}\mathrm{Pr}\left(F_{\cdot ,n-j}|F_{\cdot ,n-j+1}\right)},\end{array}$$

since the first and last column of F are known with probability 1. Note $F_{\cdot ,j}$ represents the jth column vector of the F matrix.

Let $d_i=F_{n,i}-F_{n,i-1}$ for $i=3,\dots ,n$ and $d_2=F_{n,2};$ then

$$\mathrm{Pr}\left(F_{\cdot ,n-j}|F_{\cdot ,n-j+1}\right)=\mathrm{Pr}\left(d_{n-j}|F_{\cdot ,n-j+1}\right)\mathrm{Pr}\left(F_{n-j:n-1,n-j}|d_{n-j},F_{\cdot ,n-j+1}\right).$$ (A5)

That is, the conditional probability of the $\left(n-j\right)$th column of F given the $\left(n-j+1\right)$th column of F is the product of the conditional probability of the last element of the $\left(n-j\right)$th column and the conditional probability of the rest of the $\left(n-j\right)$th column. When $d_{n-j}=2,$ the rest of the column is known with probability 1 (property 7 of the F matrix). When $d_{n-j}=1,$ the rest of the $n-j$th column has probability $1/\left|v_{n-j+1}\right|$ (property 9 of the F matrix) and when $d_{n-j}=2,$ the rest of the $n-j$th column has probability $1/\left(\begin{array}{c}\left|v_{n-j+1}\right|\\ 2\end{array}\right)$ (property 10 of the F matrix). Then rewriting Equation A5, we have

$$\mathrm{Pr}\left(F_{\cdot ,n-j}|F_{\cdot ,n-j+1}\right)=\mathrm{Pr}\left(d_{n-j+1}|F_{\cdot ,n-j+1}\right){\left(\frac{1}{\left|v_{n-j+1}\right|}\right)}^{1_{\left\{d_{n-j}=1\right\}}}{\left(\frac{1}{\left(\begin{array}{c}\left|v_{n-j+1}\right|\\ 2\end{array}\right)}\right)}^{1_{\left\{d_{n-j}=0\right\}}};$$ (A6)

since $\left|v_{n-j}\right|={\displaystyle {\sum }_{k=n-j}^n\left(1_{\left\{d_k=2\right\}}-1_{\left\{d_k=0\right\}}\right)}$ and $F_{n,k}=n-{\displaystyle {\sum }_{j=k+1}^n{d}_j},$ then

$$\mathrm{Pr}\left(d_{n-j+1}|F_{\cdot ,n-j+1}\right)=\mathrm{Pr}\left(d_{n-j+1}|F_{n,n-j+1}\right)=\mathrm{Pr}\left(d_{n-j+1}|{\displaystyle \sum _{k=n-j+2}^n{d}_k}\right),$$

and

$$\begin{array}{c}\mathrm{Pr}\left(F\right)={\displaystyle \prod _{j=1}^{n-2}\mathrm{Pr}\left(d_{n-j}|{\displaystyle \sum _{k=n-j+1}^n{d}_k}\right){\left(\frac{1}{\left|v_{n-j+1}\right|}\right)}^{1_{\left\{d_{n-j}=1\right\}}}{\left(\frac{2}{\left|v_{n-j+1}\right|\left(\left|v_{n-j+1}\right|-1\right)}\right(}^{1_{\left\{d_{n-j}=0\right\}}}}\\ ={\displaystyle \prod _{j=1}^{n-2}{\left(\frac{\left(n-{\displaystyle {\sum }_{k=n-j+1}^n{d}_k}\right)\left(n-{\displaystyle {\sum }_{k=n-j+1}^n{d}_k-1}\right)}{2}\right)}^{1_{\left\{d_{n-j}=2\right\}}}}\\ \times \hspace{0.17em}{\left(\frac{\left(n-{\displaystyle {\sum }_{k=n-j+1}^n{d}_k}\right)\left({\displaystyle {\sum }_{k=n-j+1}^n{d}_k-j}\right)}{\left|v_{n-j+1}\right|}\right)}^{1_{\left\{d_{n-j}=1\right\}}}\\ \times \hspace{0.17em}{\left(\frac{\left({\displaystyle {\sum }_{k=n-j+1}^n{d}_k-j}\right)\left({\displaystyle {\sum }_{k=n-j+1}^n{d}_k-j-1}\right)}{\left|v_{n-j+1}\right|\left(\left|v_{n-j+1}\right|-1\right)}\right)}^{1_{\left\{d_{n-j}=0\right\}}}\frac{1}{\left(\underset{2}{n-j}\right)}\\ =\frac{2^{n-2}{2}^{1-c}}{\left(n-1\right)!\left(n-2\right)!}{\displaystyle \prod _{j=2}^{n-1}{\left(\left(n-{\displaystyle \sum _{k=j+1}^n{d}_k}\right)\left(n-{\displaystyle \sum _{k=j+1}^n{d}_k-1}\right)\right)}^{1_{\left\{d_j=2\right\}}}}\\ \times \hspace{0.17em}{\left(\left(n-{\displaystyle \sum _{k=j+1}^n{d}_k})(j-n+{\displaystyle \sum _{k=j+1}^n{d}_k}\right)\right)}^{1_{\left\{d_j=1\right\}}}{\left(\left(j-n+{\displaystyle \sum _{k=j+1}^n{d}_k}\right)\left(j-n+{\displaystyle \sum _{k=j+1}^n{d}_k-1}\right)\right)}^{1_{\left\{d_j=0\right\}}}\\ \times \hspace{0.17em}{\left(\frac{1}{\left|v_{j+1}\right|}\right)}^{1_{\left\{d_j=0\right\}}+1_{\left\{d_j=1\right\}}}{\left(\frac{1}{\left|v_{j+1}\right|-1}\right)}^{1_{\left\{d_j=0\right\}}}.\end{array}$$

Since $d_n=2$ and $\left|v_n\right|=1,$ for $j=n-1,$ then $d_{n-1}$ is either 1 or 2; then

$$\begin{array}{c}\mathrm{Pr}\left(F\right)=\frac{2^{n-c-1}}{\left(n-1\right)!\left(n-2\right)!}\left(n-2\right){\left(n-3\right)}^{1_{\left\{d_{n-1}=2\right\}}}{\displaystyle \prod _{j=2}^{n-2}{\left(\left(n-{\displaystyle \sum _{k=j+1}^n{d}_k}\right)\left(n-{\displaystyle \sum _{k=j+1}^n{d}_k-1}\right)\right)}^{1_{\left\{d_j=2\right\}}}}\\ \times \hspace{0.17em}{\left(\left(n-{\displaystyle \sum _{k=j+1}^n{d}_k})(j-n+{\displaystyle \sum _{k=j+1}^n{d}_k}\right)\right)}^{1_{\left\{d_j=1\right\}}}{\left(\left(j-n+{\displaystyle \sum _{k=j+1}^n{d}_k})(j-n+{\displaystyle \sum _{k=j+1}^n{d}_k-1}\right)\right)}^{1_{\left\{d_j=0\right\}}}\\ \times \hspace{0.17em}{\left(\frac{1}{\left|v_{j+1}\right|}\right)}^{1_{\left\{d_j=0\right\}}+1_{\left\{d_j=1\right\}}}{\left(\frac{1}{\left|v_{j+1}\right|-1}\right)}^{1_{\left\{d_j=0\right\}}}.\end{array}$$

If we continue expanding the expressions, we get

$$\begin{array}{c}\mathrm{Pr}\left(F\right)=\frac{2^{n-c-1}}{\left(n-1\right)!\left(n-2\right)!}\left(n-2\right)\left(n-3\right){\left(n-4\right)}^{1_{\left\{d_{n-2}=2\right\}}+1_{\left\{d_{n-2}=1\right\}}{1}_{\left\{d_{n-1}=2\right\}}}{\left(n-5\right)}^{1_{\left\{d_{n-2}=2\right\}}{1}_{\left\{d_{n-1}=2\right\}}}\\ \times \hspace{0.17em}{\displaystyle \prod _{j=2}^{n-3}{\left(\left(n-{\displaystyle \sum _{k=j+1}^n{d}_k}\right)\left(n-{\displaystyle \sum _{k=j+1}^n{d}_k-1}\right)\right)}^{1_{\left\{d_j=2\right\}}}}\\ \times \hspace{0.17em}{\left(\left(n-{\displaystyle \sum _{k=j+1}^n{d}_k}\right)\left(j-n+{\displaystyle \sum _{k=j+1}^n{d}_k}\right)\right)}^{1_{\left\{d_j=1\right\}}}{\left(\left(j-n+{\displaystyle \sum _{k=j+1}^n{d}_k}\right)\left(j-n+{\displaystyle \sum _{k=j+1}^n{d}_k-1}\right)\right)}^{1_{\left\{d_j=0\right\}}}\\ \times \hspace{0.17em}{\left(\frac{1}{\left|v_{j+1}\right||}\right)}^{1_{\left\{d_j=0\right\}}+1_{\left\{d_j=1\right\}}}{\left(\frac{1}{\left|v_{j+1}\right|-1}\right)}^{1_{\left\{d_j=0\right\}}}\\ =\cdots \\ =\frac{2^{n-c-1}}{\left(n-1\right)!}.\end{array}$$

▪

Note that the entries of the F matrix correspond to the same quantities needed to express the transition density of an invisible event (Equation 11). We claim that the sequence of coalescent times sets $t^0,t^1,\dots ,t^{m-1}$ and $F^0,F^1,\dots ,F^{m-1}$ matrices corresponding to the ranked tree shapes of local genealogies $g_0,g_1,\dots ,g_{m-1}$ are sufficient statistics to infer $N\left(t\right)$ under the SMC′ process. We prove this through Propositions 3–6.

Proposition 3. The probability density of Tajima’s genealogy is proportional, up to a combinatorial factor, to the probability density of Kingman’s genealogy.

Proof.

$$\mathrm{Pr}\left[G^T=\left\{F,t_n,t_{n-1},\dots ,t_2\right\}|N\left(t\right)\right]=\mathrm{Pr}\left[t_n,t_{n-1},\dots ,t_2|N\left(t\right)\right]\mathrm{Pr}\left[F|t_n,t_{n-1},\dots ,t_2\right]$$

$$=\frac{n!\left(n-1\right)!}{2^{n-1}}\mathrm{Pr}\left[G=\left\{K_n,t_n,\dots ,t_2\right\}|N\left(t\right)\right]\frac{2^{n-c-1}}{\left(n-1\right)!}$$

$$=\frac{n!}{2^c}{\displaystyle \prod _{j=2}^n\frac{1}{N\left(t_j^0\right)}}\exp \left\{-{\displaystyle {\int }_{t_{j+1}^0}^{t_j^0}\frac{A^0\left(t\right)\left(A^0\left(t\right)-1\right)dt}{2N\left(t\right)}}\right\}.$$ (A7)

▪

Proposition 4. The marginal visible transition density from a local Kingman’s genealogy $g_{i-1}$ to $G_i$ is proportional to the marginal visible transition density from the corresponding local Tajima’s genealogy $g_{i-1}^T$ to $G_i^T$.

Proof. When the labeled topology of $g_{i-1}$ is the same as the labeled topology of $g_i,$ then a transition from $g_{i-1}$ to $g_i$ contains the same information about pruning location as a transition from $g_{i-1}^T$ to $g_i^T$ (Figure S1A in File S1 and Figure S2D in File S4). In fact, the $I^{i-1}\left(t\right)$ function defined in the Visible transitions subsection (Equation 8) can be defined in terms of the $F^i$ matrix and the coalescent times $t^{i-1}$ and $t^i$. In this case, for some $j\in \left\{2,\dots ,n\right\},$ $t_j^{i-1}=t_{\mathrm{del}}^i$ and $t_j^i=t_{\text{new}}^i$. Then

$$I^{i-1}\left(t\right)=\{\begin{array}{ll}0,\hfill & \text{if}\hspace{0.17em}t>\min \left(t_{\text{new}}^i,t_{\mathrm{del}}^i\right),\hfill \\ F_{l,j}^{i-1}-F_{l,j-1}^{i-1},\hfill & \text{if}\hspace{0.17em}t\in \left(t_{l+1}^{i-1},t_l^{i-1}\right)\hspace{1em}\text{for}\hspace{1em}l=j,j+1,\dots ,n.\hfill \end{array}$$

Hence, if $K_{i-1}=K_i,$ the labeled topologies of $g_{i-1}$ and $g_i,$ then

$$\mathrm{Pr}\left[G_i=\left\{K^{i-1},t^i\right\}|g_{i-1}=\left\{K^{i-1},t^{i-1}\right\},N\left(t\right)\right]=\mathrm{Pr}\left[G_i^T=\left\{F^{i-1},t^i\right\}|g_{i-1}^T=\left\{F^{i-1},t^{i-1}\right\},N\left(t\right)\right].$$

When the labeled topologies of $g_{i-1}$ and $g_i$ are different, but the children of $t_{\mathrm{del}}^i$ and the children of $t_{\text{new}}^i$ are the same, we cannot exactly identify the pruning branch and the new coalescing branch (Figure S1B in File S1) and then a transition from $g_{i-1}$ to $g_i$ contains the same information about pruning location as a transition from $g_{i-1}^T$ to $g_i^T$. Let $t_j^{i-1}=t_{\mathrm{del}}^i$ and $t_k^i=t_{\text{new}}^i;$ since the children of $t_j^{i-1}$ and $t_k^i$ are the same, it is enough to consider $F^{i-1}$. Then

$$I^{i-1}\left(t\right)=\{\begin{array}{ll}0,\hfill & \mathrm{if}\hspace{0.17em}t>\min \left(t_{\text{new}}^i,t_{\mathrm{del}}^i\right),\hfill \\ F_{l,j}^{i-1}-F_{l,j-1}^{i-1},\hfill & \mathrm{if}\hspace{0.17em}t\in \left(t_{l+1}^{i-1},t_l^{i-1}\right) \hspace{1em}\mathrm{for}\hspace{1em}l=j,j+1,\dots ,n\hfill \end{array}$$

and

$$\mathrm{Pr}\left[G_i=\left\{K^i,t^i\right\}|g_{i-1}=\left\{K^{i-1},t^{i-1}\right\},N\left(t\right)\right]=\mathrm{Pr}\left[G_i^T=\left\{F^{i-1},t^i\right\}|g_{i-1}^T=\left\{F^{i-1},t^{i-1}\right\},N\left(t\right)\right].$$

When the deleted node corresponding to $t_{\mathrm{del}}$ is a cherry and the new node corresponding to $t_{\text{new}}$ is also a cherry, there are four possible topologies $K_i$ that lead to the same ranked tree shape $F^i;$ then

$$\begin{array}{c}\mathrm{Pr}\left[G_i=g_i|g_{i-1},N\left(t\right)\right]={\left(\frac{1}{2}\right)}^{1_{\left\{t_j^{i-1}=t_{\mathrm{del}}^i\right\}}{1}_{\left\{F_{n,j}^{i-1}=F_{n,j+1}^{i-1}-2\right\}}}\times {\left(\frac{1}{2}\right)}^{1_{\left\{t_j^i=t_{\text{new}}^i\right\}}{1}_{\left\{F_{n,j}^i=F_{n,j+1}^i-2\right\}}}\\ \times \hspace{0.17em}\mathrm{Pr}\left[G_i^T=g_i^T|g_{i-1}^T,N\left(t\right)\right].\end{array}$$

▪

Proposition 5. The marginal invisible transition density from a local Kingman’s genealogy $g_{i-1}$ to $G_i$ is equal to the marginal invisible transition density from the corresponding local Tajima’s genealogy $g_{i-1}^T$ to $G_i^T$.

Proof.

$$\mathrm{Pr}\left[G_i=g_{i_1}|g_{i-1},N\left(t\right)\right]=\mathrm{Pr}\left[G_i=g_{i-1}|g_{i-1}^T,N\left(t\right)\right],$$

since all that is needed to compute the transition probability are the coalescent times and the $F^{i-1}$ matrix. Since the topology does not change, the proof follows.

▪

Proposition 6. The likelihood of partially observed embedded SMC ′ chain of local Kingman’s genealogies is proportional, up to a combinatorial factor, to the likelihood of partially observed embedded SMC ′ chain of the corresponding local Tajima’s genealogies.

Proof. The proof follows from Propositions 3–5 needed to express the likelihood of partially observed embedded SMC′ chain (Equation 13).

▪