Complexity of the simplest species tree problem

Abstract

The multispecies coalescent model provides a natural framework for species tree estimation accounting for gene-tree conflicts. Although a number of species tree methods under the multispecies coalescent have been suggested and evaluated using simulation, their statistical properties remain poorly understood. Here, we use mathematical analysis aided by computer simulation to examine the identifiability, consistency, and efficiency of different species tree methods in the case of three species and three sequences under the molecular clock. We consider four major species-tree methods including concatenation, two-step, independent-sites maximum likelihood, and maximum likelihood. We develop approximations that predict that the probit transform of the species tree estimation error decreases linearly with the square root of the number of loci. Even in this simplest case, major differences exist among the methods. Full-likelihood methods are considerably more efficient than summary methods such as concatenation and two-step. They also provide estimates of important parameters such as species divergence times and ancestral population sizes,whereas these parameters are not identifiable by summary methods. Our results highlight the need to improve the statistical efficiency of summary methods and the computational efficiency of full likelihood methods of species tree estimation.

Introduction

The multispecies coalescent (MSC) model (Rannala and Yang 2003) combines the phylogenetic process of species divergences with the population genetic process of coalescent and naturally accommodates “delayed coalescence” (also known as “incomplete lineage sorting,” Maddison 1997), the phenomenon in which gene sequences fail to coalesce in their most recent common ancestor but do so only in more ancient ancestors. Delayed coalescence causes the gene tree for a gene or genomic region to differ from the species tree and is the most important factor for gene-tree–species-tree discordance (Maddison 1997; Nichols 2001; Szöllősi et al. 2015). The MSC provides a natural framework for estimating species trees accounting for genealogical heterogeneity among genes or across the genome (Edwards 2009; Xu and Yang 2016; Kubatko 2019; Rannala et al. 2020).

Two lines of research into the MSC have provided the foundation for species tree methods. The first concerns the probabilities of different gene tree topologies (Hudson 1983; Pamilo and Nei 1988) and algorithms for their efficient calculation given the species tree (Degnan and Salter 2005; Degnan and Rosenberg 2006). The gene tree distribution can be used in the two-step method of species tree estimation, by inferring gene trees for the individual loci and then applying maximum likelihood (ML) to counts of gene tree topologies (as in stells,Wu 2012). Nevertheless, widely used two-step methods, including astral (Mirarab et al. 2014) and mp-est (Liu et al. 2010), are simpler, and estimate species trees for species triplets (assuming the molecular clock) or quartets (without the clock) and then assemble the subtrees to produce a species-tree estimate for all species. Studies of gene-tree probabilities led to the discovery of the “anomaly zone,” the region of the parameter space in which the most probable gene tree has a different topology from the species tree (Degnan and Salter 2005; Degnan and Rosenberg 2006). In the anomaly zone, the two-step method, which uses the most common gene tree as the species tree estimate, will be inconsistent.

The second line of research into MSC is the development of the joint probability distribution of the gene tree and coalescent times (Rannala and Yang 2003). This forms the basis for exact methods of inference, including ML (Yang 2002; Dalquen et al. 2017) and Bayesian methods (Liu and Pearl 2007; Heled and Drummond 2010; Yang and Rannala 2014; Ogilvie et al. 2017; Rannala and Yang 2017). Although heuristic methods use summaries of the data, exact methods use the multilocus sequence alignments directly and naturally accommodate phylogenetic reconstruction errors and uncertainties (Xu and Yang 2016; Kubatko 2019; Rannala et al. 2020).

Simulation has been used to examine the performance of different species-tree methods (e.g., Leaché and Rannala 2011; Mirarab et al. 2014; Chou et al. 2015; Xu and Yang 2016). A limitation of simulation is that it can examine only a small portion of the parameter space and the results often have limited applicability. Analytical results on the efficiency of different methods have been lacking. Here, we analyze species tree estimation under the MSC in the case of three species, with one sequence from each species per locus. We focus on closely related species and assume the JC mutation model (Jukes and Cantor 1969) and the molecular clock. We are in particular interested in the efficiency of the various methods, measured by the probability of recovering the correct species tree.

We consider four inference methods: 1) ML (a full likelihood method under the MSC applied to the multilocus sequence alignments), 2) 2-step (or majority-vote), 3) concatenation (concat), and 4) independent-sites ML (isml, also known as coalescent-aware concatenation or concat) (Xu and Yang 2016). ML is the full-likelihood method and calculates the likelihood function using the multilocus sequence alignments or a sufficient summary. The 2-step method estimates the gene tree at each locus and then uses the most common gene tree as the species tree estimate. It does not account for the uncertainties in the estimated gene trees. For the case of three species considered here, 2-step is equivalent to the maximum pseudolikelihood method (mp-est) (Liu et al. 2010). Concatenation applies ML to the concatenated sequences, assuming that the same tree underlies all sites in the super alignment. In the case considered here, concatenation is equivalent to steac (Liu et al. 2009), which uses average coalescent times over loci as data to infer a gene tree, which is the species tree estimate. Isml (or concat) estimates the species tree by ML under the assumption that all sites, both from the same locus and from different loci, have independent gene trees (Xu and Yang 2016). This was suggested as an improvement to SVDQuartets of Chifman and Kubatko (2014). All four methods considered here use ML, but the likelihood function is applied to different summaries of the same data. Here, we refer to the full-likelihood or full-data method as the ML method, whereas all other methods (2-step, concatenation, and isml) are considered heuristic summary methods: 2-step uses the (estimated) gene tree topologies, whereas concatenation and isml use the site-pattern counts pooled across loci. We derive approximations to the error rate of species tree estimation by the different methods and assess their accuracy. We use the theory to characterize the differences in the use of information in the data by different methods.

Results

Multispecies Coalescent in the Case of Three Species

For three species A, B, and C, there are three possible species trees: $S_1=((AB)C)$, $S_2=((BC)A)$, and $S_3=((CA)B)$, each with two divergence times (τ₀ and τ₁) and two population sizes (θ₀ and θ₁) (fig. 1a). Both τs and θs are measured by the expected number of mutations per site. For each species, the population size parameter is $\theta =4N\mu$, where N is the (effective) population size and μ is the mutation rate per site per generation. We consider only one sequence from each species, so that θs for the modern species are not considered. The parameters have different interpretations in different species trees: in S₁, the two ancestral species are AB and ABC so the parameters are ${\mathit{\theta }}_1$ =  $\{{\tau }_0,{\tau }_1,{\theta }_0,{\theta }_1\}$  =  $\{{\tau }_{\mathit{ABC}},{\tau }_{AB},{\theta }_{\mathit{ABC}},{\theta }_{AB}\}$.

Fig. 1.

(a) The three species trees ($S_1,S_2,S_3$) for three species ($A,B,C$) and the parameters in each MSC model. (b) The possible gene trees with coalescent times (t₀, t₁) for a locus with three sequences (a, b, c) given the species tree S₁. The probabilities for the gene trees are shown above them, where $\varphi$ = ${\mathrm{e}}^{-\frac{2}{{\theta }_1}({\tau }_0-{\tau }_1)}$ is the probability that a andb do not coalesce in population AB or over the time interval (τ₁, τ₀). Note that if the species tree is S₂ (or S₃), it will be possible for sequences b and c (or c and a) to coalesce in the time interval (τ₁, τ₀).

At each locus, three sequences (a, b, and c) are sampled, one from each species. They are related through a gene tree. The three possible gene trees are $G_1=((ab)c),\hspace{0.17em}{G}_2=((bc)a)$, and $G_3=((ca)b)$, with probabilities:

$$\begin{array}{cc}\mathbb{P}(G_1|S_1,{\mathit{\theta }}_1)\hfill & =1-\frac{2}{3}\varphi ,\hfill \\ \mathbb{P}(G_2|S_1,{\mathit{\theta }}_1)\hfill & =\mathbb{P}(G_3|S_1,{\mathit{\theta }}_1)=\frac{1}{3}\varphi ,\hfill \end{array}$$ (1)

where $\varphi$ =  ${\mathrm{e}}^{-2({\tau }_{\mathit{ABC}}-{\tau }_{AB})/{\theta }_{AB}}$ is the probability that sequences a and b do not coalesce in population AB so that all three sequences enter the ancestor ABC and the three gene trees occur with equal probability (fig. 1b) (Hudson 1983). Here, $2({\tau }_{\mathit{ABC}}-{\tau }_{AB})/{\theta }_{AB}$ is known as the internal branch length in coalescent units, as the average coalescent time in population AB is $2N_{AB}$ generations or ${\theta }_{AB}/2$ mutations per site.

For locus i, let ${\mathit{t}}_i=\{t_{i0},t_{i1}\}$ be the coalescent times (node ages) on the gene tree (fig. 1b). The joint MSC density for the gene tree and coalescent times given species tree S₁ and parameters ${\mathit{\theta }}_1$ is then:

$$\begin{array}{cc}f(G_{1a},{\mathit{t}}_i|S_1,{\mathit{\theta }}_1)\hfill & =\frac{2}{{\theta }_1}{\mathrm{e}}^{-\frac{2}{{\theta }_1}(t_{i1}-{\tau }_1)}·\frac{2}{{\theta }_0}{\mathrm{e}}^{-\frac{2}{{\theta }_0}(t_{i0}-{\tau }_0)},\hfill \\ \hfill & \hspace{1em}\hspace{1em}{\tau }_1<t_{i1}<{\tau }_0,t_{i0}>{\tau }_0,\hfill \\ f(G_k,{\mathit{t}}_i|S_1,{\mathit{\theta }}_1)\hfill & ={\mathrm{e}}^{-\frac{2}{{\theta }_1}({\tau }_0-{\tau }_1)}\hfill \\ \hfill & \times \frac{2}{{\theta }_0}\frac{2}{{\theta }_0}{\mathrm{e}}^{-\frac{6}{{\theta }_0}(t_{i1}-{\tau }_0)-\frac{2}{{\theta }_0}(t_{i0}-t_{i1})},\hfill \\ \hfill & \hspace{1em}\hspace{1em}{t}_{i1}>{\tau }_0,t_{i0}>t_{i1},\hfill \end{array}$$ (2)

for $k=1b,2,3$ (Takahata et al. 1995; Yang 2002). The probability densities for S₂ and S₃ are given similarly.

The data consist of sequence alignments at m loci. Under the JC mutation model, the data at locus i can be summarized as counts of five site patterns: xxx, xxy, yxx, xyx, and xyz, where x, y, z are any three distinct nucleotides. Let those counts be ${\mathit{x}}_i=\{x_{i0},x_{i1},x_{i2},x_{i3},x_{i4}\}$, with ${\sum }_{j=0}^4{x}_{ij}=n$ to be the number of sites (sequence length) at each locus. Let $f_{ij}=x_{ij}/n$ be the frequencies. Let data at all m loci be $\mathit{x}=\left\{{\mathit{x}}_i\right\}$.

Given the gene tree and coalescent times at locus i, the probability of the sequence data, $f({\mathit{x}}_i|G_i,{\mathit{t}}_i)$, is given by the multinomial distribution for the five site patterns. For example, given gene tree G₁ with node ages $t_{i0}$ and $t_{i1}$ (fig. 1b), the site-pattern probabilities, ${\mathit{p}}_i=\{p_{i0},p_{i1},p_{i2},p_{i3},p_{i4}\}$, are as follows:

$$\begin{array}{c}{p}_{i0}=\mathbb{P}(\mathit{xxx}|G_1,{\mathit{t}}_i)=\frac{1}{16}(1+3v^2+6u+6uv),\\ p_{i1}=\mathbb{P}(\mathit{xxy}|G_1,{\mathit{t}}_i)=\frac{1}{16}(3+9v^2-6u-6uv),\\ p_{i2}=\mathbb{P}(\mathit{yxx}|G_1,{\mathit{t}}_i)=\frac{1}{16}(3-3v^2+6u-6uv),\\ p_{i3}=\mathbb{P}(\mathit{xyx}|G_1,{\mathit{t}}_i)=p_2,\\ p_{i4}=\mathbb{P}(\mathit{xyz}|G_1,{\mathit{t}}_i)=\frac{1}{16}(6-6v^2-12u+12uv),\end{array}$$ (3)

where $u={\mathrm{e}}^{-8t_{i0}/3}$ and $v={\mathrm{e}}^{-4t_{i1}/3}$ (Yang 1994b). Note that $p_{i1}>p_{i2}=p_{i3}$ as $t_{i0}>t_{i1}$. The probabilities for gene trees G₂ or G₃ are given by symmetry. Then the sequence data or the five site-pattern counts at the locus have the multinomial probabilities:

$$\begin{array}{c}f({\mathit{x}}_i|G_1,{\mathit{t}}_i)=p_{i0}^{x_{i0}}{p}_{i1}^{x_{i1}}{p}_{i2}^{x_{i2}+x_{i3}}{p}_{i4}^{x_{i4}},\\ f({\mathit{x}}_i|G_2,{\mathit{t}}_i)=p_{i0}^{x_{i0}}{p}_{i1}^{x_{i2}}{p}_{i2}^{x_{i3}+x_{i1}}{p}_{i4}^{x_{i4}},\\ f({\mathit{x}}_i|G_3,{\mathit{t}}_i)=p_{i0}^{x_{i0}}{p}_{i1}^{x_{i3}}{p}_{i2}^{x_{i1}+x_{i2}}{p}_{i4}^{x_{i4}}.\end{array}$$ (4)

The ML Method of Species Tree Estimation

The log-likelihood function for species tree S₁ with parameters ${\mathit{\theta }}_1$ is given by summing over the gene trees and integrating over the coalescent times.

$${\ell }_1({\mathit{\theta }}_1)=\sum _{i=1}^m\hspace{0.17em}\text{log}\hspace{0.17em}f({\mathit{x}}_i|S_1,{\mathit{\theta }}_1)=\sum _{i=1}^m\hspace{0.17em}\text{log}\hspace{0.17em}\left\{\sum _{G_i}\int f(G_i,{\mathit{t}}_i|S_1,{\mathit{\theta }}_1)f({\mathit{x}}_i|G_i,{\mathit{t}}_i)\mathrm{d}{\mathit{t}}_i\right\},$$ (5)

where $f(G_i,{\mathit{t}}_i|S_1,{\mathit{\theta }}_1)$ is the MSC density for the gene tree and coalescent times at locus i (eq. 2), and $f({\mathit{x}}_i|G_i,{\mathit{t}}_i)$ is the probability of the sequence data at locus i given the gene tree (eq. 4). The log likelihood functions, ${\ell }_2({\mathit{\theta }}_2)$and ${\ell }_3({\mathit{\theta }}_3)$, for S₂ (with parameters ${\mathit{\theta }}_2$) and S₃ (with ${\mathit{\theta }}_3$) are defined similarly.

Maximizing the log-likelihood function (eq. 5) with respect to the parameters will lead to a log-likelihood value for the given species tree, and the species tree that achieves the highest $\ell$ is the ML species tree. This is not analytically tractable. The program 3 s implements the method by explicitly summing over the gene trees (G_i) and by using Gaussian quadrature to calculate the 2D integrals over ${\mathit{t}}_i$ (eq. 5) (Yang 2002; Zhu and Yang 2012; Dalquen et al. 2017). This is used in simulations.

We present two theorems for approximating the error in species tree estimation.

Theorem 1

.(a) Suppose  $z_i={(z_{i1},z_{i2},z_{i3})}^T,\hspace{0.17em}i=1,\dots ,m$, are an independent and identically distributed (i.i.d.) sample of size m from a distribution with means  $\mu ={({\mu }_1,{\mu }_2,{\mu }_2)}^T$, with  $\Delta \mu ={\mu }_1-{\mu }_2>0$, and variances  $\Sigma =\left\{{\sigma }_{jk}\right\}$, where  ${\sigma }_{11}={\sigma }_1^2,\hspace{0.17em}{\sigma }_{12}={\sigma }_{13}={\rho }_{12}{\sigma }_1{\sigma }_2,\hspace{0.17em}{\sigma }_{22}={\sigma }_{33}={\sigma }_2^2$and  ${\sigma }_{23}={\rho }_{23}{\sigma }_2^2$. Let  $\overline{z}={\{{\overline{z}}_1,{\overline{z}}_2,{\overline{z}}_3\}}^T$be the sample means, with  ${\overline{z}}_j=\frac{1}{m}\sum _{i=1}^m{z}_{ij},\hspace{0.17em}j=$1, 2, 3. For large m, $\overline{z}\sim {\mathbb{N}}_3(\mu ,\frac{1}{m}\Sigma )$. Let  $\zeta =\mathbb{P}\{{\overline{z}}_1<\max ({\overline{z}}_2,{\overline{z}}_3)\}$. Then

$$\begin{array}{cc}\zeta \hfill & \approx \Phi \left(\frac{-\Delta \mu \sqrt{m}+\sqrt{\frac{1}{\pi }({\sigma }_2^2-{\sigma }_{23})}}{\sqrt{{\sigma }_1^2-2{\sigma }_{12}+{\sigma }_2^2-\frac{1}{\pi }({\sigma }_2^2-{\sigma }_{23})}}\right)\hfill \\ \hfill & \equiv {\zeta }_N(m,\Delta \mu ,{\sigma }_1^2,{\sigma }_2^2,{\sigma }_{12},{\sigma }_{23}),\hfill \end{array}$$ (6)

where $\Phi$is the cumulative distribution function (CDF) for the normal distribution  $\mathbb{N}(0,1)$. We also write ζ_N as  ${\zeta }_N(m,\Delta \mu ,\Sigma )$.

(b) Let  $a=s_2/s_1$and  $b=\Delta \mu \sqrt{m}/s_1$, with  $s_1^2={\sigma }_1^2-2{\sigma }_{12}+{\sigma }_2^2-\frac{1}{2}({\sigma }_2^2-{\sigma }_{23})$and  $s_2^2=\frac{1}{2}({\sigma }_2^2-{\sigma }_{23})$. Then ζ is bounded by:

$$\begin{array}{cc}\Phi (-h)(1+\frac{2}{\pi }{\hspace{0.17em}\text{tan}\hspace{0.17em}}^{-1}a)\hfill & \le \zeta <2\Phi (-h)\hfill \\ \hfill & =\Phi (-h+\frac{1}{h}\text{log}\hspace{0.17em}2+o(\frac{1}{h})),\hfill \end{array}$$ (7)

where  $h=\frac{b}{\sqrt{1+a^2}}$. The equality for the lower bound holds when  $h=b=0$. We write those bounds as  ${\zeta }_{L1}\zeta {\zeta }_{U1}$, so that  $\Phi (-h)\le {\zeta }_{L1}\le \zeta <2\Phi (-h)$.

Proof.

A proof is given in Appendix A, in which, we discuss alternative approximations and also give a tighter pair of bounds $({\zeta }_{L2},{\zeta }_{U2})$ in equation (A27), with ${\zeta }_{L1}<{\zeta }_{L2}<\zeta <{\zeta }_{U2}<{\zeta }_{U1}$. □

In this paper, ζ represents the error probability of species tree estimation. Thus, the bounds $\Phi (-h)\le \zeta <2\Phi (-h)$ suggest that when $m\to \infty$, the probit transform of the species-tree error probability, ${\Phi }^{-1}(\zeta )$, where ${\Phi }^{-1}$ is the inverse CDF of $\mathbb{N}(0,1)$, decreases linearly with $\sqrt{m}$. For practical calculations for finite m in this paper, equation (6) is more accurate (see Appendix A) and will be used later.

Corollary 2.Let ( $y_0,y_1,y_2,y_3$) be random variables from the multinomial distribution MN(m, q₀, q₁, q₂, q₃), with  $q_0=1-q_1-q_2-q_3,\hspace{0.17em}{q}_1>q_2$= q₃, and  $\Delta q=q_1-q_2>0$. Then  $\mathbb{P}\{y_1<\max (y_2,y_3)\}$can be approximated by:

$$\zeta (m,q_1,q_2)=\Phi \left(\frac{-\Delta q\sqrt{m}+\sqrt{\frac{q_2}{\pi }}}{\sqrt{q_1+q_2-{(\Delta q)}^2-\frac{q_2}{\pi }}}\right),$$ (8)

$${\zeta }_{\text{ZLY}}(m,q_1,q_2)=\Phi \left(\frac{-\Delta q\sqrt{m-\frac{1}{\Delta q}}+\sqrt{\frac{q_2}{\pi }}}{\sqrt{q_1+q_2-\frac{q_2}{\pi }}}\right).$$ (9)

Proof.

Let ${\overline{z}}_j=y_j/m,j=1,2,3$ be the observed frequencies. We have ${\sigma }_{jj}=q_j(1-q_j)$ and ${\sigma }_{jk}=-q_j{q}_k$ for $j\ne k$. Then equation (8) follows from equation (6) in Theorem 1. The form ${\zeta }_{\text{ZLY}}$, an alternative to equation (8), is from Yang (1996, eq. 3), based on Zharkikh and Li(1992, eq. 20). This applies the term $1/\Delta q$ to correct for discontinuity (Fleiss et al. 2003) and ignores correlations between y₁, y₂, and y₃ as well as some terms of small probabilities. The discontinuity correction does not appear to be useful. If $m\gg 1/\Delta q$, both forms, with and without the discontinuity correction, are very close. □

The error rate for the ML method (eq. 5) is analyzed in Appendix B. When the number of loci $m\to \infty$, the MLE ${\widehat{\mathit{\theta }}}_j\to {\mathit{\theta }}_j^{*}$ in species tree S_j, j = 1, 2, 3. Note that S₁ represents the true model and ${\mathit{\theta }}_1^{*}$ are the true parameter values, while S₂ and S₃ are misspecified models and ${\mathit{\theta }}_2^{*}$ and ${\mathit{\theta }}_3^{*}$ are the “best-fitting or pseudotrue parameter values.” The Kullback–Leibler distance D₁₂ from S₂ to S₁ is:

$$\begin{array}{c}{D}_{12}=\int f(x|S_1,{\mathit{\theta }}_1^{*})\hspace{0.17em}\text{log}\hspace{0.17em}\frac{f(x|S_1,{\mathit{\theta }}_1^{*})}{f(x|S_2,{\mathit{\theta }}_2^{*})}\mathrm{dx}\hfill \\ \hfill & =\mathbb{E}(l_1({\mathit{\theta }}_1^{*}))-\mathbb{E}(l_2({\mathit{\theta }}_2^{*})),\end{array}$$ (10)

where $l_j({\mathit{\theta }}_j^{*})\equiv \hspace{0.17em}\text{log}\hspace{0.17em}f(x|S_j,{\mathit{\theta }}_j^{*})$, with x to be one data point (or site pattern counts at one locus), and where the integral means summation over all possible data outcomes at a locus. We use the per-locus log-likelihood values to compare the three species trees: ${\overline{z}}_j\equiv \frac{1}{m}{\ell }_j({\widehat{\mathit{\theta }}}_j)$, j  == 1, 2, 3. When m is large, these have the means $\mathbb{E}({\overline{z}}_j)\approx \mathbb{E}(l_j({\mathit{\theta }}_j^{*}))\equiv {\mu }_j$, with ${\mu }_1-{\mu }_2=D_{12}$, and the variance matrix $\frac{1}{m}\Sigma$, where $\Sigma =\left\{{\sigma }_{jk}\right\}$ and ${\sigma }_{jk}\equiv \text{Cov}(l_j({\mathit{\theta }}_j^{*}),l_k({\mathit{\theta }}_k^{*}))$. The error of the ML method, $e_{ML}=\mathbb{P}\{{\ell }_1({\widehat{\mathit{\theta }}}_1)<\max ({\ell }_2({\widehat{\mathit{\theta }}}_2),{\ell }_3({\widehat{\mathit{\theta }}}_3))\}$, is then given by Theorem 1 as:

$$e_{ML}=\mathbb{P}\{{\overline{z}}_1<\max ({\overline{z}}_2,{\overline{z}}_3)\}\approx {\zeta }_N(m,D_{12},\Sigma ).$$ (11)

Equation (11) cannot be used to calculate the error rate for ML as D₁₂ and σ_jk are not easily computable. It predicts a linear relationship between ${\Phi }^{-1}(e_{\text{ML}})$ and $\sqrt{m}$. This is confirmed by simulation (fig. 2a′–c′).

Precise results may be obtained in special cases. In the case of one locus (m = 1), the ML gene tree is the ML species tree except for rare data sets: the true species tree S₁ is recovered if $x_{i1}>\max (x_{i2},x_{i3})$. In rare data sets of extreme divergence, even if $x_{i1}>\max (x_{i2},x_{i3})$, ties for gene trees are possible, with the star tree being as good as the binary trees (Yang 2000), whereas ML under MSC favors S₁. One such data set is $x_i=(4,13,12,11,50)$, in which case the three gene trees as well as the star tree achieve the same likelihood, whereas ML under MSC favors S₁. However, such data sets involve sequences more divergent than random sequences have vanishingly small probability when n is large. Thus, we ignore them and consider all methods to be equivalent when m = 1. With one locus, it is impossible to identify all parameters in the MSC model: there are four parameters and only three independent site-pattern frequencies ( $f_{i0},f_{i1},f_{i2}+f_{i3}$ for S₁, for example).

The case of one site per locus (n = 1) is analyzed later in the section on isml. Numerical calculations on a model species tree are presented in table 1. They will be discussed later in comparison with other methods.

In the case of $n\to \infty$, the gene tree (including the coalescent times) at each locus is given without errors. The likelihood is then the product of MSC densities of gene trees across the loci (eq. 2). This likelihood has singularities, with one or more species trees achieving infinite likelihood (Liu et al. 2010; Yang 2014). In the case of three species considered here, only one species tree (given by the smallest coalescent time) achieves infinite likelihood and will be the unique species-tree estimate, so that the estimation can proceed despite the singularity (Yang 2014, p. 360, Problem 9.4). Let the smallest coalescent/divergence time between species across all loci be t_ab, t_bc, and t_ca. If t_ab is the smallest among the three, then species tree S₁ achieves infinite likelihood, by collapsing on the coalescent time t_ab; that is, ${\ell }_1({\widehat{\mathit{\theta }}}_1)\to \infty$ as ${\widehat{\tau }}_0={\widehat{\tau }}_1=t_{ab}$ and ${\widehat{\theta }}_1\to 0$ (see eq. 2) (Yang 2014, p.338–339), whereas the other two species trees have finite likelihood.

Given S₁ as the true species tree, both t_bc and t_ca are $>{\tau }_{\mathit{ABC}}$ (fig. 1b). If sequences a and b coalesce in population AB at any of the m loci, t_ab will be smaller than both t_bc and t_ca, and S₁ will be the ML species tree. Thus, an incorrect species tree is inferred only if a and b do not coalesce in AB at any of the m loci and are not the first to coalesce in the root population ABC. Thus,

$$e_{ML,\infty }={\varphi }^m\times \frac{2}{3},$$ (12)

where $\varphi ={\mathrm{e}}^{-\frac{2}{{\theta }_{AB}}({\tau }_{ABC}-{\tau }_{AB})}$ is the probability that a and b do not coalesce in population AB. This equation is exact and applies to both small and large m (fig. 3b).

Fig. 2.

(a–c) Species-tree estimation error (e) at three sequence lengths (n = 1, 2, 1,000) plotted against the number of loci (m) for different methods. (a$\prime$–c$\prime$) The probit transform of the species-tree error, ${\Phi }^{-1}(e)$, plotted against $\sqrt{m}$. The parameters used in the simulation are ${\tau }_0=0.02,\hspace{0.17em}{\tau }_1=0.019,\hspace{0.17em}{\theta }_0=0.01$, and ${\theta }_1=0.05$. When n = 1, all four methods (ML, 2-step, concatenation, and isml) give the same species tree estimate, while concatenation and isml are equivalent in all cases considered in this paper. The number of replicates is $R\ge {10}^4$ for ML and $\ge {10}^6$ for the other methods.

Table 1.

Probabilities ($g_1,g_2,g_3$) of Estimated Gene Trees at Different Sequence Lengths (n) and the Error Rates for the Summary Methods 2-step and isml with m = 1,000 Loci, Each with n Sites.

n	1	2	10	100	1,000	$\infty$
2-step (mp-est)
$\mathbb{P}$(tie)	0.92948	0.8673	0.57015	0.22159	0.05105	0
$g_1(n)$	0.02378	0.04474	0.14515	0.26646	0.33273	0.35947
$g_2(n)=g_3(n)$	0.02337	0.04398	0.14235	0.25598	0.30811	0.32026
$e_{2-\mathit{STEP}}$	0.642	0.633	0.597	0.470	0.260	0.114
$\zeta (m,g_1,g_2)$	0.644	0.635	0.600	0.472	0.264	0.113
${\zeta }_{\text{ZLY}}(m,g_1,g_2)$	NA	NA	0.613	0.482	0.271	0.117
$({\zeta }_{L1},{\zeta }_{U1})$	(0.635, 0.953)	(0.623, 0.935)	(0.578, 0.869)	(0.430, 0.647)	(0.219, 0.331)	(0.087, 0.132)
$({\zeta }_{L2},{\zeta }_{U2})$	(0.637, 0.729)	(0.626, 0.714)	(0.585, 0.668)	(0.446, 0.561)	(0.242, 0.328)	(0.103, 0.132)
ζ (mean2)	0.683	0.670	0.627	0.504	0.285	0.118
a	0.574051	0.574056	0.573612	0.569708	0.562911	0.555962
b	0.0678913	0.0930368	0.190376	0.527747	1.11658	1.72268
isml (concat)
$e_{\mathit{ISML}}$	0.642	0.632	0.590	0.438	0.246	0.196
ζN	0.644	0.634	0.592	0.443	0.254	0.194
${\zeta }_{N0}$	0.643	0.633	0.591	0.437	0.234	0.166
$({\zeta }_{L1},{\zeta }_{U1})$	(0.635, 0.953)	(0.622, 0.934)	(0.568, 0.854)	(0.397, 0.598)	(0.211, 0.318)	(0.157, 0.237)
$({\zeta }_{L2},{\zeta }_{U2})$	(0.637, 0.728)	(0.625, 0.713)	(0.576, 0.659)	(0.416, 0.536)	(0.233, 0.316)	(0.177, 0.237)
ζ (mean2)	0.683	0.669	0.618	0.476	0.275	0.207
a	0.574029	0.573971	0.57356	0.569747	0.558232	0.553151
b	0.067892	0.0958963	0.21228	0.607057	1.14253	1.35035

Note.—$\mathbb{P}(\text{tie})$ is the probability for ties in gene trees, with $\mathbb{P}(\text{tie})+g_1+2g_2=$ 1. The probabilities of estimated gene trees ($g_1,g_2,g_3$) as well as the error rates ($e_{2-\mathit{STEP}}$ and $e_{\text{ISML}}$) are estimated by simulation using a C program, with $\ge {10}^6$ replicates. Ties are broken evenly in the error calculation. The parameter values used are $({\tau }_0,{\tau }_1,{\theta }_0,{\theta }_1)$=(0.02, 0.019, 0.01, 0.05). The marginal (pooled) site pattern probabilities are $\overline{\mathit{p}}=$ (${\overline{p}}_0,\hspace{0.17em}{\overline{p}}_1,\hspace{0.17em}{\overline{p}}_2,\hspace{0.17em}{\overline{p}}_3,\hspace{0.17em}{\overline{p}}_4$) = (0.92831926, 0.023777106, 0.023372801, 0.023372801, 0.001158033), given by equation (13). For 2-step, at n = 1, the estimated gene tree is determined by the single site so that $g_1(1)={\overline{p}}_1$and $g_2(1)={\overline{p}}_2$, whereas at $n=\infty$, the estimated gene tree is the true gene tree, so that $g_1(\infty )=\mathbb{P}(G_1)$ and $g_2(\infty )=\mathbb{P}(G_2)$ (eq. 1). For 2-step, ${\zeta }_{\text{ZLY}}$ (eq. 9) is inapplicable at n = 1 or 2 as m = 1000 is too small. For isml, ${\zeta }_{N0}={\zeta }_N(m,\Delta \mu ,{\sigma }_1^2,{\sigma }_2^2,0,0)$ ignores the correlation (eq. 6), while ζ_N accounts for the correlation. The bounds $({\zeta }_{L1},{\zeta }_{U1})$ and $({\zeta }_{L2},{\zeta }_{U2})$ are calculated using equations (7) and (A27), with k = 2 used in ${\zeta }_{U2}$. “mean2” is the average of the tight bounds: $({\zeta }_{L2}+{\zeta }_{U2})/2$.

Fig. 3.

Error rates in species-tree estimation by ML, 2-step, and isml (=concatenation). (a) Error plotted against sequence length n when the number of loci m is fixed at 100 or 1,000, generated by simulation. (b) Error plotted against m when $n=\infty$. Error for ML is given by equation (12), whereas those for isml and 2-step are generated by simulation. (c) Error plotted against n when $nm={10}^4$ is fixed, generated by simulation. Note that all four methods are equivalent when n = 1 or $m=1$, while concatenation and isml are equivalent in all cases. Parameters used in the simulation are ${\tau }_0=0.02,\hspace{0.17em}{\tau }_1=0.019,\hspace{0.17em}{\theta }_0=0.01$, and ${\theta }_1=0.05$. The number of replicates is $R\ge {10}^4$.

Concatenation

Sequence alignments at the m loci are merged into a super-alignment of length nm, and the data are the site-pattern counts pooled across loci: $\mathit{x}.=\left\{x_{·j}\right\}$, with $x_{·j}=\sum _i{x}_{ij},\hspace{0.17em}j=0,1,\dots ,4$. The likelihood function is given by the multinomial probability of equation (4) except that $x_{·j}$ is used instead of x_ij. The ML tree is G₁ if $x_{·1}>\max (x_{·2},x_{·3})$ (Yang 1994b, 2000). We discuss the error rate of concatenation below in the section on the isml method.

We also examine biases in parameter estimation using concatenation. We use species tree S₁ with τ_ABC = 0.02, τ_AB = 0.01, θ_ABC = 0.02, and θ_AB = 0.01 to simulate $m={10}^4$ loci each with n = 250 sites. We obtain MLEs ${\widehat{t}}_0$ and ${\widehat{t}}_1$ on gene tree G₁ from the concatenated data for comparison with the MLEs ${\widehat{\tau }}_0$ and ${\widehat{\tau }}_1$ on species tree S₁ in the MSC model (eq. 5). With so much data, both concatenation and ML recover the true tree with near certainty. The MLEs under the MSC (obtained using the 3 sprogram) are very close to the true values, whereas concatenation (baseml in paml, Yang 2007) produced seriously biased estimates (table 2). Even the relative age, ${\widehat{t}}_0/{\widehat{t}}_1$ = 1.92, differs from ${\tau }_{\mathit{ABC}}/{\tau }_{AB}=2$, which means that molecular clock dating analysis using concatenated data will produce biased time estimates (Angelis and dos Reis 2015; Ogilvie et al. 2017; Tiley et al. 2020).

Table 2.

Estimates of Divergence Times (true values in parentheses) by ML under the MSC (3 s) and by Concatenation (baseml) in Two Simulated Data Sets, Each of $m={10}^4$ Loci and n = 250 Sites.

	τABC	τAB	θABC	θAB
Data/method	(0.02)	(0.01)	(0.02)	(0.01)
Data set 1, 3s	0.0201	0.0096	0.0199	0.0101
Data set 2, 3s	0.0196	0.0100	0.0201	0.0100
Data set 1, baseml	0.0298	0.0155
Data set 2, baseml	0.0298	0.0156

ISML

The isml method assumes that all sites in the super-alignment are i.i.d. Like concatenation, the data are summarized as pooled site-pattern counts, $\mathit{x}.=\{x_{·0},x_{·1},x_{·2},x_{·3},x_{·4}\}$. However, isml is coalescent-aware and uses the MSC model to calculate the probabilities for the site patterns. By averaging the conditional site-pattern probabilities of equation (3) over the MSC density of gene trees and coalescent times of equation (2), we derive the marginal site-pattern probabilities, $\overline{\mathit{p}}=({\overline{p}}_0,\dots ,{\overline{p}}_4)$, as:

$$\begin{array}{c}{\overline{p}}_0=\frac{1}{16}\left(1+18a_0+54a_{0}b+54a_0{c}_0+9c_1+9a_1\right),\\ {\overline{p}}_1=\frac{3}{16}\left(1-6a_0-18a_{0}b-18a_0{c}_0+9c_1+9a_1\right),\\ {\overline{p}}_2=\frac{3}{16}\left(1+6a_0-18a_{0}b-18a_0{c}_0-3c_1-3a_1\right),\\ {\overline{p}}_3={\overline{p}}_2,\\ {\overline{p}}_4=\frac{6}{16}\left(1-6a_0+18a_{0}b+18a_0{c}_0-3c_1-3a_1\right),\end{array}$$ (13)

where $a_0=\frac{{\mathrm{e}}^{-8{\tau }_0/3}}{3+4{\theta }_0}$, $a_1=\frac{{\mathrm{e}}^{-8{\tau }_1/3}}{3+4{\theta }_1}$, $b=\frac{{\mathrm{e}}^{-4{\tau }_1/3}}{3+2{\theta }_1}$, $c_0=2\varphi ·({\theta }_1-{\theta }_0)·\frac{{\mathrm{e}}^{-4{\tau }_0/3}}{(3+2{\theta }_0)(3+2{\theta }_1)}$, and $c_1=4\varphi ·({\theta }_1-{\theta }_0)·\frac{a_0}{3+4{\theta }_1}$, with $\varphi ={\mathrm{e}}^{-2({\tau }_0-{\tau }_1)/{\theta }_1}$. Note that $\left\{{\overline{p}}_j\right\}$ are functions of a₀, $b+c_0$ and $a_1+c_1$, although these do not appear to permit simple biological interpretations. The cases for S₂ and S₃ are given by symmetry.

The likelihood function (or the probability for the pooled site-pattern counts) for each species tree is:

$$\begin{array}{c}f(\mathit{x}.|S_1,{\mathit{\theta }}_1)={\overline{p}}_0^{x_{·0}}{\overline{p}}_1^{x_{·1}}{\overline{p}}_2^{x_{·2}+x_{·3}}{\overline{p}}_4^{x_{·4}},\\ f(\mathit{x}.|S_2,{\mathit{\theta }}_2)={\overline{p}}_0^{x_{·0}}{\overline{p}}_1^{x_{·2}}{\overline{p}}_2^{x_{·3}+x_{·1}}{\overline{p}}_4^{x_{·4}},\\ f(\mathit{x}.|S_3,{\mathit{\theta }}_3)={\overline{p}}_0^{x_{·0}}{\overline{p}}_1^{x_{·3}}{\overline{p}}_2^{x_{·1}+x_{·2}}{\overline{p}}_4^{x_{·4}}.\end{array}$$ (14)

Theorem 3

.(a) If the true species tree is S₁ with parameters  ${\mathit{\theta }}_1$, then  ${\overline{p}}_1>{\overline{p}}_2={\overline{p}}_3$. (b) Isml infers the species tree S₁ if  $x_{·1}>\max \{x_{·2},x_{·3}\}$.

Proof.

(a) Each of the marginal site pattern probabilities ${\overline{p}}_j,\hspace{0.17em}j=0,\dots ,4$, is a sum over the four gene trees of figure 1b:$G_{1a},G_{1b},G_2$ and G₃. The three gene trees $G_{1b},G_2$, and G₃ have the same densities (eq. 2). Together their contribution to the site pattern xxy is the same as that to the pattern yxx or pattern xyx. If the gene tree is $G_{1a}$ (with any coalescent times t₀ > t₁), site pattern xxy will have a higher probability than yxx or xyx, with $p_1>p_2=p_3$. Averaging over all the four gene trees, we have ${\overline{p}}_1>{\overline{p}}_2={\overline{p}}_3$.

(b) We show that if $x_{·1}>x_{·2}$, then $\ell (S_1,{\widehat{\mathit{\theta }}}_1)>\ell (S_2,{\widehat{\mathit{\theta }}}_2)$, where ${\widehat{\mathit{\theta }}}_1$ and ${\widehat{\mathit{\theta }}}_2$ are the MLEs under each species tree. First note that if $x_{·1}>x_{·2}$ and $q_1>q_2>0$, then $q_1^{x_{·1}}{q}_2^{x_{·2}}>q_1^{x_{·2}}{q}_2^{x_{·1}}$. Let $q_1={\overline{p}}_1(S_1,{\widehat{\mathit{\theta }}}_2)$and $q_2={\overline{p}}_2(S_1,{\widehat{\mathit{\theta }}}_2)$, and we have $\ell (S_1,{\widehat{\mathit{\theta }}}_2)>\ell (S_2,{\widehat{\mathit{\theta }}}_2)$. In other words, even if we use ${\widehat{\mathit{\theta }}}_2$ (the MLE for S₂) to calculate the likelihood for species tree S₁, tree S₁ will have a higher likelihood than S₂. Since ${\widehat{\mathit{\theta }}}_2$ may not be optimal for S₁, it follows that $\ell (S_1,{\widehat{\mathit{\theta }}}_1)\ge \ell (S_1,{\widehat{\mathit{\theta }}}_2)>\ell (S_2,{\widehat{\mathit{\theta }}}_2)$. □

Theorem 3 means that isml infers species tree S_j if $x_{·j}$ is the greatest among $x_{·1},\hspace{0.17em}{x}_{·2}$, and $x_{·3}$, just like concatenation.

To study the error rate for isml (or concat), let p_ij, $j=0,\dots ,4$ be the site-pattern probabilities at any locus i. Data at each locus are represented by the site-pattern frequencies $f_{ij}=x_{ij}/n$. Let $f_i=\left\{f_{ij}\right\}$ be the data at locus i. The f_i are i.i.d. among loci from a common distribution with mean $\mathbb{E}(f_{ij})={\overline{p}}_j$ and variance/covariance ${\sigma }_{jj}\equiv \mathbb{V}(f_{ij})$ and ${\sigma }_{jk}\equiv \text{Cov}(f_{ij},f_{ik})$. Let ${\overline{f}}_j=\frac{1}{m}\sum _{i=1}^m{f}_{ij}=x_{·j}/m$ be the means over loci. Here, $\left\{f_{ij}\right\}$ constitute the full data, whereas $\left\{{\overline{f}}_j\right\}$ are summaries used by isml: the species tree estimate is S_j if ${\overline{f}}_j$ is the largest among $({\overline{f}}_1,{\overline{f}}_2,{\overline{f}}_3)$. Thus, $e_{\mathit{ISML}}=\mathbb{P}({\overline{f}}_1<\max \{{\overline{f}}_2,{\overline{f}}_3\})\approx {\zeta }_N(m,{\overline{p}}_1-{\overline{p}}_2,\Sigma )$, where $\Sigma =\left\{{\sigma }_{jk}\right\}$. Below we derive the variances.

At n = 1, they are given by the multinomial distribution as:

$${\sigma }_{jj}^{(1)}={\overline{p}}_j(1-{\overline{p}}_j),\hspace{0.17em}{\sigma }_{jk}^{(1)}=-{\overline{p}}_j{\overline{p}}_k,\hspace{1em}1\le j,k\le 3.$$ (15)

At $n=\infty$, we have f_ij = p_ij, given by equation (3). The variances, denoted ${\sigma }_{jk}^{(\infty )}$, can be generated by simulating gene trees with coalescent times and calculating the site-pattern probabilities (eq. 3) (supplementary table S1, Supplementary Material online). This distribution is 3D (for $f_{i0}$, $f_{i1}$, and $f_{i2}=f_{i3}$ under S₁), indexed by four parameters (${\mathit{\theta }}_1$ in S₁), and is a mixture distribution with 4 components corresponding to the four gene trees of figure 1b. It reflects the coalescent fluctuation in gene genealogies.

For any finite $1\le n<\infty$, the variances are given by:

$$\begin{array}{c}{\sigma }_{jj}=\mathbb{V}(\mathbb{E}(f_{ij}|p_{ij}))+\mathbb{E}(\mathbb{V}(f_{ij}|p_{ij}))\\ =\mathbb{V}(p_{ij})+\mathbb{E}(p_{ij}(1-p_{ij})/n)\\ =\mathbb{V}(p_{ij})+\frac{1}{n}[\mathbb{E}(p_{ij})(1-\mathbb{E}(p_{ij}))-\mathbb{V}(p_{ij})]\\ =\frac{1}{n}{\sigma }_{jj}^{(1)}+\frac{n-1}{n}{\sigma }_{jj}^{(\infty )},\\ {\sigma }_{jk}=\text{Cov}(p_{ij},p_{ik})+\mathbb{E}(\text{Cov}(f_{ij},f_{ik}|p_{ij},p_{ik}))\\ =\text{Cov}(p_{ij},p_{ik})+\frac{1}{n}[-\mathbb{E}(p_{ij})\mathbb{E}(p_{ik})-\text{Cov}(p_{ij},p_{ik})]\\ =\frac{1}{n}{\sigma }_{jk}^{(1)}+\frac{n-1}{n}{\sigma }_{jk}^{(\infty )},\end{array}$$ (16)

where $\mathbb{E}(p_{ij})\equiv {\overline{p}}_j$ (eq. 13), whereas $\mathbb{V}(p_{ij})={\sigma }_{jj}^{\infty }$ and $\text{Cov}(p_{ij},p_{ik})={\sigma }_{jk}^{\infty }$ are the variances/covariances over the coalescent process. These are calculated for a set of parameter values in supplementary table S1, Supplementary Material online. The variances of f_ij are thus weighted averages of variances at $n=1$ and $\infty$.

The approximation $e_{\mathit{ISML}}\approx {\zeta }_N(m,{\overline{p}}_1-{\overline{p}}_2,\Sigma )$ is very accurate, with errors <0.002 in the simulation of table 1. At large n, accommodating correlation is useful as ${\zeta }_{N0}$ which ignores correlation is less accurate (see fig. 4 for the case of $n=\infty$). For example, the correlation $\rho (f_{i1},f_{i2})=\frac{{\sigma }_{12}}{\sqrt{{\sigma }_{11}{\sigma }_{22}}}$is $-0.124,-0.153$, and −0.181 at n = 1, 1,000, and $\infty$, respectively (supplementary table S1, Supplementary Material online).

We now consider parameter estimation by isml. Theorem 3 allows species tree estimation by isml without knowledge of the MLE of the parameters. With data of $x{.}_j,j=0,\dots ,4$, there are only three observations (three free proportions ${\overline{f}}_0,\hspace{0.17em}{\overline{f}}_1$, and ${\overline{f}}_2+{\overline{f}}_3$ in the case of S₁). As there are four parameters in the MSC model, it is impossible to identify all of them.

If we assume ${\theta }_0={\theta }_1=\theta$ (as in Tian and Kubatko 2016), all three parameters (${\tau }_0,{\tau }_1,\theta$) will be identifiable. As $c_0=c_1=0$, equation (13) simplifies to:

$$\begin{array}{cc}{\overline{p}}_0\hfill & =\frac{1}{16}\left(1+18a_0+54a_{0}b+9a_1\right),\hfill \\ {\overline{p}}_1\hfill & =\frac{3}{16}\left(1-6a_0-18a_{0}b+9a_1\right),\hfill \\ {\overline{p}}_2\hfill & =\frac{3}{16}\left(1+6a_0-18a_{0}b-3a_1\right)={\overline{p}}_3,\hfill \\ {\overline{p}}_4\hfill & =\frac{6}{16}\left(1-6a_0+18a_{0}b-3a_1\right),\hfill \end{array}$$ (17)

where a₀, a₁, and b are defined in equation (13) with ${\theta }_0={\theta }_1=\theta$. By equating the observed site-pattern frequencies to their expected probabilities (eq. 17), we have

$$\begin{array}{c}\frac{1}{4}(9a_1+1)={\overline{f}}_0+{\overline{f}}_1\equiv h_1,\\ \frac{1}{4}(9a_0+1)={\overline{f}}_0+\frac{1}{2}({\overline{f}}_2+{\overline{f}}_3)\equiv h_2,\\ \frac{3}{8}(-18a_{0}b+3a_1+1)={\overline{f}}_1+\frac{1}{2}({\overline{f}}_2+{\overline{f}}_3)\equiv h_3.\end{array}$$ (18)

Thus, we have a quadratic equation in $\widehat{\theta }$:

$$\begin{array}{c}4{(4h_3-2h_1-1)}^2{\widehat{\theta }}^2+[3{(4h_3-2h_1-1)}^2\\ -(4h_1-1){(4h_2-1)}^2](4\widehat{\theta }+3)=0.\end{array}$$ (19)

This always has a unique positive root. Given $\widehat{\theta }$, the estimates ${\widehat{\tau }}_0$ and ${\widehat{\tau }}_1$ are given by equation (18), which are guaranteed to be positive.

Fig. 4.

Species tree error for isml at $n=\infty$ generated by simulation (10⁸ replicates) and by approximation based on ζ_N either with or without accounting for correlations. The error goes from 0.64 (at m = 1) to 0.19 (at m = 1,000). Results for other methods for the same parameter settings are in figure 3b.

Thus, under the assumption ${\theta }_0={\theta }_1$, the isml method provides estimates of the three parameters in the model: θ, τ₀, and τ₁. As there is a one-to-one correspondence between the parameters and the multinomial proportions, the estimates are consistent and approach the true values when $m\to \infty$ for any $n\ge 1$ if the assumption of ${\theta }_0={\theta }_1$ is correct (table 3, cases c and d). However, the pooled site-pattern counts or average site-pattern frequencies are summaries of the original data and are not sufficient statistics. It then follows that the isml estimates will be less efficient and have larger asymptotic variances than the MLEs obtained from the full data under the same model assumption of ${\theta }_0={\theta }_1$ (table 3, case c). Furthermore, if ${\theta }_0\ne {\theta }_1$, assuming ${\theta }_0={\theta }_1$ will lead to biased and inconsistent parameter estimates even if the same species tree estimate is produced. In other words if ${\theta }_0\ne {\theta }_1$, the isml method assuming ${\theta }_0={\theta }_1$ will produce a consistent estimate of the species tree and inconsistent estimates of the model parameters (table 3, cases e and f).

Table 3.

Characterization of the isml Method.

	True Model	Assumption	Data Size	Parameters	isml vs. ml
(a)	${\theta }_0\ne {\theta }_1$	${\theta }_0\ne {\theta }_1$	n > 1	3 out of 4 identifiable	isml $\ne$ ml
(b)	${\theta }_0\ne {\theta }_1$	${\theta }_0\ne {\theta }_1$	n = 1	3 out of 4 identifiable	isml $=$ ml
(c)	${\theta }_0={\theta }_1$	${\theta }_0={\theta }_1$	n > 1	all 3 identifiable	isml $\ne$ ml
(d)	${\theta }_0={\theta }_1$	${\theta }_0={\theta }_1$	n = 1	all 3 identifiable	isml $=$ ml
(e)	${\theta }_0\ne {\theta }_1$	${\theta }_0={\theta }_1$	n > 1	3 out of 4 identifiable, inconsistent	isml $\ne$ ml
(f)	${\theta }_0\ne {\theta }_1$	${\theta }_0={\theta }_1$	n = 1	3 out of 4 identifiable, inconsistent	isml $=$ ml

Note.—In all cases, the species tree topology is identifiable and consistently estimated by isml when the number of loci $m\to \infty$. If the parameters are identifiable, their estimates will be consistent. When isml differs from ML and the assumed model is correct, isml is less efficient than ML for parameter estimation (case c).

Two-Step Method (Majority Vote)

In the 2-step method, we estimate gene trees at individual loci and then use the most common gene tree topology as the species tree estimate. Under JC, the ML gene tree for locus i (which is also the upgma tree) is tree G_j if x_ij is the largest among $x_{i1},x_{i2}$, and $x_{i3}$ (Yang 1994b, 2000); site patterns xxy, yxx, and xyx “support” gene trees G₁, G₂, and G₃, respectively. There is no need for numerical optimization to obtain the ML tree at each locus.

Let g₁, g₂, and g₃ be the probabilities that the estimated gene tree is G₁, G₂, and G₃, respectively; that is, $g_1=\mathbb{P}\{x_{i1}>\max (x_{i2},x_{i3})\}$, and so on. These are functions of all four parameters in the MSC model (${\mathit{\theta }}_1$) as well as the sequence length n, and can be computed numerically (Yang 2002, eq. 12) or by simulation. Under JC and the clock, $g_2=g_3<g_1<\mathbb{P}(G_1|S_1,{\mathit{\theta }}_1)$ (Yang 2002). This result has several implications. First, $g_1<\mathbb{P}(G_1)$ means that phylogenetic errors inflate gene-tree–species-tree discordance and lead to underestimation of the internal branch length in the species tree (Yang 2002). Second $g_1<\mathbb{P}(G_1)$ also means that use of estimated (rather than true) gene trees leads to reduced probability for recovering the correct species tree. Third, $g_1>g_2=g_3$ means that the 2-step estimate of the species tree is consistent even if estimated gene trees are used.

Let the number of loci at which G₁ is the ML tree be $m_1=\sum _{i=1}^m{\mathbb{I}}_{x_{i1}>\max (x_{i2}, x_{i3})}$, where the indicator function ${\mathbb{I}}_a=1$ if statement a is true and 0 otherwise. Similarly define m₂ and m₃ to be the counts for the two mismatching gene trees. The correct species tree is inferred if and only if $m_1>\max (m_2,m_3)$. Thus, the error rate can be approximated by $e_{2-\mathrm{STEP}}\approx \zeta (m,g_1,g_2)$ (eq. 8).

The accuracy of this approximation is assessed in table 1 at different values of n with m = 1,000 and with parameter values ${\tau }_0=0.02,\hspace{0.17em}{\tau }_1=0.019,\hspace{0.17em}{\theta }_0=0.01$, and ${\theta }_1=0.05$. Consider first the case of n = 1. The gene tree is resolved if the single site at the locus has site patterns 1, 2, or 3, but is unresolved if the site has patterns 0 or 4. Whether we ignore loci with ties (with site patterns 0 or 4) or break ties evenly (assigning $\frac{1}{3}$ to each gene tree) does not affect the species tree estimate. Thus, $g_1(1)={\overline{p}}_1$ and $g_2(1)={\overline{p}}_2$ (eq.13) and the error is $e_{2-\mathrm{STEP}}\approx \zeta (m,{\overline{p}}_1,{\overline{p}}_2)$. This is equivalent to $e_{\mathit{ISML}}\approx {\zeta }_N(m,{\overline{p}}_1-{\overline{p}}_2,\Sigma )$ for isml, consistent with the fact that at n = 1 all methods considered here are equivalent.

If $n=\infty$, the estimated gene trees will be the true gene trees so that $g_1=\mathbb{P}(G_1)$ and $g_2=\mathbb{P}(G_2)$. The error rate is then $\zeta (m,\mathbb{P}(G_1),\mathbb{P}(G_2))=$ζ(1,000, 0.3594737, 0.3202631) = 0.1132, close to 0.114 from simulation. At $n=1000$, the proportions of estimated gene trees are g₁ = 0.33273 and g₂ = 0.30811, so that $\zeta (m,g_1,g_2)=$ 0.264, close to 0.260 by simulation (table 1). These are much larger than 0.114 at $n=\infty$, suggesting that with n = 1,000 sites in the sequence, the estimated gene trees have substantial errors and uncertainties.

The approximations ${\zeta }_{\text{ZLY}}$ (eq. 9) and ζ (eq. 8) give nearly identical results. The error rate is found to be very sensitive to the precise values of g₁ and g₂. Overall, the approximation is good, with errors within or close to 1%.

Numerical Comparison of Different Methods

We use simulation to compare the different species-tree estimation methods and to assess the reliability of our approximations. We use a challenging species tree with parameters ${\tau }_0=0.02,\hspace{0.17em}{\tau }_1=0.019,\hspace{0.17em}{\theta }_0=0.01$, and ${\theta }_1=0.05$. The error is plotted against the number of loci (m) when the number of sites per locus is fixed at $n=$ 1, 2, or 1,000 (fig. 2).

In the case of one site per sequence (n = 1), all four methods considered in this study are equivalent, with the species tree given by the most frequent pooled site pattern (i.e., the greatest of $x_{·1},\hspace{0.17em}{x}_{·2}$, and $x_{·3}$). With one site, the independent-sites assumption is correct, and ml and isml are exactly the same. As discussed earlier, concatenation and 2-step also select the species tree according to the pooled site patterns. Treatment of ties among $x_{·1},x_{·2},x_{·3}$ has very minor effects on the error rate. For n = 1 and $m=1000$, simulation gave the error estimate $e=$ 0.642 if ties are broken evenly (table 1) or 0.641 if data sets with ties are ignored. As predicted by our theory, the probit transform of the error, ${\Phi }^{-1}(e)$, shows a linear relationship with $\sqrt{m}$ (fig. 2a′, $R^2=0.9994$).

In the case of n = 2 sites per locus, isml (=concatenation), 2-step, and ML are all distinct. To see that concatenation and 2-step may produce different species trees, consider the case of m = 3 loci and n = 2 sites. If the data set at the three loci are 11, 02, and 00, where 0–4 represent the five site patterns, concatenation will infer the correct species tree S₁ (as $x_{·1}=2,x_{·2}=1,x_{·3}=0$), whereas 2-step will have a tie between S₁ and S₂ (as $m_1=1,m_2=1,m_3=0$). If the data set at the three loci are 33, 01, and 14, concatenation will have a tie between S₁ and S₃ (as $x_{·1}=2,x_{·2}=0,x_{·3}=2$), whereas 2-step will infer the correct species tree (as $m_1=2,m_2=0,m_3=1$). We also confirm that at n = 2 ML differs from all three summary methods and can identify and consistently estimate all four parameters in the MSC model. Indeed ML is far more efficient for species tree estimation than the summary methods when n = 2 (fig. 2b and b′). Although the summary methods improve only slightly when n changes from 1 to 2, there is a major performance boost for ML (fig. 3a). This may be due to the fact that the model is fully identifiable with n = 2 but not when n = 1. The predicted linear relationship between ${\Phi }^{-1}(e)$ and $\sqrt{m}$ holds well for the three summary methods (fig. 2b′). For ML, if we remove the first two points (for m = 10 and 20), the relationship is nearly linear, with $y=-0.0022x+0.0391$, with $R^2=0.97$.

The most interesting case is with $n\gg 1$, since in real data sets n may be in the range 50–5,000, say. We used n = 1,000 in figure 2c and c′. As in the case of n = 2, there is a large performance divide between ML and the three summary methods (Isml = concat and 2-step), whereas the summary methods have similar performance. The approximate linear relationship between ${\Phi }^{-1}(e)$ and $\sqrt{m}$ holds well for all methods.

The superior performance of ML persists in the limit of $n=\infty$ (fig. 3b). For example, $e_{\text{ML},\infty }=$ 0.45 and 0.01 for ML at m = 10 and 100, respectively, compared with $e_{2-\mathrm{STEP},\infty }=$ 0.60 and 0.46 for 2-step or $e_{\mathrm{ISML},\infty }=$ 0.62 and 0.51 for isml. The differences between ML and 2-step reflect the information in the coalescent times or gene-tree branch lengths. The differences between ML and isml reflect the information in the variation of site-pattern frequencies among loci, as isml uses only the averages across loci.

Figure 3c examines the error rates of different methods, while $nm={10}^4$ is fixed. At the two ends (n = 1 or m = 1), all four methods are equivalent, with $e=$ 0.587 at n = 1 and $m={10}^4$, and $e=$ 0.646 at m = 1 and $n={10}^4$. Note that when $n=1$ and $m\to \infty$, the error $e\to 0$, while if m = 1 and $n\to \infty$, the error $e=1-g_1(n)\to 1-\mathbb{P}(G_1)=$ 0.6405. The high error at m = 1 even when $n=\infty$ is because a single gene tree (with coalescent times), even if known with certainty, does not contain much information about the MSC process. Away from the two ends (n > 1 or m > 1), ML is considerably more efficient than the summary methods (fig. 3c). The case of $m={10}^4$ (n = 1), at which $e_{\text{ML}}=$ 0.587, and the case of m = 2 (n = 5,000), at which $e_{\text{ML}}$ = 0.487, make an interesting contrast. In the first case all sites are i.i.d., while in the second, there are only two independent genes, each of 5,000 sites in complete linkage. One might expect data of independent sites to be more informative than two loci with correlated sites at the same locus (e.g., Long and Kubatko 2018), but the opposite is true. With n = 1, not all model parameters are identifiable, and this nonidentifiability issue appears to impact species tree estimation as well (Shi and Yang 2018, p. 172). With nm fixed, the smallest error $e_{\text{ML}}$ occurs at intermediate values of n and m, around $n=m=100$, although performance is similar over a large range of n (fig. 3c).

In table 1, we calculated the species-tree error probability using equations (6) and (8), as well as two pairs of bounds (${\zeta }_{L1},{\zeta }_{U1}$) and (${\zeta }_{L2},{\zeta }_{U2}$) (Theorem 1, Appendix A), for comparison with the simulation results. The asymptotic results are expected to apply when the sequence length n is fixed, whereas the number of loci $m\to \infty$. Here, m is fixed at 1,000, so that b < 2 for all cases (table 1), and is too small for the asymptotic approximations to be reliable. As a result, equations (6) and (8) are more accurate.

Discussion

Errors of Species Tree Estimation by Different Methods

Under the MSC model, data at different loci are i.i.d., so that the number of loci (m) constitutes the sample size in the statistical model. Thus, we have derived approximations to the error rate for different methods when m increases, with the sequence length n fixed. For large m, the error can be approximated by $\Phi (-c\sqrt{m})$, where c is a constant. This is seen to apply to all four methods considered in this study (ML, isml = concatenation, and 2-step) (see table 4 for a summary).

Table 4.

Summary of Analytical Approximations to Species-Tree Estimation Error by Different Methods.

Method	n = 1	$n\ge 2$	$n=\infty$
ml		eq. 11	eq. 12
2-step	$\zeta (m,{\overline{p}}_1,{\overline{p}}_2)$	$\zeta (m,g_1,g_2)$	$\zeta (m,\mathbb{P}(G_1),\mathbb{P}(G_2))$
isml/concatenation	${\zeta }_N(m,\Delta p,{\Sigma }^{(1)})$	${\zeta }_N(m,\Delta p,{\Sigma }^{(n)})$	${\zeta }_N(m,\Delta p,{\Sigma }^{(\infty )})$

Note.—For isml/concatenation, $\Delta p={\overline{p}}_1-{\overline{p}}_2$, and the variance–covariance matrix at n is ${\Sigma }^{(n)}=\frac{1}{n}{\Sigma }^{(1)}+\frac{n-1}{n}{\Sigma }^{(\infty )}$ (eq. 16). In the case of n = 1, $\zeta (m,{\overline{p}}_1,{\overline{p}}_2)={\zeta }_N(m,\Delta p,{\Sigma }^{(1)})$, and 2-step, isml, concatenation, and ml are all equivalent.

The theory for ML in Appendix B applies generally to ML selection of nonnested models, whether one model (which may and may not be the true model) fits the data better than the others, judged by the K–L divergence to the true data-generating model. In particular, the theory applies to conventional phylogenetic reconstruction without the MSC model. For example, figure 5 applies the same prediction to simulation results on four-taxa trees from Yang (1997). Previously, Susko (2011) developed a large-sample approximation to the log-likelihood difference between two trees and to the probability that each tree will be the ML tree in the case of four-species without the molecular clock. It was assumed that the internal branch length in the tree is small and approaches 0 at the rate of $n^{-\frac{1}{2}}$ or faster when the number of sites n increases. In our analysis, we take the conventional approach of fixing the parameters when the data size increases.

Fig. 5.

The probit transform of the phylogenetic reconstruction error, ${\Phi }^{-1}(e)$, is a linear function of the square root of the number of sites in the alignment ($\sqrt{n}$). Simulation results from Yang (1997, fig. 1A and B) are used in the plot. The trees used in the simulation have four taxa, with branch lengths ((0.5, 0.5):0.1, 0.5, 0.5) for tree A and ((0.5, 0.5):0.1, 0.6, 1.4) for tree B. Data are simulated under the JC+G model (Yang 1994a) and analyzed under both JC and JC+G (Jukes and Cantor 1969; Yang 1994a). Note that in (B), ML under the incorrect model (JC) is more efficient than ML under the correct model (JC+G).

We note that in problems of parameter estimation, the standard error for the parameter estimate or the width of the confidence interval typically decreases at the rate of $n^{-\frac{1}{2}}$, so that quadrupling the data size halves the interval. In contrast, the probability of recovering the best-fitting model approaches 1 much faster. As the probit transform of the error decreases linearly with $\sqrt{n}$, it will soon reach a point beyond which the precise error probability is of no practical significance: for example, ${\Phi }^{-1}(e)=-3$ means e = 0.0013, while ${\Phi }^{-1}(e)=-5$ means $e=2.9\times {10}^{-7}$. The different dynamics between model selection and parameter estimation when the data size grows is consistent with the fact that we tend to obtain extreme support for phylogenies inferred in large data sets (Yang and Zhu 2018).

Implications of Our Study to Species Tree Methods

Although the species tree problem studied here is the simplest, it has the complexities of the general problem. Furthermore, we have represented all major species tree methods in our analysis. We expect ML to be asymptotically similar to Bayesian inference as both are full-data methods.

We have assumed the JC mutation model and the molecular clock. Our results are thus applicable to shallow species phylogenies and may not apply to distantly related species for which the JC model may be inadequate for multiple-hit correction and the molecular clock may be seriously violated. In the case of three species examined in this paper, concatenation and isml always produce the same species tree estimate. However, in more general settings with four or more species and when the clock is violated and unrooted trees are used, concatenation and isml are known to be different. In particular, concatenation (as well as 2-step) can be inconsistent (Roch and Steel 2015), while isml is a coalescent-aware method and is always consistent.

The isml method considered here is similar to SVDQuartets (Chifman and Kubatko 2014). Both are summary methods based on pooled site-pattern counts. SVDQuartets is sometimes described as a site pattern-based method (e.g., Kubatko 2019). This is not a helpful description. Site-pattern counts for different loci $(\{f_{ij}\})$ are sufficient statistics under the model and carry the same amount of information as the sequence alignments at the same loci so that it makes no difference whether site patterns or sequences are used. Indeed virtually all methods involving likelihood calculation on sequences operate on site patterns instead of sites. Instead what matters is whether site patterns are pooled across loci. In the original data, the sites of the same locus share the same gene tree and the variation among loci provides information about parameters of the coalescent process such as the ancestral population sizes. Pooling sites across loci means that such information is lost (Shi and Yang 2018). As a result, the pooled site-pattern counts are unable to identify all parameters of the MSC model even if they can identify the species tree topology. Previously, Long and Kubatko (2018) found in simulations that SVDQuartets performed better in data sets of 600 coalescent-independent sites ($m=600,n=1$ in the notation of this paper) than in data of two genes each of 300 bp ($m=2,n=300$), and suggested that this is because “[t]he 600 sites observed from 600 distinct gene trees give independent genealogical information about the species tree, though indirectly, whereas the 300 sites for each of the two genes can give a reasonable indication of the individual gene trees, but still provide only two observed gene genealogies.” Our analysis suggests that this is not a correct interpretation. When the information in the data is used properly (as in the ML method), there is in fact more information in two genes each of 300 bp than in 600 independent sites (fig. 3c).

To understand the issue of parameter unidentifiability and the potential information loss for species tree estimation due to the pooling of sites across loci in SVDQuartets, consider the simple random-effects model:

$$y_{ij}=\mu +{\alpha }_i+e_{ij},\hspace{1em}i=1,\dots ,m;j=1,\dots ,n,$$ (20)

where the treatment effect ${\alpha }_i\sim \mathbb{N}(0,{\sigma }_a^2)$ and the error $e_{ij}\sim \mathbb{N}(0,{\sigma }_e^2)$. Parameters in the model include the grand mean μ and the variance components ${\sigma }_a^2$ and ${\sigma }_e^2$. It is obvious that if there are no replications within treatment (n = 1) or if the observations (y_ij) are pooled across treatments, the between-treatment variation and within-treatment errors will be confounded so that ${\sigma }_a^2$ and ${\sigma }_e^2$ will not be identifiable even though μ still is. In species tree estimation, pooling site patterns across loci (as in isml and SVDQuartets) causes some parameters of the MSC model to become unidentifiable even though the species tree still is. This issue of information loss due to averaging over the whole genome may be even more serious for methods designed for data of single nucleotide polymorphisms (SNPs) (Leaché and Oaks 2017), such as snapp (Bryant et al. 2012), because the removal of constant sites in the SNP data causes further loss of information (even if the ascertainment bias is accounted for in the method).

An important difference between isml and SVDQuartets is that isml applies ML to the pooled site-pattern counts, whereas SVDQuartets uses a criterion based on linear invariants to avoid the ML optimization (Xu and Yang 2016). Use of a non-ML criterion is expected to lead to further reduction in efficiency, in addition to information loss due to the pooling of sites across loci (Chou et al. 2015; Xu and Yang 2016; Shi and Yang 2018).

The MSC model analyzed in this paper assumes free recombination among loci and no recombination between sites of the same locus. Data for such analysis are typically loosely linked short genomic segments that are far apart from each other so that recombination within a locus is rare, whereas different loci are nearly independent (e.g.,Takahata et al. 1995; Burgess and Yang 2008; Lohse et al. 2016). Both assumptions of free recombination among loci and no recombination within locus are expected to be violated in real data analysis, and the impact of within-locus recombination is of particular concern. The ML method considered in this paper assumes no recombination (with r = 0), whereas isml (and SVDQuartets) assumes free recombination ($r=\infty$). The relative performance of the methods will depend on the true recombination rate: ML may be expected to perform better than isml if r is close to 0, while isml may be superior if r is large. At very high recombination rates, it may even be possible for ML (assuming r = 0) to be inconsistent since the method is similar to concatenation and merges sites of the same locus with different histories into one sequence. In contrast, isml is consistent for all values of r. Previously, Lanier and Knowles (2012) found in a computer simulation that species-tree estimation was robust to moderate levels of within-locus recombination (see also discussions in Edwards et al. [2016];Xu and Yang [2016]). It will be interesting to evaluate the relative performance of modern species-tree estimation methods (including isml and SVDQuartets) under realistic recombination rates.

Materials and Methods

Simulation

We use a challenging species tree with parameters ${\tau }_0=0.02,\hspace{0.17em}{\tau }_1=0.019$, ${\theta }_0=0.01$, and ${\theta }_1=0.05$ (fig. 1a). A C program is written to simulate gene trees and sequence alignments for the case of three species/sequences, under the JC model (Jukes and Cantor 1969) with the clock. To simulate the gene tree and the sequence alignment for each locus, we generate an exponential coalescent waiting time (s₁) with mean ${\theta }_1/2$. If $s_1<{\tau }_1$, the gene tree is $G_{1a}$, and another exponential waiting time s₀ is generated with mean ${\theta }_0/2$ to get $t_0={\tau }_0+s_0$ and t₁ = s₁. If $s_1>{\tau }_1$, the gene tree is one of $G_{1b},G_2,G_3$, chosen at random, and two coalescent waiting times (s₁ and s₀) are generated with means ${\theta }_0/6$ and ${\theta }_0/2$, respectively, so that $t_1={\tau }_0+s_1$ and $t_0=t_1+s_0$ (fig. 1b). The gene tree and node ages $(t_0,t_1)$ are then used to calculate the site-pattern probabilities for the locus (eq. 3), and the site-pattern counts are generated from multinomial sampling (eq. 4). Each data set consists of m loci with the sequence length of n sites. We use a large number of replicates (typically $R={10}^6$ or 10⁸) so that sampling errors due to a limited number of replicates is not a concern. Species tree estimation by concatenation (=isml) and 2-step is done by counting site patterns.

For the ML method (eq. 5), we used the simulation program MCcoal, which is part of the bpp program (Yang 2015), to simulate the gene trees and sequence alignments. The data are then analyzed using the ML program 3s (Yang 2002; Dalquen et al. 2017). The JC model is used to simulate and analyze data.

Supplementary Material

Supplementary data are available at Molecular Biology and Evolution online.

Appendix A. Proof of Theorem 1

(a) Define the random variable:

$$\begin{array}{c}y={\overline{z}}_1-\max ({\overline{z}}_2,{\overline{z}}_3)={\overline{z}}_1-\frac{1}{2}({\overline{z}}_2+{\overline{z}}_3)-\frac{1}{2}|{\overline{z}}_2-{\overline{z}}_3|\\ =y_1-\left|y_2\right|,\end{array}$$ (A1)

where $y_1={\overline{z}}_1-\frac{1}{2}({\overline{z}}_2+{\overline{z}}_3)\sim \mathbb{N}(\Delta \mu ,s_1^2/m)$ and $y_2=\frac{1}{2}({\overline{z}}_2-{\overline{z}}_3)\sim \mathbb{N}(0,s_2^2/m)$, with

$$\begin{array}{cc}\frac{1}{m}{s}_1^2\hfill & =\mathbb{V}({\overline{z}}_1)+\frac{1}{4}\mathbb{V}({\overline{z}}_2+{\overline{z}}_3)-2\text{Cov}({\overline{z}}_1,{\overline{z}}_2)\hfill \\ \hfill & =\frac{1}{m}[{\sigma }_1^2+\frac{1}{2}({\sigma }_2^2+{\sigma }_{23})-2{\sigma }_{12}],\hfill \\ \frac{1}{m}{s}_2^2\hfill & =\frac{1}{4}\mathbb{V}{({\overline{z}}_2-{\overline{z}}_3)}^2=\frac{1}{2m}({\sigma }_2^2-{\sigma }_{23}).\hfill \end{array}$$ (A2)

Here, we treat ${\overline{z}}_1,{\overline{z}}_2$ and ${\overline{z}}_3$ as normal variables, according to the central limit theorem as $m\to \infty$. As $\text{Cov}(y_1,y_2)=0$ and both y₁ are y₂ are normal variables, they are independent. Then,

$$\begin{array}{cc}\zeta \hfill & =\mathbb{P}\{y_1<|y_2\left|\right\}\hfill \\ \hfill & =\mathbb{P}\{y_2<0,y_1<-y_2\}+\mathbb{P}\{y_2>0,y_1<y_2\}\hfill \\ \hfill & =2\mathbb{P}\{y_2>0,y_1<y_2\}\hfill \\ \hfill & =2{\displaystyle {\int }_0^{\infty }\varphi }(y_2;0,\frac{1}{m}{s}_2^2)\Phi \left(\frac{y_2-\Delta \mu }{s_1/\sqrt{m}}\right)\mathrm{d}{y}_2\hfill \\ \hfill & =2{\displaystyle {\int }_0^{\infty }\varphi }(t)\Phi (at-b)\mathrm{d}t,\hfill \end{array}$$ (A3)

where $a=s_2/s_1,\hspace{0.17em}b=\Delta \mu \sqrt{m}/s_1$, and $\varphi (x;\mu ,{\sigma }^2)$ is the probability density function (PDF) for $\mathbb{N}(\mu ,{\sigma }^2)$, whereas $\varphi (x)$ is the PDF for $\mathbb{N}(0,1)$. The last integral has been studied by Yang and Rodríguez (2013, SI) in a different context and can be written as:

$$\zeta =\frac{1}{\pi }{\int }_{-\frac{\pi }{2}}^{{\hspace{0.17em}\text{tan}\hspace{0.17em}}^{-1}a}\hspace{0.17em}\text{exp}\hspace{0.17em}\left\{-\frac{b^2}{2{(\text{sin}\hspace{0.17em}\theta -a\hspace{0.17em}\text{cos}\hspace{0.17em}\theta )}^2}\right\}\mathrm{d}\theta ,$$ (A4)

or, by letting $t=a-\text{tan}\hspace{0.17em}\theta$, with dθ = −1/[(t−a)²⁺¹] dt, as:

$$\zeta =\frac{1}{\pi }{\int }_0^{\infty }\frac{1}{{(t-a)}^2+1}{\mathrm{e}}^{-\frac{b^2[{(t-a)}^2+1]}{2t^2}}\mathrm{d}t.$$ (A5)

Equations (A4) and (A5) can be calculated using Gaussian quadrature and match direct calculations using the CDF for the bivariate normal distribution for $({\overline{z}}_1-{\overline{z}}_2,{\overline{z}}_1-{\overline{z}}_3)$. When $\Delta \mu =0$, we have b = 0 and:

$$\zeta =2{\int }_0^{\infty }\varphi (t)\Phi (at)\mathrm{d}t=\frac{1}{2}+\frac{1}{\pi }{\hspace{0.17em}\text{tan}\hspace{0.17em}}^{-1}a.$$ (A6)

In the symmetrical case of $\Delta \mu =0,\hspace{0.17em}{\sigma }_1^2={\sigma }_2^2$, and ${\sigma }_{12}={\sigma }_{23}$ (with $a=\frac{1}{\sqrt{3}}$, b = 0), this gives $\frac{1}{2}+\frac{1}{\pi }{\hspace{0.17em}\text{tan}\hspace{0.17em}}^{-1}(\frac{1}{\sqrt{3}})=\frac{2}{3}$, as expected. In this case the three variables ${\overline{z}}_1,{\overline{z}}_2$ and ${\overline{z}}_3$ have the same probability of being the greatest so that the error is $\frac{2}{3}$.

To avoid numerical integration, we note that $y_2\sim \mathbb{N}(0,\frac{1}{2m}({\sigma }_2^2-{\sigma }_{23}))$, and $\left|y_2\right|$ is a folded normal variable with mean and variance:

$$\begin{array}{cc}\mathbb{E}(|y_2|)\hfill & =\sqrt{\frac{1}{m\pi }({\sigma }_2^2-{\sigma }_{23})},\hfill \\ \mathbb{V}(|y_2|)\hfill & =(\frac{1}{2m}-\frac{1}{m\pi })({\sigma }_2^2-{\sigma }_{23}).\hfill \end{array}$$ (A7)

Thus,

$$\begin{array}{cc}\mathbb{E}(y)\hfill & =\Delta \mu -\sqrt{\frac{1}{m\pi }({\sigma }_2^2-{\sigma }_{23})}.\hfill \\ \mathbb{V}(y)\hfill & =\mathbb{V}({\overline{z}}_1)+\frac{1}{4}\mathbb{V}({\overline{z}}_2+{\overline{z}}_3)+\frac{1}{4}\mathbb{V}(|{\overline{z}}_2-{\overline{z}}_3|)\hfill \\ \hfill & -\hspace{0.17em}\text{Cov}({\overline{z}}_1,{\overline{z}}_2+{\overline{z}}_3)-\text{Cov}({\overline{z}}_1,|{\overline{z}}_2-{\overline{z}}_3|)\hfill \\ \hfill & +\frac{1}{2}\text{Cov}({\overline{z}}_2+{\overline{z}}_3,|{\overline{z}}_2-{\overline{z}}_3|)\hfill \\ \hfill & =\mathbb{V}({\overline{z}}_1)+\frac{1}{4}\mathbb{V}({\overline{z}}_2+{\overline{z}}_3)+\frac{1}{4}\mathbb{V}(|{\overline{z}}_2-{\overline{z}}_3|)\hfill \\ \hfill & -\hspace{0.17em}2\text{Cov}({\overline{z}}_1,{\overline{z}}_2)-\text{Cov}({\overline{z}}_1,|{\overline{z}}_2-{\overline{z}}_3|)\hfill \\ \hfill & +\hspace{0.17em}\text{Cov}({\overline{z}}_2,|{\overline{z}}_2-{\overline{z}}_3|).\hfill \end{array}$$ (A8)

We have,

$$\begin{array}{c}\mathbb{V}({\overline{z}}_2+{\overline{z}}_3)+\mathbb{V}(|{\overline{z}}_2-{\overline{z}}_3|)=\mathbb{E}{({\overline{z}}_2+{\overline{z}}_3)}^2+\mathbb{E}{({\overline{z}}_2-{\overline{z}}_3)}^2\\ -{\mathbb{E}}^2({\overline{z}}_2+{\overline{z}}_3)-{\mathbb{E}}^2(|{\overline{z}}_2-{\overline{z}}_3|)\\ =4\mathbb{E}({\overline{z}}_2^2)-4{\mu }_2^2-\frac{4}{m\pi }({\sigma }_2^2-{\sigma }_{23})\\ =\frac{4}{m}{\sigma }_2^2-\frac{4}{m\pi }({\sigma }_2^2-{\sigma }_{23}),\\ \text{Cov}({\overline{z}}_1,|{\overline{z}}_2-{\overline{z}}_3|)=0,\\ \text{Cov}({\overline{z}}_2,|{\overline{z}}_2-{\overline{z}}_3|)=0.\end{array}$$ (A9)

Collecting all terms in equation (A8), we get

$$\mathbb{V}(y)=\frac{1}{m}\left[{\sigma }_1^2-2{\sigma }_{12}+{\sigma }_2^2-\frac{1}{\pi }({\sigma }_2^2-{\sigma }_{23})\right]$$ (A10)

If we assume that y is approximately normally distributed, as in Zharkikh and Li (1992) and Yang (1996), then equation (6) follows. Note that equation (6) can also be written as ${\zeta }_N=\Phi \left(\frac{-b+a\sqrt{2/\pi }}{\sqrt{1+a^2(1-\frac{2}{\pi })}}\right)$. Because $\left|y_2\right|$ has a folded normal distribution and is not a normal variable, the error of approximation of equation (6) does not approach zero when $m\to \infty$. For instance, in the symmetrical case ($\Delta \mu =0,\hspace{0.17em}{\sigma }_1^2={\sigma }_2^2$, and ${\sigma }_{12}={\sigma }_{23}$), equation (6) gives $\Phi (\frac{1}{\sqrt{2\pi -1}})=0.66824$, not $\frac{2}{3}$. This level of accuracy is acceptable for our calculations for finite m in this paper, as the precise value of the error is unimportant if the error is nearly zero, but equation (6) may not give the correct asymptotic error rate when $m\to \infty$ (fig. 6, a = 10).

Fig. 6.

Probit of error, ${\Phi }^{-1}(\zeta )$, plotted against b for different values of a. Six methods for calculating ζ are shown. The first five are, from top to bottom, ${\zeta }_{U1}$ (brown dashed line), ${\zeta }_{U2}$ (orange dotted, with k = 2 in eq. A27), Exact (black solid line), ${\zeta }_{L2}$ (blue dotted), and ${\zeta }_{L1}$ (purple dashed). Equation (6) (black dotted) is included as well.

(b) To study the asymptotic behavior of the error probability ζ when $m\to \infty$, we derive bounds on ζ. From equation (A3),

$$\begin{array}{cc}\zeta \hfill & =2{\int }_0^{\infty }\varphi (t){\int }_{-\infty }^{at-b}\varphi (x)\mathrm{d}x\mathrm{d}t\hfill \\ \hfill & =2{\int }_{-\infty }^{\infty }{\int }_{-\infty }^{at-b}\varphi (t)\varphi (x)\mathrm{d}x\mathrm{d}t-2{\int }_{-\infty }^0{\int }_{-\infty }^{at-b}\varphi (t)\varphi (x)\mathrm{d}x\mathrm{d}t\hfill \\ \hfill & =2S-2A,\hfill \end{array}$$ (A11)

where the first integral is $S=\Phi (-h)$, with $h=\frac{b}{\sqrt{1+a^2}}=\frac{\Delta \mu \sqrt{m}}{\sqrt{{\sigma }_1^2-2{\sigma }_{12}+{\sigma }_2^2}}$ to be the distance from the origin (0, 0) to the line $x=at-b$ (fig. 7), and the second integral is:

$$\begin{array}{cc}2A\hfill & =2{\int }_{-\infty }^0{\int }_{-\infty }^{at-b}\varphi (t)\varphi (x)\mathrm{d}x\mathrm{d}t\hfill \\ \hfill & ={\int }_{-\infty }^{-b}\varphi (x){\int }_{(x+b)/a}^{-(x+b)/a}\varphi (t)\mathrm{d}t\mathrm{d}x.\hfill \end{array}$$ (A12)

Fig. 7.

The areas of integration for integrals in Equations (A11) and (A12). The two angles are $\psi ={\hspace{0.17em}\text{tan}\hspace{0.17em}}^{-1}\frac{1}{a}$ and $\phi ={\hspace{0.17em}\text{tan}\hspace{0.17em}}^{-1}\frac{1}{ka}$, k > 1, with $\phi <\psi <\frac{\pi }{2}$. The integral over the half-plane $x<-b$ is $\Phi (-b)$, whereas the integral over the half-plane $t>(x+b)/a$ is $S=\Phi (-h)=\mathbb{P}\{{\overline{z}}_1<{\overline{z}}_2\}$. The integral over the sector $\mathit{ABA}\prime$ (the shaded area) is $2A=\mathbb{P}\{{\overline{z}}_1<{\overline{z}}_2,{\overline{z}}_1<{\overline{z}}_3\}$. This is smaller than $\Phi (-b)·\psi /\frac{\pi }{2}$ and greater than the integral over the area shaded with the brick pattern: these give the bounds (${\zeta }_{L2},{\zeta }_{U2}$) in Appendix A. The purple dashed lines are $t=x/(ka)$ and $t=-x/(ka)$. They cross the blue lines at A and $A\prime$, with the length of the line segment OA to be $r=\frac{b\sqrt{a^2{k}^2+1}}{a(k-1)}$. Note that the integral over the circle $x^2+t^2<r^2$ is $1-{\mathrm{e}}^{-\frac{1}{2}{r}^2}$.

By considering the area of integration (fig. 7), it is obvious that:

$$0<2A\le \Phi (-b)[1-\frac{2}{\pi }{\hspace{0.17em}\text{tan}\hspace{0.17em}}^{-1}a],$$ (A13)

where the equality holds when b = 0. Let,

$${\zeta }_{L2}=2\Phi (-h)-\Phi (-b)[1-\frac{2}{\pi }{\hspace{0.17em}\text{tan}\hspace{0.17em}}^{-1}a].$$

As $\Phi (-b)<\Phi (-h)$, we have:

$${\zeta }_{L2}>\Phi (-h)[1+\frac{2}{\pi }{\hspace{0.17em}\text{tan}\hspace{0.17em}}^{-1}a]\equiv {\zeta }_{L1},$$ (A14)

or

$$\Phi (-h)\le {\zeta }_{L1}\le {\zeta }_{L2}\le \zeta <2\Phi (-h)\equiv {\zeta }_{U1},$$ (A15)

as in equation (7). The equality in the lower bounds is achieved at $b=0$. Note that the bounds apply to all a > 0 and b > 0. We use the bounds (${\zeta }_{L1},{\zeta }_{U1}$) in Theorem 1 and in the calculation of table 1. The width of the interval is $\Phi (-h)[1-\frac{2}{\pi }{\hspace{0.17em}\text{tan}\hspace{0.17em}}^{-1}a]\le \Phi (-h)\le \zeta$, so that using any value inside the interval as the estimate will give an error of approximation that is smaller than the error probability ζ.

Note that the bounds $\Phi (-h)<\zeta <2\Phi (-h)$ are also given by the definition $\zeta =\mathbb{P}\{{\overline{z}}_1<{\overline{z}}_2\cup {\overline{z}}_1<{\overline{z}}_3\}$, since

$$\mathbb{P}({\overline{z}}_1<{\overline{z}}_2)<\zeta <\mathbb{P}({\overline{z}}_1<{\overline{z}}_2)+\mathbb{P}({\overline{z}}_1<{\overline{z}}_3)=2\mathbb{P}({\overline{z}}_1<{\overline{z}}_2),$$ (A16)

with $\mathbb{P}({\overline{z}}_1<{\overline{z}}_2)=\Phi (-h)$.

Next we consider the upper bound in equation (A15) when h or b is large. Note that:

$$\begin{array}{cc}\Phi (-h)\hfill & ={\int }_h^{\infty }\frac{1}{\sqrt{2\pi }}{\mathrm{e}}^{-y^2/2}\mathrm{d}y\hfill \\ \hfill & ={\int }_0^{\infty }\frac{1}{\sqrt{2\pi }}{\mathrm{e}}^{-\frac{1}{2}{(x+h)}^2}\mathrm{d}x\hfill \\ \hfill & =\frac{1}{\sqrt{2\pi }}{\mathrm{e}}^{-h^2/2}{\int }_0^{\infty }{\mathrm{e}}^{-(hx+\frac{1}{2}{x}^2)}\mathrm{d}x\hfill \\ \hfill & =\frac{1}{h\sqrt{2\pi }}{\mathrm{e}}^{-h^2/2}{\int }_0^{\infty }{\mathrm{e}}^{-t}{\mathrm{e}}^{-\frac{1}{2h^2}{t}^2}\mathrm{d}t\hfill \\ \hfill & =\frac{1}{h\sqrt{2\pi }}{\mathrm{e}}^{-h^2/2}B,\hfill \end{array}$$ (A17)

where $B={\int }_0^{\infty }{\mathrm{e}}^{-t}{\mathrm{e}}^{-\frac{1}{2h^2}{t}^2}\mathrm{d}t<{\int }_0^{\infty }{\mathrm{e}}^{-t}\mathrm{d}t=$ 1. For large h,

$$\begin{array}{cc}B\hfill & >{\int }_0^{\sqrt{h}}{\mathrm{e}}^{-t}{\mathrm{e}}^{-\frac{1}{2h^2}{t}^2}\mathrm{d}t>{\mathrm{e}}^{-\frac{1}{2h}}{\int }_0^{\sqrt{h}}{\mathrm{e}}^{-t}\mathrm{d}t\hfill \\ \hfill & ={\mathrm{e}}^{-\frac{1}{2h}}(1-{\mathrm{e}}^{-\sqrt{h}})=1-\frac{1}{2h}+o\left(\frac{1}{h}\right).\hfill \end{array}$$ (A18)

Thus, for large h, $\Phi (-h)$ is bounded by:

$$\left(1-\frac{1}{2h}+o\left(\frac{1}{h}\right)\right)\frac{1}{h\sqrt{2\pi }}{\mathrm{e}}^{-h^2/2}<\Phi (-h)<\frac{1}{h\sqrt{2\pi }}{\mathrm{e}}^{-h^2/2},$$ (A19)

or

$$\Phi (-h)=\frac{1}{h\sqrt{2\pi }}{\mathrm{e}}^{-h^2/2}+O\left(\frac{1}{h^2}{\mathrm{e}}^{-h^2/2}\right).$$ (A20)

Let $\varepsilon >0$ such that $\Phi (-(h+\varepsilon ))=\alpha \Phi (-h)$ for $0<\alpha <1$; in other words, ɛ is the offset at the probit level to reduce the probability by a fraction. From equation (A20),

$$\begin{array}{c}\frac{1}{(h+\varepsilon )\sqrt{2\pi }}{\mathrm{e}}^{-\frac{1}{2}(h^2+2\varepsilon h+{\varepsilon }^2)}+O\left(\frac{1}{{(h+\varepsilon )}^2}{\mathrm{e}}^{-\frac{1}{2}{(h+\varepsilon )}^2}\right)\hfill \\ =\frac{\alpha }{h\sqrt{2\pi }}{\mathrm{e}}^{-\frac{1}{2}{h}^2}+O\left(\frac{1}{h^2}{\mathrm{e}}^{-\frac{1}{2}{h}^2}\right).\hfill \end{array}$$ (A21)

Thus,

$$\frac{1}{h+\varepsilon }{\mathrm{e}}^{-\frac{1}{2}(h^2+2\varepsilon h+{\varepsilon }^2)}=\frac{\alpha }{h}{\mathrm{e}}^{-\frac{1}{2}{h}^2}+O\left(\frac{1}{h^2}{\mathrm{e}}^{-\frac{1}{2}{h}^2}\right),$$ (A22)

which gives $\varepsilon =-\frac{1}{h}\text{log}\hspace{0.17em}\alpha +o(\frac{1}{h})$ or

$$\Phi \left(-h+\frac{1}{h}\text{log}\hspace{0.17em}\alpha +o\left(\frac{1}{h}\right)\right)=\alpha \Phi (-h).$$ (A23)

In particular, for $\alpha =\frac{1}{2}$, we have:

$$\Phi \left(-\left(h+\frac{1}{h}\text{log}\hspace{0.17em}2+o\left(\frac{1}{h}\right)\right)\right)=\frac{1}{2}\Phi (-h).$$ (A24)

Thus, for large h, we have:

$$2\Phi (-h)=\Phi (-h+\frac{1}{h}\text{log}\hspace{0.17em}2+o(\frac{1}{h}\left)\right),$$ (A25)

as in equation (7). It may be noteworthy that for large h, a very small change at the probit level, of about $\frac{1}{h}\text{log}\hspace{0.17em}2$, changes the probability by a factor of 2.

A tighter lower bound for 2 A than zero of equation (A13) is:

$$2A>\frac{\phi }{\pi }\text{exp}\hspace{0.17em}\left\{-\frac{b^2(a^2{k}^2+1)}{2a^2{(k-1)}^2}\right\},$$ (A26)

where $\phi ={\hspace{0.17em}\text{tan}\hspace{0.17em}}^{-1}\frac{1}{ka}$ with k > 1 (fig. 7). Thus, we have a tighter pair of bounds on ζ,

$$\begin{array}{c}2\Phi (-h)-\Phi (-b)[1-\frac{2}{\pi }{\hspace{0.17em}\text{tan}\hspace{0.17em}}^{-1}a]\le \zeta <\hfill \\ 2\Phi (-h)-\frac{1}{\pi }{\hspace{0.17em}\text{tan}\hspace{0.17em}}^{-1}\frac{1}{ka}\text{exp}\{-\frac{b^2(a^2{k}^2+1)}{2a^2{(k-1)}^2}\},\hfill \end{array}$$ (A27)

where k > 1. We write this pair of bounds as ${\zeta }_{L2}<\zeta <{\zeta }_{U2}$. We have $\Phi (-b)\le \Phi (-h)\le {\zeta }_{L1}\le {\zeta }_{L2}\le \zeta <{\zeta }_{U2}<{\zeta }_{U1}=2\Phi (-h)$. These bounds, as well as the exact value and equation (6), are plotted against b in figure 6 for $a=0.01,0.1,1$ and 10.

Appendix B. The asymptotics of ML species tree estimation

The proof below borrows heavily from White (1982), Dawid (2011), and Yang and Zhu (2018). Let $S_j,j=1,2,3$ be the three species trees with parameters ${\mathit{\theta }}_j$. Note that S₁ is the true model, while S₂ and S₃ are mis-specified models. Let the data at m loci be $\mathit{x}=\left\{{\mathit{x}}_i\right\},i=1,\dots ,m$. The log-likelihood function is ${\ell }_j({\mathit{\theta }}_j)=\text{log}\hspace{0.17em}f(\mathit{x}|S_j,{\mathit{\theta }}_j)$. We also define $l_j({\mathit{\theta }}_j)=\text{log}\hspace{0.17em}f(x|S_j,{\mathit{\theta }}_j)$ for one data point (that is, site-pattern counts at any single locus), $x\equiv (x_{i0},x_{i1},x_{i2},x_{i3},x_{i4})$. When the number of loci $m\to \infty$, the MLE ${\widehat{\mathit{\theta }}}_j\to {\mathit{\theta }}_j^{*}$. We assume that both ${\widehat{\mathit{\theta }}}_j$ and ${\mathit{\theta }}_j^{*}$ are inner points in the parameter space. Whether ${\widehat{\mathit{\theta }}}_j$ is inside the parameter space or at its boundary should not affect the asymptotic rate of convergence. Here, ${\mathit{\theta }}_1^{*}$ for the true species tree S₁ is the true parameter value, whereas ${\mathit{\theta }}_2^{*}$ for S₂ (as well as ${\mathit{\theta }}_3^{*}$ for S₃) is the pseudotrue parameter value, which minimizes the Kullback–Leibler distance from the misspecified model S₂ to the true model S₁.

$$\begin{array}{c}{D}_{12}=\int f(x|S_1,{\mathit{\theta }}_1^{*})\hspace{0.17em}\text{log}\hspace{0.17em}\frac{f(x|S_1,{\mathit{\theta }}_1^{*})}{f(x|S_2,{\mathit{\theta }}_2^{*})}\mathrm{dx}\hfill \\ \hfill & =\mathbb{E}\{l_1({\mathit{\theta }}_1^{*})-(l_2({\mathit{\theta }}_2^{*})\},\end{array}$$ (A28)

where the expectation is over the true distribution $f(x|S_1,{\mathit{\theta }}_1^{*})$. D₁₃ is defined similarly, with $D_{13}=D_{12}$.

We consider the log-likelihood ratio, ${\ell }_j(\widehat{\mathit{\theta }})-{\ell }_j({\mathit{\theta }}^{*})$, given the data (x) for any of the species tree j. We drop the subscript j for clarity. As in White (1982) and Dawid (2011), we define two matrices:

$$\begin{array}{cc}I(\mathit{\theta })\hfill & =\mathbb{E}\{\nabla \hspace{0.17em}\text{log}\hspace{0.17em}f(x|\mathit{\theta })·\nabla \hspace{0.17em}\text{log}\hspace{0.17em}f{(x|\mathit{\theta })}^T\}\hfill \\ \hspace{1em}\hfill & =\mathbb{E}\{l\prime (\mathit{\theta }){(l^{\prime }(\mathit{\theta }))}^T\},\hfill \\ J(\mathit{\theta })\hfill & =\mathbb{E}\{-{\nabla }^2\hspace{0.17em}\text{log}\hspace{0.17em}f(x|\mathit{\theta })\}=\mathbb{E}\{-l″(\mathit{\theta })\},\hfill \end{array}$$ (A29)

(A29)

where the superscript T stands for transpose and where the expectation is over the true distribution, and $\nabla$ and ${\nabla }^2$ are the first and second derivatives with respect to $\mathit{\theta }$.

Apply Taylor expansion to the log likelihood around the MLE $\widehat{\mathit{\theta }}$:

$$\ell (\mathit{\theta })\approx \ell (\widehat{\mathit{\theta }})+\ell \prime (\widehat{\mathit{\theta }})(\mathit{\theta }-\widehat{\mathit{\theta }})+\frac{1}{2}{(\mathit{\theta }-\widehat{\mathit{\theta }})}^T\ell ″(\widehat{\mathit{\theta }})(\mathit{\theta }-\widehat{\mathit{\theta }}),$$ (A30)

where both the gradient and the Hessian are evaluated at the MLE ( $\widehat{\mathit{\theta }}$), with $\ell \prime (\widehat{\mathit{\theta }})=0$. Setting $\mathit{\theta }={\mathit{\theta }}^{*}$, we have:

$$\ell (\widehat{\mathit{\theta }})\approx \ell ({\mathit{\theta }}^{*})+\frac{1}{2}{(\widehat{\mathit{\theta }}-{\mathit{\theta }}^{*})}^T(-\ell ″(\widehat{\mathit{\theta }}))(\widehat{\mathit{\theta }}-{\mathit{\theta }}^{*}).$$ (A31)

Apply Taylor expansion to the derivative $\ell \prime (\mathit{\theta })$ around the MLE $\widehat{\mathit{\theta }}$ and let $\mathit{\theta }={\mathit{\theta }}^{*}$, and we have:

$$\ell \prime (\mathit{\theta })\approx \ell ″(\widehat{\mathit{\theta }})(\mathit{\theta }-\widehat{\mathit{\theta }}),$$ (A32)

and

$$\widehat{\mathit{\theta }}-{\mathit{\theta }}^{*}\approx -\ell ″{(\widehat{\mathit{\theta }})}^{-1}\ell \prime ({\mathit{\theta }}^{*}).$$ (A33)

Each of $\ell \prime (\widehat{\mathit{\theta }})$ and $\ell ″(\widehat{\mathit{\theta }})$ is a sum of m i.i.d. elements. When $m\to \infty ,-\ell ″(\widehat{\mathit{\theta }})\approx m\mathbb{E}\{-l″({\mathit{\theta }}^{*})\}=m{J}^{*}$, with $J^{*}=J({\mathit{\theta }}^{*})$ (eq. A29). Furthermore,

$$\begin{array}{cc}\mathbb{E}\{\ell \prime ({\mathit{\theta }}^{*})\}\hfill & =0,\hfill \\ \mathbb{V}\{\ell \prime ({\mathit{\theta }}^{*})\}\hfill & =m\mathbb{V}\{l\prime ({\mathit{\theta }}^{*})\}=m{I}^{*},\hfill \end{array}$$ (A34)

where $I^{*}=I({\mathit{\theta }}^{*})$ (eq. A29). Thus,

$$\sqrt{m}(\widehat{\mathit{\theta }}-{\mathit{\theta }}^{*})\underset{\hspace{0.17em}}{\overset{P}{\to }}\mathbb{N}\left(0,{(J^{*-1})}^T{I}^{*}(J^{*-1})\right).$$ (A35)

Thus, $\widehat{\mathit{\theta }}={\mathit{\theta }}^{*}+O_p(m^{-1/2})$. Equation (A31) becomes:

$$\begin{array}{c}\ell (\widehat{\mathit{\theta }})\approx \ell ({\mathit{\theta }}^{*})+\frac{1}{2}{\{\sqrt{m}(\widehat{\mathit{\theta }}-{\mathit{\theta }}^{*})\}}^T{J}^{*}\{\sqrt{m}(\widehat{\mathit{\theta }}-{\mathit{\theta }}^{*})\}\end{array}$$ (A36)

$$=\ell ({\mathit{\theta }}^{*})+O_p(1).$$

Equations (A29–A36) apply to all three species trees. In the case of S₁ (the true model), $J^{*}=I^{*}$, the Fisher information matrix, and $\ell (\widehat{\mathit{\theta }})-\ell ({\mathit{\theta }}^{*})\sim \frac{1}{2}{\chi }_d^2$. For S₂ or S₃, $\ell (\widehat{\mathit{\theta }})-\ell ({\mathit{\theta }}^{*})$ is a quadratic form of normal variates and is a mixture of noncentral ${\chi }^2$ variables with mean $\frac{1}{2}\text{tr}(I^{*}{J}^{*-1})$ and variance $\frac{1}{2}\text{tr}({(I^{*}{J}^{*-1})}^2)$, both of O(1).

Now consider using ${\overline{z}}_j\equiv \frac{1}{m}{\ell }_j({\widehat{\mathit{\theta }}}_j)$, j = 1, 2, 3, to compare species trees S₁, S₂, and S₃. We have:

$$\begin{array}{c}\mathbb{E}({\overline{z}}_j)\approx \mathbb{E}(l_j({\mathit{\theta }}_j^{*}))\equiv {\mu }_j,\\ \mathbb{V}({\overline{z}}_j)\approx \frac{1}{m}\mathbb{V}(l_j({\mathit{\theta }}_j^{*}))\equiv \frac{1}{m}{\sigma }_{jj},\\ \text{Cov}({\overline{z}}_j,{\overline{z}}_k)\approx \frac{1}{m}\text{Cov}(l_j({\mathit{\theta }}_j^{*}),l_k({\mathit{\theta }}_k^{*}))\equiv \frac{1}{m}{\sigma }_{jk}.\end{array}$$ (A37)

Thus, when the number of loci $m\to \infty ,\hspace{0.17em}\left\{{\overline{z}}_j\right\}=\{\frac{1}{m}{\ell }_j({\widehat{\mathit{\theta }}}_j)\}$ have means $({\mu }_1,{\mu }_2,{\mu }_2)$ and variance/covariance matrix $\frac{1}{m}\Sigma$, where $\Sigma =\left\{{\sigma }_{jk}\right\}$ is O(1) and independent of m. The error of the ML method, $\mathbb{P}\{{\ell }_1({\widehat{\mathit{\theta }}}_1)>\max ({\ell }_2({\widehat{\mathit{\theta }}}_2),{\ell }_3({\widehat{\mathit{\theta }}}_3))\}=\mathbb{P}\{{\overline{z}}_1>\max ({\overline{z}}_2,{\overline{z}}_3)\}$, is then given by Theorem 1 as equation (11).