Testing Multispecies Coalescent Simulators using Summary Statistics

Abstract

As genomic-scale datasets motivate research on species tree inference, simulators of the multispecies coalescent (MSC) process have become essential for the testing and evaluation of new inference methods. However, the simulators themselves must be tested to ensure that they give valid samples. This work develops methods for checking whether a collection of gene trees is in accord with the MSC model on a given species tree. When applied to well-known simulators, we find that several give flawed samples. The tests presented are capable of validating both topological and metric properties of gene tree samples, and are implemented in a freely available R package MSCsimtester so that developers and users may easily apply them.

1. Introduction

Simulation software plays an important role in the development of phylogenetic methods, providing our only means of 1) verifying that inference software performs properly on large scale data, and 2) comparing the performance of different inference methods. Although simulation studies are often performed under conditions in which model fit is much better than is likely for empirical data, these studies are nonetheless essential to methodological progress.

With increasing attention to the analysis of large multilocus datasets for which incomplete lineage sorting (ILS) may have led to discordant gene trees, simulators of the multispecies coalescent (MSC) model of ILS have become one component of the simulation pipeline. Although there are many causes of gene trees discordance other than ILS, the MSC model is sometimes viewed as the null model — to be considered before further complications such as hybridization, lateral gene transfer, gene duplication and loss, and/or population structure are invoked [1]. Inference of a species tree under the MSC can now be performed in a variety of ways [2]–[7], and methods continue to be developed and refined.

In outline, a large-scale simulation study of species tree inference methods might begin with choices of one or more fixed species trees, with branch lengths in generations, and population sizes for each branch. The number of taxa on the trees may range from small (say eight) to quite large (thousands, e.g. [6]), with anywhere from hundreds to thousands of gene trees being simulated. Sequences are then simulated on each gene tree, forming the simulated data to be analyzed. While sequence simulation software has been well vetted over the many years of development of gene tree inference methods, the same is not true of simulators producing gene trees under the MSC, and it is this component of the pipeline on which we focus.

Unfortunately, testing an MSC simulator for correctness is not simple, as the theoretical distribution of gene trees the model produces is quite complex. For instance, under the MSC on any species tree, each possible gene tree topology has positive probability, with the full probability density having a quite complicated dependence on the species tree topology, edge lengths, and population sizes. Straightforward comparison of a sample of gene trees to the theoretical distribution is simply not a practical approach. This has led some of the most careful developers to validate their software by comparing its output to that of other simulators, rather than to theoretical predictions [8]. For other MSC software, we have been unable to find any information on testing. Due to the difficulty of forming a coherent understanding of a large sample of gene trees, even knowledgeable users may not be able to spot simulation flaws, and are left to trust that the software does what is claimed.

Here we introduce several testing tools, based on theoretical distributions of summary statistics that capture either metric or topological information on the species tree parameter. Our first test is based on the distribution of pairwise distances on gene trees as developed in [9], and the second on counts of rooted triples on gene trees generated under the MSC [10]. Implemented in a freely available R package, MSCsimtester, these tests can be applied to gene tree samples from any simulator to study whether its output is in accord with the MSC. Although examining such summary statistics cannot give an ironclad guarantee of correctness, we believe they are likely to uncover most problems.

We applied our MSCsimtester tools to output from five well-known MSC simulators on a species tree: SimPhy [8], Phybase [11], Hybrid-Lambda [12], Mesquite [13], and DendroPy [14]. See Section 4 for the criterion employed in making our choices. Our goal in this short note is not to check the accuracy of all existing MSC simulators, but to introduce novel methodology for testing them and future ones.

Our tests discovered (initially) that only two of the five simulators, SimPhy and DendroPy, behave as expected under the MSC. In fact, Phybase also generates valid samples under the MSC, but its documentation was not sufficiently explicit on specifying its input, and our initial interpretation of the manual was incorrect. Our tools enabled us to catch and diagnose this user error, and the manual has since been updated after correspondence with the developer. A fourth simulator, Hybrid-Lambda [12] passes our topological tests but samples metric gene trees incorrectly. After this was uncovered in a preliminary version of this work, it has been documented on the Hybrid-Lambda website. The last, Mesquite [13], produces samples with neither gene tree topologies nor metric properties in accord with the MSC model. To the best of our knowledge, this has not been publicized. 5 While we notified the authors of these simulators of the problems in advance of this publication, the larger community should be aware of the need to interpret results of previous simulation work with them cautiously. We strongly suggest that users of other simulators, and developers of new ones, test them with the MSCsimtester tools, which are available at the Comprehensive R Archive Network (CRAN).

2. Background and testing methods

We give an informal description of the MSC model, to motivate and introduce the summary statistics upon which we focus.

Suppose first that 3 taxa are related by the species tree with topology $\left(\right(a,b),c)$. Acknowledging that species are composed of populations, we depict this by a tree whose edges are ‘pipes’ as in Figure 1. The length of each pipe is elapsed time measured in generations, and the width of the pipe represents population size, which may vary over time and edge. When individual genes are sampled from the leaves of the species tree, they trace backwards in time within the species tree until they coalesce at a common ancestral individual. Coalescence is a random process, which informally can be described as individual genes lineages choosing their ‘parent’ uniformly at random from those existing in the previous generation, a panmictic viewpoint. Thus the only population detail of importance under the MSC is size. Importantly, there is a greater chance of coalescence when populations are small, but no requirement that lineages coalesce within any specified finite time.

Fig. 1.

(Left) A metric species tree $\left(\left(a:{\ell }_a,b:{\ell }_b\right):{\ell }_v,c:{\ell }_c\right)$ drawn in black, with population sizes depicted by the widths of edges. Time is measured in generations before the present, and the vertices $v$ and $r$ are labeled, showing where the populations $a$ and $b$ merge, and further in the past where the populations $ab$ and $c$ merge. The vertex $v$ is the most recent common ancestor of species $a$ and $b,MRCA(a,b)$, and $r=MRCA(ab,c)$. With one lineage $A,B,C$ sampled from each species, $A\in a,B\in b,C\in C$, a (blue) metric gene tree depicting ancestral lineages forms within the species tree. (Right) The same metric gene tree depicted more simply.

Simplifying assumptions on population sizes are often made by modelers and programmers, such as that all populations at all times and on all edges throughout the tree are a constant $N$. More realistic is to at least allow different population sizes $N_e$ for each edge (pipe) $e$ of the species tree. While it would be highly desirable to be able to simulate gene trees under the MSC using arbitrary population size functions $N_e\left(t\right)$ varying with time, current simulators do not make this easy on a large tree. Nonetheless, our testing framework could accommodate that generality.

When gene trees are produced under the MSC they have two characteristics that can be viewed somewhat separately. One is the metric information, which is reflected in the distribution of pairwise distances between two fixed taxa across the gene trees. The second is topology, which is reflected in the distribution of rooted triple trees on three fixed taxa displayed on gene trees. In testing the performance of a simulator, it is essential that both metric and topological properties of a sample be examined.

2.1. The distribution of pairwise distances on gene trees

Consider now the species tree $\left(\left(a:{\ell }_a,b:{\ell }_b\right):{\ell }_v,c:{\ell }_c\right)$ shown in Figure 1, with root $r$ and $v$ the most recent common ancestor of $a$ and $b$. For simplicity, fix constant population sizes $N_v$ on the edge above $v$, and $N_r$ above the root $r$. Tracking the lineages of two genes $A$ and $B$ sampled from $a$ and $b$ backwards in time, they cannot coalesce until they reach the population above $v$. On the edge (population) above $v$ coalescence occurs by a Poisson process at a constant rate $1/N_v$. Thus the time $T$ to coalescence above $v$ is exponentially distributed for those times more recent than the root of the tree, $T\in \left(0,{\ell }_v\right]$. At the root $r$ of the tree there is an instantaneous change in the coalescence rate, to the value $1/N_r$, and the coalescent process begins afresh. For the purpose of illustration, assuming that $N_v>N_r$, then coalescence above the root $r$ occurs at a faster rate than below. Piecing together the 3 regions analyzed here (below $v$, between $v$ and $r$, above $r$), the density for the pairwise distance between $A$ and $B$ on gene trees is a piecewise exponential, such as that shown in Figure 2 for $N_v=2000$ and $N_r=1000$.

Fig. 2.

The plot of the probability density function of $d(A,B)$ on gene trees for the species tree shown in Figure 1, with all edge lengths equal to 1000 generations, and population size parameters $N_v=2000$ on the internal edge and $N_r=1000$ for the population ancestral to the root.

Note that the discontinuities in the density in Figure 2 occur due to 1) the impossibility of coalescence below $v$, and 2) the discontinuity in population size at $r$. For larger trees, with more edges leading from the MRCA of a pair of taxa to the root, discontinuities in the density arise whenever there is a discontinuity in the population size. More generally, note that if the population sizes vary along edges with time, then the distribution need not be piecewise exponential, but is computable from the population size functions $N_e\left(t\right)$. Finally, there is no reason to require that the species tree be ultrametric, as edge lengths are in generations, and generation time may vary on different branches. The precise form of the pairwise distance density, and its derivation, is given in Section 3, following work in [9].

2.2. The distribution of rooted triple topologies on gene trees

To test topological features of a sample of gene trees simulated under the MSC, we use the frequencies of rooted triples. For the species tree of Figure 1, again suppose the population sizes $N_v$ and $N_r$ are constant. Then the probability that lineages from $a$ and $b$ fail to coalesce in the edge of length ${\ell }_v$ between $v$ and $r$ is $e^{-x}$, where $x={\ell }_v/N_v$. The quantity $x$ here is the length of the edge in coalescent units, a unit convenient for addressing the confounding effects of population size and time. If the lineages fail to coalesce before the root, then lineages from $a,b,c$ will all be present above $r$, and all three rooted gene topologies $\left(\right(A,B),C)$, $\left(\right(A,C),B)$, and $\left(\right(B,C),A)$ are equally likely to form. This leads to the rooted triple probabilities

$$\begin{gathered} \mathbb{P}(((A,B),C))=1-\frac{2}{3}{e}^{-x}, \\ \mathbb{P}\left(\left(\left(A,C\right),B\right)\right)=\frac{1}{3}{e}^{-x}, \\ \mathbb{P}(((B,C),A))=\frac{1}{3}{e}^{-x} \end{gathered}$$ (1)

which were derived by [10]. More general formulae, accommodating larger trees and changing population sizes are derived in Section 3.

2.3. Metric and topological tests

To assess the accuracy of any MSC simulator, one first produces a large sample of gene trees from a fixed metric species tree with population size parameters. For testing using pairwise gene tree distance densities, after choosing some pair of taxa, a histogram of the pairwise distances between these taxa across the gene trees can be compared to the theoretical density. This comparison can be done visually, as major deviations from the theoretical predictions will be obvious. Additionally, one can perform a statistical test, such as that of Anderson and Darling [15], to compare the empirical distribution from the simulation to the theoretical one, giving a p-value to quantify the fit. Empirical cdfs for the Anderson-Darling test results can then be compared to the expected uniform cdf for the p-value distribution, $U\left(\mathrm{0,1}\right)$.

For verifying topological accuracy, one begins by tabulating the frequencies of the three rooted triple topologies displayed on the simulated gene trees for any (all) choice(s) of three taxa. The simulation can then be tested in two ways: First, one judges the fit of empirical frequencies to the expected ones using Pearson’s chi-squared test to obtain a p-value. Second, one finds the maximum likelihood estimator of the internal branch length $x$ from the tabulated frequencies, and compares this to the true value computed using Equations (1) or their analog for larger trees. The first test is a more formal test and has the advantage of being sensitive to imbalances between the counts for the two topologies incongruent with the species tree.

Both the metric and topological tests we propose require choosing a subset of two or three taxa. While one can apply the tests for all pairs and triples, we caution that in performing all such tests on the same data well known statistical issues arise in interpreting results of the multiple comparisons. Although methods such as the Holm-Bonferroni [16] could be applied to give conservative versions of such a family of tests, we do not believe such a rigorous approach is necessary given the goal of testing a simulator. Indeed, as our analyses of particular simulators have shown, choosing a pair of taxa connected by a many-edges path in the species tree, or a rooted triple of taxa whose internal edge and/or path from the most recent common ancestor to the root is composed of many edges in the species tree, quickly indicates problematic behavior.

3. Derivations of summary distributions

Here we derive formulas for the distributions of pairwise distances and rooted triples displayed on gene trees under the MSC, for a metric species tree with population sizes specified for each edge.

Let $\left(S,\left\{{\ell }_e\right\},\left\{N_e\right\}\right)$ be a metric species tree with population size functions, where each edge $e$ has length ${\ell }_e$ and population size $N_e:\left[0,{\ell }_e\right)\to {\mathbb{R}}^{>0}$. Here $N_e\left(t\right)$ denotes the population size for a haploid organism $t$ generations above the child node of $e$. There is also an ‘above the root’ population size function $N_r:[0,\mathrm{\infty })$. (For diploid taxa, the population sizes should be doubled). For technical reasons, we assume $1/N_e\left(t\right)$ is integrable on finite intervals.

3.1. Pairwise distance distribution

Let $v$ be the most recent common ancestor of taxa $a$ and $b$ (that is, the node on $S$ where $A$ and $B$ lineages enter the same population for the first time), and let $P_a$ be the path in $S$ from $a$ to $v,$ $P_b$ be the path in $S$ from $b$ to $v$, and $P_v$ be the path in $S$ from $v$ to the root $r$. Then $P_v=\left(e_1,e_2,\dots ,e_k\right)$, where $v$ is incident to $e_1$ and $r$ is incident to $e_k$. Finally, let $g_a={\sum }_{e\in P_a} {\ell }_a$ and $g_b={\sum }_{e\in P_b} {\ell }_b$ be the number of generations from $a$ and $b$, respectively, to $v$. Then the distance $d(A,B)$ is a random variable

$$Y=g_a+g_b+2X,$$

where $X$ is the random variable giving the time to coalescence of two lineages at $v$. Let $c\left(x\right)$ be the probability density function for $X$.

To compute $c\left(x\right)$, let $N^{\mathrm{*}}:[0,\mathrm{\infty })\to {\mathbb{R}}^{>0}$ be the piecewise ‘union’ of the $N_e$ for $e\in P_v$ and $N_r$, which with $m_0=0,m_j={\sum }_{i=1}^j {\ell }_{e_i}$ for $1\le j\le k,m_{k+1}=\mathrm{\infty }$, and $N_{k+1}$ the population function ancestral to the root is given by

$$N^{\mathrm{*}}(x)=N_{e_i}\left(x-m_{i-1}\right)$$

for $x\in \left[m_{i-1},m_i\right),1\le i\le k+1$. Since the coalescent process for two lineages in the same population of size $N^{\mathrm{*}}(x)$ occurs with instantaneous rate $1/N^{\mathrm{*}}(x)$, the probability density function is [9]

$$\begin{gathered} c(x)=\frac{1}{N^{\mathrm{*}}(x)}\mathrm{e}\mathrm{x}\mathrm{p}\left(-{\int }_0^x \frac{1}{N^{\mathrm{*}}(\tau )}d\tau \right) \\ =\left(\prod _{j=1}^{i-1} {\eta }_j\right)\frac{\mathrm{e}\mathrm{x}\mathrm{p}\left(-{\int }_0^{x-m_{i-1}} \frac{1}{N_{e_i}(\tau )}d\tau \right)}{N_{e_i}\left(x-m_{i-1}\right)}, \end{gathered}$$

for $x\in \left[m_{i-1},m_i\right)$, where

$${\eta }_i=\mathrm{e}\mathrm{x}\mathrm{p}\left(-{\int }_0^{{\ell }_i} \frac{1}{N_{e_i}\left(\tau \right)}d\tau \right)$$

is the probability that 2 lineages entering edge $e_i$ fail to coalesce on it.

Since $X=\frac{Y-g_a-g_b}{2}$, setting $g_{ab}=g_a+g_b$ this shows the probability density function for $Y$ is

$$f(y)=\{\begin{array}{l}0\text{for}y\le g_{ab},\\ \left(\prod _{j=1}^{i-1}{\eta }_j\right)\frac{\exp \left(-{\int }_0^{\frac{y-g_{ab}-2m_{i-1}}{2}}\frac{1}{N_{e_i}(\tau )}d\tau \right)}{2N_{e_i}\left(\frac{y-g_{ab}-2m_{i-1}}{2}\right)}\\ \text{for}\{\begin{array}{l}{g}_{ab}+2m_{i-1}\le y<g_{ab}+2m_i,\hfill \\ 1\le i\le k,\hfill \end{array}\\ \left(\prod _{j=1}^k{\eta }_j\right)\frac{\exp \left(-{\int }_0^{\frac{y-g_{ab}-2m_k}{2}}\frac{1}{N_r(\tau )}d\tau \right)}{2N_{e_r}\left(\frac{y-g_{ab}-2m_k}{2}\right)}\\ \text{for}{g}_{ab}+2m_k\le y.\end{array}$$

In the special case that a population size function $N_e\left(t\right)$ is constant, this shows that the corresponding piece of $f$ is a shifted, scaled, and possibly truncated exponential density.

3.2. Rooted triple frequencies

Suppose the rooted triple $\left(\right(a,b),c)$ is displayed on $S$, and let $P=\left(e_1,e_2,\dots e_i\right)$ denote the path from the most recent common ancestor of $a,b$ on $S$ to the most recent common ancestor of $a,b,c$. With the notation of the previous subsection, the probability that $A$ and $B$ lineages fail to coalesce within $P$ is ${\prod }_{j=1}^i {\eta }_j$. Note that the gene triplets $\left(\right(A,C),B)$ and $\left(\right(B,C),A))$ can only form if the $A,B$ lineages do not coalesce on $P$. Moreover, if $A,B$ have not coalesced on $P$ then by the exchangeability of lineages in the same populations under the MSC, the probability that any particular pair of $A,B,C$ coalesce first is $1/3$. Thus

$$\mathbb{P}(((A,C),B))=\mathbb{P}(((B,C),A))=\frac{1}{3}\prod _{j=1}^i {\eta }_j.$$

Since the probabilities of the three possible topologies sum to 1,

$$\mathbb{P}(((A,B),C))=1-\frac{2}{3}\prod _{j=1}^i {\eta }_j.$$

In the special case of constant population sizes ${\eta }_j=\mathrm{e}\mathrm{x}\mathrm{p}\left(-{\ell }_{e_i}/N_{e_i}\right)$. More generally, the length of $e_i$ in coalescent units is ${\int }_0^{{\ell }_i} \frac{1}{N_{e_i}\left(\tau \right)}d\tau$, and ${\prod }_{j=1}^i {\eta }_i=\mathrm{e}\mathrm{x}\mathrm{p}(-x)$ where $x$ is the length of $P$ in coalescent units.

4. Simulations and results

Using both metric and topological tests we investigated five popular simulators: SimPhy [8], Phybase [11], Hybrid-Lambda [12], Mesquite [13], amd DendroPy [14]. These simulators were chosen since they 1) allow input of a species tree, either in Newick notation or graphically, with edge lengths in generations, and 2) allow a population size $N_e$ to be assigned independently to each edge $e$ in the species tree. We consider these minimal requirements for a simulator appropriate for large scale simulations studies involving the MSC. All these software packages have many functionalities beyond MSC simulation, but our focus is restricted to the accuracy of MSC samples.

We also investigated the performance of the well-established and highly flexible ms [17] using a small species tree. However, for simulating from the MSC on a tree, ms requires careful conversion of a Newick-formatted tree for input, which violates our first criterion. This conversion, when performed manually, is prone to error, and therefore highly undesirable for many-taxon simulations. Although PhyloNet [18] provides a tool for inputting a species tree and then calling ms to generate a gene tree sample, its functionality is restricted to a constant population size on the entire species tree, violating our second criterion. On a manually produced small tree with varying population sizes, however, no aberrations were found in the gene tree samples ms produced (results not shown), as we expected.

We performed tests with a number of species trees, using a variety of choices of constant population sizes per edge. Constant population sizes were used since none of the simulators implement time-varying population sizes. Here we show only representative examples of this work, using the species tree and populations depicted in Figure 3. Additional test results, on trees with up to 6 taxa, are shown in the supplementary materials, in Figures F2–F23 and Tables T1–T4. Samples of 100,000 gene trees were simulated with Mesquite, Phybase, Simphy, DendroPy, and of size 99,999 (its maximum) for Hybrid-Lambda.

Fig. 3.

4-taxon metric species tree, with constant population sizes on each internal edge, for which data was simulated using the parameter values shown.

For assessing the accuracy of metric features of the sample, the values of $d(A,B)$ on the gene trees were extracted, and a histogram was produced. These are shown in Figure 4, with the theoretical distribution superimposed on them. There is a good match for SymPhy and DendroPy, as was seen in all our simulations with these packages. When our initial Phybase simulation did not match expectations, we learned that species tree edge lengths should be supplied as $\mu t$, where $t$ is in generations and $\mu$ is a mutation rate, while population sizes should be specified as $\theta =4\mu N_d$ where $N_d$ is diploid population size. Taking $\mu =1$ and $\theta =2N$ with $N$ the haploid population size, Phybase’s sample matched expectations well. Both the Hybrid-Lambda and Mesquite simulations show pronounced deviations from the theoretical distribution. However, when the input tree is specified with branch lengths in coalescent units, Hybrid-Lambda correctly gives a sample of gene trees whose pairwise distances in coalescent units match the theory well (results not shown); the poor fit occurs only when species tree is entered in numbers of generations and the branch-specific population sizes are entered separately.

Fig. 4.

Pairwise distance distributions for $d(A,B)$ from 100,000 simulated gene trees for the species tree and population sizes shown in Figure 3. Cumulative distribution functions are shown in the Supplement, Figure F2.

To quantify deviation of the sampled $d(A,B)$ density from the expected distribution, we apply the Anderson-Darling test. As is well known, even the small numerical errors arising from computer round-off in a simulation or its analysis may prevent the extremely close fit to theory that a large sample should exhibit. As a result, with very large samples such tests can produce misleadingly small p-values, leading to excessive rejection. To address this, we divided each of our samples into 100 subsamples of size 1000, computing a p-value for each. A good fit is then shown by a roughly uniform distribution of p-values for the subsamples, and an empirical cdf that matches that of a uniform distribution $U\left(\mathrm{0,1}\right)$. Figure 5 shows these p-value distributions and empirical cdf’s for the samples from SimPhy and Mesquite, formally confirming the conclusions already described. Similar results are presented for pairs of taxa other than $A,B$, and for other species trees, in the Supplementary materials.

Fig. 5.

Distributions and cdfs of p-values from Anderson-Darling tests comparing empirical and theoretical $d(A,B)$ densities for SimPhy (Top) and Mesquite (Bottom) on the species tree of Figure 3. The p-values were computed for 100 independent subsamples of size 1000 from the pairwise distance data underlying Figure 4. The pronounced divergence of the histogram from the expected density for Mesquite indicates poor fit. More rigorously, the deviation of Mesquite’s empirical cdf from the blue cdf for a uniform distribution of p-values shows that Mesquite’s sample is flawed.

Topological features of samples were analyzed by tabulating counts of all rooted triple topologies across the sampled gene trees and then performing a chi-squared test, and by computing the MLE of the internal edge length on the species tree triple. Results are shown in Table 1. For the programs Hybrid-Lambda, Phybase, SimPhy, and DendroPy no p-values were extreme enough to suggest poor fit. However, the p-values for Mesquite strongly suggest poor model fit, with values extremely close to 0. Internal edge length estimates were also poorest for the Mesquite sample. Note that although Hybrid-Lambda had poor performance on our metric tests with units in generations, this procedure confirms it gave a good topological sample. This is consistent with the observation that a Hybrid-Lambda sample is accurate when species tree edge lengths are given in coalescent units. Indeed, assuming the Hybrid-Lambda algorithm is based on coalescent units, its errors may occur in conversion into numbers of generations.

TABLE 1.

Rooted triple topology counts from 100,000 gene trees sampled from the MSC on the species tree of Figure 3, and the p-values from the chi-squared tests. In the last column, the internal edge length of the species tree rooted triple in coalescent units with its Maximum Likelihood estimate from the simulated datasets. Small p-values and poor internal edge length estimates indicate poor model fit. Table entries are rounded.

	$\left(\right(A,B),C)$	$\left(\right(A,C),B)$	$\left(\right(B,C),A)$	p-value	Int. edge
Expected	59564	20217	20217	-	0.500
Mesquite	56166	17424	26410	0	0.419
Hybrid-Lambda	59703	20227	20069	0.492	0.503
Phybase	59649	20122	20229	0.749	0.502
SimPhy	59800	20074	20126	0.306	0.506
DendroPy	59496	20049	20455	0.118	0.498
	$\left(\right(A,B),D)$	$\left(\right(A,D),B)$	$\left(\right(B,D),A)$	p-value	Int. edge
Expected	71027	14487	14487	-	0.833
Mesquite	70229	14807	14964	1.26e-07	0.806
Hybrid-Lambda	71116	14462	14421	0.798	0.836
Phybase	71272	14335	14393	0.219	0.842
SimPhy	71061	14324	14615	0.225	0.835
DendroPy	71006	14584	14410	0.587	0.833
	$\left(\right(A,C),D)$	$\left(\right(A,D),C)$	$\left(\right(C,D),A)$	p-value	Int. edge
Expected	52231	23884	23884	-	0.333
Mesquite	50256	24699.	25045	3.16e-35	0.293
Hybrid-Lambda	51997	24110	23892	0.203	0.328
Phybase	52367	23813	23820	0.691	0.336
SimPhy	52193	23832	23975.	0.784	0.333
DendroPy	52190	24028	23782	0.512	0.332
	$\left(\right(B,C),D)$	$\left(\right(B,D),C)$	$\left(\right(C,D),B)$	p-value	Int. edge
Expected	52231	23884	23884	-	0.333
Mesquite	54905	22466	22629	4.57e-63	0.391
Hybrid-Lambda	51844	24131	24024	0.044	0.325
Phybase	52336	23826	23838	0.801	0.336
SimPhy	52116	24035	23849	0.534	0.331
DendroPy	52383	23921	23696	0.371	0.337

5. Discussion

These results indicate that inadequate attention has been given previously to ensuring MSC simulators perform correctly. As novel methods are developed that can scale to genomic data or incorporate multilocus datasets of thousands of gene trees, it is imperative that such methodologies be tested on simulated datasets, both small and large, before practitioners can reliably trust analyses. The tests based on metric and topological summary statistics implemented in the R package MSCsimtester are a practical tool to uncover errors in simulators, and in user input to simulators. We recommend these tests be routinely used by developers of such simulators and, until software has been fully vetted, by anyone performing multispecies coalescent simulations.

Moreover, it is important that those currently conducting simulation studies, or interpreting the results of previous work (e.g., Mesquite simulations appear in [19], [20]), are aware of the problems these tests illuminate. The current version of Hybrid-Lambda is reliable only when metric species trees (and by extension networks) are specified in coalescent units, such as was done in [21], but caution is still warranted. A preliminary version of this paper showed that at least one earlier version of Mesquite produced erroneous samples, and we cannot verify that any early version performed correctly. New simulations in this work show problems remain in the current Mesquite version 3.7, and it should not be used for simulating under the MSC.

Note that our statistical methods consider distances for only one pair of taxa, or the rooted triple counts for only one choice of three taxa, at a time. When applied to many pairwise distances or rooted triple counts, even when conditioned on the species tree, these are not typically independent tests when applied to the same gene tree simulation. Although it would be desirable to test the full joint distribution as a step toward stronger testing, devising methods to do so is difficult. We believe the approach given here is adequate for revealing most simulator errors.

Finally, as the inference of species networks arising from hybridization or horizontal gene transfer receives more attention, simulators of the network multispecies coalescent (NMSC) model, such as Hybrid-Lambda, are likely to become more available. Enhancements of MSC simulators on a tree (e.g., permitting population sizes to vary with time on branch lengths) are also desirable. Developing testing methods for such software, that can be applied for general species networks and trees, will present new challenges. Understanding the distributions of summary statistics in such a setting is a considerably more complex task than that considered here.