Covariance decomposition for distance based species tree estimation

Abstract

In phylogenomics, species-tree methods must contend with two major sources of noise; stochastic gene-tree variation under the multispecies coalescent model (MSC) and finite-sequence substitutional noise. Fast agglomerative methods such as GLASS, STEAC, and METAL combine multi-locus information via distance-based clustering. We derive the exact covariance matrix of these pairwise distance estimates under a joint MSC-plus-substitution model and leverage it for reliable confidence estimation, and we algebraically decompose it into components attributable to coalescent variation versus sequence-level stochasticity. Our theory identifies parameter regimes where one source of variance greatly exceeds the other. For both very low and very high mutation rates, substitutional noise dominates, while coalescent variance is the primary contributor at intermediate mutation rates. Moreover, the interval over which coalescent variance dominates becomes narrower as the species-tree height increases. These results imply that in some settings one may legitimately ignore the weaker noise source when designing methods or collecting data. In particular, when gene-tree variance is dominant, adding more loci is most beneficial, while when substitution noise dominates, longer sequences or imputation are needed. Finally, leveraging the derived covariance matrix, we implement a Gaussian-sampling procedure to generate split support values for METAL trees and demonstrate empirically that this approach yields more reliable confidence estimates than traditional bootstrapping.

Introduction

A core problem in phylogenomics is the reconstruction of a species tree from gene alignments. A species tree encodes the evolutionary relationships and divergence times among given species, while gene trees represent the evolutionary histories of individual genes. The structure of a gene tree may differ from the species tree due to variation induced by processes such as incomplete lineage sorting (ILS), horizontal gene transfer and hybridization [1]. Finally, gene alignments are comparisons of DNA, RNA, or protein sequences across different species, from which gene trees may inferred. The task of species tree reconstruction is to infer the species tree from gene alignments.

Hence, species tree reconstruction is commonly formulated as a hierarchical statistical model with three levels, as illustrated in Fig. 1. At the top level, the species tree is the parameter of ultimate interest, to be estimated either as a point estimate or via a (approximate) posterior distribution. Given , the second level consists of m gene trees , whose likelihoods are defined by the multispecies coalescent (MSC) model, which captures the stochasticity introduced by incomplete lineage sorting (ILS) under a given species tree [2]. At the bottom level are the gene alignments , whose likelihoods depend only on the corresponding gene trees, and are computed using nucleotide substitution models such as the Jukes-Cantor (JC69) model [3], the Hasegawa-Kishino-Yano (HKY) model [4], and the Generalized Time Reversible (GTR) model [5]. In practice, we observe only the gene alignments. Depending on the method, one may first infer the gene trees and then the species tree, or infer the species tree directly from sequence data without explicit gene-tree reconstruction. While the present work focuses on the Jukes-Cantor model for simplicity and analytical tractability, more general distance corrections may be considered when substitution rates deviate from the assumptions of Jukes-Cantor model (e.g., unequal base frequencies or transition–transversion biases). In such cases, alternatives such as logdet distances [6], which enjoy desirable properties under heterogeneous substitution processes, may provide less biased distance estimates.

Fig. 1.

In phylogenetics, the goal is to reconstruct gene trees for i ∈ [m] (intermediate level) from m gene alignments (bottom level) through a nucleotide substitution model F(D|G). In phylogenomics, the aim is to infer the species tree (top level) from which these gene trees arose through the MSC model

Viewing a species tree reconstruction via a hierarchical model has several conceptual advantages: it clarifies the distinct sources of randomness (under coalescent and substitution models), provides a foundation for Bayesian approaches [7], and facilitates the development of statistical methods for quantifying uncertainty at each level. Moreover, it enables the integration of multiple data types and model components within a coherent inferential framework, as seen in widely used tools such as MrBayes [8], BEAST [9], and BPP [10].

Species tree reconstruction methods

In contrast to computationally intensive methods for species tree reconstruction, distance-based approaches based on summary statistics—such as METAL [11], GLASS [12], and STEAC [13]—offer simpler and more computationally efficient alternatives. These methods rely on estimated coalescent times or Hamming distances between species pairs, and then define summary statistics over these estimates across all genes. GLASS, which uses the minimum coalescent times across genes as its summary statistic, can provide more accurate species tree estimates than METAL or STEAC when gene tree uncertainty is minimal. In contrast, METAL and STEAC, which rely on averages as their summary measure, tend to be more robust in the presence of greater variance in gene tree estimates or forms of gene tree heterogeneity other than ILS.

Both STEAC and GLASS were originally developed as methods that take gene trees as input and produce an inferred species tree as output. Their application in phylogenomics thus depends on the accurate reconstruction of gene trees from gene sequence data. In contrast, METAL was specifically designed to infer species trees directly from gene sequences without requiring intermediate gene-tree estimation. As a result, METAL typically achieves accurate species tree reconstruction with fewer loci than methods that rely on gene-tree estimation, such as GLASS and STEAC. Notably, when working directly with sequence data, METAL requires only total sites to recover the correct species tree topology with a a specified high probability, where τ is the shortest internal branch length in the species tree. By comparison, GLASS and STEAC require and total sites respectively [11]. However, GLASS can outperform METAL in regimes when the number of bases per gene is large; as substitutional noise diminishes, GLASS can yield more accurate estimates than METAL due to its minimum distance criterion.

Despite the computational efficiency and practical appeal of distance-based methods, their behavior under different sources of uncertainty has remained only partially understood. Prior work has noted limitations of early approaches such as GLASS and proposed refinements (e.g., iGLASS [14]) to mitigate sensitivity to mutational noise. Other studies have explicitly examined the impact of gene-tree variance and substitutional error on inference accuracy, including simulation-based analyses that teased apart their relative contributions [15]. What remains less clear, however, is how these two sources of uncertainty–coalescent variation versus substitutional noise–interact across parameter regimes, and under which conditions methods such as STEAC, METAL, or more recent variants of GLASS will be most reliable. The relative influence of these error sources can shift dramatically with mutation rate, sequence length, and tree height, making a systematic understanding essential.

Although bootstrap resampling of concatenated sequences is common for assessing split support, it conflates coalescent and substitutional uncertainties and can yield misleading confidence values under extreme variance regimes. By deriving the full covariance matrix of METAL’s pairwise distances, we introduce a Gaussian-sampling procedure that produces more accurate split support estimates than traditional bootstrapping.

Contributions

Considering the hierarchical nature of species tree reconstruction, uncertainty arises from two distinct sources; (1) variation in tree topologies governed by MSC, and (2) sequence-level stochasticity introduced by the substitution process used to infer gene trees from alignments (e.g., as JC69). By explicitly deriving the covariance matrices corresponding to each source, we obtain a quantitative framework for understanding how uncertainty propagates through distance-based estimates. Related work by Guerra and Nielsen [16] has also derived the covariance of pairwise distances under the multispecies coalescent with mutation, in a more general setting that accommodates arbitrary species trees and varying ancestral population sizes.

Our analysis differs in focusing specifically on distance-based species tree reconstruction methods (METAL, GLASS, STEAC), and in decomposing the total covariance into coalescent and substitutional components to study how their relative magnitudes govern method performance. In particular, we demonstrate that METAL outperforms GLASS when substitutional variance dominates the total uncertainty, and vice versa - this finding supports practitioners in using intuition about the prominent sources of uncertainty to select a method. Furthermore, we find that METAL outperforms STEAC in all of our experiments.

This decomposition enables informed decisions about study design, such as whether to prioritize collecting longer sequences or more independent loci. When coalescent variance dominates, increasing the number of loci is most effective; when substitutional noise is dominant, longer alignments (perhaps via imputation strategies) are preferable.

In Sect. “Species tree reconstruction from gene trees”, we analyze species tree reconstruction from known gene trees under METAL, GLASS, and STEAC. We compute the covariance structure of METAL and STEAC, and show that METAL effectively interpolates between STEAC and GLASS as a function of mutation rate. In Sect. “Gene tree reconstruction from base sequences”, we study species tree reconstruction from gene base sequences, analyzing the statistical properties (signal and noise) of the distance estimates used by METAL and STEAC under finite sequence lengths. Section “Theoretical results” presents our theoretical results: the decomposition of the total covariance of METAL and STEAC into coalescent and substitutional components, analysis of the spectrum of the METAL covariance matrix, and asymptotic expressions quantifying their relative magnitudes across a range of model parameters. Section “Simulation results” contains two empirical applications. Section “Species-tree estimation” compares the performance of METAL, GLASS, and STEAC across regimes where substitutional or coalescent variance dominates, showing that the relative error of GLASS versus METAL depends critically on the proportion of variance attributable to substitutional noise. In Sect. “Bipartition confidence”, we introduce a Gaussian-sampling procedure for computing support values for METAL trees, enabled by our covariance derivations. We show empirically that this method provides more reliable split confidence than traditional bootstrapping. Finally, Sect. “Discussion” concludes the paper with a discussion of the implications, limitations, and future directions of this work.

Species tree reconstruction from gene trees

First, we introduce the notation that will be used in the rest of the paper. The task is to successfully reconstruct the species tree S = (V,E), where V and E is the set of nodes and edges in the tree respectively. The leaf set L ⊆ V corresponds to the n=|L| extant species examined. We assume throughout that the species tree is ultrametric, consistent with a molecular clock model in which all lineages evolve at the same rate, so that all root-to-leaf path lengths are equal. We define to be the length of e ∈ E in coalescent units and the diameter of the tree as .

For simplicity, we sample a single individual per species, following the approach implemented in agglomerative, distance-based frameworks such as METAL [17]. We consider m loci with associated gene trees with the same leaf set L, and define as twice the time in coalescent units to the most recent ancestor of a and b in . This is the evolutionary distance between the two leaves. In this section, we assume that the gene trees are known. Each of the three methods examined in this paper uses a dissimilarity map and subsequently performs hierarchical clustering on that distance matrix. The resulting dendrogram is an estimate of the species tree. Pseudocode for these algorithms can be found in Appendix C, Algorithm 2.

The dissimilarity maps are defined as follows:

1

2

3

for all pairs where is the probability that the bases of species a and b differ at any given site in locus i under the Jukes-Cantor evolutionary model, and μ is rate of substitution.

Note that METAL does not require prior knowledge of the mutation rate μ when applied to empirical sequence alignments. In practice, one simply computes the normalized Hamming distance between alignments of species a and b for each locus i, which is the proportion of sites that differ between those two alignments, and then averages over loci to obtain

As the number of sites per locus tends to infinity, , and thus . Under the Jukes–Cantor model [3], the p-distance is related to the Jukes-Cantor distance , which is measured in expected number of substitution time units, through

which, yields Eq. (3) by noting that .

Proposition 1 shows that, under perfect gene-tree knowledge, the METAL distance “interpolates” between STEAC and GLASS: as the substitution rate , it reduces to STEAC’s average coalescent time, whereas as , it collapses to GLASS’s minimum-coalescent-time criterion.

Proposition 1

Let be the reconstructed species tree topologies of each method provided that the correct gene trees are given, assuming that all methods use the same hierarchical clustering method, and that all the minima attained by GLASS are unique. Then,

For the second limit, it is assumed that the hierarchical clustering method is single or complete linkage clustering.

Proof

All proofs can be found in Appendix A. □

Proposition 2 then establishes that, for any four leaves a,b,c,d∈ L, the pairwise distances produced by STEAC and by METAL are both nonnegatively correlated, with STEAC correlations always at least as large as those of METAL. The covariance matrix of METAL pairwise distances–assembled from the individual covariance components derived in the proof of Proposition 2–will be used for uncertainty quantification in Sect. “Theoretical results” and to formulate the total covariance in our simulation results (Sect. “Simulation results”).

Proposition 2

Let a,b,c,d ∈ L be leaves in the species tree. Then, under MSC,

where denotes the correlation between two random variables. Moreover, as a function of t is continuous and strictly increasing on (-∞ ,0) with limit

Finally,

Gene tree reconstruction from base sequences

In the previous section, we considered properties of the distribution of gene trees under the MSC model. In this section, we investigate statistical properties of gene base sequence conditional on the gene tree or . Let be the base of the locus of species a ∈ L, where i ∈ [m], , m is the number of loci studied and is the sequence length of loci i. For notational simplicity in the formulae that follow, we assume that all loci have the same number of bases . The analysis can be readily extended to the case of loci with varying sequence lengths. The normalized Hamming distance, and Jukes-Cantor distance between species a, b∈ L are then defined as

METAL computes the average of the normalized Hamming distances, while STEAC and GLASS consider the average and minimum Jukes-Cantor distances respectively. However, a problem with using the Jukes-Cantor distances directly is that they are infinite if for any a,b ∈ L, i ∈ [m], which has a non-zero chance of occurring.

Hamming distances

Given the gene tree of locus i, the distribution of is

Effectively, is the true probability that species a and b would share the same base in locus i and . The number of bases, K, is typically large and so we can apply the common Normal approximation of the Binomial distribution,

While the normal approximation above applies to each pairwise distance individually, we require a joint multivariate description of all distances at locus i. In fact, the multivariate extension of that result is

where are vectors containing the estimated and true pairwise distances respectively and is the covariance matrix whose entries are of the form given by Proposition 3.

Proposition 3

Let a,b,c,d ∈ L be leaves of a gene tree, where the pairs a, c and b, d are sisters as shown in Fig. 2. Letting be the length of the intersection of the shortest path from a to b the shortest path from c to d, the covariance is

4

Fig. 2.

The evolutionary tree used in proposition 3. Here is the length of the intersection of the shortest path from a to b and the shortest path from c to d. All trees with 4 distinct leaves a,b,c,d ∈ L can be drawn in this way. For cherry trees, the pairs (a, c), (b, d) are sisters/cherries and the root of the 4-leaf tree is contained in the δ -segment. For comb trees, without loss of generality, the topology is (( (a, c), b), d) with a, c being sisters and the root of the tree contained in the shortest path from b to d. Note that since the paths from a to c and b, d are disconnected

Using Proposition 3, we see in particular, that the diagonal entries of coincide with the univariate variances: by expanding Eq. (4) and noting that , we have,

Jukes-Cantor distances

For the STEAC method, instead of using normalized Hamming distances, , it uses estimates for the gene tree branch lengths

5

The major issue with this approach is that if at any loci i ∈ [m] we get , then and so . Clearly, this is undesirable, especially when the number of genes is large and the number of sites per gene K is low, when such events are most common. Hierarchical clustering methods such as UPGMA would not allow clusters containing leaves a and b to ever merge.

In this paper we consider the case when K ≫ 1, when the Delta method is typically used to approximate the distribution of in Eq. (5), which dictates that

where , are vectors containing the estimated and true Jukes-Cantor distances respectively, and the row ab, column cd element of the covariance matrix is

6

where the last equality follows from Proposition 3 and the definition of . In the following section, we make use of these derived covariance matrices to compute the total covariance matrix of the METAL and STEAC pair distances.

Theoretical results

In this section, we analyze both the signal and noise of distance-based inference under the MSC+Jukes–Cantor model. The signal is quantified by the expected gap in the four-point condition for quartets, which reflects the intrinsic separation of sister pairs in the species tree. Intuitively, the greater the expected gap, the easier it is to correctly reconstruct the quartet. The noise of METAL and STEAC estimators is captured by their total covariance which can be decomposed into coalescent and substitution components, thereby quantifying the relative contributions of coalescent/genealogical and substitution/mutational uncertainty. The spectral properties of these matrices are also analyzed.

Expected gap

The following Proposition formalizes the expected four-point gap for a quartet of leaves.

Proposition 4

Let be four leaves with a and b forming a sister pair (i.e. ). Then the expected four-point gap is

7

where is the pairwise distances of the METAL estimator. If the tree has cherry topology ((a,b),(c,d)) with quartet diameter , Eq. (7) simplifies to

For a comb topology, w.l.o.g. taking the topology (( (a,b),c),d), the expression reduces to

and the same formula applies if c and d are swapped.

Decomposition of covariance

We begin by defining the total covariance of METAL distance estimates as

8

where . The following proposition shows how the total covariance can be decomposed as a coalescent and substitution covariance.

Proposition 5

The total covariance can be expressed as

9

where

10

11

and are positive semi-definite matrices.

The decomposition of Proposition 5 arises from conditioning on the latent gene trees and utilizing the law of total covariance: variation across loci due to random coalescent histories contributes the term (scaled by 1/m), while conditional variation in the site patterns within each gene tree contributes the term (scaled by 1/(mK)). Although both matrices are influenced by the coalescent and substitution processes, we use the terminology “coalescent covariance” and “substitutional covariance” to highlight which source of randomness dominates each component: reflects the variability across gene trees even with infinitely long sequences, while reflects the additional noise from having only finitely many sites per gene.

Expressions for defined in Eq. (10) have been derived in the proof of Proposition 2. Specific expressions for defined in Eq. (11), for all tree topology cases can be found in Appendix B.

The decomposition in Proposition 5 sets the stage for Proposition 6, in which we specialize to the case of a star-shaped species tree and derive explicit formulae and asymptotics for each component of covariance. These results allow use both to compare the magnitude of coalescent versus substitution uncertainty and to study how their ratio varies with the mutation rate μ and tree diameter Δ . While the star-tree setting of Proposition 6 does not capture the full complexity of species-tree inference, it provides a tractable case that highlights the relative contributions of coalescent and substitutional variance.

Proposition 6

Suppose that the species tree is a star tree with diameter Δ i.e. . Then

for with

where m is the number of loci and K is the number of sites per loci.

Moreover, , and

for all and consequently

Proposition 1 shows that in the low-mutation regime (), STEAC and METAL agree to first order, since However, a fundamental drawback of STEAC is that, for any positive value of μ , the distribution of the estimate has strictly positive probability of being infinite, because can exceed 3/4 with nonzero probability. Consequently, the variance of the STEAC estimator is infinite for all μ . This implies that with many loci the STEAC average is highly unstable: a single locus producing an infinite estimate suffices to drive the average to infinity. Although one can truncate or discard these infinite estimates, doing so introduces downward bias by systematically removing the highest inferred branch lengths. We conjecture that, in the presence of substitution uncertainty–i.e. when gene trees must be estimated rather than known a priori–STEAC is uniformly inferior to METAL for all μ ; indeed, STEAC was originally developed under the assumption of almost perfect gene-tree knowledge. We assess this conjecture empirically in Sect. “Species-tree estimation”.

Principal components of variance

As we have seen in Proposition 6, the coalescent, substitution and total covariance matrices of the METAL estimator for the star species tree only depend on . In this subsection we study the spectrum of these matrices. Specifically, for a covariance matrix , with entries

12

for pairs where , Proposition 7 derives its spectrum.

Proposition 7

The matrix , defined in Eq. (12), has eigenvalues

• the dominant eigenvalue with multiplicity 1 and corresponding first principal component

• with corresponding principal components dependent on n and independent of , and

• with multiplicity n(n-3)/2.

Note that these eigenvalues are presented in decreasing magnitude and that C is positive definite if and only if . It follows that the variance ratio of the principal component tends to as. The variance ratio of the first n most dominant eigenvectors tends to as

Proposition 7 shows that each covariance matrix in our model has exactly three distinct eigenvalues. In particular, the all-ones vector is an eigenvector corresponding to the largest eigenvalue. From the perspective of tropical geometry, plays a canonical role: the tropical projective torus is precisely the quotient that identifies any vector with its translate for all real c. Equivalently, adding a constant to every coordinate does not change the point in the toric quotient. Statistical analysis on the tropical projective torus has been explored in various works, e.g. [18–20].

Corollary 1 (to Proposition 7) further quantifies that up to one-third of the total variance of our distance-vector estimates lies along the -direction. Since the tropical projective torus “modes out” precisely the -direction, restricting our inference to removes this single principal-component subspace. As a result, the remaining variance–now confined to the -dimensional tropical torus–drops by up to one-third. Working in this quotient removes a large portion of extraneous variation while preserving all information relevant to tree topology, since adding any constant multiple of to the distance vector does not alter the inferred tree topology.

Moreover, Corollary 1 shows that when , the relative magnitude of substitution versus coalescent uncertainty scales with μ K, the expected number of substitutions per locus: as μ K increases, substitution noise becomes negligible. In particular, if

13

then the coalescent covariance term dominates the total uncertainty. Note that as the number of bases per gene K increases, the interval widens on both ends, whereas increasing Δ narrows the interval from the upper bound. Figure 3 confirms this behavior: substitution-driven variance is largest for very small and very large μ , whereas for intermediate μ the coalescent variance is predominant. Figure 4 plots the ratio of Frobenius norms , which attains its minimum at when Δ is large. Finally, Fig. 5 displays the cumulative variance explained by the leading n principal components of . Remarkably, for and large Δ , these n components capture nearly all of the METAL estimator’s variance, while for larger μ the explained fraction vanishes, in agreement with Proposition 8.

Fig. 3.

Frobenius norms of , , and as functions of the mutation rate μ , shown for three values of the species-tree diameter Δ with K = 1000. For and for , the substitution variance dominates; in the intermediate μ range, the coalescent variance prevails. As Δ increases, the interval in which dominates becomes narrower

Fig. 4.

Ratio plotted against μ for several values of Δ . Substitution variance () dominates when or , whereas coalescent variance dominates in the intermediate μ regime

Fig. 5.

Variance ratios for plotted against mutation rate μ and species-tree diameter Δ . (Left) Fraction of total variance explained by the first principal component. (Right) Fraction of total variance explained by the first n principal components. For corresponding plots of and , see Fig. 13 in the Appendix

It is worth mentioning that Proposition 8 further establishes that as , the limiting coalescent correlation matrix, as defined in Eq. (14), has rank n-1, and its nonzero principal components are the split-indicator vectors which are exactly the generators of the unique maximal cone of the ultrametric fan that contains the species-tree ultrametric. In other words, all variation of the METAL distance vector lies within the cone corresponding to the true species-tree ultrametric.

Corollary 1

Consider the matrices defined in Eqs. (8), (10), (11) for a star species tree with diameter Δ . They have the following asympototic properties

As and , the first principal component PC1=1 explains 2/(6+3Δ ) of the variance of and , and 2/9 of the variance of , while the first n principal components explain and 4/9 of the variance respectively.

Proposition 8

Consider a species tree S with positive pair distances , where Δ is the diameter of the tree and . From the covariance matrix defined in Eq. (10), if we construct the correlation matrix , where and consider the limiting case for large trees, then we let

14

Let be an internal node of the species tree S, and the sets are the sets of leaf nodes left and right of i respectively. Define the vector,

which are also known as split-indicator vectors. Then, i.e. is an eigenvector with corresponding eigenvalues . These are exactly n-1 principal components corresponding to distinct internal nodes in V\backslash L . The remaining eigenvalues are zero, and so . Finally, let the species tree pairwise distance vector be an ultrametric i.e. and suppose it does not lie on topological boundaries. Then, there exists such that for all

Finally, , where the correlation matrices are defined as above, and is the identity matrix.

Simulation results

Two applications of the derived covariance matrix in phylogenetic analysis are considered in this section. In the first application (Sect. “Species-tree estimation”), our derivations of the covariance matrix enable us to quantify uncertainty from the substitution model, which predicts the relative performance of GLASS against METAL. In the second application (Sect. “Bipartition confidence”), the covariance matrix is used to compute node-support values as confidence for each bipartition of the estimated METAL tree. We compare our method to the standard bootstrap approach. Our repository can be found in [21].

Species-tree estimation

We begin by simulating data under the multi-species coalescent (MSC) combined with a site-substitution model. Using default parameters

we first generate a species tree of diameter Δ (in coalescent units) via the Yule model. Then m gene trees are sampled under MSC on that species tree. Finally, sequences of length K are evolved along each gene tree under a substitution model with rate μ , producing m alignments.

Under these default conditions, the simulated alignments contain approximately 0.3% constant sites, corresponding to a relatively high mutation rate (μ = 1). Although this rate is higher than what is typically observed in empirical datasets, we use it deliberately to keep the simulations tractable. Using a more realistic mutation rate (e.g., μ = 0.01) would result in over 99% constant sites. In simulated data, compensating for this would require increasing the number of sites per genes Kby two orders of magnitude, making simulations computationally impractical. Moreover, the additional constant sites would contribute no information to any inference method. Hence, we adopt the higher mutation rate solely to ensure that the simulated datasets contain a reasonable proportion of informative sites, and not because the method requires unrealistically high mutation rates in practice.

From these m alignments, we infer the species tree using three approaches (see Algorithm 2): STEAC, METAL, and GLASS. In STEAC and GLASS, each gene’s pairwise Hamming distances are transformed via the inverse Jukes–Cantor formula to branch-length estimates . STEAC then averages across all m genes before applying UPGMA. GLASS takes the minimum branch gene tree branch lengths before applying UPGMA. METAL concatenates all m alignments, computes Hamming distances on the full superalignment, and reconstructs a UPGMA tree from that distance matrix. Here, UPGMA is an appropriate choice for phylogenetic inference since it will return ultrametric trees, compatible with our covariance decomposition.

Figure 6 illustrates how the relative accuracy of GLASS versus METAL depends on the fraction of variance contributed by the substitution model. As this ratio increases, METAL (which explicitly pools substitution noise across genes) consistently outperforms GLASS. To quantify reconstruction accuracy, each inferred tree is compared against the true species tree via the Robinson–Foulds (RF) distance. Over R replicates (here R=100), we obtain RF distances . The mean RF error for each method is

We consistently observe

so STEAC underperforms METAL in all tested regimes (see Fig. 14 in the Appendix). In contrast,

increases with . When substitution variance dominates, GLASS yields higher RF error than METAL. However, if coalescent variance from MSC is the primary source (for instance, when K is large or μ lies in an intermediate regime), GLASS approaches the Kullback–Leibler-optimal species-tree estimator and outperforms METAL, i.e.

In summary, the ratio

determines the preferable method: if this ratio exceeds a threshold (when μ is very low or very high, Δ is large, or K is small), METAL–by aggregating substitution noise–yields lower RF error. Otherwise, for moderate μ and large K, the coalescent term dominates and GLASS (leveraging accurate gene-tree estimates) gives superior accuracy.

Fig. 6.

Comparison of GLASS and METAL performance as a function of the ratio . When substitution-model variance constitutes a large fraction of total variance, METAL outperforms GLASS

Fig. 14.

This Figures compares STEAC to METAL, similarly to Fig. 6. In all cases METAL outperforms STEAC

Bipartition confidence

The procedure begins with data generation: we first simulate a species tree, then simulate gene trees under the multi-species coalescent (MSC) model using DendroPy [22], and finally perform sequence evolution along those gene trees using Pyvolve [23]. This pipeline is illustrated in Fig. 1.

Once we have simulated data, we obtain the METAL estimate by computing Hamming distances between the concatenated sequences. These distances serve as pairwise dissimilarities, and we reconstruct a tree via the UPGMA algorithm, following the METAL methodology.

Our main objective is to assign confidence scores to tree splits (bipartitions). The standard approach uses bootstrapping: sites from the concatenated alignment are resampled (with replacement) to generate pseudo-replicate alignments. For each replicate, we compute a new METAL tree, yielding a collection of bootstrap trees. The proportion of replicates in which a given split appears provides its bootstrap support.

As an alternative, we propose a Gaussian-sampling approach. The METAL estimator of the average pairwise distances can, for a sufficiently large number of genes, be approximated by a multivariate normal distribution. The mean of this distribution is the METAL estimate itself, and the covariance is derived from the METAL tree (in coalescent units to scale Hamming distances appropriately). The quality of this normal approximation improves as the number of genes increases.

We draw samples from this multivariate distribution to obtain synthetic vectors of pairwise distances. Each sampled distance matrix is then converted into a tree via UPGMA, exactly as in the bootstrap procedure. We estimate split support by computing the frequency of each bipartition across these “Gaussian-sampled” trees.

Figure 7 provides an illustrative comparison. The leftmost tree is the METAL estimate with bootstrap-derived support values, the middle tree shows the same METAL topology but with support values from Gaussian sampling, and the rightmost tree is the true species topology. Note that Gaussian sampling typically yields more conservative (lower) support values than bootstrapping.

Fig. 7.

Example of METAL estimate (left) with bootstrap support (percentages at each node), METAL estimate (center) with Gaussian-sampling support, and the true species tree (right). Bootstrap supports tend to be higher than Gaussian supports, but may also overestimate confidence for incorrect splits

To compare the two confidence-estimation methods quantitatively, we exploit the fact that the true species tree is known in simulation. With n = 40 taxa, there are n - 2 = 38 non-trivial splits. We label each split in the METAL estimate as correct (1) if it appears in the true tree, or incorrect (0) otherwise. Using the support scores assigned by each method, we compute the area under the ROC curve (AUC) to measure how well each set of support values discriminates correct splits from incorrect ones.

This experiment is repeated 100 times. In each series of 100 replicates, we vary exactly one simulation parameter (e.g., mutation rate μ ) while holding all other parameters (species-tree diameter Δ , number of sites per gene K, and number of genes m, number of taxa n) fixed. We then repeat separately for each of the other parameters (Δ , K, m, and n), always keeping the remaining factors constant.

Figure 8 shows boxplots of AUC scores for three methods: standard bootstrapping (blue), Gaussian sampling (green), and multilocus bootstrapping (magenta). In the left panel, we vary the number of genes m; in the right panel, we vary the number of sites per gene K. In both cases, Gaussian-sampling and multilocus bootstrapping yield higher median AUC and lower variance than standard bootstrapping. Similar figures for the other parameters can be found in Appendix D. What is interesting about parameters m and K is that the concatenated sequence length has mK bases. As the number of genes m or the number of sites per gene K increases, the runtime of standard and multilocus bootstrap procedures grows roughly in proportion to m · K, the size of the concatenated sequence. In contrast, the computational cost of the Gaussian-sampling approach is essentially unaffected by changes in m or K, since it depends only on the number of taxa n. Moreover, although standard bootstrapping requires more time as m and K increase, its ability to correctly classify true splits does not improve–in fact, its classification accuracy becomes more erratic. Nonetheless, for low values of both m and K, the results of gaussian sampling can be slightly worse than those of multilocus bootstrapping as can be seen in Fig. 17 of Appendix D.

Fig. 8.

Boxplots of AUC values comparing standard bootstrap (blue), Gaussian sampling (green), and multilocus bootstrap (magenta) support scores. (Left:) AUC distributions as the number of genes m varies (with μ , Δ , n, and K held constant). (Right:) AUC distributions as the number of sites per gene K varies (with μ , Δ , n, and m held constant). Gaussian sampling and multilocus bootstrapping yield similar median AUC and lower variability than standard bootstrapping, while Gaussian sampling retains a clear computational advantage

Fig. 17.

Boxplots of AUC values for different pairs (m, K) of number of genes and number of sites per gene, increasing together. For low values of both m and K, the results of gaussian sampling can be slightly worse than those of multilocus bootstrapping

Computational complexity comparison

For bootstrapping, each replicate requires recomputing the METAL tree from a resampled alignment of total length m · K. Computing all pairwise Hamming distances is , since there are taxon pairs. To ensure METAL is accurate with probability 1 - ɛ , one needs

where τ is the shortest internal branch length. Since , one obtains . Hence each bootstrap replicate costs , and obtaining B replicates is B times that.

In contrast, the Gaussian sampling approach requires computing a covariance matrix over

entries, resulting in a full covariance matrix of size N × N, containing elements. Computing the Cholesky decomposition of this matrix takes time, but only once. Each subsequent multivariate-normal sample (to produce one synthetic distance vector) costs . Crucially, this Gaussian method is independent of the sequence length K, making it far more efficient when K is large.

Nonetheless, when the number of taxa is very large (n ≫ 1), the one-time cost of the Cholesky decomposition becomes prohibitive, and bootstrapping may be more efficient. However, if the mutation rate is low or if , we can instead leverage the theoretical results of Sect. “Theoretical results”, specifically Corollary 1 and Proposition 8. These results describe the asymptotic spectrum of the total covariance matrix of the METAL estimator. In both of these limiting cases, we have shown in Proposition 6 that , and so that . For , there are three distinct eigenvalues with known variance ratios. As illustrated in Fig. 9, the asymptotic predictions of the eigenvalues of Corollary 1 closely match the empirical ones of a randomly generated species tree with . Knowing the principal components in advance allows us to bypass the Cholesky decomposition entirely, eliminating the most computationally expensive step of the method.

Fig. 9.

Explained variance by principal components of the total covariance matrix for a species tree with . The exact curve (blue) shows the empirical proportion of variance explained by each principal component, while the "asymptotic" curve (orange) represents theoretical predictions from Corollary 1. The close agreement validates the asymptotic spectrum in the low mutation rate regime ( )

Discussion

In this work, we explicitly formulate the covariant matrix of all pairwise distances under the MSC and substitution models for the species tree reconstruction problem. METAL constructs a distance matrix from concatenated multi-locus data [11], and then applies a clustering algorithm to infer the species tree. Our first main result is the derivation of the exact covariance matrix of METAL’s pairwise distance estimates. This covariance naturally decomposes into a term arising from coalescent genealogy variance and a term from sequence-substitution variance. We give explicit formulas for both components. We then analyze the asymptotic regime as the number of loci grows large; we show that METAL distances concetrate around expectations, and we study the spectrum of the limiting covariance matrix. The spectral analysis suggests that much of the phylogenetic signal lies in the first few principal components of the distance matrix when the number of leaves is large.

Our simulation study illustrates how the relative performance of distance-based methods depends on the underlying source of variation. In particular, METAL tends to outperform GLASS when the substitutional noise dominates, while GLASS becomes more accurate as coalescence variance takes over. This sensitivity reflects the design of each estimator; METAL effectively aggregates substitutional uncertainty across loci, whereas GLASS leverages gene-tree precision when it is available. Likewise, our results show METAL retaining strong performance even under rate variation and short branches. Finally, we introduce a new Gaussian-sampling procedure to estimate METAL split-support values, which uses the derived covariance to quantify support for inferred clades.

There are several natural extensions of our work. Our current analysis assumed a strict molecular clock, so that all lineages evolve at the same rate and that the gene and species trees are equidistant. Relaxing this assumption is important for real data and would allow us to combine our modular method with other phylogenetic inference schemes such as balanced minimum evolution [24]. Generalizing our covariance derivation to non-clock settings would be a major next step of our current analysis. Similarly, there is value in exploring other choices of distance, such as logDet distances [6], which may reduce bias in estimation. Moreover, although we derived the expected four-point gap of a quartet (i.e., the signal) in Sect. “Expected gap”, we have not yet considered the corresponding signal-to-noise ratio. Quantifying this ratio could provide insight into the likelihood of correctly reconstructing the species tree, as a higher signal relative to noise should generally increase reconstruction accuracy.

In conclusion, we have offered a systematic decomposition of the covariance structure in distance-based methods like METAL and STEAC, separating the contributions of gene-tree (coalescent) variability from sequence-level (substitutional) noise. Our framework not only illuminates how these two sources of uncertainty compare to each other, but also lays the groundwork for extending this analysis to a broader class of estimators. An important direction for future research is to determine the extent to which our findings on the balance between coalescent and substitutional effects hold under alternative models and estimators. Addressing this question will deepen our understanding of estimator performance across diverse biological scenarios.

Appendix A Proofs

Proof of Proposition 1

The definitions of the dissimilarity metrics can be found in Equations (1)–(3). We start by considering low mutation rates. Since,

and by expressing Eq. (3) as the partial sum of terms , it follows that

Therefore, in the limit , the pairwise METAL distances become proportional to the STEAC distances. Since hierarchical clustering dendrograms are invariant under scalar multiplication of the distance matrix,

For high mutation rates, note that if the GLASS minimum coalescence times are unique, i.e. for any given pair of leaves there exists such that for all , then

In other words,

Note that is a strictly increasing function. Hence, for a sufficiently large μ ,

Since the number of combinations a,b,c,d ∈ L is finite, there exists a sufficiently large μ such that the above equation holds for all a,b,c,d ∈ L. Hence, the asymptotic ordering of METAL distances agrees with the order of GLASS distances. Another way of reaching the same conclusion is using the monotone admissibility of hierarchical clustering methods such as single linkage or complete linkage [25], which allows monotonic transformation of the elements of the distance matrix without altering the clustering. □

Proof of Proposition 2

Instead of writing to refer to locus 1 specifically, we write for the remainder of the proof. First, we address the trivial case where a=b or c=d. Assume that wlog a=b, then and , which implies that

which implies that the corresponding correlations will be zero too, validating the results immediately in this case.

For the rest of the proof, assume that a ≠ b and c ≠ d.

There are four cases to consider regarding the choice of the two pairs.

• Two leaves only, where wlog a=c and b=d.

• Three leaves only, where wlog d=a and .

• Four leaves forming a cherry tree, where .

• Four leaves forming a comb tree, where .

These are the only cases to be considered; if a, b, c, d do not satisfy them, they can always be permuted so that at least one of the conditions above are satisfied.

Fig. 10 illustrates those four cases.

Fig. 10.

The four possible shapes of tree containing a,b,c,d ∈ L with

Two leaf tree   For the case where (a,b) = (c,d), the covariance expressions becomes variances, which are positive. Under the MSC , where , and so

A1

A2

where for the last result we use that the moment generating function (MGF) of is and .

For the remaining three cases, we consider the variable

where v, w, x, y ∈ L , and its moment-generating function (MGF)

The moment-generating function allows access to the covariances of interest, as the two auxiliary results below (Eqs. (A3),(A4)) show. We now show these results.

For each pair (i,j), define

with , and hence . Writing

we have

But,

Therefore

Noting that recovers

A3

Similarly, one shows that

A4

Indeed, since with ,

and

where is the MGF of , so . Subtracting gives (A4).

Three leaf tree Consider the case of a three-leaf tree where , where as shown in the second leftmost tree in Fig. 10. Define the joint random variable , where T is the first time that a pair coalesces and is that pair. In other words,

Since the first speciation event happens at time , we require that . If , then . The joint distribution function of is

where for the remainder of this proof is the set containing pairs of distinct elements of the set .

Note that , but since there are three choices of pairs, the pdf of is divided by 3 for each pair in the joint pdf above. Moreover, the conditioning probability is included.

Conditional on ,

and so

Finally, combining those two results gives us the MGF,

Taking the second derivative of this expression with respect to t at t=0 yields

and so, using Eqs. (A3), and (A4), it follows that

which are all positive. Note that the reason these results hold for the pairs (a, b) and (b, c) is that , since a and b are interchangeable. To get the correlations, we need the expression of the variances from Eqs. (A1), (A2), which yield

A5

A6

We also need to examine which is distinct from the other two cases. Here,

Following the same approach as before,

Hence, it follows that

A7

A8

A9

A10

Cherry tree We continue by considering the cherry tree as shown in Fig. 10, with , where Δ is the diameter of the subtree containing a, b, c, d. We use this for notational convenience for the remainder of the proof and should not be confused with the diameter of the species tree contatining all leaves. We also defined, and . Once again, define the joint random variable as before. For the cherry tree case, it has density function

Conditional on and the species tree S,

Note the last case where the first pair to coalesce is neither (a, b) nor (a, c). Let’s assume without loss of generality that the first pair was (a, c). Since species a and c have coalesced, we can treat them as one leaf a=c in a gene tree of 3 leaves a=c, b,d that has not been resolved yet. This gene tree is generated by the Multispecies coalescent model assuming that the species tree is the star tree, denoted S=*, since all three species can coalesce after . Therefore,

We have already computed the MGF of E for three-leaf trees. When S=*, the internal branch length vanishes i.e. τ = 0. Hence,

Hence,

From this we conclude that

A11

A12

A13

A14

A different approach is used for finding the MGF of . Define the random variable to be the number of pairs that coalesce before time Δ /2. The only pairs that could coalesce in that time interval are or (c, d) and the probabilities of each one of these pairs coalescing is independent of the other. Therefore,

If N=0, then i.e. the MGF of will be the same as with or

If N=1, either (a, b) or (c, d) have coalesced before time Δ /2 but not both, and so at time Δ /2 there will be three nodes that have not coalesced yet. Consequently, i.e. the MGF of will be the same as with τ = 0 or

For N=2, both (a, b) and (c, d) have coalesced, , and thus

Combining those three results,

The same argument can be made for the pair (a, d) and (b, c), since a, b are interchangeable. We conclude that

A15

A16

We now need to prove that the correlations in Eqs. (A15) and (A16) are non-negative and that the correlation of is an increasing function of . Note that since,

proving the monotonicity of on (-∞ , 0] is sufficient for the positivity of (A15) and (A16).

To do that, we compute the derivative,

The first inequality comes from the observation that

The second inequality stems from the fact that the function

has non-negative range, which can be proved by rewriting g as

which is clearly minimized at x,y = (1,1), and so . This concludes the proof for the case of the cherry tree.

Comb tree Finally, we conclude this proof with the last tree shape of Fig. 10, the comb tree, where and .

The joint density function of is

Furthermore,

where the three-leaf tree , for τ >0, is drawn in Fig. 11.

Fig. 11.

Three-leaf tree with internal branch length τ . Note that

The MGF of and has already been computed before (three-leaf tree) as

Hence,

and so

A17

A18

Note that is an increasing function of t on (-∞ ,0), since the first term is increasing and is also increasing.

The last case follows from the fact that a, b, c are interchangeable conditional on (a, b) not coalescing before time . Moreover, note that has MGF equal to the above , but with i.e.

Hence, it follows that

Also note that . It follows that the correlations are as follows

A19

A20

Similarly, the correlation is an increasing function of t. This concludes the proof for the case of the comb tree. □

Proof of Proposition 3

The covariance of interest can be rewritten as follows, using the properties of indicator random variables,

A21

Under the Jukes-Cantor evolutionary model, when there is a substitution, it is equally likely that the current base is substituted by any one of the four bases A, C, T, G. Since the probability of no substitution across a branch of time length t is , the probability that a randomly selected base stays the same across the same branch is

It follows that,

A22

The second term of Eq. (A21) can then be computed through a product of terms computed from Eq. (A22). We now focus on the first term of Eq. (A21).

Let r and q be the internal nodes connecting a and c and b and d respectively, such that the length of

the path from q to r is equal to . Similarly to Eq. (A22),

A23

We replace the states A, C, G, T with numbers 0, 1, 2, 3 respectively. Define the random variable and note that

Moreover, are all independent of each other since their corresponding paths are disjoint. Hence, is independent of . For the remainder of the proof \mod 4 will be dropped when ≡ is used.

Conditioning on ,

A24

where the third equality holds because of the independence of and , and the fourth equality holds because all the terms in the summand are equal and each probabilities involving sums of N being ≡ i are a third of the corresponding probabilities with .

Now the only thing left is to find the distribution of and . Without loss of generality, we will focus on the former. The probability that there are no substitutions across the paths from a to r and q to b is , since the length of the sum of the paths is . If there has been no substitution along those paths, then . Otherwise, if there has been at least one substitution, , where i=0,1,2,3 are all equiprobable. Therefore,

A25

A26

Substituting Equations (A25)–(A26) and Eq. (A23) into Eq. (A24) gives

Substituting the above to Eq. (A21) yields the desired result. □

Proof of Proposition 4

Under the Jukes–Cantor model, the expected estimated distance satisfies , where is the gene-tree distance between x and y and μ is the substitution rate. Under the multispecies coalescent (MSC), we may write , where is the species-tree path length and is the coalescent waiting time after speciation. Hence,

The constant terms (the leading 1’s in each ) cancel in the four-point expression, giving

and substituting the expectation above yields (7). For the cherry topology, , which gives the stated simplification. For the comb tree case with topology (((a,b),c),d) , and , and so the expression collapses to . □

Proof of Proposition 5

Using the law of total covariance, the total covariance between pairs of METAL distance estimates for a single loci is

27

To derive the last line we use Proposition 3 for the first term and the Jukes-Cantor formula

for the second term. Using the definitions in Equations (8), (10), (11), Eq. (9) is equivalent to (27).

It is easy to see that is positive semi-definite, as a covariance matrix of the random vector . For , observe that

is positive semidefinite i.e. for any vector , and so

This proves that is a positive semi-definite matrix. □

Proof of Proposition 6

Using the formulas from the Proof of Proposition 2 for the coalescent covariance matrix and the formulas from Appendix B for the substitution covariance matrix. Observe that for the star species tree, the three leaf covariances agree with each other for both the coalescent and the substitution matrix. So do the results for the four leaf covariances. In other words, and only depend on the total number of leaves . Also, note that and so now refer to 4-leaf, 3-leaf, and 2-leaf covariances respectively.

First, we derive the formulas of for .

A28

A29

A30

A31

A32

A33

The inequalities are derived by using for all i ≤ 2. From here, the asymptotic results as and are easy to derive. It is also clear that is decreasing with , and that . We now prove that ,

This concludes the proof. □

Proof of Proposition 7

1 First we find the spectrum of the following matrix;

We need to prove the following results for A

• i.r(A) ≤ n.
• ii.Take two distinct leaves x,y ∈ [n]. Consider the vector with , and for all distinct z,w ≠ x,y. Then is an eigenvector of A with eigenvalue n-2. In particular, the eigenvectors are linearly independent and so , where is the eigenspace of matrix Y associated with eigenvalue ρ .

The first result shows that the dimension of the eigenspace is at least by the rank-nullity theorem. The second result shows that . The dominant eigenvalue is 2n - 2 (with corresponding eigenvector ) and the corresponding eigenspace has dimension at least 1. Since the sum of the dimensions of eigenspaces is n \atopwithdelims ()2, it follows that

Proof of i. First, observe that the sum of rows is a covariance under MSC derived , since

Hence, it suffices to prove that the space spanned by the first n-1 rows for along with the row (1,… ,1) (which is the sum of all the rows of A) contains all the other rows for x, y ≠ 1. To do that, our goal is to write as a linear combination of the first n-1 rows the vector (1,… ,1) for any x,y >1, and proving element-wise that it holds. Consider the coefficients , corresponding to row , where λ is a free parameter to be determined. Also let -2 λ be the coefficient corresponding to the newly created row . Then, for all components zw with z ≠ w and z,w ∈ [n],

For the last equality to hold, note that first term vanishes if we choose and the summation term is equal to . Hence, we have proved the following result

which means that for all distinct x,y ∈ [n] we have . Hence, the rank of A cannot exceed n.

Proof of ii. We now need to verify that . First, observe that

and so it follows that

where is a column vector i.e. is a matrix of all-ones. Given that , we conclude that

which implies that the columns of are all eigenvectors of A with eigenvalue n-2. Note that i.e. the vectors constructed in statement (ii) are precisely the columns of V and hence elements in the eigenspace .

To prove that are linearly independent, consider the square matrix with . It suffices to prove that B is non-singular. Note that for all j ≠ i and and denote them as for all where N=n-1 and x> y > 0. Observing that B is a circulant matrix, its eigenvalues are

where , and ω is the -root of unity. Hence, and for . All eigenvalues are positive and so B is non-singular. Having found the spectrum of A, the task is to now find the spectrum of C. Observe that

where is a matrix whose entries are all 1.

Every eigenvalue and eigenvector that is not is transformed

In particular, if Av = λ v, then . This argument doesn’t work for since . But it’s straightforward to compute the eigenvalue directly

The eigenvalue n-2 of A becomes and the corresponding eigenvectors do not change because for all the vectors in . The remaining eigenvalue is . □

Proof of Proposition 7

For the trace and Frobenius norm, the results of Propositions 6 suffice to prove them. For the determinant, the spectral norm, and the results about variance ratios, the spectrum of those matrices is given by Proposition 7.

Regarding the variance ratio, note that Proposition 7 provides limits of variance ratios as . The variance ratio of PC1 is and the variance variance of the first n prinipal components is . For the covariance matrix where mode is either coal or sub, .

We start with , whose components are given by Equations (A28)–(A30). As ,

From that it follows that

Similarly, for , as ,

From that it follows that

This concludes the proof of the Corollary. □

Proof of Proposition 8

Two statements need to be proved; first, that is an eigenvector of for all interior nodes i, and second, that the remaining eigenvalues are zero. Observe that only takes values . The value of is 1 if and only if the topology of the leaves is either ( (a, c), (b, d) ) or ( (a, d), (b, c) ). Hence,

To prove the second statement, it suffices to prove that where the sum is over all internal nodes; that’s because the trace is the sum of eigenvalues and is a symmetric semi-positive definite matrix. The diagonal of only includes ones, and so . Let r be the root of the tree with . Denote be the sum of eigenvalues (corresponding to interior nodes) for the tree formed from the leaf set . It follows that

Using strong induction, we write and . Substituting those to the formula above yields . That concludes the inductive step. □

Appendix B Covariance calculations

Recall that using Eq. (27) we decompose the covariance into an MSC and substitution component. Formulae for the former can be found in the Proof of Proposition 2. In this section of the Appendix, the task is to compute the latter component, namely the substitution covariance matrix for all four subtree topology cases shown in Fig. 10. This substitution component can be further decomposed as follows

where is the covariance under MSC derived in Proposition 2. Therefore, the task now is to compute and for all different tree topologies.

B.1 Computing terms

First, we will state all the results for , and then provide their proofs.

• Two-leaves. If , then and so

  
• Three-leaf tree with tree topology ((a, b), c) has

  
• Four leaves (Cherry tree) with tree topology ((a, b), (c, d)) has

  
• Four leaves (Comb tree) with tree topology (((a, b), c), d) has

  

Here and for the remainder of the Appendix, Δ is the diameter of the subtree with leaves a, b, c, d and should not be confused with the diameter of the total tree, which may include more leaves and thus be larger. A more accurate notation would be , but we will write Δ for convenience.

For the 2-leaf case, note that and so

where under the MSC, whose moment generating function is 1/(1-t) which gives the 1/(1+2μ ) term in the expression above.

For the 3-leaf case, note that

which is equal to the sum of the branches of the gene tree containing the three leaves , regardless of the topology of the gene tree. Let T be the first coalescent time in the gene tree i.e. . Then, the total branch length of the gene tree generated by is

where is independent of T. Figure 12 illustrates this result.

Fig. 12.

The total length of this gene tree is if T < Δ / 2 (left) and if (right)

The pdf of T is

Therefore, we have that

B34

as stated for the 3-leaf tree.

For the 4-leaf cherry tree with topology ((a, b), (c, d)), again denote T as the first coalescence time and without loss of generality let . We also define the random variable as the sum of branch lengths of a 3-leaf gene tree, whose species tree is the star tree with zero diameter, and whose MGF is a special case of Eq. (B34) with , i.e.

B35

The central idea of this derivation is that after 2 leaves have coalesced into one, the problem is reduced to a 3-leaf case that is already computed. Note that if , then either the first pair to coalesce is either (a, b) or (c, d). Either way, . Conditioning on T and the corresponding first coalescent pair ,

where and are independent of T. It follows that,

which yields the desired result, once we substitute Eq. (B35) and note that . Similarly, note that

where is independent of T. It follows that

Note that since a and b are interchangeable in this tree topology, . This concludes the proof for the cherry tree case.

Finally, for the 4-leaf comb tree case with topology (((a, b), c), d), and noting that

where is independent of T, the desired results are derived by following the same process in the cherry tree case.

B.2 Computing terms

First, we will state all the results for , and then provide key ideas for their proof.

• Two-leaves. If , then and so the random variable in the exponent is zero, i.e.

  
• Three-leaf tree with tree topology ((a, b), c) has

  
• Four leaves (Cherry tree) with tree topology ((a, b), (c, d)) has

  
• Four leaves (Comb tree) with tree topology (((a, b), c), d) has

  

To prove the three leaf case, observe that

B36

In other words,

For the four-leaf comb and cherry case, we observe that

where is the first pair in the gene tree to coalesce and is the complimentary pair; the remaining two leaves. Conditioning on just like before yields the desired results.

Appendix C Algorithms

Algorithm 1 describes the procedure to compute the Hamming distance between two alignment sequences. This is a sub-routine within the distance-based approaches for species tree clustering - METAL, GLASS and STEAC - which are presented in Algorithm 2.

Algorithm 1.

Hamming Distance Between Two Sequences

Algorithm 2.

Species-Tree Estimation via Agglomerative Distance-Based Methods

Appendix D Additional plots

See Figs. 14, 15, 16, and 17.

Fig. 13.

Variance ratios for (top) , (middle) , and (bottom) , plotted against mutation rate μ and species-tree diameter Δ . In each row, the left panel shows the fraction of total variance explained by the first principal component, and the right panel shows the fraction of total variance explained by the first n principal components. Here, the number of sites per gene is K=1000. Since does not depend on Δ , its row contains only a single curve. For , note that when and (i.e., substitution variance dominates), its curves resemble those of , whereas for intermediate μ (when MSC variance dominates), all three models’ curves converge to that of the coalescent covariance

Fig. 15.

Boxplots of AUC values for different mutation rates μ (left), and different species tree diameter Δ (right)

Fig. 16.

Boxplots of AUC values for different number of taxa n

Author contributions

GA contributed theoretical work and computations. RY directed this project. BB and JG supervised GA. All authors reviewed the manuscript.

Funding

RY is partially funded by NSF Division of Mathematical Sciences: Statistics Program DMS 1916037. GA is funded by EPSRC through the STOR-i Centre for Doctoral Training under grant EP/L015692/1.

Data availability

DRYAD with DOI: https://doi.org/10.5061/dryad.tht76hf65

Declarations

Ethics approval and consent to participate

NA

Consent for publication

NA

Competing interests

The authors declare no competing interests.