Roadblocked monotonic paths and the enumeration of coalescent histories for non-matching caterpillar gene trees and species trees

Abstract

Given a gene tree topology and a species tree topology, a coalescent history represents a possible mapping of the list of gene tree coalescences to associated branches of a species tree on which those coalescences take place. Enumerative properties of coalescent histories have been of interest in the analysis of relationships between gene trees and species trees. The simplest enumerative result identifies a bijection between coalescent histories for a matching caterpillar gene tree and species tree with monotonic paths that do not cross the diagonal of a square lattice, establishing that the associated number of coalescent histories for n-taxon matching caterpillar trees (n ⩾ 2) is the Catalan number $C_{n-1}=\frac{1}{n}\left(\begin{array}{c}\hfill 2n-2\hfill \\ \hfill n-1\hfill \end{array}\right)$. Here, we show that a similar bijection applies for non-matching caterpillars, connecting coalescent histories for a non-matching caterpillar gene tree and species tree to a class of roadblocked monotonic paths. The result provides a simplified algorithm for enumerating coalescent histories in the non-matching caterpillar case. It enables a rapid proof of a known result that given a caterpillar species tree, no non-matching caterpillar gene tree has a number of coalescent histories exceeding that of the matching gene tree. Additional results on coalescent histories can be obtained by a bijection between permissible roadblocked monotonic paths and Dyck paths. We study the number of coalescent histories for non-matching caterpillar gene trees that differ from the species tree by nearest-neighbor-interchange and subtree-prune-and-regraft moves, characterizing the non-matching caterpillar with the largest number of coalescent histories. We discuss the implications of the results for the study of the combinatorics of gene trees and species trees.

1. Introduction

In the mathematical study of evolutionary trees, genetic lineages can be treated as evolving along the branches of a species phylogeny, a tree that represents the evolutionary relationships among a set of species [4, 12, 13]. A tree describing a set of genetic lineages that descend from a common ancestor is a gene tree, and a tree relating the species themselves is a species tree. Looking backward in time, in a gene tree of genetic lineages sampled from representative individuals of a given set of species, a pair of genetic lineages can coalesce, or find a common ancestor, only after the common ancestor of their species is reached. More generally, a set of two or more genetic lineages has a most recent common ancestor only after the most recent common ancestor of their associated species is reached.

The study of the relationship between gene trees and species trees—usually treated as binary, rooted, and leaf-labeled—has generated a number of novel combinatorial structures [3, 5, 6, 12, 20, 21, 25, 27, 28]. Among these are coalescent histories, structures that describe the possible locations on a species tree where the coalescences of a gene tree can take place [6, 16]. More precisely, for a (binary, rooted, leaf-labeled) gene tree topology G and a (binary, rooted, leaf-labeled) species tree topology S on the same set of taxa, a coalescent history f associates with each coalescence in G an edge of S, such that two properties are satisfied: (i) the species tree edge h(u) associated with a gene tree coalescence u is ancestral to all lineages that descend from u; (ii) for any pair of gene tree coalescences u, v for which u lies on a path from v to a leaf of the gene tree, h(u) lies on a path from h(v) to a leaf of the species tree. From a biological perspective, this pair of constraints encodes the rules that (i) gene lineages can coalesce only in a branch of the species tree in which it is possible for their ancestors to coexist, and that (ii) ancestors can coalesce no more recently than their descendants.

Rosenberg [16] provided a recursion that enumerates coalescent histories for arbitrary gene tree and species tree topologies. For gene tree topology G and species tree topology S, so that the taxon set of S is a superset of that of G but not necessarily the same set, let T(G, S) denote the minimal displayed subtree of S that contains all the taxa of G, that is, the subtree of S rooted at the node that corresponds to the most recent common ancestor of the taxa with the same labels as the taxa in G. Let d(G, S) ⩾ 0 denote the number of edges that separate the root of T(G, S) from the root of S. Let G_L and G_R denote the left and right subtrees of G. We define an integer parameter m ⩾ 1, and write a recursion for a function B_G,S,m:

$$B_{G,S,m}=\sum _{k=1}^m{B}_{G_L,T(G_L,S),k+d(G_L,S)}{B}_{G_R,T(G_R,S),k+d(G_R,S)},$$ (1)

The base case is obtained by setting B_G,S,m to 1 for all m in the case that G has only one taxon. With these definitions, the number of coalescent histories for gene tree topology G and species tree topology S is B_G,S,1.

Caterpillar species trees, in which an internal node exists that is descended from all other internal nodes, represent a special case in which enumeration of the coalescent histories is simpler than in the general case of arbitrary species trees. Thus, although exact and asymptotic results are known for certain additional shapes [9, 16, 18], enumerative properties have been explored most extensively for caterpillar species trees and shapes that closely resemble them [2, 6, 10, 16, 17, 19]. First, for a matching caterpillar gene tree and species tree—a caterpillar gene tree and species tree with the same labeled topology—Degnan [2] found a bijection between coalescent histories and monotonic paths on a square lattice that do not cross above the y = x diagonal, a quantity well-known to be described by the Catalan number sequence [23, item 24]. Eq. 1 recovers the Catalan numbers in this case [16, Corollary 3.5], and can be used to show that the number of coalescent histories for matching gene trees and species trees in small “caterpillar-like families” is asymptotic to a constant multiple of the Catalan numbers [16, 17]. This asymptotic behavior has been demonstrated for caterpillar-like families of arbitrary size using techniques of analytic combinatorics [10].

Enumerative results have been comparatively little studied, however, in the case that labeled gene trees and species trees disagree in topology. Than et al. [26] performed a numerical investigation, finding that the number of coalescent histories for non-matching gene tree and species tree topologies generally decreases with increasing subtree-prune-and-regraft (SPR) distance between the trees. Rosenberg & Degnan [19] demonstrated that for the caterpillar species tree topology with n ⩾ 7 taxa, there exists a non-matching gene tree topology with more coalescent histories than the matching caterpillar gene tree topology. Nevertheless, for caterpillar species tree topologies, Degnan & Rhodes [3] showed that no non-matching caterpillar gene tree topology can exceed the matching caterpillar gene tree topology in number of coalescent histories; indeed, the constructive example of Rosenberg & Degnan [19] of a non-matching gene tree topology with more coalescent histories than the matching caterpillar was not itself a caterpillar.

Here, we extend the monotonic path approach of Degnan [2] to non-matching caterpillar gene tree and species tree topologies. We show that coalescent histories for non-matching caterpillar gene tree and species tree topologies can be bijectively associated with a set of roadblocked monotonic paths that do not cross above the y = x diagonal of a square lattice. The approach immediately recovers the result of Degnan & Rhodes [3] that non-matching caterpillar gene tree topologies do not exceed the matching caterpillar gene tree topology in number of coalescent histories. It enables calculations of the number of coalescent histories for caterpillar gene tree topologies that differ from the species tree by common transformations—nearest-neighbor-interchange and subtree-prune-and-regraft. We characterize non-matching caterpillar gene trees with the largest numbers of coalescent histories, finding that the number of coalescent histories in such cases is asymptotically equivalent to that in the matching case.

2. Preliminaries

2.1. Caterpillar trees

We consider binary, rooted, leaf-labeled trees with leaf labels bijectively drawn from a label set X containing n distinct labels. For convenience, a “tree” refers to a binary, rooted, leaf-labeled tree. Trees contain two types of nodes, leaf nodes and non-leaf, or internal, nodes. Because trees are rooted, we say that a node v₁ of a tree G is descended from another node v₂ if the shortest path from v₁ to the root node contains v₂. We also say that v₂ is ancestral to v₁. Ancestor–descendant relationships also apply to pairs of edges and to pairs containing a vertex and an edge. A node or edge is trivially descended from itself, and it is also trivially ancestral to itself. The root node is an internal node.

We focus on caterpillar trees, trees in which there exists an internal node descended from all other internal nodes (Figure 1A). A caterpillar tree has exactly one cherry node, a node with exactly two descendant leaves. Among leaves, the longest path length to the root of a caterpillar tree with n leaves is n – 1.

Figure 1.

Transformations of caterpillar trees. (A) A caterpillar tree G₁. The vector of labels for G₁, in canonical order, is (A, B, C, D, E, F, G, H, I, J). The adjacent pairs of leaves are (A, B), (A, C), (B, C), (C, D), (D, E), (E, F), (F, G), (G, H), (H, I), and (I, J). (B) A tree G₂ that differs from G₁ by nearest-neighbor-interchange. Leaves E and F are exchanged. (C) A tree obtained from G₁ by forward incrementation of leaves C, D, and E. (D) A tree obtained from G₁ by reverse incrementation of leaves C, D, and E. The tree in (C) can also be viewed as the result of a subtree-prune-and-regraft operation, with the branch leading to leaf E pruned and regrafted; the tree in (D) can be viewed as the result of an SPR operation involving the leaf leading to C. In each panel, the red line indicates which leaves are permuted.

The number of distinct caterpillar trees possible for a label set X with n distinct labels is n!/2: the leaf separated from the root by only one edge has n possible labels, the leaf two edges from the root then has n – 1 possible labels, and so on. In this assignment of labels, the leaves descended from the cherry node are exchangeable. Hence, only one labeling is possible for these leaves, giving a total of n(n – 1)(n – 2) ×⋯× 3 = n!/2 labelings. These labelings represent the n!/2 caterpillar labeled topologies for label set X.

For convenience, we organize the labels in an n-leaf caterpillar tree G canonically in a vector g of length n. For i = 3, 4,…, n, entry i in the vector is the label of the leaf separated from the root by n – i + 1 edges. Entries 1 and 2 are the labels for the leaves in the cherry. Two vectors of labels g and s are considered to be equivalent if and only if one of the following two conditions holds: (1) g_i = s_i for all i, or (2) g₁ = s₂, g₂ = s₁, and g_i = s_i for each i = 3, 4,…, n.

Two leaves in a caterpillar tree are considered to be adjacent if they are separated by exactly two or three edges (Figure 1A). Equivalently, leaves are adjacent if and only if their indices in the sequence of labels for the tree differ by 1, or if one is entry 1 and the other is entry 3.

A component of a caterpillar tree is a subset of adjacent leaves, excluding from the definition the subset consisting solely of the pair of leaves in the cherry. Formally, a subset of labels X′ ⊂ X is a component of G if X′ ≠ {x₁, x₂} and for any pair of labels x₁, x₂ ∈ X′, there exists a sequence of distinct elements x₁, x_i1, x_i2,…x_ij, x₂ ∈ X′ in which each consecutive pair of elements labels adjacent leaves in G.

It is convenient to number the internal nodes of an n-leaf caterpillar tree from 1 to n – 1 in increasing order from the cherry node toward the root. These nodes are ordered by ancestor–descendant relationships, so that the node of smallest value in any nonempty subset of internal nodes descends from all other elements of the subset. We call this node the minimal node of the subset. It is also useful to consider that a tree possesses an internal edge ancestral to its root node; thus, identifying each internal node with its immediate ancestral edge, a nonempty subset of internal edges has a minimal edge.

2.2. Relationships between pairs of caterpillar trees

The labelings of distinct caterpillar trees with the same label set differ by a permutation of the vector of leaf labels. We will have occasion to examine pairs of caterpillar trees whose labelings differ by specific types of permutation: nearest-neighbor-interchange and subtree-prune-and-regraft [24].

Consider two distinct caterpillar trees G and S, bijectively labeled from the same set of n distinct labels.

Definition 1. Caterpillar trees G and S differ by a nearest-neighbor-interchange, or NNI move, if S can be obtained from G by exchanging the labels of a pair of adjacent leaves in G that are separated by exactly three edges (Figure 1B).

Note that our definition of adjacent leaves includes the leaves corresponding to labels g₁ and g₂ in the canonical ordering. This pair is the only pair of adjacent leaves that are not separated by an NNI move.

Definition 2. Caterpillar trees G and S differ by a subtree-prune-and-regraft, or SPR move, if there exists an ordered pair of edges (e₁, e₂) in G with the property that if edge e₁ is cut, edge e₂ is subdivided in two by placement of a new vertex v of degree two, and the subtree descended from e₁ is connected to vertex v such that v now has degree three and is ancestral to the subtree, then tree S is obtained (Figure 1C, 1D).

In an SPR move, note that it is possible for the edge e₂ to be the edge ancestral to the root of G.

Definition 3. Caterpillar trees G and S differ by a cyclic permutation if there exists a component G′ of G and a component S′ of S such that the labels of S′ represent a cyclic permutation of the labels of G′.

By definition of a component, this definition excludes permutations that simultaneously involve leaves separated from the root by the fewest edges and leaves separated from the root by the most edges, unless all leaves are involved.

Definition 4. Caterpillar trees G and S differ by an incrementation if they differ by a cyclic permutation and at most one label has positions in the canonical label vectors of G and S that differ by more than one.

S can differ from G by a forward or a reverse cycle or incrementation (Figure 1C, 1D). If S differs from G by a forward incrementation or cycle, then G differs from S by a reverse incrementation or cycle, and vice versa. Note that each cyclic permutation that exchanges two leaves is concurrently a forward incrementation, a reverse incrementation, and an NNI move.

We can immediately observe that a pair of caterpillar trees G and S differ by an SPR move if and only if they also differ by an incrementation of the leaf labels. SPR moves that convert caterpillars to caterpillars necessarily prune and regraft a single leaf. If a leaf is pruned from G and regrafted to S, then depending on which leaf is pruned and where it is regrafted, S can differ from G by either a forward or a reverse incrementation. Therefore, enumeration of coalescent histories in the case that caterpillar trees differ by an SPR move is performed by enumeration in the associated case of a forward or a reverse incrementation.

2.3. Coalescent histories

We study coalescent histories for a caterpillar gene tree G and a caterpillar species tree S, treated as binary, rooted, leaf-labeled caterpillar trees, each with n leaves labeled by labels bijectively drawn from the same set X. This setting corresponds to considering G to represent the tree formed by sampling a single gene lineage in each of the n species present in species tree S. Gene tree G and species tree S are said to be matching if G and S have the same labeled topology, and they are said to be non-matching otherwise.

Formally, a coalescent history can be defined as follows [19].

Definition 5. Consider an ordered pair of binary, rooted, leaf-labeled trees (G, S) whose labels are bijectively drawn from the same label set X. A coalescent history is a function h from the set of internal nodes of G to the set of internal edges of S that satisfies two conditions:

1. For each internal node v of G, all leaf labels for leaves descended from v in G label leaves descended from edge h(v) in S.

2. For all pairs of internal nodes v₁, v₂ in G, if node v₂ is descended from node v₁ in G, then edge h(v₂) is descended from edge h(v₁) in S.

An illustration appears in Figure 2. Recall that we consider that S contains an edge ancestral to its root; this edge can be the image of an internal node of G under a coalescent history mapping. Note that because an edge is trivially descended from itself, in part 2 of Definition 5, it is permissible for h(v₂) to equal h(v₁).

Figure 2.

Coalescent histories. (A) A gene tree G and species tree S with the same label set. The gene tree appears in blue, and the species tree appears in black. (B) The coalescent history depicted in (A) for (G, S). The arrows connect internal nodes of G to their associated edges in S.

We will have occasion to use the concept of a partial coalescent history.

Definition 6. Consider an ordered pair of binary, rooted, leaf-labeled trees (G, S) whose labels are drawn from the same label set X, not necessarily bijectively. A partial coalescent history is a function h from the set of internal nodes of G to the set of internal edges of S, satisfying the two conditions in Definition 5.

We say that if G is empty, then (G, S) has one partial coalescent history. For nonempty G, because the labels in G are not necessarily the same as those of S, it is possible that for some nodes v in G, S has no edge that can serve as the image of a node in G. In this case, the pair (G, S) has no partial coalescent histories. When connecting the purely graphical definition of coalescent histories in Definition 5 to the biological context in which coalescent histories arise, we say that an internal node v of G is a gene tree coalescence; the coalescence is said to occur on edge h(v) of S.

2.4. Catalan numbers and monotonic paths

We recall a number of results concerning Catalan numbers and their use in counting paths along the edges of square lattices. The Catalan sequence {C_n}_n⩾0 satisfies

$$C_n=\frac{1}{n+1}\left(\begin{array}{c}\hfill 2n\hfill \\ \hfill n\hfill \end{array}\right),$$

beginning from n = 0, with values 1, 1, 2, 5, 14, 42, 132, 429, 1430, 4862, …

Catalan numbers can be placed in the combinatorial construction known as Catalan’s triangle [14], of which we display the first six columns:

$$\begin{array}{cccccc}\hfill \hfill & \hfill \hfill & \hfill \hfill & \hfill \hfill & \hfill \hfill & \hfill 42\hfill \\ \hfill \hfill & \hfill \hfill & \hfill \hfill & \hfill \hfill & \hfill 14\hfill & \hfill 42\hfill \\ \hfill \hfill & \hfill \hfill & \hfill \hfill & \hfill 5\hfill & \hfill 14\hfill & \hfill 28\hfill \\ \hfill \hfill & \hfill \hfill & \hfill 2\hfill & \hfill 5\hfill & \hfill 9\hfill & \hfill 14\hfill \\ \hfill \hfill & \hfill 1\hfill & \hfill 2\hfill & \hfill 3\hfill & \hfill 4\hfill & \hfill 5\hfill \\ \hfill 1\hfill & \hfill 1\hfill & \hfill 1\hfill & \hfill 1\hfill & \hfill 1\hfill & \hfill 1\hfill \end{array}$$

In this triangle, the initial 1 in the lower left corner is denoted D(0, 0). Other entries are denoted D(n, k), with n as the horizontal distance from the lower left corner and k as the vertical distance from this entry.

For n, k with 0 ⩽ k ⩽ n, the entries (n, k) satisfy the recursion relation

$$D(n,k)=D(n,k-1)+D(n-1,k),$$ (2)

with initial condition D(0, 0) = 1. The general formula for D(n, k) is

$$D(n,k)=\{\begin{array}{cc}1\hfill & k=0\hfill \\ \left(\begin{array}{c}\hfill n+k\hfill \\ \hfill k\hfill \end{array}\right)-\left(\begin{array}{c}\hfill n+k\hfill \\ \hfill k-1\hfill \end{array}\right)\hfill & 1⩽k⩽n\hfill \\ 0\hfill & k>n.\hfill \end{array}\phantom{\}}$$ (3)

In particular, for k = n, we have D(n, n) = C_n.

The entry D(n, k) counts the number of monotonic paths on the lattice in the first quadrant of the (n, k) plane (including the coordinate axes) that do not cross the line k = n, where a monotonic path is a path from (0, 0) to (n, k) that proceeds by steps upward and to the right on the lattice.

We will also make use of extensions of Catalan’s triangle known as Catalan’s trapezoids of order m, which contain an initial column of m entries equal to 1, rather than a single entry [14]. Entries D_m(n, k) in Catalan’s trapezoids satisfy a version of eq. 2:

$$D_m(n,k)=D_m(n,k-1)+D_m(n-1,k).$$ (4)

We have D₁(n, k) = D(n, k). The first five columns of Catalan’s trapezoid of order 3 appear below:

$$\begin{array}{ccccc}\hfill \hfill & \hfill \hfill & \hfill \hfill & \hfill \hfill & \hfill 90\hfill \\ \hfill \hfill & \hfill \hfill & \hfill \hfill & \hfill 28\hfill & \hfill 90\hfill \\ \hfill \hfill & \hfill \hfill & \hfill 9\hfill & \hfill 28\hfill & \hfill 62\hfill \\ \hfill \hfill & \hfill 3\hfill & \hfill 9\hfill & \hfill 19\hfill & \hfill 34\hfill \\ \hfill 1\hfill & \hfill 3\hfill & \hfill 6\hfill & \hfill 10\hfill & \hfill 15\hfill \\ \hfill 1\hfill & \hfill 2\hfill & \hfill 3\hfill & \hfill 4\hfill & \hfill 5\hfill \\ \hfill 1\hfill & \hfill 1\hfill & \hfill 1\hfill & \hfill 1\hfill & \hfill 1\hfill \end{array}$$

An entry in the trapezoid can be calculated in closed form as

$$D_m(n,k)=\{\begin{array}{cc}\left(\begin{array}{c}\hfill n+k\hfill \\ \hfill k\hfill \end{array}\right)\hfill & 0⩽k<m\hfill \\ \left(\begin{array}{c}\hfill n+k\hfill \\ \hfill k\hfill \end{array}\right)-\left(\begin{array}{c}\hfill n+k\hfill \\ \hfill k-m\hfill \end{array}\right)\hfill & m⩽k⩽n+m-1\hfill \\ 0\hfill & k>n+m-1.\hfill \end{array}\phantom{\}}$$ (5)

The entry D_m(n, k) in Catalan’s trapezoid of order m counts the number of monotonic paths on the lattice in the first quadrant of the (n, k) plane (including the coordinate axes) that do not cross the line k = n + m – 1.

3. Bijection of coalescent histories and roadblocked monotonic paths

3.1. Matching gene trees and species trees

Degnan [2] proved that the number of coalescent histories for a matching caterpillar gene tree G and species tree S with n labels is the Catalan number C_n–1, demonstrating a bijection between coalescent histories and monotonic paths that do not cross the y = x diagonal of a square lattice. We will discuss this well-known correspondence, as the bijective approach is useful for the non-matching case.

Lemma 7. The coalescent histories for a matching n-leaf caterpillar gene tree G and species tree S can be bijectively associated with monotonic paths that do not cross the y = x diagonal of an (n – 1) × (n – 1) lattice.

Proof. Label the internal nodes of G sequentially from 1 to n – 1, using 1 for the internal node nearest the cherry and n – 1 for the root. For each internal node of G, identify the label for the node with the edge immediately ancestral to it. Similarly, sequentially label the internal nodes of S from 1 to n – 1, proceeding from the cherry toward the root and identifying the label for each node with its immediate ancestral edge.

For each j with 1 ⩽ j ⩽ n – 1, denote by G_j the subtree of the gene tree rooted at node j, and for each i with 1 ⩽ i ⩽ n – 1, denote by S_i the subtree of the species tree rooted at node i. We also define G₀ and S₀ to be empty subtrees of the gene tree and species tree, respectively. Denote by A_i,j the set of partial coalescent histories for (G_j, S_i). For matching G and S, for each j with 0 ⩽ j ⩽ n – 1, G_j = S_j. Hence, by definition of a coalescent history, for each internal node j ⩾ 1 of G, the image h(j) in a coalescent history h of (G, S) must be ancestral in S to all leaves of S labeled by labels in G_j. The edges of S with this property are edges j, j + 1,…, n – 1. For j ⩾ 1, we have j ⩽ h(j) ⩽ n – 1, and ∣A_i,j∣ = 0 for all (i, j) with i < j.

Each partial coalescent history in A_i,j is formed in one of two ways. Gene tree node j ⩾ 1 is mapped either to species tree internal edge i, or to one of the edges 1, 2,…, i – 1. The former case produces ∣A_i,j–1∣ partial coalescent histories, each obtained by appending the coalescence of gene tree node j to a partial coalescent history for (G_j–1, S_i). The latter case produces ∣A_{i–1, j}∣ partial coalescent histories; because no gene tree coalescences in such a partial coalescent history occur on species tree edge i, each such partial coalescent history for (G_j, S_i) is a partial coalescent history for (G_j, S_i–1). Hence, we have

$$\mid A_{i,j}\mid =\mid A_{i,j-1}\mid +\mid A_{i-1,j}\mid ,$$ (6)

with the constraint ∣A_i,j∣ = 0 for j ⩾ 1 and i < j. For j = 0 and 0 ⩽ i ⩽ n – 1, we have ∣A_i,0∣ = 1 by the convention that (G, S) has one partial coalescent history for empty G. We set ∣A_i,j∣ = 0 for all (i, j) that do not satisfy 0 ⩽ i, j ⩽ n – 1.

Recursion 6 and its base cases, with i in the role of n and j in the role of k, is precisely eq. 2. Setting i = j = n – 1, eq. 2 gives the recursion for enumerating the set of monotonic paths that do not cross the y = x diagonal of an (n – 1) × (n – 1) square lattice, a set with C_n–1 elements. In the bijection between coalescent histories and monotonic paths, each step to the right in the lattice, incrementing i, corresponds to incorporating an additional edge of the species tree as a possible location for gene tree coalescences, and each step up, incrementing j, corresponds to occurrence of a gene tree coalescence. ■

We can read a coalescent history of (G, S) from its associated monotonic path (Figure 3). For example, in a 10-leaf tree, the monotonic path that proceeds through (0,0), (3,0), (3,2), (6,2), (6,3), (7,3), (7,7), (9,7), and (9,9) has no gene tree coalescences on edge 1 of the species tree above (A, B) or on edge 2 above ((A, B), C). Gene tree coalescences (A, B) and ((A, B), C) occur on edge 3 above species tree node (((A, B), C), D). No gene tree coalescences occur on edges 4 or 5. Gene tree coalescence (((A, B), C), D) occurs on edge 6. Four gene tree coalescences occur on edge 7 above species tree node (((((((A, B), C), D), E), F), G), H). The two remaining gene tree coalescences occur on edge 9 above the species tree root.

Figure 3.

The correspondence between monotonic paths that do not cross above the y = x diagonal of an (n – 1) × (n – 1) square lattice and coalescent histories for a matching caterpillar gene tree and species tree with n = 10 leaves. The lower left corner represents the origin (0, 0). Monotonic paths from (0, 0) to (i, j) represent the partial coalescent histories A_i,j for (G_j, S_i). Values ∣A_i,j∣ are taken from eq. 2, using (i, j) in place of (n, k). Species tree internal edges are read from left to right: AB labels the species tree internal edge from which A and B descend, and each successive label indicates the internal edge ancestral both to the leaf corresponding to the associated label and to the caterpillar subtree containing all prior labels.Gene tree internal nodes are read in the same manner from bottom to top. The monotonic path shown in red indicates the locations on the species tree of the gene tree coalescences of a specific coalescent history: coalescences (A, B) and ((A, B), C) occur above species tree node (((A, B), C), D), coalescence (((A, B), C), D) occurs above species tree node ((((((A, B), C), D), E), F), G), coalescences ((((A, B), C), D), E), (((((A, B), C), D), E), F), ((((((A, B), C), D), E), F), G), and (((((((A, B), C), D), E), F), G), H) occur above species tree node (((((((A, B), C), D), E), F), G), H), and coalescences ((((((((A, B), C), D), E), F), G), H), I) and (((((((((A, B), C), D), E), F), G), H),I), J) occur above the species tree root.

The bijection between coalescent histories and monotonic paths generates a set of values of ∣A_i,j∣ that considers each i and j with 0 ⩽ i, j ⩽ n – 1 and i ⩾ j. These values can be depicted in a lattice so that the value ∣A_i,j∣ is associated with the coordinate of lattice point (i, j) (Figure 3). Indeed, they correspond exactly to the entries of Catalan’s triangle (eq. 3), with i in the role of n and j in the role of k.

The construction takes advantage of the caterpillar shape of both gene tree and species tree. Because internal nodes of a caterpillar tree can be placed in order with each entry descended from the next until the root is reached, simply stating the next leaf label suffices to specify the leaves descended from the next internal node. Movement from left to right in Figure 3 indicates movement from the cherry of the species tree toward the root, and movement from bottom to top indicates coalescence in the gene tree.

3.2. Non-matching gene trees and species trees

Our key insight is that a version of the construction of Degnan [2] linking coalescent histories and monotonic paths applies even if the gene tree and species tree are non-matching, provided that both continue to be caterpillars. Coalescent histories for non-matching caterpillars can be associated with roadblocked monotonic paths that do not cross above the y = x diagonal of an (n – 1) × (n – 1) square lattice.

Definition 8. In a lattice, a roadblocked monotonic path is a monotonic path that is not permitted to pass through certain specified lattice points. We term these lattice points roadblocks.

Consider a caterpillar gene tree G and a caterpillar species tree S, whose leaves are both bijectively associated with the same set of n leaves, but that do not necessarily match. As in Section 3.1, we associate points on the x-axis of an (n – 1) × (n – 1) lattice with species tree internal edges in S, and we associate points on the y-axis with gene tree internal nodes in G. We continue to label internal nodes of G and S in increasing order from 1 to n – 1, from the cherry to the root, indexing the gene tree internal nodes by j and the species tree internal nodes by i.

As is true in the matching case, for each j from 1 to n – 1, each coalescent history must have h(j) ⩾ j, as a gene tree internal node j must map to a species tree internal edge ancestral to at least as many leaves as descend from node j in G. Hence, each coalescent history for (G, S) corresponds to a monotonic path that has j ⩽ i and hence does not cross the y = x diagonal of the lattice. However, an additional constraint is imposed by the fact that G and S do not necessarily match.

Given G and S, let π(G) denote the permutation of the gene tree leaf labels g = (g₁, g₂,…, g_n) represented by the species tree leaf labels s = (s₁, s₂,…, s_n). The action of π sends the vector of leaf labels from one n-tuple to another, and we denote the index in S of g_k, the kth label of G, by π_k(G).

For the leaf labels g₁, g₂,…, g_n in G, let f(g_k) denote the minimal internal edge of S ancestral to leaf s_πk(G), the species tree leaf with label g_k. For a matching gene tree and species tree (G, S), π is the identity permutation so that π_k(G) = k; we then have f(g₁) = f(g₂) = 1, and f(g_k) = k – 1 for 3 ⩽ k ⩽ n.

For general (G, S) that do not necessarily match, by Definition 5, (i) if k = 1 or k = 2, then f(g_k) = max_ℓ∈{1,2} π_ℓ(G) – 1, and (ii) if 3 ⩽ k ⩽ n, then f(g_k) = max_{ℓ∈{1, 2,…, k}} π_ℓ(G) – 1. This rule encodes the fact that a gene tree coalescence can occur only on a species tree edge ancestral to all species tree leaves labeled by the elements of the set of labels for leaves descended from the gene tree coalescence.

Consider the partial coalescent histories A_i,j with i ⩾ j. As in Section 3.1, for j ⩾ 1, ∣A_i,j∣ = 0 for all (i, j) with i < j. For each j from 1 to n – 1, the minimal internal edge of S that is ancestral to all leaves labeled by labels of leaves of G that descend from gene tree internal node j is f(g_j+1). Therefore, for j ⩾ 1, we have ∣A_i,j∣ = 0 for all (i, j) with i < f(g_j+1). Note that these (i, j) are the only roadblocks: for j ⩾ 1, f(g_j+1) ⩾ j, as f(g_j+1) is one less than the maximum of j + 1 distinct elements of {1, 2,…, n– 1}, a quantity greater than or equal to j. For j ⩾ 1, because ∣A_i,j∣ = 0 for all lattice points (i, j) with i < f(g_j+1), all such points are roadblocks.

We also note that for 1 ⩽ j ⩽ j′ ⩽ n – 1, f(g_j′+1) ⩾ f(g_j+1). The set of descendant leaves of internal node j′ + 1 of G contains as a subset the descendant leaves of internal node j + 1 of G. Hence, the minimal internal edge of S ancestral to all labels that label leaves descended from internal node j′ + 1 of G has an index at least as great as the corresponding internal edge of S associated with internal node j + 1 of G. Consequently, if (i, j) is a roadblock, then because i < f(g_j+1) and f(g_j′+1) ⩾ f(g_j+1) for j′ ⩾ j, we can conclude that (i, j′) is a roadblock for each j′ with j ⩽ j′ ⩽ i.

As in Section 3.1, each partial coalescent history in A_i,j is formed in one of two ways. For j ⩾ 1, gene tree node j is mapped either to species tree internal edge i, or to one of the edges 1, 2,…,i – 1. The former case produces ∣A_i,j–1∣ partial coalescent histories, and the latter produces ∣A_i–1,j∣. Hence, the recursion ∣A_i,j∣ = ∣A_i,j–1∣ + ∣A_i–1,j∣ is still satisfied. We still have the constraints ∣A_i,j∣ = 0 for j ⩾ 1 and i < j, ∣A_i,0∣ = 1 for j = 0 and 0 ⩽ i ⩽ n – 1, and ∣A_i,j∣ = 0 for all (i, j) that do not satisfy 0 ⩽ i, j ⩽ n – 1. We also have the new constraint ∣A_i,j∣ = 0 for all (i, j) that satisfy i < f(g_j+1).

The set of roadblocks for (G, S) is defined by B_G,S = {(i, j) ∣ 1 ⩽ j ⩽ i ⩽ n – 1 and i < f(g_j+1)}. We have therefore demonstrated the following proposition.

Proposition 9. Consider a caterpillar gene tree G and a caterpillar species tree S with n leaves. (G, S) can be associated with a set of roadblocks B_G,S such that the coalescent histories for (G, S) bijectively correspond to roadblocked monotonic paths that do not cross the y = x diagonal of an (n – 1) × (n – 1) lattice.

By definition of B_G,S, we immediately see that if (i, j) is a roadblock for 1 ⩽ j ⩽ i ⩽ n – 1, then (k, j) is a roadblock as well for each k with j ⩽ k ⩽ i. We can also see that if (i, j) is a roadblock for 1 ⩽ j ⩽ i ⩽ n – 1, then (i, ℓ) is a roadblock as well for each ℓ with j ⩽ ℓ ⩽ n – 1; this result follows from the fact that f(g_j′+1) ⩾ f(g_j+1) for 1 ⩽ j ⩽ j′ ⩽ n – 1. We have the following remark.

Remark 10. Consider a caterpillar gene tree G and a caterpillar species tree S with n leaves. The roadblock set B_G,S consists of a set of points (i, j) with 1 ⩽ j ⩽ i ⩽ n – 1 such that if (i, j) ∈ B_G,S, then (i) (k, j) ∈ B_G,S for all k with j ⩽ k ⩽ i, and (ii) (i, ℓ) ∈ B_G,S for all ℓ with j ⩽ ℓ ⩽ i.

Figure 4 illustrates the correspondence between coalescent histories and roadblocked monotonic paths. In Figure 4, we have (f(g₁), f(g₂), f(g₃), f(g₄), f(g₅), f(g₆), f(g₇), f(g₈), f(g₉)) = (5, 5, 5, 5, 6, 8, 8, 9, 9). Because f(g₁₊₁) = 5, (4, 1) is a roadblock, as are (3, 1), (2, 1), and (1, 1) for the same reason ((i, j) is a roadblock if j ⩽ i < f(g_j+1)). Because f(g₂₊₁) = 5, (4, 2) is also a roadblock, as are (3, 2) and (2, 2). We can also identify (4, 2), (3, 2), and (2, 2) as roadblocks by Remark 10, as a consequence of the fact that (4, 1), (3, 1), and (2, 1) are roadblocks. Continuing through all (i, j), we identify 15 roadblocks in Figure 4.

Figure 4.

The correspondence between monotonic paths that do not cross above the y = x diagonal of an (n – 1) × (n – 1) square lattice and coalescent histories for a non-matching caterpillar gene tree and species tree with n = 10 leaves. Roadblocks are indicated by circles on lattice points; no roadblocked monotonic paths traverse the shaded regions. Otherwise, the figure design follows Figure 3.

From Proposition 9, we immediately obtain that the number of coalescent histories for (G, S) is given by the number of roadblocked monotonic paths that do not cross above the y = x diagonal of an (n – 1) × (n – 1) lattice, where the roadblocks are those in the set B_G,S. We also obtain a simple proof of the following corollary, which appeared as Remark 15 of Degnan & Rhodes [3].

Corollary 11. Consider a caterpillar gene tree G and a caterpillar species tree S with n leaves. The number of coalescent histories for (G, S) is strictly greater for G = S than for each choice of G ≠ S.

Proof. By Proposition 9, coalescent histories for (G, S) correspond to roadblocked monotonic paths that do not cross the y = x diagonal of an (n – 1) × (n – 1) lattice.

For G = S, applying Lemma 7, the number of coalescent histories is the number of monotonic paths that do not cross the y = x diagonal. Adding a roadblock to the lattice necessarily reduces the number of monotonic paths from (0, 0) to (n – 1, n – 1), as each lattice point has at least one monotonic path that passes through it. Because the number of coalescent histories for (G, S) is equal to the number of roadblocked monotonic paths on the lattice, it suffices to show that for G ≠ S, at least one lattice point is a roadblock.

Because G ≠ S, there exists some internal node j of G at least one of whose descendant leaves has a label not contained in the label set of the leaves descended from internal node j of S. This leaf has j < f(g_j+1). Hence, (j, j) is a roadblock, and (G, S) is associated with fewer monotonic paths than is (S, S). ■

3.3. Roadblock sets

Given a caterpillar species tree S, Remark 10 suggests a characterization of the possible sets of roadblocks, considering all caterpillar gene trees G. Each roadblock set has the property that within a row, all points to the left of a roadblock and on or below the y = x diagonal are also roadblocks. Within a column, all points above a roadblock and on or below the y = x diagonal are roadblocks.

Proposition 12. Consider a caterpillar species tree S with n leaves. For each caterpillar gene tree G with n leaves, denote its associated roadblock set by B_G,S. Considering all n!/2 possible caterpillar gene trees, the distinct roadblock sets are bijectively associated with the C_n–1 monotonic paths on the (n – 1) × (n – 1) lattice that do not cross the y = x diagonal.

Proof. Consider a roadblock set B_G,S. For each i from 1 to n – 2, we identify the largest j such that (i, j) is not a roadblock. Call this value j_i. A unique monotonic path connects (0, 0), (1, j₁), (2, j₂),…, (n – 2, j_n–2), (n – 1, n – 1): by Remark 10, for each i and each j > j_i, (i, j) is either a roadblock or it lies above the y = x line. Hence, denoting j₀ = 0 and j_n–1 = n – 1, for each i from 1 to n – 1, a monotonic path from (i – 1, j_i–1) to (i, j_i) must proceed horizontally by length 1 and then vertically by length j_i – j_i–1.

To show that this construction is injective, note that distinct monotonic paths are associated with distinct roadblock sets: consider a point (i, j_i) appearing in one monotonic path P₁ but not in another one, P₂. Because j_i is the largest value of j that is not a roadblock for path P₁, (i, j_i) must be a roadblock for P₂. For surjectivity, consider a monotonic path from (0, 0) to (n – 1, n – 1) that does not cross the y = x line. For each (i, j_i) in the path, 1 ⩽ i ⩽ n – 2, where j_i is the largest value of j for which the point (i, j) is in the path, we assign each point (i, ℓ) with j_i < ℓ ⩽ i to be a roadblock. ■

Figure 5 provides an illustration of Proposition 12, showing how the monotonic path associated with a roadblock set is constructed and vice versa. The monotonic path associated with a roadblock set can be viewed as the monotonic path that comes as close as possible to the roadblocks. The roadblock set for a monotonic path is the set of points above and to the left of the path.

Figure 5.

The correspondence between roadblock sets, monotonic paths that do not cross above the y = x diagonal of an (n – 1) × (n – 1) square lattice, and Dyck paths of semi-length n – 1. Given a roadblock set, the associated monotonic path is constructed by identifying for each x coordinate from 0 to n – 1 the lattice point of greatest y coordinate, and then constructing the unique monotonic path through those points. Similarly, given a monotonic path, its roadblock set is obtained by placing roadblocks at each lattice point above and to the left of the path. (A) Roadblock set symmetric across the line y = n – 1 – x. (B) Roadblock set asymmetric across the line y = n – 1 – x. (C) Roadblock set asymmetric across the line y = n – 1 – x, obtained by reflecting the roadblocks in (B) over this line. (D) Symmetric Dyck path associated with the roadblock set in (A). (E) Asymmetric Dyck path associated with the roadblock set in (B). (F) Asymmetric Dyck path associated with the roadblock set in (C), obtained by reversing the Dyck path in (E). The roadblock sets in (B) and (C) both generate 235 monotonic paths from (0, 0) to (9, 9).

The number of distinct caterpillar trees is n!/2, whereas the number of distinct roadblock sets is the smaller C_n–1. For a given caterpillar species tree, we can place the n!/2 caterpillar gene trees into equivalence classes, where two gene trees are said to be history-equivalent if and only if they are associated with the same roadblock set. Two history-equivalent caterpillar trees G₁ and G₂ have the same set of roadblocks and the same set of monotonic paths, and hence, the same set of coalescent histories, up to permutation of the leaf labels. These equivalence classes were termed history classes by Rosenberg & Tao [20], so that two caterpillars with the same roadblocks are in the same history class.

By Proposition 12, for a fixed species tree, the number of history classes considering all caterpillar trees is C_n–1; this result accords with the computation of 5 history classes for n = 4 [15, Table V] and 14 for n = 5 [20, Table 3]. We have also seen in Corollary 11 that C_n–1 is the largest possible number of coalescent histories for a pair of caterpillar trees. We now ask how many of the values 1, 2,…, C_n–1 can be the number of coalescent histories for some caterpillar gene tree and species tree.

The simplest upper bound on this quantity is C_n–1. To improve on this bound, it is convenient to use the bijection between monotonic paths that do not cross the y = x diagonal of the (n – 1) × (n – 1) lattice and Dyck paths of semilength n – 1 [22, Corollary 6.3.2]. Each monotonic path represents a series of steps by (1, 0) or (0, 1) from (0, 0) to (n – 1, n – 1), with x ⩾ y at each step. Each Dyck path represents a series of steps by (1, 1) or (1, −1) from (0, 0) to (n – 1, 0), with y ⩾ 0 at each step. The coalescent histories for (G, S) can therefore be associated with Dyck paths, where each up-step represents addition of a species in the species tree and each down-step represents a gene tree coalescence.

A Dyck path of semi-length n – 1 has 2n – 2 total up-steps and down-steps. The steps of Dyck paths can be written as a sequence, with U denoting up-steps and D denoting down-steps. A Dyck path can be reversed in the following manner: we take the sequence of U and D steps in the path, reverse the order of steps, and exchange the positions of U and D steps. Thus, a path UUUDUDDUDD becomes UUDUUDUDDD. Reversing a Dyck path corresponds to traversing the path in reverse order. A reversed Dyck path is itself a Dyck path; if the sequence of U and D steps in a Dyck path is reversed, then y ⩽ 0 at each step; exchanging the positions of the U and D steps reflects the path over the y = 0 axis.

Lemma 13. Consider a caterpillar species tree S with n leaves. Consider gene trees G₁ and G₂ with n leaves such that (i, j) is in the roadblock set B_G1,S if and only if (n – 1 – j, n – 1 – i) is in the roadblock set B_G2,S. Then (G₁, S) and (G₂, S) have the same number of coalescent histories.

Proof. We show that the coalescent histories for (G₁, S) can be bijectively associated with the coalescent histories for (G₂, S). Consider a coalescent history for (G₁, S). Identify its associated monotonic path M₁ according to Proposition 9, and identify the Dyck path P₁ associated with this monotonic path. Reverse P₁ to obtain $P_1^{\prime }$, and identify the monotonic path $M_1^{\prime }$ associated with $P_1^{\prime }$.

Because M₁ avoids each roadblock (i, j) in B_G1,S, after i + j steps, P₁ cannot have taken i up-steps and j down-steps. Because $P_1^{\prime }$ is the reverse of P₁, after 2n – 2 – i – j steps, $P_1^{\prime }$ cannot have taken n – 1 – j up-steps and n – 1 – i down-steps. The monotonic path $M_1^{\prime }$ therefore avoids the point (n – 1 – j, n – 1 – i) for each roadblock (i, j) in B_G1,S. Hence, $M_1^{\prime }$ avoids each roadblock in B_G2,S, and it therefore represents a coalescent history for G₂. Similarly, beginning from the coalescent history for (G₂, S) associated with $M_1^{\prime }$, we find that M₁ represents a coalescent history for B_G1,S. ■

The lemma demonstrates that for two roadblock sets, if their roadblocks can be obtained by transforming each roadblock (i, j) of one into a roadblock (n – 1 – j, n – 1 – i) of the other, then the associated caterpillar gene trees have the same number of coalescent histories.

Consider a set of points B on or below the y = x diagonal of the first quadrant of the (n – 1) × (n – 1) lattice (and not on lines y = 0 or x = n – 1) with the property that if (i, j) ∈ B, then (k, j) ∈ B for all k with j ⩽ k ⩽ i and (i, ℓ) ∈ B for all ℓ with j ⩽ ℓ ⩽ i. By Proposition 12, given a caterpillar species tree, B is the roadblock set for some caterpillar gene tree. We term such a set a caterpillar-friendly roadblock set.

Definition 14. Consider a caterpillar-friendly roadblock set B for the (n – 1) × (n – 1) lattice. We say that B is symmetric if for each (i, j) ∈ B, (n – 1 – j, n – 1 – i) is also in B. Otherwise, B is asymmetric.

In a symmetric caterpillar-friendly roadblock set, when the points in the roadblock set are reflected across the line y = n – 1 – x, the same roadblock set is obtained (Figure 5A). For an asymmetric caterpillar-friendly roadblock set, a different roadblock set is obtained by this reflection (Figure 5B and 5C).

For the (n – 1) × (n – 1) lattice, denote by Q_n–1 and R_n–1 the numbers of symmetric and asymmetric caterpillar-friendly roadblock sets, respectively. By Lemma 13, the asymmetric caterpillar-friendly roadblock sets can be partitioned into disjoint pairs such that the associated caterpillar gene trees for the two entries in a pair give rise to the same number of coalescent histories. Hence, considering all caterpillar gene trees and species trees, the number of distinct values possible for the number of coalescent histories is bounded above by Q_n–1 + R_n–1/2, or because Q_n–1 + R_n–1 = C_n–1, by (C_n–1 + Q_n–1)/2.

We obtain Q_n–1 by counting all ways of placing roadblocks (i, j) with i + j ⩽ n – 1. By symmetry we then assign points (n – 1 – j, n – 1 – i) to be roadblocks as well. Because of the bijection between roadblock sets and monotonic paths (Proposition 12), each set of roadblocks (i, j) with i + j ⩽ n – 1 is bijectively associated with a monotonic path from (0, 0) to a point (i, n – 1 – i) for some i with 0 ⩽ i ⩽ n – 1.

Lemma 15. The value of Q_n–1 is $\left(\begin{array}{c}\hfill n-1\hfill \\ \hfill \lfloor (n-1)∕2\rfloor \hfill \end{array}\right)$.

Proof. Using eq. 3, the number of monotonic paths from (0, 0) to (i, n – 1 – i) for some i with 0 ⩽ i ⩽ n – 1 is obtained by the sum

$$\sum _{j=0}^{\lfloor (n-1)∕2\rfloor }\left(\begin{array}{c}\hfill n-1\hfill \\ \hfill j\hfill \end{array}\right)-\sum _{j=1}^{\lfloor (n-1)∕2\rfloor }\left(\begin{array}{c}\hfill n-1\hfill \\ \hfill j-1\hfill \end{array}\right).$$

The first sum gives $2^{n-2}+\frac{1}{2}\left(\begin{array}{c}\hfill n-1\hfill \\ \hfill (n-1)∕2\hfill \end{array}\right)$ for odd n, and 2^n–2 for even n. The second sum gives $2^{n-2}-\frac{1}{2}\left(\begin{array}{c}\hfill n-1\hfill \\ \hfill (n-1)∕2\hfill \end{array}\right)$ for odd n, and $2^{n-2}-\left(\begin{array}{c}\hfill n-1\hfill \\ \hfill \lfloor (n-1)∕2\rfloor \hfill \end{array}\right)$ for even n. Combining these cases, the result follows. ■

This result appeared in Bonin et al. [1, Theorem 2.5] as the number of number of distinct first halves for Dyck paths, and in Deng et al. [7, Theorem 4.2] as the number of Dyck paths invariant under reversal.

Proposition 16. The size of the set of values that can equal the number of coalescent histories for at least one pair (G, S) consisting of an n-leaf caterpillar gene tree G and an n-leaf caterpillar species tree S is bounded above by T_n–1 = (C_n–1+Q_n–1)/2, or

$$\frac{1}{2}\left[\frac{\left(\begin{array}{c}\hfill 2n-2\hfill \\ \hfill n-1\hfill \end{array}\right)}{n}+\left(\begin{array}{c}\hfill n-1\hfill \\ \hfill \lfloor (n-1)∕2\rfloor \hfill \end{array}\right)\right].$$

This quantity, which appeared in a bijectively related context in Bonin et al. [1, Theorem 4.2], gives the number of distinct Dyck paths up to reversal. Numerical values of the formulas in Lemma 15 and Proposition 16 are shown in Table 1.

Table 1.

The number of distinct values possible for the number of coalescent histories of a caterpillar gene tree and a caterpillar species tree.

Number of leaves n	Number of distinct roadblock sets	Number of road block sets associ ated with symmet ric Dyck paths	Number of road block sets associ ated with asym metric Dyck paths	Upper bound on the number of distinct values for the number of coalescent histories	Exact number of dis tinct values for the number of coalescent histories
Notation	Cn–1	Qn–1	Pn–1	Tn–1
Formula	$\frac{\left(\begin{array}{c}\hfill 2n-2\hfill \\ \hfill n-1\hfill \end{array}\right)}{n}$	$(\begin{array}{c}\hfill n-1\hfill \\ \hfill \lfloor (n-1)∕2\rfloor \hfill \end{array})$	Cn–1 – Qn–1	(Cn–1 + Qn–1)/2
OEIS record	A000108	A001405	A306292	A007123
2	1	1	0	1	1
3	2	2	0	2	2
4	5	3	2	4	4
5	14	6	8	10	10
6	42	10	32	26	21
7	132	20	112	76	56
8	429	35	394	232	154
9	1430	70	1360	750	440
10	4862	126	4736	2494	1373
11	16796	252	16544	8524	4310
12	58786	462	58324	29624	13925

The table considers all pairs of labeled caterpillar trees, both matching and non-matching, that are possible for a given label set.

4. Non-recursive enumeration of coalescent histories

With the correspondence between coalescent histories for non-matching caterpillars and roadblocked monotonic paths established, we now turn to enumerating the coalescent histories of possibly non-matching caterpillar gene trees and species trees. We can do so recursively by enumerating roadblocked monotonic paths according to Proposition 9; we can also obtain a non-recursive formula by applying eq. 1.

Without loss of generality, considering the two subtrees immediately descended from the root of a tree, we treat the left subtree as having a number of leaves greater than or equal to that of the right subtree. The right subtree of a caterpillar tree then has a single leaf, so that in eq. 1, the right subtree G_R always has exactly one leaf in each successive step of the recursion. Hence, the term B_{GR,T (GR,S),k+d(GR,S)}, follows the base case of the recursion and is equal to 1. Eq. 1, describing the number of coalescent histories for a caterpillar gene tree G and a species tree S, then reduces to

$$B_{G,S,m}=\sum _{k=1}^m{B}_{G_L,T(G_L,S),k+d(G_L,S)},$$ (7)

with initial condition B_G,S,m = 1 for all m when G has a single leaf.

If S is also a caterpillar tree with n leaves, then we can iterate the recursion n – 1 times, at each step reducing the size of the left subtree G_L by one, until G_L has a single leaf, the base case applies, and the summand equals 1. Each iteration introduces a new summation, with its upper limit depending on the associated d(G_L, S), the number of edges that separate the root of T(G_L, S) from the root of S. Continuing to label internal nodes of G from 1 to n – 1 in increasing order from the cherry to the root, we associate internal node j of G with index k_n–j. Setting the integer parameter m equal to 1, we have

$$B_{G,S,1}=\sum _{k_1=1}^1\sum _{k_2=1}^{k_1+c_1}\sum _{k_3=1}^{k_2+c_2}\cdots \sum _{k_{n-1}=1}^{k_{n-2}+c_{n-2}}1,$$ (8)

where the constant c_j represents the number of additional edges of S that are possible locations for gene tree coalescence j but that are not possible for gene tree coalescence j + 1.

For 1 ⩽ j ⩽ n – 1, consider gene tree internal node j. Let L_j be the set of labels for all j + 1 leaves descended from j. Following the definitions in eq. 1, let T_j(G, S) denote the smallest subtree of S that has the property that each label in L_j labels one of its leaves, and let d_j denote the number of edges separating the root of T_j(G, S) from the root of S. Then d_j + 1 gives the number of edges of S on which gene tree coalescence j can occur (the +1 represents the root edge of S). The quantity u_j = n – 1 – j – d_j, equal to the number of edges of S ancestral to at least j + 1 leaves (or n – j) but on which gene tree coalescence j cannot occur, represents the number of roadblocks (i, j) with fixed j and i ⩾ j.

For j = 1, 2,…, n–2, the desired quantity c_j, the number of additional edges of S available for coalescence j but not for coalescence j + 1, equals c_j = d_j – d_j+1. We have therefore shown the following proposition.

Proposition 17. Consider a caterpillar gene tree G and a caterpillar species tree S with n leaves. The number of coalescent histories for (G, S) is obtained by eq. 8, where the vector (c₁, c₂,…, c_n–2) is obtained as a function c(G, S) that depends only on the topologies of G and S.

Note that if G and S match, then for each j from 1 to n – 1, G_j = T_j(G, S), and hence d_j = n – 1 – j, u_j = 0, and no roadblocks occur. We have c_j = 1 for each j from 1 to n – 2, and eq. 8 becomes

$$\sum _{k_1=1}^1\sum _{k_2=1}^{k_1+1}\sum _{k_3=1}^{k_2+1}\cdots \sum _{k_{n-1}=1}^{k_{n-2}+1}1,$$

equal to the Catalan number C_n–1 [16, Theorem 3.4].

We take as an example the gene tree and species tree in Figure 4. We report the values of the u_j, d_j and c_j in Table 2. The number of coalescent histories is

$$\sum _{k_1=1}^1\sum _{k_2=1}^{k_1+0}\sum _{k_3=1}^{k_2+1}\sum _{k_4=1}^{k_3+0}\sum _{k_5=1}^{k_4+2}\sum _{k_6=1}^{k_5+1}\sum _{k_7=1}^{k_6+0}\sum _{k_8=1}^{k_7+0}\sum _{k_9=1}^{k_8+0}1=235.$$

Table 2.

Quantities associated with the enumeration of coalescent histories for a caterpillar gene tree (((((((((A, F), B), C), D), G), I), H), J), E) and species tree (((((((((A, B), C), D), E), F), G), H), I), J).

Internal node index in gene tree G (j)	9	8	7	6	5	4	3	2	1
Summation index (n – j)	1	2	3	4	5	6	7	8	9
Number of roadblocks (uj)	0	1	1	2	1	1	2	3	4
Distance between root of Tj (G, S) and root of S (dj)	0	0	1	1	3	4	4	4	4
Nodes possible for coalescence j but not j + 1 (cj = dj – dj+1)	NA	0	1	0	2	1	0	0	0
Summation term	$\sum _{k_1=1}^1$	$\sum _{k_2=1}^{k_1+0}$	$\sum _{k_3=1}^{k_2+1}$	$\sum _{k_4=1}^{k_3+0}$	$\sum _{k_5=1}^{k_4+2}$	$\sum _{k_6=1}^{k_5+1}$	$\sum _{k_7=1}^{k_6+0}$	$\sum _{k_8=1}^{k_7+0}$	$\sum _{k_9=1}^{k_8+0}$

We can also obtain this result by recursive summation of roadblocked monotonic paths (Figure 4).

By exhaustive use of Proposition 17, we can evaluate all possible values of the number of coalescent histories for the n!/2 caterpillar gene trees with n leaves. This exhaustive computation applies eq. 8 with all possible vectors (c₁, c₂,…, c_n–2) that correspond to gene trees–in other words, the C_n–1 vectors with $0⩽{\sum }_{k=1}^j{c}_k⩽j$ for each j from 1 to n – 2 [23, replacing a_i in item 81 with 1 – c_k].

The upper bound from Proposition 16 on the number of distinct values for the number of coalescent histories is relatively tight for small n, but already is more than double the exact computation for n = 12 (Table 1). The smallest case in which the number of distinct values (21) differs from the upper bound (26) occurs with n = 6 leaves, in which 1, 2, 3, 4, 5, 6, 7, 9, 10, 12, 13, 14, 16, 17, 19, 22, 23, 26, 28, 32, and 42 are achievable values for the number of coalescent histories. Values 5, 9, 10, 14, and 19 are each achieved with two distinct set of roadblocks that are not equivalent when reversing their associated Dyck paths.

Because Q_n–1 ≪ C_n–1, the upper bound for the number of distinct values for the number of coalescent histories of a caterpillar pair is asymptotically equivalent to C_n–1/2, half the maximum number of coalescent histories for caterpillars. Thus, although the number of caterpillars n!/2 grows much faster than the maximal number of coalescent histories C_n–1, asymptotically only at most half the values in the range of possible values for the number of coalescent histories are achieved by actual caterpillar gene trees.

5. Special families of caterpillar gene trees and species trees

From Propositions 9 and 17, we can obtain a variety of corollaries that describe the number of coalescent histories for special pairs of non-matching caterpillar trees. For certain classes of pairs, the number of coalescent histories can be obtained in closed form.

5.1. Nearest-neighbor-interchange

For a fixed caterpillar species tree S, we first consider caterpillar gene trees G that differ from S by a single nearest-neighbor-interchange move (NNI). We have the following result.

Proposition 18. Consider a caterpillar species tree S with n leaves and a caterpillar gene tree G that differs from S by an NNI move. (i) The roadblock set B_G,S consists of a single point (i, i) on the diagonal of the square lattice, for some i with 1 ⩽ i ⩽ n – 2. (ii) The number of coalescent histories for (G, S) is C_n–1 – C_iC_n–1–i.

Proof. We use the bijection between coalescent histories and roadblocked monotonic paths (Proposition 9). We label leaves on the trees from 1 to n as in Section 3.2, using the permutation π to map the leaves of G to the leaves of S. By Definition 1, an NNI move exchanges a single pair of leaves labeled k and k + 1 in G for some k ∈ {2, 3,…, n–1}, or it exchanges leaves 1 and 3. Let k_s be the smaller of the two labels for the leaves participating in the NNI move, and let k_ℓ be the larger of the two labels. We then have π_ks(G) = k_s + 1 and π_ks+1(G) = k_s if k_s ∈ {2, 3,…, n – 1}, and π₁(G) = 3 and π₃(G) = 1 if k_s = 1.

(i) Following Section 3.2, for (G, S) differing by one NNI move, the minimal internal edge of S ancestral to leaf k_s of G is f(g_ks) = k_s if 2 ⩽ k_s ⩽ n – 1, and f(g₁) = 2 if k_s = 1. The roadblocks (i, j) in the square lattice are those points that satisfy i < f(g_j+1). By construction, (k_s – 1, k_s – 1) is the only roadblock if 2 ⩽ k_s ⩽ n – 1, and (1, 1) is the only roadblock if k_s = 1.

(ii) The number of coalescent histories for (G, S) is the number of coalescent histories for the case of no roadblocks, or C_n–1 (Lemma 7), minus the number of monotonic paths from (0, 0) to (n – 1, n – 1) that do not cross the diagonal and that pass through the roadblock. For a roadblock at (i, i), this latter quantity is C_iC_n–1–i, multiplying the number of monotonic paths C_i from (0, 0) to (i, i) that do not cross the diagonal by the number of monotonic paths from (i, i) to (n – 1, n – 1) that do not cross the diagonal. ■

Figure 6A illustrates the result of Proposition 18 with a pair (G, S) that differ by a single NNI move. The number of coalescent histories for the example is 4274, as obtained by Proposition 9. Using Proposition 18, we see that 4274 = C₉ – C₄C₅ = 4862 – 14 × 42.

Figure 6.

The number of coalescent histories for caterpillar gene tree G and species tree S that differ by nearest-neighbor-interchange (NNI) moves. (A) G and S differ by a single NNI move. (B) G and S differ by multiple disjoint NNI moves.

Extending Proposition 18, we can establish a formula for the number of coalescent histories when the permutation π performs k NNI moves, $1⩽k⩽\lfloor \frac{n-1}{2}\rfloor$, in such a way that π consists of disjoint cycles of length 2. The roadblock set for (G, S) with such a permutation contains k points on the diagonal of the (n – 1) × (n – 1) lattice. For this roadblock set, we count the monotonic paths that do not cross the diagonal by use of the inclusion–exclusion principle.

Proposition 19. Consider a caterpillar species tree S with n leaves and a caterpillar gene tree G that differs from S by k disjoint NNI moves. Then (i) the roadblock set B_G,S consists of k distinct points (i_j, i_j) on the diagonal of the square lattice, j ∈ {1, 2,…, k}, with 0 < i₁ < … i_k < n – 1. (ii) The number of coalescent histories for (G, S) can be written

$$C_{n-1}+\sum _{\ell =1}^k(-1{)}^{\ell }\left(\sum _{\begin{array}{c}\hfill (j_1,j_2,\dots ,j_{\ell })\in \{1,2,\dots ,k{\}}^{\ell }\hfill \\ \hfill 0<i_{j_1}<\dots <i_{j_{\ell }}<n-1\hfill \end{array}}{C}_{i_{j_1}}{C}_{i_{j_2}-i_{j_1}}\cdots C_{i_{j_{\ell }}-i_{j_{\ell }-1}}{C}_{n-1-i_{j_{\ell }}}\right).$$ (9)

Proof. As in Proposition 18, we rephrase a problem of enumerating coalescent histories in the language of roadblocked monotonic paths.

(i) Because the k NNI moves are disjoint, we can apply Proposition 18i sequentially k times, once for each NNI move. Each of the k NNI moves is associated with a roadblock on the diagonal of the square lattice, the location of which is determined by the identity of the pair of leaves exchanged: if k_s is the smaller of the two labels for leaves of G participating in the move, then the roadblock location is (k_s – 1, k_s – 1) for 2 ⩽ k_s ⩽ n – 1 and (1, 1) for k_s = 1. Label the roadblocks (i₁, i₁), (i₂, i₂),…, (i_k, i_k), with 0 < i₁ < i₂ < … < i_k < n – 1.

(ii) The total number of monotonic paths on the (n – 1) × (n – 1) square lattice is C_n–1 (Section 2.4). To obtain the desired quantity, we must subtract from C_n–1 the number of paths that pass through at least one of the k roadblocks. By the inclusion–exclusion principle, this quantity can be written as a sum over nonempty subsets β of the k roadblocks of the number of monotonic paths that pass through all roadblocks in β. In particular, for each j from 1 to k, denoting by β_j the set of monotonic paths that pass through roadblock (i_j, i_j), the number of monotonic paths that pass through at least one roadblock is

$$\sum _{\ell =1}^k(-1{)}^{\ell +1}\left(\sum _{\begin{array}{c}\hfill (j_1,j_2,\dots ,j_{\ell })\in \{1,2,\dots ,k{\}}^{\ell }\hfill \\ \hfill 0<i_{j_1}<\dots <i_{j_{\ell }}<n-1\hfill \end{array}}\mid {\beta }_{j_1}\cap \cdots \cap {\beta }_{j_{\ell }}\mid \right).$$

The cardinality of ∣β_j1 ⋂ ⋯ ⋂ βj_ℓ∣, representing the number of monotonic paths that pass through (0, 0), (i_j1, i_j1), …, (i_jℓ, i_jℓ), and (n – 1, n – 1), is a product of Catalan numbers, one for each pair of consecutive points that must be traversed:

$$C_{i_{j_1}}{C}_{i_{j_2}-i_{j_1}}\cdots C_{i_{j_{\ell }}-i_{j_{\ell }-1}}{C}_{n-1-i_{j_{\ell }}}.$$

The result then follows. ■

Figure 6B illustrates Proposition 19 for a case with three disjoint NNI moves. In this example, roadblocks in a 10-leaf tree appear at (1, 1), (4, 4), and (9, 9). By Proposition 19, the number of coalescent histories is C9 – (C₁C₈ + C₄C₅ + C₈C₁) + (C₁C₃C₅ + C₁C₇C₁ + C₄C₄C₁) – (C₁C₃C₄C₁) = 2179.

Proposition 18 additionally has the consequence that for a fixed caterpillar species tree S, the largest number of coalescent histories seen for a non-matching caterpillar gene tree G occurs for a gene tree that differs from S by a single NNI move.

Corollary 20. Consider a caterpillar species tree S with n leaves. Considering all possible caterpillar gene trees G ≠ S with n leaves, (i) the largest number of coalescent histories for (G, S) is obtained when the roadblock set B_G,S consists of the single point $(\frac{n}{2}-1,\frac{n}{2}-1)$ or $(\frac{n}{2},\frac{n}{2})$ for even n, or $(\frac{n-1}{2},\frac{n-1}{2})$ for odd n. (ii) It equals

$$C_{n-1}-C_{\lfloor \frac{n-1}{2}\rfloor }{C}_{\lceil \frac{n-1}{2}\rceil }.$$

Proof. First, note that for each pair (G, S) whose roadblock set B_G,S has more than one point, we can identify a pair (G′, S) whose roadblock set consists of a single point in B_G,S and that hence has at least as many coalescent histories as (G, S). By Remark 10, the roadblock in a roadblock set consisting of a single point must be located on the diagonal.

Applying Proposition 18ii, for the caterpillar gene tree G that maximizes the number of coalescent histories with fixed S, that number of coalescent histories must equal C_n–1 – C_iC_n–1–i for some i with 1 ⩽ i ⩽ n – 2. By Corollary 3.11 of Rosenberg [16], for fixed n, this quantity is maximized when $\{i,n-1-i\}=\left\{\lfloor \frac{n-1}{2}\rfloor ,\lceil \frac{n-1}{2}\rceil \right\}$.

From the proof of Proposition 18, for fixed S, this maximum is associated with the gene trees G that differ from S in that the leaves abutting the middle coalescence in the path from cherry to root of G are transposed; the case of n odd has one such coalescence and the case of n even has two. ■

Corollary 20 extends Corollary 11 by giving the exact number of coalescent histories for the pair (G, S) that has the largest number of coalescent histories among non-matching caterpillars. For odd n, the number of coalescent histories in Corollary 20 is $C_{n-1}-C_{(n-1)∕2}^2$. For example, for S = ((((A, B), C), D), E) and G = ((((A, B), D), C), E), the number of coalescent histories is $C_4-C_2^2=14-2^2=10$ coalescent histories. For even n, the number of coalescent histories in Corollary 20 is C_n–1 – C_n/2–1C_n/2. For example, for S = (((((A, B), C), D), E), F), both G = (((((A, B), D), C), E), F) and G = (((((A, B), C), E), D), F) have C₅ – C₂C₃ = 42 – 2 × 5 = 32 coalescent histories. Figure 6A gives the largest number of coalescent histories among nonmatching caterpillars with n = 10 leaves.

We can quickly observe that the largest number of coalescent histories among discordant caterpillar trees grows at the same rate as the number of coalescent histories for matching caterpillars.

Corollary 21. Considering all non-matching caterpillar pairs (G, S) with n leaves, the largest number of coalescent histories for (G, S) is asymptotic to C_n–1.

Proof. Using Stirling’s approximation, $n!~\sqrt{2\pi n}{\left(\frac{n}{e}\right)}^n$, we can verify that $C_n~4^n∕(n^{3∕2}\sqrt{\pi })$.

If n is odd, then the largest number of coalescent histories for a non-matching pair satisfies

$$C_{n-1}-C_{\frac{n-1}{2}}^2\sim \frac{4^{n-1}}{(n-1{)}^{\frac{3}{2}}\sqrt{\pi }}-\frac{4^{n-1}{2}^3}{(n-1{)}^3\pi }.$$

If n is even, then

$$C_{n-1}-C_{\frac{n}{2}-1}{C}_{\frac{n}{2}}\sim \frac{4^{n-1}}{(n-1{)}^{\frac{3}{2}}\sqrt{\pi }}-\frac{4^{n-1}{2}^3}{(n^2-2n{)}^{\frac{3}{2}}\pi }.$$

In both cases, the leading term dominates, and C_n–1 – C_{⌊(n–1)/2⌋}C_{⌈(n–1)/2⌉} ~ C_n–1. ■

5.2. Reverse incrementation of the leaf labels

Next, for a fixed caterpillar species tree S, we consider gene trees G that differ from S by incrementation.

Consider a caterpillar species tree S with n leaves and a caterpillar gene tree that differs from S by an incrementation. By definition, the leaves of some component G′ of G and S′ of S differ by cyclic permutation. Recall that an incrementation with two labels is a NNI move.

Proposition 22. If G is obtained by a reverse incrementation of S, then the roadblock set B_G,S consists of a set of consecutive points on the diagonal of the square lattice.

Proof. Consider labels k_s, k_ℓ ∈ {1, 2,…, n}, with k_s < k_ℓ and k_ℓ ≠ 2. By definition of reverse incrementation, for some component of G with leaves sequentially labeled k_s, k_s + 1,…, k_ℓ from the cherry toward the root, associated leaves of S are labeled π_ks(G) = k_s + 1, π_ks+1(G) = k_s + 2,…, π_kℓ–1(G) = k_ℓ, π_kℓ(G) = k_s.

As in the proof of Proposition 18, we compute the minimal internal edge of S ancestral to each leaf g_k of G, k ∈ {k_s, k_s + 1,…, k_ℓ}. We obtain f(g_k) = k if 2 ⩽ k ⩽ n and f(g₁) = 2 if k = 1.

The roadblocks are the points (i, j) satisfying i < f(g_j+1). We therefore find that the roadblocks are precisely those points (k_s – 1, k_s – 1),…, (k_ℓ – 2, k_ℓ – 2) if k_s > 1 and (1, 1),…, (k_ℓ – 2, k_ℓ – 2) if k_s = 1. ■

Because all roadblocks lie on the diagonal under reverse incrementation, eq. 9 can be applied to count coalescent histories. In the application of eq. 9, the distinct points on the diagonal through which monotonic paths cannot pass are (k_s – 1, k_s – 1),…, (k_ℓ – 2, k_ℓ – 2) if k_s > 1 and (1, 1),…, (k_ℓ – 2, k_ℓ – 2) if k_s = 1.

For example, in Figure 7A, the reverse incrementation of leaf labels C, D, and E has k_s = 3 and k_ℓ = 5, so that the roadblocks lie at (2, 2) and (3, 3). The number of coalescent histories is obtained by eq. 9 as C₉ – (C₂C₇ + C₃C₆) + C₂C₁C₆ = 3608.

Figure 7.

The number of coalescent histories for reverse incrementations. (A) G differs from S by a reverse incrementation. (B) G differs from S by a reverse incrementation that includes all labels. (C) G differs from S by a composition of multiple disjoint reverse incrementations.

If the reverse incrementation permutes all the labels, then all points (1, 1),…(n – 1, n – 1) are roadblocks, and the number of coalescent histories is the number of monotonic paths not crossing the diagonal that lies one unit below the y = x line (Figure 7B). As the number of monotonic paths that do not pass above a diagonal of a square lattice, this computation gives C_n–2 coalescent histories. At the same time, the inclusion–exclusion computation of eq. 9 produces a sum that traverses all subsets of the points (1, 1),…, (n – 1, n – 1).

Thus, by use of eq. 9, this construction gives a combinatorial proof of a Catalan number identity.

Corollary 23. The Catalan number C_n–2 can be written as an alternating sum of products of Catalan numbers, where the sum proceeds over all compositions of n – 1:

$$C_{n-2}=\sum _{k=1}^{n-1}(-1{)}^{k+1}\sum _{(v_1,\dots ,v_k)\in \{\mathbf{v}:{\sum }_{i=1}^k{v}_i=n-1,v_i>0\text{for all}i\}}\prod _{i=1}^k{C}_{v_k}$$

This identity can be seen as counting Dyck paths of semi-length n – 1 with no internal returns to the origin in two ways. C_n–2 gives the number of Dyck paths of semi-length n – 2, as a Dyck path of length n – 1 with no internal returns begins with an up-step that is followed by a Dyck path of semi-length n – 2 and then a a down-step. The right-hand side instead uses the inclusion–exclusion principle to perform the computation by excluding Dyck paths of semi-length n – 1 that have at least one return to the origin.

Interestingly, a reverse cycle that permutes all labels, even if it is not an incrementation, gives a Catalan number of coalescent histories, as it generates a roadblock set that consists of one or more diagonal lines. For example, with S = (((((((((A, B), C), D), E), F), G), H), I), J), the reverse incrementation G = (((((((((B, C), D), E), F), G), H), I), J), A) gives C₈ = 1430 coalescent histories (Figure 7B), the reverse cycle G = (((((((((C, D), E), F), G), H), I), J), A), B) gives C₇ = 429 coalescent histories, the reverse cycle G = (((((((((D, E), F), G), H), I), J), A), B), C) gives C₆ = 132 coalescent histories, and so on.

We note also that eq. 9 continues to apply if S differs from G by multiple disjoint reverse incrementations, as in Figure 7C, which adds a two-leaf incrementation—an NNI move—to Figure 7A. In this case, the number of coalescent histories is C₉–(C₂C₇+C₃C₆+C₇C₂)+(C₂C₁C₆+C₂C₅C₂+C₃C₄C₂)–(C₂C₁C₄C₂) = 3002.

5.3. Forward incrementation of the leaf labels

In the case that G represents a forward rather than a reverse incrementation of S, the roadblocks appear in a triangular region rather than exclusively on the diagonal of the square lattice.

Proposition 24. If G is obtained by forward incrementation of S, then the roadblock set B_G,S consists of a triangle of points on and below the diagonal of the square lattice.

Proof. Consider labels k_s, k_ℓ ∈ {1, 2, …, n}, with k_s < k_ℓ and k_ℓ ≠ 2. By definition of forward incrementation, for some component of G with leaves sequentially labeled k_s, k_s + 1, …, k_ℓ from the cherry toward the root, associated leaves of S are labeled π_ks (G) = k_ℓ, π_{ks + 1}(G) = k_s, π_ks+2(G) = k_s + 1, …, π_kℓ (G) = k_ℓ – 1.

We use Proposition 9 and compute the minimal internal edge of S ancestral to each leaf g_k of G, k ∈ {k_s, k_s + 1, …, k_ℓ}. We obtain f(g_k) = k_ℓ – 1.

The roadblocks are the points (i, j) satisfying i < f(g_j+1). Hence, the roadblocks are points (k_s – 1, k_s – 1),…, (k_ℓ – 2, k_s – 1), (k_s, k_s), …,(k_ℓ – 2, k_s), …, (k_ℓ – 2, k_ℓ – 2). ■

We can use Catalan’s trapezoids to count coalescent histories for forward incrementations, noting that every monotonic path from (0, 0) to (n – 1, n – 1) passes through exactly one point on a diagonal line from the lower right corner of the triangle of roadblocks, (k_ℓ – 2, k_s – 1) for 2 ⩽ k_s ⩽ n – 1 and (k_ℓ – 2, 1) for k_s = 1, to the bottom edge or right edge of the lattice (Figure 8A).

Figure 8.

The number of coalescent histories for forward incrementations. (A) G differs from S by a forward incrementation. All paths must pass through the dashed red line. (B) The number of paths from (4, 1) on the dashed red line to (9, 9). (C) The number of paths from (5, 0) on the dashed red line to (9, 9). (D) G differs from S by a composition of two forward incrementations. All paths must pass through the four dashed red lines. The solid line represents the Dyck path associated with the roadblock set (see Figure 5).

If 2 ⩽ k_s ⩽ n – 1, then this line has points (k_ℓ – 1 + c, k_s – 2 – c) for c = 0, 1, …, min(k_s – 2, n – k_ℓ); if k_s = 1, then the line has a single point (k_ℓ – 1, 0). We can combine the two cases with the Kronecker delta, capturing the line with the expression (k_ℓ – 1 + c, k_s – 2 + δ_ks,1 – c) for c = 0, 1, …, min(k_s – 2 + δ_ks,1, n – k_ℓ).

We can then count monotonic paths from (0,0) to some point on the line and from there to (n – 1, n – 1).

Proposition 25. Consider a caterpillar species tree S with n leaves and a caterpillar gene tree G that differs from S by a forward incrementation described by the component k_s, …, k_ℓ; of G. The number of coalescent histories for (G, S) can be written

$$\sum _{c=0}^{\text{min}(k_s-2+{\delta }_{k_s,1},n-k_{\ell })}D(k_{\ell }-1+c,k_s-2+{\delta }_{k_s,1}-c)D_{k_{\ell }-k_s-{\delta }_{k_s,1}+2+2c}(n-k_{\ell }-c,n-k_s+1-{\delta }_{k_s,1}+c).$$

where functions D and D_m follow eqs. 3 and 5, respectively.

Proof. Each monotonic path from (0, 0) to (n – 1, n – 1), proceeds through a point on the diagonal associated with the forward incrementation. The number of paths to arrive at that point from (0, 0) is tabulated by Catalan’s triangle (eq. 3), and the number of paths to reach (n – 1, n – 1) by Catalan’s trapezoids (eq. 5). ■

Figure 8A provides an example. In the figure, k_s = 3 and k_ℓ = 5, so that each roadblocked monotonic path must pass through (4, 1) or (5, 0). Figure 8B illustrates the Catalan trapezoid from (4, 1) to (9, 9), and Figure 8C illustrates the Catalan trapezoid from (5, 0) to (9, 9). Because the number of paths from (0, 0) to (4, 1) is 4 and the number of paths from (0, 0) to (5, 0) is 1, the number of coalescent histories is 4 × 572 + 1 × 429 = 2717. This value is returned by the proposition, which gives ${\sum }_{c=0}^{\min (1,5)}D(4+c,1-c)D_{4+2c}(5-c,8+c)=D(4,1)D_4(5,8)+D(5,0)D_6(4,9)=4\times 572+1\times 429=2717$.

Note that we can analyze cases with multiple disjoint forward incrementations by identifying their associated negatively sloping diagonals through which all monotonic paths must pass. The number of coalescent histories can be obtained by a nested sum counting monotonic paths that pass through exactly one point on each diagonal. Changing the perspective to consider the Dyck path associated with the roadblock set for a composition of disjoint forward incrementations, each peak in the Dyck path generates a diagonal, and we can tabulate monotonic paths that pass through points on each of these diagonals.

For example, in Figure 8D, the Dyck path associated with the roadblock set has four peaks. All monotonic paths from (0, 0) to (9, 9) must pass through two of these, at (1, 0) and (9, 8). The other two peaks generate diagonals through which all monotonic paths must pass, so that all paths must pass through (4, 1) or (5, 0) and through (8, 4) or (9, 3). The number of paths passing through (4, 1) and (8, 4) is D(4, 1)D₄(4, 3)D₅(1, 5) = 700; the number of paths through (4, 1) and (9, 3) is D(4, 1)D₄(5, 2)D₇(0, 6) = 84; the number through (5, 0) and (8, 4) is D(5, 0)D₆(3, 4)D₅(1, 5) = 175; and the number through (5, 0) and (9, 3) is D(5, 0)D₆(4, 3)D₇(0, 6) = 35. In total, the number of paths is 994.

With this perspective, we can see that such an approach to enumeration applies to any Dyck path, not just those that represent disjoint forward incrementations: for every peak in the Dyck path, a diagonal list of points is generated through which each monotonic path from (0, 0) to (n – 1, n–1) must pass. We consider all possible choices of points, one on each diagonal, and tabulate paths through those points by use of Catalan’s triangle and Catalan’s trapezoids. For a general pair of caterpillar trees, such an approach can reduce the number of summations in eq. 8 from n – 1 to the number of peaks in the associated Dyck path.

The number of Dyck paths of semilength n with exactly k peaks follows the Narayana numbers $N(n,k)=\frac{1}{n}\left(\begin{array}{c}\hfill n\hfill \\ \hfill k\hfill \end{array}\right)\left(\begin{array}{c}\hfill n\hfill \\ \hfill k-1\hfill \end{array}\right)$ [8, Section 6.1]. The mean number of peaks in a Dyck path chosen at random then follows ${\sum }_{k=1}^n\frac{k}{n}\left(\begin{array}{c}\hfill n\hfill \\ \hfill k\hfill \end{array}\right)\left(\begin{array}{c}\hfill n\hfill \\ \hfill k-1\hfill \end{array}\right)∕C_n$, which, by noting $k\left(\begin{array}{c}\hfill n\hfill \\ \hfill k\hfill \end{array}\right)=n\left(\begin{array}{c}\hfill n-1\hfill \\ \hfill k-1\hfill \end{array}\right)$ and applying eq. 5.23 in Table 169 of Graham et al. [11] to complete the summation, gives a mean of (n + 1)/2. Thus, because we consider semi-length n – 1, this approach reduces the mean number of nested summations from n – 1 in eq. 8 to n/2.

6. Discussion

We have studied coalescent histories for non-matching caterpillar gene trees and species trees, showing that as in the matching case, the number of coalescent histories for non-matching caterpillars can be computed using monotonic paths that do not cross the diagonal of a square lattice (Section 3). The recursion for the number of coalescent histories that applies for arbitrary gene trees and species trees simplifies for non-matching caterpillars to a non-recursive formula dependent only on the caterpillar topologies (Section 4). Using these results, we have counted coalescent histories for non-matching caterpillars differing by nearest-neighbor-interchange (Section 5.1). By studying reverse and forward incrementation, we have also counted coalescent histories for caterpillars differing by subtree-prune-and-regraft (Sections 5.2 and 5.3).

The bijection that connects coalescent histories and monotonic paths (Proposition 9) makes use of roadblocks, lattice points through which paths are not permitted to travel. Roadblocks occur such that if a point (i, j) is a roadblock for i ⩾ j, then (k, j) is also a roadblock for each k with j ⩽ k ⩽ i, as is (i, ℓ;) for each ℓ; with j ⩽ ℓ; ⩽ i (Remark 10). Enumeration of roadblocked monotonic paths given a roadblock set connects to Catalan’s triangle and trapezoids, enabling enumeration of the associated coalescent histories. Interestingly, the distinct roadblock sets can themselves be put into bijection with the monotonic paths that do not cross the diagonal of a square lattice, so that their number also follows the Catalan sequence (Section 3.3).

Our construction linking coalescent histories and roadblocked monotonic paths enables a simple proof of a result of Degnan & Rhodes [3] that for a fixed number of leaves, matching caterpillar trees have more coalescent histories than do non-matching caterpillar trees (Corollary 11). In particular, the lattice construction that enumerates coalescent histories for a non-matching pair of caterpillar trees contains at least one roadblock, whereas the lattice for matching caterpillars has no roadblocks and therefore has more monotonic paths. For a fixed caterpillar species tree, we have identified exactly which non-matching caterpillar gene tree generates the most coalescent histories: it is immediate that this gene tree differs from the species tree by a single NNI move, as the caterpillars differing from the species tree by one NNI move are the only ones that produce only one roadblock. We find that the specific NNI move affecting leaves nearest the “middle” of the species tree generates the largest number of coalescent histories, and that as the number of leaves increases, this value is asymptotically equivalent to the Catalan number C_n–1 (Section 5.1).

The case in which the gene tree differs from the species tree by reverse incrementation produces an elegant result. Recalling that the number of coalescent histories for matching caterpillars is described by a Catalan number, if the gene tree is obtained by a reverse incrementation affecting all leaf labels of the species tree, then the number of coalescent histories is given by the next-smaller Catalan number (Section 5.2). The case of forward incrementation is more complex, but it can be analyzed using Catalan’s trapezoids and suggests further connections to the analysis of Dyck paths (Section 5.3).

This study provides some of the first systematic closed-form results concerning coalescent histories for non-matching gene trees and species trees. Our approach applies only to caterpillars, however, as the bijection with roadblocked monotonic paths relies on the fact that the internal nodes of a caterpillar tree can be placed in a sequence such that all pairs of internal nodes have an ancestor–descendant relationship. It does suggest, however, that connections to other combinatorial structures such as Dyck paths can assist in enumerating coalescent histories for non-matching gene trees and species trees beyond use of the recursion in eq. 1.

A question that remains open is that the set of integers that could equal the number of coalescent histories for some caterpillar gene tree and species tree is unknown. Rosenberg & Degnan [19, Table 1] observed that for fixed species trees S of size n and certain values t, particularly small ones, large numbers of pairs (G, S) had exactly t coalescent histories, and Rosenberg [18] enumerated the pairs (G, S) with exactly 1 coalescent history (the lonely pairs). We and Degnan & Rhodes [3] have shown that if G and S are caterpillars, then only values t ⩽ C_n–1 can represent the number of coalescent histories. Our NNI results show that all values in the open interval (C_n–1 – C_{⌊(n–1)/2⌋}C_{⌈(n–1)/2⌉}, C_n–1) cannot be the number of coalescent histories for (G, S). For fixed caterpillar S with n leaves, it is useful to obtain the size of the set of values of t for which the pair (G, S) has exactly t coalescent histories. We observed that $(C_{n-1}+\left(\begin{array}{c}\hfill n-1\hfill \\ \hfill \lfloor (n-2)∕2\rfloor \hfill \end{array}\right))∕2$, the number of symmetric caterpillar-friendly roadblock sets plus half the number of asymmetric caterpillar-friendly roadblock sets, provides an upper bound (Section 3.3).

We note that the question of identifying the integers that represent the number of coalescent histories for some (G, S) with n leaves can be phrased entirely in terms of roadblocked monotonic paths without reference to coalescent histories. Describe a lattice as monotonically roadblocked if for each roadblock (i, j) with i ⩾ j, (k, j) is also a roadblock for each k with j ⩽ k ⩽ i, and (i, ℓ;) is a roadblock for each ℓ; with j ⩽ ℓ; ⩽ i. We seek the number of integers that represent the number of monotonic paths that do not cross the diagonal of some monotonically roadblocked lattice. That the bijection between coalescent histories and roadblocked monotonic paths raises such questions illustrates that constructions enabled by this bijection can be fruitful for studies of the properties of the paths themselves.