Enumeration of coalescent histories for caterpillar species trees and p-pseudocaterpillar gene trees

Abstract

For a fixed set X containing n taxon labels, an ordered pair consisting of a gene tree topology G and a species tree topology S bijectively labeled with the labels of X possesses a set of coalescent histories—mappings from the set of internal nodes of G to the set of edges of S describing possible lists of edges in S on which the coalescences in G take place. Enumerations of coalescent histories for gene trees and species trees have produced suggestive results regarding the pairs (G, S) that, for a fixed n, have the largest number of coalescent histories. We define a class of 2-cherry binary tree topologies that we term p-pseudocaterpillars, examining coalescent histories for non-matching pairs (G, S) in the case in which S has a caterpillar shape and G has a p-pseudocaterpillar shape. Using a construction that associates coalescent histories for (G, S) with a class of “roadblocked” monotonic paths, we identify the p-pseudocaterpillar labeled gene tree topology that, for a fixed caterpillar labeled species tree topology, gives rise to the largest number of coalescent histories. The shape that maximizes the number of coalescent histories places the “second” cherry of the p-pseudocaterpillar equidistantly from the root of the “first” cherry and from the tree root. A symmetry in the numbers of coalescent histories for p-pseudocaterpillar gene trees and caterpillar species trees is seen to exist around the maximizing value of the parameter p. The results provide insight into the factors that influence the number of coalescent histories possible for a given gene tree and species tree.

1. Introduction

In mathematical phylogenetics, a coalescent history represents the paired list of coalescences in a gene tree together with their associated edges of a species tree. Consider two binary, rooted, leaf-labeled trees, G and S, with leaves labeled by the same label set X, such that each label in X is associated with exactly one leaf of G and exactly one leaf of S. We regard G as a gene tree representing the evolution of genealogical lineages in a group of species, and S as the species tree representing the evolutionary descent of the species themselves.

For a gene tree G evolving on a species tree S, a coalescent history is a mapping from the set of internal nodes of G to the set of internal edges of S, such that two rules are followed: (i) the image of an internal node v of G is ancestral in S to each leaf of S that shares a label with some leaf descended from v in G; (ii) the image of an internal node v of G is ancestral in S to the images of each of its descendant nodes. The biological interpretation of (i) is that a set of gene lineages can only find a common ancestor on a species tree edge that it is possible for them all to reach; the interpretation of (ii) is that the gene lineages descended from a descendant node coalesce at least as recently as do the gene lineages descended from its ancestral nodes. Note that we regard a node as trivially ancestral to and descended from itself. The coalescent histories for (G, S) can be viewed as describing a discrete class of evolutionary scenarios for the lineages of G on the edges of S.

A variety of studies have enumerated the coalescent histories for pairs (G, S), both by a recursive approach that applies for all (G, S) [13, 18], and by closed-form formulas and bijective constructions suited to particular families of trees [1, 5, 6, 9, 13, 14, 15, 16]. These enumeration studies, primarily considering matching gene trees and species trees with G = S and having particular emphasis on shapes such as caterpillars, 4-pseudocaterpillars, and caterpillar-like families (Figure 1), have informally observed that in specified classes of trees, the largest number of coalescent histories tends to occur when the pair (G, S) has two features: multiple sequences exist in which the coalescences of G can be arranged, and many edges of S exist on which those coalescences can take place.

Figure 1:

Three tree shapes with n = 11 leaves. (A) Caterpillar tree shape. (B) p-pseudocaterpillar tree shape. For this tree, p = 8. (C) Caterpillar-like tree shape. The seed tree has size 9.

Rosenberg [13] observed that for small trees with at most n = 9 leaves and G = S, the largest numbers of coalescent histories among tree pairs (G, G) with fixed n were seen for trees that had structure similar to caterpillar trees, but that unlike caterpillars, had more than one possible sequence of coalescences. Rosenberg [14] and Disanto & Rosenberg [6] examined tree families (G, G), with n growing arbitrarily large in specified caterpillar-like tree families. Beginning with a seed tree, these studies generated families of increasingly large trees by sequentially adding taxa so that the next tree in a family was formed by placing the current tree and a single leaf on opposite sides of a new root. They saw that across all seed trees of a fixed small size, as the number of leaves grew without bound, the largest numbers of coalescent histories were achieved when the seed tree had many different sequences in which its coalescences could be arranged. Disanto & Rosenberg [5] constructed a tree family, the lodgepole family, that, unlike caterpillar families, grows so that as the number of leaves increases, trees accumulate both new sequences in which coalescences can be arranged and new places for them to occur. This family is the family of matching tree pairs (G, G) with the largest-known number of coalescent histories as n increases without bound.

Despite many observations suggesting that coalescent histories tend to increase in number when G has many sequences in which coalescences can be arranged and many edges on which those coalescences can occur, existing results in support of this view have focused on small trees [13] and on informal interpretations of specific families with large limits as n → ⋈ [6, 14]; no result has formally demonstrated the observation in a class of trees for a fixed finite n of arbitrary size. We devise a scenario to formalize this idea characterizing cases with the largest numbers of coalescent histories. We fix the species tree S to be a caterpillar, and we consider a family of non-matching gene trees G, the p-pseudocaterpillars. We show that in this class of non-matching pairs (G, S) with fixed n, the largest number of coalescent histories is achieved precisely when G combines the two elements: many coalescence sequences, and many edges on which those coalescences can take place.

Our approach is one of relatively few to examine enumerations of coalescent histories in the case that G is not necessarily equal to S [9, 13, 15, 16, 18]. The strategy employs a construction that enumerates coalescent histories for non-matching caterpillar trees. Generalizing a result of Degnan [1] for enumeration of coalescent histories for matching caterpillar trees, Himwich & Rosenberg [9] produced a bijection with monotonic paths for use in enumerating coalescent histories for non-matching caterpillar pairs (G, S). We use this monotonic-path construction to enumerate coalescent histories for the class of non-matching trees that considers a caterpillar species tree S and a p-pseudocaterpillar gene tree G.

Section 2 introduces definitions and notation. Section 3 gives an example that motivates the general calculation. In Section 4, we enumerate coalescent histories in the general case. Section 5 gives special cases with specified values of p. Finally, in Section 6, for a specified caterpillar species tree S of fixed size n, considering all possible values of p, we obtain the maximal number of coalescent histories across all non-matching p-pseudocaterpillar trees G. Section 7 discusses a symmetry in p for fixed n, and we conclude with a discussion in Section 8. The computations illustrate how the monotonic path approach of Himwich & Rosenberg [9] in the case of caterpillar species trees can be extended to enumerate coalescent histories in more cases beyond that of caterpillar gene trees.

2. Preliminaries

We formally define coalescent histories and p-pseudocaterpillars in Sections 2.1 and 2.2, and we introduce results concerning the Catalan numbers in Section 2.3. In Section 2.4, we describe the use of monotonic paths to enumerate coalescent histories for caterpillar tree pairs.

2.1. Coalescent histories

The definitions in this article closely follow Himwich & Rosenberg [9]. Henceforth, we treat all “trees” as binary and rooted; trees are leaf-labeled, but the labels are sometimes omitted for convenience. The set of vertices or nodes of a tree can be divided into leaf nodes and non-leaf internal nodes. For tree G, we say that a node v₁ is descended from a node v₂ if the path from v₁ to the root of G travels through v₂; v₂ is then ancestral to v₁. Ancestor–descendant relationships also apply to edge–edge pairs and edge–node pairs. A node or edge is trivially descended from and ancestral to itself. Each internal node, including the root, possesses an associated internal edge immediately ancestral to it. Thus, viewed forward in time, the root—like other internal nodes—has in-degree 1 and out-degree 2.

We consider pairs (G, S) in which G represents a gene tree, describing the descent of a set of genealogical lineages, and S represents a species tree, describing the descent of a set of species. G and S are assumed to have the same number of leaves, n. We assume that the leaf set of G and the leaf set of S are labeled by the same label set X, and that each label in X is assigned to exactly one leaf of G and to exactly one leaf of S. This assumption corresponds to an assumption that exactly one gene lineage is sampled in each of the n species.

For the pair (G, S), we can formally define the functions known as coalescent histories.

Definition 1.Consider a pair of trees (G, S) that are binary, rooted, and leaf-labeled, with the labels in bijective correspondence. A coalescent history is a function α from the set of internal nodes of G to the set of internal edges of S, satisfying two conditions:

1. For each internal node v in G, all labels for leaves descended from v in G label leaves descended from edge α(v) in S.

2. For each pair of internal nodes υ₁and υ₂in G, if υ₂is descended from υ₁, then α(v₂) is descended from α(υ₁) in S.

In this definition, nodes of G represent coalescent events for the gene lineages, and edges of S represent species tree edges along which the gene lineages evolve. A coalescent history reflects the biological process of coalescence, in which descendants cannot coalesce farther back in time than their ancestors. Ancestor–descendant relations are preserved under the mapping α.

Informally, considering the biological process of coalescence, we often refer to internal nodes as coalescences. We refer to gene tree coalescences “happening,” “occurring,” or “taking place” on species tree edges. For example, a gene tree coalescence v in G “can occur” on a species tree edge w in S if and only if there exists a coalescent history α with α(υ) = w.

2.2. Caterpillars and p-pseudocaterpillars

As our goal is to enumerate the coalescent histories in the case that G has a p-pseudocaterpillar topology and S has a caterpillar topology, we define caterpillar and p-pseudocaterpillar shapes for binary, rooted tree topologies.

Definition 2.A caterpillar tree is a binary, rooted tree that has an internal node that is descended from all other internal nodes.

In a caterpillar tree, each internal node has at least one leaf as an immediate descendant (Figure 1A). Equivalently, a caterpillar tree is a tree that has only one cherry node: an internal node with exactly two descendant leaves. Rosenberg [13] defined binary, rooted pseudocaterpillar trees with n ≥ 4 leaves as trees in which all internal nodes except one have at least one immediate leaf descendant. The node that provides the exception has two cherry nodes as its immediate descendants. We generalize the earlier definition of pseudocaterpillar trees to consider generalized pseudocaterpillar trees. To define this concept, we denote by υ_L and υ_R the left and right descendant nodes of an internal node υ.

Definition 3.A generalized pseudocaterpillar tree is a binary, rooted tree that has at least four leaves and that satisfies two conditions. (i) The tree possesses exactly two cherry nodes. (ii) For each internal node v, at least one of υ_L, υ_R has no more than two descendant leaves.

In other words, a generalized pseudocaterpillar tree is formed from a caterpillar tree, replacing one of the leaves not descended from the unique cherry node by a second cherry node (Figure 1B).

A generalized pseudocaterpillar can be described by two numbers: the total number of leaves n and the position p of the “second” cherry. To precisely identify p for a generalized pseudocaterpillar tree, we label the leaves by natural numbers starting from left to right, placing the “first” cherry—the one present in the caterpillar from which the generalized pseudocaterpillar has been generated—on the left. We define the position of the second cherry as the number corresponding to its second leaf from the left (Figure 2A). A generalized pseudocaterpillar tree with a second cherry in position p, 4 ≤ p ≤ n, is termed a p-pseudocaterpillar tree. The pseudocaterpillar trees in the sense of Rosenberg [13] are 4-pseudocaterpillars.

Figure 2:

Labeled p-pseudocaterpillar gene tree and caterpillar species tree. (A) A p-pseudocaterpillar gene tree G with coordinates for the leaves. (B) A species tree S with the internal edges labeled. Both trees have n leaves. The trees are drawn in canonical form, so that the path from the left-most leaf to the root contains all other internal nodes for the caterpillar, and all other internal nodes except one for the p-pseudocaterpillar.

Note that for convenience, our description of the value of p in a p-pseudocaterpillar tree relies on a canonical orientation of the tree. This value of p can also be identified as the number of leaves in the smallest subtree that contains both cherries of the tree.

Our interest is in the case in which the gene tree has a p-pseudocaterpillar topology for some p, and the species tree has a caterpillar topology. A species tree with n leaves has n − 1 edges on which gene tree coalescences can happen; we also label these edges with natural numbers, following the order from Degnan & Salter [3] (Figure 2B).

2.3. Catalan numbers

It is useful to introduce the Catalan number sequence 1, 1, 2, 5, 14, 42, 132, 429, …, as it features prominently in our analysis. Letting ${\mathcal{C}}_n$ be the nth Catalan number for n ≥ 0,

$${\mathcal{C}}_n=\left(\begin{array}{c}2n\\ n\end{array}\right)-\left(\begin{array}{c}2n\\ n-1\end{array}\right)=\frac{1}{n+1}\left(\begin{array}{c}2n\\ n\end{array}\right).$$ (1)

Considering the many combinatorial interpretations of this sequence [8, 17], we make use of the fact that ${\mathcal{C}}_n$ is the number of monotonic paths that travel from (0, 0) to (n, n) on a square lattice of size n × n, that begin with a 1-unit step to the right, and that do not cross the diagonal from (0, 0) to (n, n), where a monotonic path is a path that proceeds exclusively by 1-unit steps up or to the right.

A Catalan triangle is a combinatorial structure that counts monotonic paths to lattice points on or below the diagonal [12]. Entry (n, k) of the Catalan triangle gives the number of monotonic paths on the square lattice that travel from the origin to a point (n, k) and that do not travel above the y = x line. The number of such paths, which have n “right-steps” and k “up-steps,” is [12]:

$$C\left(n,k\right)=\{\begin{array}{ll}1\hfill & k=0\hfill \\ \left(\begin{array}{c}n+k\\ k\end{array}\right)-\left(\begin{array}{c}n+k\\ k-1\end{array}\right)\hfill & 1\le k\le n\hfill \\ 0\hfill & k>n.\hfill \end{array}$$ (2)

For n ≥ 1 and 1 ≤ k ≤ n, this function satisfies the first-order recurrence

$$C\left(n,k\right)=C\left(n-1,k\right)+C\left(n,k-1\right).$$

A Catalan trapezoid is obtained in a similar way, except that we allow m − 1 additional up-steps to happen starting at the origin, so that monotonic paths that do not travel above the diagonal from (0, m − 1) to (n, n + m − 1) are tabulated. The number m is called the order of the trapezoid; m = 1 corresponds to the Catalan triangle. Entry (n, k) of the Catalan trapezoid of order m is given by

$$C_t\left(n,k,m\right)=\{\begin{array}{ll}\left(\begin{array}{c}n+k\\ k\end{array}\right)\hfill & 0\le k\le m-1\hfill \\ \left(\begin{array}{c}n+k\\ k\end{array}\right)-\left(\begin{array}{c}n+k\\ k-m\end{array}\right)\hfill & m\le k\le n+m-1\hfill \\ 0\hfill & k>n+m-1,\hfill \end{array}$$ (3)

and it satisfies a similar recurrence C_t(n, k, m) = C_t(n − 1, k, m) + C_t(n, k − 1, m) for n ≥ 1 and 1 ≤ k ≤ n + m − 1. With the origin in the lower left corner, the first colums of the Catalan triangle and the Catalan trapezoid of order 3 appear below.

2.4. Bijection between coalescent histories and monotonic paths for caterpillars

Building on work of Degnan [1], the bijective construction of Himwich & Rosenberg [9] enumerates coalescent histories for pairs consisting of a caterpillar gene tree and a caterpillar species tree by bijectively associating each coalescent history with a monotonic path that does not travel above the diagonal of a square lattice. The coalescent histories are then enumerated by counting the bijectively-associated monotonic paths.

In the construction, given a caterpillar species tree S and a caterpillar gene tree G with n leaves, a square (n − 1) × (n − 1) lattice is examined. The coalescent histories for (G, S) correspond to monotonic paths from (0, 0) to (n − 1, n − 1), with each right-step corresponding to a species tree internal edge, and each up-step corresponding to a gene tree coalescence. The pair (G, S) specifies a set of roadblocks, points in the lattice through which monotonic paths are not permitted to travel. The number of coalescent histories for (G, S) then equals the number of monotonic paths that do not travel above the diagonal and that do not travel through any of the roadblocks. In the case that G and S have the same caterpillar labeled topology, no roadblocks exist, and the number of monotonic paths, and hence the number of coalescent histories, is the Catalan number ${\mathcal{C}}_{n-1}$ [1].

The construction of Himwich & Rosenberg [9] also applies to caterpillar subtrees. Suppose G possibly has fewer leaves than S, so that the label set for G is a subset of the label set for S. If we use the term partial coalescent history to describe mappings that satisfy Definition 1 except that the label set of G is a subset of the label set of S rather than a bijectively-associated label set, then the number of partial coalescent histories for a caterpillar pair (G, S) is obtained by counting roadblocked monotonic paths to an associated point that is not necessarily the point (n − 1, n − 1).

For details, see Himwich & Rosenberg [9]. We illustrate the construction in an example.

3. Example

Our approach to extending the construction of Himwich & Rosenberg [9] to count coalescent histories for a caterpillar species tree and a non-matching p-pseudocaterpillar gene tree—a tree with one extra cherry—can be understood with an example. Consider a gene tree G with 10 leaves, with cherry node (E,F) as shown in Figure 3A, and a species tree S as shown in Figure 3B.

Figure 3:

Example (gene tree, species tree) pair. (A) 6-pseudocaterpillar gene tree G with “second cherry” (E,F). The pivotal coalescence is circled in red. (B) Caterpillar species tree S.

The key to counting coalescent histories for (G, S) is to examine the specific gene tree coalescence circled in red in Figure 3A, indicating the most recent common ancestor of both cherries of G. We call this node the pivotal coalescence. In a coalescent history, this pivotal coalescence can take place on any internal edge ancestral to species F in the species tree. We partition all coalescent histories for (G, S) by the position of this pivotal coalescence. For each placement of the pivotal coalescence, we then count the number of coalescent histories by counting monotonic paths on particular diagrams for the subtrees generated by the pivotal coalescence.

Suppose the pivotal coalescence of G happens on edge 5 of S. Then all the coalescences in the “left” subtree descended from the pivotal coalescence must happen on or before edge 5. This left subtree is now a caterpillar (((A,B),C),D), coalescing on a caterpillar (((((A,B),C),D),E),F). We can now follow the construction of Himwich & Rosenberg [9] to enumerate partial coalescent histories through a bijection with monotonic paths.

In particular, the number of ways that the gene tree coalescences of (((A,B),C),D) can occur on species tree (((((A,B),C),D),E),F) is equal to the number of monotonic paths on a Catalan triangle restricted to 5 right-steps and 3 up-steps (Figure 4A). The up-steps correspond to the 3 coalescences in the subtree (((A,B),C),D), and the right-steps correspond to the 5 edges of (((((A,B),C),D),E),F) on which they can take place. Following eq. (2), the number of monotonic paths that travel from (0, 0) to (5, 3) and that do not travel above the diagonal is $\left(\begin{array}{c}8\\ 3\end{array}\right)-\left(\begin{array}{c}8\\ 2\end{array}\right)=28$. Because gene tree coalescence (E,F) must occur on species tree edge 5 when the pivotal coalescence occurs on edge 5, coalescence (E,F) does not introduce additional coalescent histories. Thus, 28 possible partial coalescent histories place the pivotal coalescence on species tree edge 5.

Figure 4:

Catalan triangle construction for enumerating coalescent histories for Figure 3. Following Himwich & Rosenberg [9], up-steps represent gene tree coalescences and are labeled in the diagram by the leaf participating in the coalescence (by both leaves for the first coalescence). Right-steps represent species tree internal edges and are labeled in the diagram by the leaf immediately descended from the associated edge (by both leaves for the first edge). The numbers indicated represent counts of monotonic paths according to eq. (2). (A) Diagram corresponding to the left subtree descended from the pivotal gene tree coalescence. (B) Diagram corresponding to the portion of the gene tree ancestral to the pivotal coalescence. The arrow indicates that all coalescences other than those depicted in the diagram have already happened by the starting point, the first species tree internal edge ancestral to F.

We now need to consider the coalescences ancestral to the pivotal coalescence. Coalescences involving leaf G can happen on edge 6 or on any edge ancestral to 6, coalescences with leaf H can happen on edge 7 or any edge ancestral to 7, provided that leaf G has already participated in a coalescence, and so on. Again following the construction of Himwich & Rosenberg [9], the possible assignments of gene tree coalescences to species tree edges in this upper part of the species tree can be described by a Catalan triangle with 4 right-steps and 4 up-steps (Figure 4B). There are 14 possible monotonic paths.

To obtain the total number of coalescent histories with pivotal coalescence on edge 5, we now multiply the two numbers we already have: for each of the 28 partial coalescent histories for coalescences descended from the pivotal coalescence, there are 14 ways for the coalescences ancestral to it to happen. Hence, 392 coalescent histories exist with pivotal coalescence on edge 5.

To obtain the total count of coalescent histories for (G, S), we must consider all other possible locations of the pivotal coalescence, and sum their associated numbers of coalescent histories. With this idea, however, we are now ready for the general case.

4. General construction

The example in Section 3 illustrates that we can enumerate coalescent histories for a p-pseudocaterpillar gene tree and a caterpillar species tree by dividing the problem into three components: placement of the pivotal coalescence, and two enumerations, one for coalescences descended from the pivotal coalescence, and the other for coalescences ancestal to it. We describe these two enumerations in full generality, and complete the calculation by summing over all placements of the pivotal coalescence.

Consider a p-pseudocaterpillar gene tree G and a caterpillar species tree S, both with n leaves, that are bijectively labeled with the same set of distinct labels. Suppose G and S have an identical leaf labeling, by which we mean that when G and S are drawn in canonical form (Figure 2), the gene tree and species tree labels are listed in the same order when reading them from left to right. Figure 3 illustrates an identical leaf labeling. Note that labelings in which the labels in one or both cherries of G are transposed with respect to S also qualify as identical.

Using our numerical labeling scheme for edges of gene trees and species trees (Figure 2), the pivotal coalescence can take place on any species tree edge from p − 1 to n − 1. Suppose it happens on edge k, p − 1 ≤ k ≤ n − 1.

4.1. Coalescences descended from the pivotal coalescence

Label by S_k the subtree of S whose root is the node immediately descended from edge k. Label the subtree of G whose root node is the pivotal coalescence by G_∗. The left subtree of G_∗, which we denote G_∗ℓ, is a caterpillar with p − 3 coalescences. By the assumption that the pivotal coalescence takes place on species tree edge k, all coalescences in G_∗ℓ must occur on edges 1, 2, …, k.

Following Himwich & Rosenberg [9], the partial coalescent histories for (G_∗ℓ, S_k), with p − 3 gene tree coalescences and k species tree edges on which they take place, correspond to monotonic paths from (0, 0) to (k, p − 3) that do not travel above the y = x line. The number of partial coalescent histories therefore corresponds to Catalan triangle entry (k, p − 3). By eq. (2), this quantity, which we denote ℓ_k, equals

$${\ell }_k=\left(\begin{array}{c}k+p-3\\ p-3\end{array}\right)-\left(\begin{array}{c}k+p-3\\ p-4\end{array}\right).$$ (4)

The right subtree of G_∗, or G_∗r, has exactly one coalescence, which can happen on any of the edges p − 1, p, …, k. Hence, the number of coalescent histories for (G_∗r, S_k) is

$$r_k=k-p+2.$$ (5)

Combining the left and right subtrees of G_∗, from eqs. (4) and (5), the number of partial coalescent histories for (G_∗, S_k) is ℓ_kr_k.

4.2. Coalescences ancestral to the pivotal coalescence

To examine coalescences ancestral to the pivotal coalescence, the pivotal coalescence can be viewed as a “leaf” of a caterpillar gene tree G^∗ whose coalescences occur on species tree edges numbered k or greater. In this view, G^∗ is the (n − p + 1)-leaf caterpillar tree in which the subtree rooted at the pivotal coalescence is replaced by a leaf, so that the pivotal coalescence is a leaf in the cherry of G^∗.

G^∗ has n − p gene tree coalescences, which take place on species tree edges k, k + 1, …, n − 1, a total of n − k edges. It is possible for multiple coalescences in G^∗ to occur on edge k; taking into account that edge k has k + 1 descendant leaves, and p − 1 coalescences have already occured including the pivotal coalescence, at most k − p + 1 coalescences of G^∗ can occur on edge k.

The coalescences of G^∗ therefore correspond to monotonic paths that do not travel above a specified diagonal of a trapezoidal lattice. The number of right-steps is n − k − 1, one for each non-root edge on which coalescences take place, and the number of up-steps is n − p, one for each gene tree coalescence in G^∗. The order of the trapezoid is k − p + 2, one more than the number of coalescences of G^∗ that can occur on the initial edge k.

We can count these monotonic paths using eq. (3), or by noting that the number of monotonic paths is symmetric with respect to interchange of the starting and ending points. In other words, the number of monotonic paths up and to the right from the lower left vertex of a trapezoid to the upper right vertex is the same as the number of monotonic paths down and to the left from the upper right vertex to the lower left vertex. By this symmetry, the number of paths on a Catalan trapezoid is then equal to one of the entries in the right-most column of some Catalan triangle. Assigning coordinates on the lattice, we have

$$C\left(n-p,n-k-1\right)=C_t\left(n-k-1,n-p,k-p+2\right).$$ (6)

A visual explanation appears in Figure 5.

Figure 5:

Monotonic path construction for coalescences ancestral to the pivotal coalescence. (A) Exchanging the starting and ending points of monotonic paths, the number of monotonic paths in a trapezoidal lattice is equal to an entry in a Catalan triangle. Following the notation of Figure 3 with n = 10 and p = 6, suppose the pivotal coalescence happens on species tree edge k = 7. Up to two additional coalescences can happen on edge k = 7, producing a trapezoid. Thus, the figure encodes partial coalescent histories for gene tree coalescences (((((A,B),C),D),E),F) with G, ((((((A,B),C),D),E),F),G) with H, (((((((A,B),C),D),E),F),G),H) with I, and ((((((((A,B),C),D),E),F),G),H),I) with J on species tree edges ancestral to A, B, C, D, E, F, G, and H. The number of monotonic paths is obtained from a Catalan trapezoid of order k − p + 2 = 3 with n − k − 1 = 2 right-steps and n − p = 4 up-steps. By a symmetry in which the number of permissible paths from starting point (0, 0) to ending point (n − k − 1, n − p) is equivalent to the number of permissible paths from the ending point to the starting point, we can count paths from this alternate perspective, evaluating entry (n − p, n − k − 1) = (4, 2) of a Catalan triangle. (B) Monotonic path construction for each of the n − p + 1 = 5 options for placement of the pivotal coalescence. For a placement of the pivotal coalescence shown in the left-hand diagram, the number of coalescent histories for (G^∗, S) is obtained by counting monotonic paths in the right-hand diagram from the lower-left vertex to an associated point on the right-hand edge.

Denote by u_k the number of coalescent histories for (G^∗, S). Using eq. (3) or the symmetry argument with eq. (2) to count monotonic paths in a triangular lattice with n − p right-steps and n − k − 1 up-steps, we have

$$u_k=C\left(n-p,n-k-1\right)=\left(\begin{array}{c}2n-p-k-1\\ n-k-1\end{array}\right)-\left(\begin{array}{c}2n-p-k-1\\ n-k-2\end{array}\right).$$ (7)

4.3. Full formula

We have shown that the number of partial coalescent histories for (G_∗, S) that place the pivotal coalescence on edge k is ℓ_kr_k, and that for each of these partial coalescent histories, the number of partial coalescent histories for (G^∗, S) is u_k. Because each coalescent history for (G, S) consists of a partial coalescent history for (G_∗, S), a placement of the pivotal coalescence, and a partial coalescent history for (G^∗, S), the number of coalescent histories for the case in which the pivotal coalescence happens on edge k is ℓ_kr_ku_k. Summing over values of k, we have proven the following theorem.

Theorem 4.Consider a caterpillar species tree S with n ≥ 4 leaves and an identically-labeled p-pseudocaterpillar gene tree G with n leaves and 4 ≤ p ≤ n. With C(n, k) as in eq. (2), the number of coalescent histories for (G, S) is

$$h\left(n,p\right)=\sum _{k=p-1}^{n-1}{\ell }_k{r}_k{u}_k=\sum _{k=p-1}^{n-1}C\left(k,p-3\right)\left(k-p+2\right)C\left(n-p,n-k-1\right).$$ (8)

Note that eq. (8) can be seen to apply for (n, p) with p = 3 and 3 ≤ p ≤ n, and hence for n = 3. In this case, G is viewed as a caterpillar gene tree whose cherry joins leaves 2 and 3. G_∗ℓ has no coalescences, so ℓ_k = 1; this enumeration accords with the definition of the function C in eq. (2), where we have C(k, p − 3) = C(k, 0) = 1 for all k.

A convenient form of eq. (8) for computation is as follows:

$$h\left(n,p\right)=\sum _{k=p-1}^{n-1}\frac{{\left(k-p+2\right)}^2\left(k-p+4\right)\left(2n-p-k-1\right)!\left(k+p-3\right)!}{\left(k+1\right)!\left(n-k-1\right)!\left(n-p+1\right)!\left(p-3\right)!}.$$ (9)

4.4. Identical and non-identical leaf labelings

The results of Himwich & Rosenberg [9] enable a result on leaf labelings. We claim that for a fixed caterpillar species tree, an identically-labeled p-pseudocaterpillar gene tree—the focus of our analysis—has strictly more coalescent histories than any non-identically-labeled p-pseudocaterpillar. The argument is that any non-identically-labeled gene tree introduces at least one “roadblock,” decreasing its associated number of monotonic paths compared to the case of identical labels.

Proposition 5.Consider a caterpillar species tree S with n ≥ 4 leaves and a value of p, 4 ≤ p ≤ n. The number of coalescent histories for (G, S), with G a p-pseudocaterpillar gene tree bijectively labeled with the same n labels as S, is bounded above by h(n, p), with equality if and only if G and S are identically labeled.

Proof. Theorem 4 demonstrates that the number of coalescent histories is h(n, p) in the identically-labeled case. We must show that a non-identically-labeled G produces fewer coalescent histories.

Fix n and p. Consider caterpillar species tree S and p-pseudocaterpillar gene tree G, bijectively labeled with n labels {A₁, A₂, …, A_n}, but not necessarily identically labeled. Suppose that from left to right, A₁, A₂, …, A_n label the leaves of S when S appears in canonical form.

In eq. (8), the form of the equation $h\left(n,p\right)={\sum }_{k=p-1}^{n-1}{\ell }_k{r}_k{u}_k$ has a sum from k = p − 1 to n − 1 of a product of three quantities. Each quantity counts the number of monotonic paths on a Catalan triangle—trivially so in the case of $r_k=C\left(k-p+2,1\right)=k-p+2$, which represents the number of monotonic paths that proceed k − p + 2 steps to the right and one step up.

If we now change G to a possibly non-identically-labeled p-pseudocaterpillar G′, then the coalescent histories can be enumerated by a corresponding decomposition $h\prime \left(n,p\right)={\sum }_{k=p-1}^{n-1}{\ell }_k^{\prime }{r}_k^{\prime }{u}_k^{\prime }$, where, with the pivotal coalescence on edge species tree edge k, ${\ell }_k^{\prime }$, $r_k^{\prime }$, and $u_k^{\prime }$ count partial coalescent histories for $\left(G_{*\ell }^{\prime },S_k\right)$, $\left(G_{*r}^{\prime },S_k\right)$ and $\left(G^{\prime *},S\right)$, respectively.

To demonstrate that h′ (n, p) < h(n, p), we argue that ${\ell }_k^{\prime }\le {\ell }_k$, $r_k^{\prime }\le r_k$ and $u_k^{\prime }\le u_k$, and that for G′≠ G, at least one of these inequalities is strict. Following the argument of Corollary 11 of Himwich & Rosenberg [9], the quantities ${\ell }_k^{\prime }$, $r_k^{\prime }$, and $u_k^{\prime }$ count monotonic paths that lie below or on the y = x line, that respectively proceed from (0, 0) to (k, p − 3), (0, 0) to (k − p + 2, 1), and (0, 0) to (n − p, n − k − 1), possibly with roadblocks

When G′ = G, no roadblocks occur, so that ${\ell }_k^{\prime }={\ell }_k$, $r_k^{\prime }=r_k$ and $u_k^{\prime }=u_k$. When G′ ≠ G, however, at least one of the following three statements holds: (i) $G_{*\ell }^{\prime }\ne G_{*\ell }$; (ii) $G_{*r}^{\prime }\ne G_{*r}$ (iii) $G^{\prime *}\ne G^{*}$. In the first case, for at least one k, a roadblock occurs in tabulating coalescent histories for G_∗ℓ, so that ${\ell }_k^{\prime }={\ell }_k$ Similarly, in the second case, for at least one k, a roadblock occurs in tabulating coalescent histories for G_∗r, so that $r_k^{\prime }\le r_k$; in the third case, for at least one k, a roadblock occurs in tabulating coalescent histories for G^∗, so that $u_k^{\prime }\le u_k$. □

Note that for the sum describing the number of coalescent histories of (G, S) to even proceed over the full range from k = p − 1 to n − 1, the first p labels of G from left to right when G is written in canonical form must be a permutation of A₁, A₂, …, A_p. Otherwise, at least one label of G must be indexed by a value that exceeds p and therefore cannot descend from edge p − 1 of S.

5. Small p

The case of identically-labeled G and S produces the largest number of coalescent histories among all p-pseudocaterpillar gene trees and caterpillar species trees with fixed (n, p) and 4 ≤ p ≤ n. Note that for p = 3, the case of identically-labeled G and S produces more coalescent histories than any non-identically-labeled pair; both G and S are caterpillars in this case, and fixing n, the number of coalescent histories for matching caterpillars exceeds that for any non-matching pair of caterpillars (Remark 15 of [2]; Corollary 11 of [9]).

We now return to the case of identically-labeled (G, S) and evaluate eq. (8) for fixed small p.

5.1. Exact formulas for fixed p

Fixing the variable p allows us to obtain exact formulas for h(n, p) as a rational function of n. The smallest case is p = 3, so that the function h(n, 3) is defined for all n ≥ 3:

$$h\left(n,3\right)=\sum _{k=2}^{n-1}C\left(k,0\right)\left(k-1\right)C\left(n-3,n-k-1\right).$$

We rewrite the summand for h in the expanded form from eq. (9). We then obtain the sum using the Wilf-Zeilberger algorithm for computing sums that involve binomial coefficients.

Proposition 6.For all n ≥ 3, the following identity holds

$$h\left(n,3\right)=\sum _{k=2}^{n-1}\frac{{\left(k-1\right)}^2\left(2n-k-4\right)!}{\left(n-2\right)!\left(n-k-1\right)!}=\frac{3\left(2n-4\right)!}{n!\left(n-3\right)!}.$$ (10)

Proof. Set m = n − 1. Let the function F (m, k) be the ratio of the summand to the right-hand side of eq. (10):

$$F\left(m,k\right)=\frac{{\left(k-1\right)}^{2}m\left(m+1\right)\left(m-2\right)!\left(2m-k-2\right)!}{6\left(m-1\right)\left(2m-3\right)!\left(m-k\right)!}.$$

This function and a proof certificate

$$R\left(m,k\right)=-\frac{\left(k-2\right)\left(2m-k-1\right)\left(k^{2}m-k^2+k-2m\right)}{2m^2\left(2m-1\right)\left(k-1\right)\left(m-k+1\right)}$$

satisfy the assumptions of the Wilf-Zeilberger theorem [11, Theorem 7.1.1]. Hence, the sum ${\sum }_{k=2}^{m}F\left(m,k\right)$ does not depend on m. We know that ${\sum }_{k=2}^{m}F\left(m,k\right)=1$ when m = 2, from which eq. (10) follows by substituting n = m + 1. □

It is convenient to write eq. (10) as a product of a rational function of n and a Catalan number,

$$h\left(n,3\right)=\frac{3\left(n-2\right)}{2\left(2n-3\right)}{\mathcal{C}}_{n-1}.$$

For other small values of p, we follow the proof in Proposition 6 to obtain analogous expressions (Table 1). The corresponding proof certificates R(m, k) appear in Appendix A.

Table 1:

Closed-form expressions for the function h(n, p) for fixed values of p (eq. (8)). Wilf-Zeilberger proof certificates appear in Appendix A. The next three terms for ${\text{lim}}_{n\to \bowtie }[h(n,p)/{\mathcal{C}}_{n-1}]$ are 179587/65536 for p = 10, 384199/131072 for p = 11, and 1631605/524288 for p = 12.

p	h(n, p)	${\lim }_{n\to \bowtie }\frac{h\left(n,p\right)}{{\mathcal{C}}_{n-1}}$
3	$\frac{3\left(n-2\right)}{2\left(2n-3\right)}{\mathcal{C}}_{n-1}$	$\frac{3}{4}$
4	$\frac{\left(19n-40\right)\left(n-3\right)}{4\left(2n-3\right)\left(2n-5\right)}{\mathcal{C}}_{n-1}$	$\frac{19}{16}$
5	$\frac{\left(49n^2-254n+315\right)\left(n-4\right)}{4\left(2n-3\right)\left(2n-5\right)\left(2n-7\right)}{\mathcal{C}}_{n-1}$	$\frac{49}{32}$
6	$\frac{\left(467n^3-4319n^2+12798n-12096\right)\left(n-5\right)}{16\left(2n-3\right)\left(2n-5\right)\left(2n-7\right)\left(2n-9\right)}{\mathcal{C}}_{n-1}$	$\frac{467}{256}$
7	$\frac{\left(1067n^4-15263n^3+78997n^2-174673n+138600\right)\left(n-6\right)}{16\left(2n-3\right)\left(2n-5\right)\left(2n-7\right)\left(2n-9\right)\left(2n-11\right)}{\mathcal{C}}_{n-1}$	$\frac{1067}{512}$
8	$\frac{\left(4751n^5-96706n^4+762163n^3-2898044n^2+5296836n-3706560\right)\left(n-7\right)}{32\left(2n-3\right)\left(2n-5\right)\left(2n-7\right)\left(2n-9\right)\left(2n-11\right)\left(2n-13\right)}{\mathcal{C}}_{n-1}$	$\frac{4751}{2048}$
9	$\frac{\left(10393n^6-284776n^5+3155822n^4-18055844n^3+56078685n^2-89321220n+56756700\right)\left(n-8\right)}{32\left(2n-3\right)\left(2n-5\right)\left(2n-7\right)\left(2n-9\right)\left(2n-11\right)\left(2n-13\right)\left(2n-15\right)}{\mathcal{C}}_{n-1}$	$\frac{10393}{4096}$

5.2. Asymptotic behavior for small p

We can extend beyond the exact formulas for h(n, p) for small p in Section 5.1 to show that for each fixed p, there exists a constant β_p such that ${\text{lim}}_{n\to \bowtie }h(n,p)\sim {\beta }_p{\mathcal{C}}_{n-1}$. The approach follows Disanto & Rosenberg [6], who considered matching gene trees and species trees in caterpillar-like families, in which trees had a caterpillar shape with the caterpillar subtree of size ℓ replaced by a “seed tree” t of size ℓ. They assumed G = S = t⁽ⁿ⁾, with t⁽ⁿ⁾ consisting of t augmented by n “caterpillar branches” appended to its root.

The framework makes use of additional definitions. An r-extended coalescent history is a coalescent history for the case in which a species tree is assumed to have its root-branch divided into r ≥ 1 components [13]. Labeling these components from 1 to r with branch 1 closest to the species tree root, an m-rooted coalescent history is an r-extended coalescent history in which the gene tree root coalesces on species tree branch m, 1 ≤ m ≤ r. The number of m-rooted coalescent histories h_n,m for G = S = t⁽ⁿ⁾ then equals h_n,m = e_n,m − e_n,m−1, with e_n,0 = 0, where e_n,m is its number of m-extended coalescent histories.

Disanto & Rosenberg [6] devised an iterative procedure for obtaining the coalescent histories for t⁽ⁿ⁺¹⁾ from the coalescent histories for t⁽ⁿ⁾, n ≥ 0. For a fixed seed tree t, the generating function for the sequence h_0,m(t) counting m-rooted coalescent histories for t is written

$$g\left(y\right)=\sum _{m=1}^{\bowtie }{h}_{0,m}\left(t\right)y^m.$$

The bivariate generating function for the sequence h_n,m(t), counting m-rooted histories for $\left(G,S\right)=\left(t^{(n)},t^{(n)}\right)$, is denoted

$$F\left(y,z\right)=\sum _{m=1}^{\bowtie }\sum _{n=0}^{\bowtie }{h}_{n,m}\left(t\right)z^n{y}^m.$$

The univariate generating function f (z) for the sequence h_n,1(t), counting coalescent histories for $\left(G,S\right)=\left(t^{(n)},t^{(n)}\right)$, satisfies

$$f\left(z\right)=\sum _{n=0}^{\bowtie }{h}_{n,1}\left(t\right)z^n=\frac{\partial F\left(0,z\right)}{\partial y}.$$

Disanto & Rosenberg [6] obtained the result

$$f\left(z\right)=\frac{g\left(\frac{1-\sqrt{1-4z}}{2}\right)}{z}.$$ (11)

By examining the expansion of f (z) around its dominant singularity, they showed that given t, there exists a positive constant β_t such that $h_{n,1}\left(t\right)~{\beta }_t{\mathcal{C}}_{n-1}$.

The construction of Disanto & Rosenberg [6] that enumerated coalescent histories of t⁽ⁿ⁺¹⁾ from those of t⁽ⁿ⁾ does not use G = S. Thus, it applies for identically-labeled caterpillar-like families generated from nonmatching seed trees t_G and t_S of the same size, as does the associated procedure for obtaining the generating function f (z). In particular, it applies for the number of coalescent histories for caterpillar-like families with p-pseudocaterpillar G and identically-labeled caterpillar S.

We first derive an expression for e_(n,p),r, the number of r-extended coalescent histories for the p-pseudocaterpillar gene tree of n ≥ p leaves on an identically-labeled caterpillar species tree of n leaves. In our notation, dividing the root-branch amounts to adding right-steps on the diagram for u_k (Section 4.2) and increasing the range of the index k. We have

$$e_{\left(n,p\right),r}=\sum _{k=p-1}^{n+r-2}C\left(k,p-3\right)\left(k-p+2\right)C_t\left(n-p,n+r-k-2,r\right).$$ (12)

To obtain this expression, note that the extension of the species tree from 1 to r branches ancestral to the root does not affect the C(k, p − 3) and k − p + 2 terms, representing coalescences descended from the pivotal coalescence. However, the Catalan trapezoid that tabulates coalescent histories ancestral to the pivotal coalescence is affected. The number of coalescences ancestral to the pivotal coalescence continues to be n − p. The number of available branches is now n + r − k − 2 instead of n − k − 1. Traversing the paths “forward,” the trapezoid has order k − p + 2, one more than the number of coalescences that can occur on the species tree branch on which the pivotal coalescence takes place, giving a count of C_t(n + r − k − 2, n − p, k − p + 2) (eq. (3)). The number of vertices on the upper edge of the trapzeoid is r, so that if paths are traversed in reverse order, the number of r-extended coalescent histories is, equivalently, C_t(n − p, n + r − k − 2, r).

The associated number of m-rooted coalescent histories, 1 ≤ m ≤ r, then satisfies

$$h_{\left(n,p\right),m}=e_{\left(n,p\right),m}-e_{\left(n,p\right),m-1}.$$ (13)

Noting that n = p for the seed tree for the p-pseudocaterpillar family and applying eqs. (12) and (13) gives generating function g_p(y),

$$g_p\left(y\right)=\sum _{m=1}^{\bowtie }{h}_{\left(p,p\right),m}{y}^m=\sum _{m=1}^{\bowtie }\frac{m\left(m+2\right)\left(2p+m-5\right)!}{\left(p-3\right)!\left(p+m-1\right)!}{y}^m.$$ (14)

For small p, the generating functions g_p can be simplified as in Table 2.

Table 2:

Generating functions. Generating function g_p(y) counts m-rooted histories h(p,p),m for caterpillar-like families with a seed p-pseudocaterpillar gene tree and identically-labeled caterpillar species tree (eq. (14)); generating function f_p(z) counts coalescent histories h_(p+k,p),1 (eq. (15)).

p	gp(y)	fp(z)
3	$\frac{y}{{\left(y-1\right)}^2}$	$\frac{2\left(-\sqrt{1-4z}+1\right)}{z{\left(\sqrt{1-4z}+1\right)}^2}$
4	$\frac{y^2-3y}{{\left(y-1\right)}^3}$	$\frac{8\left(z-\sqrt{1-4z}+1\right)}{z{\left(\sqrt{1-4z}+1\right)}^3}$
5	$\frac{2y^3-8y^2+9y}{{\left(y-1\right)}^4}$	$\frac{8\left(\sqrt{1-4z}-1\right)\left(2z-3\sqrt{1-4z}-6\right)}{z{\left(\sqrt{1-4z}+1\right)}^4}$
6	$\frac{5y^4-25y^3+44y^2-28y}{{\left(y-1\right)}^5}$	$\frac{16\left(-10z^2+33z+15z\sqrt{1-4z}-4\sqrt{1-4z}+4\right)}{z{\left(\sqrt{1-4z}+1\right)}^5}$

From g_p(y), we then obtain the generating function f_p(z) that counts coalescent histories as the numbers of leaves in the gene tree and species tree increase from p:

$$f_p\left(z\right)=\sum _{k=0}^{\bowtie }{h}_{\left(p+k,p\right),1}{z}^k=\frac{g_p\left(\frac{1-\sqrt{1-4z}}{2}\right)}{z}.$$ (15)

Using eq. (14), these generating functions can also be simplified for small p (Table 2).

Expanding the entries in Table 2, we obtain, for example:

$$f_3\left(z\right)=1+3z+9z^2+28z^3+90z^4+297z^5+1001z^6+3432z^7+O\left(z^8\right)$$

$$f_4\left(z\right)=3+11z+37z^2+124z^3+420z^4+1441z^5+5005z^6+17576z^7+O\left(z^8\right).$$

Each function gives the values h(n, p) (eq. (8)) as n is incremented beginning with n = p.

For p-pseudocaterpillar gene trees, we can compare the values for the limiting constants β_p for two choices of the species tree S: the case in which the species tree has the same p-pseudocaterpillar labeled topology, and the case of an identically-labeled caterpillar species tree. The former value, from Disanto & Rosenberg [6], exceeds the latter for p = 3 and p = 4 (Table 3). For p = 5 to p = 9, however, β_p for the non-matching caterpillar S exceeds that for a matching p-pseudocaterpillar.

Table 3:

Numerical values of the constant β_p describing ${\text{lim}}_{n\to \bowtie }[h(n,p)/{\mathcal{C}}_{n-1}]$, the asymptotic ratio of the number of coalescent histories h(n, p) to the Catalan number ${\mathcal{C}}_{n-1}$. The gene tree has a p-pseudocaterpillar topology. Values for the case that the gene tree and species tree S have a matching p-pseudocaterpillar topology are taken from Table 1 of Disanto & Rosenberg [6]; values for identically-labeled caterpillar S are taken from Table 1.

	β p
p	Matching p-pseudocaterpillar S	Caterpillar S
3	1.0000	0.7500
4	1.2500	1.1875
5	1.4375	1.5313
6	1.5938	1.8242
7	1.7305	2.0840
8	1.8535	2.3198
9	1.9663	2.5374

Rosenberg & Degnan [16] had shown that the case of a 4-pseudocaterpillar gene tree and an identically-labeled caterpillar species tree produced more coalescent histories (β_p = 1.1875) than the case of matching caterpillar gene tree and species tree (${\mathcal{C}}_{n-1}$ coalescent histories, and hence a limiting ratio of 1). The table demonstrates that p-pseudocaterpillar gene trees for each p from 5 to 9 also produce more coalescent histories than the matching caterpillar gene tree.

6. Maximal number of coalescent histories for fixed n

Applying Theorem 4, we can calculate h(n, p) systematically for small n and all p with 3 ≤ p ≤ n. Table 4 shows the values of h(n, p) for all (n, p) with n ≤ 12.

Table 4:

Values of the function h(n, p) (eq. (8)) for small values of n and p.

	p
n	3	4	5	6	7	8	9	10	11	12
3	1
4	3	3
5	9	11	9
6	28	37	37	28
7	90	124	134	124	90
8	297	420	473	473	420	297
9	1001	1441	1665	1735	1665	1441	1001
10	3432	5005	5885	6291	6291	5885	5005	3432
11	11934	17576	20930	22766	23354	22766	20930	17576	11934
12	41990	62322	74932	82537	86149	86149	82537	74932	62322	41990

The table suggests two patterns. First, we can see that a symmetry exists in which h(n, p) = h(n, n − p + 3). We will verify this symmetry in Section 7. Second, we observe that for each n, the value of p that maximizes h(n, p) lies in the middle, repeating for two adjacent values of p when n is even. We state this result formally in the following theorem.

Theorem 7.Consider a caterpillar species tree S with n ≥ 4 leaves. Among identically-labeled p-pseudocaterpillar gene trees G with n leaves and 3 ≤ p ≤ n, the value of p that maximizes the number of coalescent histories h(n, p) for (G, S) is

$$p_m=\frac{n+3}{2}$$ (16)

if n is odd. If n is even, then two adjacent maxima exist:

$$p_{m1}=\frac{n+2}{2},p_{m2}=\frac{n+4}{2}.$$ (17)

For n = 3 and n = 4, the result is trivial, as n = 3 requires p = 3, and for n = 4, h(n, 3) = h(n, 4) = 3. For n ≥ 5, the proof proceeds in three steps.

1. First, in Section 6.1, for n ≥ 5 and 4 ≤ p ≤ n, we describe a difference function D(n, p) that measures the change in the function h(n, p) when we increment p by 1 for fixed n.

2. Next, in Section 6.2, we show that the difference function D(n, p) is positive for p = 4 (Lemma 8) and negative for p = n (Lemma 9), and that it monotonically decreases as the integer p is incremented from 4 to n (Lemma 10).

3. Finally, in Section 6.3, we deduce that for fixed n ≥ 5, if D(n, p) ≠ 0 for all p, 4 ≤ p ≤ n, then a unique integer p exists at which h(n, p) is maximal; two maxima exist if D(n, p) = 0 for some p. We confirm that the maxima of h(n, p) are described by eqs. (16) and (17).

6.1. Difference function

For n ≥ 5 and 4 ≤ p ≤ n, we define the difference function of h(n, p):

$$D\left(n,p\right)=h\left(n,p\right)-h\left(n,p-1\right).$$ (18)

Because h(n, p) is defined as a sum, D(n, p) is also an expression involving a summation.

We find a closed-form expression for D(n, p). We start by expanding eq. (18) using eq. (9):

$$\begin{gathered} D\left(n,p\right)=\sum _{k=p-1}^{n-1}[\frac{{\left(k-p+2\right)}^2\left(k-p+4\right)\left(k+p-3\right)!\left(2n-p-k-1\right)!}{\left(k+1\right)!\left(p-3\right)!\left(n-k-1\right)!\left(n-p+1\right)!} \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}-\frac{{\left(k-p+3\right)}^2\left(k-p+5\right)\left(k+p-4\right)!\left(2n-p-k\right)!}{\left(k+1\right)!\left(p-4\right)!\left(n-k-1\right)!\left(n-p+2\right)!}] \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}-\frac{3\left(2n-2p+2\right)!\left(2p-6\right)!}{\left(n-p+1\right)!\left(n-p+2\right)!\left(p-4\right)!\left(p-1\right)!}. \end{gathered}$$

Notice that because the sums in expressions for h(n, p) and h(n, p − 1) have different summation limits, we obtain an additional term outside the sum. The sum in the expression for D(n, p) can be transformed into a closed form, which gives the following formula:

$$D\left(n,p\right)=\frac{2p\left(np+7p-2p^2-6\right)\left(2n-2p+2\right)!\left(2p-5\right)!}{n\left(n-1\right)\left(n-p\right)!\left(n-p+2\right)!\left(p-3\right)!p!}-\frac{3\left(2n-2p+2\right)!\left(2p-6\right)!}{\left(n-p+1\right)!\left(n-p+2\right)!\left(p-4\right)!\left(p-1\right)!}.$$ (19)

The proof appears in Appendix B.

6.2. Sign of the difference function

Using eq. (19) for D(n, p), we prove three lemmas concerning the sign of D(n, p).

Lemma 8.For p = 4, the function D(n, p) is positive for all n ≥ 5.

Proof. When we substitute p = 4 into eq. (19), we obtain

$$D\left(n,4\right)=\frac{4\left(2n-5\right)!}{n!\left(n-4\right)!}-\frac{\left(2n-6\right)!}{\left(n-2\right)!\left(n-3\right)!}=\frac{2\left(7n-15\right)\left(2n-7\right)!}{n!\left(n-5\right)!}.$$

Because n ≥ 5, all terms in this fraction are positive. □

Lemma 9.For p = n, the function D(n, p) is negative for all n ≥ 5.

Proof. Substituting p = n into eq. (19), we obtain

$$D\left(n,n\right)=-\frac{2\left(n-6\right)\left(2n-5\right)!}{n!\left(n-3\right)!}-\frac{3\left(2n-6\right)!}{\left(n-4\right)!\left(n-1\right)!}.$$

For n ≥ 7, D(n, n) is quickly seen to be a sum of two negative numbers. It remains to check the cases of n = 5 and n = 6: D(5, 5) = −2 and D(6, 6) = −9. □

Lemma 10.D′(n, p) = D(n, p) − D(n, p − 1) is negative for n ≥ 5 and 5 ≤ p ≤ n. That is, for each n ≥ 5, D(n, p) monotonically decreases as the integer p is incremented from p = 4 to p = n.

Proof. The expression for D′(n, p) can be simplified to

$$D^{\prime }\left(n,p\right)=\frac{2\left(4p^2-4np-20p+11n+27\right)\left(2n-2p+2\right)!\left(2p-8\right)!}{\left(n-p+1\right)!\left(n-p+3\right)!\left(p-4\right)!\left(p-2\right)!}.$$

Because 5 ≤ p ≤ n, the term that determines the sign of D′(n, p) is the polynomial f (n, p) = 4p² − 4np − 20p + 11n + 27 in the numerator. Solving the inequality f (n, p) < 0 for p, we obtain

$$\frac{n+5}{2}-\frac{1}{2}\sqrt{n^2-n-2}<p<\frac{n+5}{2}+\frac{1}{2}\sqrt{n^2-n-2}.$$

The left-hand term is bounded above by 3 for all n ≥ 5, and the right-hand term exceeds n for all n ≥ 5. Hence, because 5 ≤ p ≤ n, all possible values of (n, p) satisfy the inequality. □

6.3. Location of the maximum

As a result of Lemmas 8–10, for n ≥ 5, as p is incremented from 4 to n, D(n, p) monotonically decreases (Lemma 10) from a positive value at p = 4 (Lemma 8) to a negative value at p = n (Lemma 9). Hence, h(n, p) increases from p = 3 to a maximum then decreases until p = n.

Two cases are possible. Given n, a unique value p = p_m1 could exist at which D(n, p) = 0, in which case h(n, p_m1) = h(n, p_m1 − 1), and both p_m1 and p_m2 = p_m1 − 1 are maxima. Alternatively, if D(n, p) ≠ 0 for all p, then h(n, p) is maximized at the largest value of p for which D(n, p) > 0.

If n ≥ 6 is even, then inserting $p=\frac{n+4}{2}$ into eq. (19), we obtain $D\left(n,\frac{n+4}{2}\right)=0$. Hence, $h\left(n,\frac{n+4}{2}\right)=h\left(n,\frac{n+2}{2}\right)$, and maxima of h(n, p) occur at both $p_{m1}=\frac{n+4}{2}$ and $p_{m2}=\frac{n+2}{2}$.

If n ≥ 5 is odd, we show that a value p_m ≥ 4 exists for which D(n, p_m) > 0 and D(n, p_m + 1) < 0. This value p_m maximizes D(n, p).

Lemma 11.For odd n ≥ 5, with n = 2k + 1 and k ≥ 2, (i) D(n, k + 2) > 0, and (ii) D(n, k + 3) < 0.

Proof. (i) We insert (n, p) = (2k + 1, k + 2) into eq. (19), obtaining the positive quantity

$$D\left(2k+1,k+2\right)=\frac{\left(2k\right)!\left(2k-2\right)!}{\left(k-1\right)!{\left(k!\right)}^2\left(k+1\right)!}.$$

(ii) Inserting (n, p) = (2k + 1, k + 3) into eq. (19), we obtain

$$D\left(2k+1,k+3\right)=-\frac{2\left(2k\right)!\left(2k-3\right)!}{\left(k-2\right)!{\left(k!\right)}^2\left(k+1\right)!},$$

a quantity that is negative. □

We conclude h(2k + 1, k + 2) > h(2k + 1, k + 1), but h(2k + 1, k + 3) < h(2k + 1, k + 2). Hence, for odd n ≥ 5, writing $k=\frac{n-1}{2}$, $p_m=\frac{n+3}{2}$ maximizes h(n, p). The proof of Theorem 7 is complete.

6.4. Asymptotic growth of the maximal number of coalescent histories

With the value p_m that maximizes h(n, p) established, we now examine the asymptotic growth of the maximum. We quickly verify that for a fixed caterpillar species tree with n leaves, across all p-pseudocaterpillar gene trees with fixed n, the maximal number of coalescent histories exceeds the Catalan number ${\mathcal{C}}_{n-1}$ describing the number of coalescent histories for the matching caterpillar. In Section 5.2, we showed that for fixed small p ≥ 4, as n increases, the number of coalescent histories grows with a constant multiple of ${\mathcal{C}}_{n-1}$, with the constant exceeding 1. Here we show that for each n ≥ 7, the maximal number of coalescent histories, that is, h(n, p_m) for odd n and h(n, p_m1) = h(n, p_m2) for even n, exceeds the corresponding Catalan number. We abbreviate p_m = p_m1 = p_m2 for even n, so that the sequence of values of h(n, p_m) is well-defined for n ≥ 4.

Proposition 12.For odd n ≥ 7, $h(n,p_m)>{\mathcal{C}}_{n-1}$, and for even n ≥ 8, $h(n,p_m)=h(n,p_{m1})=h(n,p_{m2})>{\mathcal{C}}_{n-1}$, where h is defined by eq. (8), p_m by eq. (16), p_m1 and p_m2 by eq. (17), and ${\mathcal{C}}_{n-1}$ by eq. (1).

Proof. For n = 7, we have h(n, p_m) = h(7, 5) = 134, which exceeds C₆ = 132. For n = 8, h(n, p_m1) = h(n, p_m2) = h(8, 5) = h(8, 6) = 473, which exceeds C₇ = 429.

Lemma 4.2 of Rosenberg & Degnan [16] showed that for n ≥ 9, $h(n,4)>{\mathcal{C}}_{n-1}$. By Theorem 7, for odd n ≥ 9, h(n, p_m) > h(n, 4), and for even n ≥ 10, h(n, p_m1) = h(n, p_m2) > h(n, 4). Thus, because the maximal number of coalescent histories across all p exceeds the number for p = 4, and because the number of coalescent histories for p = 4 exceeds ${\mathcal{C}}_{n-1}$, the maximum exceeds ${\mathcal{C}}_{n-1}$. □

We introduce the definition of the exponential order of a sequence: a sequence {a_n} has exponential order k if $\lim {\sup }_{n\to \bowtie }\sqrt[n]{a_n}=k$ [7]. In other words, a_n = kⁿs(n) where s(n) is a subexponential factor with $\lim {\sup }_{n\to \bowtie }\sqrt[n]{s\left(n\right)}=1$. If sequences a_n and b_n have the same exponential order, we write $a_n\bowtie b_n$.

The Catalan numbers ${\mathcal{C}}_n$ have exponential order 4, as Stirling’s approximation to ${\mathcal{C}}_n=(2n)!/[(n+1){(n!)}^2]$ gives ${\mathcal{C}}_n\sim 4^n/\left(n^{3/2}\sqrt{\pi }\right)$. Plotting log h(n, p_m) and $\text{log}\hspace{0.17em}{\mathcal{C}}_{n-1}$ as functions of n, we see that they grow approximately linearly with similar slopes (Figure 6). We therefore claim that the sequence h(n, p_m) also has exponential order 4.

Figure 6:

For caterpillar species trees of size n, the natural logarithms of the maximal number of coalescent histories across p-pseudocaterpillar gene trees (h(n, p_m)) and the number of coalescent histories of the matching caterpillar gene tree topology (${\mathcal{C}}_{n-1}$). The quantity h(n, p_m) is computed according to Theorem 7, and ${\mathcal{C}}_{n-1}$ follows eq. (1).

Proposition 13.For caterpillar species trees with n leaves, the sequence h(n, p_m) describing the maximal number of coalescent histories across all p-pseudocaterpillar gene trees of size n has exponential order 4, so that$h\left(n,p_m\right)\bowtie {\mathcal{C}}_n$.

Proof. By Lemmas 17 and 19 in Appendix C, ${\mathcal{C}}_{n-2}\le h(n,p_m)\le n{\mathcal{C}}_{n+2}$ for n ≥ 3, from which

$$\underset{x\to \bowtie }{\lim }\sqrt[n]{{\mathcal{C}}_{n-2}}\le \underset{x\to \bowtie }{\lim }\sqrt[n]{h\left(n,p_m\right)}\le \underset{x\to \bowtie }{\lim }\sqrt[n]{n{\mathcal{C}}_{n+2}}.$$

As the left-hand and right-hand limits both equal 4, we conclude ${\lim }_{n\to \bowtie }\sqrt[n]{h\left(n,p_m\right)}=4$, $h\left(n,p_m\right)=4^{n}s\left(n\right)$ for some subexponential s(n), and $h\left(n,p_m\right)\bowtie {\mathcal{C}}_n$. □

7. Symmetry

We now verify the symmetry h(n, p) = h(n, n − p + 3) observed in Table 4 for all (n, p) with 3 ≤ p ≤ n. For convenience, given a pseudocaterpillar tree with second cherry at position p, we define its dual as the pseudocaterpillar tree with second cherry at position n − p + 3 (Figure 7).

Figure 7:

Dual p-pseudocaterpillar gene trees with the same number of coalescent histories (1665) if paired with the identically-labeled caterpillar species tree. (A) (n, p) = (9, 5). (B) (n, p) = (9, 7).

We show that for a fixed caterpillar species tree, the number of coalescent histories of an identically-labeled pseudocaterpillar is equal to the number of coalescent histories of its dual. Thus, the formula for the number of coalescent histories has a symmetry in the position of the second cherry on the p-pseudocaterpillar gene tree.

Theorem 14.For all (n, p) with 3 ≤ p ≤ n, h(n, p) = h(n, n − p + 3).

For n = 3, the claim is trivial, as p = n − p + 3 = 3. For n = 4, the claim is also trivial, as h(4, 3) = h(4, 4) = 3. For n ≥ 5, we proceed in three steps.

1. First, in Section 7.1, we introduce a dual difference function D^∗(n, p) that measures the change in h(n, n − p + 3) as p is incremented for fixed n.

2. Next, in Section 7.2, we show that the dual difference function D^∗(n, p) is equal to the difference function D(n, p) for all allowed values of (n, p).

3. Finally, in Section 7.3, we use this equality of difference functions to complete the proof of the symmetry of h(n, p).

7.1. Dual difference function

We define a function D^∗ “dual” to the difference function D (eq. (18)):

$$D^{\ast }\left(n,p\right)=h\left(n,n-p+3\right)-h\left(n,n-p+4\right).$$ (20)

The function is well defined for n ≥ 5 and 4 ≤ p ≤ n, where n − p + 3 and n − p + 4 lie in [3, n].

Using the definition of h(n, p) from eq. (9), we obtain

$$\begin{array}{l}{D}^{*}\left(n,p\right)=\sum _{k=n-p+3}^{n-1}[\frac{{\left(k-n+p-1\right)}^2\left(k-n+p+1\right)\left(k+n-p\right)!\left(n+p-k-4\right)!}{\left(k+1\right)!\left(p-2\right)!\left(n-k-1\right)!\left(n-p\right)!}\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}-\frac{{\left(k-n+p-2\right)}^2\left(k-n+p\right)\left(k+n-p+1\right)!\left(n+p-k-5\right)!}{\left(k+1\right)!\left(p-3\right)!\left(n-k-1\right)!\left(n-p+1\right)!}]+\frac{3\left(2p-6\right)!\left(2n-2p+2\right)!}{\left(p-3\right)!\left(p-2\right)!\left(n-p\right)!\left(n-p+3\right)!}.\end{array}$$

This sum can be simplified to get a closed form for D^∗:

$$D^{*}\left(n,p\right)=\frac{4\left(p-3\right)\left(n^2+2p^2-3np+5n-9p+10\right)\left(2p-7\right)!\left(2n-2p+3\right)!}{n\left(n-1\right)\left(n-p+1\right)\left(p-4\right)!\left(p-2\right)!\left(n-p\right)!\left(n-p+3\right)!}+\frac{3\left(2p-6\right)!\left(2n-2p+2\right)!}{\left(p-3\right)!\left(p-2\right)!\left(n-p\right)!\left(n-p+3\right)!}.$$ (21)

The proof appears in Appendix D.

7.2. Dual difference function is equal to the difference function

This section verifies the equality of the difference function and its dual.

Lemma 15.For all n ≥ 5 and 4 ≤ p ≤ n, the dual difference function equals the difference function

$$D^{\ast }\left(n,p\right)=D\left(n,p\right).$$

Proof. We simplify D(n, p)/D^∗(n, p) using eqs. (19) and (21), verifying that this ratio equals 1. □

7.3. Completing the proof

Rearranging terms in the definitions of the difference functions by eqs. (18) and (20), we have

$$h\left(n,p\right)-h\left(n,n-p+3\right)=h\left(n,p-1\right)-h\left(n,n-p+4\right)$$ (22)

Decrementing p from n to 4, eq. (22) gives a chain of equalities $h\left(n,n\right)-h\left(n,3\right)=h\left(n,n-1\right)-h\left(n,4\right)=h\left(n,n-2\right)-h\left(n,5\right)=\dots =h\left(n,3\right)-h\left(n,n\right).$.

In particular, for each p from 3 to n, $h\left(n,p\right)-h\left(n,n-p+3\right)=-\left[h\left(n,p\right)-h\left(n,n-p+3\right)\right]$. Both sides of this equation must then equal zero, from which we conclude $h\left(n,p\right)=h\left(n-p+3\right)$ for each p, 3 ≤ p ≤ n. The proof of Theorem 14 is complete.

Theorem 14 can strengthen Proposition 12. We now know that $h(n,p)>{\mathcal{C}}_{n-1}$ for all (n, p) with n ≥ 9 and 4 ≤ p ≤ n − 1. From Rosenberg & Degnan [16], $h(n,4)>{\mathcal{C}}_{n-1}$ for n ≥ 9. By Theorem 14, h(n, n 1) = h(n, 4) > _n−1. In the proof of Theorem 7, we show that for n odd, h(n, p) increases as p increases from 4 to $\frac{n+3}{2}$, and Theorem 14 indicates that h(n, p) decreases as p increases from $\frac{n+3}{2}$ to n − 1; similarly, for n even, h(n, p) increases as p increases from 4 to $\frac{n+2}{2}$, with $h\left(n,\frac{n+2}{2}\right)=h\left(n,\frac{n+4}{2}\right)$, then decreases as p increases from $\frac{n+4}{2}$ to n − 1. Thus, $h(n,p)>{\mathcal{C}}_{n-1}$ for all (n, p) with n ≥ 9 and 4 ≤ p ≤ n − 1.

8. Discussion

We have developed a method for counting coalescent histories in cases in which the gene tree and species tree topologies do not match, considering p-pseudocaterpillar gene trees together with an identically-labeled caterpillar species tree. Using a combinatorial construction, we find that the recursive formula from Rosenberg [13] can be evaluated non-recursively as a sum (eq. (9))—which can in turn be simplified to a closed form for fixed small p (Section 5). The number of coalescent histories h(n, p) (eq. (8)) has a symmetry in p (Theorem 14), and the maximum over values of p for each n is attained when the “second cherry” lies in the “middle” of the gene tree (Theorem 7).

Results on the value of p that maximizes h(n, p) verify an informal observation from previous studies. It has been noted that for fixed n, large numbers of coalescent histories tend to occur when two conditions are met: the number of distinct sequences in which coalescences can be arranged is large, as is the number of species tree branches describing potential placements of those coalescences [5, 6, 13, 14, 16]. For a fixed caterpillar species tree, identically-labeled p-pseudocaterpillar gene trees represent a tradeoff of these two features. As p increases, more sequences exist for coalescences descended from the pivotal coalescence. However, the number of species tree branches on which the pivotal coalescence can occur decreases, so that fewer species tree branches exist on which the larger number of coalescence sequences can occur. That h(n, p) is maximized when p lies in the “middle” aligns with the informal observation that both conditions—many coalescence sequences, and many species tree branches on which coalescences take place—are important for generating large numbers of coalescent histories.

Table 5 compares coalescent histories in three cases: matching caterpillars, matching p-pseudocaterpillars, and caterpillar species trees with identically-labeled non-matching p-pseudocaterpillar gene trees. For a caterpillar species tree, as the number of species n grows to 9 or greater, the number of coalescent histories for identically-labeled p-pseudocaterpillar gene trees with 4 ≤ p ≤ n − 1 exceeds the Catalan number of coalescent histories for the matching gene tree (Proposition 12). For fixed p, more coalescent histories can occur for the non-matching p-pseudocaterpillar gene tree and identically-labeled caterpillar species tree than for matching p-pseudocaterpillars (Tables 3 and 5).

Table 5:

Numbers of coalescent histories for matching caterpillar gene trees and species trees (${\mathcal{C}}_{n-1}$, right-hand column), caterpillar species trees and identically-labeled p-pseudocaterpillar gene trees (top entry in each cell), and matching p-pseudocaterpillar gene trees and species trees (bottom entry). Top entries are from Table 4, and bottom entries are from Rosenberg [13].

n	p							Matching caterpillar
	3	4	5	6	7	8	9
3	1							2
	2
4	3	3						5
	5	4
5	9	11	9					14
	14	13	10
6	28	37	37	28				42
	42	42	37	28
7	90	124	134	124	90			132
	132	138	130	112	84
8	297	420	473	473	420	297		429
	429	462	453	416	354	264
9	1001	1441	1665	1735	1665	1441	1001	1430
	1430	1573	1584	1511	1368	1155	858

In related work, Disanto & Munarini [4] considered matching caterpillar gene trees and species trees, identifying the leaf whose replacement by a cherry in both trees would give rise to the greatest increase in the number of coalescent histories (measured as a ratio). This speciation—the splitting of a leaf node of G and S into two child nodes—can be interpreted as extending the trees by adding the “second cherry” that converts a caterpillar into a p-pseudocaterpillar. Disanto & Munarini [4] determined the value of p with which the p-pseudocaterpillar tree pair with n leaves would have the largest number of coalescent histories. Asymptotically, this value of p is equal to $\frac{n}{2}$ [4, i^∗(n) for D_n in Table 1]—“in the middle,” as in our result in Section 6. Theorem 7 can then be seen to prove an analogous result in a nonmatching case, as the gene tree gains a cherry node whereas the species tree gains only a caterpillar leaf.

Our p = 3 case has a direct geometric interpretation in the framework of Himwich & Rosenberg [9], as it describes a non-matching pair of caterpillar trees (Figure 8). Its coalescent histories are described by monotonic paths on a lattice with a single roadblock. The p = n case, for which the number of coalescent histories is equal to the p = 3 case, can also be represented in a diagram with one roadblock, obtained by reflecting monotonic paths of the p = 3 case across y = n−1− x. Because monotonic paths not traveling above the y = x diagonal of a square lattice correspond to Dyck paths, and the coalescent histories for p = 3 correspond to Dyck paths beginning with two up-steps, the sequence a_n = h(n, 3) = h(n, n) for n ≥ 3 gives the number of Dyck paths of length n − 1 beginning with two up-steps (OEIS A000245). We can also write $h(n,3)=h(n,n)=a(n)={\mathcal{C}}_{n-1}-{\mathcal{C}}_{n-2}$.

Figure 8:

Symmetry in the number of coalescent histories for a caterpillar species tree and identically-labeled p-pseudocaterpillar gene trees, for the cases of (n, 3) and (n, n). (A) For p = 3, coalescent histories correspond to roadblocked monotonic paths on a lattice with one roadblock. The first coalescence (B,C) on a gene tree cannot happen on the species tree edge ancestral to A and B. (B) The p = n case can also be seen to correspond to roadblocked monotonic paths. The second-to-last coalescence (E,F) can occur only on the edge ancestral to the species tree root.

Our work provides an extension of an earlier study of coalescent histories for non-matching caterpillars [9]. We expect that the method we have used has potential for extension to cases with more than two cherries, with the species tree remaining a caterpillar. In such an extension, each additional cherry would generate an additional “pivotal” coalescence and an additional summation based on the placement of that coalescence.

A. Wilf-Zeilberger certificates for formulas in Table 1

This appendix gives the proof certificates for the identities in Table 1, all of which have similar proofs to Proposition 6. Only the Wilf-Zeilberger proof certificate R(m, k) differs across the cases. We list the proof certificates for the remaining identities in Table 6.

Table 6:

Wilf-Zeilberger proof certificates R(m, k) for expressions in Table 1.

p	R(m, k)
4	−{(k − 3)(k − 2m + 2)[k3(19m2 − 59m + 42) + k2(−19m2 + 86m − 72) + k(−65m2 + 134m − 68) − 13m2 + m + 14]}/[2(k − 2)k(m − 1)2(2m − 3)(19m − 2)(k − m − 1)]
5	−{(k − 4)(k − 2m + 3)[k4(49m3 − 303m2 + 578m − 330) + k3(−49m3 + 388m2 − 858m + 540) + k2(−428m3 + 2483m2 − 4470m + 2394) − 2k(59m3 + 13m2 − 459m + 342) + 4(86m3 − 477m2 + 847m − 480)]}/[2(k − 3)(k − 1)(k + 2)(m − 2)2(2m − 5)(49m2 − 58m + 3)(k − m − 1)]
6	−{(k−5)(k−2m+4)[k5(467m4−4786m3+17233m2−25394m+12600)+7k4(133m3−867m2+ 1754m − 1080) + k3(−8870m4 + 90676m3 − 324682m2 + 473420m − 230304) + k2(−9780m4 + 80459m3 − 234273m2 + 285790m − 127176) + 3k(7531m4 − 84096m3 + 314435m2 − 463542m + 227352) + 18(1705m4 − 14774m3 + 47147m2 − 65566m + 32088)]}/[2(k − 4)(k − 2)(k + 2)(k + 3)(m − 3)2(2m − 7)(467m3 − 1517m2 + 1126m − 40)(k − m − 1)]
7	−{(k − 6)(k − 2m + 5)[k6(1067m5 − 16330m4 + 94585m3 − 256250m2 + 319728m − 143640) + k5(2134m5 − 30293m4 + 163673m3 − 415318m2 + 486024m − 204120) + k4(−34377m5 + 532064m4−3106039m3+8451124m2−10543892m+4705320)+k3(−104394m5+1517539m4−8380799m3+21722234m2−26044120m+11362200)+2k2(62807m5−1114568m4+7105518m3−20362003m2 + 25974226m − 11663880) + 4k(165353m5 − 2490002m4 + 14086017m3 − 37154572m2 + 45368164m − 20112720) + 288(2001m5 − 25450m4 + 128585m3 − 322100m2 + 388324m − 173040)]}/[2(k − 5)(k − 3)(k + 2)(k + 3)(k + 4)(m − 4)2(2m − 9)(1067m4 − 6727m3 + 13027m2 − 7697m + 210)(k − m − 1)]
8	−{(k − 7)(k − 2m + 6)[k7(4751m6 − 101457m5 + 860555m4 − 3681375m3 + 8289854m2 − 9182568m + 3825360) + k6(23755m6 − 495790m5 + 4118327m4 − 17279672m3 + 38197692m2 − 41536512m + 16964640) + k5(−206286m6 + 4477837m5 − 38491018m4 + 166405493m3 − 377604962m2 + 420063216m − 174878640) − 3k4(468054m6 − 9846179m5 + 82298858m4 − 346924603m3 + 769533814m2 − 839428224m + 344802960) + k3(−228007m6 + 517454m5 + 26651409m4 − 221650784m3 + 680141896m2 − 877046808m + 386542080) + k2(13435271m6 − 292844363m5 + 2510774503m4 − 10762507513m3 + 24114363462m2 − 26483935680m + 10933917840) + k(30799166m6 − 626694930m5 + 5077031510m4 − 20873258910m3 + 45641745164m2 − 49626013680m + 20475669600) + 600(37811m6 − 656541m5 + 4768367m4 − 18525795m3 +39673190m2 −43034952m+17843760)]}/[2(k −6)(k −4)(k +2)(k +3)(k +4)(k +5)(m−5)2(2m−11)(4751m5−49196m4+178555m3−265930m2+136104m−3024)(k−m−1)]
9	−{(k − 8)(k − 2m + 7)[k8(10393m7 − 295169m6 + 3444763m5 − 21287315m4 + 74661412m3 − 147149156m2 + 148827072m − 58378320) + k7(93537m7 − 2629468m6 + 30398601m5 − 186195970m4+647532468m3−1265548552m2+1268906064m−492972480)+k6(−463559m7+13467688m6 − 160270925m5 + 1007098102m4 − 3582455306m3 + 7142486032m2 − 7285730520m + 2869943328) + k5(−7040967m7 + 199285124m6 − 2316820929m5 + 14253708290m4 − 49732292718m3 + 97411690736m2 − 97808625096m + 38061051840) + k4(−14147654m7 + 380248249m6 − 4210420826m5 + 24757920421m4 − 82938771506m3 + 157002585316m2 − 153809636904m + 59074800624) + 2k3(39720207m7 − 1170499052m6 + 14047475388m5 − 88527503360m4 + 314151700473m3 − 621935009348m2 + 628262828172m − 245216321280) + 12k2(34497145m7 − 972446354m6 + 11227172479m5 − 68469189584m4 + 236767720390m3 − 460533277076m2 + 460767938016m − 179311977936) + 72k(9940368m7 − 263143441m6 + 2885073381m5 − 16942211875m4 + 57248478267m3 − 110112505504m2 + 109806816564m − 42781636800) + 8640(57016m7 − 1276772m6 + 12427702m5 − 68227355m4 + 223726804m3 − 426503693m2 + 425617338m − 166486320)]}/[2(k − 7)(k − 5)(k + 2)(k + 3)(k + 4)(k + 5)(k + 6)(m − 6)2(2m − 13)(10393m6 − 160060m5 + 931642m4 − 2537428m3 + 3195589m2 − 1476928m + 27720)(k − m − 1)]

B. Proof of the closed form for D(n, p) from eq. (19)

In this appendix, we prove the closed-form expression for the difference function D(n, p). In particular, we focus on the term that contains a summation over k.

Lemma 16.For all (n, p) with n ≥ 5 and 4 ≤ p ≤ n, the following identity holds:

$$\begin{gathered} F\left(n,p\right)=\sum _{k=p-1}^{n-1}[\frac{{\left(k-p+2\right)}^2\left(k-p+4\right)\left(k+p-3\right)!\left(2n-p-k-1\right)!}{\left(k+1\right)!\left(p-3\right)!\left(n-k-1\right)!\left(n-p+1\right)!} \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}-\frac{{\left(k-p+3\right)}^2\left(k-p+5\right)\left(k+p-4\right)!\left(2n-p-k\right)!}{\left(k+1\right)!\left(p-4\right)!\left(n-k-1\right)!\left(n-p+2\right)!}] \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=\frac{2p\left(np+7p-2p^2-6\right)\left(2n-2p+2\right)!\left(2p-5\right)!}{n\left(n-1\right)\left(n-p\right)!\left(n-p+2\right)!\left(p-3\right)!p!}. \end{gathered}$$

Proof. Let ∆_k denote the forward difference operator in k, meaning that ∆_k(f ) = f (k + 1) − f (k). Let f (n, p, k) be the summand in the expression for F (n, p). We sum the equation

$$f\left(n,p,k\right)={\Delta }_k\left(h_{n,p}\left(k\right)\right)$$ (23)

over k, from k = p − 1 to k = n − 1. The left-hand side of eq. (23) is the summand in the statement of the lemma, and the function h_n,p(k) is the output of Gosper’s algorithm [10, 11]:

$$\begin{gathered} h_{n,p}\left(k\right)=[\left(k+1\right)\left(2n-p-k\right)(-k^3{n}^2+2k^{3}n-k^3+3k^2{n}^{2}p-9k^2{n}^2-4k^{2}np+15k^{2}n-3k^{2}p \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+6k^2-3k{n}^2{p}^2+18k{n}^{2}p-29k{n}^2+2kn{p}^2-6knp+4km-3k{p}^2+12kp-11k \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+n^2{p}^3-9n^2{p}^2+25n^{2}p-21n^2+3n{p}^2-14np+15n-p^3+6p^2-11p+6) \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\left(k+p-4\right)!\left(2n-p-k-1\right)!]/ \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\left[\left(n-1\right)n\left(p-3\right)\left(k+1\right)!\left(n-p+2\right)\left(p-4\right)!\left(n-k-1\right)\left(n-p+1\right)!\right]. \end{gathered}$$ (24)

We verify eq. (23) by using eq. (24). After summation, the left-hand side becomes F (n, p). The right-hand side telescopes, so that all terms except the first and the last cancel:

$$F\left(n,p\right)=h_{n,p}\left(n\right)-h_{n,p}\left(p-1\right).$$ (25)

We obtain the statement of the lemma by algebraic simplification of the right-hand side of eq. (25). □

C. Proofs of inequalities required for the proof of Proposition 13

In the proof of Proposition 13, we make use of lower and upper bounds for h(n, p_m). First, we prove the lower bound.

Lemma 17.For all n ≥ 3, $h(n,p_m)\ge {\mathcal{C}}_{n-2}$.

Proof. From Theorem 7, h(n, p_m) ≥ h(n, 3). Comparing h(n, 3) from eq. (10) and ${\mathcal{C}}_{n-2}$ from eq. (1), we obtain

$$\frac{h\left(n,3\right)}{{\mathcal{C}}_{n-2}}=\frac{3\left(n-2\right)}{n}\ge 1$$

for all n ≥ 3. Hence, ${\mathcal{C}}_{n-2}\le h(n,3)\le h(n,p_m)$ as desired. □

To prove the upper bound, we first need an identity concerning Catalan numbers.

Lemma 18.For n ≥ 3, the Catalan number ${\mathcal{C}}_n$ can be decomposed as a sum.

1. For even n ≥ 4, with n = 2m + 2 for m ≥ 1,

   $${\mathcal{C}}_{2m+2}=\sum _{k=m+2}^{2m+2}C\left(k,m+1\right)C\left(m,2m+2-k\right).$$
2. For odd n ≥ 3, with n = 2m + 1 for m ≥ 1,

   $${\mathcal{C}}_{2m+1}=\sum _{k=m+1}^{2m+1}C\left(k,m\right)C\left(m,2m+1-k\right).$$

Proof. (i) We use two ways of counting monotonic paths. ${\mathcal{C}}_{2m+2}$ gives the number of monotonic paths that travel from (0, 0) to (2m + 2, 2m + 2) on a square lattice, without traveling above the diagonal connecting (0, 0) to (2m + 2, 2m + 2). Each of these paths passes through exactly one vertical edge from a point (k, m + 1) to a point (k, m + 2), where k ranges from m + 2 to 2m + 2.

The number of monotonic paths that travel from (0, 0) to (k, m + 1) and that do not travel above the diagonal is C(k, m + 1). The number of monotonic paths from (k, m + 2) to (2m + 2, 2m + 2) that do not travel above the diagonal is obtained by traversing the paths in reverse order, from (2m + 2, 2m + 2) down and to the left, reaching (k, m + 2) (Figure 5A). The associated number of paths is C(m, 2m+2− k). Hence, the total number of monotonic paths from (0, 0) to (2m+2, 2m+2) that do not travel above the diagonal is ${\sum }_{k=m+2}^{2m+2}C\left(k,m+1\right)C\left(m,2m+2-k\right)$.

(ii) The argument in the odd case proceeds in the same way. Each path from (0, 0) to (2m + 1, 2m + 1) passes through exactly one vertical edge from a point (k, m) to a point (k, m + 1), where k ranges from m + 1 to 2m + 1. The number of paths from (0, 0) to (k, m) is C(k, m), and the number of paths from (k, m + 1) to (2m + 1, 2m + 1) is C(m, 2m + 1 − k). □

We are now ready to prove the upper bound.

Lemma 19.For all n ≥ 3, $h(n,p_m)\le n{\mathcal{C}}_{n+2}$.

Proof. We split the proof into two cases, according to the expressions for p_m from Theorem 7.

First, assume n is even, with n = 2m for m ≥ 2. Then p_m = m + 1, and by Theorem 4,

$$h\left(2m,m+1\right)=\sum _{k=m}^{2m-1}\left(k-m+1\right)C\left(k,m-2\right)C\left(m-1,2m-k-1\right),$$

or, equivalently,

$$h\left(2m,m+1\right)=\sum _{k=m+2}^{2m+1}\left(k-m-1\right)C\left(k-2,m-2\right)C\left(m-1,2m-k+1\right).$$

Using the decomposition in Lemma 18,

$$\begin{gathered} 2m{\mathcal{C}}_{2m+2}-h\left(2m,m+1\right)=\sum _{k=m+2}^{2m+1}[2mC\left(k,m+1\right)C\left(m,2m-k+2\right) \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}-\left(k-m-1\right)C\left(k-2,m-2\right)C\left(m-1,2m-k+1\right)] \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+\frac{2m\left(m+2\right)\left(3m+3\right)!}{\left(m+1\right)!\left(2m+3\right)!}. \end{gathered}$$

The summand in the first term is nonnegative, as function C(n, k) is monotonically increasing with respect to both arguments, and 2m ≥ k − m − 1 because k ≤ 2m + 1. The remaining term is also nonnegative. Hence, for even n we indeed have $h(n,p_m)\le n{\mathcal{C}}_{n+2}$.

Now assume n is odd, with n = 2m − 1 and m ≥ 2. We then must show $(2m-1){\mathcal{C}}_{2m+1}\ge h(2m-1,m+1)$. By Theorem 4, we have

$$h\left(2m-1,m+1\right)=\sum _{k=m}^{2m-2}\left(k-m+1\right)C\left(k,m-2\right)C\left(m-2,2m-k-2\right),$$

or, equivalently,

$$h\left(2m-1,m+1\right)=\sum _{k=m+1}^{2m-1}\left(k-m\right)C\left(k-1,m-2\right)C\left(m-2,2m-k-1\right).$$

Using the decomposition in Lemma 18, we have

$$\begin{gathered} \left(2m-1\right){\mathcal{C}}_{2m+1}-h\left(2m-1,m+1\right)=\sum _{k=m+1}^{2m-1}[\left(2m-1\right)C\left(k,m\right)C\left(m,2m-k+1\right) \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}-\left(k-m\right)C\left(k-1,m-2\right)C\left(m-2,2m-k-1\right)] \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+\frac{\left(2m-1\right)\left(2m^3+7m^2+9m+2\right)\left(3m\right)!}{m!\left(2m+2\right)!}. \end{gathered}$$

As is true in the even case, the summand is termwise nonnegative, as is the remaining term. We conclude that for odd $n,\hspace{0.17em}h(n,\hspace{0.17em}{p}_m)\le n\hspace{0.17em}{\mathcal{C}}_{n+2}$, completing the proof. □

D. Proof of the closed form for D^∗(n, p) from eq. (21)

Here we prove the closed-form expression for the dual difference function in eq. (21) from Section 7.1.

Lemma 20.For all (n, p) with n ≥ 5 and 4 ≤ p ≤ n, the following identity holds:

$$\begin{gathered} F\left(n,p\right)=\sum _{k=n-p+3}^{n-1}[\frac{{\left(k-n+p-1\right)}^2\left(k-n+p+1\right)\left(k+n-p\right)!\left(n+p-k-4\right)!}{\left(k+1\right)!\left(p-2\right)!\left(n-k-1\right)!\left(n-p\right)!} \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}-\frac{{\left(k-n+p-2\right)}^2\left(k-n+p\right)\left(k+n-p+1\right)!\left(n+p-k-5\right)!}{\left(k+1\right)!\left(p-3\right)!\left(n-k-1\right)!\left(n-p+1\right)!}] \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=\frac{4\left(p-3\right)\left(n^2+2p^2-3np+5n-9p+10\right)\left(2p-7\right)!\left(2n-2p+3\right)!}{n\left(n-1\right)\left(n-p+1\right)\left(p-4\right)!\left(p-2\right)!\left(n-p\right)!\left(n-p+3\right)!}. \end{gathered}$$

Proof. As in Appendix B, let ∆_k denote the forward difference operator in k, meaning that ∆_k(f ) = f (k + 1) − f (k). Let f (n, p, k) be the summand in the expression for F (n, p).

We sum the equation

$$f\left(n,p,k\right)={\Delta }_k\left(h_{n,p}\left(k\right)\right)$$ (26)

over k, from k = n − p + 3 to k = n − 1. The left-hand side is the summand in the statement of the lemma, and the function h_n,p(k) is the output of Gosper’s algorithm [10, 11]:

$$\begin{gathered} h_{n,p}\left(k\right)=[\left(k+1\right)(k^3{n}^2-2k^{3}n+k^3-3k^2{n}^3+3k^2{n}^{2}p+k^2{n}^2-4k^{2}np+4k^{2}n-3k^{2}p \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+6k^2+3k{n}^4-6k{n}^{3}p+4k{n}^3+3k{n}^2{p}^2-2k{n}^{2}p-2k{n}^2-2kn{p}^2+4knp+3k{p}^2 \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}-12kp+11k-n^5+3n^{4}p-3n^4-3n^3{p}^2+6n^{3}p-3n^3+n^2{p}^3-3n^2{p}^2+4n^{2}p \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}-3n^2-2np+4n-p^3+6p^2-11p+6)\left(n+k-p\right)!\left(n+p-k-4\right)!]/ \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\left[n\left(n-1\right)\left(k+1\right)!\left(p-2\right)!\left(n-k-1\right)!\left(n-p+1\right)!\right]. \end{gathered}$$ (27)

With h as in eq. (27), eq. (26) is verified algebraically. After summation of eq. (26), the left-hand side becomes F (n, p), and the right-hand side telescopes. All terms except the first and the last cancel, leaving

$$F\left(n,p\right)=h_{n,p}\left(n\right)-h_{n,p}\left(n-p+3\right).$$ (28)

The lemma then follows by algebraic simplification of the right-hand side of eq. (28). □