Figure 3. The Production of Anomalies for n-Maximally Probable Species Tree Topologies with n = 5,6,7,8 (See Table 1).

The branch lengths x, y, and λ apply to each tree: in (a) and (b), x + y denotes the length of the red internal branch, and in (c) and (d), x and y are the lengths of the deeper and shallower red internal branches, respectively; the length λ denotes the branch length between the root of the species tree and the MRCA of species A and B. For each tree, the color of a branch represents the probability that coalescences occur on the branch. On an external branch, because there is only one gene lineage, coalescences cannot occur. Prior to the root, the probability is 1 that all lineages coalesce. During the time between the root of the species tree and the divergence of A and B—and of C and D in (b–d)—the probability that any coalescences occur can be made arbitrarily close to 0 by making the internal branches sufficiently short. Similarly, by choosing x and y to be sufficiently large, the probability that all available lineages coalesce on the red branches can be made arbitrarily close to 1. In (a), the species tree can be represented as (((AB)C)Z), where Z is (DE). By making the internal branch ancestral to D and E long, the subtree Z is similar to a single taxon, and the five-taxon tree behaves like the four-taxon asymmetric tree (((AB)C)Z), which produces the anomaly ((AB)(CZ)). Thus, in (a), the AGT is ((AB)(C(DE))). Similarly, the species tree topologies in (b), (c), and (d) have the form (((AB)(CD))Z) and produce anomalies (((AB)C)(DZ)); in (b), (c), and (d) Z is (EF), (E(FG)), and ((EF)(GH)), respectively. The anomalies occur by letting internal branches in subtrees ((AB)(CD)) and Z be sufficiently short and long, respectively.