The Evolution of Hybrid Incompatibilities along a Phylogeny

Abstract

The Dobzhansky-Muller model of speciation posits that defects in hybrids between species are the result of negative epistatic interactions between alleles that arose in independent genetic backgrounds. Tests of one important prediction from this model, that incompatibilities “snowball”, have relied on comparisons of the number of incompatibilities between closely related pairs of species separated by different divergence times. How incompatibilities accumulate along phylogenies, however, remains poorly understood. We extend the Dobzhansky-Muller model to multi-species clades to describe the mathematical relationship between tree topology and the number of shared incompatibilities among related pairs of species. We use these results to develop a statistical test that distinguishes between the snowball and alternative incompatibility accumulation models, including non-epistatic and multi-locus incompatibility models, in a phylogenetic context. We further demonstrate that patterns of incompatibility sharing across species pairs can be used to estimate the relative frequencies of different types of incompatibilities, including derived-derived vs. derived-ancestral incompatibilities. Our results and statistical methods should motivate comparative genetic mapping of hybrid incompatibilities to evaluate competing models of speciation.

INTRODUCTION

The genetics of reproductive isolation provides a powerful framework for understanding the mechanisms of speciation (Dobzhansky 1937). Though other types of reproductive isolation have been investigated, most studies of speciation from a genetic perspective have targeted hybrid inviability and hybrid sterility (intrinsic postzygotic isolation; Coyne and Orr 2004; Wolf et al. 2010; Butlin et al. 2012). Hybrid dysfunction has been attributed to a large number of genomic regions across a wide variety of species (True et al. 1996; Gadau et al. 1999; Presgraves 2003; Slotman et al. 2004; Good et al. 2008; Moyle and Nakazato 2008; White et al. 2011; Burkart-Waco et al. 2012). In a growing number of cases, specific genes have been identified (Sawamura and Yamamoto 1997; Ting et al. 1998; Barbash et al. 2003; Presgraves et al. 2003; Brideau et al. 2006; Masly et al. 2006; Bomblies et al. 2007; Lee et al. 2008; Bayes and Malik 2009; Bikard et al. 2009; Ferree and Barbash 2009; Mihola et al. 2009; Phadnis and Orr 2009; Tang and Presgraves 2009). This remarkable success motivates continued development of theoretical models that generate testable predictions about the genetics of speciation.

Chief among genetic models of postzygotic isolation is one proposed by Bateson (1909), Dobzhansky (1937) and Muller (1940, 1942). This “Dobzhansky-Muller model” supposes that hybrid sterility and inviability arise from negative epistatic interactions between alleles at different loci, dubbed Dobzhansky-Muller incompatibilities [DMIs] (Turelli and Orr 2000). Incompatibility alleles arise and fix in the distinct genetic backgrounds of incipient species where they do not decrease viability or fertility. When hybridization occurs, genotypic combinations that have not been tested by natural selection form and reduce fitness. By attributing hybrid inviability and hybrid sterility to epistatic interactions between loci, the model avoids forcing any lineage to pass through an unfit intermediate at a single locus. Under the Dobzhansky-Muller model, hybrid incompatibilities result from either substitutions along different lineages (derived-derived) or from substitutions along a single lineage (derived-ancestral).

While recent theoretical work on the accumulation of hybrid incompatibilities has focused on the Dobzhansky-Muller model, there are several alternatives that do not invoke epistasis. Theory suggests such models are unlikely to produce strong reproductive isolation because they require populations to temporarily transition through unfit intermediates (Gavrilets 2003). These models have nevertheless been proposed to explain empirical results. Reproductive isolation resulting from the fixation of mildly underdominant chromosomal rearrangements has been explored as one such non-epistatic model (Kirkpatrick and Barton 2006; Coyne and Orr 2004; Walsh 1982). Experimental work in the genus Saccharomyces has led to a model of reproductive isolation involving sequence divergence and mismatch repair without epistasis (Liti 2006; Greig et al. 2002). Since these non-epistatic models invoke the progressive accumulation of mutations at single loci, hybrid incompatibilities between divergent species should accumulate linearly with time (Gourbière and Mallet 2010). This prediction has rarely been tested empirically.

Catalyzed by Orr (1995), the Dobzhansky-Muller model became the focus of mathematical theory that generated many useful insights into incompatibility evolution (Orr 1996; Gavrilets 1997; Barton 2001; Turelli and Orr 2000; Orr and Turelli 2001; Turelli et al. 2001; Kondrashov et al. 2002; Gavrilets 2004; Mendelson et al. 2004; Welch 2004; Palmer and Feldman 2009; Unckless and Orr 2009; Fierst and Hansen 2010; Bank et al. 2012; Livingstone et al. 2012). One noteworthy prediction is the “snowball effect”: the expected number of hybrid incompatibilities increases faster than linearly with time since divergence. To test this prediction, incompatibilities are counted in at least two species pairs separated by different divergence times. This approach has recently uncovered evidence for the snowball effect in Drosophila (Matute et al. 2010) and some isolation traits in Solanum (Moyle and Nakazato 2010), although other isolation traits show patterns consistent with alternative modes of evolution (Moyle and Nakazato 2010).

The potential power of comparing incompatibilities among species pairs within a clade raises the prospect of embedding the Dobzhansky-Muller model within a phylogenetic comparative framework (Moyle and Payseur 2009). The temporal dynamics of Dobzhansky-Muller incompatibility (DMI) evolution can be reconstructed by scoring incompatibilities as shared or unshared among several pairs of species with known phylogenies (Cattani and Presgraves 2009). As a result, predictions from the Dobzhansky-Muller model can be tested in a phylogenetic context. Despite widespread acceptance of the Dobzhansky-Muller model as an explanation for the evolution of postzygotic isolation and empirical success in mapping incompatibilities across a range of species, the way in which incompatibilities accumulate along phylogenies remains largely unknown.

Theory describing how incompatibilities evolve along phylogenies would also enable estimation of the relative contributions of different incompatibility classes. For example, mathematical descriptions of the Dobzhansky-Muller model have assumed that derived-derived incompatibilities have the same probability of occurring as derived-ancestral incompatibilities (Orr 1995; Turelli and Orr 2000; Orr and Turelli 2001). This has led to the prediction that derived alleles are three times more likely to participate in incompatibilities than ancestral alleles (Orr, 1995). However, substitutions at a single locus may influence the rate of evolution at epistatically linked loci (Presgraves and Stephan 2007; Schlosser and Wagner 2008; Tang and Presgraves 2009). Co-evolution of incompatible loci would result in departures from the expected proportion of derived-derived versus derived-ancestral incompatibilities.

Here, we describe how the numbers of incompatibilities shared between species pairs depend on phylogenetic history. We use these theoretical results to develop statistical tests that use the number of shared and unique incompatibilities counted between species pairs to compare models of incompatibility evolution and to estimate their parameters. We show that this comparative genetic mapping framework is a powerful approach for understanding the evolution of reproductive isolation.

METHODS AND RESULTS

Model Preliminaries

Mathematical models of DMI accumulation predict the number of incompatibilities between two lineages as a function of divergence time (Orr 1995; Orr and Turelli 2001; Welch 2004). These models consider two populations slowly diverging in allopatry from a common ancestor whereupon they independently accumulate and fix new mutations. The mathematical treatment of the model involves a number of simplifications:

• New substitutions in the two populations are at independent sites and do not overlap.

• Substitutions appear in these populations according to a Poisson process and fix instantly.

• The mean number of substitutions separating two nodes is proportional to the branch length, or evolutionary time, separating them. The proportionality constant is denoted by k. This is also the rate of the Poisson process.

• Any pair of substitutions evolving in different lineages has a small probability, p, of interacting to cause a hybrid incompatibility.

As the number of divergent sites, K, increases, the possible number of epistatic interactions between n loci increases combinatorially as $\left(\begin{array}{c}\hfill K\hfill \\ \hfill n\hfill \end{array}\right)$ (Orr 1995).

When only epistatic interactions between two loci are considered, the number of incompatibilities, I, as a function of time since divergence has expected value and variance (Orr and Turelli 2001): $\mathrm{E}\left[I\right]=\frac{p\mathrm{E}{\left[K\right]}^2}{2}$ and $\mathrm{Var}\left[I\right]=\mathrm{E}\left[I\right](1+2p\mathrm{E}[K\left]\right)$, where E[K] equals 2kt, the expected number of substitutions separating two species at time t. Similar mathematical treatment can be used to find expected values and variances in the case of epistatic interactions between more than two loci. The expected number of DMIs for interactions between n loci (see Appendix for details) is

$$\mathrm{E}\left[I_n\right]=\mathrm{E}\left[p_n\left(\begin{array}{c}\hfill K\hfill \\ \hfill n\hfill \end{array}\right)\right]=\frac{p_n}{n!}\mathrm{E}{\left[K\right]}^n$$ (1)

where I_n is the number of n-locus incompatibilities and p_n is the probability that an n-locus interaction between substitutions results in a hybrid incompatibility. This relationship can also be employed to determine the variance in the number of n-locus DMIs,

$$\mathrm{Var}\left[I_n\right]=\frac{p_n(1-p_n)}{n!}\mathrm{E}{\left[K\right]}^n+{\left(\frac{p_n}{n!}\right)}^2\left(\sum _{k=1}^n{\left(\begin{array}{c}\hfill n\hfill \\ \hfill k\hfill \end{array}\right)}^{2}k!\mathrm{E}{\left[K\right]}^{2n-k}\right)$$ (2)

It is important to remember that these values are expectations and variances for incompatibilities involving exactly n loci only. If higher-order interactions are thought to contribute, a model for the total number of DMIs will need to quantify the contributions of different multi-locus interactions to the total number of incompatibilities (e.g., the relative frequencies of two- and three-locus incompatibilities).

Incompatibilities on a Three-Species Tree

Our goal is to understand predictions of the Dobzhansky-Muller model for the evolution of DMIs in a multi-species clade, including predictions about the number and phyletic distribution of incompatibilities among hybrids between different species pairs. We consider the simplest case with phylogenetic structure where three species have diverged from a recent common ancestor. Assumptions from the previous mathematical treatment (Orr 1995; Orr and Turelli 2001), including the gradual divergence of populations in allopatry and the instantaneous fixation of mutations, continue to apply.

On this simple tree (see Figure 1) we label the two most recently diverged sister taxa Species A and Species B; and we label the third taxon Species C. We call the time of the event where A and B diverge T_d and define t₂ to be the elapsed time from T_d to present. We define t₁ to be the elapsed time from the initial divergence at the root to T_d. We assume that the substitution rate per genome per unit time, k, is the same across every branch in the tree.

Figure 1.

A three-species tree on which hybrid incompatibilities are considered. T_d marks the time when Species A and B diverge, t₁ marks the time from the initial divergence of Species C to T_d, and t₂ marks the time from T_d to present. The number of substitutions from each branch, as referenced in the text, K_L, K_R, K_a, K_b, and K_c, are also labeled on their respective branches. Note that K_t = K_L + K_R.

Pairs of taxa from these three lineages yield three classes of hybrids: AB, AC, and BC. When considering each of these hybrid classes individually, the expected number of DMIs follows the behavior described by Turelli and Orr (2001). That is, each of the two taxa forming a hybrid can be considered separately as two populations that have diverged from a common ancestor. The total time since divergence of the taxa that form the AB hybrid is t₂ and the total time since divergence of the taxa that form the AC and BC hybrids is t₁ + t₂. The expected values and variances for the number of incompatibilities can be calculated as described in previous work (Turelli and Orr 2001; Table 1). However, the numbers of DMIs between different hybrid classes are not independent, but depend on the topology of the phylogeny. To understand covariation between the numbers of DMIs, we must first describe how incompatibilities are shared between different species pairs.

Table 1.

Expectation and variance values for the two-locus model; k is the substitution rate per genome per unit time, t₁ is the time from the root to T_d, t₂ is the time from T_d to the present, and p is the probability of an incompatibility.

	Expectation	Variance
IAB	2p(kt2)2	2p(kt2)2(1+4pkt2)
IAC	2p(kt1+kt2)2	2p(kt1+kt2)2(1+4p(kt1+kt2))
IBC	2p(kt1+kt2)2	2p(kt1+kt2)2(1+4p(kt1+kt2))
IsharedA	$\left(\frac{p}{2}\right){\left({\mathit{kt}}_2\right)}^2$	$\left(\frac{p}{2}\right){\left({\mathit{kt}}_2\right)}^2(1+2{\mathit{pkt}}_2)$
IsharedB	$\left(\frac{p}{2}\right){\left({\mathit{kt}}_2\right)}^2$	$\left(\frac{p}{2}\right){\left({\mathit{kt}}_2\right)}^2(1+2{\mathit{pkt}}_2)$
IsharedC	$\left(\frac{p}{2}\right){({\mathit{kt}}_2+2{\mathit{kt}}_1)}^2$	$\left(\frac{p}{2}\right){({\mathit{kt}}_2+2{\mathit{kt}}_1)}^2(1+2p(2{\mathit{kt}}_1+{\mathit{kt}}_2\left)\right)$

Shared Incompatibilities

We call an incompatibility shared between different hybrids if it involves the same interaction between the same set of loci in each of the hybrids. With three species, there are three possible pairwise comparisons: AB with AC, AB with BC, and AC with BC. We assume the same alleles that cause an incompatibility in one hybrid are sufficient to cause the same incompatibility in another hybrid. As a consequence of our modeling assumptions, we disregard incompatibilities that are the result of convergence since new incompatibilities must arise from independently occurring substitutions.

Thus, shared incompatibilities can only exist among species pairs if they share some of the same substitutions. This leads to a distinction in how derived-derived and derived-ancestral incompatibilities are shared between hybrids (Figure 2). Notice in our example that there is no way for a DMI to be shared among all three species pairs. Since derived-derived incompatibilities involve interactions between substitutions that have occurred in different lineages, they can only be shared between species pairs when there is shared evolutionary history among all parental species. In contrast, because derived-ancestral incompatibilities involve substitutions that have occurred on a single lineage, they can be shared if at least one parent from each pair share evolutionary history. Thus, sharing of derived-derived incompatibilities is more restrictive.

Figure 2.

Diagram of derived-derived (left) and derived-ancestral (right) interactions involving the last substitution in Species C. Open circles on the tree indicate where substitutions have occurred on a lineage. Closed circles on the tree indicate the corresponding ancestral allele on other lineages. Solid arrows represent interactions that would lead to shared incompatibilities between the AC and BC hybrid while dashed arrows represents interactions unique to a single hybrid.

Consider the shared incompatibilities between the AC and BC hybrid in our three-species example. Interactions between substitutions that arose before T_d, labeled K_t = K_L + K_R in Figure 1, can result in shared incompatibilities that are both derived-derived and derived-ancestral since all parental species of AC and BC share the same history before T_d. Substitutions that arose after T_d on Species C’s lineage, K_c, can participate in shared derived-ancestral incompatibilities with other substitutions in K_c, and shared derived-derived incompatibilities by interacting with substitutions in K_L and K_R. In contrast, substitutions that arose after T_d in Species A and B’s lineage, K_a and K_b respectively, will not result in any shared DMIs.

The total number of shared two-locus incompatibilities between AC and BC, I_sharedC, is then

$$I_{\mathit{sharedC}}=I_{K_t\times K_t}+I_{K_c\times K_c}+I_{K_c\times K_t}$$ (3)

where I_{Kt × Kt} represents the incompatibilities from substitutions before T_d, I_{Kc × Kc} represents the derived-ancestral incompatibilities from substitutions on the C branch after T_d, and I_{Kc × Kt} represents the interactions from substitutions on the C lineage after T_d with all substitutions that arose before T_d.

We can calculate the expected value of I_sharedC by using information about the expected number of substitutions on each branch. Let K_t be the total number of substitutions between the two lineages before T_d; there are $\left(\begin{array}{c}\hfill K_t\hfill \\ \hfill 2\hfill \end{array}\right)$ possible two locus incompatibilities involving only the K_t substitutions. Let K_c be the number of substitutions on the C lineage after T_d; there are $\left(\begin{array}{c}\hfill K_c\hfill \\ \hfill 2\hfill \end{array}\right)$ possible two locus derived-ancestral incompatibilities involving only the K_c substitutions. All interactions between the K_t and K_c substitutions are also shared; there are (K_t)(K_c) such shared interactions. Given the number of substitutions, the expected value of I_sharedC is then

$$\begin{array}{cc}\hfill \mathrm{E}[I_{\mathit{sharedC}}\mid \mathrm{K}]& =p\left[\left(\begin{array}{c}\hfill K_t\hfill \\ \hfill 2\hfill \end{array}\right)+\left(\begin{array}{c}\hfill K_c\hfill \\ \hfill 2\hfill \end{array}\right)+K_t{K}_c\right]\hfill \\ \hfill & =p\left(\begin{array}{c}\hfill K_t+K_c\hfill \\ \hfill 2\hfill \end{array}\right)\hfill \end{array}$$ (4)

This simplification shows that the number of shared interactions is equivalent to the number of combinations between substitutions shared by both pairs of parents. Under the assumption that the substitution rate k is the same across every branch of the tree,

$$\mathrm{E}\left[K_t\right]=2{\mathit{kt}}_1\mathrm{E}\left[K_c\right]={\mathit{kt}}_2$$ (5)

we can calculate E[I_sharedC] using Eq. (1), because K_t + K_c is Poisson distributed with mean E[K_t]+E[K_c]=k(2t₁+t₂)

$$\begin{array}{cc}\hfill \mathrm{E}\left[I_{\mathit{sharedC}}\right]=\mathrm{E}\left[\mathrm{E}[I_{\mathit{sharedC}}\mid \mathrm{K}]\right]& =\mathrm{E}\left[p\left(\begin{array}{c}\hfill K_t+K_c\hfill \\ \hfill 2\hfill \end{array}\right)\right]\hfill \\ \hfill & =\frac{p}{2}\mathrm{E}{[K_t+K_c]}^2\hfill \\ \hfill & =\frac{p}{2}{\left(\mathrm{E}\right[K_t]+\mathrm{E}[K_c\left]\right)}^2=\frac{p}{2}{(2{\mathit{kt}}_1+{\mathit{kt}}_2)}^2\hfill \end{array}$$ (6)

We now consider the incompatibilities shared between the AB and AC hybrids. While interactions between K_L and K_R can exist in the AC hybrid, they are absent in the AB hybrid, since the substitutions before divergence are fixed in both parental species. The only shared incompatibilities are derived-ancestral incompatibilities arising from K_a. By the same reasoning, the incompatibilities shared between the AB and BC hybrid involve only derived-ancestral interactions arising from K_b substitutions. The total number of possible two-locus incompatibilities shared between the AB and AC hybrids, I_sharedA, and between the AB and BC hybrids, I_sharedB, is then

$$I_{\mathit{sharedA}}=I_{K_a\times K_a}{I}_{\mathit{sharedB}}=I_{K_b\times K_b}$$ (7)

where I_{Ka × Ka} and I_{Kb × Kb} are the number of derived-ancestral incompatibilities that have arisen from substitutions after T_d in the A and B lineages respectively.

The expected value of I_sharedA and I_sharedB given information on the number of substitutions in each branch, K, is then

$$\mathrm{E}[I_{\mathit{sharedA}}\mid \mathrm{K}]=p\left(\begin{array}{c}\hfill K_a\hfill \\ \hfill 2\hfill \end{array}\right)\mathrm{E}[I_{\mathit{sharedB}}\mid \mathrm{K}]=p\left(\begin{array}{c}\hfill K_b\hfill \\ \hfill 2\hfill \end{array}\right)$$

Again, under the assumption that the substitution rate, k, is the same across every branch of the tree

$$\mathrm{E}\left[K_a\right]=\mathrm{E}\left[K_b\right]={\mathit{kt}}_2$$ (8)

We then calculate E[I_sharedA] and E[I_sharedB] using Eq. (1), substituting from Eq. (8),

$$\begin{array}{c}\hfill \mathrm{E}\left[I_{\mathit{sharedA}}\right]=\mathrm{E}\left[\mathrm{E}[I_{\mathit{sharedA}}\mid \mathrm{K}]\right]=\frac{p}{2}\mathrm{E}{\left[K_a\right]}^2\hfill \\ \hfill \mathrm{E}\left[I_{\mathit{sharedB}}\right]=\mathrm{E}\left[\mathrm{E}[I_{\mathit{sharedB}}\mid \mathrm{K}]\right]=\frac{p}{2}\mathrm{E}{\left[K_b\right]}^2\hfill \\ \hfill \mathrm{E}\left[I_{\mathit{sharedA}}\right]=\mathrm{E}\left[I_{\mathit{sharedB}}\right]=\frac{p}{2}{\left({\mathit{kt}}_2\right)}^2\hfill \end{array}$$ (9)

We now consider the variance of these shared incompatibilities, following Eq. (2). We do so below for the variance of I_sharedC. Recall from Eq. (4) that the number of shared interactions is equal to the number of substitutions shared between both pairs of parents, K_t + K_c; applying Eq. (2),

$$\begin{array}{cc}\hfill \mathrm{Var}\left[I_{\mathit{sharedC}}\right]& =\frac{p(1-p)}{2}\mathrm{E}{[K_t+K_c]}^2+{\left(\frac{p}{2}\right)}^2\left(\sum _{k=1}^2{\left(\begin{array}{c}\hfill 2\hfill \\ \hfill k\hfill \end{array}\right)}^{2}k!\mathrm{E}{[K_t+K_c]}^{4-k}\right)\hfill \\ \hfill & =\frac{p}{2}\mathrm{E}{[K_t+K_c]}^2(1+2p\mathrm{E}[K_t+K_c\left]\right)\hfill \end{array}$$

Using the independence of K_t and K_c and substituting from Eq. (5), the variance of I_sharedC is

$$\begin{array}{cc}\hfill \mathrm{Var}\left[I_{\mathit{sharedC}}\right]& =\frac{p}{2}{\left(\mathrm{E}\left[K_t\right]+\mathrm{E}\left[K_c\right]\right)}^2\left(1+2p\left(\mathrm{E}\right[K_t]+\mathrm{E}[K_c\left]\right)\right)\hfill \\ \hfill & =\frac{p}{2}{(2{\mathit{kt}}_1+{\mathit{kt}}_2)}^2(1+2p(2{\mathit{kt}}_1+{\mathit{kt}}_2\left)\right)\hfill \end{array}$$ (10)

The calculation for I_sharedB and I_sharedA follow closely and the results are presented in Table 1.

Covariance between Incompatibilities

From the pairwise comparison of three species, we have derived six different measures of the number of DMIs between three species: the incompatibilities in each of the three species pairs and the incompatibilities shared among the three species pairs. Next, we consider the covariance between each of these six measures of DMIs. Recall that each interaction between a set of substitutions can be modeled as a Bernoulli trial with probability p of becoming an incompatibility. While this assumption is unlikely to be true across the genome, empirical data to guide the development of a more sophisticated alternative is absent and previous theoretical treatments have maintained this assumption (Orr 1995; Turelli and Orr 2001; Welch 2004). Given the substitutions on each branch, the formation of one incompatibility is then assumed to be independent from another. The relationship between different measures of the number of incompatibilities is then dependent only on which incompatibilities are shared. We partition the incompatibilities of each measure into the independent components that can be shared between measures.

$$\begin{array}{cc}\hfill I_{\mathit{sharedC}}& =I_{K_t\times K_t}+I_{K_c\times K_c}+I_{K_c\times K_t}\hfill \\ \hfill I_{\mathit{sharedB}}& =I_{K_b\times K_b}\hfill \\ \hfill I_{\mathit{sharedA}}& =I_{K_a\times K_a}\hfill \\ \hfill I_{\mathit{AB}}& =I_{K_a\times K_a}+I_{K_b\times K_b}+I_{K_a\times K_b}\hfill \\ \hfill I_{\mathit{AC}}& =I_{\mathit{sharedC}}+I_{K_a\times K_a}+I_{K_a\times K_t}+I_{K_a\times K_c}\hfill \\ \hfill I_{\mathit{BC}}& =I_{\mathit{sharedC}}+I_{K_b\times K_b}+I_{K_b\times K_t}+I_{K_b\times K_c}\hfill \end{array}$$ (11)

This partitioning follows the logic from the calculation of the shared incompatibilities. The measures of shared incompatibilities involve interactions as described in Eqs. (3) and (7).

These relationships allow us to calculate the covariance given the number of substitutions. The calculation of this sum is demonstrated below for the covariance between I_sharedC and I_AC. We begin by partitioning the covariance following Eq. (11),

$$\begin{array}{cc}\hfill \mathrm{Cov}[I_{\mathit{sharedC}},I_{\mathit{AC}}]& =\mathrm{Var}\left[I_{\mathit{sharedC}}\right]+\mathrm{Cov}[I_{\mathit{sharedC}},I_{K_a\times K_a}]+\mathrm{Cov}[I_{\mathit{sharedC}},I_{K_a\times K_t}+I_{K_a\times K_c}]\hfill \\ \hfill & =\mathrm{Var}\left[I_{\mathit{sharedC}}\right]+\mathrm{Cov}[I_{\mathit{sharedC}},I_{K_a\times K_t}+I_{K_a\times K_c}]\hfill \end{array}$$ (12a)

Notice that the second term, the covariance of I_sharedC with I_{Ka × Ka}, is zero as there are no substitutions that participate in both sets of incompatibilities. Since the variance of IsharedC was calculated earlier, this leaves only the calculation of the third term, which can be determined by conditioning on K.

$$\begin{array}{cc}\hfill \mathrm{Cov}& [I_{\mathit{sharedC}},I_{K_a\times K_t}+I_{K_a\times K_c}]=\hfill \\ \hfill & \mathrm{E}\left[\mathrm{Cov}[I_{\mathit{sharedC}},I_{K_a\times K_t}+I_{K_a\times K_c}\mid \mathrm{K}]\right]+\mathrm{Cov}\left[\mathrm{E}[I_{\mathit{sharedC}}\mid \mathrm{K}],\mathrm{E}[I_{K_a\times K_t}+I_{K_a\times K_c}\mid \mathrm{K}]\right]\hfill \end{array}$$

We need only consider the second term, as the expected covariance given K for the first term is zero.

$$\mathrm{Cov}\left[\mathrm{E}\right[I_{\mathit{sharedC}}\mid \mathrm{K}],\mathrm{E}[I_{K_a\times K_t}+I_{K_a\times K_c}\mid \mathrm{K}\left]\right]=\mathrm{Cov}\left[p\left(\begin{array}{c}\hfill K_t+K_c\hfill \\ \hfill 2\hfill \end{array}\right),p(K_a{K}_t+K_a{K}_c)\right]$$

We calculate this using the definition of covariance, then expand and simplify with Eq. (1)

$$\begin{array}{cc}\hfill & =p^2\mathrm{E}\left[\left(\begin{array}{c}\hfill K_t+K_c\hfill \\ \hfill 2\hfill \end{array}\right)(K_a{K}_t+K_a{K}_c)\right]-p^2\mathrm{E}\left[\left(\begin{array}{c}\hfill K_t+K_c\hfill \\ \hfill 2\hfill \end{array}\right)\right]\mathrm{E}\left[K_a{K}_t+K_a{K}_c\right]\hfill \\ \hfill & =p^2\mathrm{E}\left[K_a\right]{\left(\mathrm{E}\left[K_c\right]+\mathrm{E}\left[K_t\right]\right)}^2\hfill \end{array}$$ (12b)

Substituting Eq. (12b) and the variance of I_sharedC from Eq. (10) into Eq. (12a) yields the total covariance. We can also substitute from Eqs. (5) and (8) to arrive at the covariance in terms of k, p, t₁ and t₂.

$$\begin{array}{cc}\hfill \mathrm{Cov}[I_{\mathit{sharedC}},I_{\mathit{AC}}]& =p^2\mathrm{E}\left[K_a\right]{\left(\mathrm{E}\left[K_c\right]+\mathrm{E}\left[K_t\right]\right)}^2+\frac{p}{2}{\left(\mathrm{E}\left[K_t\right]+\mathrm{E}\left[K_c\right]\right)}^2\left(1+2p\left(\mathrm{E}\right[K_t]+\mathrm{E}[K_c\left]\right)\right)\hfill \\ \hfill & =\frac{p}{2}{\left(\mathrm{E}\left[K_t\right]+\mathrm{E}\left[K_c\right]\right)}^2\left(1+2p\left(\mathrm{E}\right[K_t]+\mathrm{E}[K_a]+\mathrm{E}[K_c\left]\right)\right)\hfill \\ \hfill & =\frac{p}{2}{(2{\mathit{kt}}_1+{\mathit{kt}}_2)}^2(1+4k({\mathit{kt}}_1+{\mathit{kt}}_2\left)\right)\hfill \end{array}$$ (13)

The 14 other covariances, out of 15 unique covariances from the 6 measures of incompatibility, are provided in Table 2 (derivations are shown in supplementary material).

Table 2.

Covariance values for the two-locus model.

	Covariance
IAB, IAC	$\left(\frac{p}{2}\right){\left({\mathit{kt}}_2\right)}^2(1+8p({\mathit{kt}}_1+{\mathit{kt}}_2\left)\right)$
IAB, IBC	$\left(\frac{p}{2}\right){\left({\mathit{kt}}_2\right)}^2(1+8p({\mathit{kt}}_1+{\mathit{kt}}_2\left)\right)$
IAB, IsharedA	$\left(\frac{p}{2}\right){\left({\mathit{kt}}_2\right)}^2(1+4{\mathit{pkt}}_2)$
IAB, IsharedB	$\left(\frac{p}{2}\right){\left({\mathit{kt}}_2\right)}^2(1+4{\mathit{pkt}}_2)$
IAC, IsharedA	$\left(\frac{p}{2}\right){\left({\mathit{kt}}_2\right)}^2(1+4p({\mathit{kt}}_1+{\mathit{kt}}_2\left)\right)$
IBC, IsharedB	$\left(\frac{p}{2}\right){\left({\mathit{kt}}_2\right)}^2(1+4p({\mathit{kt}}_1+{\mathit{kt}}_2\left)\right)$
IAC, IsharedC	$\left(\frac{p}{2}\right){({\mathit{kt}}_2+2{\mathit{kt}}_1)}^2(1+4p({\mathit{kt}}_1+{\mathit{kt}}_2\left)\right)$
IBC, IsharedC	$\left(\frac{p}{2}\right){({\mathit{kt}}_2+2{\mathit{kt}}_1)}^2(1+4p({\mathit{kt}}_1+{\mathit{kt}}_2\left)\right)$
IAC, IBC	$\left(\frac{p}{2}\right)({\mathit{kt}}_2+2{\mathit{kt}}_1)\left(8p{\left({\mathit{kt}}_1\right)}^2+(2{\mathit{kt}}_1+{\mathit{kt}}_2)(1+8{\mathit{pkt}}_2)\right)$
IAB, IsharedA	0
IAC, IsharedB	0
IBC, IsharedA	0
IsharedA, IsharedB	0
IsharedA, IsharedC	0
IsharedB, IsharedC	0

Alternative Models on a Simple Tree

The treatment of the two-locus Dobzhansky-Muller model described above can be modified to include additional scenarios that describe the evolution of incompatibilities. We consider scenarios that involve different probabilities of derived-derived and derived-ancestral incompatibilities, three-locus interactions, and incompatibility accumulation that is linear with time.

Different Probabilities of Derived-Derived and Derived-Ancestral Incompatibilities

The number of incompatibilities does not depend on where substitutions fall on the two lineages in the two-locus Dobzhansky-Muller model (Muller 1942). Though interactions of alleles that arise in the same lineage are distinguished from interactions of alleles that arise from different lineages, it has been assumed that both types of interactions, derived-ancestral and derived-derived, are equally likely to produce an incompatibility (Orr 1995; Orr and Turelli 2001; Welch 2004). By relaxing this assumption, we may be able to recapture variation from population processes neglected by assumptions in the mathematical formulation of the Dobzhansky-Muller model. This would mean that the number of incompatibilities is no longer independent of where substitutions occur with respect to the lineages.

Recall that the classical mathematical treatment of the Dobzhansky-Muller model assumes the instantaneous and independent fixation of substitutions. If fixation is less than instantaneous, alleles that arise sequentially on a single lineage have a chance of co-existing in the same genome at the same time. By coexisting in a single lineage, the allelic combination would be subject to selection, precluding them from participating in a DMI and reducing the overall probability of derived-ancestral incompatibilities. Alternatively, substitutions on a single lineage may co-evolve such that a substitution at one incompatibility-forming allele favors substitutions at interacting alleles. This type of dependence would create an imbalance of substitutions on one lineage and manifest as a greater probability of derived-ancestral incompatibilities.

We define two new variables; let p_a be the probability that an untested derived-ancestral combination leads to an incompatibility, and p_d be the probability that an untested derived-derived combination leads to an incompatibility. Departing from our three taxa case momentarily, we reconsider the number of two-locus DMIs in two divergent populations with these new parameters. Let K_L be the number of substitutions on one branch of the tree and K_R be the number of substitutions on the other. Since interactions between substitutions from the same branch lead to derived-ancestral incompatibilities while interactions between substitutions from different branches lead to derived-derived incompatibilities, we have

$$\begin{array}{cc}\hfill \mathrm{E}\left[I\right]& =I_{K_L\times K_L}+I_{K_R\times K_R}+I_{K_L\times K_R}\hfill \\ \hfill & =\mathrm{E}\left[p_a\left(\begin{array}{c}\hfill K_L\hfill \\ \hfill 2\hfill \end{array}\right)\right]+\mathrm{E}\left[p_a\left(\begin{array}{c}\hfill K_R\hfill \\ \hfill 2\hfill \end{array}\right)\right]+\mathrm{E}\left[p_d{K}_L{K}_R\right]\hfill \\ \hfill & =\frac{p_a}{2}(\mathrm{E}{\left[K_L\right]}^2+\mathrm{E}{\left[K_R\right]}^2)+p_d\mathrm{E}\left[K_L\right]\mathrm{E}\left[K_R\right]\hfill \end{array}$$ (14)

Since we have maintained the assumption of equal rates of substitution on all branches,

$$\mathrm{E}\left[K_L\right]=\mathrm{E}\left[K_R\right]=\mathit{kt}$$ (15)

and the expected number of incompatibilities is

$$\mathrm{E}\left[I\right]=(p_a+p_d){\left(\mathit{kt}\right)}^2$$ (16)

It is clear from Eq. (16) that it is impossible to estimate p_a independently from p_d given only an estimate of kt and a measure of the number of incompatibilities between two species. Because the probabilities occur together as a sum, it is only possible to estimate (p_a+p_d). We show below this is not the case when more than two species are considered and that separate estimates of p_a and p_d can be made.

Returning to our example of three species on a known tree, we apply Eq. (16) to derive the expected number of incompatibilities in the AB, AC, and BC hybrids by simply substituting the correct divergence times.

$$\begin{array}{cc}\hfill \mathrm{E}\left[I_{\mathit{AB}}\right]& =(p_a+p_d){\left(\mathit{kt}\right)}^2\hfill \\ \hfill \mathrm{E}\left[I_{\mathit{AC}}\right]& =(p_a+p_d){({\mathit{kt}}_1+{\mathit{kt}}_2)}^2\hfill \\ \hfill \mathrm{E}\left[I_{\mathit{BC}}\right]& =(p_a+p_d){({\mathit{kt}}_1+{\mathit{kt}}_2)}^2\hfill \end{array}$$ (17)

Recall that hybrids of the AB and AC species pairs as well as hybrids of the AB and BC species pairs share only their derived-ancestral incompatibilities, I_shA and I_shB respectively. Following Eq. (9), the expected number of shared incompatibilities is

$$\mathrm{E}\left[I_{\mathit{shA}}\right]=\mathrm{E}\left[I_{\mathit{shB}}\right]=\frac{p_a}{2}{\left({\mathit{kt}}_2\right)}^2$$

However, shared incompatibilities between hybrids of the AC and BC species pair, I_shC, involve both derived-derived and derived-ancestral incompatibilities. To consider these, let K_L and K_R now represent the number of substitutions on their respective branches as in Figure 1. Following Eq. (3), we partition I_Kt in I_shC into I_KL and I_KR.

$$I_{\mathit{sharedC}}=I_{K_t\times K_t}+I_{K_c\times K_c}+I_{K_c\times K_t}$$

E[I_{KC × KC}] consists of (p_a/2) E[K_c]² derived-ancestral incompatibilities, and we have already solved for I_{Kt × Kt} in Eq. (14), leaving us only to partition I_{Kc × Kt}. K_c and K_L are substitutions from branches on opposite sides of the root while K_c and K_R are on branches from the same side, making the incompatibilities that form from the former, derived-derived, and the latter, derived-ancestral.

$$\begin{array}{cc}\hfill I_{K_c\times K_t}& =I_{K_c\times K_L}+I_{K_c\times K_R}\hfill \\ \hfill \mathrm{E}\left[I_{K_c\times K_t}\right]& =\mathrm{E}\left[I_{K_c\times K_L}\right]+\mathrm{E}\left[I_{K_c\times K_R}\right]\hfill \\ \hfill & =\mathrm{E}\left[p_d{K}_c{K}_L\right]+\mathrm{E}\left[p_a{K}_c{K}_R\right]\hfill \end{array}$$

Combining this with our previous results and substituting from Eqs. (5) and (15) results in

$$\mathrm{E}\left[I_{\mathit{shC}}\right]=\frac{p_a}{2}{\left({\mathit{kt}}_2\right)}^2+(p_a+p_d)({\left({\mathit{kt}}_1\right)}^2+({\mathit{kt}}_1\left)\right({\mathit{kt}}_2\left)\right)$$ (18)

The expected values for all six incompatibility measures are listed in Table 3. Similar reasoning and methods can be used to calculate the variance and covariances between the numbers of incompatibilities for this model. The results of these calculations are presented in Table 4.

Table 3.

Expectation and variance values for the Dobzhansky-Muller model where the probability of derived-ancestral incompatibilities, p_a, is different from the probability of derived-derived incompatibilities, p_d.

	Expectation	Variance
IAB	(pa+pd(kt2)2	(pa+pd)(kt2)2(1+2kt2(pa+pd))
IAC	(pa+pd)(kt1+kt2)2	(pa+pd)(kt1+kt2)2(1+2(kt1+kt2)(pa+pd))
IBC	(pa+pd)(kt1+kt2)2	(pa+pd)(kt1+kt2)2(1+2(kt1+kt2)(pa+pd))
IsharedA	$\left(\frac{p_a}{2}\right){\left({\mathit{kt}}_2\right)}^2$	$\left(\frac{p_a}{2}\right){\left({\mathit{kt}}_2\right)}^2(1+2p_a{\mathit{kt}}_2)$
IsharedB	$\left(\frac{p_a}{2}\right){\left({\mathit{kt}}_2\right)}^2$	$\left(\frac{p_a}{2}\right){\left({\mathit{kt}}_2\right)}^2(1+2p_a{\mathit{kt}}_2)$
IsharedC	$\left(\frac{p_a}{2}\right){\left({\mathit{kt}}_2\right)}^2+(p_a+p_d)\left({\left({\mathit{kt}}_1\right)}^2+\left({\mathit{kt}}_1\right)\left({\mathit{kt}}_2\right)\right)$	$\left(\frac{p_a}{2}\right){\left({\mathit{kt}}_2\right)}^2(1+2p_a{\mathit{kt}}_2)+\left({\mathit{kt}}_1\right)\left({\mathit{kt}}_2\right)\left(p_a+p_d+{\mathit{kt}}_2\left(3p_a^2+2p_a{p}_d+p_d^2\right)\right)+{\left({\mathit{kt}}_1\right)}^2(p_a+p_d)(1+(2{\mathit{kt}}_1+3{\mathit{kt}}_2\left)\right(p_a+p_d\left)\right)$

Table 4.

Covariance values for the p_a ≠ p_d Dobzhansky-Muller model.

	Covariance
IAB, IAC	$\left(\frac{p_a}{2}\right){\left({\mathit{kt}}_2\right)}^2(1+4p_d({\mathit{kt}}_1+{\mathit{kt}}_2\left)\right)+\left(p_a^2+p_d^2\right){\left({\mathit{kt}}_2\right)}^2({\mathit{kt}}_1+{\mathit{kt}}_2)$
IAB, IBC	$\left(\frac{p_a}{2}\right){\left({\mathit{kt}}_2\right)}^2(1+4p_d({\mathit{kt}}_1+{\mathit{kt}}_2\left)\right)+\left(p_a^2+p_d^2\right){\left({\mathit{kt}}_2\right)}^2({\mathit{kt}}_1+{\mathit{kt}}_2)$
IAB, IsharedA	$\left(\frac{p_a}{2}\right){\left({\mathit{kt}}_2\right)}^2(1+2{\mathit{kt}}_2(p_a+p_d\left)\right)$
IAB, IsharedB	$\left(\frac{p_a}{2}\right){\left({\mathit{kt}}_2\right)}^2(1+2{\mathit{kt}}_2(p_a+p_d\left)\right)$
IAC, IsharedA	$\left(\frac{p_a}{2}\right){\left({\mathit{kt}}_2\right)}^2(1+2({\mathit{kt}}_1+{\mathit{kt}}_2\left)\right(p_a+p_d\left)\right)$
IBC, IsharedB	$\left(\frac{p_a}{2}\right){\left({\mathit{kt}}_2\right)}^2(1+2({\mathit{kt}}_1+{\mathit{kt}}_2\left)\right(p_a+p_d\left)\right)$
IAC, IsharedC	$\left(\left(\frac{p_a}{2}\right){\left({\mathit{kt}}_2\right)}^2+{\mathit{kt}}_1({\mathit{kt}}_1+{\mathit{kt}}_2)(p_a+p_d)\right)(1+2({\mathit{kt}}_1+{\mathit{kt}}_2\left)\right(p_a+p_d\left)\right)$
IBC, IsharedC	$\left(\left(\frac{p_a}{2}\right){\left({\mathit{kt}}_2\right)}^2+{\mathit{kt}}_1({\mathit{kt}}_1+{\mathit{kt}}_2)(p_a+p_d)\right)(1+2({\mathit{kt}}_1+{\mathit{kt}}_2\left)\right(p_a+p_d\left)\right)$
IAC, IBC	$\left(\frac{p_a}{2}\right)\left({\left({\mathit{kt}}_2\right)}^2+2{\left({\mathit{kt}}_1\right)}^2+2\left({\mathit{kt}}_1\right)\left({\mathit{kt}}_2\right)\right)+p_a{\mathit{kt}}_1({\mathit{kt}}_1+{\mathit{kt}}_2)+2p_a{p}_d{({\mathit{kt}}_1+{\mathit{kt}}_2)}^2(2{\mathit{kt}}_1+{\mathit{kt}}_2)+\left(p_a^2+p_d^2\right){({\mathit{kt}}_1+{\mathit{kt}}_2)}^2(2{\mathit{kt}}_1+{\mathit{kt}}_2)$
IAB, IsharedA	0
IAC, IsharedB	0
IBC, IsharedA	0
IsharedA, IsharedB	0
IsharedA, IsharedC	0
IsharedB, IsharedC	0

Notice that when p_a = p_d = p, this model reduces to the formulation presented in Section 2. The observation that predicted quantities change when p_a ≠ p_d raises the possibility of estimating the fractions of derived-derived and derived-ancestral incompatibilities. If the assumption that substitutions are independent and accumulate proportional to branch length holds, then the ratio of these parameters represents the ratio of derived-derived and derived-ancestral incompatibilities.

Three-Locus Incompatibilities

Most of the developments in both the theoretical and empirical understanding of reproductive isolation under a Dobzhansky-Muller framework have focused on epistasis between two loci (Turelli and Orr 2001; Mendelson et al. 2004; Turelli and Moyle 2007; Gourbière and Mallet 2010; Moyle and Nakazato 2010; Matute et al 2010). This focus is likely explained by the challenge of mapping higher-order epistatic incompatibilities in crosses with practical sample sizes. There is, however, no theoretical reason to believe DMIs responsible for reproductive isolation are mainly the result of two-locus interactions. DMIs involving a greater number of loci may actually be more common because a greater number of participating loci allow a greater number of unconstrained evolutionary paths (Cabot et al. 1994; Orr 1995).

We expand the Dobzhansky-Muller model and incorporate three-locus incompatibilities by modeling the total number of DMIs as a result of both the interactions between two loci and the interactions between three loci. Let untested allele pairs and triples from these substitutions have a probability, p₂ and p₃, of resulting in an incompatibility respectively. Note that both types of incompatibilities arise from the same set of substitutions. The expected numbers of incompatibilities and shared incompatibilities follow from Eq. (1), as a sum of terms from n = 2 and n = 3 because of the linearity of the expectation operator.

For example, the expected number of incompatibilities between species A and species B, I_AB, from our three species example can be calculated as

$$I_{\mathit{AB}}=I_{\mathit{AB}}^{\left(2\right)}+I_{\mathit{AB}}^{\left(3\right)}$$ (19)

where I_AB⁽²⁾ are the two-locus incompatibilities between species A and B, and I_AB⁽³⁾ three-locus incompatibilities between species A and B. Given information on the number of substitutions, K, the expectation is

$$\begin{array}{cc}\hfill \mathrm{E}[I_{\mathit{AB}}\mid \mathrm{K}]& =\mathrm{E}\left[I_{\mathit{AB}}^{\left(2\right)}\mid \mathrm{K}\right]+\mathrm{E}\left[I_{\mathit{AB}}^{\left(3\right)}\mid \mathrm{K}\right]\hfill \\ \hfill & =p_2\left(\begin{array}{c}\hfill K_a+K_b\hfill \\ \hfill 2\hfill \end{array}\right)+p_3\left(\begin{array}{c}\hfill K_a+K_b\hfill \\ \hfill 3\hfill \end{array}\right)\hfill \end{array}$$

Applying Eq. (1) and then substituting for E[K_a] and E[K_b],

$$\begin{array}{cc}\hfill \mathrm{E}\left[I_{\mathit{AB}}\right]& =\frac{p_2}{2!}{\left(\mathrm{E}\right[K_a]+\mathrm{E}[K_b\left]\right)}^2+\frac{p_3}{3!}{\left(\mathrm{E}\right[K_a]+\mathrm{E}[K_b\left]\right)}^3\hfill \\ \hfill & =2p_2{\left({\mathit{kt}}_2\right)}^2+\frac{4p_3}{3}\left({\mathit{kt}}_2\right)3\hfill \end{array}$$ (20)

The calculation of the variances and covariances for the number of incompatibilities in this model are more involved because of statistical interaction between the n = 2 and n = 3 terms. These values can still be computed with the same techniques used in the previous sections but we do not reproduce them here. The results are presented, along with other expected values for this model, in Tables 5 and 6.

Table 5.

Expectation and variance values for the three-locus model with probability p₃ for three-locus incompatibilities and probability p₂ for two-locus incompatibilities.

	Expectation	Variance
IAB	$2p_2{\left({\mathit{kt}}_2\right)}^2+\left(\frac{4p_3}{3}\right){\left({\mathit{kt}}_2\right)}^3$	$2p_2{\left({\mathit{kt}}_2\right)}^2(1+4p_2{\mathit{kt}}_2)+\left(\frac{4p_3}{3}\right){\left({\mathit{kt}}_2\right)}^3(1+6p_3{\mathit{kt}}_2(1+{\mathit{kt}}_2\left)\right)+8p_2{p}_3{\left({\mathit{kt}}_2\right)}^3(1+2{\mathit{kt}}_2)$
IAC	$2p_2{({\mathit{kt}}_1+{\mathit{kt}}_2)}^2+\left(\frac{4p_3}{3}\right){({\mathit{kt}}_1+{\mathit{kt}}_2)}^3$	$2p_2{({\mathit{kt}}_1+{\mathit{kt}}_2)}^2(1+4p_2({\mathit{kt}}_1+{\mathit{kt}}_2\left)\right)+\left(\frac{4p_3}{3}\right){({\mathit{kt}}_1+{\mathit{kt}}_2)}^3(1+6p_3({\mathit{kt}}_1+{\mathit{kt}}_2\left)\right(1+{\mathit{kt}}_1+{\mathit{kt}}_2\left)\right)+8p_2{p}_3{({\mathit{kt}}_1+{\mathit{kt}}_2)}^3(1+2({\mathit{kt}}_1+{\mathit{kt}}_2\left)\right)$
IBC	$2p_2{({\mathit{kt}}_1+{\mathit{kt}}_2)}^2+\left(\frac{4p_3}{3}\right){({\mathit{kt}}_1+{\mathit{kt}}_2)}^3$	$2p_2{({\mathit{kt}}_1+{\mathit{kt}}_2)}^2(1+4p_2({\mathit{kt}}_1+{\mathit{kt}}_2\left)\right)+\left(\frac{4p_3}{3}\right){({\mathit{kt}}_1+{\mathit{kt}}_2)}^3(1+6p_3({\mathit{kt}}_1+{\mathit{kt}}_2\left)\right(1+{\mathit{kt}}_1+{\mathit{kt}}_2\left)\right)+8p_2{p}_3{({\mathit{kt}}_1+{\mathit{kt}}_2)}^3(1+2({\mathit{kt}}_1+{\mathit{kt}}_2\left)\right)$
IsharedA	$\left(\frac{p_2}{2}\right){\left({\mathit{kt}}_2\right)}^2+\left(\frac{p_3}{6}\right){\left({\mathit{kt}}_2\right)}^3$	$\left(\frac{p_2}{2}\right){\left({\mathit{kt}}_2\right)}^2(1+2p_2{\mathit{kt}}_2)+\left(\frac{p_3}{12}\right){\left({\mathit{kt}}_2\right)}^3(2+3p_3{\mathit{kt}}_2(2+{\mathit{kt}}_2\left)\right)+p_2{p}_3{\left({\mathit{kt}}_2\right)}^3(1+{\mathit{kt}}_2)$
IsharedB	$\left(\frac{p_2}{2}\right){\left({\mathit{kt}}_2\right)}^2+\left(\frac{p_3}{6}\right){\left({\mathit{kt}}_2\right)}^3$	$\left(\frac{p_2}{2}\right){\left({\mathit{kt}}_2\right)}^2(1+2p_2{\mathit{kt}}_2)+\left(\frac{p_3}{12}\right){\left({\mathit{kt}}_2\right)}^3(2+3p_3{\mathit{kt}}_2(2+{\mathit{kt}}_2\left)\right)+p_2{p}_3{\left({\mathit{kt}}_2\right)}^2(1+{\mathit{kt}}_2)$
IsharedC	$\left(\frac{p_2}{2}\right){(2{\mathit{kt}}_1+{\mathit{kt}}_2)}^2+\left(\frac{p_3}{6}\right){(2{\mathit{kt}}_1+{\mathit{kt}}_2)}^3$	$\left(\frac{p_2}{2}\right){(2{\mathit{kt}}_1+{\mathit{kt}}_2)}^2(1+2p_2(2{\mathit{kt}}_1+{\mathit{kt}}_2\left)\right)+\left(\frac{p_3}{12}\right){(2{\mathit{kt}}_1+{\mathit{kt}}_2)}^3(2+3p_3(2{\mathit{kt}}_1+{\mathit{kt}}_2\left)\right(2+2{\mathit{kt}}_1+{\mathit{kt}}_2\left)\right)+p_2{p}_3(1+2{\mathit{kt}}_1+{\mathit{kt}}_2){(2{\mathit{kt}}_1+{\mathit{kt}}_2)}^3$

Table 6.

Covariance values for the three-locus model.

	Covariance
IAB, IAC	$\left(\frac{p_3}{6}\right){\left({\mathit{kt}}_2\right)}^3(1+12p_3({\mathit{kt}}_1+{\mathit{kt}}_2\left)\right(1+2{\mathit{kt}}_1+2{\mathit{kt}}_2\left)\right)+\left(\frac{p_2}{2}\right){\left({\mathit{kt}}_2\right)}^2(1+8p_2)({\mathit{kt}}_1+{\mathit{kt}}_2))+p_2{p}_3{\left({\mathit{kt}}_2\right)}^2({\mathit{kt}}_1+2{\mathit{kt}}_2\left)\right(1+4{\mathit{kt}}_1+4{\mathit{kt}}_2)$
IAB, IBC	$\left(\frac{p_3}{6}\right){\left({\mathit{kt}}_2\right)}^3(1+12p_3({\mathit{kt}}_1+{\mathit{kt}}_2\left)\right(1+2{\mathit{kt}}_1+2{\mathit{kt}}_2\left)\right)+\left(\frac{p_2}{2}\right){\left({\mathit{kt}}_2\right)}^2(1+8p_2)({\mathit{kt}}_1+{\mathit{kt}}_2))+p_2{p}_3{\left({\mathit{kt}}_2\right)}^2({\mathit{kt}}_1+2{\mathit{kt}}_2\left)\right(1+4{\mathit{kt}}_1+4{\mathit{kt}}_2)$
IAB, IsharedA	$\left(\frac{p_3}{6}\right){\left({\mathit{kt}}_2\right)}^3(1+6p_3({\mathit{kt}}_2\left)\right(1+{\mathit{kt}}_2\left)\right)+\left(\frac{p_2}{2}\right){\left({\mathit{kt}}_2\right)}^2(1+4p_2{\mathit{kt}}_2)+\left(\frac{3p_2{p}_3}{2}\right){\left({\mathit{kt}}_2\right)}^3(1+2{\mathit{kt}}_2)$
IAB, IsharedB	$\left(\frac{p_3}{6}\right){\left({\mathit{kt}}_2\right)}^3(1+6p_3({\mathit{kt}}_2\left)\right(1+{\mathit{kt}}_2\left)\right)+\left(\frac{p_2}{2}\right){\left({\mathit{kt}}_2\right)}^2(1+4p_2{\mathit{kt}}_2)+\left(\frac{3p_2{p}_3}{2}\right){\left({\mathit{kt}}_2\right)}^3(1+2{\mathit{kt}}_2)$
IAC, IsharedA	$\left(\frac{p_3}{6}\right){\left({\mathit{kt}}_2\right)}^3(1+6p_3({\mathit{kt}}_1+{\mathit{kt}}_2\left)\right(1+{\mathit{kt}}_1+{\mathit{kt}}_2\left)\right)+\left(\frac{p_2}{2}\right){\left({\mathit{kt}}_2\right)}^2(1+4p_2({\mathit{kt}}_1+{\mathit{kt}}_2\left)\right)+\left(\frac{p_2{p}_3}{2}\right){\left({\mathit{kt}}_2\right)}^2(1+2{\mathit{kt}}_1+2{\mathit{kt}}_2)(2{\mathit{kt}}_1+3{\mathit{kt}}_2)$
IBC, IsharedB	$\left(\frac{p_3}{6}\right){\left({\mathit{kt}}_2\right)}^3(1+6p_3({\mathit{kt}}_1+{\mathit{kt}}_2\left)\right(1+{\mathit{kt}}_1+{\mathit{kt}}_2\left)\right)+\left(\frac{p_2}{2}\right){\left({\mathit{kt}}_2\right)}^2(1+4p_2({\mathit{kt}}_1+{\mathit{kt}}_2\left)\right)+\left(\frac{p_2{p}_3}{2}\right){\left({\mathit{kt}}_2\right)}^2(1+2{\mathit{kt}}_1+2{\mathit{kt}}_2)(2{\mathit{kt}}_1+3{\mathit{kt}}_2)$
IAC, IsharedC	$\left(\frac{p_3}{6}\right){\left({\mathit{kt}}_2\right)}^3(1+6p_3({\mathit{kt}}_1+{\mathit{kt}}_2\left)\right(1+{\mathit{kt}}_1+{\mathit{kt}}_2\left)\right)+\left(\frac{p_2}{2}\right){\left({\mathit{kt}}_2\right)}^2(1+4p_2({\mathit{kt}}_1+{\mathit{kt}}_2\left)\right)+\left(\frac{p_2{p}_3}{2}\right){\left({\mathit{kt}}_2\right)}^2(1+2{\mathit{kt}}_1+2{\mathit{kt}}_2)(2{\mathit{kt}}_1+3{\mathit{kt}}_2)$
IBC, IsharedC	$\left(\frac{p_3}{6}\right){(2{\mathit{kt}}_1+{\mathit{kt}}_2)}^3(1+6p_3({\mathit{kt}}_1+{\mathit{kt}}_2\left)\right(1+{\mathit{kt}}_1+{\mathit{kt}}_2\left)\right)+\left(\frac{p_2}{2}\right){(2{\mathit{kt}}_1+{\mathit{kt}}_2)}^2(1+4p_2({\mathit{kt}}_1+{\mathit{kt}}_2\left)\right)+\left(\frac{p_2{p}_3}{2}\right){(2{\mathit{kt}}_1+{\mathit{kt}}_2)}^2(1+2{\mathit{kt}}_1+2{\mathit{kt}}_2)(4{\mathit{kt}}_1+3{\mathit{kt}}_2)$
IAC, IBC	$\left(\frac{p_3}{6}\right)(2{\mathit{kt}}_1+{\mathit{kt}}_2)\left({(2{\mathit{kt}}_1+{\mathit{kt}}_2)}^2+p_3\left(12{({\mathit{kt}}_1+{\mathit{kt}}_2)}^2\left(2{\mathit{kt}}_1+{\mathit{kt}}_2+2{({\mathit{kt}}_1+{\mathit{kt}}_2)}^2\right)\right)\right)+\left(\frac{p_2}{2}\right)(2{\mathit{kt}}_1+{\mathit{kt}}_2)\left(8p_2{\left({\mathit{kt}}_1\right)}^2+(2{\mathit{kt}}_1+{\mathit{kt}}_2)(1+8p_2{\mathit{kt}}_2)\right)+2p_2{p}_3(2{\mathit{kt}}_1+{\mathit{kt}}_2)({\mathit{kt}}_1+{\mathit{kt}}_2)\left(4{\left({\mathit{kt}}_1\right)}^2+(2{\mathit{kt}}_1+{\mathit{kt}}_2)(1+4{\mathit{kt}}_2)\right)$
IAB, IsharedA	0
IAC, IsharedB	0
IBC, IsharedA	0
IsharedA, IsharedB	0
IsharedA, IsharedC	0
IsharedB, IsharedC	0

Notice that unlike the model where derived-ancestral incompatibilities are allowed to occur with a different probability than derived-derived incompatibilities, the two parameters in this model, p₂ and p₃, are not directly comparable. There are many more untested allele combinations that are mathematically possible and even if two-locus and three-locus interactions contributed equally to the total number of incompatibilities, p₂ would be orders of magnitude greater than p₃.

Non-Epistatic Models: Linear Incompatibility Accumulation

To model the linear accumulation of hybrid incompatibilities among three species, we begin with the same framework developed for the two-locus epistatic model. But instead of interacting with other substitutions, we imagine each new substitution has a small probability of causing an incompatibility. This probability, which we denote p₁, represents the chance that a single substitution in one lineage causes an incompatibility in a hybrid with any other lineage. While this value appears similar to the probability, p, in the two-locus model, it is not analogous and parameterizes a single locus model that does not involve interaction.

The number of incompatibilities predicted in a hybrid of two species by the single locus model has expectation and variance

$$\mathrm{E}\left[I\right]=2p_1\mathit{kt}\mathrm{Var}\left[I\right]=2p_1\mathit{kt}$$ (21)

where t is the amount of time since the divergence of the two species. Consider now the behavior of the number of incompatibilities in the three species example. As in the two-locus interaction model, the number of incompatibilities when considering each hybrid individually follows the expectations and variance described above. However, the manner in which incompatibilities are shared is much simplified. Since each incompatibility involves only a single allele, incompatibilities are shared as long as they exist in both hybrids. For example, the incompatibilities shared between the AB and AC hybrids, I_sharedA, include all incompatibilities that arose from K_a substitutions. The set of these sharing relationships for the single locus model is summarized below; contrast this with the two-locus Dobzhansky-Muller relationships presented in Eqs. (11).

$$\begin{array}{cc}\hfill & {\mathrm{I}}_{\mathit{sharedC}}=I_{K_t}+I_{K_c}\hfill \\ \hfill & I_{\mathit{sharedA}}=I_{K_a}\hfill \\ \hfill & I_{\mathit{sharedB}}=I_{K_b}\hfill \\ \hfill & I_{\mathit{AB}}=I_{K_a}+I_{K_b}\hfill \\ \hfill & I_{\mathit{AC}}=I_{\mathit{sharedC}}+I_{K_a}\hfill \\ \hfill & I_{\mathit{BC}}=I_{\mathit{sharedC}}+I_{K_b}\hfill \end{array}$$ (22)

Using the relationships in Eqs. (21) and (22), the expectation, variance, and covariances of the number of incompatibilities can be calculated for the single locus model; results are presented in Tables 7 and 8.

Table 7.

Expectation and variance values for the single-locus model of incompatibility accumulation with probability p1 of incompatibility formation for each new substitution.

	Expectation	Variance
IAB	2p1kt2	2p1kt2
IAC	2p1(kt1 + kt2)	2p1(kt1 + kt2)
IBC	2p1(kt1 + kt2)	2p1(kt1 + kt2)
IsharedA	p1(2kt1 + kt2)	p1(2kt1 + kt2)
IsharedB	p 1 kt 2	p 1 kt 2
IsharedC	p 1 kt 2	p 1 kt 2

Table 8.

Covariance values for the single-locus model.

	Covariance
IAB, IAC	p 1 kt 2
IAB, IBC	p 1 kt 2
IAB, IsharedA	p 1 kt 2
IAB, IsharedB	p 1 kt 2
IAC, IsharedA	p 1 kt 2
IBC, IsharedB	p 1 kt 2
IAC, IsharedC	p1(2kt1 + kt2)
IBC, IsharedC	p1(2kt1 + kt2)
IAC, IBC	p1(2kt1 + kt2)
IAB, IsharedA	0
IAC, IsharedB	0
IBC, IsharedA	0
IsharedA, IsharedB	0
IsharedA, IsharedC	0
IsharedA, IsharedC	0

Inference Methods and Simulation

Maximum-Likelihood Estimators and the AIC

We have described how hybrid incompatibilities are distributed across a three-taxon phylogeny under different models of incompatibility accumulation. We use these properties to develop a method that compares the goodness-of-fit between models and estimates model parameters. We continue to focus on the three species example with six observations of incompatibilities; however, it is mathematically trivial to extend this method to a larger number of related species.

We first consider the probability distribution of a vector of observable incompatibility measures. A set of incompatibility measures, e.g. II = {I_AB, I_AC, I_BC, I_sharedA, I_sharedB, I_sharedC}, is the result of a single instance of the incompatibility accumulation process. The set of observations from a genetic mapping study can be thought of as a single vector on a distribution of possible observations. This distribution of possible observations is a function of the nature of hybrid incompatibility formation. Recall that the distribution for the number of substitutions between species has been modeled to be Poisson and the distribution for the number of incompatibilities given the number of substitutions has been modeled to be binomial; when a hybrid from a single pair of species is considered, the distribution of the number of DMIs has been shown to be well approximated by a Gaussian (Orr and Turelli 2001). We apply this idea and approximate the distribution of the vector of incompatibility measures, II, with a multivariate Gaussian.

We have mathematically formulated the expectations and covariances of incompatibility measures for a selection of different models (results summarized in Tables 1 through 8). By constructing a vector mean and covariance matrix from these results, the probability distribution function of possible incompatibility observations can be defined for any given model. Since the calculated expectations and covariances for each model are themselves functions of the model parameters, the probability distribution function can be written explicitly as a function of the model parameters.

From this explicit probability distribution function, we define a likelihood function for a vector of observed incompatibilities and a set of model parameters. For a given model with six observations, we have

$$L(x,\theta )=\frac{1}{{\left(2\pi \right)}^3\sqrt{\mid \Sigma \mid }}\exp \left[\frac{1}{2}{(x-\mu (\theta \left)\right)}^{\mathrm{T}}{\left[\Sigma \right(\theta \left)\right]}^{-1}(x-\mu (\theta \left)\right)\right]$$ (23)

where x is a vector of observed incompatibilities, μ is the vector of expected incompatibilities, Σ is the covariance matrix, ∣Σ∣ is the determinant of the covariance matrix, and θ is a set of parameters in the model.

We find a set of parameters that maximizes this likelihood function for an observed vector of incompatibilities. This is the maximum-likelihood estimator for the model parameters

$$\widehat{\theta }=\underset{\theta }{\mathrm{argmax}}\left[L\right(x,\theta \left)\right]$$ (24)

The relative goodness of fit for each model can be compared by applying the Akaike Information Criterion (Akaike 1974), taking into account that the single-locus and two-locus models each have a single parameter, p₁ and p respectively, while the p_a ≠ p_d and the three-locus models each have two parameters, p_a, p_d and p₂, p₃ respectively.

Calculating the likelihood for the single-locus model is complicated by the rank of the covariance matrix. Based on how we have modeled single-locus incompatibilities to accumulate, there are three independent measures of incompatibility (see Eq. (22)). As a result, the dimensionality of the covariance matrix exceeds the rank, making the matrix singular and the likelihood ill-defined. To remedy this, we add to the single-locus model a small, bounded parameter, δ, that we call the error term. We envision an empirical measurement error of each observation that scales with the theoretical variance. The error term captures this variance and conveniently increases the rank of the covariance matrix,

$$\Sigma =\left(\begin{array}{cccccc}{\sigma }_{\mathit{ab}}^2(1+\delta )\hfill & {\sigma }_{\mathit{ab},\mathit{ac}}\hfill & {\sigma }_{\mathit{ab},\mathit{bc}}\hfill & {\sigma }_{\mathit{ab},\mathit{shC}}\hfill & {\sigma }_{\mathit{ab},\mathit{shB}}\hfill & {\sigma }_{\mathit{ab},\mathit{shA}}\hfill \\ {\sigma }_{\mathit{ab},\mathit{ac}}\hfill & {\sigma }_{\mathit{ac}}^2(1+\delta )\hfill & {\sigma }_{\mathit{ac},\mathit{bc}}\hfill & {\sigma }_{\mathit{ac},\mathit{shC}}\hfill & {\sigma }_{\mathit{ac},\mathit{shB}}\hfill & {\sigma }_{\mathit{ac},\mathit{shA}}\hfill \\ {\sigma }_{\mathit{ab},\mathit{bc}}\hfill & {\sigma }_{\mathit{ac},\mathit{bc}}\hfill & {\sigma }_{\mathit{bc}}^2(1+\delta )\hfill & {\sigma }_{\mathit{bc},\mathit{shC}}\hfill & {\sigma }_{\mathit{bc},\mathit{shB}}\hfill & {\sigma }_{\mathit{bc},\mathit{shA}}\hfill \\ {\sigma }_{\mathit{ab},\mathit{shC}}\hfill & {\sigma }_{\mathit{ac},\mathit{shC}}\hfill & {\sigma }_{\mathit{bc},\mathit{shC}}\hfill & {\sigma }_{\mathit{shC}}^2(1+\delta )\hfill & {\sigma }_{\mathit{shC},\mathit{shB}}\hfill & {\sigma }_{\mathit{shC},\mathit{shA}}\hfill \\ {\sigma }_{\mathit{ab},\mathit{shB}}\hfill & {\sigma }_{\mathit{ac},\mathit{shB}}\hfill & {\sigma }_{\mathit{bc},\mathit{shB}}\hfill & {\sigma }_{\mathit{shC},\mathit{shB}}\hfill & {\sigma }_{\mathit{shB}}^2(1+\delta )\hfill & {\sigma }_{\mathit{shB},\mathit{shA}}\hfill \\ {\sigma }_{\mathit{ab},\mathit{shA}}\hfill & {\sigma }_{\mathit{ac},\mathit{shA}}\hfill & {\sigma }_{\mathit{bc},\mathit{shA}}\hfill & {\sigma }_{\mathit{shC},\mathit{shA}}\hfill & {\sigma }_{\mathit{shB},\mathit{shA}}\hfill & {\sigma }_{\mathit{shA}}^2(1+\delta )\hfill \end{array}\right)$$ (25)

where σ², σ, and δ correspond to the respective variances, covariances, and error factor.

To summarize, given a particular set of incompatibility observations, the maximum-likelihood estimators are calculated for each model under consideration and the model that yields the lowest AIC is preferred. We have not rigorously justified the approximation that the distribution of a vector of incompatibilities should be a multivariate Gaussian, but the effectiveness of the inference procedure in distinguishing between different models of incompatibility accumulation can be determined by its application to simulated data.

Simulation and Results

To test our inference method, we ran simulations of hybrid incompatibility accumulation on three-species trees under the single-locus, two-locus, three-locus, and p_a ≠ p_d models we described earlier. We simulated the number of substitutions on each branch by drawing from a Poisson with a mean proportional to the branch length and substitution rate, kt. From these numbers of substitutions, we calculated the number of potential interactions based on the model of incompatibility accumulation being simulated. The number of incompatibilities for each branch was then drawn from a binomial, using the number of potential interactions and the model’s probability (or probabilities) that an interaction becomes an incompatibility as parameters. The maximum-likelihood estimators and AIC values for each of the models were then calculated from the results of each simulation to measure the method’s ability to correctly identify the underlying simulated model.

Since power depends on the total number of incompatibilities simulated, we divided the simulations into 4 groups, corresponding to 20, 50, 100, and 150 expected incompatibilities between the two most diverged species by adjusting the probability that an interaction led to an incompatibility. For each of these groups, we ran 2000 simulations on 15 trees of differing internal branch length while holding the longest branch length constant at k(t₁+t₂) = 10000. For simulations of models with more than one parameter, the three-locus and p_a ≠ p_d models, the total number of expected incompatibilities was held constant while the contribution from each component was randomly varied. The same constraint was applied to p₂ and p₃ for the three-locus model.

Our estimation of θ is complicated by the fact that the covariance matrix, Σ, depends on its estimation, that is Σ = Σ(θ). In order to deal with this, we use an optimization routine; our maximum-likelihood estimators were calculated numerically using the bounded BFGS algorithm as implemented in the Python programming language’s SciPy library. The error term, δ, for calculation of the single-locus model’s maximum likelihood, was bounded to [1 × 10⁻⁴, 1]. The error term is bounded so that it does not dominate the parameter estimates when the model fits poorly. One concern with adding a bounded parameter is that it will penalize the AIC by adding an additional parameter to the model without allowing the parameter to fully contribute to increasing the likelihood. However, we will see this is not a concern since the single-locus model is overwhelmingly supported when the underlying simulation is single-locus.

The first simulation result we consider is the relative fit of the single-locus and two-locus models when the simulation is single-locus. Figure 3(top) shows the AIC values calculated for the single-locus model to be uniformly lower than the AIC values calculated for the two-locus model. Comparisons of the single-locus AIC with other model AICs show a similar pattern; that is, when the underlying mode of simulated incompatibility accumulation is linear, the single-locus model is always favored. In contrast, two-locus AIC values are not always lower than the single-locus AIC when the underlying mode of incompatibility simulation involves two-locus epistasis (Figure 3(middle)). Nevertheless, the two-locus model is correctly favored over the single-locus model in greater than 80% of simulations even with the longest internal branch length and only 20 expected incompatibilities between the most divergent species (Figure 3(bottom)). These results suggest high power to detect the acceleration in incompatibility accumulation predicted by the Dobzhansky-Muller model. In both comparisons, the distinction between the AIC values increases with a greater number of expected incompatibilities and with greater divergence time between the most closely related species.

Figure 3.

Results from simulations of single-locus and two-locus incompatibility accumulation. Plots are divided horizontally into four columns based on model parameters that correspond to, from left to right, 20, 50, 100, and 150 expected incompatibilities between the most diverged species. Simulations on fifteen three-species trees of differing times since most recent divergence, T_d, are represented in each individual plot. Boxes show median and quartile values while whiskers show range. Highlighted values correspond to AIC calculated for a single-locus model while those in black correspond to AIC calculated for a two-locus model. (top) AIC values from a simulation of single-locus incompatibility accumulation. (middle) AIC values from a simulation of two-locus incompatibility accumulation. (bottom) Percentages of two-locus simulations where each model is favored by having lower AIC. (In single-locus simulations, the single-locus model is always favored.)

Differences in AIC values between simulations of Dobzhansky-Muller-type models (two-locus, three-locus, and p_a ≠ p_d) are much smaller, making the distinction between these models less clear (Figure 4). Nevertheless, Dobzhansky-Muller-type models are consistently favored over the single-locus model in these simulations. As we might expect, the underlying model of simulation is correctly favored more frequently when a greater number of total incompatibilities is simulated. Unlike the comparison between the single-locus and two-locus models, shorter relative internal branch lengths do not necessarily result in a clearer distinction between the different Dobzhansky-Muller-type models. While the two-locus model is still most easily identified with decreasing relative internal branch length, the power to correctly identify the other two Dobzhansky-Muller-type models is not monotonic with relative internal branch length.

Figure 4.

Comparison of favored models for Dobzhansky-Muller-type model simulations. Plots are divided horizontally into four columns based on model parameters that correspond to, from left to right, 20, 50, 100, and 150 expected incompatibilities between the most diverged species. Simulations on fifteen three-species trees of differing times since most recent divergence, T_d, are represented in each individual plot. Each bar shows the percentage of simulations where each model had the lowest calculated AIC among compared models. (top) Comparison of favored models from simulations of the two-locus model. (middle) Comparison of favored model from simulations of the p_a ≠ p_d model. (bottom) Comparison of favored models from simulations of the three-locus model.

We further investigated the nested Dobzhansky-Muller-type models, the p_a ≠ p_d and three-locus models, to determine the power of the inference method to identify more complex models under various parameter values. For these two models, we ran 5000 simulations on each of the 15 trees of varying internal branch length. The percentage of incompatibilities attributable to any single parameter was randomly assigned, by randomly selecting the ratio of p_a to p_d or p₂ to p₃ respectively, while the total number of expected incompatibilities was fixed for the 4 groups (20, 50, 100, and 150 expected incompatibilities). The results were binned into ten fractions based on the percent that each parameter contributed to the total number of incompatibilities. Figure 4 is a representative plot showing the percentage of simulations where the more complex model is favored over simpler Dobzhansky-Muller-type models. There is no simple relationship between power and relative internal branch length. As expected, the p_a ≠ p_d model is more likely to be favored when there is a greater imbalance between derived-derived and derived-ancestral incompatibilities. Similarly, the three-locus model is more likely to be favored when three-locus interactions account for a greater fraction of the total number of incompatibilities. These results indicate that evidence for complex models of DMI accumulation can be inferred under the right conditions.

DISCUSSION

The comparison of patterns of reproductive isolation across species has generated tremendous insights into the speciation process (Haldane 1922; Coyne and Orr 1989, 1997; Sasa et al. 1998; Presgraves 2002; Lijtmaer et al. 2003; Price and Bouvier 2003; Bolnick and Near 2005). The benefits of the comparative approach extend to genetic mapping of isolation phenotypes (Moyle and Payseur 2009). Our results demonstrate that different modes of incompatibility evolution leave contrasting patterns of incompatibility sharing across species pairs, in a manner that depends on the phylogeny. This finding has important implications for genetic studies of speciation. First, genetic models of speciation can be directly tested using comparative mapping data by counting the number of shared and unique incompatibilities between each lineage. Though the Dobzhanksy-Muller model enjoys considerable support from genetic mapping studies, its evolutionary predictions (e.g., Orr 1995) have only recently been tested, with mixed results (Matute et al. 2010; Moyle and Nakazato 2010). Second, our results situate the evolving genetic architecture of reproductive isolation in its phylogenetic context. As with any trait, the genetic changes that generate isolation evolve along a tree: statistical inferences must take into account this fact. Finally, we see that clades of species with different phylogenetic structures (but similar ages) are expected to show distinct patterns. Our results provide guidance on choosing species groups for reconstructing the evolution of reproductive barriers from a genetic perspective.

Assumptions and Challenges

Two major categories of assumptions underlie the approach described here, those associated with our mathematical extension of the Dobzhansky-Muller model and those associated with our phylogenetic test. Much of the power from our comparative framework comes from modeling shared incompatibilities; this extension of the Dobzhansky-Muller model assumes that the same allelic combinations in different species pairs result in the same incompatibilities. Our extended model also assumes that all of the incompatibilities measured from different species pairs are distributed as a multivariate normal. The validity of this latter assumption should improve with larger numbers of incompatibilities.

Our phylogenetic test assumes that relative divergence time and tree topology estimates are accurate. Relative divergence times can be estimated from substitution rates. For example, the number of synonymous substitutions per synonymous site, K_s, has been used as a proxy for divergence time in recent empirical studies of incompatibility accumulation (Matute et al., 2010; Moyle and Nakazato, 2010). Obtaining an accurate phylogeny can be complicated by gene flow and incomplete lineage sorting, which can cause the phylogeny to vary across the genome, especially in closely related species (Maddison 1997; Pollard et al. 2006; White et al. 2009). Phylogenies at incompatibility loci are the most relevant for the evolution of reproductive isolation, but phylogenies from randomly chosen loci may often be the only ones available. The maximum-likelihood method used in our test also implicitly weights each of the incompatibility measurements equally. This assumption, that each measure is equally accurate, is most reasonable when species pairs have similar divergence times so that deviations from the expected values are equally distributed among the measurements.

Our method currently assumes that all incompatibilities are counted, but this will rarely be true in practice. Sample size constraints and error in genetic mapping generally prohibit the identification of all loci that affect genetically complex traits such as hybrid inviability and hybrid sterility. This problem is exacerbated for epistatic loci, where genetic identification requires the comparison of multi-locus genotypes, further partitioning the sample and requiring large numbers of individuals to find both incompatibility partners. A reduced count of total incompatibilities should not bias the results, as long as incompatibilities that arise on different parts of the phylogeny are equally detectable. Similarly, if we assume that incompatibilities are independent, finding one of the multiple loci that comprise an incompatibility should not introduce a bias. Nevertheless, the effects on method performance when only a subset of incompatibility loci is detected should be examined.

A further challenge comes from the designation of an incompatibility as shared. Because genetic mapping often associates large chromosomal regions with the phenotype, positional overlaps are not always shared genetic changes. Although the ideal solution is to identify or finely map the causative mutations, approaches that statistically evaluate the evidence for shared loci are available (Li et al. 2005; Broman et al. 2012). Extending these methods to treat sets of epistatic loci would be helpful for studies of the genetics of speciation.

Studying the Genetics of Speciation in its Phylogenetic Context

Our simulations show that the Dobzhansky-Muller model and a non-epistatic alternative in which isolation mutations accumulate linearly with time can be reliably distinguished with comparative mapping data. The distinction is clearest in phylogenies with substantial external branch lengths. This is because incompatibilities accumulate stochastically and there is greater uncertainty in the number of incompatibilities when few exist between the most recently diverged species.

Our approach reveals additional characteristics of incompatibility evolution that have long been of interest in speciation genetics. Coevolution of incompatibility substitutions (Tang and Presgraves 2009) – so that one substitution affects the fate of the other – increases the proportion of incompatibilities that arise along one lineage (derived-ancestral incompatibilities). Alternatively, when incompatibility substitutions are evolutionarily independent, a higher fraction of derived-derived incompatibilities is predicted because derived alleles can be incompatible with both derived and ancestral alleles (while ancestral alleles can only be incompatible with derived alleles) (Orr 1995). Identifying both partner loci and reconstructing the phylogenetic placement of substitutions is a powerful approach for distinguishing the two incompatibility types (Cattani and Presgraves 2009), but this is a daunting task for many species. Our method allows estimation of the frequencies of derived-derived and derived-ancestral incompatibilities across the genome. Furthermore, whether incompatibilities mostly involve pairs of substitutions or instead feature higher-order interactions measures the complexity of disrupted genetic networks in hybrids. Incompatibilities featuring changes at more than two loci are well-known (Cabot et al. 1994; Davis et el. 1994; Palopoli and Wu 1994; Perez and Wu 1995; Orr and Irving 2001; Phadnis and Orr 2009; Phadnis 2011) and predicted to be common (Orr 1995), but their overall contribution to hybrid dysfunction compared to pairwise interactions has not been quantified in the same hybrids. These genetic characterizations of the speciation process would be difficult to draw from individual mapping studies, but are possible with comparative mapping data.

The shape of the tree also affects the ability to differentiate between these flavors of the Dobzhansky-Muller model. Phylogenies with greater external branch length offer higher power to find unequal evolutionary rates for ancestral-derived and derived-derived incompatibilities. In contrast, the ability to detect a significant contribution from three-locus incompatibilities is greatest for phylogenies with intermediate values of internal branch length. This reflects a balance between measuring the number of incompatibilities involving substitutions on the internal branch, which improves with longer internal branch lengths, and measuring the number of incompatibilities between the most recently diverged species, which improves with longer external branch lengths. An exception to these patterns is observed when one species pair has very recently diverged; in this case, the p_a ≠ p_d model is favored in all simulations of complex Dobzhansky-Muller type models. This is likely due to the greater freedom of p_a in the p_a ≠ p_d model when t₂ is small. Comparing the two-parameter models (Table 3 and Table 4), we see that I_sharedA and I_sharedB depend only on a single parameter, p_a, in the p_a ≠ p_d model whereas all six measures involve both parameters in the three-locus model.

In summary, our results suggest guidelines for selecting species groups for comparative mapping of reproductive isolation. If the primary goal is to distinguish epistatic from non-epistatic models, clades with long internal branches should be avoided. When attempting to gauge the contribution of three-locus incompatibilities or to compare the frequencies of derived-derived and derived-ancestral incompatibilities, groups of species with variable divergence times should be favored. Finally, the addition of data from mapping studies of taxa on the same phylogeny will undoubtedly improve the power to distinguish between models, as it will increase the total number of incompatibilities measured. Extensions to our approach are worthy of consideration. The method of analysis we employed, incorporating shared incompatibilities and the expected covariance between different species pairs, can be generalized to more complex models. While we have mathematically described only the incorporation of phylogenetic information from three species, the calculation for additional species pairs is straightforward. The primary barrier to the addition of more species in the analysis is the difficulty in collecting incompatibility data from more species pairs. This is apparent when we consider the addition of a fourth species to the phylogeny. For the complete set of incompatibilities measured between a phylogeny with four species, a genetic analysis of the isolating traits in six hybrids is necessary – this is double the number from a phylogeny with only three species. Nevertheless, we anticipate that the insights into speciation uniquely provided by comparative genetic mapping will motivate the increased application of this framework.

Appendix.

Below we derive Equation (1), the expected number of DMIs when considering interactions between n loci as a function of the expected number of substitutions separating two lineages.

$$\begin{array}{cc}\hfill \mathrm{E}\left[I\right]=\mathrm{E}\left[\mathrm{E}\right[I\mid K\left]\right]& =\mathrm{E}\left[p_n\left(\begin{array}{c}\hfill K\hfill \\ \hfill n\hfill \end{array}\right)\right]\hfill \\ \hfill & =\mathrm{E}\left[\frac{p_n}{n!}{\left(K\right)}_n\right]\hfill \end{array}$$

where (x)_n is the falling factorial, (x)_n = x(x-1)…(x-n+1). Using the definition of the expectation and the fact that K is Poisson distributed, let λ = E[K]

$$\begin{array}{cc}\hfill \mathrm{E}\left[{\left(K\right)}_n\right]& =\sum _{k=0}^{\infty }k(k-1)\cdots (k-n+1)\frac{{\lambda }^k}{k!}{e}^{-\lambda }\hfill \\ \hfill & ={\lambda }^n\sum _{j=0}^{\infty }\frac{{\lambda }^j}{j!}{e}^{-\lambda }\hfill \\ \hfill & ={\lambda }^n\hfill \end{array}$$ (A1)

Recognizing the summation as over the probability mass function of a Poisson distribution, the expected number of DMIs is then

$$\mathrm{E}\left[I\right]=\frac{p_n}{n!}\mathrm{E}{\left[K\right]}^n$$

We derive below Equation (2), the variance in the number of DMIs when considering interactions between n loci as a function of the expected number of substitutions between two lineages. We begin with the law of total variance, conditioning on the total number of substitutions between the lineages, K,

$$\mathrm{Var}\left[I\right]=\mathrm{E}\left[\mathrm{Var}[I\mid K]\right]+\mathrm{Var}\left[\mathrm{E}[I\mid K]\right]$$

Since the number of incompatibilities given the number of substitutions is binomially distributed, each interaction is a Bernoulli trial with probability p_n of becoming an incompatibility. The variance in the number of incompatibilities given K is then the product of p_n(1-p_n) and the number of interactions.

$$=\mathrm{E}\left[p_n(1-p_n)\left(\begin{array}{c}\hfill K\hfill \\ \hfill n\hfill \end{array}\right)\right]+\mathrm{Var}\left[\mathrm{E}[I\mid K]\right]$$

Using Eq. (A1) to simplify the left hand term and recognizing the expected number of incompatibilities given K is the product of p_n and the number of substitutions,

$$=\frac{p_n(1-p_n)}{n!}\mathrm{E}{\left[K\right]}^n+\mathrm{Var}\left[p_n\left(\begin{array}{c}\hfill K\hfill \\ \hfill n\hfill \end{array}\right)\right]$$

Using the definition of variance on the right hand term and Eq. (A1) again

$$\begin{array}{cc}\hfill & =\frac{p_n(1-p_n)}{n!}\mathrm{E}{\left[K\right]}^n+p_n^2\left(\mathrm{E}\left[{\left(\begin{array}{c}\hfill K\hfill \\ \hfill n\hfill \end{array}\right)}^2\right]-\mathrm{E}{\left[\left(\begin{array}{c}\hfill K\hfill \\ \hfill n\hfill \end{array}\right)\right]}^2\right)\hfill \\ \hfill & =\frac{p_n(1-p_n)}{n!}\mathrm{E}{\left[K\right]}^n+{\left(\frac{p_n}{n!}\right)}^2\left[\mathrm{E}\left[{\left(K\right)}_n^2\right]-\mathrm{E}{\left[K\right]}^{2n}\right]\hfill \end{array}$$

The product of two falling factorials can be expressed with connection coefficients as,

$$\begin{array}{cc}\hfill {\left(x\right)}_m{\left(x\right)}_n=\sum _{k=0}^m\left(\begin{array}{c}\hfill m\hfill \\ \hfill k\hfill \end{array}\right)& \left(\begin{array}{c}\hfill n\hfill \\ \hfill k\hfill \end{array}\right)k!{\left(x\right)}_{m+n-k}\hfill \\ \hfill & =\frac{p_n(1-p_n)}{n!}\mathrm{E}{\left[K\right]}^n+{\left(\frac{p_n}{n!}\right)}^2\left(\mathrm{E}\left[\sum _{k=0}^n{\left(\begin{array}{c}\hfill n\hfill \\ \hfill k\hfill \end{array}\right)}^{2}k!{\left(K\right)}_{2n-k}\right]-\mathrm{E}{\left[K\right]}^{2n}\right)\hfill \end{array}$$

Applying Eq. (A1) and reindexing the sum yields

$$\mathrm{Var}\left[I\right]=\frac{p_n(1-p_n)}{n!}\mathrm{E}{\left[K\right]}^n+{\left(\frac{p_n}{n!}\right)}^2\left(\sum _{k=1}^n{\left(\begin{array}{c}\hfill n\hfill \\ \hfill k\hfill \end{array}\right)}^{2}k!\mathrm{E}{\left[K\right]}^{2n-k}\right)$$

For n = 2,

$$\mathrm{Var}\left[I\right]=\frac{1}{2}{p}_2\mathrm{E}{\left[K\right]}^2(1+2p_2\mathrm{E}[K\left]\right)$$

For n = 3,

$$\mathrm{Var}\left[I\right]=\frac{1}{12}{p}_3\mathrm{E}{\left[K\right]}^3\left(2+3p_3\mathrm{E}\left[K\right](2+\mathrm{E}[K\left]\right)\right)$$

For n = 4,

$$\mathrm{Var}\left[I\right]=\frac{1}{72}{p}_4\mathrm{E}{\left[K\right]}^4\left(3+p_4\mathrm{E}\left[K\right](12+9\mathrm{E}[K]+2\mathrm{E}{\left[K\right]}^2)\right)$$