On the Evolution of Epistasis III

Abstract

Whether interaction between genes is better represented by synergistic or antagonistic epistasis has been a focus of experimental research in bacterial population genetics. Our previous research on evolution of modifiers of epistasis in diploid systems has indicated that the strength of positive or negative epistasis should increase provided linkage disequilibrium is maintained. Here we study a modifier of epistasis in fitness between two loci in a haploid system. Epistasis is modified in the neighborhood of a mutation selection balance. We show that when linkage in the three-locus system is tight, an increase in the frequency of a modifier allele that induces either more negative or more positive epistasis is possible. Epistasis here can be measured on either an additive or multiplicative scale.

0. Introduction

The evolution of epistasis is a topic largely absent from classical population genetic literature (Hansen et al., 2006). The subject has, however, received increased attention in the past decade as relevant data on haploid genetic systems have become available (Elena and Lenski 1997, 2001; Wilke and Adami 2001; You and Yin 2002; Sanjuan et al. 2004; Bonhoeffer et al. 2004; Burch and Chao 2004; Weinreich and Chao 2005; Shapiro et al. 2006). “Functional” (Hansen and Wagner 2001) or “physiological” (Cheverud and Routman 1995) epistasis refers to non-additive interaction among genes in production of phenotypes (see also Weinreich et al. 2005). This kind of epistasis is not a population property and is independent of allele frequencies. It can be expressed in terms of the parameters used to describe the phenotype of a multilocus individual.

Epistsis may occur at the genotypic level in determining fitness while not being detectable at the level of phenotype, as shown by Lunzer et al. (2005). In this paper we shall be concerned with the genotype-fitness relationship as well as the role of recombination. In Lunzer et al. (2005), the locations of the variants were specified by amino-acids in a single enzyme; recombination was, therefore, not important in their chemostat competition experiments with E. coli.

In theoretical work on the evolution of recombination, the shape of the fitness as a function of the number of deleterious mutations has been shown to be critical (Feldman et al. 1980; Lenski et al. 1999; Otto and Lenormand 2002; Keightley and Otto 2006). With negative epistasis, for example, log fitness declines faster than linearly with increasing mutation number, and under such conditions recombination (or sex) is favored (Eshel and Feldman 1970; Feldman et al. 1980; Kondrashov 1988). Positive epistasis, on the other hand, will put sex at a disadvantage. Thus how the shape of this function evolves is also important. An approximate numerical treatment by Azevedo et al. (2006) concluded that when deleterious mutations can recombine there should be selection towards negative (synergistic) epistasis, but without recombination, positive epistasis should evolve. Understanding how epistasis evolves has broader implications, as epistatic interactions are “central to the evolution of genetic recombination” (Michalakis and Roze 2004). It should be noted, however, that in experiments with the RNA bacteriophage ϕ6, Froissart et al. (2004) were unable to detect any advantage to reassortment even in the presence of negative epistasis.

This manuscript is the third in a series of papers studying the evolution of epistasis. In our first two papers (Liberman and Feldman 2005, 2006) we considered a diploid population, where the character under consideration is determined by two “major” loci with two possible alleles (A and a, B and b) at each locus and symmetric epistatic fitnesses. A third “modifier” locus is introduced, whose sole function is to determine the extent of epistasis in the selection at the major loci. By studying the equilibrium structure of such an epistatic-modifying system, we have shown that under well-specified conditions a modifying allele that increases epistasis succeeds. In other words, genetic interactions tend to become stronger.

In the present manuscript we assume that the population is haploid, rather than diploid. In this case viability selection alone cannot produce a stable polymorphism. Therefore we assume that at the two major loci there is a mutation-selection balance that produces a Hardy-Weinberg (HW) polymorphic equilibrium. As before, the modifier locus affects the extent of epistasis at the major loci. We analyze the external stability of the HW polymorphism towards the invasion of a new allele that modifies the existing epistasis. This model can be compared with our recent haploid analysis in which there are an arbitrary number of loci, each subject to the same rate of mutation to deleterious alleles (Desai et al. 2007). Fitness of each mutant was (1 - s), but the haplotype with k mutations had fitness e^-sk(1+ε), where ε describes the epistasis; ε > 0 entails negative or synergistic epistasis with ε < 0 representing positive or antagonistic epistasis. Antagonistic epistasis was shown to evolve under some simplifying assumptions on the relative rates of mutation and selection that we do not make in the present two-locus analysis.

The paper is organized as follows. Section 1 introduces the two-locus, two-allele haploid selection model. It is verified that with additive selection the mean fitness increases. In Section 2 we introduce independent unidirectional deleterious mutations at each locus (from A → a, B → b, each at rate μ) and describe the possible mutation-selection equilibria. In Section 3 a third modifier locus that controls epistasis is introduced and its dynamics explicated. In Section 4 the existence of a class of HW equilibria is revealed, in Section 5 we review the results, and in Sections 6, 7, and 8 we analyze the external stability of these HW polymorphic equilibria, thus determining the fate of a modifier allele that changes the extent of additive, or multiplicative, epistasis at the major loci. Detailed proofs of the main results are given in supplemental online material (SOM), sections A-E.

1. Two-locus haploid selection models

Consider a large population of haploid individuals and a character determined by two diallelic loci. Let A, a be the possible alleles at the first locus and B, b those at the second locus. Generations are non-overlapping, and the life cycle consists of random mating of gametes, recombination (in a diploid phase), Mendelian segregation, and viability selection in the haploid phase, in that order. We use the following notation:

$$\begin{array}{ccccc}\hfill \text{Gamete}:\hfill & \hfill AB\hfill & \hfill Ab\hfill & \hfill aB\hfill & \hfill ab\hfill \\ \hfill \text{Frequency}:\hfill & \hfill x_1\hfill & \hfill x_2\hfill & \hfill x_3\hfill & \hfill x_4\hfill \\ \hfill \text{Viability}:\hfill & \hfill w_1\hfill & \hfill w_2\hfill & \hfill w_3\hfill & \hfill w_4\hfill \end{array}$$ (1.1)

Let r (0 < r < 1) be the recombination rate between the two loci. The frequencies $x_1^{\prime },x_2^{\prime },x_3^{\prime },x_4^{\prime }$ of the four gametes in the next generation are given by

$$\begin{array}{c}\hfill w{x}_1^{\prime }=w_1(x_1-rD)=w_1{\widehat{x}}_1\hfill \\ \hfill w{x}_2^{\prime }=w_2(x_2+rD)=w_2{\widehat{x}}_2\hfill \\ \hfill w{x}_3^{\prime }=w_3(x_3+rD)=w_3{\widehat{x}}_3\hfill \\ \hfill w{x}_4^{\prime }=w_4(x_4-rD)=w_4{\widehat{x}}_4\hfill \end{array}$$ (1.2)

where D = x₁x₄ - x₂x₃ is the linkage disequilibrium and w, the normalizing factor, is

$$w=\sum _{i=1}^4{w}_i{\widehat{x}}_i=\sum _{i=1}^4{w}_i{x}_i-rD(w_1-w_2-w_3+w_4).$$ (1.3)

Unless mentioned to the contrary, it will be assumed that w_i ≠ w_j for i ≠ j.

The expression

$$e_{\mathrm{add}}=w_1-w_2-w_3+w_4$$ (1.4)

is the usual measure of additive epistasis associated with the fitness parameters w₁, w₂, w₃, w₄. Similarly

$$e_{\mathrm{mult}}=w_1{w}_4-w_2{w}_3$$ (1.5)

is the measure of multiplicative epistasis.

Our interest centers on the evolution of epistasis. Therefore we would like to know, as a baseline, how a population evolves under non-epistatic selection, either additive or multiplicative, that is where e_add = 0 or e_mult = 0, respectively. We have the following two-part result.

Result 1

1. With additively non-epistatic selection, the mean fitness increases. That is

   $$\sum _{i=1}^4{w}_i{x}_i^{\prime }\ge \sum _{i=1}^4{w}_i{x}_i$$ (1.6)

   with equality if and only if there is fixation in one of the four gametes. Moreover there is global convergence to fixation in the gamete with maximum viability.

2. With multiplicatively non-epistatic selection, the only possible equilibria are the four fixations. Moreover the fixation in the gamete with maximum viability is the only locally stable equilibrium.

The proof is given in SOM, section A.

In view of Result 1, non-epistatic haploid selection does not allow polymorphic equilibrium. A possible source of polymorphism would be mutation, giving rise to a mutation-selection balance. Suppose, therefore, that in addition to haploid selection we have at each of the two loci independent unidirectional mutation A → a and B → b. Let μ be the mutation rate, assumed to be the same at each locus. Then using the previous notation, the transformation of frequencies is given by

$$\begin{array}{c}w{x}_1^{\prime }={(1-\mu )}^2{w}_1(x_1-rD)\hfill \\ w{x}_2^{\prime }=\mu (1-\mu )w_1(x_1-rD)+(1-\mu )w_2(x_2+rD)\hfill \\ w{x}_3^{\prime }=\mu (1-\mu )w_1(x_1-rD)+(1-\mu )w_3(x_3+rD)\hfill \\ w{x}_4^{\prime }={\mu }^2{w}_1(x_1-rD)+\mu w_2(x_2+rD)+\mu w_3(x_3+rD)+w_4(x_4-rD),\hfill \end{array}$$ (1.7)

where $w={\sum }_{i=1}^4{w}_i{\widehat{x}}_i={\sum }_{i=1}^4{w}_i{x}_i-rD{e}_{\mathrm{add}}$ is the normalizing factor.

For the mutation-selection balance, suppose that w₁ > w₂, w₃ > w₄ so that AB is the gamete with the maximum fitness and ab the gamete with minimum fitness. These inequalitites will be assumed throughout the paper. Under these assumptions, the equilibrium x* = (1, 0, 0, 0), corresponding to fixation in AB, is stable. Let T_μ be the transformation of frequencies given in (1.7), then T_μx* = ((1 - μ)², μ(1 - μ), μ(1 - μ), μ²), and when 0 < μ < 1, T_μx* involves all four gametes with positive frequencies. Therefore from small perturbation theory (e.g., Karlin and McGregor 1971, 1972) we know that for small μ there is a locally stable polymorphism near x* = (1, 0, 0, 0). Thus in the haploid selection model with unidirectional mutation, a stable mutation-selection balance exists. Of course, the cases of non-epistatic selection are included in this result.

Although it is difficult to specify the equilibrium points of T_μ explicitly, some classifications are possible. Direct computation produces the following result (see SOM, section B).

Result 2

The haploid selection-mutation model (1.7) has a polymorphic equlibrium with linkage equilibrium D = 0 if and only if selection is multiplicatively non-epistatic, in which case the mutation-selection equilibrium ${\mathbf{x}}^{\ast }=(x_1^{\ast },x_2^{\ast },x_3^{\ast },x_4^{\ast })$ is unique. It exists when

$$(1-\mu )w_1>w_2,(1-\mu )w_1>w_3,{(1-\mu )}^2{w}_1>w_4,$$ (1.8)

and it satisfies

$$\begin{array}{c}{x}_2^{\ast }=\frac{\mu w_1{x}_1^{\ast }}{(1-\mu )w_1-w_2},x_3^{\ast }=\frac{\mu w_1{x}_1^{\ast }}{(1-\mu )w_1-w_3},\hfill \\ x_4^{\ast }=\frac{{\mu }^2{w}_1{x}_1^{\ast }}{{(1-\mu )}^2{w}_1-w_4}\left[1+\frac{w_2}{(1-\mu )w_1-w_2}+\frac{w_3}{(1-\mu )w_1-w_3}\right].\hfill \end{array}$$ (1.9)

Remark 1

Result 2 entails that when there is multiplicative epistasis (e_mult ≠ 0), equilibria with D = 0 must be boundary equilibria with x₁ = 0 and either x₂ = 0 or x₃ = 0. In fact three boundary equilibria with D = 0 are possible: (0, 0, 0, 1), $(0,0,{\tilde{x}}_3,{\tilde{x}}_4)$, $(0,{\tilde{\tilde{x}}}_2,0,{\tilde{\tilde{x}}}_4)$ where

$${\tilde{x}}_3=\frac{(1-\mu )w_3-w_4}{w_3-w_4},{\tilde{x}}_4=\frac{\mu w_3}{w_3-w_4},$$ (1.10)

and

$${\tilde{\tilde{x}}}_2=\frac{(1-\mu )w_2-w_4}{w_2-w_4},{\tilde{\tilde{x}}}_4=\frac{\mu w_2}{w_2-w_4}.$$ (1.11)

As w₁ > w₂, w₃ > w₄, $(0,0,{\tilde{x}}_3,{\tilde{x}}_4)$ exists if (1 - μ)w₃ > w₄ and $(0,{\tilde{\tilde{x}}}_2,0,{\tilde{\tilde{x}}}_4)$ exists if (1 - μ)w₂ > w₄. Of course these equilibria may also exist when there is no multiplicative epistasis (e_mult = 0).

When a mutation-selection equilibrium exists for the recursion system (1.7), we have (see SOM, section B) the following result.

Result 3

At mutation-selection balance from (1.7), we must have

$$w>(1-\mu )w_2,w>(1-\mu )w_3.$$ (1.12)

If w₄ < w₂, w₃ < w₁, then

$$w>w_4.$$ (1.13)

The sign of the linkage disequilibrium at a polymorphic equilibrium depends on the fitness parameters, as follows (see SOM, section B).

Result 4

(Felsenstein 1965, Feldman et al. 1980.) In the haploid mutation-selection model (1.7) at any interior equilibrium x = (x₁,x₂,x₃,x₄) with x_i ≠ 0 for all i the linkage disequilibrium D = x₁ x₄ - x₂ x₃ satisfies

$$D(w_1{w}_4-w_2{w}_3)>0.$$ (1.14)

with D = 0 if and only if w₁ w₄ = w₂ w₃.

Thus the sign of D, at any polymorphic equilibrium, coincides with the sign of the multiplicative epistasis e_mult.

2. The epistasis modification model

Suppose that in addition to the two “major” loci there is a third modifier locus, with n possible alleles M₁, M₂,..., M_n, whose sole function is to control epistasis among the haploid fitnesses at the two major loci. Let the frequencies of the four gametes AB, Ab, aB, ab at the major loci, in the present generation, be x₁, x₂, x₃, x₄, respectively. There is no mutation at the modifier locus. Then in this three-locus framework we can write

$$\begin{array}{ccccc}\hfill \text{Gamete}:\hfill & \hfill AB{M}_i\hfill & \hfill Ab{M}_i\hfill & \hfill aB{M}_i\hfill & \hfill ab{M}_i\hfill \\ \hfill \text{Fitness}:\hfill & \hfill w_1^i\hfill & \hfill w_2^i\hfill & \hfill w_3^i\hfill & \hfill w_4^i\hfill \\ \hfill \text{Frequency}:\hfill & \hfill x_1{p}_i\hfill & \hfill x_2{q}_i\hfill & \hfill x_3{r}_i\hfill & \hfill x_4{s}_i\hfill \end{array}$$ (2.1)

where 0 ≤ p_i, q_i, r_i, s_i ≤ 1 and ${\sum }_i{p}_i={\sum }_i{q}_i={\sum }_i{r}_i={\sum }_i{s}_i=1$. The population state in the present generation is therefore specified by the five vectors x, p, q, r, s where:

$$\begin{array}{c}\mathbf{x}=(x_1,x_2,x_3,x_4)\hfill \\ \mathbf{p}=(p_1,p_2,\dots ,p_n)\hfill \\ \mathbf{q}=(q_1,q_2,\dots ,q_n)\hfill \\ \mathbf{r}=(r_1,r_2,\dots ,r_n)\hfill \\ \mathbf{s}=(s_1,s_2,\dots ,s_n).\hfill \end{array}$$ (2.2)

The three-locus recombination distribution is determined by the four crossover probabilities

$$c_{00},c_{01},c_{10},c_{11},$$ (2.3)

where c₀₀ + c₀₁ + c₁₀ + c₁₁ = 1. Here c₀₀ is the probability that there is no recombination between A/a and B/b nor between B/b and M_i/M_j. Similarly c₀₁ is the probability that no recombination occurs between A/a and B/b but recombination occurs beween B/b and M_i/M_j. There is a corresponding interpretation for c₁₀, while c₁₁ is the probability that recombination occurs both between A/a and B/b and between B/b and M_i/M_j. With this representation

$$r=c_{10}+c_{11},R=c_{01}+c_{11},$$ (2.4)

where r is the recombination rate between the two major loci and R is the frequency of recombination between the modifier locus and the two major loci. As in the previous section we assume that μ is the rate of mutation A → a, B → b at each locus and that recombination events are not affected by the modifier locus.

We observe the population at the gametic stage and in the course of one generation the population undergoes random mating, recombination, Mendelian segregation, haploid selection, and mutation, in that order. For ease of notation we represent the transformation of frequencies before mutation. Starting with the population state x, p, q, r, s we have:

$$\begin{array}{ccccc}\hfill \text{Gamete}:\hfill & \hfill AB{M}_i\hfill & \hfill Ab{M}_i\hfill & \hfill aB{M}_i\hfill & \hfill ab{M}_i\hfill \\ \hfill {}^{″}{\text{New}}^{″}\text{frequency after selection and before mutation}:\hfill & \hfill {\tilde{x}}_1{\tilde{p}}_i\hfill & \hfill {\tilde{x}}_2{\tilde{q}}_i\hfill & \hfill {\tilde{x}}_3{\tilde{r}}_i\hfill & \hfill {\tilde{x}}_4{\tilde{s}}_i\hfill \\ \hfill {}^{″}{\text{New}}^{″}\text{frequency after mutation}:\hfill & \hfill x_1^{\prime }{p}_i^{\prime }\hfill & \hfill x_2^{\prime }{q}_i^{\prime }\hfill & \hfill x_3^{\prime }{r}_i^{\prime }\hfill & \hfill x_4^{\prime }{s}_i^{\prime }.\hfill \end{array}$$ (2.5)

For the “new” frequencies before mutation we can use the transformation of frequencies described in Liberman and Feldman (1986). Following that transformation and using the vector notation

$${\mathbf{w}}_k=(w_k^1,w_k^2,\dots ,w_k^n)\text{for}k=1,2,3,4$$ (2.6)

we have:

$$\begin{array}{cc}\hfill w{\tilde{x}}_1\tilde{\mathbf{p}}=& x_1^2\mathbf{p}\circ {\mathbf{w}}_1+x_1{x}_2[(c_{00}+c_{10})\mathbf{p}\circ {\mathbf{w}}_1+(c_{01}+c_{11})\mathbf{q}\circ {\mathbf{w}}_1]\hfill \\ \hfill & +x_1{x}_3[(c_{00}+c_{11})\mathbf{p}\circ {\mathbf{w}}_1+(c_{01}+c_{10})\mathbf{r}\circ {\mathbf{w}}_1]\hfill \\ \hfill & +x_1{x}_4[c_{00}\mathbf{p}\circ {\mathbf{w}}_1+c_{01}\mathbf{s}\circ {\mathbf{w}}_1]\hfill \\ \hfill & +x_2{x}_3[c_{11}\mathbf{q}\circ {\mathbf{w}}_1+c_{10}\mathbf{r}\circ {\mathbf{w}}_1]\hfill \\ \hfill w{\tilde{x}}_2\tilde{\mathbf{q}}=& x_2^2\mathbf{q}\circ {\mathbf{w}}_2+x_1{x}_2[(c_{01}+c_{11})\mathbf{p}\circ {\mathbf{w}}_2+(c_{00}+c_{10})\mathbf{q}\circ {\mathbf{w}}_2]\hfill \\ \hfill & +x_2{x}_4[(c_{00}+c_{11})\mathbf{q}\circ {\mathbf{w}}_2+(c_{01}+c_{10})\mathbf{s}\circ {\mathbf{w}}_2]\hfill \\ \hfill & +x_1{x}_4[c_{11}\mathbf{p}\circ {\mathbf{w}}_2+c_{10}\mathbf{s}\circ {\mathbf{w}}_2]\hfill \\ \hfill & +x_2{x}_3[c_{00}\mathbf{q}\circ {\mathbf{w}}_2+c_{01}\mathbf{r}\circ {\mathbf{w}}_2]\hfill \\ \hfill w{\tilde{x}}_3\tilde{\mathbf{r}}=& x_3^2\mathbf{r}\circ {\mathbf{w}}_3+x_1{x}_3[(c_{01}+c_{10})\mathbf{p}\circ {\mathbf{w}}_3+(c_{00}+c_{11})\mathbf{r}\circ {\mathbf{w}}_3]\hfill \\ \hfill & +x_3{x}_4[(c_{00}+c_{10})\mathbf{r}\circ {\mathbf{w}}_3+(c_{01}+c_{11})\mathbf{s}\circ {\mathbf{w}}_3]\hfill \\ \hfill & +x_1{x}_4[c_{10}\mathbf{p}\circ {\mathbf{w}}_3+c_{11}\mathbf{s}\circ {\mathbf{w}}_3]\hfill \\ \hfill & +x_2{x}_3[c_{00}\mathbf{r}\circ {\mathbf{w}}_3+c_{01}\mathbf{q}\circ {\mathbf{w}}_3]\hfill \\ \hfill w{\tilde{x}}_4\tilde{\mathbf{s}}=& x_4^2\mathbf{s}\circ {\mathbf{w}}_4+x_2{x}_4[(c_{00}+c_{11})\mathbf{s}\circ {\mathbf{w}}_4+(c_{01}+c_{10})\mathbf{q}\circ {\mathbf{w}}_4]\hfill \\ \hfill & +x_3{x}_4[(c_{00}+c_{10})\mathbf{s}\circ {\mathbf{w}}_4+(c_{01}+c_{11})\mathbf{r}\circ {\mathbf{w}}_4]\hfill \\ \hfill & +x_1{x}_4[c_{01}\mathbf{p}\circ {\mathbf{w}}_4+c_{00}\mathbf{s}\circ {\mathbf{w}}_4]\hfill \\ \hfill & +x_2{x}_3[c_{10}\mathbf{q}\circ {\mathbf{w}}_4+c_{11}\mathbf{r}\circ {\mathbf{w}}_4].\hfill \end{array}$$ (2.7)

Here ∘ is the Schur product of vectors, namely, if a = (a₁, a₂, ..., a_n) and b = (b₁, b₂, ..., b_n) then

$$\mathbf{a}\circ \mathbf{b}=(a_1{b}_1,a_2{b}_2,\dots ,a_n{b}_n),$$ (2.8)

and w is a normalizing factor such that

$${\tilde{x}}_1\sum _{i=1}^n{\tilde{p}}_i+{\tilde{x}}_2\sum _{i=1}^n{\tilde{q}}_i+{\tilde{x}}_3\sum _{i=1}^n{\tilde{r}}_i+{\tilde{x}}_4\sum _{i=1}^n{\tilde{s}}_i=1.$$ (2.9)

The new frequencies after mutation are

$$\begin{array}{c}w{x}_1^{\prime }{\mathbf{p}}^{\prime }={(1-\mu )}^{2}w{\tilde{x}}_1\tilde{\mathbf{p}}\hfill \\ w{x}_2^{\prime }{\mathbf{q}}^{\prime }=\mu (1-\mu )w{\tilde{x}}_1\tilde{\mathbf{p}}+(1-\mu )w{\tilde{x}}_2\tilde{\mathbf{q}}\hfill \\ w{x}_3^{\prime }{\mathbf{r}}^{\prime }=\mu (1-\mu )w{\tilde{x}}_1\tilde{\mathbf{p}}+(1-\mu )w{\tilde{x}}_3\tilde{\mathbf{r}}\hfill \\ w{x}_4^{\prime }{\mathbf{s}}^{\prime }={\mu }^{2}w{\tilde{x}}_1\tilde{\mathbf{p}}+\mu w{\tilde{x}}_2\tilde{\mathbf{q}}+\mu w{\tilde{x}}_3\tilde{\mathbf{r}}+w{\tilde{x}}_4\tilde{\mathbf{s}},\hfill \end{array}$$ (2.10)

where w is the same normalizing factor as in (2.7).

3. Existence of Hardy-Weinberg equilibria

A special class of equilibria of the system (2.10) are the Hardy-Weinberg (HW) equilibria, where the modifier locus is in linkage equilibrium with the major loci. The three-locus gametic frequencies at these equilibria are given by a Kronecker product vector x ⊗ y, where

$$\begin{array}{c}\hfill \mathbf{x}=(x_1,x_2,x_3,x_4),\mathbf{y}=(y_1,y_2,\dots ,y_n),\hfill \\ \hfill \text{and}x\otimes y=(x_1{y}_1,x_1{y}_2\dots x_1{y}_n\dots x_4{y}_1\dots x_4{y}_n).\hfill \end{array}$$ (3.1)

Thus, the frequencies of the 4n three-locus gametes are

$$\begin{array}{ccccc}\hfill \text{Gamete}:\hfill & \hfill AB{M}_i\hfill & \hfill Ab{M}_i\hfill & \hfill aB{M}_i\hfill & \hfill ab{M}_i\hfill \\ \hfill \text{Frequency}:\hfill & \hfill x_1{y}_i\hfill & \hfill x_2{y}_i\hfill & \hfill x_3{y}_i\hfill & \hfill x_4{y}_i.\hfill \end{array}$$ (3.2)

Comparing (3.2) with (2.1), at the HW equilibria we have p = q = r = s = y. Using the transformation equations (2.7) and (2.10), at equilibrium we have, e.g.

$$\begin{array}{cc}\hfill w{x}_1\mathbf{y}=& {(1-\mu )}^2\mathbf{y}\circ {\mathbf{w}}_1[x_1^2+x_1{x}_2+x_1{x}_3+x_1{x}_4(c_{00}+c_{01})+x_2{x}_3(c_{10}+c_{11})]\hfill \\ \hfill =& {(1-\mu )}^2\mathbf{y}\circ {\mathbf{w}}_1[x_1-rD].\hfill \end{array}$$ (3.3)

Thus, using (2.4), the four equilibrium equations become

$$\begin{array}{c}w{x}_1\mathbf{y}={(1-\mu )}^2\mathbf{y}\circ {\mathbf{w}}_1[x_1-rD]\hfill \\ w{x}_2\mathbf{y}=\mu (1-\mu )\mathbf{y}\circ {\mathbf{w}}_1[x_1-rD]+(1-\mu )\mathbf{y}\circ {\mathbf{w}}_2[x_2+rD]\hfill \\ w{x}_3\mathbf{y}=\mu (1-\mu )\mathbf{y}\circ {\mathbf{w}}_1[x_1-rD]+(1-\mu )\mathbf{y}\circ {\mathbf{w}}_3[x_3+rD]\hfill \\ w{x}_4\mathbf{y}={\mu }^2\mathbf{y}\circ {\mathbf{w}}_1[x_1-rD]+\mu \mathbf{y}\circ {\mathbf{w}}_2[x_2+rD]+\mu \mathbf{y}\circ {\mathbf{w}}_3[x_3+rD]+\mathbf{y}\circ {\mathbf{w}}_4[x_4-rD]\hfill \end{array}$$ (3.4)

Summing all components for each of the equations in (3.4) using ${\sum }_{i=1}^n{y}_i=1$, we find

$$\begin{array}{c}w{x}_1={(1-\mu )}^2(\mathbf{y},{\mathbf{w}}_1)[x_1-rD]\hfill \\ w{x}_2=\mu (1-\mu )(\mathbf{y},{\mathbf{w}}_1)[x_1-rD]+(1-\mu )(\mathbf{y},{\mathbf{w}}_2)[x_2+rD]\hfill \\ w{x}_3=\mu (1-\mu )(\mathbf{y},{\mathbf{w}}_1)[x_1-rD]+(1-\mu )(\mathbf{y},{\mathbf{w}}_3)[x_3+rD]\hfill \\ w{x}_4={\mu }^2(\mathbf{y},{\mathbf{w}}_1)[x_1-rD]+\mu (\mathbf{y},{\mathbf{w}}_2)[x_2+rD]+\mu (\mathbf{y},{\mathbf{w}}_3)[x_3+rD]+(\mathbf{y},{\mathbf{w}}_4)[x_4-rD],\hfill \end{array}$$ (3.5)

where (a, b) is the scalar product of the two vectors a = (a₁,..., a_n) and b = (b₁,..., b_n), namely

$$(\mathbf{a},\mathbf{b})=\sum _{i=1}^n{a}_i{b}_i.$$ (3.6)

Observe that equations (3.5) are equivalent to the equilibrium equations at the major loci resulting from (1.7) with associated fitness parameters

$$\begin{array}{ccccc}\text{Gamete}:\hfill & \hfill AB\hfill & \hfill Ab\hfill & \hfill aB\hfill & \hfill ab\hfill \\ \text{Fitness}:\hfill & \hfill (\mathbf{y},{\mathbf{w}}_1)\hfill & \hfill (\mathbf{y},{\mathbf{w}}_2)\hfill & \hfill (\mathbf{y},{\mathbf{w}}_3)\hfill & \hfill (\mathbf{y},{\mathbf{w}}_4)\hfill \end{array}$$ (3.7)

where

$$(\mathbf{y},{\mathbf{w}}_k)=\sum _{i=1}^n{w}_k^i{y}_i$$

is the average fitness of the k-th gamete with weights given by y = (y₁, y₂ ... y_n).

Now substituting (3.5) into (3.4) gives for the first equation

$${(1-\mu )}^2(\mathbf{y},{\mathbf{w}}_1)[x_1-rD]\mathbf{y}={(1-\mu )}^2\mathbf{y}\circ {\mathbf{w}}_1[x_1-rD],$$ (3.8)

and since 0 < μ < 1 we have

$$[x_1-rD][(\mathbf{y},{\mathbf{w}}_1)\mathbf{y}-\mathbf{y}\circ {\mathbf{w}}_1]=0.$$ (3.9)

Similarly, for the second equation we have

$$\mu (1-\mu )[x_1-rD][(\mathbf{y},{\mathbf{w}}_1)\mathbf{y}-\mathbf{y}\circ {\mathbf{w}}_1]+(1-\mu )[x_2+rD][(\mathbf{y},{\mathbf{w}}_2)\mathbf{y}-\mathbf{y}\circ {\mathbf{w}}_2]=0.$$ (3.10)

Using (3.9) and 0 < μ < 1, (3.10) gives

$$[x_2+rD][(\mathbf{y},{\mathbf{w}}_2)\mathbf{y}-\mathbf{y}\circ {\mathbf{w}}_2]=0.$$ (3.11)

In general we derive the following four equations at equilibrium:

$$\begin{array}{c}\hfill [x_1-rD][(\mathbf{y},{\mathbf{w}}_1)\mathbf{y}-\mathbf{y}\circ {\mathbf{w}}_1]=0\hfill \\ \hfill [x_2+rD][(\mathbf{y},{\mathbf{w}}_2)\mathbf{y}-\mathbf{y}\circ {\mathbf{w}}_2]=0\hfill \\ \hfill [x_3+rD][(\mathbf{y},{\mathbf{w}}_3)\mathbf{y}-\mathbf{y}\circ {\mathbf{w}}_3]=0\hfill \\ \hfill [x_4-rD][(\mathbf{y},{\mathbf{w}}_4)\mathbf{y}-\mathbf{y}\circ {\mathbf{w}}_4]=0.\hfill \end{array}$$ (3.12)

We assume that at the major loci we have an interior equilibrium, which entails that x_k ∓ rD ≠ 0 (± depending on k) for all k. We conclude that a HW equilibrium exists if and only if for each k = 1, 2, 3, 4

$$(\mathbf{y},{\mathbf{w}}_k)\mathbf{y}=\mathbf{y}\circ {\mathbf{w}}_k.$$ (3.13)

Observe that the equilibrium equations (3.13) are exactly the one-locus multiallellic equilibrium equations with n alleles M₁, M₂, ..., M_n and haploid fitness parameters $w_k^1,w_k^2,\dots ,w_k^n$. We have called these HW equilibria, when they exist, viability analogous HW (VAHW) equilibria (Liberman and Feldman, 1986). We summarize these findings in the following:

Result 5

If a frequency vector y exists such that (y,w_k)y=y∘w_k for k = 1, 2, 3, 4, and x = (x₁,x₂,x₃,x₄), the frequencies vector of AB, Ab, ab, and ab, satisfies the two-locus mutation-selection equilibrium equations (3.5) with fitness parameters (y, w_k) for k = 1, 2, 3, 4, then x ⊗ y is a VAHW equilibrium of the three-locus system (2.10).

Remark 5

The VA equations (3.13) are of the form

$$(\mathbf{w},\mathbf{z})\mathbf{z}=\mathbf{w}\circ \mathbf{z}$$ (3.14)

where w = (w₁, w₂,..., w_n) and z = (z₁, z₂,..., z_n). Observe that if $\stackrel{‒}{w}=(\mathbf{w},\mathbf{z})$, then

$$\stackrel{‒}{w}{z}_i=w_i{z}_{i}i=1,2,\dots ,n.$$ (3.15)

Hence, unless w₁ = w₂ = ... = w_n, at least one of z_i-s must be zero. In fact, if I ⊂ {1, 2,..., n} is the set such that w_i = w_j for i, j ∈ I, then any solution z^I of (3.14) corresponding to I has

$$z_i^I=0\text{for}i\notin I,z_i^I\ge 0\text{for}i\in I\text{with}\sum _{i\in I}{z}_i^I=1.$$ (3.16)

Therefore, unless there are special relations among the vectors w₁, w₂, w₃, w₄, the only solutions of the four equations (3.13) are the n fixations in any one of the n alleles M₁, M₂,..., M_n at the modifier locus. On the other hand, if, e.g., w_k = w_ke, where e = (1, 1,..., 1) for all k = 1, 2, 3, 4, such that the fitnesses of the four gametes AB, Ab, aB, ab at the major loci are the same for each allele at the modifier locus, then any vector y with non-negative components and ${\sum }_{i=1}^n{y}_i=1$ is a solution of the four equations (3.13).

4. External stability of a VAHW equilibrium

Let x ⊗ y be a VAHW equilibrium where y = (y₁, y₂,..., y_n) satisfies the equilibrium equations (3.14) and x solves the equations (3.5) with fitness parameters

$$w_k=(\mathbf{y},{\mathbf{w}}_k)k=1,2,3,4.$$ (4.1)

Following Remark 5, only one of y₁, y₂,..., y_n is non-zero. Suppose now that a new allele M_i, where i is not one of the indices 1, 2,..., n, is introduced at the modifier locus, with new fitness parameters

$$w_1^i,w_2^i,w_3^i,w_4^i.$$ (4.2)

Mutation does not occur at the modifier locus after the initial appearance of M_i. The linear approximation for p_i, q_i, r_i, s_i, the frequencies of the new rare allele M_i among the four gametes AB, Ab, aB, ab, respectively, is derived from (2.7) and (2.10) and reduces to the following equations:

$$\begin{array}{cc}\hfill w{x}_1{p}_i^{\prime }={(1-\mu )}^2{w}_1^i& \{x_1^2{p}_i+x_1{x}_2[(c_{00}+c_{10})p_i+(c_{01}+c_{11})q_i]\hfill \\ \hfill & +x_1{x}_3[(c_{00}+c_{11})p_i+(c_{01}+c_{10})r_i]\hfill \\ \hfill & +x_1{x}_4[c_{00}{p}_i+c_{01}{s}_i]\hfill \\ \hfill & +x_2{x}_3[c_{11}{q}_i+c_{10}{r}_i]\}\hfill \\ \hfill w{x}_2{q}_i^{\prime }=\mu (1-\mu )w_1^i& \{x_1^2{p}_i+x_1{x}_2[(c_{00}+c_{10})p_i+(c_{01}+c_{11})q_i]\hfill \\ \hfill & +x_1{x}_3[(c_{00}+c_{11})p_i+(c_{01}+c_{10})r_i]\hfill \\ \hfill & +x_1{x}_4[c_{00}{p}_i+c_{01}{s}_i]\hfill \\ \hfill & +x_2{x}_3[c_{11}{q}_i+c_{10}{r}_i]\}\hfill \\ \hfill +(1-\mu )w_2^i& \{x_2^2{q}_i+x_1{x}_2[(c_{01}+c_{11})p_i+(c_{00}+c_{10})q_i]\hfill \\ \hfill & +x_2{x}_4[(c_{00}+c_{11})q_i+(c_{01}+c_{10})s_i]\hfill \\ \hfill & +x_1{x}_4[c_{11}{p}_i+c_{10}{s}_i]\hfill \\ \hfill & +x_2{x}_3[c_{00}{q}_i+c_{01}{r}_i]\hfill \\ \hfill w{x}_3{r}_i^{\prime }=\mu (1-\mu )w_1^i& \{x_1^2{p}_i+x_1{x}_2[(c_{00}+c_{10})p_i+(c_{01}+c_{11})q_i]\hfill \\ \hfill & +x_1{x}_3[(c_{00}+c_{11})p_i+(c_{01}+c_{10})r_i]\hfill \\ \hfill & +x_1{x}_4[c_{00}{p}_i+c_{01}{s}_i]\hfill \\ \hfill & +x_2{x}_3[c_{11}{q}_i+c_{10}{r}_i]\}\hfill \\ \hfill +(1-\mu )w_3^i& \{x_3^2{r}_i+x_1{x}_3[(c_{01}+c_{10})p_i+(c_{00}+c_{11})r_i]\hfill \\ \hfill & +x_3{x}_4[(c_{00}+c_{10})r_i+(c_{01}+c_{11})s_i]\hfill \\ \hfill & +x_1{x}_4[c_{10}{p}_i+c_{11}{s}_i]\hfill \\ \hfill & +x_2{x}_3[c_{00}{r}_i+c_{01}{q}_i]\hfill \\ \hfill w{x}_4{s}_i^{\prime }={\mu }^2{w}_1^i& \{x_1^2{p}_i+x_1{x}_2[(c_{00}+c_{10})p_i+(c_{01}+c_{11})q_i]\hfill \\ \hfill & +x_1{x}_3[(c_{00}+c_{11})p_i+(c_{01}+c_{10})r_i]\hfill \\ \hfill & +x_1{x}_4[c_{00}{p}_i+c_{01}{s}_i]\hfill \\ \hfill & +x_2{x}_3[c_{11}{q}_i+c_{10}{r}_i]\}\hfill \\ \hfill +{\mu }^2{w}_2^i& \{x_2^2{q}_i+x_1{x}_2[(c_{01}+c_{11})p_i+(c_{00}+c_{10})q_i]\hfill \\ \hfill & +x_2{x}_4[(c_{00}+c_{11})q_i+(c_{01}+c_{10})s_i]\hfill \\ \hfill & +x_1{x}_4[c_{11}{p}_i+c_{10}{s}_i]\hfill \\ \hfill & +x_2{x}_3[c_{00}{q}_i+c_{01}{r}_i]\}\hfill \\ \hfill +\mu w_3^i& \{x_3^2{r}_i+x_1{x}_3[(c_{01}+c_{10})p_i+(c_{00}+c_{11})r_i]\hfill \\ \hfill & +x_3{x}_4[(c_{00}+c_{10})r_i+(c_{01}+c_{11})s_i]\hfill \\ \hfill & +x_1{x}_4[c_{10}{p}_i+c_{11}{s}_i]\hfill \\ \hfill & +x_2{x}_3[c_{00}{r}_i+c_{01}{q}_i]\}\hfill \\ \hfill +w_4^i& \{x_4^2{s}_i+x_2{x}_4[(c_{00}+c_{11})s_i+(c_{01}+c_{10})q_i]\hfill \\ \hfill & +x_3{x}_4[(c_{00}+c_{10})s_i+(c_{01}+c_{11})r_i]\hfill \\ \hfill & +x_1{x}_4[c_{01}{p}_i+c_{00}{s}_i]\hfill \\ \hfill & +x_2{x}_3[c_{10}{q}_i+c_{11}{r}_i]\}.\hfill \end{array}$$ (4.3)

Let L be the matrix corresponding to the linear approximation (4.3) and write

$$\left[\begin{array}{c}\hfill p_i^{\prime }\hfill \\ \hfill q_i^{\prime }\hfill \\ \hfill r_i^{\prime }\hfill \\ \hfill s_i^{\prime }\hfill \end{array}\right]=\mathbf{L}\left[\begin{array}{c}\hfill p_i\hfill \\ \hfill q_i\hfill \\ \hfill r_i\hfill \\ \hfill s_i\hfill \end{array}\right].$$ (4.4)

Then the external stability of x ⊗ y is determined by the largest root in absolute value of the fourth degree polynomial Q(z), where

$$Q\left(z\right)=\det [\mathbf{L}-z\mathbf{I}].$$ (4.5)

As the leading coefficient (of z⁴) in Q(z) is 1, the value of Q(z) as z → ∞ is positive. Here I is the 4 × 4 identity matrix. We shall be especially interested in (4.5) at z = 1, namely Q(1), since if Q(1) < 0, then Q(z) has at least one positive root greater than 1.

We are interested in the influence of the sign and amount of epistasis introduced by the new allele M_i on the external stability of the VAHW equilibrium x ⊗ y. We will assume that M_i determines the associated fitness parameters in a linear way, namely

$$w_k^i=w_k+w_k{\delta }_k{\epsilon }_{i}k=1,2,3,4,$$ (4.6)

where the δ_k-s can be either positive or negative, and ε_i > 0. Under this assumption Q(1) is given by the following.

Result 6

Under the linear selection modification assumption (4.6) for small ε_i, and up to non-linear terms in ε_i,

$$Q\left(1\right)=\frac{{(1-\mu )}^2{w}_1}{w^4{x}_1}{\epsilon }_i\det \left(\mathbf{N}\right)$$ (4.7)

where N is the 4 × 4 matrix given by

$$\begin{array}{c}{n}_{11}=-x_1{c}_{01}\hfill \\ n_{12}=(x_1+x_3)c_{11}\hfill \\ n_{13}=(x_1+x_2)c_{10}\hfill \\ n_{14}={\delta }_1(x_1-rD)\hfill \\ n_{21}=-(1-\mu )w_2(x_1+x_2)(c_{01}+c_{10})\hfill \\ n_{22}=(1-\mu )w_2(c_{00}+c_{11})-w\hfill \\ n_{23}=0\hfill \\ n_{24}=(1-\mu )w_2[{\delta }_1(x_1-rD)+{\delta }_2(x_2+rD)]\hfill \\ n_{31}=-(1-\mu )w_3(x_1+x_3)(c_{01}+c_{11})\hfill \\ n_{32}=0\hfill \\ n_{33}=(1-\mu )w_3(c_{00}+c_{10})-w\hfill \\ n_{34}=(1-\mu )w_3[{\delta }_1(x_1-rD)+{\delta }_3(x_3+rD)]\hfill \\ n_{41}=-\mu w_2(x_1+x_2)(c_{01}+c_{10})-\mu w_3(x_1+x_3)(c_{01}+c_{11})-w_4[x_4+x_2(c_{00}+c_{11})+x_3(c_{00}+c_{10})+x_1{c}_{00}]+w\hfill \\ n_{42}=\mu w_2(c_{00}+c_{11})-w_4[(x_2+x_4)(c_{00}+c_{11})+(x_1+x_3)c_{00}]+w\hfill \\ n_{43}=\mu w_3(c_{00}+c_{10})-w_4[(x_3+x_4)(c_{00}+c_{10})+(x_1+x_2)c_{00}]+w\hfill \\ n_{44}=\mu w_2[{\delta }_1(x_1-rD)+{\delta }_2(x_2+rD)]+\mu w_3[{\delta }_1(x_1-rD)+{\delta }_3(x_3+rD)]+w_4{\delta }_4(x_4-rD)\hfill \end{array}$$ (4.8)

The proof is given in SOM, section C.

5. The evolution of an epistasis modifier—Results

Does a new modifier allele M_i determining new fitness parameters $w_1^i,w_2^i,w_3^i,w_4^i$ that induce “more” positive or “more” negative epistasis invade a population that is at a VAHW equilibrium? Using the additive and the multiplicative epistasis measures defined in (1.4) and (1.5), the new epistasis measures with the M_i allele present at the modifier locus are, respectively,

$$\begin{array}{c}{e}_{\mathrm{add}}^i=w_1^i-w_2^i-w_3^i+w_4^i\hfill \\ e_{\mathrm{mult}}^i=w_1^i{w}_4^i-w_2^i{w}_3^i.\hfill \end{array}$$ (5.1)

Since the modifier allele M_i determines the fitness parameters linearly as in (4.6), the new measure of additive epistasis is

$$e_{\mathrm{add}}^i=(w_1-w_2-w_3+w_4)+(w_1{\delta }_1-w_2{\delta }_2-w_3{\delta }_3+w_4{\delta }_4){\epsilon }_i.$$ (5.2)

As (w₁ - w₂ - w₃ + w₄) is the “present” measure of additive epistasis, the change in additive epistasis is (w₁δ₁ - w₂δ₂ - w₃δ₃ + w₄δ₄)ε_i. If (w₁ - w₂ - w₃ + w₄) > 0, for example, then as we assume that ε_i > 0, there is “more” or “less” additive epistasis depending on whether (w₁δ₁ - w₂δ₂ - w₃δ₃ + w₄δ₄) is positive or negative, respectively.

Similarly, with multliplicative epistasis the new measure is

$$e_{\mathrm{mult}}^i=(w_1{w}_4-w_2{w}_3)+(w_1{w}_4{\delta }_1-w_2{w}_3{\delta }_2-w_2{w}_3{\delta }_3+w_1{w}_4{\delta }_4){\epsilon }_i+(w_1{w}_4{\delta }_1{\delta }_4-w_2{w}_3{\delta }_2{\delta }_3){\epsilon }_i^2.$$ (5.3)

In our analysis we assume ε_i to be positive and “small” and ignore non-linear terms in ε_i. If w₁w₄ - w₂w₃ > 0, for example, then up to non-linear terms in ε_i, the change in multiplicative epistasis is determined by the sign of the term (w₁w₄δ₁ - w₂w₃δ₂ - w₂w₃δ₃ + w₁w₄δ₄), and “more” multiplicative epistasis corresponds to this term being positive while there is “less” multiplicative epistasis when this term is negative.

The fate of the new modifier alleles is determined by eigenvalues of the linear approximation matrix L of (4.3) or the sign of Q(1) of (4.7). Unfortunately the general analysis of the eigenvalues of L and the sign of Q(1) is too complicated. However, we can provide some results that shed light on the general trend of evolution. Specifically, we will discuss the following cases:

1. R ≈ 0, where the modifier locus and the major loci are tightly linked;

2. r ≈ 0, where the two major loci are tightly linked;

3. Modifier changing only one fitness parameter.

6. The case R ≈ 0

When R = 0 we have

$$c_{00}=1-r,c_{01}=0,c_{10}=r,c_{11}=0.$$ (6.1)

As a result, the matrix N of (4.8) that determines the external stability of a VAHW equilibrium x ⊗ y, has the special form

$$\mathbf{N}=\left[\begin{array}{cccc}\hfill 0\hfill & \hfill 0\hfill & \hfill n_{13}\hfill & \hfill n_{14}\hfill \\ \hfill n_{21}\hfill & \hfill n_{22}\hfill & \hfill 0\hfill & \hfill n_{24}\hfill \\ \hfill 0\hfill & \hfill 0\hfill & \hfill n_{33}\hfill & \hfill n_{34}\hfill \\ \hfill n_{41}\hfill & \hfill n_{42}\hfill & \hfill n_{43}\hfill & \hfill n_{44}\hfill \end{array}\right].$$ (6.2)

It is easily seen that in this case det(N) = -N₁ · N₂ where

$$N_1=\det \left[\begin{array}{cc}\hfill n_{13}\hfill & \hfill n_{14}\hfill \\ \hfill n_{33}\hfill & \hfill n_{34}\hfill \end{array}\right],N_2=\det \left[\begin{array}{cc}\hfill n_{21}\hfill & \hfill n_{22}\hfill \\ \hfill n_{41}\hfill & \hfill n_{42}\hfill \end{array}\right].$$ (6.3)

Let p = x₁ + x₂, then from (4.8) we have

$$\begin{array}{cc}{n}_{13}=rp\hfill & n_{14}={\delta }_1(x_1-rD)\hfill \\ n_{33}=(1-\mu )(1-r)w_3-w\hfill & n_{34}=(1-\mu )w_3[{\delta }_1(x_1-rD)+{\delta }_3(x_3+rD)],\hfill \end{array}$$ (6.4)

and

$$\begin{array}{cc}{n}_{21}=-(1-\mu )w_{2}rp\hfill & n_{22}=(1-\mu )w_2(1-r)-w\hfill \\ n_{41}=-\mu w_{2}rp-w_4(1-rp)+w\hfill & n_{42}=\mu w_2(1-r)-w_4(1-r)+w.\hfill \end{array}$$ (6.5)

Direct computation of N₂, using the normalizing factor w of (1.3) reveals that

$$N_2=(w-w_4)[w-(1-\mu )w_2](1-r)+rw[(w_1-w_2)(x_1-rD)+(w_3-w_4)(x_3+rD)].$$ (6.6)

As we assume w₁ > w₂, w₃ > w₄, at equilibrium w > w₄, w > (1 - μ)w₂ and we conclude that N₂ > 0 for all values of r and μ.

Following Result 6, a new modifier allele will invade the population if N₁ > 0. Now

$$N_1={\delta }_1(x_1-rD)\{w-(1-\mu )w_3[1-r(1+p)]\}+{\delta }_3(x_3+rD)(1-\mu )w_{3}rp.$$ (6.7)

Thus, when r = 0 N₁ is positive if and only if δ₁ > 0, whereas when r > 0, N₁ is positive if

$${\delta }_3>-{\delta }_1\frac{x_1-rD}{x_3+rD}\cdot \frac{w-(1-\mu )[1-r(1+p)]w_3}{(1-\mu )rp{w}_3}.$$ (6.8)

Thus when r = 0 and δ₁ > 0, with all the other δ_k values small in absolute value, then a new allele associated with more positive additive or multiplicative epistasis invades when ε_i > 0 and small. Moreover, when r > 0 condition (6.8) is valid when δ₁ > 0 and δ₃ < 0 but small in absolute value, but also when δ₁ < 0 and δ₃ > 0 and large enough. In these two cases when |δ₂| and |δ₄| are small, the modifier allele can invade if it introduces more positive as well as “more” negative additive or multiplicative epistasis. Thus, using small perturbation theory, we have the following result.

Result 7

Suppose the modifier locus is tightly linked to the two major loci (R ≈ 0) and let x ⊗ y be a VAHW equilibrium. Then it is possible that anew modifier allele that induces more positive or more negative additive or multiplicative epistasis invades the population for all values of r and μ for which x ⊗ y exists and is stable.

7. The case r ≈ 0

If the two major loci are absolutely linked (r = 0), then the equilibrium equations at the major loci reduce to

$$\begin{array}{c}w{x}_1=(1-\mu )w_1{x}_1\hfill \\ w{x}_2=\mu (1-\mu )w_1{x}_1+(1-\mu )w_2{x}_2\hfill \\ w{x}_3=\mu (1-\mu )w_1{x}_1+(1-\mu )w_3{x}_3\hfill \\ w{x}_4={\mu }^2{w}_1{x}_1+\mu w_2{x}_2+\mu w_3{x}_3+w_4{x}_4.\hfill \end{array}$$ (7.1)

Solving the equilibrium equations yields the four types of equilibria represented in Table 1. It should be noted that the “corner” and the two “edge” equilibria have linkage equilibrium (D = 0), whereas the “face” equilibrium has negative linkage equilibrium (D < 0). Following Result 4, at the polymorphic equilibrium the sign of D coincides with the sign of the multiplicative epistasis e_mult = w₁w₄ - w₂w₃, with D = 0 when e_mult = 0 and selection is multiplicative.

Table 1.

Possible equilibria when r = 0

Type	Frequencies	Existence Conditions
Corner	x̄ = (0, 0, 0, 1)	none
Edge	$\begin{array}{ccc}\hfill \tilde{\mathbf{x}}=(0,0,{\tilde{x}}_3,{\tilde{x}}_4),\hfill & \hfill {\tilde{x}}_3=\frac{(1-\mu )w_3-w_4}{w_3-w_4},\hfill & \hfill {\tilde{x}}_4=\frac{\mu w_3}{w_3-w_4}\hfill \\ \hfill \widetilde{\tilde{\mathbf{x}}}=(0,{\tilde{\tilde{x}}}_2,0,{\tilde{\tilde{x}}}_4),\hfill & \hfill {\tilde{\tilde{x}}}_2=\frac{(1-\mu )w_2-w_4}{w_2-w_4},\hfill & \hfill {\tilde{\tilde{x}}}_4=\frac{\mu w_2}{w_2-w_4}\hfill \end{array}$	(1 - μ)w3 > w4 (1 - μ)w2 > w4
Face	$\widehat{\mathbf{x}}=(0,\alpha ,\alpha ,1-2\alpha ),\alpha =\frac{(1-\mu )w_2-w_4}{2(w_2-w_4)},1-\alpha =\frac{\mu w_2}{w_2-w_4}$	w2 = w3, (1 - μ)w2 > w4
Polymorphism*	${\mathbf{x}}^{\ast }=(x_1^{\ast },x_2^{\ast },x_3^{\ast },x_4^{\ast })$	(1 - μ)w1 > w2, (1 - μ)w1 > w3 (1 - μ)2w1 > w4

^*With the polymorphism x* we have $x_2^{\ast }=\frac{\mu w_1{x}_1^{\ast }}{(1-\mu )w_1-w_2},x_3^{\ast }=\frac{\mu w_1{x}_1^{\ast }}{(1-\mu )w_1-w_3},x_4^{\ast }=\frac{{\mu }^2{w}_1{x}_1^{\ast }}{{(1-\mu )}^2{w}_1-w_4}\left[1+\frac{w_2}{(1-\mu )w_1-w_2}+\frac{w_3}{(1-\mu )w_1-w_3}\right]$.

In Table 2 we list sufficient conditions for a new modifier allele to invade, i.e., the VAHW equilibrium is externally unstable. The complete details of the analysis are given in SOM, section D. As all four types of equilibria are independent of R, we chose sufficient conditions that are also independent of R. Other sufficient conditions that depend on R are also available and can be obtained using SOM, section D.

Table 2.

Sufficient conditions for external instability

Equilibrium	Sufficient condition for external instability
Corner x̄ = (0, 0, 0, 1)	δ4 > 0
Edge x̃ = (0, 0, x̃3, x̃4)	δ3 > 0, w3δ3x3 + w4δ4x4 > 0
Face x̂ = (0, α, α, 1 - 2α)	δ2 = δ3 > 0, w2δ2x2 + w3δ3x3 + w4δ4x4 > 0
Polymorphism† ${\mathbf{x}}^{\ast }=(x_1^{\ast },x_2^{\ast },x_3^{\ast },x_4^{\ast })$	δ1 > 0, δ2 > 0, δ3 > 0, δ4 > 0 or δ1 > 0, ${\sum }_{k=1}^4{w}_k{\delta }_k{x}_k>0,{\sum }_{k=1}^4{\delta }_k{x}_k<0$

^†The conditions are given for the special case where w₂ = w₃ in which case $x_2^{\ast }=x_3^{\ast }$.

Following Table 2, and the additive and multiplicative measures of epistasis associated with a new modifying allele described in (5.2) and (5.3), it is easily seen that a new modifier that changes epistasis to be “more positive” or “more negative,” both in the additive and multiplicative sense, can invade. For example, if the present VAHW equilibrium x ⊗ y has x = x*, the polymorphic equilibrium of Table 1, and originally there is positive additive epistasis (w₁ - w₂ - w₃ + w₄) ≥ 0, or positive multiplicative epistasis w₁w₄ - w₂w₃ ≥ 0, then from Table 2 if δ₁, δ₄ > 0 and δ₂, δ₃ > 0 but small, a modifier allele that increases epistasis (see (5.2) and (5.3)) will invade. Similarly, if originally we have negative epistasis, (w₁ - w₂ - w₃ + w₄) ≤ 0 or w₁w₄ - w₂w₃ ≤ 0, and δ₂, δ₃ > 0 with δ₁, δ₄ > 0 but small, then a modifier that induces “more negative” epistasis invades.

Using small perturbation theory the same phenomenon occurs also when r ≈ 0, and we have the following result:

Result 8

If the two major loci are tightly linked (r ≈ 0), then a new modifier allele associated with more positive or more negative additive or multiplicative epistasis can invade a population which is at a VAHW equilibrium.

8. Modification of one fitness parameter

We now discuss the four special cases where the new modifier allele changes only one of the fitness parameters. Here the only constraints on recombination are $0\le R\le \frac{1}{2}$ and $0\le r\le \frac{1}{2}$.

Case I

δ₁ = δ₂ = δ₃ = 0, w₄δ₄ =1

In this case the modifier allele M_i does not alter the fitness parameters of the gametes AB, Ab, aB but changes the fitness of ab from w₄ to w₄ + ε_i. Thus in the presence of M_i the additive epistasis measure is

$$e_{\mathrm{add}}^i=(w_1-w_2-w_3+w_4)+{\epsilon }_i,$$ (8.1)

while the multiplicative epistasis measure is

$$e_{\mathrm{mult}}^i=(w_1{w}_4-w_2{w}_3)+w_1{\epsilon }_i.$$ (8.2)

As we assume ε_i > 0, in both of these cases M_i introduces “more” positive epistasis. In this case we have the following result (see SOM, section E).

Result 9

If selection modification is of the form in Case I, then the VAHW equilibrium x⊗y is externally unstable towards the introduction of “more” positive (additive or multiplicative) epistasis. In particular the population evolves away from no (additive or multiplicative) epistasis towards positive epistasis.

Remark 6

Although Result 7 holds for small values of ε_i, Result 9 for Case I is true for any ε_i. This is easily seen from the expansion of Q(1) given in SOM, section C under the assumptions of Case I.

Case II

δ₁ = δ₂ = δ₄ = 0, δ₃w₃ = 1

In this case the modifier allele M_i changes only the fitness of aB from w₃ to w₃ + ε_i. Thus the total additive epistasis becomes

$$e_{\mathrm{add}}^i=(w_1-w_2-w_3+w_4)-{\epsilon }_i,$$ (8.3)

and the multiplicative epistasis is

$$e_{\mathrm{mult}}^i=(w_1{w}_4-w_2{w}_3)-w_2{\epsilon }_i.$$ (8.4)

Hence, in both cases when ε_i > 0, M_i reduces the amount of positive epistasis or produces more negative epistasis. In this case we have (see SOM, section E)

Result 10

If the selection modification follows Case II, then any VAHW equilibrium x ⊗ y is externally unstable towards the introduction of more negative, but small, epistasis provided the mutation rate μ is sufficiently small.

Case III

δ₁ = δ₃ = δ₄ = 0, δ₂w₂ = 1

In our model there is complete symmetry between the second and the third gametes Ab and aB respectively. Therefore the results of Case II apply also to this case; that is, Result 10 also holds under the assumptions of Case III.

Case IV

δ₂ = δ₃ = δ₄ = 0, δ₁w₁ = 1

Here the new measures of epistasis are

$$e_{\mathrm{add}}^i=(w_1-w_2-w_3+w_4)+{\epsilon }_i,e_{\mathrm{mult}}^i=(w_1{w}_4-w_2{w}_3)+w_4{\epsilon }_i,$$ (8.5)

as the modifier allele M_i changes only the fitness of AB from w₁ to w₁ + ε_i. Hence epistasis becomes more positive when ε_i > 0. In this case we have a similar result to Result 9, namely:

Result 11

If selection modification follows Case IV, then the VAHW equilibrium x ⊗ y is externally unstable towards the introduction of “more” additive or multiplicative epistasis, provided mutation rates are sufficiently small.

The proof is given in SOM, section E.

9. Conclusions

A general analysis of the general problem with arbitrary recombination rates and general fitnesses and fitness perturbations appears to be intractable. In the case of no recombination our results are completely in line with those of Desai et al. (2007) in that a modifier increasing the level of antagonistic epistasis will be favored. When both R and r are very close to zero we again have evolution of more positive (multiplicative or additive) epistasis. However, when either r or R or both are larger, it is possible that more antagonistic or more synergistic epistasis evolves. This possibility was not seen in the multilocus analysis of Desai et al. The apparent discrepancy occurs because the fitness models are not exactly the same; by changing w₄, for example, in the present paper we have affected s and ε in their terms, not just the ε parameter.

Our results appear to differ from those of Azevedo et al. (2006), although these authors did not use a modifier gene to alter epistasis. They found that with recombination there should generally be selection towards (negative) synergistic epistasis. At best their conclusion is incomplete; as revealed by our modifier model the amount of recombination among the interacting genes can be important. Our analyses address evolution of epistasis between a single pair of diallelic loci, while the simulations by Azevedo et al. tracked ten diallelic loci. In principle, there could be 2¹⁰ - 10 - 1 (i.e., 1,013) epistatic interactions with ten diallelic loci, and it may be that the discrepancy between our results and theirs is due to their summary of these interactions by a single scalar.

In their work with deleterious mutations in E. coli, Elena and Lenski (1997) found no signal of epistasis in one set of experiments, but both positive and negative epistasis in another set of experiments. A recent reanalysis of these data (Beerenwinkel et al. 2007) found that among 27 double mutants there were more positive epistatic interactions between more deleterious mutations. The overall fitness landscape in therefore more complicated than the widely used form e^-sk(1+ε) (e.g., Desai et al. 2007). However, neither the data of Elena and Lenski nor the analysis by Beerenwinkel et al. really address how the shape of the fitness landscape, and in particular the signs of pairwise epistasis measures, might evolve. Our analysis suggests that across a genome with multiple mutations, since either positive or negative pairwise epistasis might evolve, a complex fitness landscape should be expected.

All of our results assume the population is initially in the neighborhood of a mutation selection balance. We cannot say what the dynamics of epistasis might be if it changes during a transient phase of the evolution. These haploid results would appear to confirm and extend our earlier findings for diploid populations, namely that an increased level of interaction between genes is likely to be favored by evolution. In the diploid case (Liberman and Feldman 2006) we saw that this was not necessarily a consequence of an increase in the mean fitness with increasing epistasis. It remains to be seen whether the same is true in the haploid model. It would also be interesting if, as was seen in Liberman and Feldman (2006), the way in which the equilibrium mean fitness changed with epistasis depended on the recombination rates.