On the evolution of epistasis II: A generalized Wright-Kimura framework

Abstract

The evolution of fitness interactions between genes at two major loci is studied where the alleles at a third locus modify the epistatic interaction between the two major loci. The epistasis is defined by a parameter ε and a matrix structure that specifies the nature of the interactions. When ε = 0 the two major loci have additive fitnesses, and when these are symmetric the interaction matrices studied here produce symmetric viabilities of the Wright (1952)-Kimura (1956) form. Two such interaction matrices are studied, for one of which epistasis as measured by |ε| always increases, and for the other it increases when the linkage between the major loci is tight enough and there is initial linkage disequilibrium. Increase of epistasis does not necessarily coincide with increase in equilibrium mean fitness.

1. Introduction

The term “evolvability” has been used to describe the potential for change in the additive genotypic contribution to phenotypic variance (e.g., Carter et al. 2005 and references therein). Hansen and Wagner (2001) proposed that the role of epistasis in evolvability could be studied by allowing the effect of each gene to be a linear function of the effects of other genes—the “multilinear genotype-phenotype” map. This model was also used by Carter et al. (2005) to claim that one form of linear interaction, called “directional epistasis,” is important in evolution and, in particular, the evolution of the additive component of genotypic variance. Both linkage and linkage disequilibrium were claimed to be unimportant to response to multilinear epistatic selection.

The focus of these and subsequent studies (Hansen et al. 2006) has been on the directionality of the epistasis, namely the extent to which effects of different genes reinforce each other along a trajectory of directional selection on a phenotype to which they contribute. Epistasis here is a component of the variance in the trait under selection, and its effect is modeled in the analysis of variance framework (see also Barton and Turelli 2005). This kind of epistasis is called “statistical” (e.g., Carter et al. 2005, Weinreich et al. 2005) because of its essential dependence on allele frequencies at the interacting loci. “Physiological” (Cheverud and Routman 1995) or “functional” (Hansen and Wagner 2001) epistasis, on the other hand, describes how the loci interact functionally, in our case to produce individual fitnesses. Weinreich et al. (2005) regard the latter as a more inclusive concept of epistasis because it does not depend on allele frequencies at the interacting loci and therefore, unlike statistical epistasis, is not a population property. We focus here on functional epistasis and its evolutionary dynamics, extending the modifier approach we initiated in Liberman and Feldman (2005). We define two-locus fitnesses with explicit epistatic interactions betweeen them and ask whether modifier alleles that induce departures from purely additive fitness effects will succeed. This allows us to examine explicitly the effects of linkage disequilibrium and linkage among the major and modifier loci on the dynamics of epistasis.

To construct this model to test the evolution of epistasis, we consider two major loci with alleles A₁, A₂ at the first and B₁, B₂ at the second. A third locus with alleles M₁, M₂ modifies the epistasis between the first two. If the order of the loci is ABM, then r is the recombination rate between A and B with R that between B and M. We shall assume throughout that there is no interference so that the probability of a double recombination event is rR.

The viabilities of the genotypes at the major loci are represented by the 4 × 4 fitness matrix W given as:

It is assumed that W is symmetric, namely, w_ij = w_ji for all i, j and that it shows no position effect, that is w₁₄ = w₂₃. Liberman and Feldman (2005) represent the way the third locus determines the amount of epistasis by assuming that W takes the form

$$\mathbf{W}={\mathbf{W}}_{\text{add}}+{\epsilon }_j\mathbf{\Delta },$$ (1.2)

where

$${\mathbf{W}}_{\text{add}}=\left[\begin{array}{cccc}{\alpha }_1+{\beta }_1& {\alpha }_1+{\beta }_2& {\alpha }_2+{\beta }_1& {\alpha }_2+{\beta }_2\\ {\alpha }_1+{\beta }_2& {\alpha }_1+{\beta }_3& {\alpha }_2+{\beta }_2& {\alpha }_2+{\beta }_3\\ {\alpha }_2+{\beta }_1& {\alpha }_2+{\beta }_2& {\alpha }_3+{\beta }_1& {\alpha }_3+{\beta }_2\\ {\alpha }_2+{\beta }_2& {\alpha }_2+{\beta }_3& {\alpha }_3+{\beta }_2& {\alpha }_3+{\beta }_3\end{array}\right],$$ (1.2a)

$$\mathbf{\Delta }=\left[\begin{array}{rrrr}\hfill 1& \hfill -1& \hfill -1& \hfill 1\\ \hfill -1& \hfill 1& \hfill 1& \hfill -1\\ \hfill -1& \hfill 1& \hfill 1& \hfill -1\\ \hfill 1& \hfill -1& \hfill -1& \hfill 1\end{array}\right],$$ (1.2b)

and in (1.2) ε_j takes the values ε₁, ε₂, or ε₃ depending on whether the genotype at the modifier locus is M₁M₁, M₁M₂, or M₂M₂, respectively. The fitness scheme (1.2) is the most general two-locus two-allele fitness scheme with no fitness differences between coupling and repulsion genotypes such that the row-wise additive epistasis measures, defined by Bodmer and Felsenstein (1967)

$$e_i=w_{i1}-w_{i2}-w_{i3}+w_{i4}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i=1,2,3,4$$ (1.3)

satisfy e₁ = –e₂ = –e₃ = e₄ = 4ε (see also Puniyani et al. 2004). The epistatic parameter ε = e₁/4 plays a key role in our model but is difficult to characterize as directional or non-directional, in the terms of Carter et al. (2005). We shall see, however, that the Δ matrix which determines how ε affects the fitnesses, can play an important role in its evolution. In our framework, it is Δ which defines the architecture of interactions. Later in this paper we consider other constructions for the epistatic parameter that use interaction matrices different from (1.2b).

For this three-locus system, Liberman and Feldman (2005) determined conditions for the increase of a rare allele M₂ in a system in which M₁ is close to fixation and showed that if |ε₂ – ε₁| is sufficiently small an allele M₂ with ε₂ > ε₁ will invade when both r and R are positive, and when the major loci are in linkage equilibrium (D = 0) with M₁ fixed and the allele frequencies at the major loci are not equal. They also showed that in the special case where W_add is alsosymmetric, α₁ = α₃ and β₁ = β₃, in which case for all ε W = W_add + εΔ is a special case of the Lewontin and Kojima (1960) symmetric fitness matrix, if the initial state where M₁ is fixed is in linkage disequilibrium (D ≠ 0) with the major loci, then for positive r and R, M₂ increases in frequency if (ε₂ - ε₁) is small and ε₂ >ε₁ > 0.

It should be pointed out that the claim in Liberman and Feldman (2005) is not that increase of epistasis is always favored, but that for the Lewontin-Kojima fitness scheme, positive and negative linkage disequilibrium and all corresponding recombination rates lead to an increase in epistasis. These results are interesting in view of the arguments of Wagner and Altenberg (1996), Orr (2000), Welch and Waxman (2003), and Pepper (2003), which appear to suggest that the fitness contributions due to separate loci should become independent over the course of evolution. That is, the evolution of genotypic contributions to different phenotypes should be in the direction of modularity; that is, evolution should act to decrease the degree of interaction between genes that contribut too different phenotypes.

The results of Liberman and Feldman (2005) depend on the choice of the Δ matrix in (1.2). For example, if for any s, t we choose

$$\mathbf{\Delta }(s,t)=s\mathbf{\Delta }+t\mathbf{E},$$ (1.4)

where Δ is given in (1.2b) and E is the matrix whose entries are all 1, then the fitness matrix

$$\mathbf{W}(s,t)={\mathbf{W}}_{\text{add}}+\epsilon \mathbf{\Delta }(s,t)$$ (1.5)

has the same properties as W = W(1, 0), namely equal fitnesses for the two double heterozygotes, and the epistasis measures in (1.3) satisfy e₁ = –e₂ = –e₃ = e₄, for all values of s and t. However, the fate of a new modifier allele M₂ introduced near an equilibrium where M₁ is fixed may depend on the values of s and t. Specifically, if s = −1, t = 0, for example, an allele M₂ with ε₂ < ε₁ will invade for sufficiently small |ε₂ - ε₁| when both R and r are positive.

2. The interaction matrix Δ

The result of Liberman and Feldman (2005) for the case where the major loci are in linkage disequilibrium at the equilibrium with M₁ fixed at the modifier locus utilizes knowledge of a class of equilibria which can be obtained in closed form, in terms of r, for the Lewontin and Kojima (1960) symmetric viability model. The Lewontin-Kojima model is a special case of the general two-locus symmetric viability model introduced by Bodmer and Felsenstein (1967), where the fitness matrix W takes the form

$$\mathbf{W}=\left[\begin{array}{cccc}1-\delta & 1-\beta & 1-\gamma & 1\\ 1-\beta & 1-\alpha & 1& 1-\gamma \\ 1-\gamma & 1& 1-\alpha & 1-\beta \\ 1& 1-\gamma & 1-\beta & 1-\delta \end{array}\right].$$ (2.1)

If we denote by x₁, x₂, x₃, x₄ the frequencies of the four gametes AB, Ab, aB, ab, respectively, then this two-locus, two-allele viability model with recombination rate r between the two loci has symmetric equilibria (x̂₁, x̂₂, x̂₃, x̂₄) with

$${\widehat{x}}_1={\widehat{x}}_4=\frac{1}{4}+\widehat{D},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\widehat{x}}_2={\widehat{x}}_3=\frac{1}{4}-\widehat{D},$$ (2.2)

where D̂ = x̂₁x̂₄ – x̂₂x̂₃ is a solution of the cubic equation

$$64\ell D^3-16m{D}^2-4(\ell -8r)D+m=0$$ (2.3)

with ℓ= 2(β + γ) – (α + δ), m = δ – α (see Karlin and Feldman 1970).

Observe that W of (2.1) shows no position effect and the row measures of epistasis are

$$\begin{array}{l}{e}_1=e_4=(\beta +\gamma )-\delta \\ e_2=e_3=\alpha -(\beta +\gamma ).\end{array}$$ (2.4)

When m = δ – α = 0 we have the Lewontin-Kojima model. In this case e₁ = –e₂ = –e₃ = e₄ and the cubic equation has the solutions

$$\widehat{D}=0\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{or\hspace{0.17em}}\hspace{0.17em}\hspace{0.17em}\widehat{D}=\pm \frac{1}{4}\sqrt{1-\frac{8r}{\ell }}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{for\hspace{0.17em}}\hspace{0.17em}\hspace{0.17em}r\le \frac{\ell }{8}.$$ (2.5)

When l = 0 but δ ≠ α, the matrix (2.1) is the Wright (1952) and Kimura (1956) symmetric viability model. Under this model the four row measures of epistasis are the same (so that in (2.4), e₁ = e₂ = e₃ = e₄), and the cubic equation reduces to the quadratic

$$D^2-\frac{2r}{m}D-\frac{1}{16}=0\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}(m\ne 0).$$ (2.6)

Since $\left|D\right|<\frac{1}{4}$, this quadratic has only one solution, whose value depends on m:

$$\begin{array}{l}\widehat{D}=\frac{r}{m}-\sqrt{\frac{1}{16}+{\left(\frac{r}{m}\right)}^2},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}m>0\\ \widehat{D}=\frac{r}{m}+\sqrt{\frac{1}{16}+{\left(\frac{r}{m}\right)}^2},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}m<0.\end{array}$$ (2.7)

(See Karlin and Feldman 1970, Bürger 2000.) Using the methods of Karlin and Feldman (1970), Bürger (2000) records two additional equilibria that are asymmetric. That is, they are not of the form (2.2) with (2.5). Bürger shows that these asymmetric equilibria have a small parametric range of stability. We concentrate our analysis on the symmetric equilibria and will comment on the results for the asymmetric equilibria in the section on the numerical work.

We can therefore apply the analysis of Liberman and Feldman (2005) to the class of fitness matrices W = W_add + εΔ where W_add is specified in (1.2a) and $\mathbf{\Delta }={[{\delta }_{ij}]}_{i,j=1}^4$ takes the symmetric form (2.1), namely δ_ij = δ_ji and

$${\delta }_{11}={\delta }_{44},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\delta }_{22}={\delta }_{33},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\delta }_{14}={\delta }_{23},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\delta }_{13}={\delta }_{24},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\delta }_{34}={\delta }_{12}$$ (2.8)

with the added condition that δ_i1 – δ_i2 – δ_i3 + δ_i4 is the same for all rows i = 1, 2, 3, 4. With such a Δ, each fitness matrix W of the form W = W_add + εΔ has all four row measures of epistasis the same (e₁ = e₂ = e₃ = e₄). Observe that if α₁ = α₃ and β₁ = β₃ in W_add (see equ. 1.2a), then (2.8) ensures that W = W_add + εΔis a Wright-Kimura symmetric viability matrix for all ε values.

3. A new class of fitness matrices

Let $\mathbf{\Delta }={\left[{\delta }_{ij}\right]}_{i,j}^4=1$ be a symmetric matrix that satisfies the relations (2.8) with δ_i1 – δ_i2 – δ_i3 + δ_i4 the same for all i. Then Δ can be represented as

$$\mathbf{\Delta }=\left[\begin{array}{cccc}a& b& c& e\\ b& d& e& c\\ c& e& d& b\\ e& c& b& a\end{array}\right]$$ (3.1)

where d = 2(b + c) –(a + 2e). We will concentrate on two examples of the class (3.1). The first is

$$\overline{\mathbf{\Delta }}=\left[\begin{array}{rrrr}\hfill 1& \hfill 0& \hfill 0& \hfill 1\\ \hfill 0& \hfill -3& \hfill 1& \hfill 0\\ \hfill 0& \hfill 1& \hfill -3& \hfill 0\\ \hfill 1& \hfill 0& \hfill 0& \hfill 1\end{array}\right]$$ (3.2)

and the second is

$$\widehat{\mathbf{\Delta }}=\left[\begin{array}{rrrr}\hfill 1& \hfill -1& \hfill -1& \hfill 1\\ \hfill -1& \hfill -7& \hfill 1& \hfill -1\\ \hfill -1& \hfill 1& \hfill -7& \hfill -1\\ \hfill 1& \hfill -1& \hfill -1& \hfill 1\end{array}\right].$$ (3.3)

Observe that Δ̂ = 2Δ̄ – E. In both cases we will follow the analysis developed in Liberman and Feldman (2005) for the evolution of the “new” modifier allele introduced into a population at an equilibrium with the modifier allele fixed at M₁.

Let $x_1^{\ast },x_2^{\ast },x_3^{\ast },x_4^{\ast }$ be the equilibrium frequencies of the four gametes ABM₁, AbM₁, aBM₁, abM₁, corresponding to the fitness matrix ${\mathbf{W}}^{\ast }=\parallel w_{ij}^{\ast }\parallel ={\mathbf{W}}_{\text{add}}+{\epsilon }_1\mathbf{\Delta }$ determined by the genotype M₁M₁ at the modifier locus, where Δ is given either in (3.2) or (3.3). $D^{\ast }=x_1^{\ast }{x}_4^{\ast }-x_2^{\ast }{x}_3^{\ast }$ is the linkage disequilibrium associated with this equilibrium.

The “external” stability of the equilibrium to invasion by M₂ is determined by the eigenvalues of the matrix of the linear approximation to the transformation of frequencies “near” the equilibrium. The eigenvalues of this matrix are the roots of its characteristic polynomial Q(z). The sign of Q(1) is important because Q(z) is a fourth-degree polynomial, so that if Q(1) < 0 it has at least one root that is greater than unity, which entails that M₂ will increase when rare. In Liberman and Feldman (2005) it is verified that neglecting non-linear terms in (ε₂ – ε₁) we have

$$Q(1)=\frac{({\epsilon }_2-{\epsilon }_1)}{x_1^{\ast }{x}_2^{\ast }{x}_3^{\ast }{x}_4^{\ast }}det(\stackrel{\sim }{\mathbf{M}}),$$ (3.4)

where M̃ is given by

$$\stackrel{\sim }{\mathbf{M}}=\left[\begin{array}{cccc}{\delta }_1^{\ast }{x}_1^{\ast }-r{\ell }^{\ast }& A_1& B_1& C_1\\ {\delta }_2^{\ast }{x}_2^{\ast }+r{\ell }^{\ast }& D_1& C_2& B_2\\ {\delta }_3^{\ast }{x}_3^{\ast }+r{\ell }^{\ast }& C_2& D_2& A_2\\ {\delta }^{\ast }& -r{d}^{\ast }& -r{d}^{\ast }& r{d}^{\ast }\end{array}\right],$$ (3.5)

where ${\ell }^{\ast }={\delta }_{14}{x}_1^{\ast }{x}_4^{\ast }-{\delta }_{23}{x}_2^{\ast }{x}_3^{\ast },\hspace{0.17em}{d}^{\ast }=w_{14}^{\ast }{x}_1^{\ast }{x}_4^{\ast }-w_{23}^{\ast }{x}_2^{\ast }{x}_3^{\ast }$, and

$$\begin{array}{l}{\delta }_k^{\ast }=\underset{\ell =1}{\overset{4}{\Sigma }}{\delta }_{k\ell }{x}_{\ell }^{\ast }\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}k=1,2,3,4\\ {\delta }^{\ast }=\underset{k=1}{\overset{4}{\Sigma }}{\delta }_k^{\ast }{x}_k^{\ast }.\end{array}$$ (3.6)

Also

$$\begin{array}{l}{A}_1=w_{12}^{\ast }{x}_1^{\ast }{x}_2^{\ast }R+w_{23}^{\ast }{x}_2^{\ast }{x}_3^{\ast }rR\\ A_2=w_{34}^{\ast }{x}_3^{\ast }{x}_4^{\ast }R+w_{14}^{\ast }{x}_1^{\ast }{x}_4^{\ast }rR\\ B_1=w_{13}^{\ast }{x}_1^{\ast }{x}_3^{\ast }\left[(1-r)R+r(1-R)\right]+w_{23}{x}_2^{\ast }{x}_3^{\ast }r(1-R)\\ B_2=w_{24}^{\ast }{x}_2^{\ast }{x}_4^{\ast }\left[(1-r)R+r(1-R)\right]+w_{14}{x}_1^{\ast }{x}_4^{\ast }r(1-R)\\ C_1=w_{14}{x}_1^{\ast }{x}_4^{\ast }(1-r)R\\ C_2=w_{23}{x}_2^{\ast }{x}_3^{\ast }(1-r)R\\ D_1=w_{22}^{\ast }{x_2^{\ast }}^2+w_{12}^{\ast }{x}_1^{\ast }{x}_2^{\ast }(1-R)+w_{24}{x}_2^{\ast }{x}_4^{\ast }\left[(1-r)(1-R)+rR\right]+\\ +w_{23}^{\ast }{x}_2^{\ast }{x}_3^{\ast }(1-r)(1-R)-w^{\ast }{x}_2^{\ast }\\ D_2=w_{33}^{\ast }{x_3^{\ast }}^2+w_{13}{x}_1^{\ast }{x}_3^{\ast }\left[(1-r)(1-R)+rR\right]+w_{34}^{\ast }{x}_3^{\ast }{x}_4^{\ast }(1-R)+\\ +w_{23}^{\ast }{x}_2^{\ast }{x}_3^{\ast }(1-r)(1-R)-w^{\ast }{x}_3^{\ast }.\end{array}$$ (3.7)

The mean fitness at equilibrium, w*, is

$$w^{\ast }=\underset{k,\ell =1}{\overset{4}{\Sigma }}{w}_{k\ell }^{\ast }{x}_k^{\ast }{x}_{\ell }^{\ast }.$$ (3.8)

4. Evolution of epistasis “near” equilibrium with D* = 0; ε₁ = 0

If ε₁ (corresponding to M₁M₁) is zero, then the initial model is additive and with overdominance at both major loci, when r > 0, D* = 0 is the unique interior equilibrium. When D* = 0 at equilibrium, then also d* = 0 (since $w_{14}^{\ast }=w_{23}^{\ast }$), and det(M̃) of (3.4) and (3.5) is given by

$$det(\stackrel{\sim }{\mathbf{M}})=-{\delta }^{\ast }\left|\begin{array}{ccc}{A}_1& B_1& C_1\\ D_1& C_2& B_2\\ C_2& D_2& A_2\end{array}\right|.$$ (4.1)

Liberman and Feldman (2005) show that if D* = 0, then

$$\begin{array}{l}{C}_1=C_2=C\hfill \\ D_1+A_1+B_2+C=0\hfill \\ D_2+A_2+B_1+C=0,\hfill \end{array}$$ (4.2)

and in fact

$$\left|\begin{array}{ccc}{A}_1& B_1& C_1\\ D_1& C_2& B_2\\ C_2& D_2& A_2\end{array}\right|>0$$ (4.3)

provided R > 0. When R = 0, then C₁ = C₂ = 0 and also A₁ = A₂ = 0 and, due to (4.2), D₁ + B₂ = 0, which implies that

$$\left|\begin{array}{ccc}{A}_1& B_1& C_1\\ D_1& C_2& B_2\\ C_2& D_2& A_2\end{array}\right|=\left|\begin{array}{ccc}0& B_1& 0\\ D_1& 0& B_2\\ 0& D_2& 0\end{array}\right|=\left|\begin{array}{ccc}0& B_1& 0\\ D_1+B_2& 0& B_2\\ 0& D_2& 0\end{array}\right|=0.$$ (4.4)

Thus when R > 0 and D* = 0 the sign of det(M̃) is the sign of –δ* Using the representation of Δ̄ given in (3.2) we have

$$\begin{array}{r}\hfill {\overline{\delta }}_1^{\ast }={\overline{\delta }}_4^{\ast }=x_1^{\ast }+x_4^{\ast }\\ \hfill {\overline{\delta }}_2^{\ast }=-3x_2^{\ast }+x_3^{\ast }\\ \hfill {\overline{\delta }}_3^{\ast }=-3x_3^{\ast }+x_2^{\ast }\end{array}.$$ (4.5)

Therefore

$${\overline{\delta }}^{\ast }=\underset{k=1}{\overset{4}{\Sigma }}{\delta }_k^{\ast }{x}_k^{\ast }={\left(x_1^{\ast }+x_4^{\ast }\right)}^2-3\left({x_2^{\ast }}^2+{x_3^{\ast }}^2\right)+2x_2^{\ast }{x}_3^{\ast }$$ (4.6)

$$={\left(x_1^{\ast }+x_4^{\ast }\right)}^2-{\left(x_2^{\ast }+x_3^{\ast }\right)}^2-2{\left(x_2^{\ast }-x_3^{\ast }\right)}^2$$ (4.7)

or, since $x_1^{\ast }+x_2^{\ast }+x_3^{\ast }+x_4^{\ast }=1$,

$${\overline{\delta }}^{\ast }=\left(x_1^{\ast }-x_2^{\ast }-x_3^{\ast }+x_4^{\ast }\right)-2{\left(x_2^{\ast }-x_3^{\ast }\right)}^2.$$ (4.8)

As $d^{\ast }=w_{14}^{\ast }{x}_1^{\ast }{x}_4^{\ast }-w_{23}{x}_2^{\ast }{x}_3^{\ast }$ and $w_{14}^{\ast }=w_{23}^{\ast }$, we also have D* = 0, and we can represent the $x_k^{\ast }$’s as

$$x_1^{\ast }=p^{\ast }{q}^{\ast },\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{x}_2^{\ast }=p^{\ast }(1-q^{\ast }),\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{x}_3^{\ast }=(1-p^{\ast })q^{\ast },\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{x}_4^{\ast }=(1-p^{\ast })(1-q^{\ast }),$$ (4.9)

where $p^{\ast }=x_1^{\ast }+x_2^{\ast },q^{\ast }=x_1^{\ast }+x_3^{\ast }$. Hence

$${\overline{\delta }}^{\ast }=(2p^{\ast }-1)(2q^{\ast }-1)-2{(p^{\ast }-q^{\ast })}^2.$$ (4.10)

Unlike the case discussed in Liberman and Feldman (2005), δ̄* is not always positive, but will be positive, for example, when p* ≈ q* and both $p^{\ast },q^{\ast }>\frac{1}{2}$ or $p^{\ast },q^{\ast }<\frac{1}{2}$. When δ̄* > 0 then Q(1) < 0 if ε₂ > 0 provided ε₂ is small and R > 0. As Q(1) < 0 implies that the equilibrium is externally unstable, we again find in this case that a modifier allele that increases epistasis (ε₂ > 0) is favored.

To fully specify when δ̄* of (4.10) is positive, let

$$P=p^{\ast }-\frac{1}{2},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}Q=q^{\ast }-\frac{1}{2}.$$ (4.11)

Then it is easily seen that δ̄* > 0 when

$$\begin{array}{l}0<P<\frac{1}{2},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\left(2-\sqrt{3}\right)P<Q<min\left[\frac{1}{2},\left(2+\sqrt{3}\right)P\right]\\ -\frac{1}{2}<P<0,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}max\left[-\frac{1}{2},\left(2+\sqrt{3}\right)P\right]<Q<\left(2-\sqrt{3}\right)P.\end{array}$$ (4.12)

If we use the fitness scheme W = W_add + εΔ̂ , then as Δ̂ = 2Δ̄ – E we have

$${\widehat{\delta }}^{\ast }=2{\overline{\delta }}^{\ast }-1.$$ (4.13)

Thus δ̂* > 0 when 2δ̄* > 1, and although there are cases where δ̂* ≤ 0, there is a “wide” range of values for which δ̂* > 0, in which case an allele that increases epistasis can invade the population.

The exact conditions for δ̂* > 0 are

$$\begin{array}{l}\frac{1}{\sqrt{12}}<P<\frac{1}{2},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}2P-\sqrt{3P^2-\frac{1}{4}}<Q<min\left[\frac{1}{2},2P+\sqrt{3P^2-\frac{1}{4}}\right]\\ -\frac{1}{2}<P<-\frac{1}{\sqrt{12}},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}max\left[-\frac{1}{2},2P-\sqrt{3P^2-\frac{1}{4}}\right]<Q<2P+\sqrt{3P^2-\frac{1}{4}}.\end{array}$$ (4.14)

We therefore have the following result.

Result 1

Under both selection schemes W = W_add + εΔ̄ or W = W_add + εΔ̂, suppose that ε₁, corresponding to M₁M₁, is zero. Then a modifier allele M₂ that increases epistasis, introduced near the D* = 0 equilibrium, will invade the population provided R > 0 under conditions (4.12) and (4.14), respectively.

Remark 1

If ε₁ = 0 and ε₂ < 0, then M₂ induces negative epistasis and M₂ will increase if δ̄* or δ̂* are negative in this case. Conditions for δ̄* < 0 and δ̂* < 0 analogous to (4.12) and (4.14), respectively, can be derived.

5. Evolution of epistasis near D* ≠ 0 equilibria

We will limit our analysis to the case where, in W_add of (1.2a), we have α₁ = α₃ and β₁ = β₃, in which case the fitness matrix W = W_add + εΔ with Δ of the form (2.8) is a Wright-Kimura symmetric viability matrix for all feasible values of ε. We then assume that the equilibrium frequencies $x_1^{\ast },x_2^{\ast },x_3^{\ast },x_4^{\ast }$ are those of the symmetric polymorphism where

$$x_1^{\ast }=x_4^{\ast }=\frac{1}{4}+D^{\ast },\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{x}_2^{\ast }=x_3^{\ast }=\frac{1}{4}-D^{\ast }.$$ (5.1)

Here $D^{\ast }=x_1^{\ast }{x}_4^{\ast }-x_2^{\ast }{x}_3^{\ast }$ is given by

$$D^{\ast }=\frac{r}{m}+\sqrt{\frac{1}{16}+{\left(\frac{r}{m}\right)}^2},$$ (5.2)

where m = –4ε₁/(α₂ + β₂ + ε₁) is negative when ε₁ > 0. We assume in what follows that ε₁ > 0, but similar analysis can be carried out if ε₁ < 0. Assume then thatm < 0 and hence D*> 0.

Since W* = W_add + ε₁Δ is a Wright-Kimura fitness matrix, $w_{12}^{\ast }=w_{34}^{\ast }$ and $w_{13}^{\ast }=w_{24}^{\ast }$. Thus it follows from (3.7) that in this case

$$A_2-A_1=rR{d}^{\ast },\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{B}_2-B_1=r(1-R)d^{\ast },\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{D}_1=D_2.$$

Also

$$A_1+A_2=2A_1+rR{d}^{\ast },\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{B}_1+B_2=2B_1+r(1-R)d^{\ast }$$

and since C₁ + C₂ = (1 – r)Rd* we can write

$$C_1+C_2=2C-(1-r)d^{\ast }$$ (5.3)

where C = C₁.

The computation of det(M̃) in (3.5) as given in Liberman and Feldman (2005) can be applied here to give

$$det(\stackrel{\sim }{\mathbf{M}})=T_1\times T_2,$$ (5.4)

where

$$T_1={\delta }^{\ast }(A_1+B_1)+r{d}^{\ast }\left({\delta }_1^{\ast }{x}_1^{\ast }+{\delta }_4^{\ast }{x}_4^{\ast }-2r{\delta }_{14}{D}^{\ast }\right)$$ (5.5)

and

$$T_2=(A_1+C)\left[(1-r)R-r(1-R)\right]d^{\ast }-(B_1+C)\left\{2(A_1+C)-\left[(1-r)R-rR\right]d^{\ast }\right\},$$ (5.6)

with T₂ < 0 for d* > 0 when R > 0. In the case at hand, $d^{\ast }=w_{14}^{\ast }{D}^{\ast }>0$. As $x_1^{\ast }=x_4^{\ast },\hspace{0.17em}{x}_2^{\ast }=x_3^{\ast }$, then from (3.8) in the case Δ = Δ̄, and using (5.1)

$${\overline{\delta }}^{\ast }=2\left(x_1^{\ast }-x_2^{\ast }\right)$$ (5.7)

$$=4D^{\ast }.$$ (5.8)

In addition

$${\overline{\delta }}_1^{\ast }{x}_1^{\ast }+{\overline{\delta }}_4^{\ast }{x}_4^{\ast }-2r{\delta }_{14}{D}^{\ast }=2{\overline{\delta }}_1^{\ast }{x}_1^{\ast }-2r{D}^{\ast }$$ (5.9)

and in fact

$${\overline{\delta }}_1^{\ast }{x}_1^{\ast }+{\overline{\delta }}_4^{\ast }{x}_4^{\ast }-2r{\delta }_{14}{D}^{\ast }=2\left[\frac{1}{8}+(1-r)D^{\ast }+2{(D^{\ast })}^2\right].$$ (5.10)

When R > 0, we have A₁ > 0 and B₁ > 0. Hence we conclude that

$$T_1={\overline{\delta }}^{\ast }(A_1+B_1)+r{d}^{\ast }\left({\overline{\delta }}_1^{\ast }{x}_1^{\ast }+{\overline{\delta }}_4^{\ast }{x}_4^{\ast }-2r{\delta }_{14}{D}^{\ast }\right)>0.$$ (5.11)

Since

$$Q(1)=\frac{({\epsilon }_2-{\epsilon }_1)}{x_1^{\ast }{x}_2^{\ast }{x}_3^{\ast }{x}_4^{\ast }}det(\stackrel{\sim }{\mathbf{M}})$$

and det(M̃) < 0 when R > 0, then Q(1) < 0 when ε₂ > ε₁ > 0 provided (ε₂ – ε₁) is small and R > 0. If m > 0 and D* < 0, it is not difficult to show that T₁ < 0, T₂ < 0. In this case ε₁ < 0 and M₂ increases if ε₂ < ε₁ and negative epistasis increases.

We can therefore conclude that if ε₁ > 0 and (ε₂ – ε₁) is small and R > 0, a modifier allele that increases epistasis will invade the population for all values of r > 0. In fact the above result also holds when r = 0 but requires a separate analysis since when r = 0, $D^{\ast }=\frac{1}{4}$, and $x_2^{\ast }=x_3^{\ast }=0$. The analysis is given in Appendix A. We have thus secured the following result.

Result 2

Under the fitness scheme W = W_add + εΔ̄, a modifier allele that increases epistasis invades the population when the population is initially at the symmetric equilibrium given by (4.1), for any positive values of r and R.

The situation is different with the fitness scheme W = W_add + εΔ̂. In this case Δ̂ = 2Δ̄ – E and we have

$${\widehat{\delta }}^{\ast }=2{\overline{\delta }}^{\ast }-1=8D^{\ast }-1$$ (5.12)

as δ̄* = 4D*. Moreover

$${\widehat{\delta }}_1^{\ast }={\widehat{\delta }}_4^{\ast }=2(x_1^{\ast }-x_2^{\ast })=4D^{\ast }.$$ (5.13)

So

$${\widehat{\delta }}_1^{\ast }{x}_1^{\ast }+{\widehat{\delta }}_4^{\ast }{x}_4^{\ast }-2r{\delta }_{14}{D}^{\ast }=8D^{\ast }\left(\frac{1}{4}-D^{\ast }\right)-2r{D}^{\ast }=2D^{\ast }(1-r)+8{(D^{\ast })}^2.$$ (5.14)

Hence T₁ of (5.5) is given by

$$T_1=(8D^{\ast }-1)(A_1+B_1)+2r{w}_{14}^{\ast }{D}^{\ast }\left[(1-r)+4D^{\ast }\right].$$ (5.15)

T₁ is positive when $D^{\ast }>\frac{1}{8}$. As m < 0, it is easily seen that D* given in (5.2) is a decreasing function of r. Therefore there must exist a value of r, which we denote as r_crit, such that $D^{\ast }>\frac{1}{8}$ is equivalent to r < r_crit, and we have the following result.

Result 3

Under the fitness scheme W = W_add + εΔ̂ , when the population is at the symmetric equilibrium (5.1) with D* > 0 (and hence d* > 0), a modifier allele that increases epistasis can invade the population if the recombination rate r between the two major loci is smaller than a critical value r_crit.

The above analysis secures Result 3 for both R and r positive. The same result holds for r = 0 and any R ≥ 0, as explained in Appendix A.

Remark 2

When m > 0 and D* < 0 (i.e., d* < 0), the argument is more complex. It can be shown that T₂ < 0 when d* < 0 andR > 0 and that T₁ < 0 for all r with $0<r<\frac{1}{2}$. Now m > 0 entails that ε₁ < 0, and therefore, from (3.4), Q(1) < 0 if ε₂ < ε₁; that is, the allele M₂ succeeds if it increases the negative epistasis.

6. A two-parameter class of Δ matrices

Consider the two parameter class Δ(s, t) of matrices defined as

$$\mathbf{\Delta }(s,t)=s\overline{\mathbf{\Delta }}-t\mathbf{E},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}s>0,\hspace{0.17em}t\ge 0.$$ (6.1)

Then Δ̄ = Δ(1, 0) and Δ̂=Δ(2, 1). For the class of fitness matrices of the form W = W_add + ε Δ(s, t) we have the following general result.

Result 4

Let the fitness scheme W be of the form W = W_add + εΔ(s, t) such that W is of the Wright-Kimura form. Suppose the population is at a symmetric equilibrium with positive linkage disequilibrium (D* > 0) and is fixed on the allele M₁ at the modifier locus with associated epistasis ε₁ > 0. Then a new allele M₂ with epistasis ε₂ > ε₁ and (ε₂ – ε₁) small will invade if R > 0 provided that D* > t/(4s) for all 0 ≤ r < r_crit and s > t.

For the proof, note that

$${\overline{\delta }}^{\ast }(s,t)=4s{D}^{\ast }-t,$$ (6.2)

$${\overline{\delta }}_1^{\ast }(s,t)={\overline{\delta }}_4^{\ast }(s,t)=\frac{s-2t}{2}+2s{D}^{\ast }.$$ (6.3)

As $x_1^{\ast }=x_4^{\ast }=\frac{1}{4}+D^{\ast }$we have

$${\overline{\delta }}_1^{\ast }(s,t)x_1^{\ast }+{\overline{\delta }}_4^{\ast }(s,t)x_4^{\ast }-2r(s-t)D^{\ast }=\frac{s-2t}{4}+2(s-t)(1-r)D^{\ast }+4s{(D^{\ast })}^2.$$ (6.4)

When r > 0, from (2.7) (A₁ + B₁) > 0. Also as $D^{\ast }(D^{\ast }<\frac{1}{4})$ is a decreasing function of r, and as s > t, that is, $\frac{t}{4s}<\frac{1}{4}$, there is an r_crit such that 4sD* – t > 0 when r < r_crit. Therefore when $D^{\ast }>\frac{t}{4s}$, following (6.4) we have

$$\frac{s-2t}{4}+4s{(D^{\ast })}^2>\frac{s-2t}{4}+\frac{t^2}{4s}=\frac{{(s-t)}^2}{4s}.$$ (6.5)

So in fact

$$T_1={\overline{\delta }}^{\ast }(s,t)(A_1+B_1)+r{w}_{14}^{\ast }{D}^{\ast }\left[{\overline{\delta }}_1^{\ast }(s,t)x_1^{\ast }+{\overline{\delta }}_4^{\ast }(s,t)x_4^{\ast }-2r(s-t)D^{\ast }\right]$$ (6.6)

is positive when 0 < r < r_crit. Thus as T₂ is always negative, det(M̃) = T₁· T₂ < 0, which proves our result.

As before, if ε₁ < 0 and D* < 0, then M₂ invades if ε₂ < ε₁ < 0 so that negative epistasis becomes stronger.

In Appendix A we show that in the case r = 0 the four eigenvalues for the local stability to invasion by M₂ can be obtained explicitly, without the assumption that (ε₂ – ε₁) is small. In fact, lim_R→0 lim_r→0 Q(1; r, R) = O(ε₂ –ε₁)² , and because we have neglected terms O(ε₂ –ε₁)² and higher order in (ε₂ – ε₁)_, using Q(1) for R = 0 will be misleading. Appendix A shows that when r = 0, if s > t then M₂ increases when rare if ε₂ > ε₁ > 0.

7. Discussion and numerical treatment

As pointed out by Hansen et al. (2006), the evolution of epistasis is not part of the corpus of classical population genetics. The statistical framework, exemplified by the recent work of Hansen et al., defines epistasis with respect to a reference genotype. This is difficult to define in our context, although at a viability-analogous Hardy-Weinberg equilibrium (Liberman and Feldman 2005) an average epistasis has a meaning and might serve as a reference value. Indeed, in Liberman and Feldman (2005) it was shown that for the Lewontin-Kojima fitness scheme, increase of epistasis relative to this average occurred.

Linkage disequilibrium is important in our analysis and for the results. This appears not to be the case in analysis of statistical epistasis (Barton and Turelli 2004, Carter et al. 2005, Hansen et al. 2006). Again, this is because we focus on initial perturbation of functional epistasis. Further, linkage is expected to be important in our analysis, at least for initial rates of increase. It turns out from our results to be qualitatively even more important as it may determine the direction of change of epistasis. Again, this appears not to be the case for statistical epistasis. Reconciliation of results obtained for evolution of statistical and functional epistasis remains to be achieved.

A possible explanation for the increase of epistasis seen in Liberman and Feldman (2005) is that the mean fitness is an increasing function of the amount of epistasis. We have examined the mean fitness in the Wright-Kimura model studied here as a function of ε when the interaction matrix is either Δ̂ or Δ̄ In the former case we can write the equilibrium value of the mean fitness W̄ in the case of Δ̂ as

$$\overline{W}=\frac{{\alpha }_1+{\beta }_1+{\alpha }_2+{\beta }_2}{2}-\stackrel{\sim }{r}-\epsilon +\sqrt{4{\epsilon }^2+{\stackrel{\sim }{r}}^2},$$ (7.1)

where r̃ = r(α₂ + β₂ + ε). Then

$$\frac{\partial \overline{W}}{\partial \epsilon }=(4\epsilon +r\stackrel{\sim }{r}){\left(4{\epsilon }^2+{\stackrel{\sim }{r}}^2\right)}^{-1/2}-1-r,$$ (7.2)

which is positive at r = 0 and negative at $r=\frac{1}{2}$. The value of r where (7.2) is zero is called r_Wcrit and can be obtained as a power series in ε/(α₂ + β₂ + ε). In the case of Δ̄ however, we can show that no such critical value exists for $r\in [0,\frac{1}{2}]$.

Of course we are interested in comparing r_crit and r_Wcrit in the case Δ̂, and as can be of seen in the numerical examples of the supplemental material, which is available online at http://charles.stanford.edu/supplementary/, the two critical values of the recombination rate are different. This suggests that the underlying reason for increase of epistasis for r < r_crit (or decrease when r > r_crit) is not a corresponding increase in the mean fitness. In examples I–IV of the supplemental materials, for which ε₁ > 0, we see that r_Wcrit > r_crit, although we have not shown that with interaction matrix Δ̂ this inequality always holds.

The supplemental material includes eight numerical examples: in examples I–V the interaction matrix isΔ̂ (equ. 2.3) and we take ε₁ > 0. Example VI was chosen to conform to the quadratic optimum construction of Burger (2000) for which the initial equilibrium is asymmetric. Example VII has interaction matrixΔ̄(equ. 2.2). Example VIII is chosen to illustrate a type of polymorphism of M₁ and M₂ discussed by Liberman and Feldman (2005). In each case the epistasis-modifying allele M₂ was introduced near the equilibrium with M₁M₁ and epistasis ε₁, with M₁M₂ and M₂M₂ producing epistasis values ε₂ and ε₃, respectively. For examples I–IV and VII, ε₁>0 while ε₁= 0 in example V and ε₁<0 in examples VI and VIII. The additive fitness matrices W_add for examples I, II, V, and VII are the same, as are those for examples III and IV. In all cases these additive matrices have α₁= α₃ and β₁= β₃ producing matrices W of the Wright-Kimura form.

The supplemental tables summarize the evolution at the M₁/M₂ locus. Tables I, V, VIII, and XII record the final frequency vectors x̂_M1 and x̂_M2 of the M₁- and M₂- linked chromosomes, the largest eigenvalue, λ, of the local stability matrix near fixation of M₁, the final mean fitness W̄_f and final linkage disequilibrium D̂, at R = 0 for examples I–IV, respectively. Tables II, VI, IX, and XIII correspond to examples I–IV, respectively, and show λ as a function of r and R. It appears that λ is monotonic in both R and r with the direction of monotonicity determined by whether ε₂ is greater or less than ε₁. For example, when 0 < ε₂ < ε₁, λ is a monotone decreasing function of r, and when ε₂ <ε₁ > 0 it increases in r.

In example I, ε₂ = 0.2 >ε₁ = 0.1. From Table I we see that λ> 1 for r < r_crit = 0.043121, and M₂ finally fixes, while for r > 0.043121, M₂ is lost. In example II, ε₂ = 0.05 <ε₁ = 0.1, and in Table V we see that for r < r_crit = 0.028842, M₂ is lost and M₁ fixes while for r > 0.028842, M₂ fixes. Because the additive components of the fitnesses are the same in examples I and II, these results suggest that r_crit is dependent on ε₂ and probably on (ε₂ – ε₁). The pair of examples III and IV is analogous to the pair I and II and illustrates similar behaviors. The quantity r_crit is defined as the r value at which λ = 1, but in general we cannot find r_crit explicitly. Tables IV, VII, X, and XIV record as r_crit the value of r that numerically yields λ closest to 1. It appears from the numerical work that r_crit is monotone decreasing in R.

Table III shows the dependence of λ on ε₂ for different values of r at R = 0 for examples I and II. For ε₂ > ε₁ > 0, λ starts from a value larger than 1 and decreases below 1 as r is increased, while for ε₂ < ε₁, λ starts below 1 and increases above 1 as r is increased, confirming that the sign of λ-1 is indeed determined by the sign of (ε₂ - ε₁) (and r of course). Table IX is the analogue of Table III for fitness examples III and IV.

From the previous two paragraphs, we may speculate that if $\frac{1}{2}>r$, R ≥ 0, the form of λ is

$$\lambda =1+({\epsilon }_2-{\epsilon }_1)g(r,R,{\epsilon }_1,{\epsilon }_2)$$ (7.3a)

with g(r, R, ε₁, ε₂) such that

$$g(0,0,{\epsilon }_1,{\epsilon }_2)>0,$$ (7.3b)

$$\frac{\partial g}{\partial r}<0,\frac{\partial g}{\partial R}<0,\frac{{\partial }^{2}g}{\partial r\partial R}>0.$$ (7.3c)

In example V we chose ε₁ = 0, which ensures that the initial equilibrium with M₁ fixed has D* = 0. In this case, with interaction matrix Δ̂, the initial allele frequencies at both loci are equal and for r > 0, M₂ cannot invade if it increases epistasis (from inequalities 4.16). At r = 0 the leading eigenvalue for invasion by M₂ is 1, but the initial small numerical perturbation places the frequencies in the domain of attraction to the high complementarity equilibrium (0.5, 0, 0, 0.5) with M₂ fixed, since M₂M₂ has the largest value of epistasis.

Example VI produces asymmetric equilibria for Bürger’s (2000) quadratic optimum model. This requires a very delicate choice of parameters because, as shown by Bürger, the set of parameters producing a stable asymmetric equilibrium is small. In this case ε₁ < 0 and as seen in Table XVIIM₂ increases if it produces more negative epistasis.

Example VII is included as a case where the interaction matrix is Δ̄(equ. 2.2). Here, with ε₃ > ε₂ > ε₁ > 0, λ > 1 for all r, R, as shown in the analysis.

Supplemental example III has the heterozygote M₁M₂ producing the greatest positive epistasis. We expect M₂ to increase when rare provided r < r_crit, and we see in Tables XIII and IX that this is indeed the case. We would also expect M₁ to increase when rare for $r<r_{\text{crit}}^{\ast }$ with $r_{\text{crit}}^{\ast }$ different from r_crit because ε₁ is different from ε₃. This explains the coexistence of M₁ and M₂, which we expect to occur if $r<min(r_{\text{crit}},r_{\text{crit}}^{\ast })$.

The final example VIII has negative epistasis with ε₂ produced by M₁M₂ being less than both ε₁ and ε₃. The interaction matrix here is Δ̂, and in this case there is no r_crit < 0.5. Thus, for all r and R we expect, and indeed obtain, polymorphism. The form of the polymorphism is viability analogous Hardy-Weinberg (VAHW; Liberman and Feldman 1986, 2005) with the ratio of the frequencies x̂_i,M₁ and x̂_i,M₂ is the same for all i and equal to (ε₂ - ε₃)/(2ε₂ -ε₁ -ε₃).

The Wright-Kimura fitness scheme discussed here allows considerable flexibility in the choice of the Δ matrix. We can say that an epistasis-increasing allele will invade, at least if the linkage between the loci whose interactions it controls is sufficiently tight. This distinguishes the Wright-Kimura symmetric scheme from the Lewontin-Kojima model for which there appears to be no constraint on this linkage for invasion by an allele that increases epistasis and that is introduced near a stable state with linkage disequilibrium (Liberman and Feldman 2005).

Since the evolution of epistasis may depend on the linkage between the interacting genes, and on their linkage disequilibrium, we suggest that the problems of modularity and evolvability are more complex than hitherto thought. The interaction between the fitness elements of W precludes us from saying that the selection is directional. In fact, some degree of overdominance exists in the basic fitness scheme that maintains polymorphism in the first place. We do not include mutation in these analyses. Discussion of epistasis, canalization, and evolvability usually hinge on the sensitivity of the fitness of a mutant to the occurrence of one or more further mutations. In our system of fitnesses, changing of A to a or B to b has little effect because the fitnesses are symmetric. It would be of great interest to extend this class of analyses to asymmetric schemes where such substitutions might have an effect. One could then compare this class of models more directly with mutation-selection-recombination systems with respect to evolvability and canalization. Our results suggest that evolution towards modularity is a delicate matter of the interaction between selection, linkage, and history. Starting near D = 0 may produce entirely different trajectories for epistasis from those starting near D ≠ 0, and as shown here these trajectories may depend on r.

Appendix A: The case r ≈ 0

When r = 0, the linear approximation determining the external stability of the equilibrium $x_1^{\ast }=x_4^{\ast }=\frac{1}{2},\hspace{0.17em}{x}_2^{\ast }=x_3^{\ast }=0$ is given in equations (4.7) in Liberman and Feldman(2005) and has a matrix of the form

$$\left|\begin{array}{cccc}{S}_1& S_2& S_3& S_4\\ 0& U& 0& 0\\ 0& 0& V& 0\\ T_1& T_2& T_3& T_4\end{array}\right|.$$ (A.1)

Also when the fitness matrix is a Wright-Kimura symmetric fitness matrix, then

$$S_1=T_4,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{S}_2=T_3,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{S}_3=T_2,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{S}_4=T_1.$$ (A.2)

It is easily seen that the four eigenvalues associated with (A.1) are

$${\lambda }_1=U,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\lambda }_2=V,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\lambda }_3=S_1+S_4,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\lambda }_4=S_1-S_4.$$ (A.3)

Now using (4.7) in Liberman and Feldman (2005) we find

$$\begin{array}{l}{w}^{\ast }U=\left(\frac{1}{2}{\overline{w}}_{21}+\frac{1}{2}{\overline{w}}_{24}\right)(1-R),\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{w}^{\ast }V=\left(\frac{1}{2}{\overline{w}}_{31}+\frac{1}{2}{\overline{w}}_{34}\right)(1-R)\\ w^{\ast }{S}_1=\frac{1}{2}{\overline{w}}_{11}+\frac{1}{2}{\overline{w}}_{14}(1-R),\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{w}^{\ast }{S}_4=\frac{1}{2}{\overline{w}}_{14}R.\end{array}$$ (A.4)

If W*= W_add + ε₁Δ (s, t), W̄ = W_add + ε₂Δ (s, t), then referring to (4.2) and (4.5) of Liberman and Feldman (2005) we have

$$\begin{array}{l}U=\left[\frac{\frac{1}{2}{w}_{21}^{\ast }+\frac{1}{2}{w}_{24}^{\ast }}{w^{\ast }}+\frac{{\epsilon }_2-{\epsilon }_1}{2w^{\ast }}({\delta }_{21}+{\delta }_{24})\right](1-R)\\ V=\left[\frac{\frac{1}{2}{w}_{31}^{\ast }+\frac{1}{2}{w}_{34}^{\ast }}{w^{\ast }}+\frac{{\epsilon }_2-{\epsilon }_1}{2w^{\ast }}({\delta }_{31}+{\delta }_{34})\right](1-R)\\ S_1+S_4=\frac{\frac{1}{2}{w}_{11}^{\ast }+\frac{1}{2}{w}_{14}^{\ast }}{w^{\ast }}+\frac{{\epsilon }_2-{\epsilon }_1}{2w^{\ast }}({\delta }_{11}+{\delta }_{14})\\ S_1-S_4=\frac{\frac{1}{2}{w}_{11}^{\ast }+\frac{1}{2}{w}_{14}^{\ast }(1-2R)}{w^{\ast }}+\frac{{\epsilon }_2-{\epsilon }_1}{2w^{\ast }}\left[{\delta }_{11}+{\delta }_{14}(1-2R)\right].\end{array}$$ (A.5)

Now $\frac{1}{2}{w}_{k1}^{\ast }+\frac{1}{2}{w}_{k4}^{\ast }=w_k^{\ast }$ for k = 1, 2, 3, 4 and $w_1^{\ast }=w_4^{\ast }=w^{\ast }$ where $w_2^{\ast },w_3^{\ast }<w^{\ast }$ because of internal stability. Thus since δ₂₁ = δ₂₄ = δ₃₁ = δ₃₄ = -t, δ₁₁ = δ₁₄ = (s – t), we have

$$\begin{array}{ll}{\lambda }_1=\left(\frac{w_2^{\ast }}{w^{\ast }}-t\frac{{\epsilon }_2-{\epsilon }_1}{w^{\ast }}\right)(1-R),\hfill & {\lambda }_2=\left(\frac{w_3^{\ast }}{w^{\ast }}-t\frac{{\epsilon }_2-{\epsilon }_1}{w^{\ast }}\right)(1-R)\hfill \\ {\lambda }_3=1+\frac{{\epsilon }_2-{\epsilon }_1}{w^{\ast }}(s-t),\hfill & {\lambda }_4=1-\frac{R{w}_{14}^{\ast }}{w^{\ast }}+\frac{{\epsilon }_2-{\epsilon }_1}{w^{\ast }}(s-t)(1-R).\hfill \end{array}$$ (A.6)

From (A.4) we know that all eigenvalues are positive. If s > t, then when ε₂ > ε₁ > 0, λ₃> 1, and when ε₂ < ε₁ with |ε₂ - ε₁| small all eigenvalues are positive and less than one. Thus, when r = 0 and s > t, a modifier allele that increases epistasis is favored, whereas a modifier allele that reduces epistasis cannot invade the population. As the eigenvalues are continuous functions of r, the same result holds for r ≈ 0 for any $0\le R\le \frac{1}{2}$ provideds > t. Thus Results 3 and 4 are true for any $0\le R\le \frac{1}{2}$ and 0 ≤ r < r_crit where r_crit depends on the interaction matrix Δ.