The effect of gene interactions on the long-term response to selection

Significance

The role of gene interactions in the response to selection has long been a controversial subject; whereas some have claimed they are not relevant for adaptation, others have argued that their long-term effects are of high significance. In this manuscript, we derive simple and general predictions for the effect of gene interactions on the long-term response to selection in two extreme regimes. We show that, when the dynamics of allele frequencies are dominated by genetic drift, the long-term response is surprisingly simple, depending only on the initial components of the trait variance, regardless of the detailed genetic architecture. In the opposite regime, when selection dominates the dynamics of allele frequencies, the long-term response depends only on the genotype−phenotype.

Abstract

The role of gene interactions in the evolutionary process has long been controversial. Although some argue that they are not of importance, because most variation is additive, others claim that their effect in the long term can be substantial. Here, we focus on the long-term effects of genetic interactions under directional selection assuming no mutation or dominance, and that epistasis is symmetrical overall. We ask by how much the mean of a complex trait can be increased by selection and analyze two extreme regimes, in which either drift or selection dominate the dynamics of allele frequencies. In both scenarios, epistatic interactions affect the long-term response to selection by modulating the additive genetic variance. When drift dominates, we extend Robertson’s [Robertson A (1960) Proc R Soc Lond B Biol Sci 153(951):234−249] argument to show that, for any form of epistasis, the total response of a haploid population is proportional to the initial total genotypic variance. In contrast, the total response of a diploid population is increased by epistasis, for a given initial genotypic variance. When selection dominates, we show that the total selection response can only be increased by epistasis when some initially deleterious alleles become favored as the genetic background changes. We find a simple approximation for this effect and show that, in this regime, it is the structure of the genotype−phenotype map that matters and not the variance components of the population.

The relation between an organism’s genotype and its phenotype is immensely complicated, yet quantitative genetics predicts the correlations between relatives, and the response to selection over tens of generations, based primarily on an additive model. How do interactions between genes affect the response to selection? This question is not easy to answer: Although the additive model can be represented by a few parameters, there are an enormous number of possible relationships between genotype and phenotype. Some insight may come from studying specific well-understood systems, for example, regulation of gene expression by binding of transcription factors, or folding of RNA molecules. Here, we take the alternative approach, seeking statistical regularities, while making minimal assumptions about the nature of epistasis.

There has been a long-standing debate about the effect of genetic interactions on adaptation (1–7). Although some claim that they are unimportant because they contribute very little to the total genetic variance of a population, and consequently to its short-term response, others claim that their long-term effects can be substantial. At the heart of the debate is the distinction between what has been called physiological and statistical epistasis. The former is independent of the state of the population, whereas the latter is the statistical contribution of gene interactions to the trait variance, which depends on the allele frequencies (8, 9).

In the absence of new mutations, the total response to selection is limited by the initial standing variation. Selection acts on the variation present in the population, and will typically act to reduce it; the extent to which it does so depends on the allele frequencies, and on the relation between genotype and phenotype (i.e., the “genetic architecture”). Interactions between genes (epistasis) are a key component of this genetic architecture, and their effect on the response to selection has been controversial. On the one hand, artificially selected populations are usually well approximated by the infinitesimal model, which, in its simplest form, assumes infinitely many loci of small and additive effect (10, 11). This suggests that gene interactions do not play an important role. On the other hand, biology can hardly be additive, and genomes are finite. If the trait depended on a small number of additive alleles, these would quickly fix—contradicting the robust observation of sustained response to selection. One possibility is that epistasis sustains additive genetic variance for longer: Alleles that were initially deleterious or near-neutral may acquire favorable effects as the genetic background changes, “converting” epistatic variance into additive, and so prolonging the response to selection.

Note that, if the map between genotype and phenotype were arbitrary, no predictions could be made: The fittest genotype could have any value, and genotypes can be organized in any way over the fitness landscape. We will make a statistical argument, which assumes that epistasis primarily involves low-order interactions, and that these can be treated as, to some degree, independent of each other. In particular, we assume that there is no systematic tendency for alleles with a positive marginal effect to interact positively (or negatively), on average; if there were, the rate of evolution would clearly accelerate (or decelerate).

Although much effort has been spent measuring the sign of epistasis among new mutations (12, 13), or among segregating variants (14–16), it is not clear that this would tell us about the natural pattern over a wider range. A population under selection can reach trait values far beyond the initial trait variance it displays, so the epistasis among variants in the existing population may not predict longer-term evolution. Here, we analyze two extreme limits: the limit of very weak selection, in which drift dominates the dynamics of the genetic variances, and the deterministic limit, where selection is the only force. We investigate the effect of epistasis on the long-term response of a population, and the mechanistic basis for these effects.

Results

The Interaction Between Epistasis and Random Drift.

Robertson (17) showed that, in the weak selection limit $\left(N_{e}s\ll 1\right)$, where $N_e$ is the effective population size and s is the selection coefficient at a single locus, the total response of an additive trait in a finite diploid population is simply proportional to the additive variance initially present, provided there is no dominance. He used the fact that the probability of fixation of an allele can be written as $u\left(p_0\right)=\left(1-\exp \left(-4N_{e}s{p}_0\right)\right)/(1-\text{exp}\left(-4N_{e}s\right)$) = $p_0+2p_0\left(1-p_0\right)N_{e}s+O\left({\left(N_{e}s\right)}^2\right)$. The total expected response, summed over all generations until every allele is lost or fixed, is $R^{\infty }={\displaystyle {\sum }_i{\alpha }_i\left(u\left(p_0\right)-p_0\right)}$, where ${\alpha }_i$ is the additive effect of each allele on the trait. When selection is weak relative to random drift, this can be approximated as $R^{\infty }\approx$ $2{\displaystyle {\sum }_i{\alpha }_i{p}_0\left(1-p_0\right)N_e{s}_i}$. If the selection gradient on the trait is β, the selection favoring an allele with effect ${\alpha }_i$ is $s_i=\beta {\alpha }_i$, and so $R^{\infty }\approx 2{\displaystyle {\sum }_i{\alpha }_i^2{p}_0\left(1-p_0\right)\beta N_e}=2\beta N_e{V}_A=2N_e{R}^0$, where $V_A^0={\displaystyle {\sum }_i{\alpha }_i^2{p}_0\left(1-p_0\right)}$ is the initial additive genetic variance, and $R^0=\beta V_A^0$ is the response in the first generation.

This calculation is valid in the limit of weak selection at every locus, reflected in the fact that the probability of fixation is approximated by the first-order perturbation to neutrality. It is also only strictly valid when the trait is additive, excluding any form of interaction between genes. Indeed, it seems very hard to generalize to allow for epistasis, because, then, the calculation of the fixation probability of an allele depends on the trajectories of the alleles at all other loci it interacts with.

An alternative derivation is possible, which focuses on the change in variance components due to drift. Consider an additive trait that is determined by very many loci, each under negligibly weak selection. Drift will disperse allele frequencies, decreasing the additive variance by a factor $\left(1-1/2N_e\right)$ per generation. Therefore, the total selection response is just $R^{\infty }={\displaystyle {\sum }_{t=0}^{\infty }\beta V_A^0{\left(1-1/2N_e\right)}^t}$ = $\beta 2N_e{V}_A^0$, as before. Both arguments show that the total response is proportional to the initial additive variance in the population (which, for an additive genetic architecture with no dominance, where heterozygotes display the mean phenotype of the homozygotes, corresponds to the total initial genetic variance). This is to be expected, because both assume weak selection at individual loci, so that the variance declines primarily through random drift.

What is the effect of epistasis on the total response of the population? Classical quantitative genetics (e.g., ref. 18) uses statistical arguments to derive expressions for the expected components of genetic variance in an inbred population. Barton and Turelli (19) derive these using an explicit population genetic model, assuming two alleles per locus; Hill et al. (20) show how these are related to the classical expressions, and that they apply without any constraint on the distribution of allelic effects at each locus. In the following, we make use of these results to derive expressions for the expected long-term response of populations with arbitrary genetic architectures. We first focus on haploid populations and then extend our results to diploid populations.

The Long-Term Response for Haploids.

Suppose that a haploid population initially has additive genetic variance $V_A$, pairwise epistatic variance $V_{AA}$, and so on for higher-order epistatic components; it is convenient to denote the kth order variance component as $V_{A\left(k\right)}$. Throughout, we will assume that the population is always at linkage equilibrium (such that $\langle p_i^{t+1}{p}_j^{t+1}\rangle =\langle p_i^{t+1}\rangle \langle p_j^{t+1}\rangle$, where the expectation $\langle \cdot \rangle$ is taken over multiple realizations of the process). If the expected change in allele frequency due to selection at any locus ($\langle \mathrm{\Delta }{p}_i\rangle =\beta A_i{p}_i^t{q}_i^t$, where β is the selection gradient and $A_i$ is the effect of locus i on the trait) is small compared with its variance ($\text{Var}\left(p_i^{t+1}\right)=\langle {\left(p_i^{t+1}\right)}^2\rangle -{\langle p_i^{t+1}\rangle }^2=p_i{q}_i/N$; according to the Wright−Fisher model), we can assume that the dynamics of the variance components are dominated by drift. This does not necessarily mean that selection is weak on the trait; if the trait is affected by many loci, these conditions can be met at any individual locus even with strong selection on the trait (the infinitesimal model). Under these conditions, after an arbitrary period of inbreeding, the expected additive variance of a haploid population is

$$\begin{array}{c}\langle V_A^t\rangle =\left(1-F_t\right)\left(V_A^0+2F_t{V}_{AA}^0+3F_t^2{V}_{AAA}^0\dots \right)\\ =\left(1-F_t\right){\displaystyle \sum _{k=1}^{\infty }k{F}_t^{k-1}{V}_{A\left(k\right)}^0}\end{array}$$

where $F_t$ is the probability of identity between two genes at a locus (equation 29 of ref. 19, but see also Dynamics of Variance Components Under Genetic Drift). Assuming a uniform rate of inbreeding, so that $1-F_t={\left(1-1/N_e\right)}^t$, the total response to directional selection β is

$$\begin{array}{c}\langle R^{\infty }\rangle =\beta {\displaystyle \sum _{t=0}^{\infty }{V}_A^t}=\beta {\displaystyle \sum _{t=0}^{\infty }\left(1-F_t\right)}{\displaystyle \sum _{k=1}^{\infty }k{F}_t^{k-1}{V}_{A\left(k\right)}^0}\\ =\beta N_e{\displaystyle \sum _{k=1}^{\infty }{V}_{A\left(k\right)}^0}=\beta N_e{V}_G^0.\end{array}$$ [1]

Remarkably, the total response to directional selection is proportional to the initial total genotypic variance, $V_G^0={\displaystyle {\sum }_{k=1}^{\infty }{V}_{A\left(k\right)}^0}$—a simple extension of Robertson’s (17) result. The change in mean can thus be very much greater than the change in the first generation, if $N_e$ is large (Fig. 1). For a given initial total genetic variance, $V_G^0$, the initial response is slower with epistasis, because it is proportional only to the additive component, $V_A^0$. However, as allele frequencies fluctuate randomly, epistasis generates additional additive variance, such that the total selection response is the same. For a given initial genotypic variance, epistasis slows the initial response, but does not affect its final value, whereas, for a given initial additive variance, epistasis increases the final response by $V_G^0/V_A^0$, compared with an additive genetic architecture.

Fig. 1.

Long-term trait value reached by finite populations with a pairwise genetic architecture as a function of the initial fraction of epistatic variance $V_{AA}/V_G^0$, for different population sizes as indicated. For each population size, we increase the variance in interaction strength, keeping the variance of main effects constant at ${\sigma }_{\alpha }=1/10$, from ${\sigma }_{\mathit{ϵ}}={10}^{-3}$ to ${\sigma }_{\mathit{ϵ}}=1/2$, resulting in an initial genetic variance ranging from $V_G=2$ to $V_G\approx 3$. Initial epistatic variance saturates with the strength of interactions (see Calculating Expectations for the Initial Components of Variance). Lines represent the theoretical prediction from Eq. 1. Selection strength was set to $\beta ={10}^{-3}$. Circles represent means of ${10}^4$ simulations, and error bars (too small to be seen) represent the SE.

Our result is based on the dynamics of genetic variances, but Robertson’s (17) result (reproduced at the beginning of the section) suggests an alternative view: As alleles fluctuate in frequency, they change the average effects of alleles at other loci, thereby changing their probability of fixation. In particular, as we show in The Interaction Between Strong Directional Selection and Epistasis, alleles that are initially deleterious but become beneficial as the response unfolds should contribute to this increase in long-term response. This view predicts that an increase in long-term response should be correlated with an increase in the number of alleles that are beneficial at the end of the response, which we confirmed through simulations (Fig. S1).

Fig. S1.

Average increase in the number of alleles that are beneficial: mean number of alleles that switch sign from negative to positive, calculated from the same simulations as in Fig. 1.

The Long-Term Response for Diploids.

For a diploid population, with no dominance (meaning that the phenotype of the heterozygote is simply the mean of the homozygous phenotypes), epistasis has a stronger effect. From equation 45 of Barton and Turelli (19),

$$\begin{array}{c}\langle V_A^t\rangle =\left(1-F_t\right)\left(V_A^0+2\left(2F_t\right)V_{AA}^0+3{\left(2F_t\right)}^2{V}_{AAA}^0\dots \right)\\ =\left(1-F_t\right){\displaystyle \sum _{k=1}^{\infty }k{\left(2F_t\right)}^{k-1}{V}_{A\left(k\right)}^0}.\end{array}$$

Assuming that $1-F_t={\left(1-1/2N_e\right)}^t$, the total response to directional selection β is

$$\begin{array}{c}\langle R^{\infty }\rangle =\beta {\displaystyle \sum _{t=0}^{\infty }{V}_A}=\beta {\displaystyle \sum _{t=0}^{\infty }\left(1-F_t\right)}{\displaystyle \sum _{k=1}^{\infty }k{\left(2F_t\right)}^{k-1}{V}_{A\left(k\right)}^0}\\ =2N_e\beta {\displaystyle \sum _{k=1}^{\infty }{2}^{k-1}{V}_{A\left(k\right)}^0>2N_e\beta V_G^0}.\end{array}$$

Now, epistasis can increase the total response disproportionately. This is because we are calculating the contribution of epistatic components to the variance of a population that will ultimately be completely inbred—which amplifies the genetic variance contributed by each locus by a factor of 2 per order of interaction. (To see this, compare the variance of the trait in a diploid population at Hardy−Weinberg proportions, $q^2:2pq:p^2$, with that after complete inbreeding, $q:0:p$). However, it is unlikely that higher-order components will be large enough for this effect to be substantial (6). With two alleles per locus, the kth order variance is proportional to $\sim {\left(pq\right)}^k$, the product of allele frequencies across the interacting loci. Because $pq<1/4$, this decreases at least as $2^{-2k}$.

We have assumed constant directional selection, β, on the trait. However, the result readily extends to more complex forms of selection. Assuming that fitness increases monotonically with the trait value, we can transform to a scale on which log fitness depends linearly on the trait. On this new scale, the variance components will be different, but our general results show how the total response depends on these components, when measured on the appropriate scale.

The Interaction Between Strong Directional Selection and Epistasis.

We now turn to the deterministic case, where selection is the predominant process. Imagine that, in the initial haploid population, alleles at n loci have marginal effects ${\alpha }_i$, so that, with a selection gradient β, they have selective advantage $\beta {\alpha }_i$. We label the alleles by their initial effect, so that ${\alpha }_i>0$. If the trait is additive, and the population infinitely large, then, if selection remains constant, the fittest genotype $\left\{\mathrm{1,1},\dots \right\}$ will eventually fix, and the mean will increase by $R^{\infty }={\displaystyle {\sum }_i{\alpha }_i{q}_i^0}$, where $q_i=1-p_i$. However, with epistasis, the marginal effect of alleles will change through time. If these effects retain the same sign, the same final state will be reached, but it will have a different trait value. If some effects change sign, then a different genotype may be reached. There may still be a single global fitness peak, or there may be multiple peaks, so that the outcome is sensitive to the initial allele frequencies. However, we avoid considering this complication by simply comparing the final state of populations with and without epistasis. More precisely, we compare the total change in mean of an epistatic trait with the total change in mean of an additive trait, starting with the same allele frequencies and marginal effects.

Typically, the initial distribution of allele frequencies will be U-shaped, as expected for a population in a stationary distribution with low mutation rates per site (N_eμ < 1 for haploids). Thus, an enormous range of variation can be released as initially rare alleles increase, and are brought together by recombination. We will assume that the population is so large that alleles that are initially deleterious, but will ultimately be favored, will not be lost from the population. This is not realistic, but it is equivalent to assuming a low rate of recurrent mutation, such that alleles are continually reintroduced, so that, if they are favored, they will become common enough to increase deterministically. We assume that epistasis is not so strong that the marginal effects of alleles fluctuate more rapidly than the timescale needed for them to establish.

If most alleles are very rare or very common, they will spend a relatively short time passing from low to high frequency. Thus, we can caricature the process as a series of separate substitutions; at each substitution, the marginal effects of all of the other alleles change. This will change the time at which favorable alleles substitute, and may cause them to lose their advantage entirely. This caricature is not quite the same as the Strong Selection Weak Mutation approximation (21), in which populations are assumed to be fixed for a single genotype, and to evolve by fixation of new mutations, because rare alleles still follow a complex dynamic. In reality, substitutions may overlap, and their effects change continuously. This limiting scenario is helpful in understanding the following argument, but, because we focus on the final outcome, we do not need to assume that alleles substitute one by one. All we need to know is what is the difference between the final and initial effects of an allele.

Assuming only pairwise interactions, two alleles per locus, and linkage equilibrium throughout (see Materials and Methods), the ultimate change in mean is

$$R^{\infty }={\displaystyle \sum _i{\alpha }_i\left(p_i^{\infty }-p_i^0\right)}+\frac{1}{2}{\displaystyle \sum _{i\ne j}{\mathit{ϵ}}_{ij}\left(p_i^{\infty }{p}_j^{\infty }-p_i^0{p}_j^0\right)}.$$

At any time, allele i has selective advantage $s_i=\beta A_i$, where $A_i={\alpha }_i+{\displaystyle {\sum }_j{\mathit{ϵ}}_{ij}{p}_j}$ is the average effect of locus i on the trait mean that changes linearly with the frequencies of all of the other alleles (see Materials and Methods). Recall that we labeled alleles such that, initially, $A_i>0$ for all alleles. For simplicity, assume that the set $S_0$ of alleles starts at low frequency, and a set $S_1$ starts at high frequency. Therefore, $n_0$ alleles are expected to sweep from low to high frequency, thereby perturbing the selection on all other alleles. Provided that epistasis is sufficiently weak that no selection coefficients change sign, this set will sweep to fixation, and the difference between the response of an additive trait with the same initial effects (and therefore necessarily with the same initial additive variance) will be $\mathrm{\Delta }R={\displaystyle {\sum }_{ij\in S_0}{\mathit{ϵ}}_{ij}}$. Therefore, as long as epistasis is not directional, epistasis will, on average, make no difference to the total response. More precisely, provided that the total pairwise epistasis, summed over all pairs of alleles that sweep from low to high frequency, is zero, epistasis will have no net effect on the selection response.

This calculation is based on the assumption that additive effects cannot change sign; if interactions are strong enough to turn an initially beneficial allele, $k,$ into a deleterious one as frequencies at other loci change, the long-term response can be different, because the difference between the two architectures is now $\mathrm{\Delta }R=-{\alpha }_k+{\displaystyle {\sum }_{j\ne k}{\mathit{ϵ}}_{kj}}$. By assuming that the epistatic coefficients are independent random variables, we can approximate the probability that epistasis will change the ultimate genotype, and the expected change in trait mean that results.

The probability that an allele switches sign as the remaining favorable alleles increase is simply the probability that the sum of all interactions is less than the negative of its main effect: $p_{\text{flip}}=J_{\mathit{ϵ}}\left(-{\alpha }_k\right)={\displaystyle {\int }_{-\infty }^{-{\alpha }_k}{P}_{\mathit{ϵ}}\left(x\right)dx}$, where $J_{\mathit{ϵ}}$ is the cumulative probability distribution and $P_{\mathit{ϵ}}$ is the probability density of the sum of all epistatic interactions with the ith locus, and the $n_0$ loci that are sweeping from low to high frequency. We assume that the number of interactions is large enough, and that they are sufficiently independent, that this is approximately Gaussian: ε = ${\displaystyle {\sum }_{i\ne j\in S_0}{\mathit{ϵ}}_{ij}\sim \mathcal{N}\left(0,\sqrt{n_0}{\sigma }_{\mathit{ϵ}}\right)}$, where ${\sigma }_{\mathit{ϵ}}$ is the standard deviation (SD) of the ${\mathit{ϵ}}_{ij}$. Note that we only need to assume that the ${\mathit{ϵ}}_{ij}$ are drawn independently from a distribution with finite variance, so that their sum approaches a Gaussian. Note also that one of the $n_1$ alleles that is at high frequency will interact with all of the $n_0$ alleles that sweep from low to high frequency, whereas one of the $n_0$ sweeping alleles will only interact with the $n_0-1$ other alleles that are also sweeping. However, we assume that $n_0\gg 1$, and so approximate $n_0-1$ by $n_0$ throughout.

We can now calculate the average probability of an allelic reversal by integrating over the distribution of the main effects, $P_{\alpha }\left[\alpha \right]$,

$$\langle p_{\text{flip}}\rangle ={\displaystyle {\int }_0^{\infty }{J}_{\mathit{ϵ}}\left(-\alpha \right)P_{\alpha }\left(\alpha \right)d\alpha }.$$ [2]

This can be calculated explicitly for specific choices for $P_{\alpha }$,

$$\langle p_{\text{flip}}\rangle =\frac{1}{\pi }{\text{tan}}^{-1}\left(\mathrm{ℰ}\right)\hspace{1em}\hspace{1em}\hspace{1em}\left(\text{half}-\text{Gaussian}\right)$$

$$\langle p_{\text{flip}}\rangle =\frac{1}{2}\left(1-e^{{\mathrm{ℰ}}^2}\text{erfc}\left[\mathrm{ℰ}\right]\right)\hspace{1em}\hspace{1em}\hspace{1em}\left(\text{Exponential}\right)$$

where $\mathrm{ℰ}=\sqrt{n_0}{\sigma }_{\mathit{ϵ}}/{\sigma }_{\alpha }$ is the ratio of the SDs of the strength of genetic interactions versus the main effects, the relative “strength” of genetic interactions. These two functions necessarily increase linearly from zero for small $\mathrm{ℰ}$, and saturate at $1/2$ for large $\mathrm{ℰ}$; they differ by, at most, $4.4\text{\%}$. Comparing with numerical simulations, we see that this calculation captures the expected number of alleles that would be deleterious, compared with the original ultimate genotype (in which alleles remain positive throughout the evolutionary response) (Fig. 2). However, it assumes that all loci respond independently to the changing genetic background. In reality, as alleles switch sign and go extinct, other loci will be affected as well. When comparing to the actual global peak of the fitness landscape, we see that the actual number of alleles that change sign is less than predicted by our approximation of independence (Eq. 2). This is because, for each allele that reverses its sign, an interaction is effectively removed, thereby reducing $p_{\text{flip}}$ for the remaining loci. Numerically, we found that a good approximation to the expected number of allelic reversals is $1-p_{\text{flip}}\left(1-p_{\text{flip}}\right)$ of that assuming independence, independently of number of loci (Fig. 2A). This expression should converge to the one assuming independence at low values of $p_{\text{flip}}$, which we confirmed through simulations (Fig. 2A). Each of these allelic reversals (or “flips”) will translate into some change in trait value compared with an additive population. For each allele, this increase will be $\mathrm{\Delta }{z}_i={\displaystyle {\sum }_{j\ne i}{\mathit{ϵ}}_{ij}}$, and so the average contribution of each allele that switches sign will be

$$\langle \mathrm{\Delta }{\text{z}}_i\rangle =\frac{1}{J_{\mathit{ϵ}}\left(-\alpha \right)}{\displaystyle {\int }_0^{\infty }{\displaystyle {\int }_0^{-\alpha }\mathit{ϵ}{P}_{\mathit{ϵ}}\left(\mathit{ϵ}\right)P_{\alpha }\left(\alpha \right)d}\mathit{ϵ}d\alpha }.$$ [3]

For specific distributions of main effects,

$$\langle \mathrm{\Delta }{\text{z}}_i\rangle =\frac{1}{\langle p_{\text{flip}}\rangle }\frac{{\mathrm{ℰ}}^2{\sigma }_{\alpha }}{\sqrt{2\pi }\sqrt{1+{\mathrm{ℰ}}^2}}\hspace{1em}\hspace{1em}\hspace{1em}\left(\text{half}‐\text{Gaussian}\right)$$ [4]

$$\langle \mathrm{\Delta }{\text{z}}_i\rangle =\frac{1}{\langle p_{\text{flip}}\rangle }\frac{e^{{\mathrm{ℰ}}^2}\text{erfc}\left[\mathrm{ℰ}\right]{\mathrm{ℰ}}^2{\sigma }_{\alpha }}{\sqrt{2}}\hspace{1em}\hspace{1em}\hspace{1em}\left(\text{Exponential}\right).$$ [5]

These functions are again very similar, tending to zero for weak epistasis, and to $c{\sigma }_{\alpha }+\sqrt{2/\pi }\sqrt{n_0}{\sigma }_{\mathit{ϵ}}$ for large $\mathrm{ℰ}$, where $c=0.505$ for the half-Gausssian, and 0.444 for the exponential. Finally, the total trait increase due to allelic reversals is simply the expected number of alleles that reverse times the increase due to each of those,

$$\langle \mathrm{\Delta }R\rangle =n\langle p_{\text{flip}}\rangle \langle \mathrm{\Delta }{z}_i\rangle .$$ [6]

Substituting from Eqs. 2 and 4, we see that $\langle \mathrm{\Delta }R\rangle$ increases close to linearly with ${\sigma }_{\mathit{ϵ}}$, as $\sim n\sqrt{2/\pi }\sqrt{n_0}{\sigma }_{\mathit{ϵ}}$ (Fig. 2B). Note that, whereas only the $n_0$ loci that sweep from low to high frequency contribute (via epistasis) to changes in marginal effects, all $n=n_0+n_1$ loci may be affected by these changes. An initially favorable allele that is still at low frequency when it loses its advantage will never appear, whereas one that is already at high frequency will sweep back down to low frequency (recall that we assume either an extremely large population or recurrent mutation, so that deleterious alleles never get fixed). Either way, any of the n initially favored alleles that lose their advantage will be eliminated. The previous calculations are upper bounds for the effect of epistasis on the long-term response of a population, because we considered that $n_0$ loci have low initial allele frequencies. If initial allele frequencies are appreciable, part of the interaction effect is absorbed by the initial average effect of the loci, and alleles may not reverse their sign, but this can be seen as a reduction in $n_0$.

Fig. 2.

(A) Mean fraction of alleles that reverse their sign when all initially beneficial alleles go to fixation (black circles) or in the genotype of highest trait value (gray circles), as a function of the relative strength of genetic interactions $\mathrm{ℰ}=\sqrt{n_0}{\sigma }_{\mathit{ϵ}}/{\sigma }_{\alpha }$. The black line denotes the prediction assuming independence of alleles, and the gray dashed line is the same prediction multiplied by $1-p_{\text{flip}}\left(1-p_{\text{flip}}\right)$. (B) Expected increase in trait value if all of the initially beneficial alleles fix (black circles) or at the actual global peak (gray circles). Dashed line corresponds to the prediction obtained based on the gray line in A. All points correspond to the mean of 100 trials, randomizing the coefficients for the indicated variances for ${\mathit{ϵ}}_{ij}$ and ${\alpha }_i$ for $n=10$ loci.

When drift dominates, the ultimate response depends only on initial variance components. When selection dominates, the effect of epistasis on the ultimate response depends on reversals in selection on individual alleles, so that a different genotype is reached; this depends on the ratio of SDs of epistatic versus main effects, multiplied by the square root of the number of interacting alleles that sweep from low to high frequency ($\mathrm{ℰ}=\sqrt{n_0}{\sigma }_{\mathit{ϵ}}/{\sigma }_{\alpha }$), as we showed above. The transition between the two regimes depends only on the selection strength each locus experiences (Fig. 3); the trait can be under high selection strength, but, as long as it is highly polygenic, the prediction derived for the drift-dominated regime will hold (Fig. 3) because, for the same initial genetic variance, the selection coefficients scale as $n^{-1/2}$ with the number of loci. In the limit of infinite numbers of genes, this prediction would hold regardless of the selection gradient on the trait. In the selection-dominated regime, the distribution of allele frequencies is mostly irrelevant because the ultimate response depends on alleles that are initially vanishingly rare (or absent if we allow mutation). The key parameter is $\mathrm{ℰ}=\sqrt{n_0}{\sigma }_{\mathit{ϵ}}/{\sigma }_{\alpha }$, which is not constrained by the initial variance components: If alleles are at extreme frequency, the epistatic variance must be small, even if $\mathrm{ℰ}$ is large. However, we can relate this parameter to variance components in the F2 population formed by crossing the ancestral with the derived population (see Measuring Strength of Interactions from F2 Crosses and Fig. S2). This implies that the initial epistatic variance does not predict the long-term response of the population, because the population may contain strong interactions (large $\mathrm{ℰ}$) that are not manifested as epistatic variance (which depends on allele frequencies) but can contribute to the long-term response as allele frequencies reach appreciable levels.

Fig. 3.

The transition from the drift-dominated to the selection-dominated regimes. Stochastic simulations with increasing scaled selection ($N_e\beta$) showing the total trait increase in units of initial SD of trait value for $n=50$ (squares) and $n=\mathrm{1,000}$ (circles) loci. Dotted line corresponds to the drift-dominated prediction ($R=\beta N_e{V}_G^0$), and dashed lines correspond to the deterministic limit taken from numerical analysis, for an epistatic architecture (black) and an additive architecture (gray) of the same initial genetic variance. Initial allele frequencies were sampled from a beta distribution with mean $1/4$ and variance $1/16$, ${\sigma }_{\alpha }=1/10$, and ${\sigma }_{\mathit{ϵ}}=0.5$. Simulations were performed either with $\beta ={10}^{-2}$ or $\beta ={10}^{-5}$ and varying $N_e$ to yield the reported product $N_e\beta$.

Fig. S2.

The ratio of epistatic ($V_{AA}^H$) versus additive ($V_A^H$) variances of a population obtained by crossing an ancestral population with a derived population that differ at $n_0$ loci, as a function of the strength of genetic interactions $\mathrm{ℰ}$. Line corresponds to the prediction $\mathrm{ℰ}/\left[8\left(1+\mathrm{ℰ}\right)\right]$. Circles represent means of 1,000 trials of random pairwise genetic architectures with ${\alpha }_i$ sampled from a Gaussian distribution of zero mean and ${\sigma }_{\alpha }=1/10$ and ${\mathit{ϵ}}_{ij}$ sampled from a Gaussian distribution of zero mean and ${\sigma }_{\mathit{ϵ}}$ varied from $1/100$ to $1/2$.

Discussion

The role of epistasis in evolution has long been controversial. Wright (1) argued that epistasis would cause populations to become trapped at local “adaptive peaks” and proposed that a “shifting balance” between selection and random drift could allow them to explore alternative peaks, so as to move toward the global optimum. This theory motivated much work on the structure of natural populations, yet it remains unclear whether adaptation is significantly slowed by trapping on local peaks (1, 22). Mayr (23) criticized the supposed neglect of epistasis by “bean-bag genetics,” provoking a robust defense by Haldane (24). More recently, it has been proposed that epistatic variance can be “converted” into additive variance following a bottleneck, aiding adaptation (25). The failure of large genome-wide association studies to assign much heritable variance to specific loci (the so-called “missing heritability”) has been attributed to epistasis (26, 27), although this explanation is unnecessary (28). Overall, the practical success of the additive model in quantitative genetics appears hard to reconcile with the strong molecular interactions between genes.

We investigate how epistasis affects the response to selection, by asking a simple and clearly defined question: By how much does epistasis influence the ultimate change in the mean of a selected trait? We compare the effects of directional selection on two populations that initially have the same genetic variance for a trait; in one, inheritance is strictly additive, whereas, in the other, there can be strong gene interactions. We find simple results in the two extreme cases, where either drift or selection dominate.

In the first case, where drift is stronger than selection on individual alleles, the outcome can be predicted from the initial variance components. This seems remarkable but can be understood as a perturbation to neutrality: When selection is spread over very many loci, its effect on any one locus is weak relative to drift, and so the variance components are hardly perturbed by selection. This is an extension of the infinitesimal model to nonadditive inheritance (29). For haploids, the total selection response is proportional to the initial genotypic variance (including both additive and nonadditive components). For diploids, kth-order components of variance have effect multiplied by $2^k$. Nevertheless, it is extremely difficult to find plausible models in which such higher-order variance components are significant (10). One way to see this is to imagine a population in which all additive effects are zero—as would be the case at an equilibrium under balancing selection. However, any change in allele frequencies will necessarily generate nonzero additive effects, and, consequently, substantial additive variance. Higher-order epistatic variance is also likely to be small if alleles are at extreme frequencies: the kth-order epistasis is proportional to the product of $\langle pq\rangle$ across k loci, and so cannot be large (10).

In contrast, when selection is strong relative to drift $\left(Ns\gg 1\right)$, the population will fix at an adaptive peak. This requires strong selection on each allele ($Ns\gg 1$), but also recurrent mutation, so any allele that is favored will eventually succeed. Thus, the issue is primarily about the genotype−phenotype map rather than any population genetic process. We show that, unless epistasis is systematically biased, the ultimate response is increased, on average, only if the fittest peak is changed, because epistasis changes some alleles from being favored to being deleterious. The expected response can be predicted by a simple argument, which assumes that the net effect of the changing background is a normally distributed perturbation, and that the chance that selection on an allele changes sign is simply the chance that this random epistatic perturbation exceeds the main effect of the allele.

When drift dominates, our conclusions follow simply from the variance components, without further assumptions. When selection dominates, we assume that epistasis is not systematically biased toward (or against) interactions between favorable alleles. If a systematic bias is allowed, then epistasis can have an arbitrarily large effect. To see this, imagine a trait that is some function $g\left[z\right]$ of an additive trait, z. If $g\left[z\right]$ curves upward, then the selection response will accelerate, and can become much larger than the corresponding additive model with the same initial additive variance. Indeed, if we are allowed to assign trait values to genotypes arbitrarily, we can construct paths that follow any pattern of change through time. Our conclusions about the ultimate effects of selection depend on the assumption that epistasis adds a random perturbation, without any definite bias. Empirically, this seems to be the consensus: Epistasis is pervasive between genetic loci but is generally nondirectional in the sense that there is no prevalent pattern of either positive or negative interactions (30).

Throughout this paper, we assumed that the populations remained at linkage equilibrium. However, linkage disequilibrium (LD) will be generated in a number of ways. Drift alone will produce some LD, although this should be symmetric around no disequilibrium and so should not, on average, have strong effects. In the selection-dominated regime, however, LD should be generated consistently in a directional fashion. Nagylaki’s theorem (31) states that, under weak selection, as we assume here, the population is guaranteed to approach and remain close to linkage equilibrium. Furthermore, the fact that we assume that interactions have no preferred direction should further hinder the buildup of LD. Nevertheless, LD will affect the response: A truly infinite population with no recombination (full linkage) is guaranteed to find the global peak, and linkage can only affect the rate at which this is approached. In this sense, linkage can help because recombining populations can get “trapped” at local peaks.

Do gene interactions and epistasis affect the long-term response? In the drift-dominated regime, which can apply even when selection on the trait is strong for polygenic traits, the long-term response is merely proportional to the initial genetic variance. Epistatic architectures can reach higher trait values compared with additive populations of the same additive variance, because the former necessarily harbor more genetic variance. However, this effect will be small and the response slower because epistatic variance typically represents a small fraction of the total genetic variance. In the selection-dominated regime, it is the specifics of the interaction structure that matter. Substantial increases of the long-term response can be reached when interactions are strong and induce allelic reversals, but the initial epistatic variance of the population is not predictive of this (see Deterministic Limits as a Function of Allele Frequency Distribution and Figs. S3 and S4). These results set expectations for the effect of epistasis on the long-term response under directional selection and help reconcile the success of the infinitesimal additive model (11) with the biological fact that genetic interactions are pervasive in nature.

Fig. S3.

(A) Long-term response for a population with initial allele frequency distribution as denoted in the figure ($E\left[p\right]$ and $Var\left(p\right)$ denote the mean and variance of a beta distribution). Number of loci is set to $n=50$, ${\sigma }_{\alpha }=1/10$, ${\sigma }_{\mathit{ϵ}}=1/2$, and $\beta =1/100$. (B) The initial fraction of epistatic variance $V_{\text{AA}}$ present in the population. Dashed line is $Var\left(p\right)=p^2$, which crosses the isocontours of fraction of epistatic variance at their highest value.

Fig. S4.

Difference in long-term response of an epistatic population and an additive one of the same initial additive variance as a function of the number of loci for ${\sigma }_{\alpha }=1/10$ and (from bottom to top) ${\sigma }_{\mathit{ϵ}}=\left\{\frac{{\sigma }_{\alpha }}{10},\hspace{0.17em}2\frac{{\sigma }_{\alpha }}{10},\hspace{0.17em}3\frac{{\sigma }_{\alpha }}{10},\hspace{0.17em}4\frac{{\sigma }_{\alpha }}{10},\hspace{0.17em}\text{and}\hspace{0.17em}5\frac{{\sigma }_{\alpha }}{10}\right\}$. Initial allele frequencies were drawn from a Beta distribution with mean m = 1/4 and variance v = 1/20.

Materials and Methods

Simulations.

Except where noted, we assume a haploid population of effective size $N_e$; similar results can be obtained for a diploid population with $2N_e$ copies of each gene, provided there is no dominance. We also assume that the trait is under weak directional selection, with selection gradient β, and that the population is close to linkage equilibrium. We ignore the environmental component of variance, because this does not affect the response to directional selection. Our results for finite populations under weak selection are independent of the trait architecture. However, for simulations, we assume that the trait of a diallelic haploid genotype of n loci $\underset{¯}{X}=\left(X_1,\dots ,X_n\right)$ is given by $z\left(\underset{¯}{X}\right)={\displaystyle {\sum }_{i=1}^n{\alpha }_i{X}_i}+1/2{\displaystyle {\sum }_{ij=1}^n{\mathit{ϵ}}_{ij}{X}_i{X}_j}$. Assuming that the population is at linkage equilibrium, one can write, for the mean trait value, $\overline{z}={\displaystyle {\sum }_i{\alpha }_i{p}_i}+1/2{\displaystyle {\sum }_{i\ne j}{\mathit{ϵ}}_{ij}{p}_i{p}_j}$. The effect of a particular locus on the trait mean is then $A_i=\partial \overline{z}/\partial p_i={\alpha }_i+{\displaystyle {\sum }_j{\mathit{ϵ}}_{ij}{p}_j}$. We typically assume that main effects ${\alpha }_i$ and epistatic coefficients ${\mathit{ϵ}}_{ij}$ are randomly sampled from Gaussian distributions $N(0,{\sigma }_{\alpha }$ and $N\left(0,{\sigma }_{\mathit{ϵ}}\right)$, respectively.

We assume directional selection for increased trait values such that the fitness of a particular genotype $\underset{¯}{X}$ is $w\left(\underset{¯}{X}\right)=1+\beta z\left(\underset{¯}{X}\right)$. At linkage equilibrium, the mean fitness of the population can then be written as $\overline{w}=1+\beta \overline{z}$. The change in allele frequency at any locus is then $\mathrm{\Delta }{p}_i=p_i{q}_i\left(d\ln \overline{w}\right)/d{p}_i=\frac{\beta }{\overline{w}}{A}_i{p}_i{q}_i$. The dynamics of macroscopic variables, such as trait mean, variance, etc., can be derived from this expression.

Finite Population Simulations.

The simulations were performed by keeping track only of allele frequencies, thereby abolishing LD. Every generation, allele frequencies are updated according to the deterministic expectation, $\mathrm{\Delta }{p}_i=\beta A_i{p}_i\left(1-p_i\right)$, where $A_i={\displaystyle {\sum }_{ij}{\mathit{ϵ}}_{ij}{p}_j}$. A binomial sample of N copies is then sampled with this probability of success for each locus, and the allele frequencies are updated according to this sample. We repeat this procedure until no genetic variation exists at any locus, i.e., the population is fixed for one genotype.

Deterministic Numerical Simulations.

Every iteration, we iterate the deterministic recursion $\mathrm{\Delta }{p}_i=\beta A_i{p}_i\left(1-p_i\right)/\overline{w}$, where $A_i={\alpha }_i+{\displaystyle {\sum }_{j\ne i}{\mathit{ϵ}}_{ij}{p}_j}$. The recursion is iterated for a set number of generations chosen so that the remaining variance in the population is vanishingly small.

Fitness Landscape Analysis.

To find the absolute maximum trait increase, we associate a trait value $z={\displaystyle {\sum }_{i=1}^n{\alpha }_i{X}_i}+1/2{\displaystyle {\sum }_{ij}{\mathit{ϵ}}_{ij}{X}_i{X}_j}$ to every genotype $\left(X_1,\dots ,X_n\right)$ and exhaustively search for the maximum of the landscape.

Dynamics of Variance Components Under Genetic Drift

For a haploid population with n biallelic loci, any trait can be defined as

$$z\left(\underset{¯}{X}\right)={\displaystyle \sum _i{\alpha }_i{X}_i}+{\displaystyle \sum _{i>j}{\alpha }_{ij}{X}_i{X}_j}+{\displaystyle \sum _{i>j>k}{\alpha }_{ijk}{X}_i{X}_j{X}_k+\cdots }.$$

Under linkage equilibrium, the mean and variance of the trait distribution over the population can be written as

$$\begin{array}{c}\overline{z}={\displaystyle \sum _i{\alpha }_i{p}_i}+{\displaystyle \sum _{i>j}{\alpha }_{ij}{p}_i{p}_j}+{\displaystyle \sum _{i>j>k}{\alpha }_{ijk}{p}_i{p}_j{p}_k+\cdots }\\ \text{Var}\left(z\right)=V_A+V_{AA}+\cdots +V_{A_{\left(k\right)}}\\ ={\displaystyle \sum _i{\left(A_i^{\left(1\right)}\right)}^2{V}_i}+{\displaystyle \sum _{i>j}{\left(A_{ij}^{\left(2\right)}\right)}^2{V}_i{V}_j+\cdots }\end{array}$$

where

$$\begin{array}{c}{A}_i^{\left(1\right)}=\frac{\partial \overline{z}}{\partial p_i}={\alpha }_i+{\displaystyle \sum _j{\alpha }_{ij}{p}_j}+{\displaystyle \sum _{jk}{\alpha }_{ijk}{p}_j{p}_k+\cdots }\\ A_{ij}^{\left(2\right)}=\frac{{\partial }^2\overline{z}}{\partial p_i\partial p_j}={\alpha }_{ij}+{\displaystyle \sum _k{\alpha }_{ijk}{p}_k+\cdots }\\ ⋮\\ A_{ij\cdots n}^{\left(n\right)}=\frac{{\partial }^n\overline{z}}{\partial p_i\partial p_j\cdots \partial p_n}.\end{array}$$

If binomial sampling (genetic drift) is the main force determining dynamics at every locus, and the population is random mating and at linkage equilibrium, we can write, for the expectation at the next generation (for multiple realizations of the same process) under the Wright-Fisher model,

$$\begin{array}{c}\langle p_{t+1}\rangle =p_t\\ \text{Var}\left(p_{t+1}\right)=\frac{1}{N}{p}_t\left(1-p_t\right).\end{array}$$

Using these expressions, we can calculate the expectation of the genetic variance at each locus in the next generation,

$$\begin{array}{c}\langle V_{t+1}\rangle =\langle p_{t+1}\rangle -\langle p_{t+1}^2\rangle =\langle p_{t+1}\rangle -\mathrm{Var}\left(p_{t+1}\right)-{\langle p_{t+1}\rangle }^2=p_t\left(1-p_t\right)-\frac{1}{N}{p}_t\left(1-p_t\right)\\ =\left(1-\frac{1}{N}\right)V_t\\ \text{or},\text{conversely},\\ \langle p_{t+1}^2\rangle ={\langle p_t\rangle }^2+\text{Var}\left(p_{t+1}\right)=p_t^2+\frac{1}{N}{p}_t\left(1-p_t\right)=p_t^2+\frac{1}{N}{V}_t.\end{array}$$ [S1]

Because $\overline{z}$ does not contain terms in higher powers of each of the allele frequencies (that is, it does not contain terms in $p_i^2$), every $A_{i\cdots k}$ does not depend on the allele frequencies of any of the $i\cdots k$ loci. This means that the expectation $\langle {\left(A_{i\cdots k}\right)}^2{V}_i\cdots V_k\rangle =\langle {\left(A_{i\cdots k}\right)}^2\rangle \langle V_i\rangle \cdots \langle V_k\rangle$.

Each of the $A_i^2$ is of the form

$$A_i^2={\alpha }_i^2+{\displaystyle \sum _{j>i}{\alpha }_{ij}^2{p}_j^2}+{\displaystyle \sum _{k>j>i}{\alpha }_{ijk}^2{p}_j^2{p}_k^2+\cdots +\text{cross}‐\text{terms}}.$$

The cross-terms are all linear on the allele frequencies $p_j,\cdots ,p_k$ and so, in expectation, remain the same in the next generation because $\langle p_i^{t+1}\rangle =p_i^t$. The terms of $A_i^2$ that are quadratic in $p_i$ are changed in the next generation, in expectation, as in Eq. S1. With a bit of algebra (cumbersome using this notation, but see ref. 19), we can see that the expectation for ${\langle A_i^2\rangle }_{t+1}$ is inflated by a fraction of how much the locus i contributes to all of the higher-order components of variance ($V_{AA}$, etc.).

For example, for a genetic architecture involving only pairwise interactions, the contribution of a locus to additive variance is

$$\begin{array}{c}{\left(V_A\right)}_i={\left({\alpha }_i+{\displaystyle \sum _j{\alpha }_{ij}{p}_j}\right)}^2{V}_i\\ =\left({\alpha }_i^2+{\displaystyle \sum _j{\alpha }_{ij}^2{p}_j^2}+2{\alpha }_i{\displaystyle \sum _j{\alpha }_{ij}{p}_j}+\left({\displaystyle \sum _j{\alpha }_{ij}{p}_j}\right)\left({\displaystyle \sum _{k\ne j}{\alpha }_{ik}{p}_k}\right)\right)V_i.\end{array}$$

This contribution will be in expectation in the next generation,

$$\begin{array}{c}{\langle {\left(V_A\right)}_i\rangle }_{t+1}=({\alpha }_i^2+{\displaystyle \sum _j{\alpha }_{ij}^2{\langle p_j^2\rangle }_{t+1}}+2{\alpha }_i{\displaystyle \sum _j{\alpha }_{ij}{\langle p_j\rangle }_{t+1}}\\ +\left({\displaystyle \sum _j{\alpha }_{ij}{\langle p_j\rangle }_{t+1}}\right)\left({\displaystyle \sum _{k\ne j}{\alpha }_{ik}{\langle p_k\rangle }_{t+1}}\right)){\langle V_i\rangle }_{t+1}\\ =({\alpha }_i^2+{\displaystyle \sum _j{\alpha }_{ij}^2\left({\left(p_j^t\right)}^2+\text{Var}\left(p_j^t\right)\right)}+2{\alpha }_i{\displaystyle \sum _j{\alpha }_{ij}{p}_j^t}\\ +\left({\displaystyle \sum _j{\alpha }_{ij}{p}_j^t}\right)\left({\displaystyle \sum _{k\ne j}{\alpha }_{ik}{p}_k^t}\right))\left(1-\frac{1}{N}\right)V_i^t\\ ={\left(A_i^t\right)}^2\left(1-\frac{1}{N}\right)V_i^t+{\displaystyle \sum _j{\alpha }_{ij}^2\text{Var}\left(p_j^{t+1}\right)\left(1-\frac{1}{N}\right)V_i^t}\\ ={\left(A_i^t\right)}^2\left(1-\frac{1}{N}\right)V_i^t+{\displaystyle \sum _j{\alpha }_{ij}^2\left(\frac{1}{N}\right)p_j^t\left(1-p_j^t\right)\left(1-\frac{1}{N}\right)V_i^t}\\ =\left(1-\frac{1}{N}\right){\left(V_A^t\right)}_i+\frac{1}{N}\left(1-\frac{1}{N}\right)\underset{\text{The}\hspace{0.17em}\text{locus’}\hspace{0.17em}\text{contribution}\hspace{0.17em}\text{to}\hspace{0.17em}{V}_{AA}}{\underbrace{{\displaystyle \sum _j{\alpha }_{ij}^2{V}_j^t{V}_i^t}}}.\end{array}$$

In general, for arbitrary orders of interactions, one obtains

$${\langle V_{A\left(j\right)}\rangle }_{t+1}={\left(1-\frac{1}{N}\right)}^j{\displaystyle \sum _{k=0}^{n-j}\left(\begin{array}{c}k-j\\ j\end{array}\right){\left(\frac{1}{N}\right)}^k{V}_{A\left(j+k\right)}^t}$$

and, in particular, for the additive variance,

$${\langle V_A\rangle }_{t+1}=\left(1-\frac{1}{N}\right){\displaystyle \sum _{k=1}^{\infty }k{\left(\frac{1}{N}\right)}^{k-1}{V}_{A\left(k\right)}^t}.$$

Calculating Expectations for the Initial Components of Variance over the Allele Frequency Distribution

The initial epistatic variance present in a population can be written as

$$V_{\text{epi}}=\frac{1}{2}{\displaystyle \sum _{ij}{p}_i{q}_i{p}_j{q}_j{\mathit{ϵ}}_{ij}^2}$$

$$V_{\text{epi}}={\displaystyle \sum _{ij}{p}_i{q}_i{p}_j{q}_j{\mathit{ϵ}}_{ij}^2}={\displaystyle \sum _{ij}\psi \left(p_i,p_j\right){\mathit{ϵ}}_{ij}^2}.$$

If we assume that the initial allele frequencies are independently distributed, we can average over the allele frequencies and write

$$\begin{array}{c}{\overline{V}}_{\text{epi}}=\overline{p}\left(1-\overline{p}\right)\left(\overline{p}\left(1-\overline{p}\right)-2V_p-V_p^2\right){\displaystyle \sum _{ij}{\mathit{ϵ}}_{ij}^2}\\ {\overline{V}}_{\text{epi}}=\overline{\psi }\left(p\right){\displaystyle \sum _{ij}{\mathit{ϵ}}_{ij}^2},\end{array}$$

where $\overline{p}$ and $V_p=\overline{p^2}-{\overline{p}}^2$ are the mean and variance of the distribution of allele frequencies. In the same manner, if we assume that epistatic interactions are much stronger than the background independent effects ${\alpha }_i$, the initial additive variance can be written as

$$\begin{array}{c}{\overline{V}}_A=\left({\overline{p}}^2-V_p\right)\left(\overline{p}\left(1-\overline{p}\right)-V_p\right){\displaystyle \sum _{ij}{\mathit{ϵ}}_{ij}^2+2{\overline{p}}^2\left(\overline{p}\left(1-\overline{p}\right)-V_p\right)}{\displaystyle \sum _i\left({\displaystyle \sum _{j,k\ne j}{\mathit{ϵ}}_{ij}{\mathit{ϵ}}_{ik}}\right)}\\ .\overline{V_A}=\overline{\varphi }\left(p\right){\displaystyle \sum _{ij}{\mathit{ϵ}}_{ij}^2+\overline{\phi }\left(p\right)}{\displaystyle \sum _i\left({\displaystyle \sum _{j,k\ne j}{\mathit{ϵ}}_{ij}{\mathit{ϵ}}_{ik}}\right)}\end{array}$$

where $\psi \left(p_i,p_j\right)=p_i{q}_i{p}_j{q}_j$, $\varphi \left(p_i,p_j\right)=p_i{q}_i{p}_j^2$ and $\phi \left(p_i,p_j,p_k\right)=p_i{q}_i{p}_j{p}_k$.

$$\begin{array}{c}\langle {\overline{V}}_A\rangle ={\displaystyle \sum _i\left(\overline{p}\left(1-\overline{p}\right)-V_p\right)\langle {\alpha }_i^2\rangle +\overline{p}\left(\overline{p}\left(1-\overline{p}\right)-V_p\right)}{\displaystyle \sum _j\langle {\mathit{ϵ}}_{ij}^2\rangle }\\ +2\overline{p}\left(\overline{p}\left(1-\overline{p}\right)-V_p\right){\displaystyle \sum _j\langle {\alpha }_i{\mathit{ϵ}}_{ij}\rangle +2{\overline{p}}^1\left(\overline{p}\left(1-\overline{p}\right)-V_p\right)}\times {\displaystyle \sum _j{\displaystyle \sum _{k>j}\langle {\mathit{ϵ}}_{ij}{\mathit{ϵ}}_{ik}\rangle }}.\end{array}$$

Using $\overline{\psi }={\left[\overline{p}\left(1-\overline{p}\right)-\left(V_p\right)\right]}^2$, $\overline{\varphi }=\left(V_p+{\overline{p}}^2\right)\left[\overline{p}\left(1-\overline{p}\right)-V_p\right]$, and $\overline{\phi }={\overline{p}}^2\left[\overline{p}\left(1-\overline{p}\right)-V_p\right]$, the average total genetic variance can then be written as

$$\begin{array}{c}\langle {\overline{V}}_G\rangle =\left(\overline{\psi }+\overline{\varphi }\right){\displaystyle \sum _{ij}\langle {\mathit{ϵ}}_{ij}^2\rangle }+\overline{\phi }{\displaystyle \sum _i{\displaystyle \sum _{j,k\ne j}\langle {\mathit{ϵ}}_{ij}{\mathit{ϵ}}_{ik}\rangle }}\\ \langle {\overline{V}}_G\rangle ={\displaystyle \sum _i\left(\overline{p}\left(1-\overline{p}\right)-V_p\right)\langle {\alpha }_i^2\rangle +\overline{p}\left[\overline{p}\left(1-\overline{p}\right)-V_p\right]}{\displaystyle \sum _{ij}\langle {\mathit{ϵ}}_{ij}^2\rangle }\\ +{\overline{p}}^2\left[\overline{p}\left(1-\overline{p}\right)-V_p\right]{\displaystyle \sum _i{\displaystyle \sum _{j,k\ne j}\langle {\mathit{ϵ}}_{ij}{\mathit{ϵ}}_{ik}\rangle +2\overline{p}\left(\overline{p}\left(1-\overline{p}\right)-V_p\right)}}{\displaystyle \sum }_j\langle {\alpha }_i{\mathit{ϵ}}_{ij}\rangle \end{array}$$

and, realizing that, because all of the ${\mathit{ϵ}}_{ij}$ are independent and identically distributed, all terms $\langle {\mathit{ϵ}}_{ij}^2\rangle$ are identical and all terms $\langle {\mathit{ϵ}}_{ij}{\mathit{ϵ}}_{ik}\rangle =0$ and $\langle {\alpha }_i{\mathit{ϵ}}_{ij}\rangle =0$, we can write, for the initial fraction of epistatic variance,

$$\begin{array}{c}\frac{\langle {\overline{V}}_{\text{epi}}\rangle }{\langle {\overline{V}}_G\rangle }=\frac{\overline{\psi }}{\overline{\varphi }+\overline{\psi }}=\frac{\overline{p}\left(1-\overline{p}\right)-V_p}{\overline{p}\left(1+\overline{p}\right)+V_p}\\ \frac{\langle {\overline{V}}_{\text{epi}}\rangle }{\langle {\overline{V}}_{\text{G}}\rangle }=-\frac{\left(n-1\right){\sigma }_{\text{epi}}^2\left(p^2-p+V_p\right)}{\left(n-1\right){\sigma }_{\text{epi}}^2\left(p^2+p+V_p\right)+4{\sigma }_{\text{add}}^2}.\end{array}$$

This allows us to quantify the amount of epistatic variance initially present in the population, as a function only of the mean and variance of the initial allele distribution. In the limit of strong epistasis or many loci, this expression reduces to a limit that depends only on the initial allele frequency distribution,

$${\frac{\langle {\overline{V}}_{\text{epi}}\rangle }{\langle {\overline{V}}_{\text{G}}\rangle }|}_{n\to \infty }={\frac{\langle {\overline{V}}_{\text{epi}}\rangle }{\langle {\overline{V}}_{\text{G}}\rangle }|}_{\frac{{\sigma }_{\text{epi}}}{{\sigma }_{\text{add}}}\to \infty }=\frac{\overline{p}\left(1-\overline{p}\right)-V_p}{\overline{p}\left(1+\overline{p}\right)+V_p}.$$

Deterministic Limits as a Function of Allele Frequency Distribution

The long-term response of a population in the deterministic regime is mostly determined by the fitness landscape the population evolves on (see The Interaction Between Strong Directional Selection and Epistasis). When initial allele frequencies are not vanishingly rare, the long-term response will be below the limits we show in The Interaction Between Strong Directional Selection and Epistasis, because the population starts at a higher trait value. Fig. S3 shows the long-term response as a function of the allele frequency distribution for strong $\mathrm{ℰ}$.

Measuring Strength of Interactions from F2 Crosses

One can measure the critical parameter $\mathrm{ℰ}$ by crossing ancestral populations with derived populations. These differ at $n_0$ loci, which will be at $p_i=1/2$ in the hybrid population. Therefore, $V_A^H={\displaystyle {\sum }_i{\left(A_i+{\mathit{ϵ}}_i\right)}^2}\left(1/4\right)=\left(n_0/4\right)\left({\sigma }_{\alpha }^2+n_0{\sigma }_{\mathit{ϵ}}^2\right)=\left(n_0{\sigma }_{\alpha }^2/4\right)\left(1+{\mathrm{ℰ}}^2\right),V_{AA}^H=\left(1/2\right){\displaystyle {\sum }_{i\ne j}{\mathit{ϵ}}_{ij}^2}{\left(1/4\right)}^2=\left[n_0\left(n_0-1\right)/32\right]{\sigma }_{\mathit{ϵ}}^2\approx n_0{\mathrm{ℰ}}^2{\sigma }_{\alpha }^2/32$. Therefore, $V_{AA}^H/V_A^H\approx \left(n_0/8\right)\left[{\mathrm{ℰ}}^2/\left(1+{\mathrm{ℰ}}^2\right)\right]$. We also have that $\mathrm{\Delta }R\approx n_0{\sigma }_{\alpha }$, so that $V_A^H/\mathrm{\Delta }{R}^2\approx \left(1/4n_0\right)\left(1+{\mathrm{ℰ}}^2\right)$—which is just a haploid version of the Wright−Castle−Lande estimator. We see that, if the ratio of epistatic to additive variance in the hybrid population is $\approx$ 1, then $\mathrm{ℰ}\approx \sqrt{8/n_0}$, which is small.