Balancing Selection in Species with Separate Sexes: Insights from Fisher’s Geometric Model

Abstract

How common is balancing selection, and what fraction of phenotypic variance is attributable to balanced polymorphisms? Despite decades of research, answers to these questions remain elusive. Moreover, there is no clear theoretical prediction about the frequency with which balancing selection is expected to arise within a population. Here, we use an extension of Fisher’s geometric model of adaptation to predict the probability of balancing selection in a population with separate sexes, wherein polymorphism is potentially maintained by two forms of balancing selection: (1) heterozygote advantage, where heterozygous individuals at a locus have higher fitness than homozygous individuals, and (2) sexually antagonistic selection (a.k.a. intralocus sexual conflict), where the fitness of each sex is maximized by different genotypes at a locus. We show that balancing selection is common under biologically plausible conditions and that sex differences in selection or sex-by-genotype effects of mutations can each increase opportunities for balancing selection. Although heterozygote advantage and sexual antagonism represent alternative mechanisms for maintaining polymorphism, they mutually exist along a balancing selection continuum that depends on population and sex-specific parameters of selection and mutation. Sexual antagonism is the dominant mode of balancing selection across most of this continuum.

ONE of the primary goals of population genetics is to evaluate how processes of mutation, selection, migration, recombination, and genetic drift account for the maintenance of genetic variation in natural populations. This “great obsession” with genetic variation (Gillespie 2004, p. 154) is reflected by a massive body of theoretical and empirical research that seeks to connect empirical patterns of population and quantitative genetic variability with the specific evolutionary processes that might account for them.

Balancing selection—selection that maintains genetic variation at a locus—could potentially account for an important fraction of observed quantitative genetic variability (Dobzhansky 1955; Lewontin 1974; Charlesworth and Hughes 1999). Although relatively few unambiguous cases of balancing selection have been documented to date (see Charlesworth 2006; Hedrick 2012; Johnston et al. 2013; Leffler et al. 2013), empirical methods for detecting their population genetic signatures are highly conservative and may fail to identify many (or most) instances of balancing selection within a genome (Charlesworth 2006; Charlesworth and Charlesworth 2010). Furthermore, balanced polymorphisms may plausibly account for several empirical observations that are not easily explained by alternative models of variation maintained by recurrent mutation. For example, the high genetic variance in life history and reproductive traits (Houle 1992; Pomiankowski and Moller 1995) and the presence of intermediate-frequency alleles with large phenotypic effects (Long et al. 2000) are each potentially attributable to a subset of loci that segregate for balanced alleles (Charlesworth and Hughes 1999; Turelli and Barton 2004).

Population genetic theory provides a complementary approach for assessing the plausibility of different scenarios of balancing selection, and indeed there are many formal models that delineate the parameter conditions required for balancing selection to operate (e.g., Wright 1969; Crow and Kimura 1970; Prout 2000). However, most theory cannot predict the probability of balancing selection because it does not incorporate details of the fitness effect distribution among random alleles and genotypes in a population. Such details ultimately determine the relative likelihood of different forms of selection at individual loci.

A recent study by Sellis et al. (2011) provides new insight into the probability of balancing selection by modeling the frequency of heterozygote advantage in Fisher’s geometric model (FGM) (Fisher 1930; Orr 2005a,b)—an influential theoretical framework for the fitness effect distribution of random mutations. Sellis et al. (2011) showed that, among loci contributing to adaptive evolutionary change, the probability of heterozygote advantage tends to increase with the “standardized size” (Orr 1998) of adaptive mutations. Under Fisher’s original scaling, mutation size is given by $x=r\sqrt{n}/(2z),$ where r is the absolute phenotypic effect size of the mutation, n is the number of traits, and z represents the displacement of the population from a fitness optimum within phenotypic space (see Fisher 1930, pp. 38–41; Orr 1998). A mutation of size r is effectively small when the distance to the optimum is large (e.g., in the limit r/z → 0) and effectively large when the distance to the optimum is small (e.g., r → z) (for a clear discussion of x and its biological meaning, see Orr 1998, pp. 937–938). Adaptation—the evolutionary movement of a population toward its optimum—causes the distance to the optimum to shrink (Orr 1998) and thereby increases the scaled sizes of random mutations. Poorly adapted populations that are far from an optimum may initially experience little opportunity for heterozygote advantage, yet adaptive evolution will ultimately overturn such initially unfavorable conditions. During the course of an adaptive walk within phenotypic space, mutations eventually become sufficiently “large” for heterozygote advantage to be likely (Sellis et al. 2011; for related results, see Manna et al. 2011).

In species with separate sexes, balancing selection can potentially arise by way of heterozygote advantage or by sexual antagonism (a.k.a. “intralocus sexual conflict”), in which the best genotype for females differs from the best genotype for males (Rice 1992; Chippindale et al. 2001). Whether balanced, sexual antagonistic alleles are common within populations has important and wide-ranging biological implications for the genetic basis of quantitative traits (Rice 1984; Turelli and Barton 2004; Bonduriansky and Chenoweth 2009), genetic loads and extinction risk (Kokko and Brooks 2003; Whitlock and Agrawal 2009; Connallon et al. 2010; Holman and Kokko 2013), mating system evolution (Seger and Trivers 1986; Albert and Otto 2005; Blackburn et al. 2010; Roze and Otto 2012), and the evolution of genomes and genetic systems (Charlesworth and Charlesworth 1980; Day and Bonduriansky 2004; Ellegren and Parsch 2007; van Doorn and Kirkpatrick 2007, 2010; Mank 2009; Connallon and Clark 2010, 2011, 2013a; Parsch and Ellegren 2013; Wright and Mank 2013; Charlesworth et al. 2014; Kirkpatrick and Guerrero 2014). Although prior theory clearly defines the parameter criteria for balancing selection by sexual antagonism (e.g., Kidwell et al. 1977; Pamilo 1979; Patten and Haig 2009; Unckless and Herren 2009; Fry 2010; Patten et al. 2010, 2013; Arnqvist 2011; Connallon and Clark 2012; Mullon et al. 2012; Jordan and Charlesworth 2012), it remains unclear how often such conditions might be expected to arise in dioecious populations. A recent extension of Fisher’s geometric model provides a theoretical framework for predicting the sex-specific distribution of mutant fitness effects (Connallon and Clark 2014), yet this theory has not addressed opportunities for balancing selection.

Here, we present new theoretical predictions for the probability of balancing selection in a species with distinct sexes. Our models build upon a recent extension of Fisher’s geometric model (Connallon and Clark 2014) that incorporates two evolutionarily important features of dioecious species: (1) sexually dimorphic fitness landscapes, which generate sex-specific patterns of selection on phenotypes expressed by each sex, and (2) sex-by-genotype interactions (a form of gene-by-environment interaction during development), in which mutations have sexually dimorphic effects on female and male phenotypes. We first extend Fisher’s original scaling function by developing a mathematical expression for the standardized mutation size in a dioecious population. We show that this new scaling function can be used to calculate several interesting metrics in FGM, including (1) the probability that random mutations experience positive, purifying, or balancing selection and (2) the critical mutation size that maximizes the probability of balancing selection. We show that sex differences in selection on phenotypes and genotype-by-sex interactions each tend to inflate the standardized mutation size and that these effects increase opportunities for balancing selection in the FGM framework. Finally, we demonstrate that, for most of the parameter space of sex-specific selection and gene-by-sex interactions, sexual antagonism is a pervasive feature of balancing selection.

Model

FGM with two sexes

The basic model analyzed here is a diploid extension of the haploid, two-sex FGM model that was recently developed by Connallon and Clark (2014). Male and female phenotypes are each characterized by a vector of n trait values, with each vector representing a specific location in n-dimensional phenotypic space. For mathematical convenience, and following prior work in Fisher’s model (Fisher 1930; Orr 1998; Sellis et al. 2011; Connallon and Clark 2014), we assume that (1) sex-specific fitness is a function of the Euclidean distance to the relevant optimum and (2) each mutation has a random, unbiased orientation within phenotypic space. Relaxation of these assumptions—for example, by allowing for elliptical fitness surfaces and/or mutational biases—often decreases the “effective” dimensionality in Fisher’s geometric model (see Waxman and Welch 2005; Martin and Lenormand 2006; Martin 2014), such that n may be interpreted as the effective number (rather than the total number) of independent traits.

The initial population is assumed to be fixed for a wild-type genotype, for which males and females express phenotypic values of A_m and A_f, respectively, where A_j = {x_1j, x_2j, … , x_nj} is a vector of n trait values for the jth sex; j = {f, m}. Optimal trait combinations for males and females are depicted by vectors O_m and O_f, which describe the locations of sex-specific optima for the n traits. The direction of selection acting in the two sexes can be described using a pair of vectors (one for each sex) that point from A_j to O_j (Figure 1). The lengths of the vectors are given by z_m and z_f and represent the displacement of each sex from its optimum (i.e., the Euclidean distance between A_j and O_j). The angle between vectors is θ_sel, which potentially takes any value between zero (where directions of sex-specific selection are the same in males and females) and π (where the sexes are selected in opposite directions). The degree of correlation between male and female orientations of directional selection is ρ_sel = cos(θ_sel) (Rodgers and Nicewander 1988; Connallon and Clark 2014). Sex-specific fitness of females and males, respectively, follows a Gaussian function

$$w_f=\exp \left(-{\mathit{\omega }}_f{z}_f^2\right)$$ (1a)

and

$$w_m=\exp \left({-{\mathit{\omega }}_m{z}_m}^2\right),$$ (1b)

where ω_f and ω_m are positive constants that describe the rate of fitness decline away from each optimum.

Figure 1.

Fisher’s geometric model with two sexes (an example with two traits). For the jth sex, individuals homozygous for a wild-type allele express phenotype A_j and are separated from their optimum (O_j) by distance z_j. Mutations alter the phenotype of their carriers, with mutant homozygotes expressing a phenotype at a distance r_j from the ancestral homozygote (an example mutant homozygote is represented by the solid diamond). Heterozygotes are similarly oriented away from A_j, but have a modified distance of hr_j, where h is the dominance coefficient with respect to the mutant phenotypic effect. For the example shown, heterozygotes will express an intermediate phenotype between A_j and the solid diamond, and for most values of h (0 < h < 1), the heterozygote phenotype will be closer to the optimum than either homozygote genotype will be. In such cases, there will be heterozygote advantage for fitness in the jth sex. Females and males may be characterized by distinct fitness optima and unique positions in phenotypic space with respect to each genotype (see Connallon and Clark 2014).

The phenotypic effects of each mutation can be similarly summarized using sex-specific vectors (Figure 1). Each mutation has a magnitude, with r_j representing the phenotypic (i.e., Euclidean) distance between A_j and the location of individuals that are homozygous for the mutation. The magnitude in a heterozygote is r_jh, where h represents the dominance of the mutation with respect to the phenotype (i.e., 0 < h < 1, with h = 0, h = 1/2, and h = 1 corresponding to recessive, additive or codominant, and dominant effects, respectively). Mutational orientations are assumed to be random and unbiased in n-dimensional space. The degree of similarity between the sexes, for each mutation’s orientation in phenotypic space, depends on the between-sex correlation of phenotypic effects within individual trait axes. For a mutation with magnitude r_m and r_f, ρ_mut represents both the phenotypic correlation between the sexes for each trait and the mean degree of correlation between mutant vectors {ρ_mut = E[cos(θ_mut)], where θ_mut is the angle between the vectors of a mutation, and E[] denotes the expectation (see Connallon and Clark 2014)}.

Taking both selection and mutation into consideration, we can calculate distances between the optimum O_j and the phenotypic position of individuals that carry zero, one, or two copies of a given mutation. Letting O_j′ = O_j – A_j = {o_1j, o_2j, … , o_nj}, the distance to the optimum in heterozygous males and females will be

$$\begin{array}{c}{z}_{m\left(\text{het}\right)}=\sqrt{z_m^2+{\left(h{r}_m\right)}^2-\frac{2h{r}_m}{\sqrt{{\displaystyle {\sum }_i{m}_i^2}}}{\displaystyle \sum _{i=1}^n{o}_{\mathit{im}}{m}_i}}\\ \mathrm{\approx }\sqrt{z_m^2+{\left(h{r}_m\right)}^2-\frac{2h{r}_m}{\sqrt{n}}{\displaystyle \sum _{i=1}^n{o}_{\mathit{im}}{m}_i}}\end{array}$$ (2a)

and

$$\begin{array}{c}{z}_{f\left(\text{het}\right)}=\sqrt{z_f^2+{\left(h{r}_f\right)}^2-\frac{2h{r}_f}{\sqrt{{\displaystyle {\sum }_i{f}_i^2}}}{\displaystyle \sum _{i=1}^n{o}_{\mathit{if}}{f}_i}}\\ \mathrm{\approx }\sqrt{z_f^2+{\left(h{r}_f\right)}^2-\frac{2h{r}_f}{\sqrt{n}}{\displaystyle \sum _{i=1}^n{o}_{\mathit{if}}{f}_i}}\end{array}$$ (2b)

(Appendix A), where the values of m_i and f_i are obtained from n independent draws from a bivariate standard normal distribution with cov(m_i, f_i) = ρ_mut (Connallon and Clark 2014). These relationships naturally emerge from the mutation algorithm described in the Simulations section, with approximations on the right-hand side of Equations 2a and 2b accurate for values of n ≥ 10 (Waxman and Welch 2005; Connallon and Clark 2014). Distances to the optima for homozygous carriers of the mutation, z_j(hom), can be obtained by setting h = 1 in Equation 2. Heterozygous and homozygous selection coefficients for mutations can be calculated directly from Equations 1 and 2 (Appendix A), with homozygous selection coefficients defined as s_j2 = exp[ω_j(z_j² – z_j(hom)²)] – 1 and heterozygous selection coefficients defined as s_j1 = exp[ω_j(z_j² – z_j(het)²)] – 1.

Criteria for positive selection and balancing selection

Balancing selection criteria follow the well-established theory of sex-specific selection at a diploid locus with two alleles (e.g., Kidwell et al. 1977). Let B₁ represent an ancestral allele and B₂ be the derived allele. The relative fitnesses for the three genotypes are provided in Table 1. We assume that generations are discrete and follow the life cycle of birth, selection prior to mating, and random mating among breeding adults. In cases where selection differs between males and females, the frequency of alleles inherited from mothers and from fathers will differ. Following convention, we define q_m and q_f as the frequency of B₂ alleles among the male and female gametes that contribute to a given generation.

Table 1. Sex-specific fitnesses and genotype frequencies.

	Genotype
	B1B1	B1B2	B2B2
Female fitness:	1	1 + sf1	1 + sf2
Male fitness:	1	1 + sm1	1 + sm2
Zygotic frequency:	(1 – qm)(1 – qf)	qm(1 – qf) + qf(1 – qm)	qmqf

Following standard criteria for determining evolutionary stability (see Otto and Day 2007), a new (rare) mutation will experience selection to increase in frequency when the boundary equilibrium q_m, q_f = 0 is unstable. The criterion for selection to favor invasion of a rare mutation is

$$0<s_{f1}+s_{m1}.$$ (3)

Mutations that meet this criterion can be further divided into mutually exclusive subsets: (1) positively selected mutations that are also favored to fix in the population and (2) mutations under balancing selection, which can invade from low frequency, but are not favored to fix. For the latter case, selection favors the maintenance of ancestral (wild-type) and mutant alleles. Balancing selection occurs when both boundary equilibria (q_m, q_f = 0; q_m, q_f = 1) are unstable, a condition often referred to as a “protected polymorphism” (Prout 1968). The condition for balancing selection is

$$0<s_{f1}+s_{m1}>s_{f2}\left(1+s_{m2}-s_{m1}\right)+s_{m2}\left(1+s_{f2}-s_{f1}\right).$$ (4)

We define positive selection when q_m, q_f = 0 is unstable (B₂ invades when rare) and q_m, q_f = 1 is stable (B₁ does not invade when rare).

Simulations

In our subsequent analytical results, we assume that (1) mutations have small fitness effects, as is generally expected (e.g., Orr 2005a,b, 2006; analytical results are appropriate for selection coefficients within the approximate range s_f1, s_f2, s_m1, s_m2 < 0.1), and (2) dimensionality is sufficiently high so that the approximations in Equation 2 remain valid [i.e., that n > 10 (Waxman and Welch 2005; Connallon and Clark 2014)]. Analytical results were verified using an exact simulation approach described in Connallon and Clark (2014), with adjustments for diploid inheritance. Without any loss of generality, we define the phenotypic positions of wild-type males and females, and their respective optima, so that A_f = A_m = {0, 0, …, 0}, O_f = {z_f, 0, …, 0}, and O_m = {z_mcos(θ_sel), z_msin(θ_sel), 0, …, 0}. Mutation magnitudes for each sex (r_m, r_f) were generated using a bivariate gamma distribution with equal male and female marginal distributions (values of r_m and r_f were obtained using the “mixture approach” algorithm, described in Michael and Schucany 2002). Within the jth sex, the phenotypic location of individuals homozygous for a random mutation is described by a vector, y_j = {y_1j, y_2j, … , y_nj}, with the elements of the vector representing a set of phenotypic changes in the n traits. For a mutation with magnitude r_m in males, y_im = r_mm_i/M, where $M^2={\displaystyle {\sum }_{i=1}^n{m}_i^2},$ and the m_i are independent draws from a standard normal distribution. This sampling strategy generates random, uniformly distributed points on the surface of an n sphere with radius r_m (Muller 1959; Marsaglia 1972; Connallon and Clark 2014). For a mutation with magnitude r_f in females, y_if = r_ff_i/F, where $F^2={\displaystyle {\sum }_{i=1}^n{f}_i^2},$ and the f_i are independent draws from a standard normal distribution with cov(m_i, f_i) = ρ_mut. For a mutation with homozygous effect y_j = {y_1j, y_2j, … , y_nj}, the corresponding heterozygous effect is modified by dominance, so that y_jh = {y_1jh, y_2jh, … , y_njh} (following Sellis et al. 2011). Once the y_im and y_if are specified for an individual mutation, it is straightforward to calculate heterozygous and homozygous selection coefficients for each sex, by using Equations 1 and 2 (see Appendix A).

Results

Mutation size and the probability of balancing selection

The proportion of random mutations that experience positive selection, purifying selection, and balancing selection can be directly calculated using the selection criteria outlined above (Equations 3 and 4) and the distribution of fitness effects that emerges from the two-sex Fisher’s geometric model (Equations 1 and 2) (see Appendix B). Following Fisher’s original concept of the standardized mutation size (Fisher 1930; Orr 1998), let x_A represent an adjusted standardized “size” that accounts for sex differences in directional selection and sex-by-genotype phenotypic effects of mutations. This adjusted mutation size is defined as

$$x_{\text{A}}=\frac{\left({\mathit{\omega }}_m{r}_m^2+{\mathit{\omega }}_f{r}_f^2\right)\sqrt{n}}{2\sqrt{{\left({\mathit{\omega }}_m{z}_m{r}_m\right)}^2+{\left({\mathit{\omega }}_f{z}_f{r}_f\right)}^2+2{\mathit{\omega }}_m{z}_m{r}_m{\mathit{\omega }}_f{z}_f{r}_f{\mathit{\rho }}_{\mathit{mf}}}}.$$ (5)

Mutation size is now a function of sex-specific selection and mutation parameters (ω_j, r_j, z_j), dimensionality (n), and the between-sex correlation of fitness effects: ρ_mf = ρ_selρ_mut (see Model section). Note that, by removing all possible sex differences (letting ρ_mf = 1, ω_m = ω_f, r = r_m = r_f, z = z_m = z_f), Equation 5 collapses to Fisher’s (1930) original expression: x = r√(n)/(2z).

The individual probabilities of purifying, positive, and balancing selection are simple functions of the new mutation size scaling (Appendix B). The probability that selection favors invasion of a random mutation with size x_A is

$$\mathrm{Pr}\left(s_{m1}+s_{f1}>0\right)\mathrm{\approx }\frac{1}{\sqrt{2\mathit{\pi }}}{\displaystyle {\int }_{x_{\text{A}}h}^{\mathrm{\infty }}{e}^{-\left(t^2/2\right)}dt}=1-\mathrm{\Phi }\left[x_{\text{A}}h\right],$$ (6)

where Φ(x_Ah) is the cumulative standard normal distribution, and h is the dominance of the mutation relative to the wild-type allele (0 < h < 1). In the absence of sex differences, where x_A = r√(n)/(2z), Equation 6 is identical to previous, high-dimension approximations (n > 10) for the probability of a beneficial mutation in Fisher’s geometric model (Fisher 1930; Kimura 1983; Orr 1998; Sellis et al. 2011).

Alleles selected to invade the population may go to fixation or evolve to an intermediate balanced polymorphic state. These outcomes—positive and balancing selection, respectively—are subsets of the probability space described by Equation 6 and are mutually exclusive. The probability that a random mutation evolves under balancing selection is

$$\begin{array}{c}\mathrm{Pr}\left(\mathrm{balancing}\right)\mathrm{\approx }\frac{1}{\sqrt{2\mathit{\pi }}}{\displaystyle {\int }_{x_{\text{A}}h}^{x_{\text{A}}(\left(1-h^2\right)/\left(1-h\right))}{e}^{-\left(t^2/2\right)}dt}\\ =\mathrm{\Phi }\left[x_{\text{A}}\frac{1-h^2}{1-h}\right]-\mathrm{\Phi }\left[x_{\text{A}}h\right],\end{array}$$ (7)

where Φ again refers to the cumulative standard normal. The probability of positive selection is therefore Pr(positive) = Pr(s_f1 + s_m1 > 0) – Pr(balancing) = 1 – Φ[x_A(1 – h²)/(1 – h)]. Relative probabilities of the three modes of selection—purifying, positive, and balancing—are shown in Figure 2A. Positive selection is more common than balancing selection when scaled mutation size is small (∼x_A < 1/2). For larger values of x_A, a majority of adaptive mutations evolve under balancing selection, rather than positive selection, with the probability of balancing selection ultimately converging, with increasing mutation size, to Pr(balancing) = 1 – Φ(x_Ah). In other words, as mutation size increases, an ever-increasing share of adaptive mutations (mutations favored to invade when rare) will experience balancing selection.

Figure 2.

Balancing selection in Fisher’s geometric model. (A) Probabilities of purifying selection (black), positive selection (red), or balancing selection (blue), as a function of the scaled mutation of size, x_A. The dotted line represents the probability that selection will favor evolutionary invasion of a random mutation of size x_A (this is the sum of individual probabilities of positive and balancing selection). Results are for the case of codominant expression in heterozygotes (h = 1/2), and curves were generated using Equations 6 and 7. Simulations (not shown) confirm that these results are extremely accurate at moderate to high dimensionality (i.e., n > 10). (B) The maximum probability of balancing selection (black curve) and the critical value of x_A $\left({\widehat{x}}_{\text{A}}\right)$ that maximizes the probability of balancing selection (dotted gray line). Results are based on Equations 7 and 8.

As originally noted by Fisher (1930), mutations with small phenotypic effects (x_A → 0) are likely to improve fitness, whereas large mutations have small probabilities of doing so. A scaled mutation size near zero will therefore maximize the probability of positive selection and minimize the probability of purifying selection. In contrast, the probability of balancing selection is maximized for mutations of intermediate size. The value of x_A that maximizes the probability of balancing selection is

$${\widehat{x}}_{\text{A}}=\left(1-h\right)\sqrt{\frac{2}{1-h^2\left(3-2h\right)}\ln \left[\frac{\left(1-h^2\right)}{h\left(1-h\right)}\right]}$$ (8)

(Appendix C). In the absence of dominance between mutant and wild-type alleles (h = 1/2), this reduces to ${\widehat{x}}_{\text{A}}=\sqrt{\ln \left(3\right)}\mathrm{\approx }1.048,$ with partial recessivity (h < 1/2) increasing this critical mutation effect size and partial dominance decreasing it (Figure 2B, gray dotted line). The maximum probability of balancing selection can also be calculated by evaluating Equation 7, using the critical mutation size of Equation 8. When phenotypic effects are codominant (h = 1/2), a maximum of ∼24% of mutations experience balancing selection; the probability of balancing selection further increases with partial recessivity and decreases with partial dominance (Figure 2B).

Mean mutation size in species with separate sexes

Mutation size (x_A) clearly influences the probability that random mutations experience purifying, positive, or balancing selection. Small-sized mutations evolve primarily under positive or purifying selection (in the limit, x_A → 0, there is an ∼50% probability of each type of selection). For larger x_A values, most mutations experience purifying selection, but those that do not generally evolve under balancing selection (see above; Figure 2). How will sex-specific selection and sex-by-genotype effects influence the average magnitude of x_A? As demonstrated below, both factors inflate the scaled mutation size and should thereby expand opportunities for balancing selection relative to positive selection.

An intuitive way to view the scaled mutation size is by rewriting it as a function of the strength of directional selection in each sex. The strength of directional selection in the jth sex can be quantified as β_j = |dln(w_j)/dz_j| = 2ω_jz_j, which is conceptually similar to the selection gradient of quantitative genetics (Lande 1980; Falconer and Mackay 1996) and is roughly proportional to the expected rate of adaptation in Fisher’s geometric model (Orr 2000; Connallon and Clark 2014). Mean mutation size, as a function of the strength and orientation of sex-specific selection, is

$$E\left(x_{\text{A}}\right)=E\left[\frac{\left({\mathit{\omega }}_m{r}_m^2+{\mathit{\omega }}_f{r}_f^2\right)\sqrt{n}}{\sqrt{{\left({\mathit{\beta }}_m{r}_m\right)}^2+{\left({\mathit{\beta }}_f{r}_f\right)}^2+2{\mathit{\beta }}_m{\mathit{\beta }}_f{r}_m{r}_f{\mathit{\rho }}_{\mathit{mf}}}}\right],$$ (9)

which takes into account the probability density of male and female mutational magnitudes among random mutations. To illustrate how sex differences in selection or mutational effects will affect E(x_A), it is useful to consider two extremes of the possible values of β_m relative to β_f: (1) highly asymmetric directional selection (i.e., β_m/β_f → 0 or β_m/β_f → ∞), which may be plausible immediately after a change in the environment, and (2) equally strong directional selection in each sex (β_m/β_f → 1), which is the long-run expectation during adaptive evolution (Lande 1980; Connallon and Clark 2014).

When directional selection is highly asymmetric, the scaled mutation size reduces to x_A ≈ (ω_mr_m² + ω_fr_f²)(n)^0.5/(β_jr_j), where j refers to the sex experiencing directional selection (there is no directional selection in the other sex if it is at its optimum, which we assume here). We arbitrarily present results for the case where males alone experience directional selection (β_m/β_f → ∞; similar results are obtained for the opposite case of β_m/β_f → 0). The average scaled size of a mutation can be approximated using a Taylor series expansion of x_A, to second order with respect to the mean mutation size of each sex. We assume that mutation size follows a bivariate gamma distribution of mutational magnitudes, with equal male and female marginal distributions [E(r) = E(r_m) = E(r_f); var(r) = var(r_m) = var(r_f)] and a between-sex correlation of mutational magnitudes of corr(r_m, r_f) = cov(r_m, r_f)/var(r). The mean size of a random mutation is then

$$\begin{array}{c}E\left(x_{\text{A}}\right)\mathrm{\approx }\frac{E\left(r\right)\left({\mathit{\omega }}_m+{\mathit{\omega }}_f\right)\sqrt{n}}{{\mathit{\beta }}_m}\\ +\frac{2{\mathit{\omega }}_f\sqrt{n}}{{\mathit{\beta }}_m}\frac{\mathrm{var}\left(r\right)\left[1-\mathrm{corr}\left(r_m,r_f\right)\right]}{E\left(r\right)}\\ =E\left(x_m\right)\left(1+\frac{{\mathit{\omega }}_f}{{\mathit{\omega }}_m}+\frac{{\mathit{\omega }}_f}{{\mathit{\omega }}_m}\frac{2\left[1-\mathrm{corr}\left(r_m,r_f\right)\right]}{k}\right)\end{array}$$ (10)

(Appendix D), where k is the shape parameter of the gamma distribution (k > 0), and $x_m=r_m\sqrt{n}/\left(2z_m\right)$ is the male-specific scaled mutation size. From Equation 10 we can visualize three factors that inflate the mean scaled mutation size: E(x_A) increases as the relative strength of stabilizing selection in females increases (ω_f/ω_m increases), and as the distribution of mutation magnitudes becomes increasingly leptokurtic (where k decreases, which skews the distribution toward small values), and the correlation between sexes becomes small to negative [corr(r_m, r_f) decreases]. The scaled mutation size is always elevated as a consequence of selection in females, and the magnitude of this elevation is substantial when the ratio ω_f/ω_m is moderate to large [e.g., for ω_f/ω_m > 1/2 and corr(r_m, r_f) > 0, E(x_A) is inflated by at least 50% relative to the case of no selection in females].

For the scenario of equally strong directional selection in each sex (β = β_m = β_f), the average scaled mutation size becomes

$$\begin{array}{l}E\left(x_{\text{A}}\right)\mathrm{\approx }\mathit{\omega }\frac{E\left(r\right)\sqrt{n}}{\mathit{\beta }}\\ \times \left[\frac{2}{\sqrt{2\left(1+{\mathit{\rho }}_{\mathit{mf}}\right)}}+\frac{\left[1-\mathrm{corr}\left(r_m,r_f\right)\right]\left(1+3{\mathit{\rho }}_{\mathit{mf}}\right)}{2k\left(1+{\mathit{\rho }}_{\mathit{mf}}\right)\sqrt{2\left(1+{\mathit{\rho }}_{\mathit{mf}}\right)}}\right]\end{array}$$ (11)

(Appendix D), where ω = (ω_m + ω_f)/2. E(x_A) now depends on the interaction between three forms of correlation between the sexes: (1) the correlation of mutant magnitudes, corr(r_m, r_f); (2) the correlation of male and female selection orientations, cos(θ_sel); and (3) the mean correlation of mutational orientations, E[cos(θ_mut)] {recall that ρ_mf = cos(θ_sel)E[cos(θ_mut)]}. E(x_A) is inflated by sex differences in selection or mutational effects when the bracketed term in Equation 11 is >1. When mutant phenotypic effects are strongly correlated between the sexes [corr(r_m, r_f), cos(θ_mut) → 1], the bracketed term reduces to 2[2 + 2cos(θ_sel)]^–1/2, which increases as the orientations of selection become increasingly divergent between the sexes (θ_sel > 0). When phenotypic correlations between the sexes are weak [corr(r_m, r_f), cos(θ_mut) → 0], E(x_A) is elevated by a factor of (1 + 4k)/(2k√2) or at least ∼1.4-fold. Figure 3 shows the magnitude of increase for E(x_A), across a range of conditions for sex-specific selection and mutational effects. E(x_A) is inflated across the entire range of parameter space and particularly so when the sexes are selected in different directions (cos(θ_sel) << 1) or the phenotypic effects of mutations are strongly decoupled between the sexes (0 < ρ_mut = corr(r_m, r_f) = E[cos(θ_mut)] << 1).

Figure 3.

Sex differences in the direction of selection, and sex-by-genotype effects of mutations, elevate mean mutation size, E(x_A). Solid curves show the elevation of E(x_A) relative to an idealized population in which the selection and mutational effects are identical between the sexes. The fold change to E(x_A) is based on numerical evaluation of the bracketed term in Equation 11, with ρ_mut = corr(r_m, r_f).

Mechanisms of balancing selection

There are two general mechanisms that may account for individual episodes of balancing selection in the two-sex geometric model considered here. Variation may be maintained by heterozygote advantage in both sexes (s_m2 < s_m1 > 0 and s_f2 < s_f1 > 0) or by sexually antagonistic (SA) selection, which in its broadest sense occurs when male and female fitnesses are maximized by a different genotype at a locus. Sexual antagonism models can be further divided into two subforms: (1) “directional” SA selection, where the fitness of males and females is maximized by different homozygous genotypes at the locus, leading to positive selection in one sex and purifying selection in the other (s_m2 < s_m1 < 0 < s_f1 < s_f2 or s_f2 < s_f1 < 0 < s_m1 < s_m2), and (2) “mixed” SA selection, where there is positive or purifying selection in one sex and heterozygote advantage in the other (s_m2 < s_m1 > 0 or s_f2 < s_f1 > 0).

To estimate the relative frequency of each balancing selection mechanism, we simulated random mutations in Fisher’s geometric model and classified the proportion of balancing selection cases that were attributable to heterozygote advantage, directional SA selection, or mixed SA selection. We first considered a scenario where females were perfectly adapted and males were displaced from their fitness optimum (z_f = 0 and z_m > 0; equivalent results apply when z_f > 0 and z_m = 0). Under this strongly asymmetric scenario, all mutations are deleterious to females and heterozygote advantage in both sexes is not possible. Directional SA selection becomes the dominant mechanism of balancing selection when scaled mutation sizes are small or mutations have strongly correlated phenotypic effects between the sexes (e.g., Figure 4, top left). As phenotypic correlations decrease and the size of mutations increases, the mixed SA selection scenario predominates (Figure 4; Supporting Information, Figure S1, Figure S2, and Figure S3).

Figure 4.

Balancing selection mechanisms and their relative probabilities. SA selection mechanisms (open and shaded areas) account for the vast majority of balancing selection cases. Heterozygote advantage becomes the dominant form of balancing selection in the restrictive case where directional selection and mutant phenotypic effects are strongly concordant between the sexes (e.g., z_m → z_f and ρ_sel, ρ_mut → 1). For each parameter set (z_m, z_f, ρ_sel, ρ_mut), 500,000 balanced polymorphisms were randomly generated using the simulation approach described in the text. These results were used to calculate the proportion of balancing selection arising from heterozygote advantage, mixed SA selection, and directional SA selection. For each parameter combination, mutation magnitudes were generated using a bivariate exponential distribution (gamma with shape parameter k = 1), with equal marginal distributions and correlation of corr(r_m, r_f) = ρ_mut. Small mutations use E[r] = 0.05, medium mutations use E[r] = 0.2, and large mutations use E[r] = 0.4 {these absolute mutation sizes correspond to male-specific scaled sizes of E[x_m] = E(r)(n^0.5)/(2z_m) = 0.25, E[x_m] = 1, and E[x_m] = 2, respectively}. Representative results are shown for n = 25 and ω_m = ω_f = 1/2, with additional results presented in Figure S1, Figure S2, and Figure S3).

We also considered scenarios where both sexes were under similarly strong directional selection (β_f = β_m). In this case, heterozygote advantage remains rare unless the orientation of directional selection and the phenotypic effects of mutations are both strongly correlated between the sexes (e.g., ρ_sel, ρ_mut → 1; Figure 4, Figure S1, Figure S2, and Figure S3). Under the remaining conditions, sexual antagonism is the predominant driver of balancing selection, with the mixed SA selection model representing the most common mechanism across most of the parameter space. Directional SA selection dominates when males and females are selected in different directions, mutational effects are strongly correlated between the sexes, and the scaled mutation size is small (Figure 4, Figure S1, Figure S2, and Figure S3).

The efficacy of balancing selection

The above results (e.g., Equation 7) describe the probability of balancing selection in Fisher’s geometric model, but they do not account for evolutionary dynamics under balancing selection, in which genetic drift is expected to play an important role (Robertson 1962; Ewens and Thomson 1970; Nei and Roychoudhury 1973). To identify criteria for effective balancing selection—the degree to which allele frequencies in a finite population converge to the deterministic equilibrium, $\widehat{q}$—we analyzed the stationary distribution for random mutations within the FGM framework. For a diploid locus in a Wright–Fisher population of effective size N_e, the stationary distribution of a B₂ allele is proportional to

$$f\left(q\right)\mathrm{\propto }\frac{\exp \left[4N_{\text{e}}{\displaystyle \int \left(\mathrm{\Delta }q/q\left(1-q\right)\right)dq}\right]}{q\left(1-q\right)}$$ (12)

(Ewens 2004; Rice 2004; Charlesworth and Charlesworth 2010), where q is the population frequency of B₂, and ∆q is the expected frequency change of B₂ over a single generation. Assuming selection coefficients are small (|s_f1|, |s_f2|, |s_m1|, |s_f2| << 1), the expected frequency change is

$$\mathrm{\Delta }q\mathrm{\approx }\frac{\left[2\left(s_{m1}+s_{f1}\right)-\left(s_{f2}+s_{m2}\right)\right]}{2}q\left(1-q\right)\left(\widehat{q}-q\right),$$ (13)

where $\widehat{q}=\left(s_{f1}+s_{m1}\right)/\left(2\left(s_{m1}+s_{f1}\right)-\left(s_{f2}+s_{m2}\right)\right)$ represents the polymorphic equilibrium under balancing selection (see Appendix E).

The relative importance of selection vs. drift depends on the density of the stationary distribution near the equilibrium$\left(\widehat{q}\right)$, relative to the density near q = 0 and q = 1, with effectively strong selection leading to a high relative density near $\widehat{q}$ (Ewens 2004, pp. 26 and 27). The ratio of the density near the equilibrium, relative to the density near the boundaries, can be described by the ratio

$$\begin{array}{c}R=\frac{{\displaystyle {\int }_{\widehat{q}-\mathit{\epsilon }}^{\widehat{q}+\mathit{\epsilon }}f\left(q\right)dq}}{{\displaystyle {\int }_{0.001-\mathit{\epsilon }}^{0.001+\mathit{\epsilon }}f\left(q\right)dq}+{\displaystyle {\int }_{0.999-\mathit{\epsilon }}^{0.999+\mathit{\epsilon }}f\left(q\right)dq}}\\ \mathrm{\approx }\frac{\mathrm{1}}{\mathrm{1}{\mathrm{0}}^{\mathrm{3}}\widehat{q}\left(1-\widehat{q}\right)\left(\exp \left[-N_{\text{e}}\mathit{\alpha }{\widehat{q}}^2\right]+\exp \left[N_{\text{e}}\mathit{\alpha }{\left(1-\widehat{q}\right)}^2\right]\right)}\\ +\hspace{0.5em}O\left({\mathit{\epsilon }}^3\right)\end{array}$$ (14)

(Appendix F), where α = [2(s_m1 + s_f1) – (s_f2 + s_m2)] represents the net strength of selection at the locus, and 2ε is an arbitrarily small allele frequency interval. Equation 14 summarizes how the efficacy of balancing selection increases with the net strength of selection (α), as equilibrium values become increasingly intermediate ($\widehat{q}$ → 1/2) and with increasing N_e. While there is no specific cutoff to delineate weak from strong balancing selection, values of R > 10 [assuming a sufficiently intermediate equilibrium, i.e., $\widehat{q}$(1 – $\widehat{q}$) > 0.001] are generally sufficient for maintaining polymorphism, and we use this cutoff as a guideline for interpreting subsequent results. [Note that this guideline shares properties with other diffusion-based metrics, such as mean heterozygosity under balancing selection and allele fixation rates or times at polymorphic loci (cf. Robertson 1962; Ewens and Thomson 1970; Nei and Roychoudhury 1973; Hedrick 1974; Ewens 2004; Connallon and Clark 2012; Mullon et al. 2012).]

To evaluate opportunities for effective balancing selection, and to contrast the efficacy under distinct mechanisms of sex-specific selection, we simulated random mutations and characterized the distribution of $\widehat{q}$ and α for mutations meeting balancing selection criteria (Equation 4). We again considered cases of symmetric vs. asymmetric directional selection (β_f/β_m = 0 and β_f/β_m = 1) across a broad range of mutation sizes, selection and mutation correlations between the sexes (ρ_sel, ρ_mut), and dominance (h). Specific values of $\widehat{q}$ and α vary widely among mutations subject to balancing selection, with sexually antagonistic mutations consistently more susceptible to drift than mutations under heterozygote advantage (Figure 5 and Figure S4). Mutations subject to directional SA selection (positive selection in one sex, in purifying selection in the other) are associated with substantially lower values of net selection (α), on average, and are the most likely to evolve primarily by drift (Figure 5 and Figure S4; note that the α-values are plotted on a logarithmic scale in Figure 5, Figure S4, Figure S5, Figure S6, Figure S7, and Figure S8). Mutations causing heterozygote advantage in one sex (mixed SA selection) or both sexes (heterozygote advantage) have similar equilibria and net selection strengths, although mixed SA mutations are somewhat more susceptible to drift. Overall, these qualitative differences among balancing selection mechanisms are robust to variability in male and female fitness landscapes (z_m, z_f, ρ_sel), the strength of genetic correlations between the sexes (ρ_mut), dimensionality (n), and dominance (h) (see Figure S4, Figure S5, Figure S6, Figure S7, and Figure S8).

Figure 5.

The efficacy of selection differs between mechanisms of balancing selection. Results show the distributions (means and 75% confidence intervals) of α and $\widehat{q}$ for mutations under each of the three forms of balancing selection. The areas above the solid curves represent the parameter space for effectively strong selection, based on Equation 14 with criteria R > 10 (see the text surrounding Equation 14). Gray curves (lower) represent the parameter space for a population with effective size N_e = 100,000; black curves (upper) are for populations with N_e = 10,000. For each balancing selection mechanism and for each parameter set (z_m, z_f, ρ_sel), 10,000 balanced polymorphisms were randomly generated using the simulation approach described in the main text. These results were used to calculate means and confidence intervals. For each parameter combination, mutation magnitudes were generated using a bivariate exponential distribution (gamma with shape parameter k = 1), with equal marginal distributions and correlation of ρ_mut = corr(r_m, r_f). Three different mean mutation sizes were used: small mutations use E[r] = 0.05, medium mutations use E[r] = 0.2, and large mutations use E[r] = 0.4 {these absolute mutation sizes correspond to male-specific scaled sizes of E[x_m] = E(r)(n^0.5)/(2z_m) = 0.25, E[x_m] = 1, and E[x_m] = 2, respectively}. Representative results are shown for ρ_mut = 0.9, n = 25, and ω_m = ω_f = 1/2, with additional results presented in Figure S4, Figure S5, Figure S6, Figure S7, and Figure S8.

Discussion

Our analysis yields four theoretical insights. First, there are ample opportunities for balancing selection in dioecious populations. Criteria for balancing selection at a locus are easily met for mutations of intermediate size (on the relevant scale of Fisher’s geometric model: x_A, for a dioecious population), with a maximum probability typically on the order of 25% (conditional on dominance; Figure 2). For all but small-effect mutations, balancing selection is more common than positive selection and may therefore be an important feature of adaptive evolutionary change, as also recently demonstrated for the monoecious case by Sellis et al. (2011). Second, mean mutation size, E(x_A), is elevated by divergent directional selection on male and female phenotypes and by sexually dimorphic phenotypic effects of random mutations (sex-by-genotype effects). Consequently, the relative probability of balancing vs. positive selection should increase in species with separate sexes. Third, under most of the parameter space of sex-specific selection and sex-by-genotype effects, balancing selection primarily involves sexually antagonistic alleles. Finally, in a finite population, the effective strength of sexually antagonistic balancing selection is expected to generally be weaker than that of heterozygote advantage, particularly under directional SA selection (the mixed SA selection model shows a smaller departure from the het. advantage model). Therefore, SA alleles under balancing selection may often exhibit population frequencies that are far from equilibrium (see Livingstone 1992; Connallon and Clark 2013b), evolutionary dynamic properties that mimic those of neutral or weakly deleterious alleles (Connallon and Clark 2012), and relatively weak molecular population genetic signatures of partial selective sweeps (Connallon and Clark 2013b; see Sellis et al. 2011 for an additional discussion of balancing selection signals in FGM).

Fitness trade-offs and opportunities for balancing selection

Several studies have reported sexually antagonistic genetic variation for fitness, or fitness components, in animal and plant populations (reviewed in Bonduriansky and Chenoweth 2009; Kirkpatrick 2009; van Doorn 2009; Pennell and Morrow 2013), yet the genetic basis of this quantitative genetic variation is not well known. Much of this phenotypic variation could be attributable to many loci that segregate for low-frequency sexually antagonistic alleles that are maintained at a balance between recurrent mutation, net purifying selection, and genetic drift. Alternatively, the variation could be primarily attributable to a small set of loci that segregate for intermediate-frequency sexually antagonistic alleles under balancing selection.

Population genetics models are often used to evaluate the plausibility of sexual antagonism as a common mechanism for balancing selection (Kidwell et al. 1977; Pamilo 1979; Curtsinger 1980; Prout 2000; Patten and Haig 2009; Unckless and Herren 2009; Fry 2010; Arnqvist 2011; Jordan and Charlesworth 2012; Patten et al. 2013), and most of this theory has focused on “SA directional” selection models (but see Kidwell et al. 1977; Gavrilets and Rice 2006; Mokkonen et al. 2011), where only a small fraction of the conceivable parameter space for sex-specific fitness and dominance will generate balancing selection at a diploid locus (Prout 2000; Patten and Haig 2009). A broader class of fitness trade-off scenarios for balancing selection—e.g., between environments that vary over time or space (Levene 1953; Haldane and Jayakar 1963) and antagonistic pleiotropy between traits (Rose 1982; Curtsinger et al. 1994)—suffers from similarly severe parameter restrictions (Prout 2000). On the other hand, fitness landscape topography biases the range of fitness effect values that beneficial alleles are likely to take (Orr 2005a,b, 2010; Manna et al. 2011; Sellis et al. 2011), with mutations that are subject to fitness trade-offs (including sexually antagonistic alleles) preferentially occupying portions of the conceivable parameter space that are the most amenable to balancing selection (Gillespie 1978; Fry 2010).

Our analysis of the dioecious Fisher’s geometric model, coupled with the recent monoecious analysis by Sellis et al. (2011), suggests that balancing selection may arise naturally by way of the structure of multidimensional fitness landscapes. Because high-dimensional systems like FGM ensure that improvements along some trait axes are coupled with maladaptation along others (Orr 1998), adaptive mutations and alleles subject to balancing selection are nearly always subject to fitness trade-offs. Sex differences in selection (ρ_sel < 1) and sex-by-genotype effects on trait variation (ρ_mut < 1) increase the effective dimensionality of Fisher’s geometric model and thereby expand opportunities for fitness trade-offs between individuals of a population. In models that ignore the effects of sex, balancing selection by heterozygote advantage requires that the population be relatively close to a local optimum in the fitness landscape (Sellis et al. 2011). Such restrictions are alleviated in the two-sex FGM, whose net effect is to elevate the standardized mutation size and consequently the importance of balancing relative to positive selection during the process of adaptation.

Whereas fitness trade-offs are expected to manifest across the full parameter space for balancing selection in Fisher’s geometric model, the nature of the underlying trade-offs varies as a function of specific properties of mutation and the phenotypic orientation of sex-specific selection. The classical heterozygote balance model (Fisher 1922) dominates when mutant phenotypic effects are strongly correlated between the sexes and directional selection is similar in strength and orientation between males and females (i.e., ρ_mut, ρ_sel >> 0). Sexual antagonism, in the absence of heterozygote advantage in either sex (SA directional), dominates as patterns of directional selection diverge between the sexes (ρ_sel decreases), but mutational effects remain tightly coupled (ρ_mut >> 0). A mixed scenario of sexual antagonism—with heterozygote advantage in one sex and positive/purifying selection in the other (SA mixed)—represents the primary mechanism for maintaining balanced polymorphisms when mutant phenotypic effects are poorly correlated between the sexes.

We can tentatively gauge opportunities for each mode of balancing selection by considering relevant empirical estimates of selection and mutational effects on phenotypes expressed by both sexes. Current estimates of sex-specific selection reveal widespread asymmetries in the strength or direction of selection on individual phenotypic traits (Cox and Calsbeek 2009; Stulp et al. 2012), and recent estimates of multivariate selection from humans and two insect species provide concrete examples of sex differential selection within multivariate trait space [ρ_sel ≈ –0.61 for three traits of the Indian meal moth (Lewis et al. 2011), ρ_sel ≈ –0.73 for seven traits in Drosophila serrata (Gosden et al. 2012), and ρ_sel ≈ 0.22 for seven traits in a human population (Stearns et al. 2012)]. Although this set of studies is small, selection patterns are consistently divergent between the sexes, which implies that mutations with sexually antagonistic fitness effects should be common (Connallon and Clark 2014) and that sexual antagonism should play an important role in generating balancing selection (i.e., SA directional or SA mixed mechanisms, as shown here). Estimates of quantitative genetic variability among male and female traits mostly show strong and positive phenotypic correlations between the sexes [e.g., detectable, but modest gene-by-sex effects (Poissant et al. 2010; Houle and Fierst 2012; Griffin et al. 2013)], which tentatively suggests an important potential role for SA directional balancing selection. There is, however, a great need for studies that measure the joint effects of random mutations on male and female traits (Houle and Fierst 2012). These should prove valuable for further evaluating predictions of balancing selection that emerge from Fisher’s geometric model, as well as the severity of genetic constraints to sex-specific adaptation and the evolution of sexual dimorphism.

Conclusions

Balancing selection may be common within diploids (Sellis et al. 2011) and typically involves alleles with sexually antagonistic fitness effects. The genetic architecture of sex-specific fitness is likely to be complex, with processes of recurrent mutation, balancing selection, and genetic drift each potentially contributing to the maintenance of fitness variation (Bonduriansky and Chenoweth 2009; Connallon and Clark 2012). Our results show that the parameter criteria for sexually antagonistic balancing selection are not difficult to meet, and consequently intermediate-frequency sexually antagonistic alleles may contribute substantially to the high levels of quantitative genetic variance that are commonly observed in life-history traits and fitness components (Houle 1992; Charlesworth and Charlesworth 1999; Charlesworth and Hughes 1999). However, the common assumption that balanced polymorphisms are maintained at or near equilibrium may be unrealistic, particularly with regard to sexually antagonistic alleles, which are particularly susceptible to genetic drift (see Hesketh et al. 2013).

Finally, both theory and data suggest that the severity of sexual antagonism should generally increase during the course of adaptation, with sexually antagonistic mutations being rare in poorly adapted populations and common in well-adapted ones (e.g., Long et al. 2012; Berger et al. 2014; Connallon and Clark 2014), but this is not always so (see Delcourt et al. 2009; Punzalan et al. 2014). The relative probability of balancing vs. positive selection similarly increases during the course of adaptation in Fisher’s geometric model [e.g., because the average mutation size increases as the distance to the optimum shrinks (Sellis et al. 2011); see above]. Taken together, we expect that both sexual antagonism and balancing selection will be most rare following an abrupt change in the environment, in which both sexes are displaced from their fitness optima. As the population adapts, sexual antagonism and balancing selection should become increasingly important in shaping patterns of population genetic variation, leading to a reconfiguration of the sex-specific distribution of fitness effects among random mutations (Connallon and Clark 2014) and of the genetic basis of phenotypic (including fitness) variance between poorly and well-adapted populations.

Appendix A: Movements in Phenotypic Space and Selection Coefficients

Let A_j = {x_1j, x_2j, … , x_nj} represent the phenotypic value expressed by wild-type individuals of the jth sex (j = {f, m}); O_j = {O_1j, O_2j, … , O_nj} is the location of its optimum, and O_j′ = O_j – A_j = {o_1j, o_2j, … , o_nj} is a rescaled optimum that becomes useful elow. Consider a mutation with a homozygous magnitude of r_m and r_f and heterozygous magnitude of r_mh and r_fh. The phenotypic value expressed by a heterozygote will be A_j + hy_j, and a homozygote will be A_j + y_j, where y_j = {y_1j, y_2j, … , y_nj} is a mutation vector that describes the set of phenotypic changes in each of the n traits (this is described in the Simulations section of the main text). The Euclidean distance between a wild-type individual of sex j and the respective phenotypic optimum is

$$z_j=\sqrt{{\displaystyle {\sum }_{i=1}^n{(O_{\mathit{ij}}-x_{\mathit{ij}})}^2}}.$$

The distance between a heterozygote and its optimum is

$$\begin{array}{c}{z}_{j\left(\text{het}\right)}=\sqrt{{\displaystyle {\sum }_{i=1}^n{[O_{\mathit{ij}}-(x_{\mathit{ij}}+h{y}_{\mathit{ij}})]}^2}}\\ =\sqrt{{\displaystyle {\sum }_{i=1}^n{(O_{\mathit{ij}}-x_{\mathit{ij}})}^2}+{\displaystyle {\sum }_{i=1}^n{(h{y}_{\mathit{ij}})}^2}-2h{\displaystyle {\sum }_{i=1}^n{o}_{\mathit{ij}}{y}_{\mathit{ij}}}}\\ =\sqrt{z_j{}^2+{(h{r}_j)}^2-2h{\displaystyle {\sum }_{i=1}^n{o}_{\mathit{ij}}{y}_{\mathit{ij}}}}.\end{array}$$

The exact result in Equation 2 is obtained by substituting y_im = r_mm_i/M, where M² = ∑m_i², and y_if = r_ff_i/F, where F² = ∑f_i² (as described in the Simulations section). With sufficiently high dimensionality (n > 10 works well) m_i/M and f_i/F converge to m_i/√(n) and f_i/√(n) (Connallon and Clark 2014), which accounts for the final approximations in Equation 2. The same approach can be taken to obtain values of z_j(hom).

For a random mutation, the fitness of the three genotypes will be w_{j(wild type)} = exp[–ω_jz_j²], w_j(het) = exp[–ω_jz_j(het)²], and w_j(hom) = exp[–ω_jz_j(hom)²]. Selection coefficients are scaled relative to the wild-type fitness. The heterozygote and homozygote selection coefficients for a random mutation are (respectively) s_j1 = w_j(het)/w_{j(wild type)} – 1 = exp[ω_j(z_j² – z_j(het)²)] – 1, and s_j2 = w_j(hom)/w_{j(wild type)} – 1 = exp[ω_j(z_j² – z_j(hom)²)] – 1.

Appendix B: Modes of Selection and Their Probabilities

Define t = ln(1 + s) as the natural logarithm of fitness for a genotype with relative fitness 1 + s. Under weak selection (s << 1), t is well approximated as t = ln(1 + s) ∼ s. The criterion for balancing selection at a diploid locus can be approximated as

$$0\mathrm{<}{t}_{\text{mhet}}+t_{\text{fhet}}\mathrm{>}{t}_{\text{mhom}}+t_{\text{fhom}},$$ (B1)

where t_mhet = ln(1 + s_m1), t_fhet = ln(1 + s_f1), t_mhom = ln(1 + s_m2), and t_fhom = ln(1 + s_f2). The criterion for invasion of a new mutation is modified to

$$0\mathrm{<}{t}_{\text{mhet}}+t_{\text{fhet}}.$$ (B2)

Let the mutation effect size in heterozygotes be r_mh and r_fh, where h is the dominance coefficient (assumed to fall in the range 0 < h < 1), and the mutation size in homozygotes be r_m and r_f. From Equation B1, along with Equation 2 and its surrounding text from the main text, the criteria for balancing selection can be restated as

$$\frac{h({\omega }_m{r}_m^2+{\omega }_f{r}_f^2)\sqrt{n}}{2}<{\omega }_m{r}_m{\displaystyle \sum _i{o}_{\mathit{mi}}{m}_i}+{\omega }_f{r}_f{\displaystyle \sum _i{o}_{\mathit{fi}}{f}_i}<\frac{(1-h^2)({\omega }_m{r}_m^2+{\omega }_f{r}_f^2)\sqrt{n}}{2(1-h)}.$$ (B3)

Terms m_i and f_i are standard normal variables with cov(m_i, f_i) = ρ_mut. The function in the middle is therefore normally distributed with respective mean of zero and variance of

$$\begin{array}{c}\mathrm{var}=\mathrm{var}\left({\omega }_m{r}_m{\displaystyle \sum _i{o}_{\mathit{mi}}{m}_i}+{\omega }_f{r}_f{\displaystyle \sum _i{o}_{\mathit{fi}}{f}_i}\right)\\ ={({\omega }_m{r}_m)}^2\left({\displaystyle \sum _i{o}_{mi}^2\text{var}(m_i)}\right)+{({\omega }_f{r}_f)}^2\left({\displaystyle \sum _i{o}_{fi}^2\text{var}(f_i)}\right)+2{\omega }_m{\omega }_f{r}_m{r}_f\left({\displaystyle \sum _i{o}_{\mathit{mi}}{o}_{\mathit{fi}}\text{cov}(m_i,f_i)}\right)\\ ={({\omega }_m{r}_m)}^2\left({\displaystyle \sum _i{o}_{mi}^2}\right)+{({\omega }_f{r}_f)}^2\left({\displaystyle \sum _i{o}_{fi}^2}\right)+2{\omega }_m{\omega }_f{r}_m{r}_f{\rho }_{\text{mut}}\left({\displaystyle \sum _i{o}_{\mathit{mi}}{o}_{\mathit{fi}}}\right)\\ ={({\omega }_m{r}_m{z}_m)}^2+{({\omega }_f{r}_f{z}_f)}^2+2{\omega }_m{\omega }_f{z}_m{z}_f{r}_m{r}_f{\rho }_{\text{mut}}\text{cos}({\theta }_{\text{sel}}),\end{array}$$

where ρ_mf = ρ_mutcos(θ_sel). After dividing the terms of the inequality (B3) by √(var), the function in the middle is now distributed as a standard normal [N(0, 1)], and we can now solve for the probability of balancing selection,

$$\text{Pr}\left(\text{bal.}\right)\approx \frac{1}{\sqrt{2\pi }}{\displaystyle {\int }_b^a{e}^{-\left(t^2/2\right)}dt}=\text{Φ}(a)-\text{Φ}(b),$$

where

$$a=\frac{(1-h^2)}{(1-h)}\frac{({\omega }_m{r}_m^2+{\omega }_f{r}_f^2)\sqrt{n}}{2\sqrt{{({\omega }_m{z}_m{r}_m)}^2+{({\omega }_f{z}_f{r}_f)}^2+2{\omega }_m{z}_m{r}_m{\omega }_f{z}_f{r}_f{\rho }_{\mathit{mf}}}}=\frac{(1-h^2)}{(1-h)}{x}_{\text{A}}$$

and

$$b=h\frac{({\omega }_m{r}_m^2+{\omega }_f{r}_f^2)\sqrt{n}}{2\sqrt{{({\omega }_m{z}_m{r}_m)}^2+{({\omega }_f{z}_f{r}_f)}^2+2{\omega }_m{z}_m{r}_m{\omega }_f{z}_f{r}_f{\rho }_{\mathit{mf}}}}=h{x}_{\text{A}},$$

which accounts for Equation 7 in the main text. The same basic approach can be taken to calculate the probability that selection will favor invasion of a rare mutation. In this case, Pr(s_m1 + s_f1 > 0) = 1 – Φ(b), as in Equation 6.

Appendix C: The Maximum Probability of Balancing Selection

To find the specific value of x_A that maximizes the probability of balancing selection, we restate Equation 7 as

$$\mathrm{\Phi }\left(\frac{(1-h^2)}{(1-h)}{x}_{\text{A}}\right)-\mathrm{\Phi }(h{x}_{\text{A}})=\frac{1}{2}\left[\text{erf}\left(\frac{(1-h^2)}{(1-h)\sqrt{2}}{x}_{\text{A}}\right)-\text{erf}\left(\frac{h}{\sqrt{2}}{x}_{\text{A}}\right)\right],$$

which has a first derivative, with respect to x_A, of

$$\frac{d}{d{x}_{\text{A}}}\left[\mathrm{\Phi }\left(\frac{(1-h^2)}{(1-h)}{x}_{\text{A}}\right)-\mathrm{\Phi }(h{x}_{\text{A}})\right]=\frac{h}{\sqrt{2\pi }}\exp \left(-\frac{h^2}{2}{x}_{\text{A}}^2\right)\left[\frac{(1-h^2)}{h(1-h)}\exp \left(-x_{\text{A}}^2\left[\frac{1-h^2(3-2h)}{2{(1-h)}^2}\right]\right)-1\right].$$

The maximum probability of balancing selection is found by setting this function to zero and solving for x_A, which yields

$${\widehat{x}}_{\text{A}}=(1-h)\sqrt{\frac{2}{1-h^2(3-2h)}\ln \left[\frac{(1-h^2)}{h(1-h)}\right]}.$$

Appendix D: Mean Standardized Mutation Size

For β_m > 0 and β_f = 0 (z_f = 0), we can approximate E[x_A] by performing a second-order Taylor series expansion in the vicinity of r_m, r_f = E(r) and take the expectation, which leads to

$$\begin{array}{c}E\left[x_{\text{A}}\right]=\frac{E(r){\omega }_m\sqrt{n}}{{\beta }_m}+E\left[\frac{{\omega }_f{r}_f^2\sqrt{n}}{{\beta }_m{r}_m}\right]\\ \approx \frac{E(r)({\omega }_m+{\omega }_f)\sqrt{n}}{{\beta }_m}\left[1+\frac{{\omega }_{f}E\left\{{[r_m-E(r)]}^2+{[r_f-E(r)]}^2-2[r_m-E(r)][r_f-E(r)]\right\}}{({\omega }_m+{\omega }_f){[E(r)]}^2}\right]\\ =\frac{E(r)({\omega }_m+{\omega }_f)\sqrt{n}}{{\beta }_m}\left[1+\frac{2{\omega }_f[\text{var}(r)-\text{cov}(r_m,r_f)]}{({\omega }_m+{\omega }_f){[E(r)]}^2}\right].\end{array}$$

Equation 10 in the main text can be found by substituting β_m = 2ω_mz_m, E(r) = kθ, var(r) = kθ², and corr(r_m, r_f) = cov(r_m, r_f)/var(r) and rearranging. Using the same approach for β = β_m = β_f, we get

$$\begin{array}{c}E\left[x_{\text{A}}\right]=E\left[\frac{({\omega }_m{r}_m^2+{\omega }_f{r}_f^2)\sqrt{n}}{\beta \sqrt{r_m^2+r_f^2+2r_m{r}_f{\rho }_{\mathit{mf}}}}\right]\\ \approx \frac{E(r)({\omega }_m+{\omega }_f)\sqrt{n}}{\beta \sqrt{2(1+{\rho }_{\mathit{mf}})}}\left[1+\frac{(1+3{\rho }_{\mathit{mf}})E\left\{{[r_m-E(r)]}^2+{[r_f-E(r)]}^2-2[r_f-E(r)][r_m-E(r)]\right\}}{8(1+{\rho }_{\mathit{mf}}){[E(r)]}^2}\right]\\ =\frac{E(r)({\omega }_m+{\omega }_f)\sqrt{n}}{\beta \sqrt{2(1+{\rho }_{\mathit{mf}})}}\left[1+\frac{(1+3{\rho }_{\mathit{mf}})[\text{var}(r)-\text{cov}(r_f,r_m)]}{4(1+{\rho }_{\mathit{mf}}){[E(r)]}^2}\right].\end{array}$$

Substituting E(r) = kθ, var(r) = kθ², and corr(r_m, r_f) = cov(r_m, r_f)/var(r) and simplifying yields Equation 11 in the main text.

Appendix E: Allele Frequency Dynamics and Equilibrium of B₂ Under Balancing Selection

Among zygotes, let q_m be the frequency of B₂ transmitted from males and q_f be the frequency of B₂ transmitted from females. The genotypic frequencies among zygotes are q_mq_f = [B₂B₂]; q_m(1 – q_f) + q_f(1 – q_m) = [B₁B₂]; (1 – q_m)(1 – q_f) = [B₁B₁]. Let q = (q_m + q_f)/2, and 2ε = q_m – q_f. Assuming weak selection (Nagylaki 1979; Charlesworth and Charlesworth 2010), we approximate the allele frequency dynamics to first order in ε. Following selection, the expected allele frequency change in males and females will be

$$\begin{array}{c}\mathrm{\Delta }{q}_m=\frac{(q^2-{\epsilon }^2)(1+s_{m2})+[q(1-q)+{\epsilon }^2](1+s_{m1})}{1+s_{m2}(q^2-{\epsilon }^2)+2s_{m1}[q(1-q)+{\epsilon }^2]}-q-\epsilon \approx \frac{q(1-q)[s_{m1}+q(s_{m2}-2s_{m1})]}{1+q^2{s}_{m2}+2q(1-q)s_{m1}}-\epsilon \\ \mathrm{\Delta }{q}_f=\frac{(q^2-{\epsilon }^2)(1+s_{f2})+[q(1-q)+{\epsilon }^2](1+s_{f1})}{1+s_{f2}(q^2-{\epsilon }^2)+2s_{f1}[q(1-q)+{\epsilon }^2]}-q+\epsilon \approx \frac{q(1-q)[s_{f1}+q(s_{f2}-2s_{f1})]}{1+q[s_{f2}q+2s_{f1}(1-q)]}+\epsilon .\end{array}$$

The net change after a generation is (Δq_m + Δq_f)/2,

$$\begin{array}{c}\mathrm{\Delta }q=\frac{\mathrm{\Delta }{q}_f+\mathrm{\Delta }{q}_m}{2}\approx \frac{[2(s_{m1}+s_{f1})-(s_{f2}+s_{m2})]q(1-q)(\widehat{q}-q)}{2}\\ \widehat{q}=\frac{s_{f1}+s_{m1}}{2(s_{m1}+s_{f1})-(s_{f2}+s_{m2})},\end{array}$$

where $\widehat{q}$ is the balanced polymorphic equilibrium when it exists.

Appendix F: Criteria for Effective Balancing Selection

Following Ewens (2004, pp. 26 and 27), we wish to know the relative proportion of the density, f(q), that is found near the deterministic equilibrium value ($\widehat{q}$), compared to the allele frequency boundaries q = 0 and q = 1. Using a second-order Taylor series expansion, we can approximate the area under the stationary distribution near arbitrary frequency point a:

$$\begin{array}{c}{\displaystyle {\int }_{a-\epsilon }^{a+\epsilon }f(q)dq}\propto {\displaystyle {\int }_{a-\epsilon }^{a+\epsilon }\frac{\exp [-\alpha {(q-\widehat{q})}^2]}{q(1-q)}dq}\\ \approx f(a){\displaystyle {\int }_{a-\epsilon }^{a+\epsilon }dq}+\left(\frac{\partial f(a)}{\partial q}\right){\displaystyle {\int }_{a-\epsilon }^{a+\epsilon }(q-a)dq}+\left(\frac{1}{2}\frac{{\partial }^{2}f(a)}{\partial q^2}\right){\displaystyle {\int }_{a-\epsilon }^{a+\epsilon }{(q-a)}^{2}dq}\\ =2\epsilon f(a)+\frac{2{\epsilon }^3}{3}\left(\frac{1}{2}\frac{{\partial }^{2}f(a)}{\partial q^2}\right).\end{array}$$

By substituting a = 0.001, a = 0.999, and a = $\widehat{q}$ (as appropriate), we obtain Equation 14 of the main text.