Bistable Mutation-Selection Equilibria and Violations of Fisher’s Theorem in Tetraploids: Insights from Nonlinear Dynamics

Abstract

Polyploidy and whole genome duplication (WGD) are widespread biological phenomena with substantial cellular, meiotic, and genetic effects. Despite their prevalence and significance across the tree of life, population genetics theory for polyploids is not well developed. The lack of theoretical models limits our understanding of polyploid evolution and restricts our ability to harness polyploidy for crop improvement amidst increasing environmental stress. To address this gap, we developed and analyzed deterministic models of mutation-selection balance for tetraploids under polysomic (autotetraploid) and disomic (allotetraploid) inheritance patterns and arbitrary dominance relationships. We also introduced a new mathematical framework based on ordinary differential equations and nonlinear dynamics for analyzing the models. We find that autotetraploids approach Hardy-Weinberg Equilibrium 33% faster than allotetraploids, but the different tetraploid inheritance models show little differences in mutation load and allele frequency at mutation-selection balance. Our model also reveals two bistable points of mutation-selection balance for dominant alleles with biased mutation rates over a wide range of selection coefficients in the tetraploid models compared to bistability in only a narrow range for diploids. Finally, using discrete time simulations, we explore the temporal dynamics of allele frequency and fitness change and compare these dynamics to the predictions of Fisher’s Fundamental Theorem of Natural Selection. While Fisher’s predictions generally hold, we show that the bistable dynamics for dominant mutations fundamentally alter the associated temporal dynamics. Overall, this work develops foundational theoretical models that will facilitate the development of population genetic models and methodologies to study evolution in empirical tetraploid populations.

Introduction

Whole genome duplication (polyploidy) represents one of the most dramatic mutational events in evolution, instantaneously doubling chromosome number and fundamentally altering meiotic (Lloyd and Bomblies, 2016), cellular (Fernandes Gyorfy et al., 2021), and evolutionary processes (Otto, 2007; Van De Peer et al., 2017). This genomic restructuring poses an apparent paradox: while theory predicts that reduced efficacy of selection in polyploids should lead to the accumulation of deleterious mutations and higher mutation load at mutation-selection balance (Otto and Whitton, 2000; Otto, 2007), polyploid lineages flourish across the tree of life, especially among flowering plants (Van De Peer et al., 2017; One Thousand Plant Transcriptomes Initiative, 2019; McKibben et al., 2024). Understanding how polyploidy reshapes mutation-selection dynamics — particularly through the masking of recessive deleterious alleles as ploidy increases, which can reduce mutation load in polyploids (Otto and Whitton, 2000) — is important to understanding the success of polyploid species. However, quantitative frameworks for predicting these altered evolutionary equilibria remain incomplete, especially for allopolyploids with disomic inheritance that comprise many polyploid species (Barker et al., 2016; Dunn and Sethuraman, 2024). Despite polyploidy’s applications in agriculture (Renny-Byfield and Wendel, 2014; Sattler et al., 2016; Trojak-Goluch et al., 2021), its role in adaptating to environmental stress (Van De Peer et al., 2021; Turcotte et al., 2024; Blake-Mahmud et al., 2025), and its emerging relevance to cancer biology (Bielski et al., 2018; Matsumoto et al., 2021) and tissue repair (Bailey et al., 2021), we lack unified theoretical models that can predict how the balance between mutation and selection differs across ploidy — a gap that limits our ability to understand foundational evolutionary processes in polyploids.

Polyploids are broadly categorized as either autopolyploid or allopolyploid based on their underlying genetics and chromosome pairing behavior during meiosis. Specifically, genetic or cytological definitions group auto- and allopolyploids into those with polysomic and disomic inheritance, respectively (Ramsey and Schemske, 2002; Parisod et al., 2010). Polysomic inheritance occurs when all of the chromosomes pair together in meiosis at random (i.e. with no preferential pairing) so that all possible allelic combinations occur with equal probability. Alternatively, disomic inheritance occurs when there are clearly distinguishable pairs of chromosomes (i.e. subgenomes) that always pair together and have inheritance patterns similar to that of diploids. Thus, under disomic inheritance, some allelic combinations in the resulting gametes do not occur because alleles in the same subgenome cannot segregate into the same gamete.

The distinction between auto- and allopolyploidy has substantialy shaped both empirical research (Barker et al., 2016; Doyle and Sherman-Broyles, 2017) and theoretical modeling (Haldane, 1926; Wright, 1938; Hill, 1970). Notably, most of these models focus exclusively on polysomic inheritance patterns (i.e., autopolyploidy) and do not consider disomic inheritance (i.e., allopolyploidy). This may be because each independent subgenome/pair of homologs under disomic inheritance can be treated as a diploid which, in turn, connects allopolyploids to the wealth of population genetics theory and methods developed for diploids. This treatment is often helpful for bioinformatic, genomic, and inference analysis (Blischak et al., 2023). However, this approach may also obscure fundamental evolutionary processes, such as selection that operates on the integrated phenotype arising from the combination of subgenomes, which may be impacted by subgenome dominance and homoeolog expression bias (Grover et al., 2012; Wang et al., 2022; Zhang et al., 2023) or masking effects (Vekemans et al., 2025). Despite being more common in the empirical literature (Soltis et al., 2016; Spoelhof et al., 2017), allopolyploids have received little formal theoretical attention. Although models for polyploids have advanced to account for gametic/panmictic disequilibrium (Rowe, 1982; Rowe and Hill, 1984), population structure (Ronfort et al., 1998), double reduction (Butruille and Boiteux, 2000; Huang et al., 2019), and multiple loci (Griswold and Williamson, 2017; Mostafaee and Griswold, 2019; Griswold and Asif, 2023), all of these advances assume polysomic inheritance patterns. As a result, key questions about the dynamics of mutation and natural selection in disomic polyploids remain unanswered. Understanding these dynamics is important not only for insight into the evolutionary trajectories of allopolyploids, but also for interpreting empirical patterns of genetic variation in species with complex genomic structures and demographic histories (Conover and Wendel, 2022).

Among the principal goals of population genetics theory is the prediction and interpretation of patterns of genetic variation within and across populations. Mutation-selection balance (MSB) is one of the foundational concepts for understanding this variation and occurs when the rate at which variation is introduced by mutation is equal to the rate at which it is removed by purifying selection (Crow and Kimura, 1970). In this context, points of MSB primarily arise as equilibria in deterministic models under an infinite population size limit (Moran, 1977) (see Bürger (2000) for a comprehensive review of such models), but a more general concept of dynamic MSB has also been developed for finite populations when considering both deleterious and beneficial mutations (Haigh, 1978; Goyal et al., 2012). The deterministic regime which we adopt here was first considered for a diploid population by Haldane (1927) and led to approximations for the segregating frequency of a mutant allele at MSB as $\sqrt{\mu /s}$ for fully recessive mutants and $\mu /sh$ for partially to fully dominant mutants, where $\mu$, $s$, and $h$ are the mutation rate, selection coefficient, and dominance coefficient, respectively (Crow and Kimura, 1970). Ronfort (1999) extended these results to polysomic tetraploids with the approximations $\sqrt[4]{\mu /s}$ for fully recessive mutants and ${\mu /sh}_1$ for partial or fully dominant mutants where $h_1$ is the dominance coefficient for an individual with a single mutant copy. Identifying MSB helps to quantify the expected amount of segregating genetic variation at a locus and, by extension, mutation load. This, in turn, facilitates comparisons of the expected variation due to MSB with the variation present in genomic data (e.g. (Simmons and Crow, 1977)). Although variation in real populations can also be maintained by balancing or linked selection (Marion and Noor, 2023), models of MSB have been used to argue for the maintenance of sex and recombination (Charlesworth, 1990; Chasnov, 2000) and were recently applied to specific classes of mutations (Weghorn et al., 2019) and cancer progression (Persi et al., 2021). Beyond predicting patterns of genetic variation, understanding mutation-selection dynamics is crucial for applications ranging from crop breeding (Dwivedi et al., 2023) to conservation genetics (Willi et al., 2022), where the maintenance of genetic diversity under changing environmental conditions is critical.

Here, we address the theoretical gap of mutation-selection balance for polyploids by developing a deterministic mutation-selection model for allotetraploids adapted from the two-locus, two-allele models for diploids developed by Kimura (1956) and Lewontin and Kojima (1960). We also present a robust approach based on nonlinear dynamics for identifying mutation-selection equilibria under arbitrary dominance models for tetraploids with disomic inheritance patterns (i.e. allotetraploids) and compare this analysis to that for previously developed models of tetraploids with polysomic inheritance patterns (i.e. autotetraploids) and diploids. Using this approach, we find that when selection is parameterized to act solely on the total dosage of an allele (i.e. selection acts equally on both subgenomes), ploidy is the primary determinant of equilibrium allele frequency and mutation load at MSB for both auto- and allotetraploids. Furthermore, for completely dominant deleterious mutations with unequal forward and back mutation rates, multiple points of MSB can coexist in both the tetraploid and diploid models. Bistable equilibria have not been previously observed in autotetraploids and could have important implications for evolutionary trajectories. These bistable equilibria arise from a pair of saddle-node bifurcations and are present for a wide range of parameter space for the tetraploid models. We also validate the model analysis by comparing numerical estimates for mutation load from our model to previously derived analytical approximations (Haldane, 1937; Ronfort, 1999). Finally, we investigate the temporal dynamics of how allele frequency and fitness change over time and compare these results to predictions from Fisher’s Fundamental Theorem of Natural Selection. For alleles under purifying selection, we assess the impacts of bistability and bifurcations on the rate of approach to mutation-selection equilibria. Our framework provides quantitative foundations for understanding how ploidy shapes the fundamental balance between mutation and selection, with immediate applications to both crop improvement and the prediction of evolutionary trajectories in polyploid populations.

Methods

General Model Description and Notation

Here, we introduce a general mathematical model for the effects of selection and mutation on changes over generations in gamete frequency for diploids, autotetraploids (tetraploids with polysomic inheritance), and allotetraploids (tetraploids with disomic inheritance). We extend a classical mutation-selection framework to model a biallelic locus in infinite populations of auto- and allotetraploids with random mating and discrete non-overlapping generations. For each generation $t$, we begin with gamete frequencies $g^t$, progress through the processes of random mating, selection, meiosis, and mutation, and conclude with gamete frequencies $g^{t+1}$ in generation $t+1$.

$$g^t\stackrel{\text{Random Mating}}{\to }G\stackrel{\text{Selection}}{\to }{G}^{\prime }\stackrel{\text{Meiosis}}{\to }{g}^{\prime }\stackrel{\text{Mutation}}{\to }{g}^{t+1}$$

In general, $g$ will be used to refer to gamete frequencies and $G$ for genotype frequencies.

Under this modeling approach, the gamete frequencies in the next (discrete) generation can be written as an iterative map or difference equation of the form

$${\mathbf{g}}^{\mathbf{t}+\mathbf{1}}=\mathbf{f}\left({\mathbf{g}}^{\mathbf{t}},\mathbf{p}\right)$$ (1)

where $\mathbf{g}$ is a vector of gamete frequencies and $\mathbf{p}$ is a vector of parameter values for mutation, selection, and dominance.

Then, to find the change in gamete frequency in a single generation, we have

$$\frac{\text{Δ}\mathbf{g}}{\text{Δ}t}={\mathbf{g}}^{\mathbf{t}+\mathbf{1}}-{\mathbf{g}}^{\mathbf{t}}$$ (2)

where $\text{Δ}t$ is one unit of discrete generational time. Because the dynamics are more easily modeled by differential equations, we follow the analysis of Hofbauer (1985) to replace the discrete time difference ${\mathbf{g}}^{\mathbf{t}+\mathbf{1}}-{\mathbf{g}}^{\mathbf{t}}$ with the continuous time derivative $\dot{\mathbf{g}}=d\mathbf{g}/dt$ by taking the limit $\text{Δ}t\to 0$. Then, we have a system of ordinary differential equations (ODEs) given by:

$$\dot{\mathbf{g}}={\mathbf{g}}^{\mathbf{t}+\mathbf{1}}-{\mathbf{g}}^{\mathbf{t}}$$ (3)

This construction of ODEs is equivalent to that presented for diploids by Bürger (1998).

In the following section, we present the allotetraploid model in detail and define the notation and parameters used throughout the paper. The complete diploid and autotetraploid models are included in the supplementary material, and the differences between these and the allotetraploid model are briefly noted in “Comparison to Diploid and Autotetraploid Models.”

Allotetraploid One-Locus Model

We model a single biallelic locus in an allotetraploid population in which each individual has two distinct subgenomes with independent segregation between them. That is, we do not allow for homoeologous exchanges or homoeologous gene conversion. Our model is conceptually similar to the two-locus diploid model first presented by Kimura (1956). Much of the notation is borrowed from Lewontin and Kojima (1960) and Gerard (2023).

Notation and Random Mating

Let $\mathbf{g}$ be a 2×2 matrix of diploid gamete frequencies with elements $g_{ij}$ where $i$, $j\in \{0,1\}$ denotes the number of derived (‘A’) alleles in the subgenomes of each gamete.

$$\mathbf{g}=\left[\begin{array}{ll}{g}_{00}\hfill & g_{01}\hfill \\ g_{10}\hfill & g_{11}\hfill \end{array}\right]$$ (4)

The allele frequencies in each subgenome (i.e. $p_a$, $q_a$, $p_b$, $q_b$) can be determined by computing the marginal frequencies of $\mathbf{g}$. Here, $p$ denotes the frequency of the ancestral allele (‘a’) and $q$ the frequency of the derived allele (‘A’). The allele frequencies in the $a$ subgenome are the marginals over the rows of $\mathbf{g}$, and the allele frequencies in the $b$ subgenome are the marginals over the columns.

Let $\mathbf{G}$ be a 3×3 matrix of genotype frequencies. Each element, $G_{ij}$ where $i$, $j\in \{0,1,2\}$, denotes the genotype frequency with $i$ and $j$ derived alleles in subgenomes $a$ and $b$, respectively. Similarly to how the subgenome allele frequencies can be obtained from the marginal distributions of $\mathbf{g}$, the genotype frequencies within subgenomes $a$ and $b$ can be obtained by summing over the rows and columns of $\mathbf{G}$, respectively. Under random mating, $\mathbf{G}$ is defined by a two-dimensional discrete linear convolution of $\mathbf{g}$, i.e. $\mathbf{G}={\mathbf{g}}_{\ast }^2\mathbf{g}$ (Gerard, 2023).

$$\begin{gathered} \mathbf{G}=\left[\begin{array}{lll}{G}_{00}\hfill & G_{01}\hfill & G_{02}\hfill \\ G_{10}\hfill & G_{11}\hfill & G_{12}\hfill \\ G_{20}\hfill & G_{21}\hfill & G_{22}\hfill \end{array}\right] \\ =\left[\begin{array}{ccc}{g}_{00}^2& 2g_{00}{g}_{01}& g_{01}^2\\ 2g_{00}{g}_{10}& 2\left(g_{00}{g}_{11}+g_{01}{g}_{10}\right)& 2g_{01}{g}_{11}\\ g_{10}^2& 2g_{10}{g}_{11}& g_{11}^2\end{array}\right] \end{gathered}$$ (5)

This linear convolution is equivalent to calculating genotype frequencies as the expectation of two independent draws from a multinomial distribution of the gamete frequencies in $\mathbf{g}$. The genotype frequencies $G_{ij}$ are expanded in terms of the gamete frequencies $g_{ij}$ in the second matrix above and in Column 3 of Table 1.

Table 1:

Allotetraploid Model Summary for Mutation-Selection Balance

Allotetraploid Genotype	Genotype Frequency	Expressed In Gamete Frequencies As	Relative Fitness	Produces Gametes In Proportion $g_{00}:g_{10}:g_{01}:g_{11}$
$\text{AA}\Vert \text{AA}$	$G_{00}$	$g_{00}{g}_{00}$	1	$1:0:0:0$
$\text{AA}\Vert \text{Aa}$	$G_{01}$	$2g_{00}{g}_{01}$	$1-s{h}_1$	$\frac{1}{2}:0:\frac{1}{2}:0$
$\text{Aa}\Vert \text{AA}$	$G_{10}$	$2g_{00}{g}_{10}$	$1-s{h}_1$	$\frac{1}{2}:\frac{1}{2}:0:0$
$\text{AA}\Vert \text{aa}$	$G_{02}$	$g_{01}{g}_{01}$	$1-s{h}_2$	$0:0:1:0$
$\text{aa}\Vert \text{AA}$	$G_{20}$	$g_{10}{g}_{10}$	$1-s{h}_2$	$0:1:0:0$
$\text{Aa}\Vert \text{Aa}$	$G_{11}$	$2g_{01}{g}_{10}+$	$1-s{h}_2$	$\frac{1}{4}:\frac{1}{4}:\frac{1}{4}:\frac{1}{4}$
$\text{Aa}\Vert \text{aa}$	$G_{12}$	$2g_{01}{g}_{11}$	$1-s{h}_3$	$0:0:\frac{1}{2}:\frac{1}{2}$
$\text{aa}\Vert \text{Aa}$	$G_{21}$	$2g_{10}{g}_{11}$	$1-s{h}_3$	$0:\frac{1}{2}:0:\frac{1}{2}$
$\text{aa}\Vert \text{aa}$	$G_{22}$	$g_{11}{g}_{11}$	$1-s$	$0:0:0:1$

Allotetraploid Selection

We model selection acting on each tetraploid genotype as a 3×3 matrix of relative fitnesses, $\mathbf{S}$, such that each element is the relative fitness of its respective element in $\mathbf{G}$ (Column 4 of Table 1). We model the strength of selection in allotetraploids as being determined only by the dosage of the selected allele and being independent of the dosage distribution across subgenomes. Thus, we assume that there is no functional difference between alleles residing on different subgenomes and that variants in each subgenome are seen equally by selection. In this case, the matrix of relative fitnesses is of the form

$$\mathbf{S}=\left[\begin{array}{ccc}1& 1-s{h}_1& 1-s{h}_2\\ 1-s{h}_1& 1-s{h}_2& 1-s{h}_3\\ 1-s{h}_2& 1-s{h}_3& 1-s\end{array}\right],$$ (6)

where $s$ is the selection coefficient of a deleterious mutation and $h_1$, $h_2$, $h_3$ are the dominance coefficients for individuals with one, two, and three copies of the selected allele, respectively. Note that the sum of the subscripts in $\mathbf{G}$ corresponds to the subscript of the dominance coefficient in $\mathbf{S}$ for all heterozygous genotypes. For example, $G_{11}$ has two deleterious alleles, one in each subgenome, which corresponds to a dominance coefficient of $h_2$ in our model of selection.

We define $\overline{\omega }$ to be the average fitness of the population, calculated as the sum of the elements of $\mathbf{S}$ with corresponding weights $\mathbf{G}$, which is

$$\begin{gathered} \overline{\omega }={\text{Σ}}_i{\text{Σ}}_j{G}_{ij}{S}_{ij} \\ =g_{00}^2+2\left(g_{00}{g}_{10}+g_{00}{g}_{01}\right)\left(1-s{h}_1\right) \\ +\left(2g_{00}{g}_{11}+2g_{01}{g}_{10}+g_{01}^2+g_{10}^2\right)\left(1-s{h}_2\right) \\ +2\left(g_{01}{g}_{11}+g_{10}{g}_{11}\right)\left(1-s{h}_3\right)+g_{11}^2(1-s) \end{gathered}$$ (7)

and can be simplified to

$$\begin{array}{l}\overline{\omega }=1-2s\left[h_1\left(g_{00}{g}_{10}+g_{00}{g}_{01}\right)+h_2\left(g_{00}{g}_{11}+g_{01}{g}_{10}\right)\right.\\ \left.+h_3\left(g_{01}{g}_{11}+g_{10}{g}_{11}\right)\right]-s\left(h_2\left(g_{01}^2+g_{10}^2\right)+g_{11}^2\right).\end{array}$$ (8)

Then, we define $\mathbf{W}$ to be a matrix of normalized fitnesses, with $\overline{\omega }$ as the standard normalizing factor. So, $\mathbf{W}=\mathbf{S}/\overline{\omega }$ gives

$$\mathbf{W}=\left[\begin{array}{lll}{\omega }_0\hfill & {\omega }_1\hfill & {\omega }_2\hfill \\ {\omega }_1\hfill & {\omega }_2\hfill & {\omega }_3\hfill \\ {\omega }_2\hfill & {\omega }_3\hfill & {\omega }_4\hfill \end{array}\right],$$ (9)

where ${\omega }_i$ is the normalized fitness effect for a genotype with $i$ derived alleles. Finally, the normalized genotype frequencies after selection, ${\mathbf{G}}^{\prime }$, are

$${\mathbf{G}}^{\prime }=\mathbf{G}\odot \mathbf{W}=\left[\begin{array}{lll}{G}_{00}^{\prime }\hfill & G_{01}^{\prime }\hfill & G_{02}^{\prime }\hfill \\ G_{10}^{\prime }\hfill & G_{11}^{\prime }\hfill & G_{12}^{\prime }\hfill \\ G_{20}^{\prime }\hfill & G_{21}^{\prime }\hfill & G_{22}^{\prime }\hfill \end{array}\right],$$ (10)

where $\odot$ denotes the Hadamard product which is an element-wise multiplication of two matrices. For example, $G_{11}^{\prime }=G_{11}{\omega }_2$. The $i{j}^{th}$ element of ${\mathbf{G}}^{\prime }$ is simply

$$G_{ij}^{\prime }=\frac{G_{ij}{S}_{ij}}{\overline{\omega }}=G_{ij}{\omega }_{i+j}.$$

Allotetraploid Meiosis

To model gamete segregation for a disomic allotetraploid, we use the following system of equations

$$g_{00}^{\prime }=G_{00}^{\prime }+\frac{1}{2}{G}_{01}^{\prime }+\frac{1}{2}{G}_{10}^{\prime }+\frac{1}{4}{G}_{11}^{\prime }$$ (11a)

$$g_{10}^{\prime }=\frac{1}{2}{G}_{10}^{\prime }+G_{20}^{\prime }+\frac{1}{4}{G}_{11}^{\prime }+\frac{1}{2}{G}_{21}^{\prime }$$ (11b)

$$g_{01}^{\prime }=\frac{1}{2}{G}_{01}^{\prime }+G_{02}^{\prime }+\frac{1}{4}{G}_{11}^{\prime }+\frac{1}{2}{G}_{12}^{\prime }$$ (11c)

$$g_{11}^{\prime }=\frac{1}{4}{G}_{11}^{\prime }+\frac{1}{2}{G}_{12}^{\prime }+\frac{1}{2}{G}_{21}^{\prime }+G_{22}^{\prime }$$ (11d)

where the coefficients for each term are the proportions of that gamete generated from the given allotetraploid genotype. For example, the coefficient for $G_{11}^{\prime }$ is $\frac{1}{4}$ for all gametes because the $G_{11}$ genotype produces one of each gamete type, so the proportion of each gamete is one in four. These segregation frequencies are listed in Column 5 of Table 1 and are equivalent to those for a diploid two-allele, two-locus model (Li, 1972). The four simplified equations above are equivalent to equations 10.a - 10.d studied by Lewontin and Kojima (1960) when the recombination rate, $R$, is 0.5 (i.e. loci are on separate chromosomes, as is the case for homoeologous loci in an allopolyploid).

Allotetraploid Mutation

We define the rate of forward mutation from the ancestral to derived state as $\mu$ and the rate of back mutation as $v$. We assume that the mutation rates are equal between subgenomes and that mutation acts only on the gametes after meiotic segregation. The following set of equations model the transition probabilities between types of gametes in terms of $\mu$ and $v$.

$$\begin{gathered} g_{00}^{t+1}={(1-\mu )}^2{g}_{00}^{\prime }+(1-\mu )\nu g_{01}^{\prime } \\ +\nu (1-\mu )g_{10}^{\prime }+{\nu }^2{g}_{11}^{\prime } \end{gathered}$$ (12a)

$$\begin{gathered} g_{01}^{t+1}=(1-\mu )\mu g_{00}^{\prime }+(1-\mu )(1-\nu )g_{01}^{\prime } \\ +\nu \mu g_{10}^{\prime }+(1-\nu )\nu g_{11}^{\prime } \end{gathered}$$ (12b)

$$\begin{gathered} g_{10}^{t+1}=\mu (1-\mu )g_{00}^{\prime }+\mu \nu g_{01}^{\prime } \\ +(1-\nu )(1-\mu )g_{10}^{\prime }+(1-\nu )\nu g_{11}^{\prime } \end{gathered}$$ (12c)

$$\begin{gathered} g_{11}^{t+1}={\mu }^2{g}_{00}^{\prime }+\mu (1-\nu )g_{01}^{\prime } \\ +(1-\nu )\mu g_{10}^{\prime }+{(1-\nu )}^2{g}_{11}^{\prime } \end{gathered}$$ (12d)

For example, in (12a), $g_{01}^{\prime }$ will result in a $g_{00}^{t+1}$ gamete if two specific events occur i) no forward mutation occurs in the $a$ subgenome (hence the $1-\mu$ factor) and ii) back mutation does occur in the $b$ subgenome (hence the $v$ factor).

Allotetraploid Changes in Gamete Frequencies

Combining the steps of random mating, selection, meiosis, and mutation described above, we model the change in gamete frequencies between subsequent generations by subtracting a $g_{ij}^t$ term from (12) to obtain the following system of ODEs for allotetraploids:

$${\dot{g}}_{00}=g_{00}^{t+1}-g_{00}^t$$ (13a)

$${\dot{g}}_{01}=g_{01}^{t+1}-g_{01}^t$$ (13b)

$${\dot{g}}_{10}=g_{10}^{t+1}-g_{10}^t$$ (13c)

$${\dot{g}}_{11}=g_{11}^{t+1}-g_{11}^t.$$ (13d)

This system of equations can then be reduced to a system of three equations in three variables by progressive substitution of equations (5), (10), (11), and (12) into equation (13) and finally substituting $g_{10}=1-g_{00}-g_{01}-g_{11}$ into the resulting system and removing the ${\dot{g}}_{10}$ equation. Note that this accounts for the linear constraint that the gamete frequencies must sum to one. The choice of which gamete frequency and corresponding equation to remove is arbitrary, but the removal of $g_{10}$ and ${\dot{g}}_{10}$ is reflected in our corresponding numerical analysis.

Comparison to Diploid and Autotetraploid Models

The notation and model structure for both diploids and autotetraploids is conceptually similar to that presented above for allotetraploids. The primary difference is that diploids and autotetraploids lack any subgenomic structure and, thus, gamete and genotype frequencies (i.e. $\mathbf{g}$ and $\mathbf{G}$ ) are modeled as vectors instead of matrices. By extension, selection is also modeled as a vector of fitnesses instead of as a matrix. Importantly, the diploid and autotetraploid models use slightly different notation for gametes and genotypes; instead of an $ij$ subscript which delineates between subgenomes, both models use a single $i$ subscript for the total number of derived alleles in a gamete or genotype. Additionally, while traditional diploid models focus on changes in allele frequencies, we instead refer to gamete frequencies for semantic and notational consistency across all three models. Gamete and allele frequencies are equivalent for the diploid model, and, when allele frequencies are of interest, we use the gamete frequencies in the tetraploid models to calculate allele frequency as a summary statistic. To make the distinction between alleles and gametes clear, we use the notation from Lewontin and Kojima (1960) and other two-locus, two-allele diploid models. For comparison with Table 1, general summaries of the diploid and autotetraploid models are in Tables S1 and S2.

Numerical Model Analysis

Our models capture the dynamical interplay between mutation and selection at a single locus in populations of diploids, autotetraploids, and allotetraploids. One of the primary objectives in this setting is the identification of points of MSB. Mathematically, these correspond to points where the change in gamete frequencies is simultaneously zero for all gamete types in the model. So, points of MSB are fixed points or equilibria solutions of the systems of ODEs.

Our model is nonlinear in both the gamete frequencies and parameters, which makes analyzing the systems of ODEs difficult; in particular, no analytical solutions exist. However, progress toward locating fixed points and evolutionary trajectories of populations can still be made. We adopt a nonlinear dynamics framework and apply classic techniques from this field to analyze our model with a focus on qualitative features such as fixed points, their stability, and bifurcations (Strogatz, 2018).

We first introduce some additional notation before sketching the numerical approaches used in analyzing the models. Let the vectorized system of ODEs be denoted as $\dot{\mathbf{g}}$. This vectorized function, $\dot{\mathbf{g}}$, will take two vector inputs, $\mathbf{g}$ and $\mathbf{p}$, where $\mathbf{g}\in {ℝ}^{n-1}$ is a vector of $n-1$ gamete frequencies and $\mathbf{p}\in {ℝ}^m$ is a vector of $m$ parameters. Note that both $\dot{\mathbf{g}}$ and $\mathbf{g}$ have $n-1$ elements for a model with $n$ gametic types because the linear constraint that the gamete frequencies must sum to 1 is first applied to the system of ODEs. For example, in the allotetraploid model, $\dot{\mathbf{g}}=\left[{\dot{g}}_{00},{\dot{g}}_{01},{\dot{g}}_{11}\right]$, $\mathbf{g}=\left[g_{00},g_{01},g_{11}\right]$, and $\mathbf{p}=\left[s,h_1,h_2,h_3,\mu ,\nu \right]$.

A fixed point, ${\mathbf{g}}^{\ast }\in {ℝ}^{n-1}$, occurs where $\dot{\mathbf{g}}\left({\mathbf{g}}^{\ast },{\mathbf{p}}^{\ast }\right)=\mathbf{0}$ for some specific ${\mathbf{p}}^{\ast }\in {ℝ}^m$. Importantly, the fixed points and their stability are equivalent for the iterative map/difference equation (1) and differential equations (3). As applied to our analysis, the approximation of iterative maps by differential equations holds for weak evolutionary forces (i.e. $s$, $\mu$, $\nu \ll 1$). More formal analysis of this weak evolution limit has been performed in the differential equations framework (Hadeler, 1981; Hofbauer, 1985; Bürger, 1998) and also in the context of replicator dynamics from evolutionary game theory (Losert and Akin, 1983).

Because no analytical solution exists for the fixed points of $\dot{\mathbf{g}}$, we estimate the fixed points numerically using vpasolve in Matlab2023b. We first evaluate the system of ODEs for some set of parameters by calculating $\dot{\mathbf{g}}\left(\cdot ,{\mathbf{p}}^{\ast }\right)$. This gives a system of nonlinear, algebraic equations in the elements of $\mathbf{g}$ which is then passed to vpasolve without an initial guess. Because the systems are relatively small with three or fewer equations and, when solving $\dot{\mathbf{g}}\left({\mathbf{g}}^{\ast },{\mathbf{p}}^{\ast }\right)=\mathbf{0}$, each ${\dot{g}}_i$ can be written as a polynomial expression in terms of the elements of $\mathbf{g}$, vpasolve returns all (real) fixed points ${\mathbf{g}}^{\ast }$ for a given ${\mathbf{p}}^{\ast }$.

In order to classify the linear stability of each fixed point ${\mathbf{g}}^{\ast }$, we linearize the system about each ${\mathbf{g}}^{\ast }$ by keeping only $\mathcal{O}(\epsilon )$ terms in the Taylor expansion. This is equivalent to computing the Jacobian matrix, $\mathbf{J}$ at $\left({\mathbf{g}}^{\ast },{\mathbf{p}}^{\ast }\right)$. The $i{j}^{th}$ element of $\mathbf{J}$ is $\frac{\partial {\dot{g}}_i}{\partial g_j}$ where ${\dot{g}}_i$ is the $i^{th}$ differential equation of $\dot{\mathbf{g}}$ and $g_j$ is the $j^{th}$ gamete type in $\mathbf{g}$. The solutions of the resulting linear approximation characterized by $\mathbf{J}$ are fully specified by the eigenvalues ${\lambda }_i$ and eigenvectors ${\mathbf{v}}_i$ of $\mathbf{J}$. We use the Matlab function eig to numerically find the eigenvalues and eigenvectors of $\mathbf{J}$ evaluated at a fixed point.

Then, we classify the linear stability of each fixed point. In the biological context, a population near a stable fixed point will move toward it and a population near an unstable fixed point will move away from it. Mathematically, a fixed point is (asymptotically) stable if the real part of all eigenvalues is negative (i.e. $\mathrm{Re}\left({\lambda }_i\right)<0$, $\forall i\in \{1,\dots ,n-1\}$). Alternatively, a fixed point is unstable if at least one eigenvalue has positive real part (i.e. $\exists {\lambda }_i$ s.t. $\mathrm{Re}\left({\lambda }_i\right)>0$). A third, intermediate case called marginal stability can also occur when the eigenvalues all have a non-positive real part and at least one eigenvalue has an identically zero real part (i.e. $\mathrm{Re}\left({\lambda }_i\right)\le 0$, $\forall i\in \{1,\dots ,n-1\}$ and $\exists {\lambda }_i$ s.t. $\mathrm{Re}\left({\lambda }_i\right)=0$). This results in a (zero-eigenvalue) bifurcation at the fixed point.

Mutation Load

In our model, we use relative fitness to parameterize selection. While this is useful within each individual model, it makes explicit comparisons between populations difficult because the fittest genotype in each model need not have the same absolute fitness. Instead, we calculate mutation load, $L$, which is appropriately normalized to facilitate comparisons across models and populations. Mutation load is considered the expense, measured in terms of decreased fitness due to deleterious mutations, for the opportunity to evolve via beneficial mutations (Muller, 1950; Crow, 1970). We use the most common definition of mutation load which is the fraction by which fitness is reduced in a population of interest relative to a reference population. That is

$$L=\frac{{\omega }_1-{\omega }_2}{{\omega }_1}=\frac{{\omega }_{max}-\overline{\omega }}{{\omega }_{max}}$$ (14)

where ${\omega }_1={\omega }_{max}$ is the fitness of the reference population and ${\omega }_2=\overline{\omega }$ is the fitness of the population of interest. ${\omega }_1={\omega }_{max}$ is most often taken to be one which represents the fitness of a population absent any mutations.

Results

Neutral Dynamics

We begin by exploring the simplest case without selection or mutation $(s=\mu =\nu =0)$ and build further complexity in sections below. In this case, the system only captures the effects of random mating and meiosis, so the population will evolve toward Hardy-Weinberg Equilibrium (HWE).

Panmictic Disequilibrium

While a single generation of random mating in diploids produces genotype frequencies in HWE, polyploid populations approach HWE asymptotically. The rate of approach can be described using panmictic disequilibrium, a set of statistics denoted by ${\text{Δ}}_{ij,t}$ which are calculated as the difference between the observed gamete frequency at generation $t$ minus the HWE gamete frequency (given the observed allele frequency). In general, panmictic disequilibrium is

$${\text{Δ}}_{ij,t}=g_{ij,t}-{\widehat{g}}_{ij},$$ (15)

where $g_{ij,t}$ denotes the ordered gamete frequency of type $ij$ at generation $t$ and ${\widehat{g}}_{ij}$ denotes the HWE frequency of the gamete of type $ij$ (Gallais, 2003). At HWE, ${\widehat{g}}_{ij}=p_i{p}_j$ where $p_i$, $p_j$ are the allele frequencies of types $i$, $j$, respectively. ${\text{Δ}}_{ij,t}$ is closely related to linkage disequilibrium between two loci in diploids. The primary distinction is that panmictic disequilibrium is measured for multiple chromosomes at a single locus instead of multiple loci on a single chromosome (as in linkage disequilibrium). Similar to how linkage disequilibrium decays over subsequent generations and recombination events, so too does ${\text{Δ}}_{ij,t}$ decay over time. However, this decay is dependent on the proportion of gametes which are identical by descent (IBD) from the previous generation instead of the recombination rate between loci (Slatkin, 2008).

Let $c$ denote the proportion of gametes which are IBD from the gametes in the previous generation. Then, $1-c$ of the gametes are not IBD (i.e. are produced from a random sampling of alleles). So,

$$g_{ij,t+1}=c{g}_{ij,t}+(1-c){\widehat{g}}_{ij}.$$

Subtracting a ${\widehat{g}}_{ij}$ term from both sides gives

$$g_{ij,t+1}-{\widehat{g}}_{ij}=c\left(g_{ij,t}-{\widehat{g}}_{ij}\right)$$

which is equivalent to

$${\text{Δ}}_{ij,t+1}=c{\text{Δ}}_{ij,t}$$

(see (15)). Thus, it is clear that ${\text{Δ}}_{ij}$ decays geometrically at rate $c$ (Gallais, 2003). For a panmictic tetraploid population with polysomic inheritance, this set of statistics has been shown to decay geometrically with a rate of $c=1/3$ (Haldane, 1930) because two in six (or one in three) gametes are IBD in an autotetraploid individual.

For an allotetraploid individual, there are four possible gametes because chromosomes from the same subgenome (i.e. with the same immutable centromere) cannot segregate together during meiosis I. Two of these four gametes are IBD to the gametes which formed the allotetraploid individual. Thus, the proportion of gametes which are IBD is one in two and ${\text{Δ}}_{ij,t}$ decays geometrically at a rate of $c=1/2$. This geometric decay applies only to the overall genotypes of the allotetraploid; the genotypes of the (diploid) subgenomes each reach HWE in a single generation. So, the decay of ${\text{Δ}}_{ij,t}$ is associated with the asymptotic breakdown of correlation between subgenomes (Gerard, 2023). Overall, an autotetraploid population will approach HWE 33% faster than an allotetraploid population, but any substantive amount of panmictic disequilibrium is removed by random mating in a few generations (Fig. 1).

Figure 1: Decay of panmictic disequilibrium over time.

Populations have no selection nor mutation and start with $p=q=0.5$ and only homozygotes present in the first generation.

Efficacy of Selection and Bistable Equilibria

Starting in this section, we consider non-neutral cases with mutation, where $\mu$, $\nu$, $s\ne 0$. With both mutation and selection present, we are interested in identifying points of mutation-selection balance (MSB), comparing these points between the diploid and tetraploid models, and investigating how the location of these points shift as parameters vary.

We begin by considering how points of MSB shift as the strength of selection $s$ varies when forward and backward mutation rates are equal $(\mu =\nu )$ (Fig. 2). In this case, the auto- and allotetraploids have nearly identical equilibrium allele frequencies and mutation loads (Fig. 2). Notably, the two tetraploid models show very slight differences at MSB across dominance classes with the differences being $\mathcal{O}(\mu )$ and $\mathcal{O}\left({\mu }^2\right)$ for allele frequency and mutation load, respectively (Figs. S1, S2). As expected, the frequency of the deleterious allele decreases as $s$ increases, but the response to selection is not equal between the tetraploid and diploid models (Fig. 2A-E). The tetraploid populations generally experience a smaller change in allele frequency for the same change in the strength of selection. Additionally, for weak selection (approximately $s<\mu$), the equilibria arise from a balance between mutational pressures with the approximate equilibrium given by $q\approx \mu /(\mu +\nu )$ (Crow and Kimura, 1970). In this case where $\mu =\nu$, it is straightforward to see that the expected allele frequency for a neutral or nearly neutral allele (i.e. $s\ll 1$) is $q=0.5$ (Fig. 2A-E).

Figure 2: Mutation-selection equilibria for equal mutation rates.

Both allele frequency and mutation load are shown. For all panels, $\mu =\nu ={10}^{-8}$. Dominance coefficients for the partially recessive column are $h_1=0.02$, $h=h_2=0.1$, and and $h_3=0.4$ for the partially dominant column are $h_1=0.6$, $h=h_2=0.9$, and $h_3=0.98$.

Furthermore, both the auto- and allotetraploids have a higher proportion of deleterious alleles segregating at MSB compared to diploids facing the same mutation and selection pressures, regardless of dominance; selection is less efficacious at purging deleterious alleles at higher ploidies. However, these higher allele frequencies do not necessarily imply a lower fitness. This is because tetraploids can mask the effects of recessive and partially recessive deleterious mutations; specifically, in line with theoretical predictions from Otto and Whitton (2000), we find that tetraploids have a mutation load lower than the expected $L=4\mu$ for partially recessive mutations provided that $h_1<h/2$ (Fig. 2G). In the fully recessive case, we find that the tetraploid populations are more fit than the diploid populations for all $s$ when measured by mutation load $L$ (Fig. 2F). Remarkably, under the special case of equal mutation rates, $L$ is up to twice as large for diploids as for tetraploids at intermediate strengths of selection (Fig. 2F). Haldane (1937) first analytically calculated $L=\mu$ in the recessive case for diploids, and Ronfort (1999) extended this result to any ploidy level, which agrees with our numerical estimates of $L$ for large $s$. Note that because our model includes back mutation, our numerical estimates of $L$ disagree with this analytical approximation for small $s$. For the cases with any dominance, load is approximately twice as large for the tetraploid versus diploid populations (Fig. 2G-J). This is because the effective mutation rate in the tetraploid models is twice that in the diploid models, because there are twice as many loci/chromosomes to mutate. For non-recessive dominance, Otto and Whitton (2000) calculated $L\approx k\mu$ for a population of $k$-ploid individuals, which generally aligns with our results (but note the deviation for partially recessive mutants). Surprisingly, we find a non-monotonic relationship between $s$ and $L$ for dominant mutations in the tetraploids near $s={10}^{-7}$ (Figs. 2J, S3).

We now turn our attention to a case of biased mutation rates such that $\mu >\nu$ (Fig. 3). In this region of parameter space, most of the same patterns and results occur as under the equal mutation rate regime. For example, the deleterious allele has a higher segregating frequency at MSB in tetraploids, and this frequency decreases as the strength of selection increases. Additionally, the same approximations for mutation load hold for stronger strengths of selection, and for $s<\mu$ the populations exist in a mutational equilibrium at $q=\mu /(\mu +\nu )$, which for $\mu =2\times {10}^{-8}$ and $\nu ={10}^{-9}$ gives $q\approx 0.95$. For the recessive and additive mutations, there is still a single, globally stable equilibrium which all populations will evolve toward. However, for dominant mutations, we find a more interesting result; for some values of $s$, there are two stable fixed points (Fig. 3C). This is termed bistability and was originally reported in diploids by Haldane (1927) and can be understood by decomposing the differential equation into components due to mutation and selection (Fig. S4). For these dominant mutations, the presence of bistability indicates that populations facing the same selective and mutation pressures, but with different initial allele frequencies, may evolve toward different equilibria (Fig. S5). So, in some cases, a population can evolve toward a less fit state with a higher deleterious allele frequency, despite facing the pressure of purifying selection. For example, a population just above the unstable equilibrium (dashed line) will evolve up toward a stable fixed point near $q\approx 0.95$ (Fig. 3C).

Figure 3: Mutation-selection equilibria for biased mutation rates.

Both allele frequency and mutation load are shown. For all panels, $\mu =2\times {10}^{-8}$ and $\nu ={10}^{-9}$. The solid lines represent stable equilibria. The dashed lines in (C) represent unstable equilibria. In (F), load is only shown for the stable equilibria.

Efficiency of Selection and Fisher’s Theorem

To investigate the temporal dynamics as populations approach MSB equilibrium, we developed a discrete time simulation that tracks the evolution of gamete frequencies (and, by extension, allele frequencies) for a specified number of generations and arbitrary initial conditions. We compare the results of these simulations of diploid and tetraploid populations against the predictions of Fisher’s Fundamental Theorem of Natural Selection (FTNS), with a focus on the impact that bifurcations and bistability have on the applicability of FTNS. We first consider a case with balanced mutation rates that are weak relative to the strength of selection (Fig. 4). Although mutation is present in this case, Fisher’s FTNS still accurately predicts the (partial) increase in mean fitness due to additive genetic variance primarily because $\mu$, $\nu \ll s$. This implies that natural selection is the predominant determinant of the long-term temporal dynamics and any initial standing variation outweighs the relatively small influx of new mutations.

Figure 4: Fitness trajectories across models and dominance for weak, unbiased mutation.

Trajectories of population fitness over time for $s=.01$, $\mu ={10}^{-8}$, and $\nu ={10}^{-8}$. All populations begin in Hardy-Weinberg Equilibrium with an initial allele frequency between 0 to 1. The coloration denotes the initial allele frequency when the simulation started (i.e. at $t=0$).

Both diploids and autotetraploids show the same general pattern: recessive mutations are more efficiently selected out of the population when considering population fitness (Fig. 4). That is, fitness more quickly reaches the optimal value for more recessive mutations. However, this does not directly correspond to a more efficient purging of the deleterious allele from the population (Fig. S6). Instead, deleterious mutations with additive dominance are most effectively removed by natural selection, but each remaining copy of the mutation still has an impact on the fitness of the population (whereas a recessive mutation only affects the population fitness if it is present in a homozygous state). Similar to how overall mutation load is masked by tetraploids for recessive mutations, the rate of fitness increase is, in effect, “masked” so that it is identical between the diploid and autotetraploid populations (Fig. 4). This equal rate of evolution as measured by fitness is accompanied by equivalent fitness variances across the three models, in line with the predictions of Fisher’s FTNS (Fig. S7). So, for recessive mutations, purifying selection is equally effective and efficient in the diploid and tetraploid models. For additive and dominant mutations, the tetraploids do evolve more slowly as measured by both fitness and allele frequency, which reflects the decreased efficiency of selection at higher ploidy levels predicted by evolutionary theory (Ronfort, 1999; Otto and Whitton, 2000). These differences in how quickly fitness changes are mediated by different characteristics of the fitness variance trajectories for additive and dominant mutations. Although the tetraploid models reach a maximum fitness variance at a later generation for both additive and dominant mutations, the magnitude of the fitness variance is also lower for additive mutations in the tetraploid models compared to the diploid model (Fig. S7).

Next, we consider a case with biased mutation rates that are comparable to the strength of selection (Fig. 5). Under these parameters, we would expect some deviations from the predictions of Fisher’s FTNS due to the influx of deleterious mutations and a more even balance between mutation and selection in driving the temporal dynamics. For recessive mutations, we see a similar “masking” effect in which the tetraploids evolve at an equivalent rate when measured by population fitness (Fig. 5). However, unlike the above, allele frequency also decreases more quickly for recessive mutations than additive or dominant mutations (Fig. S8). In addition to this difference, the tetraploid and diploid populations seem to evolve on similar time scales for dominant mutations, as opposed to the diploids evolving more quickly than the tetraploids under an equal mutation rate regime. Furthermore, the rate of fitness change no longer directly corresponds to the magnitude of fitness variance in the population (Fig. S9). Thus, in the presence of stronger, biased mutation rates and bistability, both Fisher’s FTNS and traditional predictions about the efficiency of natural selection in tetraploids compared to diploids break down.

Figure 5: Fitness trajectories across models and dominance for strong, biased mutation.

Trajectories of population fitness over time for $s=.05$, $\mu =5\times {10}^{-3}$, and $\nu ={10}^{-4}$. All populations begin in Hardy-Weinberg Equilibrium with an initial allele frequency between 0 to 1. The coloration denotes the initial allele frequency when the simulation started (i.e. at $t=0$).

Discussion

We have developed the first comprehensive mathematical framework for comparing mutation-selection dynamics across diploid, autotetraploid, and allotetraploid systems, addressing a critical theoretical gap. Extensions of classical one and two-locus biallelic models for diploids, our deterministic models provide quantitative foundations for understanding how ploidy level fundamentally reshapes evolutionary equilibria. Although autotetraploids approach HWE faster than allotetraploids (Fig. 1), our unifying framework reveals that auto- and allotetraploids exhibit remarkably similar evolutionary trajectories despite their distinct meiotic behaviors, with both types showing higher segregating allele frequencies compared to diploids at mutation-selection balance (Fig. 2). Additionally, for completely recessive mutations, both tetraploid models are at least as fit as and evolve at equivalent rates as diploid populations (Figs. 2, 4). This suggests that, at this fundamental level, ploidy itself — and the associated masking of deleterious alleles — is more important than the specific mode of inheritance. This result provides a crucial theoretical baseline, isolating the effect of ploidy from other confounding factors. Perhaps most interestingly, we found bistability across all three models for dominant deleterious mutations with biased mutation rates (Fig. 3). Beyond altering the equilibrium structure, this bistability also has substantial impacts on the temporal dynamics of evolution under biased mutation rates with recessive mutations evolving faster than additive or dominant ones (Fig. 5).

However, this near equivalence is predicated on our idealized model which considers a single locus in a population with an infinite size and disregards differences between subgenomes in allotetraploids. In nature, established allopolyploids often exhibit subgenome dominance, where homoeologous genes from one subgenome are expressed at universally higher (or lower) levels on average, and biased fractionation, leading to differential gene loss between the two subgenomes (Edger et al., 2017). These processes could result from unequal selective pressures on alleles or unequal mutation rates depending on their subgenomic location, scenarios that our flexible framework could be extended to explore in future work.

Mutation-Selection Balance and Bistable Equilibria

To better understand how bistability may impact a population’s trajectory, we consider a scenario in which the strength of selection, $s$, changes over time; such changes in selective pressures have been shown in recent work on the joint distribution of fitness effects (Huang et al., 2021) and can be mediated by epistatic interactions with a new variant (Johnson et al., 2023), changes in DNA methylation and gene expression levels (Flores et al., 2013), or, as we outline below, environmental change (Wade and Kalisz, 1990). Consider a changing environment which makes a trait less adaptive for an individual’s survival and induces a corresponding increase in $s$. So, if a population was in equilibrium at approximately $s={10}^{-5}$, $q=0.95$ and the environment changes such that the strength of selection is now $s=5\times {10}^{-5}$, the population would experience a substantial reduction in both allele frequency and mutation load at equilibrium from $q\approx 0.95$ to $q\approx 0$ and $L\approx 2\times {10}^{-5}$ to $L\approx 8\times {10}^{-8}$ (Fig. 3C,F). These large changes due to a small change in a parameter are characteristic of bifurcations and could have substantial impacts on fitness, especially when considered at multiple sites across the genome.

The band of $s$ values for which bistability exists is notably larger for tetraploids than for diploids (Fig. 3). Thus, we expect the initial allele frequency to play a more substantial role in the long-term evolutionary trajectory of tetraploids. This is especially important to consider given the extreme bottlenecks neopolyploids undergo shortly after formation which further amplifies the impact of the founding individuals (Ramsey and Schemske, 2002). Additionally, neopolyploids rely almost exclusively on the genetic variation carried over from founding individuals. With this in mind, we expect that bistability will amplify the influence of diploid progenitors and their genetic variation on the evolutionary trajectories of neopolyploid populations. This may result in a higher mutation load following the bottleneck, even with the increased masking of recessive or partially recessive mutants and reduced inbreeding depression during a bottleneck (Layman and Busch, 2018). In addition to the interactions of bistability with small population sizes, recent work has focused on the interactions of the increased masking of recessive alleles with low recombination rates in small population sizes, revealing that pseudo-overdominance is much more likely in autotetraploid populations than diploid ones (Booker and Schrider, 2025b). In addition to its implications for neopolyploids, bistability is also relevant to other contexts such as linked selection. Bistability may impact the ability of purifying selection to effectively purge deleterious alleles linked to beneficial mutations because these deleterious alleles will reach a high segregating frequency due to linked selection. Once the linked (dominant) deleterious allele crosses the threshold of the unstable equilibrium, it will remain at a high frequency near the mutational equilibrium and be unable to be removed by purifying selection. Thus, bistability, in conjunction with other evolutionary processes such as bottlenecks and linked selection, has the potential to alter a population’s long-term evolution. We emphasize that these dynamics are observed in our infinite population size model and, thus, only account for the average change in allele frequency in each generation. Future work should incorporate stochastic genetic drift, which is especially important in small populations to better understand how finite population sizes may affect or interact with these dynamics. In particular, in a finite population size model, the unstable manifold which separates the two stable equilibria will become increasingly blurred as the population size decreases.

Even though dominant deleterious mutations are thought to be rare (Eyre-Walker and Keightley, 2007), the bistable dynamics described above for dominant deleterious mutations create important alternative evolutionary trajectories where populations with identical selective pressures but different initial allele frequencies evolve toward dramatically different equilibria — a phenomenon particularly pronounced in polyploids due to their wider parameter ranges for maintaining bistability compared to diploids. This bistability may contribute to increased genetic variation in polyploids (Soltis and Soltis, 2000) which has been hypothesized to play a substantial role in the success and prevalence of polyploids among domesticated crops (Salman-Minkov et al., 2016; Qi et al., 2021). Furthermore, because of its potential impact on genetic variation, bistability may impact range expansion which is thought to be facilitated by increased plasticity and genetic variance at higher ploidy levels (Ramsey, 2011), although empirical evidence for this hypothesis is mixed (Martin and Husband, 2009) and simulations suggest that inheritance patterns (i.e. disomic vs. tetrasomic) may play a crucial role in range expansion (Booker and Schrider, 2025a). Overall, coupled with complex connections between dominance and selection and difficulties in estimating genome-wide dominance (Huber et al., 2018; Kyriazis and Lohmueller, 2024) (see Di and Lohmueller (2024) for a recent review), the presence of bistability in our model makes understanding the distribution of dominance effects even more important for predicting the evolution of polyploid populations over time, as slight differences in initial conditions between populations can lead to profoundly different fitness outcomes.

Bistability and Temporal Dynamics

Fisher first stated the Fundamental Theorem of Natural Selection (FTNS) in 1930 as “the rate of increase of fitness of any species is equal to the genetic variance in fitness” (p. 46) (Fisher, 1930). Fisher’s work and, in particular, his FTNS subsequently sparked substantial research and development in population genetics. However, what Fisher precisely meant by his theorem has long been debated among experts (Edwards, 1994). The ‘modern’ interpretation of the theorem (Okasha, 2008) focuses on the partial change in mean fitness due to additive genetic variance and was originally introduced by Price (1972) and later clarified by Ewens (1989). This interpretation can also be framed in the context of adaptation from standing variation (Queller, 2017). Although the debate over the theorem’s proper interpretation has continued (Ewens and Lessard, 2015; Edwards, 2016; Ewens, 2024), extensions of the theorem have recently extended beyond the component of additive genetic variance to explicitly account for the influx of deleterious de novo mutations (Basener and Sanford, 2018) and balancing selection (Grafen, 2021). Here, we will adopt the ‘modern’ interpretation which argues that Fisher only accounted for the (partial) increase in mean fitness due to additive (genetic) variance and does not consider changes attributable to the environment or other stochasticities (but see Gaynor et al. (2025) for the importance of environmental stochasticty in polyploid establishment).

For small, equal mutation rates, the diploid and tetraploid populations evolve as expected; “masking” effects account for equal rates of evolution for recessive mutations and the diploids evolve more quickly for additive and dominant mutations (Fig. 4). Alternatively, for larger, biased mutation rates which admit bistability, the dynamics are notably different. In particular, for both additive and dominant mutations, the diploids and tetraploids evolve on similar time scales (Fig. 5). This is likely due to the presence of bistability and associated bifurcations that distort the temporal dynamics over the transient period before the population reaches an equilibrium. In particular, because the rates of change are all simultaneously zero at an equilibrium, the change in allele frequency for a population near an equilibrium is very small. So, the populations which start near the unstable equilibrium initially evolve very slowly. In fact, the bistability and bifurcations present in the dominant case are also likely responsible for the relatively slower dynamics for additive mutations (compared to recessive ones). For example, consider a population with partial dominance (not shown here) right where a bifurcation emerges so that for dominance coefficients closer to additivity there is a single stable equilibrium and for dominance coefficients closer to full dominance there are two stable equilibria. In this case, the rates of change are very nearly zero at the emerging bifurcation (because the bifurcation point must be an equilibrium itself, the rates of change are near zero at this point). So, for additive mutations, a population starting at a high allele frequency must pass through the ‘ghost’ left behind by the bifurcation resulting in a long transient period before reaching the MSB equilibrium. This is one example of critical slow down near a bifurcation which is a well-documented phenomenon in biological models (Van Nes and Scheffer, 2007; Chisholm and Filotas, 2009; Dakos and Bascompte, 2014) and in dynamical systems more broadly (Mandel and Erneux, 1987; Tredicce et al., 2004; Strogatz, 2018). Thus, Fisher’s FTNS does not apply in this case as the existence of bistable mutation-selection equilibria obscures the relationship between additive genetic variance and the efficiency of natural selection.

Conclusion

Our unified mathematical framework provides the first comprehensive comparison of mutation-selection dynamics across diploid, autotetraploid, and allotetraploid systems under idealized conditions. Within our deterministic, single-locus model, auto- and allotetraploid systems exhibit remarkably similar evolutionary trajectories, suggesting that ploidy level itself — rather than specific inheritance mechanisms — may be the primary determinant of fundamental mutation-selection dynamics. Most notably, our analysis reveals widespread bistability in polyploid systems and associated breakdowns of Fisher’s Fundamental Theorem. While these findings emerge from simplified theoretical conditions, they provide important foundations for understanding when and how genome duplication alters evolutionary processes.

Our flexible framework can be extended to more complex models, including unequal mutation or selection across subgenomes of allotetraploids; unequal selection may arise as a result of homoeolog expression biases, neofunctionalization, or subfunctionalization, among other mechanisms (Adams et al., 2003; Des Marais and Rausher, 2008). Future extensions of our framework could incorporate linkage effects, environmental variation, under/overdominance, or homoeologous exchange (Mason and Wendel, 2020) and may reveal additional complex dynamics including limit cycles, Hopf bifurcations, or additional fixed points not seen under our model of selection but that have previously been documented in two-locus, biallelic diploid models similar to our model for allotetraploids (Akin, 1982; Pontz et al., 2018).

The substantial theoretical gap in polyploid population genetics relative to diploids (Meirmans et al., 2018) has limited our ability to predict and interpret evolutionary patterns in polyploid species. This theory is critical for identifying the action of mutation, selection, and drift in evolution, especially among many complicating processes in polyploids such as double reduction, dosage effects, and biased fractionation. Our nonlinear dynamics framework begins to address this need and provides a foundation for analyzing deterministic mutation-selection models for diploids, autotetraploids, and allotetraploids. To complement our deterministic analysis, future work should incorporate population genomic inference methods and stochastic models to explore the subtleties of polyploid evolution, particularly as high-quality polyploid genomic data become increasingly available. The insights from our model and similar work could have relevant applications for economically important crops, especially in light of how dominance substantially alters allele frequency dynamics and, thus, may influence selective breeding in crops with recent histories of whole genome duplication (Renny-Byfield and Wendel, 2014). As theoretical understanding advances through incorporation of additional biological complexity, these foundational insights may prove essential for predicting polyploid evolutionary trajectories in natural populations and agricultural systems.

Data and Code Availability

Code implementing all analyses in this manuscript is available at https://github.com/conJUSTover/Mutation_Selection_Balance_Autos_Allos.