Elevated mutation near crossovers inhibits the evolution of recombination

Abstract

Recombination diversifies offspring genomes and helps ensure chromosome segregation during meiosis. Mutation rates are elevated near crossovers due to the induction of double-strand breaks and their imperfect repair, a byproduct of recombination typically ignored by theory designed to explain its evolution. To examine the evolutionary role of mutagenic recombination, we analyze a population genetic model in which a modifier locus controls both the rate of recombination between two loci experiencing viability selection and the rate of mutation at those loci. Analytical and numerical results demonstrate that the advantage of recombination conferred by its capacity to remove epistatic, deleterious variants is overcome by the selective cost of even small increases in the mutation rate. By incorporating the mutagenic effects of recombination, our analysis extends a rich body of theory to acknowledge a molecular feature inherent in the formation of crossovers. Our findings suggest that higher recombination rate evolves by altering steps in the crossover pathway that are less likely to inflict mutational damage.

Introduction

During meiosis, homologous chromosomes exchange DNA via crossovers, creating new combinations of genetic variants. Recombination improves the efficiency of natural selection, shapes genomic patterns of variation, and (in many species) helps to ensure that chromosomes segregate properly (Fisher 1930; Muller 1932; Hill and Robertson 1966; Felsenstein 1974; Feldman et al. 1980; Maynard Smith and Haigh 1974; Begun and Aquadro 1992; Charlesworth et al 1993; Barton 1995; Feldman et al. 1996; Otto and Barton 1997; Otto and Feldman 1997; Hassold and Hunt 2001; Page and Hawley 2003; Cutter and Payseur 2013; Charlesworth and Jensen 2021). Therefore, explaining heterogeneity in recombination rate observed along the genome, among individuals, and between species (McVean et al. 2004; Coop and Przeworski 2007; Smukowski and Noor 2011; Stapley et al. 2017; Haenel et al. 2018; Henderson and Bomblies 2021; Johnston 2024) is a goal with broad evolutionary significance.

A rich body of theory delineates conditions under which the recombination rate is expected to evolve (reviewed by Dapper and Payseur 2017). Population genetic models focus on the ability of recombination to alter genotype frequencies in offspring, which generates indirect selection on the recombination rate (Barton 1995; Feldman et al. 1996). Modeling the fate of alleles at modifier loci that change the recombination rate indicates that recombination’s propensity to break down linkage disequilibrium (LD) generated by natural selection and genetic drift confers advantages in a variety of scenarios. When inter-locus combinations of deleterious alleles decrease fitness more than predicted by their individual effects (synergistic or negative epistasis), recombination expands genetic variance by reducing LD, increasing the capacity of populations to purge deleterious variants (Feldman et al. 1980; Otto and Feldman 1997; Barton and Charlesworth 1998). When selection fluctuates temporally or spatially, recombination separates sets of maladaptive alleles, thereby raising the mean fitness among offspring (Barton 1995; Otto and Michalakis 1998). Recombination also frees beneficial variants from LD with deleterious mutations generated by the interaction between genetic drift and selection (Hill and Roberston 1968; Peck 1994; Otto and Barton 2001; Keightley and Otto 2006).

Although mutation is often a key ingredient in models of the evolution of recombination, recombination and mutation are routinely treated as independent processes. Consideration of the molecular and cellular steps that lead to the formation of crossovers challenges this assumption. Recombination requires the programmed generation of DNA damage in the form of double-strand breaks (DSBs), a minority of which are resolved as crossovers between homologous chromosomes (Keeney et al. 1997; Martini et al. 2011; Cole et al. 2012; Yokoo et al. 2012; Hunter 2015; Varas et al. 2015; Gray and Cohen 2016). The repair of DSBs associated with recombination is prone to error (Macaisne et al. 2018; Rodgers and McVey 2016; Hanscom and McVey 2020), causing recombination to be mutagenic. The most direct evidence for this claim comes from experiments in budding yeast. After Magni and Von Borstel (1962) used the frequency of reversion of auxotrophic alleles to conclude that the mutation rate is higher during meiosis than mitosis, Magni (1963) showed that a large excess of meiotic revertant cells was associated with a crossover. Mitotic budding yeast cells with DSBs experimentally induced near revertible alleles have a 100-fold higher reversion rate than cells without induced DSBs (Strathern et al. 1995). Using a forward mutation reporter, Rattray et al. (2015) found an elevated mutation rate at a meiotic recombination hotspot (compared to a coldspot). The higher mutation rate is dependent on the protein that generates DSBs during meiosis (SPO11), tying mutagenic recombination to the processing of DSBs (Rattray et al. 2015).

Comparing the locations of crossovers and de novo mutations observed in human pedigrees provides additional evidence that recombination is mutagenic. Halldorsson et al. (2019) estimated that within 1kb of crossovers, maternal and paternal mutation rates are 58.4x and 41.5x their respective genome-wide rates. Examining the overlap between mutations and recombination hotspots (inferred from binding of the recombination intermediate protein DMC1), Hinch et al. (2023) suggested that 1 in 4 sperm and 1 in 12 eggs harbors a new mutation specifically due to DSB repair.

Current models of the evolution of recombination ignore the possibility that recombination is mutagenic. To address this gap in the field, we extend population genetic theory to simultaneously consider changes in the recombination rate and changes in the mutation rate. We discover new dynamics that further delineate the conditions required for recombination to evolve.

Model and Results

Our goal is to determine how mutagenic recombination affects the evolution of recombination driven by selection against deleterious variants. We build on the modeling framework established by Feldman et al. (1980) and Otto and Feldman (1997), who demonstrated that weak, synergistic epistasis against deleterious mutations favors the evolution of higher recombination.

We assume a haploid population that is large enough that genetic drift can be ignored. We analyze a three-locus modifier model, following changes in haplotype frequencies from one generation to the next that result from mutation, recombination, and selection (in that order). The model is illustrated in Figure 1. The modifier locus ($M$) controls the rates of mutation and recombination at two loci that affect viability ($A$ and $B$). The $A$ and $B$ alleles each experience recurrent mutation to deleterious alleles ($a$ and $b$, respectively) at rate $u$ for haplotypes containing the resident allele at the modifier locus ($M$) and at rate $u(1+d)$ for haplotypes containing the new modifier allele ($m$). Reverse mutations are very rare and are ignored. The $d$ parameter captures the proportional change in the mutation rate caused by the new modifier allele. There is no mutation at the modifier locus. The three loci are arranged in the following order on the chromosome: $M-A-B$. After mutation occurs at the viability loci, haploid individuals mate at random to form diploids. During meiosis, the modifier locus recombines with the first viability locus ($A$) at rate $R$. The rate of recombination between viability loci $A$ and $B$ is determined by the genotype at the modifier locus: $MM$ individuals recombine at rate $r_1,Mm$ individuals recombine at rate $r_2$, and $mm$ individuals recombine at rate $r_3$. Following recombination, selection acts on haploid viability according to the haplotype at loci $A$ and $B$. The viabilities (fitnesses) of $AB,Ab,aB$, and $ab$ individuals are $1,1-s,1-s$, and $(1-s{)}^2+\epsilon$, respectively, where $s$ denotes the loss of viability caused by selection and $\epsilon$ denotes epistasis between the two loci. When $\epsilon$ is negative, the fitness of the $ab$ combination is less than expected from the product of the fitnesses of the constituent alleles (synergistic or negative epistasis). With two alleles at each of three loci, there are eight possible haplotypes $\{MAB,MAb,MaB,Mab,mAB,mAb,maB,\mathrm{a}\mathrm{n}\mathrm{d}mab\}$ whose frequencies among adult haploids are $x_1,x_2,x_3,x_4,x_5,x_6,x_7$, and $x_8$. Parameter abbreviations are summarized in Table 1.

Figure 1. Illustration of the model.

A modifier locus (shown in blue) controls rates of recombination and mutation at two loci (red) that undergo recurrent mutation to deleterious alleles. Parameter abbreviations are described in Table 1.

Table 1.

Abbreviations for model parameters.

Abbreviation	Parameter
$R$	Rate of recombination between modifier locus ($M$) and first viability locus ($A$)
$r_1$, $r_2$, $r_3$	Rates of recombination between first viability locus ($A$) and second viability locus ($B$) for the $\mathit{MM}$, $\mathit{Mm}$, and $\mathit{mm}$ genotypes (respectively) at the modifier locus
$u$	Baseline mutation rate to deleterious alleles at the two viability loci ($A$ and $B$)
$d$	Proportional change in mutation rate conferred by the $m$ allele at the modifier locus
$W_1$	Fitness of haplotype with one deleterious allele ($\mathit{Ab}$ and $\mathit{aB}$)
$W_2$	Fitness of haplotype with two deleterious alleles ($\mathit{ab}$)
$\mathit{s}$	Strength of selection against deleterious alleles ($\mathit{a}$ and $\mathit{b}$)
$\mathit{\epsilon }$	Strength of epistasis; difference in fitness of haplotype with two deleterious alleles ($\mathit{ab}$) from that expected if fitness is multiplicative

We construct a series of eight recursion equations that describe changes in the eight haplotype frequencies from one generation to the next (Appendix). We seek to determine the conditions under which a modifier allele, $m$, that alters both the recombination rate and the mutation rate, will spread through the population. When the resident allele ($M$) is fixed and selection is strong relative to mutation, the system of recursions for the four haplotypes containing $M\left\{x_1,x_2,x_3,x_4\right\}$ equilibrates at a balance between mutation, recombination, and selection at the viability loci ($A$ and $B$). We use the frequency of deleterious alleles ($\hat{p}$) and linkage disequilibrium between alleles ($\widehat{D}$) at the two viability loci to describe the polymorphic equilibrium. Assuming epistasis is weak, $\hat{p}$ and $\widehat{D}$ can be approximated to order $\epsilon$ (Appendix) as:

$$\hat{p}=\frac{u(1-s)}{s(1-u)}+\epsilon \frac{u^2\left(r_1+s-r_{1}s-u\right)}{s^3(1-u)\left((1-u{)}^2-\left(1-r_1\right)(1-s{)}^2\right)}+\vartheta \left({\epsilon }^2\right)$$

$$\hat{D}=\epsilon \frac{(s-u{)}^2{u}^2}{s^4(1-u{)}^2\left((1-u{)}^2-\left(1-r_1\right)(1-s{)}^2\right)}+\vartheta \left({\epsilon }^2\right).$$ (1)

The first part of the expression for $\hat{p}$ reduces to the familiar $\frac{u}{s}$ (Haldane 1937) when $u$ is small relative to $s$. [Note: The denominators in this expression contained a typo in Otto and Feldman (1997), with the mutation rate dropped from $(1-u{)}^2-\left(1-r_1\right)(1-s{)}^2$.]

To evaluate the stability of this resident equilibrium to invasion of the modifier allele, $m$, that affects both recombination and mutation, we construct the stability matrix containing linearized recursions for the frequencies of the four haplotypes that harbor $m\left\{x_5,x_6,x_7,x_8\right\}$ (Appendix). The leading eigenvalue of this matrix $\left({\lambda }_L\right)$ evaluated at the resident equilibrium $(\hat{p},\widehat{D})$ determines the fate of the new modifier allele: invasion occurs when ${\lambda }_L>1$, whereas invasion is resisted when ${\lambda }_L<1$. The expression for ${\lambda }_L$ for this stability matrix is long and difficult to interpret. To make progress, we assume that epistasis $(\epsilon )$ and the mutation effect of the modifier allele $(d)$ are similarly weak. Then, ${\lambda }_L$ can be approximated to order $\epsilon$ (Appendix) as:

$$\begin{gathered} {\lambda }_L= \\ 1 \\ -d\frac{(s-u)u\left[\left(2(1-R)\left(1-r_2\right)+R{r}_2\right)((1-u)-(1-R)(1-s))+(1-R)r_2(1-R+Rs-u)\right]}{X(1-u)(R+s-Rs-u)} \\ -\epsilon \left(r_2-r_1\right)Y\frac{(s-u{)}^2{u}^2}{s^{2}X(R+s-Rs-u)((1-u{)}^2-(1-R)\left(1-r_2\right)(1-s{)}^2)((1-u{)}^2-(1-\mathrm{r}1)(1-s{)}^2)} \\ +\vartheta \left({\epsilon }^2\right), \end{gathered}$$ (2)

where

$$\left.X=(1-u)-\left((1-R)\left(1-r_2\right)+R{r}_2\right)(1-s)\right),$$

and

$$Y=(1-u{)}^2-(1-s{)}^2\left(1-r_2\right)+R(1-s{)}^2\left(2-3r_2\right)-R^2(1-s{)}^2\left(1-2r_2\right).$$

The fractions in equation (2) are positive, assuming that the mutation rate is small relative to the remaining parameters. As noted by Otto and Feldman (1997), $Y$ is also positive for tightly linked modifiers ($R$ small) and remains so even if $R=\frac{1}{2}$ when epistasis is weak, as assumed in (2).

The signs of the two main parts of expression (2) thus determine the fate of a new modifier allele. The sign of the term involving $\epsilon$ depends on the difference in recombination frequency between modifier genotypes ($r_2-r_1$). When recombination increases with a new modifier allele ($r_2>r_1$) and $\epsilon$ is weak and negative, the last term is positive, pushing in the direction of invasion (Otto and Feldman 1997; who also show that the $\epsilon$-term disfavors increased recombination when epistasis is strong, unless the modifier is very tightly linked). By contrast, the sign of the term involving $d$ is determined by the sign of $d$ (this term does not depend on the difference in recombination frequency between genotypes). When the new modifier allele increases the mutation rate ($d>0$), the $d$-term is negative, bolstering the stability of the resident equilibrium. When the new modifier allele decreases the mutation rate ($d<0$), the $d$-term is positive, favoring invasion. Importantly, the relative magnitudes of these two terms differ, with the $d$-term proportional to $u$ and the $\epsilon$-term generally much smaller and proportional to $u^2$. This difference is more easily seen by assuming weak selection (order $\epsilon$) and even weaker mutation rates (order ${\epsilon }^2$), in which case the $d$-term reduces to

$$-dsu\frac{2(1-R)R\left(1-r_2\right)+R^2{r}_2+(1-R{)}^2{r}_2}{R\left(R+r_2-2R{r}_2\right)},$$

while the $\epsilon$-term reduces to

$$-\epsilon \left(r_2-r_1\right)u^2\frac{r_2+R\left(1-r_2\right)+R(1-R)\left(1-2r_2\right)}{R{r}_1\left(R+r_2-R{r}_2\right)\left(R+r_2-2R{r}_2\right)}.$$

For biologically reasonable values of $d$ and $\epsilon$, the $d$-term will be larger in magnitude than the $\epsilon$-term. Consequently, the fate of the new modifier allele is determined disproportionately by its effect on mutation. Therefore, a modifier allele that increases both the recombination rate and the mutation rate is not expected to spread through the population.

To understand how the effects of mutagenic recombination scale with the number of loci experiencing viability selection, we developed a simpler approximation for the $d$-term of ${\lambda }_L$ in (2) and compared it to results from a two-locus model of a mutation rate modifier (Appendix). Our findings indicate that the $d$-term scales nearly linearly with the number of viability loci (Appendix).

The analytical results presented above offer general insights into the evolution of recombination in the realm where epistasis and changes in mutation rate are weak. To explore the fate of the new modifier allele under a wider array of conditions, we solve numerically for the resident equilibrium ($\hat{p}$ and $\widehat{D}$) and then for ${\lambda }_L$ of the stability matrix under a variety of parameter values.

Numerical results confirm that an allele that increases the recombination rate without changing the mutation rate spreads through the population when $\epsilon$ is negative and not too strong (Figure 2) (Otto and Feldman 1997). If the allele also elevates the mutation rate even by a small factor $(0.01\mathrm{\%};d=0.0001)$, it is no longer favored over the same range of $\epsilon$ (Figure 2). Our results also confirm that the invasion ability of an allele that increases recombination decreases with its recombinational distance from the first viability locus $(R)$ (Figure 3) (Otto and Feldman 1997). When the same allele increases the mutation rate by a very small factor ($0.001\mathrm{\%};d=0.00001$), it does not invade, unless the modifier locus is closely linked to the first viability locus (Figure 3). Although an allele that confers a larger increase in recombination rate ($r_2$ vs. $r_1$) invades more readily, the expansion must be very large for an allele that also raises the mutation rate by a small factor $(0.01\mathrm{\%};d=0.0001)$ to spread, even with tight linkage between the modifier locus and the first viability locus (Figure 4). Across a range of selection coefficients $(s)$, the resident equilibrium is stable to the introduction of a modifier allele that increases the recombination rate and increases the mutation rate by a small factor ($0.01\mathrm{\%};d=0.0001$) (Figure 5). A higher baseline mutation rate $(u)$ reduces the proportional increase caused by a given value of $d$, expanding the parameter space over which a modifier allele that increases mutation by a small factor $(d=0.0001)$ invades, but this advantage is quickly overcome by reasonable elevation in the mutation rate (Figure 6). Numerical analyses assuming a variety of parameter combinations reveal linear effects of $d$ on ${\lambda }_L$, in contrast to the non-linear effects of the other parameters.

Figure 2. Effect of epistasis.

Invasion of a modifier of recombination with $r_1=0.10,R=0.01,s=0.01,u=0.0001$. Invasion occurs when ${\lambda }_L-1$ is positive. When a modifier increases the recombination rate by 20% ($r_2=0.12$) and does not affect the mutation rate ($d=0$), it invades if epistasis ($\epsilon$) is weak and negative (blue), consistent with Otto and Feldman (1997). When the modifier increases the mutation rate by even a small factor ($0.01\mathrm{\%};d=0.0001$), it does not invade (red).

Figure 3. Effect of linkage of the modifier to loci experiencing viability selection.

Invasion of a modifier of recombination with $r_1=0.10,s=0.01,\epsilon =-0.01,u=0.0001$. Invasion occurs when ${\lambda }_L-1$ is positive. A modifier that increases the recombination rate by 20%$\left(r_2=0.12\right)$ and does not affect the mutation rate ($d=0$) invades as long as it is closely linked ($R$) to the first viability locus. A similar modifier that increases the mutation rate by a very small factor $(0.001\mathrm{\%};d=0.00001)$ only invades if it is very closely linked to the first viability locus.

Figure 4. Effect of amount of increase in recombination.

Invasion of a modifier of recombination with $r_1=0.01,R=0.01,s=0.05,\epsilon =-0.01,u=0.0001$. Invasion occurs when ${\lambda }_L-1$ is positive. Holding the effect on mutation rates constant, a modifier with a stronger effect on recombination rate invades more readily. A modifier that increases the mutation rate by a small factor $(0.01\mathrm{\%};d=0.0001)$ does not invade unless it increases the recombination rate by a very large amount.

Figure 5. Effect of selection.

Invasion of a modifier of recombination with $r_1=0.10,R=0.01,\epsilon =-0.001,u=0.0001$. Invasion occurs when ${\lambda }_L-1$ is positive. A modifier that increases the recombination rate by 20% $\left(r_2=0.12\right)$ and increases the mutation rate by a small factor $(0.01\mathrm{\%};d=0.0001)$ does not invade, regardless of the selection coefficient.

Figure 6. Effect of baseline mutation.

Invasion of a modifier of recombination with $r_1=0.10,R=0.01,\epsilon =-0.001,s=0.01$. Invasion occurs when ${\lambda }_L-1$ is positive. With a larger baseline mutation rate $(u)$, a modifier that increases the recombination rate by 20% $\left(r_2=0.12\right)$ and increases the mutation rate by a small factor $(0.01\mathrm{\%};d=0.0001)$ invades, whereas a modifier that increases the mutation rate by one factor more $(0.1\mathrm{\%};d=0.001)$ does not invade.

Overall, our numerical results are consistent with our analytical results in showing that the advantage of increased recombination in the face of synergistic epistasis among deleterious mutations is eliminated or substantially diminished by the elevated mutagenesis inherent in the formation of crossovers.

Discussion

The process of forming crossovers elevates the mutation rate (Magni 1963; Strathern et al. 1995; Rattray et al. 2015; Halldorsson et al. 2019; Hinch et al. 2023). Our analysis demonstrates an important evolutionary consequence of this phenomenon. Even small increases in the mutation rate substantially narrow the zone in which higher recombination rates can evolve. This conclusion adds a fresh perspective to an expansive body of research that delineates the conditions for recombination rate to evolve.

Our results suggest that evolutionary expansions of the genetic map are likely to be accomplished through those changes that simultaneously minimize associated mutagenesis. Some genome-wide association studies (GWAS) of recombination and some patterns of molecular evolution broadly match this prediction. Although candidate genes for variation in crossover number within populations (identified by GWAS) function at multiple steps in the pathway, there is an enrichment for genes that act after DSB generation, particularly those involved in the decision to repair DSBs as crossovers (Johnston 2024; Payseur 2025). Rapidly evolving recombination genes in mammals are functionally clustered around the crossover/non-crossover choice and the formation of the synaptonemal complex (Dapper and Payseur 2019), although signatures of positive selection are spread more evenly across the recombination pathway in birds (Szasz-Green et al. 2025). To the extent that these patterns point away from variants that increase recombination by adding DSBs, they are consistent with our theoretical findings.

At the same time, increasing DSBs could have additional fitness benefits that balance the cost of higher mutation. For example, meiotic DSBs aid chromosome homology searching and pairing in many species (Gray and Cohen 2006), including those with strong evidence for mutagenic recombination (e.g. yeast and humans). Models that explicitly treat the evolution of DSBs could provide new insights into the evolution of recombination.

Our model focuses on the advantage of elevating recombination when there is synergistic epistasis against deleterious mutations. Other scenarios may more strongly favor increased recombination, including small population size (Otto and Barton 2001; Barton and Otto 2005), environments that fluctuate temporally or spatially (Barton 1995; Otto and Michalakis 1998; Lenormand and Otto 2000), non-random mating (Roze and Lenormand 2005), and selective interference among mutations (Keightley and Otto 2006; Roze 2021). By assuming that mutations are always deleterious with a constant strength of selection, our model is also restricted to a zone in which increases in mutation rate are inherently disfavored (Karlin and McGregor 1974; Altenberg et al. 2017). Modifier alleles that confer higher mutation rates along with more recombination could evolve more readily when mutations have a distribution of fitness effects or in heterogeneous environments where adaptive mutations are possible.

Still, we expect that the mutagenic effects of recombination will usually impede its evolution. Perhaps theoretical and empirical patterns concerning the evolution of recombination rate should be revisited in the light of this conclusion.

Appendix

Recursions

Our analysis follows changes in three-locus haplotype frequencies from one generation to the next that result from mutation, recombination, and selection. With two alleles at each of the three loci, there are eight possible haplotypes $\{MAB,MAb,MaB,Mab,mAB,mAb,maB,mab\}$ whose frequencies among adult haploids are $\left\{x_1,x_2,x_3,x_4,x_5,x_6,x_7,x_8\right\}$.

Haplotype frequencies after mutation are:

$$x_{1,m}=x_1(1-u{)}^2$$

$$x_{2,m}=x_1(1-u)+x_2(1-u)$$

$$x_{3,m}=x_{1}u(1-u)+x_3(1-u)$$

$$x_{4,m}=x_1{u}^2+x_{2}u+x_{3}u+x_4$$

$$x_{5,m}=x_5[1-u(1+d){]}^2$$

$$x_{6,m}=x_5[1-u(1+d)][u(1+d)]+x_6[1-u(1+d)]$$

$$x_{7,m}=x_{5}u(1+d)[1-u(1+d)]+x_7[1-u(1+d)]$$

$$x_{8,m}=x_5[u(1+d){]}^2+x_{6}u(1+d)+x_{7}u(1+d)+x_8$$

Haplotype frequencies following recombination are:

$$\begin{gathered} x_{1,r}=x_{1,m}-r_1\left(x_{1,m}{x}_{4,m}-x_{2,m}{x}_{3,m}\right)-\left(r_2+R-2r_{2}R\right)\left(x_{1,m}{x}_{6,m}-x_{2,m}{x}_{5,m}\right) \\ -R\left(x_{1,m}{x}_{7,m}-x_{3,m}{x}_{5,m}\right)-\left(R-r_{2}R\right)\left(x_{1,m}{x}_{8,m}-x_{4,m}{x}_{5,m}\right) \\ +r_{2}R\left(x_{3,m}{x}_{6,m}-x_{2,m}{x}_{7,m}\right)+r_2\left(x_{2,m}{x}_{7,m}-x_{1,m}{x}_{8,m}\right) \end{gathered}$$

$$\begin{gathered} x_{2,r}=x_{2,m}+r_1\left(x_{1,m}{x}_{4,m}-x_{2,m}{x}_{3,m}\right)+\left(r_2+R-2r_{2}R\right)\left(x_{1,m}{x}_{6,m}-x_{2,m}{x}_{5,m}\right) \\ -R\left(x_{2,m}{x}_{8,m}-x_{4,m}{x}_{6,m}\right)+\left(R-r_{2}R\right)\left(x_{3,m}{x}_{6,m}-x_{2,m}{x}_{7,m}\right) \\ -r_{2}R\left(x_{1,m}{x}_{8,m}-x_{4,m}{x}_{5,m}\right)+r_2\left(x_{1,m}{x}_{8,m}-x_{2,m}{x}_{7,m}\right) \end{gathered}$$

$$\begin{gathered} x_{3,r}=x_{3,m}+r_1\left(x_{1,m}{x}_{4,m}-x_{2,m}{x}_{3,m}\right)-\left(r_2+R-2r_{2}R\right)\left(x_{3,m}{x}_{8,m}-x_{4,m}{x}_{7,m}\right) \\ +R\left(x_{1,m}{x}_{7,m}-x_{3,m}{x}_{5,m}\right)-\left(R-r_{2}R\right)\left(x_{3,m}{x}_{6,m}-x_{2,m}{x}_{7,m}\right) \\ +r_{2}R\left(x_{1,m}{x}_{8,m}-x_{4,m}{x}_{5,m}\right)-r_2\left(x_{3,m}{x}_{6,m}-x_{4,m}{x}_{5,m}\right) \end{gathered}$$

$$\begin{gathered} x_{4,r}=x_{4,m}-r_1\left(x_{1,m}{x}_{4,m}-x_{2,m}{x}_{3,m}\right)+\left(r_2+R-2r_{2}R\right)\left(x_{3,m}{x}_{8,m}-x_{4,m}{x}_{7,m}\right) \\ +R\left(x_{2,m}{x}_{8,m}-x_{4,m}{x}_{6,m}\right)+\left(R-r_{2}R\right)\left(x_{1,m}{x}_{8,m}-x_{4,m}{x}_{5,m}\right) \\ -r_{2}R\left(x_{3,m}{x}_{6,m}-x_{2,m}{x}_{7,m}\right)-r_2\left(x_{4,m}{x}_{5,m}-x_{3,m}{x}_{6,m}\right) \end{gathered}$$

$$\begin{gathered} x_{5,r}=x_{5,m}-r_3\left(x_{5,m}{x}_{8,m}-x_{6,m}{x}_{7,m}\right)+\left(r_2+R-2r_{2}R\right)\left(x_{1,m}{x}_{6,m}-x_{2,m}{x}_{5,m}\right) \\ +R\left(x_{1,m}{x}_{7,m}-x_{3,m}{x}_{5,m}\right)+\left(R-r_{2}R\right)\left(x_{1,m}{x}_{8,m}-x_{4,m}{x}_{5,m}\right) \\ -r_{2}R\left(x_{3,m}{x}_{6,m}-x_{2,m}{x}_{7,m}\right)-r_2\left(x_{4,m}{x}_{5,m}-x_{3,m}{x}_{6,m}\right) \end{gathered}$$

$$\begin{gathered} x_{6,r}=x_{6,m}+r_3\left(x_{5,m}{x}_{8,m}-x_{6,m}{x}_{7,m}\right)-\left(r_2+R-2r_{2}R\right)\left(x_{1,m}{x}_{6,m}-x_{2,m}{x}_{5,m}\right) \\ +R\left(x_{2,m}{x}_{8,m}-x_{4,m}{x}_{6,m}\right)-\left(R-r_{2}R\right)\left(x_{3,m}{x}_{6,m}-x_{2,m}{x}_{7,m}\right) \\ +r_{2}R\left(x_{1,m}{x}_{8,m}-x_{4,m}{x}_{5,m}\right)-r_2\left(x_{3,m}{x}_{6,m}-x_{4,m}{x}_{5,m}\right) \end{gathered}$$

$$\begin{gathered} x_{7,r}=x_{7,m}+r_3\left(x_{5,m}{x}_{8,m}-x_{6,m}{x}_{7,m}\right)+\left(r_2+R-2r_{2}R\right)\left(x_{3,m}{x}_{8,m}-x_{4,m}{x}_{7,m}\right) \\ -R\left(x_{1,m}{x}_{7,m}-x_{3,m}{x}_{5,m}\right)+\left(R-r_{2}R\right)\left(x_{3,m}{x}_{6,m}-x_{2,m}{x}_{7,m}\right) \\ -r_{2}R\left(x_{1,m}{x}_{8,m}-x_{4,m}{x}_{5,m}\right)+r_2\left(x_{1,m}{x}_{8,m}-x_{2,m}{x}_{7,m}\right) \end{gathered}$$

$$\begin{gathered} x_{8,r}=x_{8,m}-r_3\left(x_{5,m}{x}_{8,m}-x_{6,m}{x}_{7,m}\right)-\left(r_2+R-2r_{2}R\right)\left(x_{3,m}{x}_{8,m}-x_{4,m}{x}_{7,m}\right) \\ -R\left(x_{2,m}{x}_{8,m}-x_{4,m}{x}_{6,m}\right)-\left(R-r_{2}R\right)\left(x_{1,m}{x}_{8,m}-x_{4,m}{x}_{5,m}\right) \\ +r_{2}R\left(x_{3,m}{x}_{6,m}-x_{2,m}{x}_{7,m}\right)+r_2\left(x_{2,m}{x}_{7,m}-x_{1,m}{x}_{8,m}\right) \end{gathered}$$

Following selection, haplotype frequencies in the next generation are:

$$x_1^{\prime }=\frac{x_{1,r}}{\bar{W}}$$

$$x_2^{\prime }=\frac{x_{2,r}{W}_1}{\bar{W}}$$

$$x_3^{\prime }=\frac{x_{3,r}{W}_1}{\bar{W}}$$

$$x_4^{\prime }=\frac{x_{4,r}{W}_2}{\bar{W}}$$

$$x_5^{\prime }=\frac{x_{5,r}}{\bar{W}}$$

$$x_6^{\prime }=\frac{x_{6,r}{W}_1}{\bar{W}}$$

$$x_7^{\prime }=\frac{x_{7,r}{W}_1}{\bar{W}}$$

$$x_8^{\prime }=\frac{x_{8,r}{W}_2}{\bar{W}}$$

where $\bar{W}$ is average fitness computed as the sum of haplotype fitnesses weighted by haplotype frequencies, $W_1=1-s$, and $W_2=(1-s{)}^2+\epsilon$.

Equilibria

We first seek equilibria when the resident allele ($M$) is fixed in the population. To simplify the analysis, we assume that selection acts symmetrically, which reduces the system of four recursions involving $M$ to two recursions (following $x_1$ and $x_2$), and we replace haplotype frequencies with expressions involving the frequency of deleterious alleles ($p=p_a=p_b=1-\left.\left(x_1+x_2\right)\right)$ and the linkage disequilibrium between alleles $\left(D=D_{AB}\right)$. To find approximations for equilibrium values of $p$ and $D$, we use a perturbation analysis assuming epistasis is weak ($\epsilon$ is small). We rewrite $p$ and $D$ as power functions of $\epsilon$, insert them into the recursions, take Taylor series around $\epsilon =0$, and solve for the terms in zeroth order and first order of $\epsilon$. This procedure leads to equation (1) for the equilibrium in the main text.

Stability to Introduction of Allele that Modifies both Mutation and Recombination

To determine the conditions under which an allele that modifies both mutation and recombination will spread through the population, we use a perturbation analysis to approximate the leading eigenvalue (${\lambda }_L$) of the Jacobian matrix evaluated at the resident equilibrium. We ask how small changes in $\epsilon$ or $d$ perturb ${\lambda }_L$ away from the boundary between stability and invasion (${\lambda }_L=1$). In the characteristic polynomial evaluated at the resident equilibrium, we replace ${\lambda }_L$ with 1 plus a small term of order $\epsilon$ and replace $d$ with a small term of order $\epsilon$. Assuming $\epsilon$ and $d$ are small, taking Taylor series around $\epsilon =0$, and solving for the small term of order $\epsilon$ leads to expression (2) for ${\lambda }_L$ in the main text.

Effects of Modifier of Mutation Rate in Two-Locus vs. Three-Locus Models

To understand how the effects of changing the mutation rate are expected to scale to a larger number of selected loci, we analyzed a simpler, two-locus model, in which a modifier controls the mutation rate at a single locus. The two-locus model features the same parameters and setup as the three-locus model, except the modifier does not affect recombination and viability is determined by a single locus. With two alleles at each locus, there are four haplotypes $\{MA,Ma,mA,\mathrm{a}\mathrm{n}\mathrm{d}ma\}$ with frequencies $x_1,x_2,x_3$, and $x_4$. Following mutation at the viability locus, haplotype frequencies are:

$$x_{1,m}=x_1(1-u)$$

$$x_{2,m}=x_{1}u+x_2$$

$$x_{3,m}=x_3[1-u(1+d)]$$

$$x_{4,m}=x_3[u(1+d)]+x_4$$

Following recombination between the modifier locus and the viability locus, haplotype frequencies are:

$$x_{1,r}=x_{1,m}{x}_{1,m}+x_{1,m}{x}_{2,m}+x_{1,m}{x}_{3,m}+(1-R)x_{1,m}{x}_{4,m}+R{x}_{2,m}{x}_{3,m}$$

$$x_{2,r}=x_{1,m}{x}_{2,m}+R{x}_{1,m}{x}_{4,m}+x_{2,m}{x}_{2,m}+(1-R)x_{2,m}{x}_{3,m}+x_{2,m}{x}_{4,m}$$

$$x_{3,r}=x_{1,m}{x}_{3,m}+R{x}_{1,m}{x}_{4,m}+(1-R)x_{2,m}{x}_{3,m}+x_{3,m}{x}_{3,m}+x_{3,m}{x}_{4,m}$$

$$x_{4,r}=(1-R)x_{1,m}{x}_{4,m}+R{x}_{2,m}{x}_{3,m}+x_{2,m}{x}_{4,m}+x_{3,m}{x}_{4,m}+x_{4,m}{x}_{4,m}$$

Following selection, haplotype frequencies in the next generation are:

$$x_1^{\prime }=\frac{x_{1,r}}{\bar{W}}$$

$$x_2^{\prime }=\frac{x_{2,r}{W}_1}{\bar{W}}$$

$$x_3^{\prime }=\frac{x_{3,r}}{\bar{W}}$$

$$x_4^{\prime }=\frac{x_{4,r}{W}_1}{\bar{W}}$$

where $\bar{W}$ is average fitness computed as the sum of haplotype fitnesses weighted by haplotype frequencies, and $W_1=1-s$.

Solving for the equilibrium value of $x_1$ with the resident allele ($M$) fixed ($x_3=x_4=0$), leads to this expression:

$${\hat{x}}_1=\frac{s-u}{s(1-u)}$$

As expected, the expression approaches $1-\frac{u}{s}$ when $u$ is small (Karlin and McGregor 1974). To determine the stability of this equilibrium, we construct the $2\times 2$ Jacobian matrix for the two haplotype frequencies $\left(x_3,x_4\right)$ involving the modifier allele that affects the mutation rate $(m)$ and evaluate this matrix at the resident equilibrium. Solving for ${\lambda }_L$, then assuming $u$ is small and biological constraints on recombination, selection, and mutation are satisfied ($0\le R\le \frac{1}{2},s>0,u>0$), we use a Taylor series to obtain:

$${\lambda }_L=1-dsu\frac{(1-R)}{R+s-Rs}+\vartheta \left(u^2\right)$$

Stability to invasion by the modifier allele is determined by the sign of $d$. When the modifier allele increases the mutation rate ($d>0),{\lambda }_L<1$, and it does not invade. When the modifier allele decreases the mutation rate $(d<0),{\lambda }_L>1$, and it spreads through the population.

To consider the effects of mutation on two viability loci, we assume that the first viability locus is distance $R$ from the modifier locus, and the second viability locus is twice this distance, which (using Haldane’s mapping function) is $2R(1-R)$. Then, we further approximate the $d$-term above separately for the two viability loci, assuming very weak mutation (order ${\epsilon }^2$), weak selection (order $\epsilon$), and weak effects of the modifier allele on mutation and recombination (order $\epsilon )$. Summing results for the two loci, we obtain this expression for the effects of mutation on ${\lambda }_L$:

$$-dsu\left(\frac{1-R}{R}+\frac{1-(2R(1-R))}{2R(1-R)}\right)=-dsu\frac{3-6R+4R^2}{2R(1-R)}+\vartheta \left({\epsilon }^5\right)$$

Under the same assumptions, the $d$-term derived from the three-locus model equals the exact same quantity. This result suggests that the effects of mutagenic recombination on the spread of a recombination rate modifier scale in a nearly linear manner with the number of viability loci.

Data and Code Availability

Two Mathematica notebooks, “ThreeLocus_RecombinationMutation_10.31.25.nb” and “TwoLocus_Mutation_10.31.25.nb,” provide additional details on models and their analyses.