Unified reduction principle for the evolution of mutation, migration, and recombination

Significance

Evolution by Darwinian natural selection can not only shape how organisms survive and reproduce, but also affect transmission of genetic and other information between generations. Modifier-gene models for the evolution of information transmission have revealed a universal tendency for more faithful transmission to evolve in populations at equilibrium where natural selection is balanced by errors in information transmission. This is shown to be a very general property of models that include mutation and migration under selection and recombination under selection on diploids. The breadth of this reduction principle focuses attention on the departures from its mathematical assumptions, which may explain those biological phenomena of information transmission between generations for which the reduction principle fails.

Abstract

Modifier-gene models for the evolution of genetic information transmission between generations of organisms exhibit the reduction principle: Selection favors reduction in the rate of variation production in populations near equilibrium under a balance of constant viability selection and variation production. Whereas this outcome has been proven for a variety of genetic models, it has not been proven in general for multiallelic genetic models of mutation, migration, and recombination modification with arbitrary linkage between the modifier and major genes under viability selection. We show that the reduction principle holds for all of these cases by developing a unifying mathematical framework that characterizes all of these evolutionary models.

The theory of the reduction principle and its departures has its beginnings, like so many other threads of mathematical evolutionary theory, in Fisher’s The Genetical Theory of Natural Selection (1). Fisher describes how the asymmetric flow from fitter genotypes to less-fit genotypes due to homologous recombination between pairs of polymorphic loci creates an “agency” to reduce recombination between genes. Fisher notes, however, that due to the ubiquity of chromosomal crossing-over “it is an inevitable inference that some other cause must induce an equally powerful selection in favor of crossing over.” Fisher continues, “the mathematical difficulties of an exact investigation are worthy of a far more extended treatment.” (ref. 1, p. 119). Decades after Fisher’s observations, the mathematical difficulties have begun to be resolved.

Recombination affects the distribution of genotypes among an organism’s offspring. As such, it can be regarded as a process that affects information transmission between generations. Mutation is another such process. With migration, the geographic location, in addition to genetic information, is transmitted between generations. The evolutionary dynamics of the control of information transmission between generations are the subject of modifier-gene theory.

In evolutionary genetics, modifier theory is the study of the dynamics of a gene (or genes) whose alleles modify the rate of a process that changes genetic information between organismal generations, information that is represented in the state of “major genes” that are under selection. Modifier genes can evolve even when they have no direct effects on fitness, and the primary focus of modifier-gene theory is the dynamics of such “neutral” modifier genes. Neutral modifier alleles change in frequency due to their associations with genes that are under direct selection.

The first mathematical model for the evolution of a modifier gene was that of Nei (2), who assumed that in a large diploid population, a gene with alleles $M$ and $m$ controlled the rate of recombination between two major genes $A/a$ $B/b$ that were under viability selection. Feldman (3) framed the analysis of the evolution of recombination in terms of conditions for $m$ to invade a population fixed on $MM$ at an equilibrium of the $A/a$ and $B/b$ genes under viability selection (4). In this case, if $Mm$ produces recombination $r_2$, allele $m$ invades $MM$, which produces recombination rate $r_1$, if $A/a$ and $B/b$ are in linkage disequilibrium and $r_2<r_1$.

In subsequent studies of the evolution of modifiers of recombination rates, mutation rates, and migration rates, the same result kept appearing: In populations at an equilibrium balance between natural selection and either mutation, recombination, or migration, a new modifier allele can invade if it reduces the rate of the modified process and cannot invade if it increases that rate (5–9). In ecological models, where the term “migration” is usually replaced by “dispersal,” and in which genetics play no role, reduced dispersal has also been shown to evolve (10). A comparison between such ecological, game-theoretic, or adaptive-dynamics arguments and the formal population genetic arguments that characterize modifier theory was made in Feldman et al. (11). The generality of the result prompted Feldman et al. (12) to call this phenomenon a “reduction principle.” Fisher’s agency was in fact observed to be a general outcome of the evolutionary dynamics of genetic information transmission.

The possibility that a mathematical unity underlies the reduction principle motivated the study by Altenberg (13). What emerged is that the behavior of modifier-gene models possesses a universal part and a particular part. The universal part—the mathematical unity that underlies the reduction principle—is reflected in an important theorem of Karlin (14), which can be summarized as “mixing reduces growth.” Karlin developed the theorem to study the effect of population subdivision on the preservation of genetic diversity, a problem seemingly unrelated to modifier-gene theory. The particular part of modifier-gene models concerns the details of the genetics, for example, how the modifier gene is linked to the major loci, whether the organism is diploid or haploid, the number of information-altering processes, the kind of selection on the major genes, and other such details. These details greatly complicate the mathematical analysis.

The reduction principle was proved to hold for modifiers of mutation, recombination, and migration (dispersal) (13, 15) for arbitrary numbers of genotypes and selection regimes, but only in the case of tightly linked modifiers or modifiers with extreme reduction, by applying Karlin’s theorem. The universal aspect of the reduction principle was shown, but under restricted conditions.

In the converse direction, the particular part of modifier-gene models was solved in a series of papers that include arbitrary numbers of modifier-gene alleles, arbitrary linkage to the major loci, arbitrary selection regimes, and arbitrary control by the modifier locus, in the case of modifiers of recombination (16), mutation (17), and migration (18). The main analytic technique there—the analysis of spectral radii through the characteristic polynomial—limited the universal part to two alleles at the major loci in the case of recombination and mutation modification and two patches in the case of migration modification (19).

The mathematical complications in the particular parts of modifier models were resolved for diploid modifiers (20) through application of an extension of Karlin’s theorem to essentially nonnegative matrices. We apply this method here to multiallele genetic models that include mutation modifiers with haploid selection and migration modifiers with haploid selection. Further, we give the complete results for modifiers of recombination under diploid selection, a case that, although discussed in Altenberg (20), has not been explicitly shown.

We first introduce separately the modification models for mutation, migration, and recombination and exhibit the local stability matrices that determine whether a new modifier allele will invade. Next, we show that there is a unified structure that characterizes these matrices and review the mathematical tools needed to bound the spectral radii of these structurally unified matrices. Finally, we formulate what we call the “unified reduction principle” and outline its simple proof.

The Models

In what follows we describe models for the modification of mutation, migration, and recombination. In all three models, we consider a large population and a character under selection that is determined by a “major locus” (“major loci” in the recombination case). The genotypes determining this character are multiallelic. Linked to it is a “modifier locus” that has no direct effect on fitness and whose function is to determine the rate of mutation, migration, or recombination. We assume that the modifier locus has two alleles, $M$ and $m$, and that initially only $M$ is present. Suppose that the population evolves to a stable equilibrium when only $M$ is present; we check the local stability of this equilibrium to the introduction of the allele $m$ at the modifier locus. We call this “external stability.”

The Mutation Modification Model.

Here we consider a large population of haploids and a character determined by a major locus with $n$ possible alleles $A_1,A_2,\mathrm{\dots },A_n$. Linked to this locus is a modifier locus with two possible alleles, $M$ and $m$, that determine the mutation rates between the $n$ alleles $A_1,A_2,\mathrm{\dots },A_n$. Specifically, when $M(m)$ is present with probability $1-{\mu }_M(1-{\mu }_m)$, the allele $A_i$ does not mutate, and with probability $\frac{{\mu }_M}{n-1}\left(\frac{{\mu }_m}{n-1}\right)$ $A_i$ mutates to $A_j$ for any $j\ne i$. As the modifier locus has no direct effect on fitness, the fitnesses of the genotypes $A_{i}M$ and $A_{i}m$ have the same value $w_i$ $(i=\mathrm{\hspace{0.17em}1},2,\mathrm{\dots },n)$. Recombination occurs between the two loci at rate $r$ ($0\le r\le \mathrm{\hspace{0.17em}1}$), such that the outcome of the mating between $A_{i}M$ and $A_{j}m$ is $\frac{1}{2}{A}_{i}M+\frac{1}{2}{A}_{j}m$ with probability $(1-r)$ and $\frac{1}{2}{A}_{i}m+\frac{1}{2}{A}_{j}M$ with probability $r$.

Let $x_i$ and $y_i$ be the frequencies of $A_{i}M$ and $A_{i}m$, respectively, in the present generation with $0\le x_i$, $y_i\le \mathrm{\hspace{0.17em}1}$ for $i=\mathrm{\hspace{0.17em}1},2,\mathrm{\dots },n$ and ${\sum }_{i=1}^n(x_i+y_i)=\mathrm{\hspace{0.17em}1}$. Let $\text{𝐱}={(x_1,x_2,\mathrm{\dots },x_n)}^{\text{T}}$ and $\text{𝐲}={(y_1,y_2,\mathrm{\dots },y_n)}^{\text{T}}$ (T is the transpose operation) and write $(\text{𝐱},\text{𝐲})$ as the frequency vector in the present generation. Then, after selection, random mating, recombination, segregation, and mutation, in this order, $(\text{𝐱}\prime ,\text{𝐲}\prime )$ in the next generation are given by

$$x_i^{\prime }=(1-{\mu }_M)\left[\frac{w_i{x}_i}{w}-r{\displaystyle \sum _{j=1}^n}\frac{w_i{w}_j}{w^2}{D}_{ij}\right]$$ [1]

$$+\frac{{\mu }_M}{n-1}\sum _{j\ne i}\left[\frac{w_j{x}_j}{w}-r{\displaystyle \sum _{k=1}^n}\frac{w_j{w}_k}{w^2}{D}_{jk}\right]$$

$$y_i^{\prime }=(1-{\mu }_m)\left[\frac{w_i{y}_i}{w}+r{\displaystyle \sum _{j=1}^n}\frac{w_i{w}_j}{w^2}{D}_{ij}\right]$$ [2]

$$+\frac{{\mu }_m}{n-1}\sum _{j\ne i}\left[\frac{w_j{y}_j}{w}+r{\displaystyle \sum _{k=1}^n}\frac{w_j{w}_k}{w^2}{D}_{jk}\right]$$

for all $i,j=1,2,\mathrm{\dots },n$, where

$$D_{ij}=x_i{y}_j-y_i{x}_j,\hspace{1em}{D}_{ji}=-D_{ij},\hspace{1em}{\displaystyle \sum _{i,j=1}^n}{D}_{ij}=0,$$ [3]

and

$$w={\displaystyle \sum _{i=1}^n}{w}_i(x_i+y_i)$$ [4]

is the “mean fitness.” The $D_{ij}$ values for $i\ne j$ ($D_{ii}=0$) are the linkage disquilibria.

Let $({\mathrm{𝐱}}^{\ast },0)$ be the equilibrium of the system of Eqs. 1 and 2, where only $M$ is present. In this case $\mathrm{𝐲}=0$, $D_{ij}=0$ for all $i$ and $j$, and ${\mathrm{𝐱}}^{\ast }$ satisfies the equilibrium equations

$$w{x}_i=(1-{\mu }_M)w_i{x}_i+\frac{{\mu }_M}{n-1}\sum _{j\ne i}{w}_j{x}_j;\hspace{1em}i=1,\hspace{0.17em}2,\mathrm{\dots },n,$$ [5]

which we can write as

$$\text{𝐱}=\left[(1-{\mu }_M)\text{𝐈}+{\mu }_M\text{𝐒}\right]\stackrel{\sim }{\text{𝐃}}x,$$ [6]

where I is the $n\times n$ identity matrix, $\stackrel{\sim }{\text{𝐃}}$ is the diagonal matrix

$$\stackrel{\sim }{\text{𝐃}}=\text{diag}(\frac{w_1}{w},\frac{w_2}{w},\mathrm{\dots },\frac{w_n}{w})$$ [7]

with $w={\sum }_{i=1}^n{w}_i{x}_i$, and S is given by

$$\text{𝐒}=\frac{1}{n-1}\left[\begin{array}{ccccc}\hfill 0\hfill & \hfill 1\hfill & \hfill 1\hfill & \hfill \mathrm{\dots }\hfill & \hfill 1\hfill \\ \hfill 1\hfill & \hfill 0\hfill & \hfill 1\hfill & \hfill \mathrm{\dots }\hfill & \hfill \mathrm{⋮}\hfill \\ \hfill 1\hfill & \hfill 1\hfill & \hfill 0\hfill & \hfill \mathrm{\dots }\hfill & \hfill \mathrm{⋮}\hfill \\ \hfill \mathrm{⋮}\hfill & \hfill \mathrm{⋮}\hfill & \hfill \mathrm{⋮}\hfill & \hfill \mathrm{\ddots }\hfill & \hfill \mathrm{⋮}\hfill \\ \hfill 1\hfill & \hfill \mathrm{\dots }\hfill & \hfill \mathrm{\dots }\hfill & \hfill \mathrm{\dots }\hfill & \hfill 0\hfill \end{array}\right],$$ [8]

with zeroes on the diagonal and $\frac{1}{n-1}$ elsewhere. If $\stackrel{\sim }{\text{𝐃}}$ is a scalar matrix, then $w_i=w$ for all $i$ and there is only one equilibrium $(\frac{1}{n},\frac{1}{n},\mathrm{\dots },\frac{1}{n})$ that does not depend on ${\mu }_M$. We exclude this case and assume that $\stackrel{\sim }{\text{𝐃}}$ is a nonscalar matrix.

The local stability of $({\mathbf{x}}^{\ast },\text{𝟎})$ to the introduction of the allele $m$ at the modifier locus is determined by the linear approximation ${\text{𝐋}}^{\ast }$ of the transformation Eqs. 1 and 2 near $({\mathbf{x}}^{\ast },\text{𝟎})$. Specifically, let $(\text{𝐱},\mathbf{y})=({\mathbf{x}}^{\ast },\text{𝟎})+(\mathit{ϵ},𝜹)$, where $\mathit{ϵ}={({ϵ}_1,{ϵ}_2,\mathrm{\dots },{ϵ}_n)}^{\text{T}}$, $𝜹={({\delta }_1,{\delta }_2,\mathrm{\dots },{\delta }_n)}^{\text{T}}$ with ${ϵ}_i$ and ${\delta }_i$ positive and small and ${\sum }_{i=1}^n({ϵ}_i+{\delta }_i)=\text{𝟎}$. Then $(\text{𝐱}\prime ,\text{𝐲}\prime )=({\text{𝐱}}^{\ast },0)+(\mathit{ϵ}\prime ,𝜹\prime )$, where $\mathit{ϵ}\prime ,𝜹\prime$ have the same properties as $\mathit{ϵ}$ and $𝜹$. Therefore, up to first-order terms in $\mathit{ϵ}$ and $𝜹$ we can write

$$(\mathbf{x}\prime ,\mathbf{y}\prime )=({𝐱}^{\ast },\text{𝟎})+{\text{𝐋}}^{\ast }(\mathit{ϵ},𝜹),$$ [9]

and the stability of $({\mathbf{x}}^{\ast },\text{𝟎})$ is determined by the eigenvalues of the $2n\times 2n$ matrix ${\text{𝐋}}^{\ast }$.

The scenario in Eq. 9 is well known from modifier theory (e.g., ref. 19), and the matrix ${\text{𝐋}}^{\ast }$ is known to have the block structure

$${\text{𝐋}}^{\ast }=\left[\begin{array}{cc}\hfill {\text{𝐋}}_{\text{in}}^{\ast }\hfill & \hfill \begin{array}{ccc}\hfill \oplus \hfill & \hfill \mathrm{\dots }\hfill & \hfill \oplus \hfill \\ \hfill \oplus \hfill & \hfill \mathrm{\dots }\hfill & \hfill \oplus \hfill \end{array}\hfill \\ \hfill \begin{array}{ccc}\hfill 0\hfill & \hfill \mathrm{\dots }\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill \mathrm{\dots }\hfill & \hfill 0\hfill \end{array}\hfill & \hfill {\text{𝐋}}_{\text{ex}}^{\ast }.\hfill \end{array}\right]$$ [10]

Because the linear approximation to the dynamics of the rare haplotype frequencies $y_i$ (those carrying $m$) in Eq. 2 near $({\text{𝐱}}^{\ast },\text{𝟎})$ does not involve $\mathit{ϵ}$, the entries marked $\oplus$ do not affect the eigenvalues of ${\text{𝐋}}^{\ast }$, which are therefore those of the two $n\times n$ submatrices ${\mathbf{L}}_{\text{in}}^{\ast }$ and ${\mathbf{L}}_{\text{ex}}^{\ast }$. ${\mathbf{L}}_{\text{in}}^{\ast }$ determines the internal stability of ${\text{𝐱}}^{\ast }$ confined to the boundary where only $M$ is present, and ${\mathbf{L}}_{\text{ex}}^{\ast }$ is the external stability matrix, namely the linear approximation to the evolution near ${\mathbf{x}}^{\ast }$ involving only the gametes $A_{1}m,A_{2}m,\mathrm{\dots },A_{n}m$. Because we assume that ${\mathbf{x}}^{\ast }$ is internally stable, the stability of ${\text{𝐱}}^{\ast }$ is determined by the eigenvalues of ${\mathbf{L}}_{\text{ex}}^{\ast }$. $𝜹\prime ={\mathbf{L}}_{\text{ex}}^{\ast }𝜹$ is given, by Eq. 2, as

$${\delta }_i^{\prime }=(1-{\mu }_m)\left[\frac{w_i{\delta }_i}{w^{\ast }}+r{\displaystyle \sum _{j=1}^n}\frac{w_i{w}_j}{{(w^{\ast })}^2}{D}_{ij}^{\ast }\right]+\frac{{\mu }_m}{n-1}\sum _{j\ne i}\left[\frac{w_j{\delta }_j}{w^{\ast }}+r{\displaystyle \sum _{k=1}^n}\frac{w_j{w}_k}{{(w^{\ast })}^2}{D}_{jk}^{\ast }\right],$$ [11]

where

$$D_{ij}^{\ast }=x_i^{\ast }{\delta }_j-x_j^{\ast }{\delta }_i,\hspace{1em}{D}_{jk}^{\ast }=x_j^{\ast }{\delta }_k-x_k^{\ast }{\delta }_j,$$ [12]

and

$$w^{\ast }={\displaystyle \sum _{i=1}^n}{w}_i{x}_i^{\ast }.$$ [13]

Using Eqs. 12 and 13, Eq. 11 can be rewritten as

$${\delta }_i^{\prime }=(1-{\mu }_m)\left[\frac{w_i{\delta }_i}{w^{\ast }}(1-r)+r\frac{w_i{x}_i^{\ast }}{w^{\ast }}{\displaystyle \sum _{j=1}^n}\frac{w_j{\delta }_j}{w^{\ast }}\right]+\frac{{\mu }_m}{n-1}\sum _{j\ne i}\left[\frac{w_j{\delta }_j}{w^{\ast }}(1-r)+r\frac{w_j{x}_j^{\ast }}{w^{\ast }}\sum _{k=1}^n\frac{w_k{\delta }_k}{w^{\ast }}\right].$$ [14]

If $r=0$, Eq. 14 reduces to

$${\delta }_i^{\prime }=(1-{\mu }_m)\frac{w_i{\delta }_i}{w^{\ast }}+\frac{{\mu }_m}{n-1}\sum _{j\ne i}\frac{w_j{\delta }_j}{w^{\ast }},i=\mathrm{1,2},\mathrm{\dots },n,$$ [15]

which, on comparison with Eqs. 5 and 6, we can write in matrix notation as

$$𝜹\prime =\left[(1-{\mu }_m)\mathbf{I}+{\mu }_m\mathbf{S}\right]\mathbf{D}𝜹,$$ [16]

where D and S are given as in Eqs. 7 and 8, respectively, with $w$ replaced by $w^{\ast }$, and

$$𝜹={({\delta }_1,{\delta }_2,\mathrm{\dots },{\delta }_n)}^{\text{T}},\hspace{1em}𝜹\prime ={({\delta \prime }_1,{\delta \prime }_2,\mathrm{\dots },{\delta \prime }_n)}^{\text{T}}.$$ [17]

At $r=\mathrm{\hspace{0.17em}1}$, Eq. 14 reduces for $i=1,2,\mathrm{\dots },n$ to

$${\delta }_i^{\prime }=(1-{\mu }_m)\frac{w_i{x}_i^{\ast }}{w^{\ast }}{\displaystyle \sum _{j=1}^n}\frac{w_j{\delta }_j}{w^{\ast }}+\frac{{\mu }_m}{n-1}{\displaystyle \sum _{j\ne i}}\frac{w_j{x}_j^{\ast }}{w^{\ast }}\sum _{k=1}^n\frac{w_k{\delta }_k}{w^{\ast }},$$ [18]

which, in matrix notation, can be written as

$$𝜹\prime =\left[(1-{\mu }_m)\mathbf{I}+{\mu }_m\mathbf{S}\right]{\mathbf{Q}}^{\ast }\mathbf{D}𝜹.$$ [19]

Here ${𝐐}^{\ast }$ is the matrix all of whose columns are

$$\mathbf{D}{\mathbf{x}}^{\ast }=\left[\begin{array}{c}\hfill \frac{w_1{x}_1^{\ast }}{w^{\ast }}\hfill \\ \hfill \frac{w_2{x}_2^{\ast }}{w^{\ast }}\hfill \\ \hfill \mathrm{⋮}\hfill \\ \hfill \frac{w_n{x}_n^{\ast }}{w^{\ast }}\hfill \end{array}\right].$$ [20]

As ${\mathbf{L}}_{\text{ex}}^{\ast }$ is linear in $r$, for $0\le r\le \mathrm{\hspace{0.17em}1}$ we write ${\mathbf{L}}_{\text{ex}}^{\ast }=\mathbf{M}({\mu }_m,r)\mathbf{D}$, where

$$\mathbf{M}({\mu }_m,r)=(1-r)\left[(1-{\mu }_m)\mathbf{I}+{\mu }_m\mathbf{S}\right]+r\left[(1-{\mu }_m){\mathbf{Q}}^{\ast }+{\mu }_m\mathbf{S}{\mathbf{Q}}^{\ast }\right].$$ [21]

Observe that by Eqs. 8 and 20, S and ${\mathbf{Q}}^{\ast }$ are column-stochastic matrices, ${\mathbf{Q}}^{\ast }$ is a positive matrix, and S is a nonnegative irreducible matrix. Also it is easily seen that

$${\mathbf{Q}}^{\ast }\mathbf{D}{\mathbf{x}}^{\ast }=\mathbf{D}{\mathbf{x}}^{\ast },$$ [22]

and

$$\mathbf{M}({\mu }_M,r)\mathbf{D}{\mathbf{x}}^{\ast }={\text{𝐱}}^{\ast }\hspace{1em}\text{for}\hspace{1em}0\le r\le 1.$$ [23]

From Eqs. 6 and 23, it is shown that when ${\mu }_m={\mu }_M$, ${\mathbf{L}}_{\text{ex}}^{\ast }{\text{𝐱}}^{\ast }={\text{𝐱}}^{\ast }$, so that ${\mathbf{L}}_{\text{ex}}^{\ast }$ has an eigenvalue 1 for ${\mu }_m={\mu }_M$ with $0\le r\le \mathrm{\hspace{0.17em}1}$.

The Migration Modification Model.

Here we assume a large population of haploids that occupies two demes and a modifier locus with two alleles $M$ and $m$ that determine the migration rates between the demes to be ${\nu }_M$ and ${\nu }_m$, respectively. Let the fitnesses of the genotypes $A_{i}M$ and $A_{i}m$ be $w_i$ in deme 1 and $v_i$ in deme 2 for $i=1,2,\mathrm{\dots },n$. Assume that the recombination rate between the major locus and the modifier locus is $r$ in both demes. Let the frequencies of $A_{i}M$ and $A_{i}m$ in deme 1 be $x_i$ and $y_i$, respectively, and in deme 2, $a_i$ and $b_i$, respectively, for $i=1,2,\mathrm{\dots },n$ with ${\sum }_{i=1}^n(x_i+y_i)=1$ and ${\sum }_{i=1}^n(a_i+b_i)=1$. Then following selection, recombination and segregation, and migration (in that order), after one generation the new frequencies $x_i^{\prime },y_i^{\prime },a_i^{\prime },b_i^{\prime }$ are given by

$$x_i^{\prime }=(1-{\nu }_M)\left[\frac{w_i{x}_i}{w}-r\sum _{j=1}^n\frac{w_i{w}_j}{w^2}(x_i{y}_j-x_j{y}_i)\right]+{\nu }_M\left[\frac{v_i{a}_i}{v}-r\sum _{j=1}^n\frac{v_i{v}_j}{v^2}(a_i{b}_j-a_j{b}_i)\right]$$ [24]

$$y_i^{\prime }=(1-{\nu }_m)\left[\frac{w_i{y}_i}{w}+r\sum _{j=1}^n\frac{w_i{w}_j}{w^2}(x_i{y}_j-x_j{y}_i)\right]+{\nu }_m\left[\frac{v_i{b}_i}{v}+r\sum _{j=1}^n\frac{v_i{v}_j}{v^2}(a_i{b}_j-a_j{b}_i)\right]$$ [25]

$$a_i^{\prime }=(1-{\nu }_M)\left[\frac{v_i{a}_i}{v}-r\sum _{j=1}^n\frac{v_i{v}_j}{v^2}(a_i{b}_j-a_j{b}_i)\right]+{\nu }_M\left[\frac{w_i{x}_i}{w}-r\sum _{j=1}^n\frac{w_i{w}_j}{w^2}(x_i{y}_j-x_j{y}_i)\right]$$ [26]

$$b_i^{\prime }=(1-{\nu }_m)\left[\frac{v_i{b}_i}{v}+r\sum _{j=1}^n\frac{v_i{v}_j}{v^2}(a_i{b}_j-a_j{b}_i)\right]+{\nu }_m\left[\frac{w_i{y}_i}{w}+r\sum _{j=1}^n\frac{w_i{w}_j}{w^2}(x_i{y}_j-x_j{y}_i)\right]$$ [27]

for $i=1,2,\mathrm{\dots },n$, where

$$w=\sum _{i=1}^n{w}_i(x_i+y_i),\hspace{1em}v={\displaystyle \sum _{i=1}^n}{v}_i(a_i+b_i)$$ [28]

are the mean fitnesses in each of the two demes.

On the boundary where only modifier allele $M$ is present, namely $y_i=\mathrm{\hspace{0.17em}0}$, $b_i=\mathrm{\hspace{0.17em}0}$ for $i=\mathrm{\hspace{0.17em}1},2,\mathrm{\dots },n$, the equilibrium equations resulting from Eqs. 24 and 26 are

$$\begin{array}{c}{x}_i=(1-{\nu }_M)\frac{w_i{x}_i}{w}+{\nu }_M\frac{v_i{a}_i}{v}\hfill \\ \\ a_i=(1-{\nu }_M)\frac{v_i{a}_i}{v}+{\nu }_M\frac{w_i{x}_i}{w}\hfill \end{array}i=1,2,\mathrm{\dots },n.$$ [29]

These equations can be written in matrix notation as

$$\left[\begin{array}{c}\hfill \text{𝐱}\hfill \\ \hfill \text{𝐚}\hfill \end{array}\right]=\left[(1-{\nu }_M)\text{𝐈}+{\nu }_M\text{𝐒}\right]\text{𝐃}\left[\begin{array}{c}\hfill 𝐱\hfill \\ \hfill 𝐚\hfill \end{array}\right],$$ [30]

where $\text{𝐱}={(x_1,x_2,\mathrm{\dots },x_n)}^{\text{T}}$, $\text{𝐚}={(a_1,a_2,\mathrm{\dots },a_n)}^{\text{T}}$, I is the $2n\times 2n$ identity matrix,

$$\text{𝐃}=\text{diag}(\frac{w_1}{w},\frac{w_2}{w},\mathrm{\dots },\frac{w_n}{w},\frac{v_1}{v},\frac{v_2}{v},\mathrm{\dots },\frac{v_n}{v}),$$ [31]

and

$$\text{𝐒}=\left[\begin{array}{cc}\hfill 0\hfill & \hfill {\text{𝐈}}_n\hfill \\ \hfill {\text{𝐈}}_n\hfill & \hfill 0\hfill \end{array}\right],$$ [32]

where ${\text{𝐈}}_n$ is the $n\times n$ identity matrix. S is a reducible column-stochastic matrix, which can be rearranged as a diagonal block matrix, with $n$ $2\times 2$ blocks of $\left[\begin{array}{cc}\hfill 0\hfill & \hfill 1\hfill \\ \hfill 1\hfill & \hfill 0\hfill \end{array}\right]$ with associated submatrices of D given by $D_i=\text{diag}(\frac{w_i}{w},\frac{v_i}{v})$ for $i=1,2,\mathrm{\dots },n$. We assume that these are non-scalar matrices; that is, for each $i$, $\frac{w_i}{w}\ne \frac{v_i}{v}$. Otherwise, for some $i$, $x_i=a_i$ is independent of the migration, a case we exclude.

This equilibrium $({\text{𝐱}}^{\ast },\text{𝟎},{\text{𝐚}}^{\ast },\text{𝟎})$ exists and is internally stable (21). Its external stability is determined by ${\text{𝐋}}_{\text{ex}}^{\ast }$, where

$$\left[\begin{array}{c}\hfill \text{𝐲}\prime \hfill \\ \hfill \text{𝐛}\prime \hfill \end{array}\right]={\text{𝐋}}_{\text{ex}}^{\ast }\left[\begin{array}{c}\hfill 𝐲\hfill \\ \hfill \text{𝐛}\hfill \end{array}\right],\begin{array}{cc}𝐲={(y_1,y_2,\mathrm{\dots },y_n)}^{\text{T}},\hfill & 𝐲\prime ={(y_1^{\prime },y_2^{\prime },\mathrm{\dots },y_n^{\prime })}^{\text{T}},\hfill \\ \text{𝐛}={(b_1,b_2,\mathrm{\dots },b_n)}^{\text{T}},\hfill & \text{𝐛}\prime ={(b_1^{\prime },b_2^{\prime },\mathrm{\dots },b_n^{\prime })}^{\text{T}}\hfill \end{array}.$$ [33]

Thus, using Eqs. 25 and 27, ${\text{𝐋}}_{\text{ex}}^{\ast }$ is derived from

$$y_i^{\prime }=(1-{\nu }_m)\left[\frac{w_i{y}_i}{w^{\ast }}+r\sum _{j=1}^n\frac{w_i{w}_j}{{(w^{\ast })}^2}(x_i^{\ast }{y}_j-x_j^{\ast }{y}_i)\right]+{\nu }_m\left[\frac{v_i{b}_i}{v^{\ast }}+r\sum _{j=1}^n\frac{v_i{v}_j}{{(v^{\ast })}^2}(a_i^{\ast }{b}_j-a_j^{\ast }{b}_i)\right]$$ [34]

$$b_i^{\prime }=(1-{\nu }_m)\left[\frac{v_i{b}_i}{v^{\ast }}+r\sum _{j=1}^n\frac{v_i{v}_j}{{(v^{\ast })}^2}(a_i^{\ast }{b}_j-a_j^{\ast }{b}_i)\right]+{\nu }_m\left[\frac{w_i{y}_i}{w^{\ast }}+r\sum _{j=1}^n\frac{w_i{w}_j}{{(w^{\ast })}^2}(x_i^{\ast }{y}_j-x_j^{\ast }{y}_i)\right]$$ [35]

for $i=\mathrm{\hspace{0.17em}1},2,\mathrm{\dots },n$, where

$$w^{\ast }=\sum _{i=1}^n{w}_i{x}_i^{\ast },\hspace{1em}{v}^{\ast }=\sum _{i=1}^n{v}_i{a}_i^{\ast }.$$ [36]

Observe that when $\left[\begin{array}{c}\hfill 𝐲\hfill \\ \hfill 𝐛\hfill \end{array}\right]=\left[\begin{array}{c}\hfill {\text{𝐱}}^{\ast }\hfill \\ \hfill {𝐚}^{\ast }\hfill \end{array}\right]$ and ${\nu }_m={\nu }_M$, from Eq. 34 and 35, $\left[\begin{array}{c}\hfill 𝐲\prime \hfill \\ \hfill 𝐛\prime \hfill \end{array}\right]$ is given by

$$y_i^{\prime }=(1-{\nu }_m)\frac{w_i{x}_i^{\ast }}{w^{\ast }}+{\nu }_m\frac{v_i{a}_i^{\ast }}{v^{\ast }}$$

$$b_i^{\prime }=(1-{\nu }_m)\frac{v_i{a}_i^{\ast }}{v^{\ast }}+{\nu }_m\frac{w_i{x}_i^{\ast }}{w^{\ast }},$$ [37]

and using the equilibrium equations, Eq. 29 implies that $\text{𝐲}\prime ={\text{𝐱}}^{\ast }$, $\text{𝐛}\prime ={\text{a}}^{\ast }$ so that ${\text{𝐋}}_{\text{ex}}^{\ast }\left[\begin{array}{c}\hfill {\text{𝐱}}^{\ast }\hfill \\ \hfill {𝐚}^{\ast }\hfill \end{array}\right]=\left[\begin{array}{c}\hfill {\text{𝐱}}^{\ast }\hfill \\ \hfill {𝐚}^{\ast }\hfill \end{array}\right]$ for $0\le r\le \mathrm{\hspace{0.17em}1}$. When $r=0$, ${\text{𝐋}}_{\text{ex}}^{\ast }$ in Eqs. 34 and 35 reduce to

$$\begin{array}{cc}{y}_i^{\prime }=(1-{\nu }_m)\frac{w_i{y}_i}{w^{\ast }}+{\nu }_m\frac{v_i{b}_i}{v^{\ast }}\hfill & \\ & \\ b_i^{\prime }=(1-{\nu }_m)\frac{v_i{b}_i}{v^{\ast }}+{\nu }_m\frac{w_i{y}_i}{w^{\ast }}\hfill & \end{array}i=1,2,\mathrm{\dots },n.$$ [38]

Hence, when $r=\mathrm{\hspace{0.17em}0}$, ${\text{𝐋}}_{\text{ex}}^{\ast }$ can be written in matrix notation as

$${\mathbf{L}}_{\text{ex}}^{\ast }=\left[(1-{\nu }_m)\text{𝐈}+{\nu }_m\text{𝐒}\right]\text{𝐃},$$ [39]

where I, S, and D are as in Eqs. 31 and 32.

For $r=1$, from Eqs. 34 and 35, ${\text{𝐋}}_{\text{ex}}^{\ast }$ is (for $i=\mathrm{1,2},\mathrm{\dots },n$)

$$\begin{array}{c}{y}_i^{\prime }=(1-{\nu }_m)\frac{w_i{x}_i^{\ast }}{w^{\ast }}\sum _{j=1}^n\frac{w_j}{w^{\ast }}{y}_j+{\nu }_m\frac{v_i{a}_i^{\ast }}{v^{\ast }}\sum _{j=1}^n\frac{v_j}{v^{\ast }}{b}_j\\ b_i^{\prime }=(1-{\nu }_m)\frac{v_i{a}_i^{\ast }}{v^{\ast }}\sum _{j=1}^n\frac{v_j}{v^{\ast }}{b}_j+{\nu }_m\frac{w_i{x}_i^{\ast }}{w^{\ast }}\sum _{j=1}^n\frac{w_j}{w^{\ast }}{y}_j.\end{array}$$ [40]

Here we used $\frac{w_i{y}_i}{w^{\ast }}{\sum }_{j=1}^n\frac{w_j{x}_j^{\ast }}{w^{\ast }}=\frac{w_i{y}_i}{w^{\ast }}$ and $\frac{v_i{b}_i}{v^{\ast }}{\sum }_{j=1}^n\frac{v_j{a}_j^{\ast }}{v^{\ast }}=\frac{v_i{b}_i}{v^{\ast }}$ because $w^{\ast }={\sum }_{j=1}^n{w}_j{x}_j^{\ast }$ and $v^{\ast }={\sum }_{j=1}^n{v}_j{a}_j^{\ast }$. In matrix notation, when $r=1$, we can represent ${\text{𝐋}}_{\text{ex}}^{\ast }$ as

$${\text{𝐋}}_{\text{ex}}^{\ast }=\left[(1-{\nu }_m)\text{𝐐}+{\nu }_m\stackrel{\sim }{\text{𝐐}}\right]\text{𝐃},$$ [41]

where D is as in Eq. 31 but with $w=w^{*}$, $v=v^{*}$, and

$$\text{𝑸}=\left[\begin{array}{cc}\hfill {\text{𝐐}}_1\hfill & \hfill \text{𝟎}\hfill \\ \hfill \text{𝟎}\hfill & \hfill {\text{𝐐}}_2\hfill \end{array}\right],\stackrel{\sim }{\text{𝐐}}=\left[\begin{array}{cc}\hfill \text{𝟎}\hfill & \hfill {\text{𝐐}}_2\hfill \\ \hfill {\text{𝐐}}_1\hfill & \hfill \text{𝟎}\hfill \end{array}\right].$$ [42]

${\text{𝐐}}_1$ has all columns equal to ${[\frac{w_1{x}_1^{\ast }}{w^{\ast }},\frac{w_2{x}_2^{\ast }}{w^{\ast }},\mathrm{\dots },\frac{w_n{x}_n^{\ast }}{w^{\ast }}]}^{\text{T}}$ and ${\text{𝐐}}_2$ has all columns equal to ${[\frac{v_1{a}_1^{\ast }}{v^{\ast }},\frac{v_2{a}_2^{\ast }}{v^{\ast }},\mathrm{\dots },\frac{v_n{a}_n^{\ast }}{v^{\ast }}]}^{\text{T}}$.

Combining Eqs. 39 and 41, and because ${\text{𝐋}}_{\text{ex}}^{\ast }$ is linear in $r$ for $0\le r\le \mathrm{\hspace{0.17em}1}$, we obtain in general that ${\text{𝐋}}_{\text{ex}}^{\ast }=\text{𝐌}({\nu }_m,r)\text{𝐃}$, where

$$\text{𝐌}({\nu }_m,r)=(1-r)\left[(1-{\nu }_m)\text{𝐈}+{\nu }_m\text{𝐒}\right]+r\left[(1-{\nu }_m)\text{𝐐}+{\nu }_m\stackrel{\sim }{\text{𝐐}}\right].$$ [43]

Note that S, Q, and $\stackrel{\sim }{\text{𝐐}}$ are column-stochastic nonnegative matrices. Also, because $\text{𝐐𝐃}\left[\begin{array}{c}\hfill {\text{𝐱}}^{\ast }\hfill \\ \hfill {𝐚}^{\ast }\hfill \end{array}\right]=\text{𝐃}\left[\begin{array}{c}\hfill {\text{𝐱}}^{\ast }\hfill \\ \hfill {𝐚}^{\ast }\hfill \end{array}\right]$ and ${\text{𝐋}}_{\text{ex}}^{\ast }\left[\begin{array}{c}\hfill {\text{𝐱}}^{\ast }\hfill \\ \hfill {𝐚}^{\ast }\hfill \end{array}\right]=\left[\begin{array}{c}\hfill {\text{𝐱}}^{\ast }\hfill \\ \hfill {𝐚}^{\ast }\hfill \end{array}\right]$ when ${\nu }_m={\nu }_M$, for all $0\le r\le \mathrm{\hspace{0.17em}1}$, we have

$$\left[(1-{\nu }_M)\text{𝐈}+{\nu }_M\text{𝐒}\right]\text{𝐃}\left[\begin{array}{c}\hfill {\text{𝐱}}^{\ast }\hfill \\ \hfill {𝐚}^{\ast }\hfill \end{array}\right]=\left[\begin{array}{c}\hfill {\text{𝐱}}^{\ast }\hfill \\ \hfill {𝐚}^{\ast }\hfill \end{array}\right],\left[(1-{\nu }_M)\text{𝐐}+{\nu }_M\stackrel{\sim }{\text{𝐐}}\right]\text{𝐃}\left[\begin{array}{c}\hfill {\text{𝐱}}^{\ast }\hfill \\ \hfill {𝐚}^{\ast }\hfill \end{array}\right]=\left[\begin{array}{c}\hfill {\text{𝐱}}^{\ast }\hfill \\ \hfill {𝐚}^{\ast }\hfill \end{array}\right].$$ [44]

The Recombination Modification Model.

Consider a diploid population and a character determined by two major loci with alleles $A_1,A_2,\mathrm{\dots },A_g$ at the first locus and $B_1,B_2,\mathrm{\dots },B_h$ at the second one. These loci are subject to viability selection. A modifier locus, which is linked to the major locus, has two possible alleles $M$ and $m$ and is not under direct selection; its only function is to determine the recombination rate between the two major loci. There are then three classes of genotypes

$$\frac{A_i{B}_{j}M}{A_k{B}_{l}M},\frac{A_i{B}_{j}M}{A_k{B}_{l}m},\frac{A_i{B}_{j}m}{A_k{B}_{l}m},$$ [45]

all of which have the same positive fitness, which we denote by $w(i,j;k,l)$.

With three loci there are two recombination intervals ${\mathrm{\Delta }}_1$ and ${\mathrm{\Delta }}_2$ as shown below:

[46]

Recombination (crossover) may occur in either, both, or neither of the two intervals ${\mathrm{\Delta }}_1$ and ${\mathrm{\Delta }}_2$. Thus, there are four possible crossover events,

$$(0,0),\hspace{1em}(0,1),\hspace{1em}(1,0),\hspace{1em}(1,1),$$ [47]

where $(\mathrm{0,0})$ denotes no crossover in either ${\mathrm{\Delta }}_1$ or ${\mathrm{\Delta }}_2$, $(\mathrm{0,1})$ denotes no crossover in ${\mathrm{\Delta }}_1$ but crossover in ${\mathrm{\Delta }}_2$, etc. There are four corresponding crossover probabilities,

$$C(\mathrm{0,0}),\hspace{1em}C(\mathrm{0,1}),\hspace{1em}C(\mathrm{1,0}),\hspace{1em}C(\mathrm{1,1}),$$ [48]

with $C(\mathrm{0,0})+C(\mathrm{0,1})+C(\mathrm{1,0})+C(\mathrm{1,1})=\mathrm{\hspace{0.17em}1}$. The modifier locus determines the crossover probabilities according to the genotypes $MM$, $Mm$, $mm$, denoted 1, 2, 3, respectively. Thus, we have three sets of crossover probabilities,

$$C^i(\mathrm{0,0}),\hspace{1em}{C}^i(\mathrm{0,1}),\hspace{1em}{C}^i(\mathrm{1,0}),\hspace{1em}{C}^i(\mathrm{1,1}),\hspace{1em}i=1,2,3.$$ [49]

Note that

$$r_i=C^i(1,0)+C^i(1,1)\hspace{1em}i=1,2,3$$ [50]

is the recombination rate between the two major loci in the presence of genotype $i$ at the modifier locus, and

$$r=C^i(0,1)+C^i(1,1)\hspace{1em}i=1,2,3$$ [51]

is the recombination rate between the major loci and the modifier locus; $r$ is assumed to take the same value independently of the genotype at the modifier locus.

At each generation the gametes $A_i{B}_{j}M$ and $A_i{B}_{j}m$ undergo random union, viability selection, recombination, and segregation (in this order) to form the new generation. Let $x_{ij}$ denote the frequency of gamete $A_i{B}_{j}M$, $y_{ij}$ that of gamete $A_i{B}_{j}m$ in the present generation, and $x_{ij}^{\prime }$ and $y_{ij}^{\prime }$ be the corresponding frequencies in the next generation. The evolution is determined by the transformation from $(\text{𝒙},\text{𝒚})$ to $(\text{𝐱}\prime ,\text{𝐲}\prime )$.

For the external stability of equilibria on the boundary, where only $M$ is present, to the introduction of $m$ at the modifier locus, we concentrate on the following transformations. We first describe the transformation T from $(\text{𝐱},\text{𝟎})$ to $(\text{x}\prime ,\text{𝟎})$ on the boundary where only $M$ is present to find any equilibrium $({\text{𝐱}}^{\ast },\text{𝟎})$ on this boundary. Then, assuming that $({\text{𝐱}}^{\ast },\text{𝟎})$ exists and is internally stable, we characterize the external local stability of $({\text{𝐱}}^{\ast },\text{𝟎})$ in terms of the linear transformation ${\text{𝐋}}_{\text{ex}}^{\ast }$. This will give us the conditions under which allele $m$ will invade the population fixed on $M$ at $({\text{𝐱}}^{\ast },\text{𝟎})$.

To these ends we need two segregation tables: Table 1, for T, gives the probabilities that the gamete $A_i{B}_{j}M$ is produced by different genotypes carrying only the $M$ allele at the modifier locus. Table 2, for ${\text{𝐋}}_{\text{ex}}^{\ast }$, gives the probabilities of obtaining the gamete $A_i{B}_{j}m$ from different genotypes that are heterozygous $Mm$. We are not concerned with genotype $mm$ during the initial increase (or decrease) of $m$ near $({\text{𝐱}}^{\ast },\text{𝟎})$.

Table 1.

Recombination with $M/M$

Genotype	$A_i{B}_{j}M$ probability
$\frac{A_i{B}_{j}M}{A_k{B}_{l}M}\hspace{0.17em}k\ne i,l\ne j$	$C^1\left(0\text{,}\mathrm{\hspace{0.17em}0}\right)+C^1\left(0\text{,}\mathrm{\hspace{0.17em}1}\right)$
$\frac{A_i{B}_{j}M}{A_i{B}_{l}M}\hspace{0.17em}l\ne j$	1
$\frac{A_i{B}_{j}M}{A_k{B}_{j}M}\hspace{0.17em}k\ne i$	1
$\frac{A_i{B}_{j}M}{A_i{B}_{j}M}$	1
$\frac{A_i{B}_{l}M}{A_k{B}_{j}M}\hspace{0.17em}k\ne i,l\ne j$	$C^1\left(1\text{,}\mathrm{\hspace{0.17em}0}\right)+C^1\left(1\text{,}1\right)$

Table 2.

Recombination with $M/m$

Genotype	$A_i{B}_{j}m$ probability
$\frac{A_i{B}_{j}m}{A_k{B}_{l}M}\hspace{0.17em}k\ne i\text{,}l\ne j$	$C^2\left(0\text{,}\mathrm{\hspace{0.17em}0}\right)$
$\frac{A_i{B}_{j}M}{A_k{B}_{l}m}\hspace{0.17em}k\ne i\text{,}l\ne j$	$C^2\left(0\text{,}\mathrm{\hspace{0.17em}1}\right)$
$\frac{A_i{B}_{j}m}{A_i{B}_{l}M}\hspace{0.17em}l\ne j$	$C^2\left(0\text{,}\mathrm{\hspace{0.17em}0}\right)+C^2\left(1\text{,}\mathrm{\hspace{0.17em}0}\right)$
$\frac{A_i{B}_{j}M}{A_i{B}_{l}m}\hspace{0.17em}l\ne j$	$C^2\left(0\text{,}\mathrm{\hspace{0.17em}0}\right)1+C^2\left(1\text{,}\mathrm{\hspace{0.17em}1}\right)$
$\frac{A_i{B}_{j}m}{A_k{B}_{j}M}\hspace{0.17em}k\ne i$	$C^2\left(0\text{,}\mathrm{\hspace{0.17em}0}\right)+C^2\left(1\text{,}\mathrm{\hspace{0.17em}1}\right)$
$\frac{A_i{B}_{j}M}{A_k{B}_{j}m}\hspace{0.17em}k\ne i$	$C^2\left(0\text{,}\mathrm{\hspace{0.17em}1}\right)+C^2\left(1\text{,}\mathrm{\hspace{0.17em}0}\right)$
$\frac{A_i{B}_{j}m}{A_i{B}_{j}M}\hspace{0.17em}k\ne i\text{,}l\ne j$	1
$\frac{A_i{B}_{l}m}{A_k{B}_{j}M}\hspace{0.17em}k\ne i\text{,}l\ne j$	$C^2\left(1\text{,}\mathrm{\hspace{0.17em}1}\right)$
$\frac{A_i{B}_{l}M}{A_k{B}_{j}m}\hspace{0.17em}k\ne i\text{,}l\ne j$	$C^2\left(1\text{,}\mathrm{\hspace{0.17em}0}\right)$

From Table 1 we derive the transformation T from $(\text{𝐱},\text{𝟎})$ to $(\text{𝐱}\prime ,\text{𝟎})$, namely

$$w{x}_{ij}^{\prime }=\left[C^1(\mathrm{0,0})+C^1(\mathrm{0,1})\right]{\displaystyle \sum _{k,l}}w(i,j;k,l)x_{ij}{x}_{kl}+\left[C^1(\mathrm{1,0})+C^1(\mathrm{1,1})\right]{\displaystyle \sum _{k,l}}w(i,l;k,j)x_{il}{x}_{kj},$$ [52]

for all $i=\mathrm{\hspace{0.17em}1},2,\mathrm{\dots },g$, $j=\mathrm{\hspace{0.17em}1},2,\mathrm{\dots },h$. From Eq. 50 $C^1(\mathrm{1,0})+C^1(\mathrm{1,1})=r_1$, so we can rewrite Eq. 52 as

$$w{x}_{ij}^{\prime }=(1-r_1){\displaystyle \sum _{k,l}}w(i,j;k,l)x_{ij}{x}_{kl}+r_1{\displaystyle \sum _{k,l}}w(i,l;k,j)x_{il}{x}_{kj}.$$ [53]

Let $w(i,j)={\sum }_{k,l}w(i,j;k,l)x_{kl}$ be the marginal fitness of chromosomes carrying $A_i{B}_j$. Then Eq. 53 can be written (for all $i$ and $j$) as

$$w{x}_{ij}^{\prime }=(1-r_1)w(i,j)x_{ij}+r_1{\displaystyle \sum _{k,l}}w(i,l;k,j)x_{il}{x}_{kj},$$ [54]

where $w={\sum }_{i,j,k,l}w(i,j;k,l)x_{ij}{x}_{kl}={\sum }_{i,j}w(i,j)x_{ij}$ is the mean fitness.

Because $({\text{𝐱}}^{\ast },\text{𝟎})$ is an equilibrium on the boundary where only $M$ is present, ${\text{𝐱}}^{\ast }$ satisfies the equations

$$w^{\ast }{x}_{ij}^{\ast }=(1-r_1)w^{\ast }(i,j)x_{ij}^{\ast }+r_1{\displaystyle \sum _{k,l}}w(i,l;k,j)x_{il}^{\ast }{x}_{kj}^{\ast },$$ [55]

where $w^{\ast }(i,j)={\sum }_{k,l}w(i,j;k,l)x_{kl}^{\ast }$ and $w^{\ast }={\sum }_{i,j}{w}^{\ast }(i,j)x_{ij}^{\ast }$.

In the same way, Table 2 determines the external local stability given by the transformation $\text{y}\prime ={\text{𝐋}}_{\text{ex}}^{\ast }\text{y}$, where

$$w^{\ast }{\text{y}}_{ij}^{\prime }=C^2(\mathrm{0,0}){\displaystyle \sum _{k,l}}w(i,j;k,l)y_{ij}{x}_{kl}^{\ast }+C^2(\mathrm{0,1}){\displaystyle \sum _{k,l}}w(i,j;k,l)x_{ij}^{\ast }{y}_{kl}+C^2(\mathrm{1,0}){\displaystyle \sum _{k,l}}w(i,l;k,j)x_{il}^{\ast }{y}_{kj}+C^2(\mathrm{1,1}){\displaystyle \sum _{k,l}}w(i,l;k,j)y_{il}{x}_{kj}^{\ast }.$$ [56]

Observation 1.

When $r_2=r_1$, ${\text{𝐋}}_{ex}^{\ast }{\text{𝐱}}^{\ast }={\text{𝐱}}^{\ast }$ for all $0\le r\le \mathrm{\hspace{0.17em}1}$.

In fact, substituting ${\text{𝐱}}^{\ast }$ for $𝐲$ in Eq. 56 we have

$${\left({\text{𝑤}}^{\ast }{\text{L}}_{\text{ex}}^{\ast }{x}^{\ast }\right)}_{ij}=\left[C^2(\mathrm{0,0})+C^2(\mathrm{0,1})\right]{\displaystyle \sum _{k,l}}w(i,j;k,l)x_{ij}^{\ast }{x}_{kl}^{\ast }+\left[C^2(\mathrm{1,0})+C^2(\mathrm{1,1})\right]{\displaystyle \sum _{k,l}}w(i,l;k,j)x_{il}^{\ast }{x}_{kj}^{\ast }.$$ [57]

However, $C^2(\mathrm{1,0})+C^2(\mathrm{1,1})=r_2$ and $C^2(\mathrm{0,0})+C^2(\mathrm{0,1})=1-r_2$. Therefore, if $r_2=r_1$, from Eq. 53, as $\text{𝐓}{\text{𝐱}}^{\ast }={\text{𝐱}}^{\ast }$, Eq. 57 reduces to ${(w^{\ast }{\text{𝐋}}_{\text{ex}}^{\ast }{\text{𝐱}}^{\ast })}_{ij}=w^{\ast }{x}_{ij}^{\ast }$. Hence, when $r_2=r_1$, ${\text{𝐋}}_{\text{ex}}^{\ast }{\text{𝐱}}^{\ast }={\text{𝐱}}^{\ast }$ for all $0\le r\le \mathrm{\hspace{0.17em}1}$.

In what follows we assume that crossover events in the two intervals ${\mathrm{\Delta }}_1$ and ${\mathrm{\Delta }}_2$ are independent; that is, for $i=\mathrm{1,2,3}$,

$$\begin{array}{ccc}{C}^i(\mathrm{0,0})=(1-r_i)(1-r),\hfill & C^i(\mathrm{0,1})=(1-r_i)r,\hfill & \\ C^i(\mathrm{1,0})=r_i(1-r),\hfill & C^i(\mathrm{1,1})=r_{i}r.\hfill & \end{array}$$ [58]

In classical terms, we assume no interference in ${\mathrm{\Delta }}_1$ and ${\mathrm{\Delta }}_2$. Under this assumption, we make the following observation.

Observation 2.

With no recombination interference, the external stability matrix ${\text{𝐋}}_{\text{ex}}^{\ast }$ can be represented as ${\text{𝐋}}_{\text{ex}}^{\ast }=\text{𝐌}(r_2,r)\text{𝐃},$ where D is a diagonal matrix

$$\text{𝐃}=\text{diag}[\frac{w^{\ast }(\mathrm{1,1})}{w^{\ast }},\mathrm{\dots },\frac{w^{\ast }(1,h)}{w^{\ast }},\mathrm{\dots },\frac{w^{\ast }(g,1)}{w^{\ast }},\mathrm{\dots },\frac{w^{\ast }(g,h)}{w^{\ast }}],$$ [59]

and

$$\text{𝐌}(r_2,r)=(1-r)\left[(1-r_2)\text{𝐈}+r_2\text{𝐒}\right]+r\left[(1-r_2)\text{𝐐}+r_2\stackrel{\sim }{\text{𝐒}}\right].$$ [60]

Here, I is the identity matrix, S and $\stackrel{\sim }{\text{𝐒}}$ are nonnegative column-stochastic matrices, and Q is a positive column-stochastic matrix. In addition,

$$\left[(1-r_1)\text{𝐈}+r_1\text{𝐒}\right]\text{𝐃}{\text{𝐱}}^{\ast }={\text{𝐱}}^{\ast },\hspace{1em}\left[(1-r_1)\text{𝐐}+r_1\stackrel{\sim }{\text{𝐒}}\right]\text{𝐃}{\text{𝐱}}^{\ast }={\text{𝐱}}^{\ast },$$ [61]

and

$$\text{𝐐𝐃}{\text{𝐱}}^{\ast }={\text{𝐱}}^{\ast }.$$ [62]

The proof is given in Proof of Observation 2.

Note that the diagonal matrices ${\text{𝐃}}_j$, where

$${\mathbf{D}}_j=\text{diag}[\frac{w^{\ast }(1,j)}{w^{\ast }},\frac{w^{\ast }(2,j)}{w^{\ast }},\mathrm{\dots },\frac{w^{\ast }(g,j)}{w^{\ast }}],j=\mathrm{1,2},\mathrm{\dots },h,$$ [63]

are nonscalar matrices. In fact, the equilibrium equations Eq. 55 can be written as

$$w^{\ast }{x}_{ij}^{\ast }=w^{\ast }(i,j)x_{ij}^{\ast }-r_1{D}_{ij}^{\ast },$$ [64]

and it is easily seen that ${\sum }_i{D}_{ij}^{\ast }=\mathrm{\hspace{0.17em}0}$. Indeed, if ${\text{𝐃}}_j$ is a scalar matrix, then $w^{\ast }(1,j)=w^{\ast }(2,j)=\mathrm{\cdots }=w^{\ast }(g,j)=c$, and from Eq. 64 we would have $w^{\ast }{\sum }_i{x}_{ij}^{\ast }=c{\sum }_i{x}_{ij}^{\ast }$. Because all $x_{ij}^{\ast }$ are positive, $c=w^{\ast }$, and Eq. 64 implies that $D_{ij}^{\ast }=\mathrm{\hspace{0.17em}0}$ for all $i$ and $j$ so that $x_{ij}^{\ast }$ does not depend on the modified recombination parameter $r_1$. We exclude such cases and can assume that all of the ${\text{𝐃}}_j$ matrices in Eq. 63 are nonscalar.

Proof of Observation 2

We start with $r=0$, in which case $C^2(\mathrm{0,1})=C^2(\mathrm{1,1})=0$, $C^2(\mathrm{0,0})=1-r_2$, and $C^2(\mathrm{1,0})=r_2$. Then from Eq. 57, ${\text{𝐋}}_{\text{ex}}^{\ast }$ is given by

$$w^{\ast }{y}_{ij}=(1-r_2){\displaystyle \sum _{k,l}}w(i,j;k,l)x_{kl}^{\ast }{y}_{ij}+r_2{\displaystyle \sum _{k,l}}w(i,l;k,j)x_{il}^{\ast }{y}_{kj}$$ [S1]

for all $i,j$. As ${\sum }_{k,l}w(i,j;k,l)x_{kl}^{\ast }=w^{\ast }(i,j)$, we can write Eq. S1 as

$$y_{ij}^{\prime }=(1-r_2)\frac{w^{\ast }(i,j)}{w^{\ast }}{y}_{ij}+r_2{\displaystyle \sum _k}\left({\displaystyle \sum _l}\frac{w(i,l;k,j)x_{il}^{\ast }}{w^{\ast }(k,j)}\right)\frac{w^{\ast }(k,j)}{w^{\ast }}{y}_{kj}.$$ [S2]

Thus, when $r=0$, ${\text{𝐋}}_{\text{ex}}^{\ast }=\text{𝐌}(r_2,0)\text{𝐃}$, where

$$\text{𝐌}(r_2,0)=(1-r_2)\text{𝐈}+r_2\text{𝐒}.$$ [S3]

I is the $gh\times gh$ identity matrix, and S is the $gh\times gh$ block matrix having $h$ blocks, each a $g\times g$ matrix corresponding to the $g$ gametes $A_1{B}_j,A_2{B}_j,\mathrm{\dots },A_g{B}_j$ for $j=1,2,\mathrm{\dots },h$, with $(i,k)$ elements ${\sum }_l\frac{w(i,l;k,j)x_{il}^{\ast }}{w^{\ast }(k,j)}$. Thus, $S$ is nonnegative, and it is column stochastic because

$$\sum _{i=1}^g\left(\sum _{l=1}^h\frac{w(i,l;k,j)x_{il}^{\ast }}{w^{\ast }(k,j)}\right)=\frac{w^{\ast }(k,j)}{w^{\ast }(k,j)}=1.$$ [S4]

When $r=1$, $C^2(\mathrm{0,0})=C^2(\mathrm{1,0})=0$, $C^2(\mathrm{0,1})=1-r_2$, and $C^2(\mathrm{1,1})=r_2$, and ${\text{𝐋}}_{\text{ex}}^{\ast }$ from Eq. 56 becomes

$$w^{\ast }{y}_{ij}^{\prime }=(1-r_2){\displaystyle \sum _{k,l}}w(i,j;k,l)x_{ij}^{\ast }{y}_{kl}+r_2{\displaystyle \sum _{k,l}}w(i,l;k,j)x_{kj}^{\ast }{y}_{il},$$ [S5]

which we rewrite as

$$y_{ij}^{\prime }=(1-r_2){\displaystyle \sum _{k,l}}\frac{w(i,j;k,l)x_{ij}^{\ast }}{w^{\ast }(k,l)}\frac{w^{\ast }(k,l)}{w^{\ast }}{y}_{kl}+r_2{\displaystyle \sum _l}\left({\displaystyle \sum _k}\frac{w(i,l;k,j)x_{kj}^{\ast }}{w^{\ast }(i,l)}\right)\frac{w^{\ast }(i,l)}{w^{\ast }}{y}_{il}.$$ [S6]

Thus, when $r=1$, ${\text{𝐋}}_{\text{ex}}^{\ast }=\text{𝐌}(r_2,1)\text{𝐃}$, where

$$\text{𝐌}(r_2,1)=(1-r_2)\text{𝐐}+r_2\stackrel{\sim }{\text{𝐒}},$$ [S7]

Q is a $gh\times gh$ positive matrix whose $(i,j;k,l)$ entry is $\frac{w(i,j;k,l)x_{ij}^{\ast }}{w^{\ast }(k,l)}$, and it is column stochastic because

$${\displaystyle \sum _{l,j}}\frac{w(i,j;k,l)x_{ij}^{\ast }}{w^{\ast }(k,l)}=\frac{w^{\ast }(k,l)}{w^{\ast }(k,l)}=1.$$ [S8]

$\stackrel{\sim }{\text{𝐒}}$ is a $gh\times gh$ block matrix having $g$ $(h\times h)$ blocks corresponding to the $h$ gametes $A_i{B}_1,A_i{B}_2,\mathrm{\dots },A_i{B}_h$ for $i=1,2,\mathrm{\dots },g$, with $(l,j)$ elements ${\sum }_k\frac{w(i,l;k,j)x_{kj}^{\ast }}{w^{\ast }(i,l)}$. So $\stackrel{\sim }{\text{𝐒}}$ is nonnegative and column stochastic because

$${\displaystyle \sum _j}{\displaystyle \sum _k}\frac{w(i,l;k,j)x_{kj}^{\ast }}{w^{\ast }(i,l)}=\frac{w^{\ast }(i,l)}{w^{\ast }(i,l)}=1.$$ [s9]

Finally, because ${\text{𝐋}}_{\text{ex}}^{\ast }$ is linear in $r$, we conclude that ${\text{𝐋}}_{\text{ex}}^{\ast }=\text{𝐌}(r_2,r)\text{𝐃}$, where $\text{𝐌}(r_2,r)$ is given in Eq. 60. By Observation 1, ${\text{𝐋}}_{\text{ex}}^{\ast }{\text{𝐱}}^{\ast }={\text{𝐱}}^{\ast }$ when $r_2=r_1$ for all $0\le r\le 1$, and hence Eq. 62 follows from Eq. 61. Also, from Eq. S5 and the definition of Q, we have

$${\left(\text{𝐐𝐃}{\text{𝐱}}^{\ast }\right)}_{ij}={\displaystyle \sum _{k,l}}\frac{w(i,j;k,l)}{w^{\ast }}{x}_{ij}^{\ast }{x}_{kl}^{\ast }=\frac{w^{\ast }(i,j)}{w^{\ast }}{x}_{ij}^{\ast },$$ [S10]

which proves that $\text{𝐐𝐃}{\text{𝐱}}^{\ast }=\text{𝐃}{\text{𝐱}}^{\ast }$.

A Unified Mathematical Structure for Invasion

Although the three models for modification of mutation, migration, and recombination are different in structure, they share similar mathematical representations for their equilibrium equations and external stability matrices.

In fact, if $\alpha$ is the modified parameter (mutation rate, migration rate, or recombination rate) and its value with the $M$ allele fixed at the modifier locus is ${\alpha }^{\ast }$, then the associated boundary equilibrium ${\text{𝐱}}^{\ast }$ satisfies the equation

$$\left[(1-{\alpha }^{\ast })\text{𝐈}+{\alpha }^{\ast }\text{𝐒}\right]\text{𝐃}{\text{𝐱}}^{\ast }={\text{𝐱}}^{\ast },$$ [65]

where I is the identity matrix, S is a nonnegative column-stochastic matrix, and D is a positive nonscalar diagonal matrix. S is an irreducible matrix in the mutation modification models. In the recombination and migration modification cases, S is a reducible matrix of irreducible blocks, and each part of D associated with each block of S is a nonscalar matrix.

In all three modification models the external stability matrix ${\text{𝐋}}_{\text{ex}}^{\ast }$ has a similar representation: ${\text{𝐋}}_{\text{ex}}^{\ast }=\text{𝐌}(\alpha ;r)\text{𝐃}$, where

$$\text{𝐌}(\alpha ;r)=(1-r)\left[(1-\alpha )\text{𝐈}+\alpha \text{𝐒}\right]+r\left[(1-\alpha )\text{𝐐}+\alpha \stackrel{\sim }{\text{𝐒}}\right].$$ [66]

Here $r$ is the recombination rate between the major locus (loci) and the modifier locus. In Eq. 66, Q and $\stackrel{\sim }{\text{𝐒}}$ are nonnegative column-stochastic matrices and for $0<\alpha <\mathrm{\hspace{0.17em}1}$, $(1-\alpha )\text{𝐐}+\alpha \stackrel{\sim }{\text{𝐒}}$ is a positive matrix. In addition, with ${\alpha }^{\ast }$ produced by the modifier genotype $MM$,

$$\left[(1-{\alpha }^{\ast })\text{𝐈}+{\alpha }^{\ast }\text{𝐒}\right]\text{𝐃}{\text{𝐱}}^{\ast }={\text{𝐱}}^{\ast },\left[(1-{\alpha }^{\ast })\text{𝐐}+{\alpha }^{\ast }\stackrel{\sim }{\text{𝐒}}\right]\text{𝐃}{\text{𝐱}}^{\ast }={\text{𝐱}}^{\ast };$$ [67]

therefore $\text{𝐌}({\alpha }^{\ast };r)\text{𝐃}{\text{𝐱}}^{\ast }={\text{𝐱}}^{\ast }$ for all $r$, and also

$$\text{𝐐𝐃}{\text{𝐱}}^{\ast }=\text{𝐃}{\text{𝐱}}^{\ast }.$$ [68]

Our goal is to determine when the equilibrium ${\text{𝐱}}^{\ast }$, which we assume exists and is internally stable, is externally stable. Mathematically we want to characterize the spectral radius of ${\text{𝐋}}_{\text{ex}}^{\ast }$ as a function of the modified parameter $\alpha$. Because ${\text{𝐋}}_{\text{ex}}^{\ast }$ is a nonnegative matrix, by the Perron–Frobenius theory the spectral radius of ${\text{𝐋}}_{\text{ex}}^{\ast }$ is its largest positive eigenvalue. Moreover, as ${\text{𝐋}}_{\text{ex}}^{\ast }{\text{𝐱}}^{\ast }={\text{𝐱}}^{\ast }$ when $\alpha ={\alpha }^{\ast }$, the spectral radius of ${\text{𝐋}}_{\text{ex}}^{\ast }$ is 1. Thus, we want to find the spectral radius of ${\text{𝐋}}_{\text{ex}}^{\ast }$ when $\alpha <{\alpha }^{\ast }$ and when $\alpha >{\alpha }^{\ast }$.

The mathematical tools for analyzing the spectral radius of matrices like ${\text{𝐋}}_{\text{ex}}^{\ast }$ were developed by Karlin (14) in his investigation of migration models and later extended by Altenberg (20, 22). We review these tools in the next section.

Mathematical Tools

Karlin (14) proved the following theorem.

Theorem 1.

Karlin’s theorem: Let S be an arbitrary nonnegative irreducible column-stochastic matrix and consider the family of matrices

$$\text{𝐌}(\alpha )=(1-\alpha )\text{𝐈}+\alpha \text{𝐒},\hspace{1em}\alpha >0.$$ [69]

Then, for any diagonal nonscalar matrix D with positive terms on the diagonal, the spectral radius of $\text{𝐌}(\alpha )\text{𝐃}$ is strictly decreasing as $\alpha$ increases.

Observe that in our modification models, when $r=\mathrm{\hspace{0.17em}0}$, ${\text{𝐋}}_{\text{ex}}^{\ast }$ is $\left[(1-\alpha )\text{𝐈}+\alpha \text{𝐒}\right]\text{𝐃}$, and we can apply Karlin’s theorem provided S and D satisfy the conditions of the theorem. Indeed, with mutation modification, S and D satisfy these conditions, and because the spectral radius of ${\text{𝐋}}_{\text{ex}}^{\ast }$ is 1 when $\alpha ={\alpha }^{\ast }$, by the theorem the spectral radius of ${\text{𝐋}}_{\text{ex}}^{\ast }$ is less than 1 if $\alpha >{\alpha }^{\ast }$ and larger than 1 if $\alpha <{\alpha }^{\ast }$. Thus, when $r=0$, there is selection in favor of reduced mutation rates, and the reduction principle applies for mutation modification. For recombination or migration modification, S can be a reducible matrix and Karlin’s theorem does not yield strict monotonicity of the spectral radius with respect to $\alpha$ without additional assumptions on D. Moreover, Karlin’s theorem cannot be directly applied to the case when $r>\mathrm{\hspace{0.17em}0}$ or to the case where in Eq. 69 $\alpha >\mathrm{\hspace{0.17em}1}$ and $\text{𝐌}(\alpha )$ has negative entries on the diagonal and nonnegative entries elsewhere. This is exactly the situation in which Altenberg (20, 22) generalized Karlin’s theorem.

An essentially nonnegative matrix [also referred to as a Metzler–Leontief (ML) matrix] is a square real matrix all of whose off-diagonal elements are nonnegative. The spectral bound of a matrix A, denoted by $\sigma (\text{𝐀})$, is defined as $\sigma (\text{𝐀})={\max }_i\{\text{Re}({\lambda }_i)\}$, where $\{{\lambda }_i\}$ are the eigenvalues of A and $\text{Re}(z)$ is the real part of the complex number $z$.

Let $\text{𝐁}=[b_{ij}]$ be an ML matrix where $b_{ij}\ge \mathrm{\hspace{0.17em}0}$ for all $i\ne j$. We summarize some of the properties of ML matrices: (i) $\sigma (\text{𝐁})$ is an eigenvalue of B. (ii) If $\text{𝐁}\text{𝐳}=\lambda \text{𝐳}$ and z is a positive vector, then $\lambda =\sigma (\text{𝐁})$. (iii) If B is a nonnegative matrix, then $\sigma (\text{𝐁})=\rho (\text{𝐁})$, the spectral radius of $𝐁$.

Altenberg (20) proved the following generalization of Karlin’s theorem.

Theorem 2.

Let U be an ML matrix that is irreducible and column stochastic. Let $\text{𝐂}(\beta )$ be the family of matrices

$$\text{𝐂}(\beta )=(1-\beta )\text{𝐈}+\beta \text{𝐔},\hspace{1em}\beta >0.$$ [70]

Then, for any nonscalar positive diagonal matrix D, the spectral bound $\sigma \left[\text{𝐂}(\beta )\text{𝐃}\right]$ is strictly decreasing as $\beta$ increases.

The case where U is reducible was also analyzed by Altenberg (20). Specifically, if U is a reducible column-stochastic ML matrix of the form

$$\text{𝐔}=\left[\begin{array}{cccc}\hfill {\text{𝐔}}_1\hfill & \hfill 0\hfill & \hfill \mathrm{\dots }\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill {\text{𝐔}}_2\hfill & & \hfill 0\hfill \\ \hfill \mathrm{⋮}\hfill & & \hfill \mathrm{\ddots }\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill {\text{𝐔}}_l\hfill \end{array}\right],$$ [71]

then the previous result holds provided each of the submatrices ${\text{𝐃}}_1,{\text{𝐃}}_2,\mathrm{\dots },{\text{𝐃}}_l$ of D associated with each of the blocks ${\text{𝐔}}_1,{\text{𝐔}}_2,\mathrm{\dots },{\text{𝐔}}_l$, respectively, are nonscalar matrices.

The Unified Reduction Principle

Theorem 3.

Consider a population with a multiallelic major locus and a biallelic (with alleles $M$ and $m$) modifier locus and a stable equilibrium ${\text{𝐱}}^{\ast }$, where only the $M$ modifier allele is present, producing the mutation, migration, or recombination rate ${\alpha }^{\ast }$. Then ${\text{𝐱}}^{\ast }$ is stable to the introduction of the $m$ allele at the modifier locus, with associated rate $\alpha$, if $\alpha >{\alpha }^{\ast }$, and it is unstable if $\alpha <{\alpha }^{\ast }$.

Proof: Because ${\text{𝐱}}^{\ast }$ is internally stable when only $M$ is present, its external stability to introduction of $m$ is determined by the spectral radius of ${\text{𝐋}}_{\text{ex}}^{\ast }$, where ${\text{𝐋}}_{\text{ex}}^{\ast }=\text{𝐌}(\alpha ;r)\text{𝐃}$. $\text{𝐌}(\alpha ;r)$ is given in Eq. 66, and $r$ is the recombination rate between the major locus and the modifier locus.

When $r=0$, we have ${\text{𝐋}}_{\text{ex}}^{\ast }=\text{𝐌}(\alpha ;0)\text{𝐃}$ with

$$\text{𝐌}(\alpha ,0)=(1-\alpha )\text{𝐈}+\alpha \text{𝐒},\hspace{1em}0<\alpha <1,$$ [72]

where S is nonnegative, column stochastic, and irreducible in the case of mutation modification. D is a positive diagonal nonscalar matrix. In the recombination and migration modification model, S is a reducible block matrix, and the associated submatrices of D are positive diagonal nonscalar matrices.

Thus, following Karlin and Altenberg, the spectral radius of $\text{𝐌}(\alpha ;0)\text{𝐃}$ in Eq. 72 is a decreasing function of $\alpha$. (It is a nonnegative matrix for $0<\alpha <\mathrm{\hspace{0.17em}1}$.) However, the spectral radius of ${\text{𝐋}}_{\text{ex}}^{\ast }$ is 1 when $\alpha ={\alpha }^{\ast }$. Therefore, when $r=0$,

$${\alpha }^{\ast }>\alpha ⟹\rho ({\text{𝐋}}_{\text{ex}}^{\ast })<1,\hspace{1em}{\alpha }^{\ast }<\alpha ⟹\rho ({\text{𝐋}}_{\text{ex}}^{\ast })>1.$$ [73]

Hence, when $r=\mathrm{\hspace{0.17em}0}$, ${\text{𝐱}}^{\ast }$ is externally stable when $\alpha >{\alpha }^{\ast }$ and unstable when $\alpha <{\alpha }^{\ast }$ and there is selection in favor of a modifier allele with smaller $\alpha$.

If $r>\mathrm{\hspace{0.17em}0}$ and ${\text{𝐋}}_{\text{ex}}^{\ast }=\text{𝐌}(\alpha ;r)\text{𝐃}$, where

$$\text{𝐌}(\alpha ;r)=(1-r)\left[(1-\alpha )\text{𝐈}+\alpha \text{𝐒}\right]+r\left[(1-\alpha )\text{𝐐}+\alpha \stackrel{\sim }{\text{𝐒}}\right],$$ [74]

then because $(1-\alpha )\text{𝐐}+\alpha \stackrel{\sim }{\text{𝐒}}$ is a positive matrix, $\text{𝐌}(\alpha ;r)$is a positive irreducible column-stochastic matrix and D is a positive diagonal nonscalar matrix. Let

$$\text{𝐂}(\beta )=(1-\beta )\text{𝐈}+\beta \text{𝐌}(r;\alpha ),\hspace{1em}\beta >0.$$ [75]

Then

$$\text{𝐂}\left(\frac{{\alpha }^{\ast }}{\alpha }\right)=\left(1-\frac{{\alpha }^{\ast }}{\alpha }\right)\text{𝐈}+(1-r)\left[\frac{{\alpha }^{\ast }}{\alpha }(1-\alpha )\text{𝐈}+{\alpha }^{\ast }\text{𝐒}\right]+r\left[\frac{{\alpha }^{\ast }}{\alpha }(1-\alpha )\text{𝐐}+{\alpha }^{\ast }\stackrel{\sim }{\text{𝐒}}\right].$$ [76]

Observe that $\frac{{\alpha }^{\ast }}{\alpha }(1-\alpha )=\left(\frac{{\alpha }^{\ast }}{\alpha }-1\right)+(1-{\alpha }^{\ast })$. Hence

$$\text{𝐂}\left(\frac{{\alpha }^{\ast }}{\alpha }\right)=\left(1-\frac{{\alpha }^{\ast }}{\alpha }\right)\text{𝐈}+\left(\frac{{\alpha }^{\ast }}{\alpha }-1\right)\left[(1-r)\text{𝐈}+r\text{𝐐}\right]+(1-r)\left[(1-{\alpha }^{\ast })\text{𝐈}+{\alpha }^{\ast }\text{𝐒}\right]+r\left[(1-{\alpha }^{\ast })\text{𝐐}+{\alpha }^{\ast }\stackrel{\sim }{\text{𝐒}}\right].$$ [77]

Therefore,

$$\text{𝐂}\left(\frac{{\alpha }^{\ast }}{\alpha }\right)\text{𝐃}{\text{𝐱}}^{\ast }=\left(1-\frac{{\alpha }^{\ast }}{\alpha }\right)\text{𝐃}{\text{𝐱}}^{\ast }+\left(\frac{{\alpha }^{\ast }}{\alpha }-1\right)\left[(1-r)\text{𝐃}{\text{𝐱}}^{\ast }+r\text{𝐐𝐃}{\text{𝐱}}^{\ast }\right]+(1-r)\left[(1-{\alpha }^{\ast })\text{𝐈}+{\alpha }^{\ast }\text{𝐒}\right]\text{𝐃}{\text{𝐱}}^{\ast }+r\left[(1-{\alpha }^{\ast })\text{𝐐}+{\alpha }^{\ast }\stackrel{\sim }{\text{𝐒}}\right]\text{𝐃}{\text{𝐱}}^{\ast }.$$ [78]

Following Eqs. 67 and 68 we have

$$\text{𝐂}\left(\frac{{\alpha }^{\ast }}{\alpha }\right)\text{𝐃}{\text{𝐱}}^{\ast }=\left(1-\frac{{\alpha }^{\ast }}{\alpha }\right)\text{𝐃}{\text{𝐱}}^{\ast }+\left(\frac{{\alpha }^{\ast }}{\alpha }-1\right)\text{𝐃}{\text{𝐱}}^{\ast }+(1-r){\text{𝐱}}^{\ast }+r{\text{𝐱}}^{\ast },$$ [79]

or $\text{𝐂}\left(\frac{{\alpha }^{\ast }}{\alpha }\right)\text{𝐃}{\text{𝐱}}^{\ast }={\text{𝐱}}^{\ast }$.

Hence ${\text{𝐱}}^{\ast }$ is a positive eigenvector of $\text{𝐂}\left(\frac{{\alpha }^{\ast }}{\alpha }\right)\text{𝐃}$ with eigenvalue 1, and so $\sigma \left(\text{𝐂}\left(\frac{{\alpha }^{\ast }}{\alpha }\right)\text{𝐃}\right)=1$. Therefore, using Altenberg’s generalization of Karlin’s theorem for ML matrices [also Schreiber and Lloyd-Smith (23)], we see that $\sigma (\text{𝐂}(\beta )\text{𝐃})$ is strictly decreasing as $\beta$ increases. Thus,

$$\frac{{\alpha }^{\ast }}{\alpha }<1⟹1=\sigma \left(\text{𝐂}\left(\frac{{\alpha }^{\ast }}{\alpha }\right)\text{𝐃}\right)>\sigma (\text{𝐂}(1)\text{𝐃}),\frac{{\alpha }^{\ast }}{\alpha }>1⟹1=\sigma \left(\text{𝐂}\left(\frac{{\alpha }^{\ast }}{\alpha }\right)\text{𝐃}\right)<\sigma (\text{𝐂}(1)\text{𝐃}).$$ [80]

Because ${\text{𝐋}}_{\text{ex}}^{\ast }=\text{𝐂}(1)\text{𝐃}=\text{𝐌}(\alpha ;r)\text{𝐃}$ is a positive matrix, $\sigma [\text{𝐂}(1)\text{𝐃}]=\rho \left({\text{𝐋}}_{\text{ex}}^{\ast }\right)$ is the spectral radius of ${\text{𝐋}}_{\text{ex}}^{\ast }$. We therefore conclude that

$$\begin{array}{c}{\alpha }^{\ast }<\alpha ⟹\rho \left({\text{𝐋}}_{\text{ex}}^{\ast }\right)<1\\ {\alpha }^{\ast }>\alpha ⟹\rho \left({\text{𝐋}}_{\text{ex}}^{\ast }\right)>1.\end{array}$$

Combining Eqs. 73 and 80 we have proven the reduction principle for the three models for all $0\le r\le 1$.

Discussion

Through a simplification of the technique developed by Altenberg (20) we have shown that the reduction principle is a general property of mutation, migration, and recombination modifiers in multiallele models with arbitrary recombination between the modifier and major genes. These cases thus combine the “universal part” of the reduction principle with several additional “particular parts.”

A version of the reduction principle appears in a number of models without explicit genetics, principally ecological models for the evolution of dispersal. In ecological models, where the term migration is usually replaced by dispersal and in which genetics play no role, reduced dispersal has also been shown to evolve (10). Karlin’s theorem was independently discovered by Kirkland et al. (24) and applied to discrete-patch models of dispersal evolution. The reduction principle was found to hold in reaction–diffusion models of dispersal, where space is continuous, and dispersal is represented not by a finite matrix but by a differential or integral operator (10, 25–28). These results were incorporated into the reduction principle by Altenberg (29) through a theorem that generalizes Karlin’s theorem to all resolvent-positive operators, including second-order differential operators, Schrödinger operators, and nonlocal dispersal kernels.

Another kind of information that may be transmitted between organisms is cultural information. The reduction principle was demonstrated (ref. 13, pp. 203–206) to hold for a model of a vertically transmitted cultural trait whose faithfulness of transmission is determined by another vertically transmitted cultural trait. The applicability of the reduction principle to cultural inheritance is a largely unexplored area.

The mathematical proof of the reduction principle is valid under the minimal assumptions of infinite population size, constant-viability selection, a population at equilibrium under a balance between selection and a process that alters transmitted information, and equal scaling of transition probabilities during replication [referred to as “linear variation” (20)]. Violations of the reduction principle entail specific departures from these assumptions.

The principle sources of departure are as follows: (i) Allele frequencies at the major loci are not at equilibrium, due to being in a transient phase, or at a periodic or chaotic attractor, or due to fluctuating selection, or due to genetic drift. (ii) The modifier gene does not scale the transition probabilities between genotypes equally (or between other types of transmitted information, such as patch location), due to the presence of other transforming processes [the principle of partial control (13, 30)] or biases in the direction of mutation or dispersal. (iii) Natural selection acts directly on the modifier gene rather than indirectly as a consequence of transmission modification.

A modifier allele that increases recombination may invade if introduced while the major loci are proceeding toward fixation (31–33). Increased recombination may also evolve when the major loci are under cyclically fluctuating selection, either exogenously caused (34) or induced by host–parasite dynamics (35–37). Similarly, mutation-increasing alleles may invade under some patterns of fluctuating selection on the major gene (38–44). However, the direction of change in mutation rate is sensitive to the form of selection on the major gene(s) (44–46). Finally, migration-increasing alleles can succeed under some forms of temporal variation in the geographic pattern of selection on the major genes (30, 46, 47). In general, the reduction principle does not hold if selection occurs at the level of fertility differences between mating pairs or if there is mixed inbreeding and outcrossing with viability selection (48–50).

A further class of models for which the reduction principle fails includes two (or perhaps more) kinds of information transmission among the major loci. An important example has the major loci subject to both mutation and viability selection and the modifier affecting recombination among the major loci. In this case, an allele reducing recombination can succeed if there is diminishing-returns epistasis among the major loci, but increased recombination may succeed if there is synergistic epistasis (12, 51, 52).

Frequency-dependent selection brings no new behavior to the dynamics if the new modifier allele is introduced into a population at a stable equilibrium with viability differences among the major loci (13, 15). However, it is possible for frequency-dependent selection to create periodic or chaotic attractors (53–55) where the genotype frequencies are constantly changing. In such cases modifiers that increase mutation rates may invade (56, 57).

Departures from the reduction principle represent a wide diversity of biological phenomena and processes. We may ask whether in mathematical models that show these departures from the reduction principle there is a universal part as in the reduction principle itself. Very few relevant results have been obtained. We point to one general theorem (47) showing that when growth rates among multiple patches alternate every generation between $w_i$ and $1/w_i$, then the opposite of the reduction principle holds, and populations can be invaded by new strains that increase the dispersal rate. Such results, which are true also for mutation modifiers (39, 40, 43), appear to be sensitive to the symmetry assumptions on the selection regimes.

Here we have expanded the class of models for which the reduction principle holds. A complete picture of the conditions under which the reduction principle or departures from it hold remains a largely open theoretical question.