Assortative mating enhances postzygotic barriers to gene flow via ancestry bundling

Significance

Interspecific hybridization is now known to be common, raising the question of how species are maintained in the face of gene flow. Assortative mating among species can limit hybridization in the first place and can also help purge introgressed DNA if hybrids have reduced mating success. Here, we explore a further, indirect role of assortative mating as a barrier to gene flow. Under assortative mating, parents covary in their ancestry, and therefore “bundle” ancestry together in their offspring. This increases the population’s ancestry variance, allowing selection to purge introgressed DNA more efficiently. Using theoretical and empirical analyses, we show that the bundling effect can greatly increase the purging of introgressed DNA, contributing substantially to the maintenance of distinct species.

Abstract

Hybridization and subsequent genetic introgression are now known to be common features of the histories of many species, including our own. Following hybridization, selection often purges introgressed DNA genome-wide. While assortative mating can limit hybridization in the first place, it is also known to play an important role in postzygotic selection against hybrids and, thus, the purging of introgressed DNA. However, this role is usually thought of as a direct one: a tendency for mates to be conspecific reduces the sexual fitness of hybrids, reducing the transmission of introgressed ancestry. Here, we explore a second, indirect role of assortative mating as a postzygotic barrier to gene flow. Under assortative mating, parents covary in their ancestry, causing ancestry to be “bundled” in their offspring and later generations. This bundling effect increases ancestry variance in the population, enhancing the efficiency with which postzygotic selection purges introgressed DNA. Using whole-genome simulations, we show that the bundling effect can comprise a substantial portion of mate choice’s overall effect as a postzygotic barrier to gene flow. We then derive a simple method for estimating the impact of the bundling effect from standard metrics of assortative mating. Applying this method to data from a diverse set of hybrid zones, we find that the bundling effect increases the purging of introgressed DNA by between 1.2-fold (in a baboon system with weak assortative mating) and 14-fold (in a swordtail system with strong assortative mating). Thus, assortative mating’s bundling effect contributes substantially to the genetic isolation of species.

The tendency for mating pairs to be conspecific—assortative mating—is widespread among sexual organisms and can arise by a number of mechanisms, including active mate choice, physiological constraints on the compatibility of mates, and temporal or microspatial segregation of populations (1–4). Assortative mating is an important barrier to gene flow between species (5). In this capacity, it can act prezygotically, preventing the formation of hybrid offspring, and postzygotically, with hybrids suffering reduced mating success (6–8). Following admixture, postzygotic factors can cause deleterious effects in hybrids, leading to selection against introgressed DNA. Alongside potentially reduced mating success, these factors include incompatibility of genetic variants from the two parent species (9–11), maladaptation of introgressed alleles to the recipient species’ ecology (12), and higher genetic load in the donor species (13, 14). Recent evidence suggests that the deleterious effect of postzygotic factors can often be spread across a large number of genomic loci (14–16) (e.g., ∼1,000 loci for Neanderthal-human introgression [14]).

When introgressed ancestry is deleterious at many loci throughout the genome, the rate at which it is purged by selection is proportional to the variance across individuals in how much introgressed DNA they carry (13, 17, 18) (Materials and Methods). In light of this, we reasoned that while assortative mating can contribute directly to the purging of introgressed DNA in a given generation—via the reduced sexual fitness of hybrids—it can also contribute indirectly to purging in the next and later generations by altering how introgressed DNA is packaged among offspring. Specifically, positive assortative mating “bundles” like-with-like ancestry in the formation of offspring, increasing population-wide ancestry variance in offspring and later generations (19, 20) (Fig. 1). This increased ancestry variance enhances the efficiency with which postzygotic selection of various kinds purges introgressed DNA in these later generations. Therefore, there exist two mechanisms by which assortative mating can act as a barrier to gene flow between species: a direct, “sexual selection” mechanism and an indirect, “bundling” mechanism.

Fig. 1.

Assortative mating bundles introgressed DNA together, increasing the efficiency with which it is purged by selection. Introgressed alleles initially appear in the population in perfect cis-LD (cis covariance) with one another; recombination subsequently breaks down these cis covariances over time. Assortative mating generates positive ancestry correlations between mating pairs so that offspring inherit maternal and paternal genomes that covary in their proportions of introgressed DNA (i.e., that covary “in trans”). These trans covariances are subsequently converted to new cis covariances, as covarying maternal and paternal genomes recombine into the same gametes. The result is that relative to random mating, assortative mating causes introgressed DNA to become more densely concentrated in a smaller number of individuals. This bundling effect increases the variance across individuals in how much deleterious introgressed DNA they carry, and therefore increases the rate at which introgressed DNA is purged by selection.

Results

To study the contribution of these two mechanisms to the genetic isolation of species, we considered a model in which a recipient and donor species experience a single pulse of admixture, resulting in a fraction of donor DNA admixing into the recipient species’ gene pool. Introgressed alleles at many loci are assumed to reduce viability in the recipient species, such that an individual with introgressed genomic fraction $I$ has relative viability fitness $1-IS$, where $S$ is the strength of viability selection against introgressed ancestry. (This situation could arise, for example, if the donor species is poorly adapted to the recipient species’ local environment.)

Although we later develop calculations to measure the importance of the bundling effect that are agnostic with respect to the mechanisms driving assortative mating, for concreteness, we first consider a model in which assortative mating arises via active preference-trait mate choice. Males and females can both exercise mate choice in nature, but for simplicity in our models, females are assumed to be the choosy sex (our results do not depend on this assumption). Mate choice occurs according to a fixed relative preference model (21) based on ancestry, with a female preferring to mate with males from the species that matches her majority ancestry. Specifically, if a female’s introgressed fraction is $I_f$, then her probability of mating with a given male of introgressed fraction $I_m$ is proportional to

$${\alpha }^{4\left(\frac{1}{2} - I_f\right)\left(\frac{1}{2} - I_m\right)},$$ [1]

where $\alpha >1$ quantifies the overall strength of mate choice in the system. Thus, if a female is of 100% recipient-species ancestry ($I_f=0$), she prefers males of 100% recipient-species ancestry ($I_m=0$) over fully hybrid males ($I_m=1/2$) by a factor of $\alpha$ and over 100% donor-species males ($I_m=1$) by a factor of ${\alpha }^2$. In contrast, fully hybrid females ($I_f=1/2$) are indiscriminate in mate choice. We consider alternative specifications of mate choice later.

We performed whole-genome simulations of this model (22) and observed rapid purging of introgressed ancestry following the initial admixture pulse (Fig. 2). This purging is due to 1) viability selection, 2) sexual selection induced by the direct effect of mate choice, and 3) the enhancement of 1) and 2) by the bundling effect of assortative mating induced by mate choice. To isolate the contributions of mate choice’s sexual selection and bundling effects, we used simulation experiments to artificially eliminate the bundling effect while preserving the fitness consequences of mate choice for males. Each generation, we calculated the sexual fitness of every adult male under the model of mate choice described above (averaged over the population of adult females) and reassigned these sexual fitnesses to viability fitnesses that took effect in an additional round of viability selection. Mating pairs were then formed at random among surviving males and females. This procedure preserves the sexual selection effect of mate choice, since attractive males still enjoy the same higher fitness, but it eliminates the bundling effect of mate choice because the offspring generation is produced by random mating.

Fig. 2.

The bundling effect of assortative mating is an important contributor to the genetic isolation of species. (A) In our model, introgressed ancestry is purged as a result of viability selection and sexual selection and their enhancement by the bundling effect of mate choice. These three effects can be distinguished using simulation experiments that artificially preserve the sexual selection induced by mate choice but remove its bundling effect. In the absence of the bundling effect, the purging of introgressed ancestry (yellow line) is substantially slower and ultimately less profound than in the presence of the bundling effect (purple line). The simulations here assume that all loci are unlinked ($r=1/2$), $\alpha =4$, $S=0.1$, and there is an initial admixture proportion of 20%. The y axis is log-scaled so that trajectory slopes represent proportionate rates of purging. (B) The bundling effect contributes substantially to mate choice’s overall effect in cases of realistic recombination processes. Its contribution is especially large in the case of humans, a high-recombination species.

We found that in simulations with this unbundling procedure, substantially less introgressed DNA was purged than in simulations with unmanipulated mate choice. Consider the case displayed in Fig. 2A, where all loci are unlinked. In the absence of mate choice, the introgressed fraction would be reduced by viability selection alone from 20 to 16% after 25 generations (Fig. 2A). In the presence of mate choice, the introgressed fraction is, in fact, reduced to just 3%, so the overall effect of mate choice is an additional 13 percentage points of purging. However, if we remove the bundling effect of mate choice, the additional purging across 25 generations is just 8.5 percentage points (Fig. 2A). Therefore, in this case, the bundling effect accounts for more than one-third of mate choice’s overall effect. Put differently, viability selection would need to be 6.8-fold stronger to match the amount of purging after 25 generations under full mate choice but only 4.5-fold stronger if the bundling effect of mate choice is removed (SI Appendix, Fig. S1). Thus, bundling amplifies mate choice’s effect on the strength of selection against introgressed DNA by ∼50% in this case. As expected, the contribution of bundling increases with the strength of assortative mating (SI Appendix, Fig. S2).

The results described above are for a highly stylized genome, where all loci are unlinked. To see if the bundling effect can be important in more realistic genomes, we repeated our simulations using linkage maps for a high-recombination species (humans [23]) and a low-recombination species (Drosophila melanogaster [24]) (Materials and Methods). We found that the bundling effect constitutes a large proportion of mate choice’s overall effect in the human case and a smaller proportion in D. melanogaster (Fig. 2B). This implies that recombination plays an important role in the bundling effect, which, in turn, highlights a role for linkage disequilibria (LD) among introgressed alleles.

The increase in the rate of purging due to the bundling effect derives from increased ancestry variance in the population caused by assortative mating (Fig. 3A). In our unbundling simulations, which impose random mating, the population ancestry variance—normalized for the overall proportion of introgressed ancestry—matched that observed in simulations without mate choice (Fig. 3A). In contrast, when the bundling effect is preserved, the ancestry variance substantially exceeds this random-mating expectation (Fig. 3A).

Fig. 3.

The bundling effect increases ancestry variance by generating ancestry covariance between and within maternally and paternally inherited genomes. Evolution of the population’s overall ancestry variance, trans covariance, and cis covariance under the three scenarios of no mate choice, full mate choice, and mate choice with the bundling effect removed. Since introgressed ancestry is purged at different rates in the three cases, and since the overall introgressed fraction $I_t$ influences the range of possible ancestry variances and covariances, we normalize each variance and covariance trajectory by the variance expected from the introgressed fraction alone, absent any LD: $I_t\left(1-I_t\right)$. (A) Recombination rapidly breaks down the initial ancestry variance in the population. The decay of the variance is, however, substantially slower with mate choice (purple line) than without mate choice (blue line). This increased variance is due to the bundling effect: the normalized trajectory under mate choice with the bundling effect removed is the same as that under no mate choice. (B) One component of the increased variance caused by the bundling effect is due to ancestry covariances across maternally and paternally inherited genomes, which arise because mate choice causes mating pairs to have correlated ancestries. Without mate choice, or with its bundling effect removed, the trans covariance is zero (except in the first generation, where individuals are all of one species or the other). (C) The second component of the bundling effect is its effect on ancestry covariances within haploid genomes. Cis covariances are initially large in all scenarios (since introgressed alleles appear in the population in perfect LD), but they are rapidly broken down over time by recombination. Mate choice decelerates this decay by generating trans covariances, which recombination converts into cis covariances. Parameters are as in Fig. 2A.

We formalized this intuition by decomposing the overall population ancestry variance into three components (Materials and Methods): 1) a small component due to heterozygosity at different loci; 2) a component due to ancestry covariance between individuals’ maternally and paternally inherited genomes (trans-LD; Fig. 1); and 3) a component due to ancestry covariance within individuals’ maternally and paternally inherited genomes (cis-LD; Fig. 1). (These covariances have been given various names in the literature (e.g., gametic phase disequilibrium for cis-LD and nongametic LD for trans-LD) (25). The sum of cis- and trans-LD has been referred to as “composite” LD (25).) In the first generation after admixture, all introgressed alleles have been inherited from donor-species parents and, therefore, lie in perfect cis-LD with one another. Recombination breaks down this initial cis-LD quickly over subsequent generations (Fig. 3C), reducing ancestry variance and thus slowing the rate of purging (e.g., refs. 13, 18). If mating were random, trans-LD would be zero in all generations after the initial admixture event (Fig. 3B), so the rate of decay of overall ancestry variance would depend only on the changing frequencies of introgressed alleles (component 1 above) and the reduction in their cis-LD by recombination (component 3). The bundling effect of assortative mating impedes the decay of ancestry variance in two ways: by generating trans-LD (component 2; Fig. 3B) and by slowing down the decay of cis-LD (component 3; Fig. 3C).

The “trans channel” of the bundling effect is a direct consequence of assortative mating: mating pairs have disproportionately similar ancestry, so offspring inherit maternal and paternal genomes with correlated ancestry (i.e., trans-LD; Fig. 1 and SI Appendix, Fig. S3). While the degree of trans-LD in a given generation—and, thus, the strength of the trans channel—does not depend on recombination in the previous generation, the “cis channel” is driven largely by recombination converting trans-LD from the previous generation into cis-LD (Fig. 1) (as shown in previous models of mate choice; refs. 26–28).

Our results thus far have excluded sex chromosomes, which are of particular interest as they are enriched for genes involved in mate choice (29) and show distinct patterns of cis-LD and trans-LD in models of sexual selection (28, 30). Incorporating sex chromosomes into our model, we first consider the case where there is autosomal recombination in the heterogametic sex. In this case, sex chromosomes tend to purge introgressed ancestry at a higher rate than autosomes (SI Appendix, Fig. S4 A and B) because sex chromosomes do not recombine in the heterogametic sex and therefore maintain longer, more deleterious introgressed linkage blocks than autosomes (18). Interestingly, we find that Z chromosomes (in female-heterogametic taxa, such as birds) purge more introgressed DNA than X chromosomes (in male-heterogametic taxa, such as mammals) (SI Appendix, Fig. S4), with the importance of the bundling effect concomitantly larger for Z chromosomes. This difference can be explained by the fact that under an even sex ratio, two-thirds of Z chromosomes each generation are in males, on whom—in our model—selection against introgressed ancestry is stronger than in females because of the additional effect of sexual selection. In contrast, under male heterogamety, two-thirds of X chromosomes are in females, on whom selection against introgressed ancestry is weaker.

Consistent with this logic, when we consider the case where there is no autosomal recombination in the heterogametic sex (as in male Drosophila and female Lepidoptera), we find that the X chromosome purges less introgressed DNA than the autosomes, while the Z chromosome purges more. In this case, the X and Z chromosomes recombine more than the autosomes because they spend two-thirds of the time in the recombining sex (vs. one-half for autosomes). This promotes slower purging of introgressed ancestry (18). Counteracting this force in female-heterogametic systems, the Z chromosome spends more time than autosomes do in the more strongly selected sex (males); in contrast, the X chromosome spends less time in the more strongly selected sex.

Our model assumes female mate choice. While male mate choice is known to play an important role in many systems (31), sexual selection tends to be stronger for males than for females (32). Therefore, all else equal, our results show that the influence of mate choice—and, concomitantly, its bundling effect—on the purging of sex-linked introgressed ancestry is stronger in female-heterogametic species than in male-heterogametic species.

Throughout, we have assumed a simple model in which genome-wide ancestry determines the mating preferences of females, the attractiveness of males, and viability. In a genetically more realistic model, separate loci would underlie these three distinct traits. To check that our results are robust to consideration of this more realistic scenario, we augmented our model to include female preference loci, male trait loci, and loci at which introgressed alleles reduce viability. We found that the degree of purging of introgressed ancestry and the importance of mate choice’s bundling effect were similar to our baseline simulations (SI Appendix, Fig. S5). This can be explained by the fact that most purging of introgressed DNA happens extremely quickly after admixture (13, 18), before the preference, trait, and ancestry loci have a chance to become fully dissociated by recombination.

We have considered a particular model of mate choice in which a female prefers to mate with males of the species matching her majority ancestry. In an alternative model, a female is disproportionately likely to mate with males with the same ancestry as her (33). This might occur, for example, when mating is based on matching a polygenic trait like body size (34) or reproductive timing (35), or when hybrids display a novel trait that is disfavored in mate choice by the parental species but favored by fellow hybrids (36). Under such a model of mate choice (Materials and Methods), we find that the relative importance of the bundling effect is even greater than under our baseline “preference for conspecifics” model (SI Appendix, Fig. S6B). This is because in a model where hybrid females prefer hybrid males, the ancestry covariance among mating pairs (and, therefore, the trans-LD in offspring) is especially large. However, despite the increased importance of the bundling effect in this model, introgressed ancestry is not purged at an especially high rate (SI Appendix, Fig. S6B) because hybrid males are not of especially low fitness—being favored in mating by hybrid females (33). Similar results are observed for “sexual imprinting” models in which a female prefers to mate with males who have similar ancestry fractions to her father (SI Appendix, Fig. S6C) or mother (SI Appendix, Fig. S6D) (37, 38).

Thus far, we have considered only additive viability selection against introgressed ancestry. An alternative possibility is that the deleterious viability effects of introgressed alleles are largely recessive (13). In that case, we might predict the bundling effect—and, in particular, its trans channel—to have an especially large influence on the rate of purging of introgressed ancestry, as it generates an excess of homozygosity. In fact, in simulations of this scenario, we observe only a modest increase in the importance of the bundling effect (SI Appendix, Fig. S7) relative to the additive case. The reason is that the overall influence of the trans channel is dominated by its effect on the $L(L-1)\approx L^2$ possible trans associations across locus pairs, rather than its effect on the $L$ possible within-locus trans associations (for which dominance is relevant) (39).

Another possible model of selection against introgression is that introgressed alleles reduce viability because of deleterious epistatic interactions with recipient-species alleles (9, 10). By promoting the association of like-with-like ancestry, assortative mating is expected to reduce the efficiency of selection against such cross-species Dobzhansky-Muller incompatibilities (DMIs). In the presence of other forms of selection against introgressed ancestry (e.g., sexual selection induced by mate choice), the overall impact of the bundling effect will depend on whether these other forms of selection outweigh selection against introgressed ancestry due to DMIs. Consistent with this logic, when we alter our model to include a large number of DMIs and begin our simulations with an introgressed fraction near 1/2 (so that sexual selection against introgressed ancestry is initially weak), we find that the purging of introgressed ancestry is initially slowed by ancestry bundling, as assortative mating reduces the efficiency of selection against incompatible introgressed alleles (SI Appendix, Fig. S8). However, as the introgressed fraction declines and sexual selection against introgressed ancestry intensifies, the bundling effect switches to accelerating the purging of introgressed ancestry (SI Appendix, Fig. S8). It has been argued that persistent introgression of donor-species alleles should often cause the recipient-species alleles with which they are incompatible to be eliminated by selection, collapsing the postzygotic barrier to gene flow (40, 41). By accelerating the wholesale purging of introgressed alleles, the bundling effect can dampen their impact on the frequencies of recipient-species incompatible alleles, promoting the maintenance of incompatibility-based postzygotic barriers and thus species boundaries.

In general, the strength of ancestry-based assortative mating can be quantified by the correlation coefficient of ancestry proportions between mates $\rho$. In the Materials and Methods, we show that if the ancestry correlation among mating pairs is $\rho$, then the bundling effect causes a factor $1+\rho$ more introgressed ancestry to be purged in the generation immediately following assortative mating and a factor $\sim 1/(1-\rho )$ more introgressed ancestry to be purged in the long run. The latter calculation relies on some potentially unrealistic assumptions, notably that selection does not alter ancestry variance appreciably within each generation, but we may conservatively treat $1+\rho$ and $1/(1-\rho )$ as lower and upper bounds, respectively, for the overall impact of the bundling effect.

The most direct way to measure $\rho$ is to estimate the ancestry proportions of mates. This requires detailed knowledge of mating pairs. One system where this is possible is the long-term study of baboons in Kenya’s Amboseli basin (42). There, yellow baboons (Papio cynocephalus) and anubis baboons (P. anubis) hybridize (42, 43), and genomic analyses indicate that the minor anubis ancestry has been purged over time (44). Tung et al. (45) used long-term observations of mating behavior in this system to investigate the determinants of mating success and mate pair composition, revealing ancestry-based assortative mating. Using data from Tung et al. (45), we calculate an ancestry correlation of $\rho =0.195$ among putative mating pairs (Materials and Methods and SI Appendix, Fig. S9A). If anubis ancestry is deleterious, this estimate translates to a ∼20 to 24% increase in the purging of anubis ancestry due to the bundling effect.

Even in cases where mating is not observed, $\rho$ can still be measured by estimating the ancestry fractions of mothers and offspring. In a hybrid population of the swordtail fishes Xiphophorus birchmanni and X. cortezi in Hidalgo, Mexico, genomic evidence suggests that the minor-parent ancestry (birchmanni) is deleterious to individuals of predominantly major-parent ancestry (cortezi) (46). Powell et al. (47) measured mother-offspring ancestry differences to infer paternal ancestry, finding evidence for strong ancestry-based assortative mating in this system (consistent with other swordtail systems; refs. 8, 49). Using the mother-offspring data of Powell et al. (47), we calculate an ancestry correlation among mating pairs of $\rho =0.928$ (Materials and Methods and SI Appendix, Fig. S9B). This estimate corresponds to a ∼2- to 14-fold increase in the purging of introgressed ancestry, revealing that in systems with strong but imperfect premating isolation, any ancestry that does introgress across the prezygotic barrier will experience much more rapid postzygotic purging because of ancestry bundling.

Finally, to obtain a broad sense of the quantitative importance of the bundling effect, we turn to a meta-analysis of assortative mating in avian hybrid zones (50), which found an average correlation coefficient between mates of 0.44 (33, 50) (Materials and Methods). Substituting this value into our calculations above, we estimate that the bundling effect of assortative mating increases the purging of introgressed DNA by 44 to 80%.

Discussion

When two species come into secondary contact and hybridize, DNA introgressed from one species into the other is often deleterious in the recipient species’ genomic and environmental backgrounds at many loci across its genome, and is therefore purged over time by selection (51). This purging inhibits the genomic collapse of the two species into one. Here, we have shown that assortative mating, in addition to the direct effects it can have in reducing gene flow between species (i.e., limiting hybridization and reducing the sexual fitness of hybrids), has an important indirect effect. Assortative mating “bundles” introgressed ancestry in offspring and later generations, increasing ancestry variance in the population and thus improving the efficiency with which selection purges deleterious introgressed DNA. We have used simulations, analytical calculations, and empirical measurements from natural hybrid systems to show that this indirect bundling effect can account for a substantial portion of assortative mating’s overall effect as a barrier to gene flow.

Positive assortative mating can be generated through a variety of mechanisms, including active preference-trait mate choice, physiological constraints on the compatibility of mates, and temporal or microspatial segregation of populations (1–4), with important consequences for the speciation process (4, 52). In our simulations, we modeled assortative mating as arising from an active preference for mating with conspecifics. We found that the importance of the bundling effect was relatively insensitive to behavioral and genetic details of the preference-trait system, as long as the preference and trait were sufficiently polygenic. This suggests that our results should extend to other kinds of mate choice, active or not.

While our simulations employed specific models of mate choice, our analytical calculations revolve around a general metric for assortative mating: the ancestry correlation between mates, $\rho$. These calculations, therefore, apply broadly to any system of assortative mating, regardless of the underlying mechanism. We applied these calculations to two empirical situations where $\rho$ could be estimated. We found, in a swordtail system with extremely strong assortative mating, that the bundling effect might greatly enhance the purging of alleles that introgress across the strong prezygotic barrier. In a baboon system with weaker assortative mating, we found that the bundling effect might increase the amount of purging by as much as 20 to 25%. As the corpus of genomic data from hybrid zones and other systems of introgression continues to grow, measurements of $\rho$ from diverse taxa will become available, allowing for a broad view of the importance of the bundling effect in the purging of introgressed ancestry.

In the scenario we considered, there is a fitness cost to individuals with hybrid ancestry, which the bundling effect exacerbates. In the particular models of mate choice that we simulated, mate choice also led to hybrid individuals suffering a further reduction in fitness due to sexual selection, being disfavored in mate choice by all or the majority of the population. In contrast, other systems of assortative mating could favor individuals with hybrid ancestry, if the number of matings they lose from their major ancestry population is more than compensated for by the number of matings they gain from their minor ancestry population. However, importantly, the operation of the bundling effect does not depend on the assumption that hybrids suffer reduced sexual fitness—as long as they suffer a net fitness cost overall, the bundling mechanism will enhance selection against hybrid ancestry. Indeed, as revealed in our simulations, the overall effect of assortative mating can, in general, be partitioned into a direct sexual selection effect and the bundling effect, which is, therefore, separable in principle from any sexual selection induced by assortative mating.

In this regard, it is interesting to note that in our mate choice simulations, the bundling effect had the greatest impact when hybrids preferred to mate with other hybrids, despite sexual selection against hybrids in this scenario being weaker than in our baseline scenario, where hybrids are relatively indiscriminate in mate choice. The reason is that like-with-like assortative mating generates a stronger ancestry correlation among mates, with mate ancestry matching occurring across the entire spectrum of ancestries. Therefore, even though introgressed ancestry is purged more slowly than in the scenario where hybrids are indiscriminate (owing to stronger sexual selection against hybrid ancestry in that scenario), the contribution of the bundling effect to this purging is proportionately greater. This result extends to any system of assortative mating under which hybrids disproportionately mate with hybrids, including “passive” systems based on spatial or temporal behavioral phenotypes for which hybrids show intermediate values (e.g., ref. 35).

In some situations, assortative mating can actually facilitate gene flow between species—for example, if it increases the sexual fitness of hybrids and does so sufficiently to compensate for their reduced fitness due to natural selection. For example, there are a number of cases where sexual selection has favored the asymmetric introgression of display traits across species boundaries (53, 54). Additionally, the introgression of preference alleles from one population into the other can facilitate the subsequent introgression of the preferred trait via sexual selection (55–57). Such scenarios involve more complicated dynamics than we have modeled, including situations where selection favors introgression of certain parts of the genome but not others. It would be interesting to understand how ancestry bundling induced by assortative mating affects these dynamics.

Under the scenario of polygenic selection against introgressed DNA, most purging occurs—and many of the genomic signatures of introgression are set up—in the first few generations after initial hybridization, when introgressed blocks are large and variably sized (13, 18, 51). The influence of the bundling effect is concomitantly most acute in these early generations (Figs. 2 and 3). These observations suggest that we can learn about the long-term consequences of hybridization—and how these consequences are affected by factors such as the recombination process and the mating system—from the synthesis of theory and empirical field studies of the viability and mating success of early-generation hybrids.

Conclusions.

Models of speciation with gene flow, and the maintenance of species after secondary contact, have revealed these processes to be highly sensitive to the degree of gene flow between the species involved (1, 5, 52, 58). Any mechanism that reduces gene flow between species, therefore, can contribute meaningfully to their isolation. We have shown that the bundling effect of assortative mating can substantially accelerate the purging of introgressed DNA. By thus reducing gene flow, the bundling effect could play an important role in the genetic isolation of hybridizing species.

Materials and Methods

Admixture Pulse.

The model organism is a diploid sexual with a genome of length $L=1,000$ loci. In our simulations, the “recipient-species” population is of size $N=100,000$ and experiences a sudden pulse of introgression such that a fraction $I_0=0.2$ of the generation-0 population are of 100% donor-species ancestry. Using the SLiM 3.3 simulation software (22), we track the overall introgressed fraction in subsequent generations $I_t$. The results displayed in the figures are averages across 100 trials.

Viability Selection.

Introgressed alleles across the genome reduce viability in the recipient species, with the deleterious effect equal and additive across and within loci. Thus, if a fraction $I$ of an individual’s diploid genome is introgressed, the individual’s viability is $1-IS$, where $S$ is the strength of selection. In our simulations, $S=0.1$.

Mate Choice.

In our initial model, females engage in mate choice based on ancestry. All the models of mate choice that we consider are fixed relative preference models (21): each adult female has a strength of preference for every adult male in the population and chooses to mate with a given male with probability proportional to her strength of preference for him. The expected number of matings is the same for each adult female, so only viability selection operates among females. In contrast, some males have a higher expected number of matings than others, so both viability and sexual selection operate in males. Our baseline “preference for conspecifics” model for the strength of a female’s mating preference is

$${\alpha }^{4\left(\frac{1}{2}-I_f\right)\left(\frac{1}{2}-I_m\right)},$$

where $I_f$ is the female’s introgressed fraction, $I_m$ is the male’s introgressed fraction, and $\alpha$ is the overall strength of mating preferences in the population. In the simulations displayed in the main text, $\alpha =4$; we explore various values of $\alpha$ in SI Appendix, Fig. S2.

The “like-with-like” mating model that we consider (SI Appendix, Fig. S6B) is

$${\alpha }^{-2\left|I_f-I_m\right|}.$$

The “sexual imprinting” models that we consider (SI Appendix, Fig. S6 C and D) are

$${\alpha }^{-2\left|I_{\mathit{parent}}-I_m\right|},$$

where $I_{\mathit{parent}}$ is the introgressed fraction of the female’s father (SI Appendix, Fig. S6C) or mother (SI Appendix, Fig. S6D).

The Rate of Purging Is Proportional to the Population Ancestry Variance.

Treating the ancestry proportion of an individual as a phenotype, the change in mean ancestry ($I_t$) from one generation to the next can be written, using the breeder’s equation, as

$$I_{t+1}-I_t=\beta V_t,$$

where $V_t$ is the variance in ancestry proportion across individuals, as the ancestry proportion is a perfectly additive phenotype, and $\beta$ is the directional selection gradient on ancestry. Under our additive model of viability selection against introgression, with no mate choice, $\beta =-\frac{S{I}_t}{1-S{I}_t}\approx -S{I}_t$ (18), so the decrease in introgressed ancestry is proportional to the variance in ancestry proportion. Under more complex models of selection (e.g., viability selection due to DMIs and sexual selection due to ancestry-based mate choice), our selection gradient would no longer take this simple form. However, the selection gradient can always be empirically calculated as the slope of fitness regressed on individual ancestry proportion; it will then naturally include fitness components due to mate choice and may depend on the ancestry composition of the population. The breeder’s equation will always predict the change in mean ancestry in the next generation as the product of the selection gradient and the population variance in ancestry. Thus, in a generation, we can always conceptualize the effect of mate choice in terms of its effect on ancestry variance and its effect on the strength of directional selection.

Decomposition of the Population Variance in Ancestry.

An individual’s introgressed fraction $I$ can be decomposed into maternal (m) and paternal (p) contributions:

$$I=\frac{1}{2}\left(I^m+I^p\right)=\frac{1}{2}\left(\frac{1}{L}{\displaystyle {\sum }_{l=1}^L{i}_l{}^m}+\frac{1}{L}{\displaystyle {\sum }_{l=1}^L{i}_l{}^p}\right)=\frac{1}{2L}{\displaystyle {\sum }_{l=1}^L\left(i_l{}^m+i_l{}^p\right)},$$

where $I^m$ and $I^p$ are the introgressed fractions of the maternally and paternally inherited genomes of the individual, respectively, and $i_l{}^m$ and $i_l{}^p$ are indicator variables for whether the maternally and paternally inherited alleles at locus $l$ are introgressed.

The ancestry variance across all individuals is

$$\mathit{Var}\left(I\right)=\frac{1}{4}[\frac{1}{L^2}{\sum }_{l=1}^L\mathit{Var}\left(i_l{}^m\right)+\frac{1}{L^2}{\sum }_{l=1}^L\mathit{Var}\left(i_l{}^p\right)+\frac{1}{L^2}{\sum }_{l\ne l^{\prime }}\mathit{Cov}(i_l{}^m,i_{l^{\prime }}{}^m)+\frac{1}{L^2}{\sum }_{l\ne l^{\prime }}Cov(i_l{}^p,i_{l^{\prime }}{}^p)+2Cov(I^m,I^p)].$$ [2]

The term $\mathit{Cov}\left(I^m,I^p\right)$ in Eq. 2 is the ancestry covariance between maternally and paternally inherited genomes (i.e., the overall trans-LD). The terms $\frac{1}{L^2}{\displaystyle {\sum }_{l\ne l^{\prime }}\mathit{Cov}\left(i_l{}^m,i_{l^{\prime }}{}^m\right)}$ and $\frac{1}{L^2}{\displaystyle {\sum }_{l\ne l^{\prime }}\mathit{Cov}\left(i_l{}^p,i_{l^{\prime }}{}^p\right)}$ in Eq. 2 are the ancestry covariances within maternal and paternal genomes (i.e., the cis-LD). In the absence of trans-LD and cis-LD, ancestry variance would simply be a function of the allele frequencies at different loci: $\frac{1}{4L^2}\left[{\displaystyle {\sum }_{l=1}^L\mathit{Var}\left(i_l{}^m\right)}+{\displaystyle {\sum }_{l=1}^L\mathit{Var}\left(i_l{}^p\right)}\right]$ in Eq. 2.

The variance, trans-LD, and cis-LD values displayed in Fig. 3 were calculated as above and normalized by dividing through by a factor of $I_{t }(1-I_t),$ where $I_t=E[I]$ is the population’s introgressed fraction in generation $t$. This normalization accounts for the fact that the variances and covariances scale with the overall frequency of introgressed ancestry; under this normalization, the overall variance, trans-LD, and cis-LD are the same for the “no mate choice” and “mate choice with bundling removed” cases in Fig. 3, despite introgressed ancestry being purged at a higher rate in the latter case owing to the additional effect of sexual selection.

Sex Chromosomes.

In the configurations of our model that involve sex chromosomes, the ancestry fraction of a heterogametic individual is calculated as $I =\left(A{I}^A+X{I}^X\right)/\left(A+X\right)$, where A and X are the autosomal and X-linked (or Z-linked) fractions of total haploid genome length, respectively, and $I^A$ and $I^X$ are the individual’s autosomal and X-linked introgressed fractions. Notice that in treating the hemizygous X equivalently to the autosomes in this calculation, we are assuming full-dosage compensation. For the stylized genomes we consider in SI Appendix, Fig. S4, $A=X=1/2$.

Recombination Maps.

Loci were assumed to be spaced evenly along the physical (bp) genome. For stylized recombination processes, we assumed that all loci were unlinked (e.g., Figs. 2A and 3) or that the rate of recombination between adjacent locus pairs on the same chromosome was constant (SI Appendix, Fig. S4). In the case of realistic recombination processes (Fig. 2B), we interpolated empirical linkage maps along our evenly spaced loci. For humans, we used the male and female maps generated by Kong et al. (23); for D. melanogaster, we used the female linkage map produced by Comeron et al. (24). We ignored crossover interference in our simulations.

DMIs.

We considered only pairwise DMIs between donor- and recipient-species alleles (i.e., no higher-order epistasis). Our simulations involved $L=1,000$ loci, harboring $D=100$ nonoverlapping DMI locus pairs, and 800 “ancestry loci.”

Suppose that the incompatible alleles at loci $l_1$ and $l_2$ are A and B, with a and b being the alternative alleles at these loci, respectively. We considered two dominance cases for our DMIs (59). In the first case, all DMIs are of intermediate epistatic dominance, so that the genotypes AaBb, AABb, AaBB, and AABB suffer viability reductions $s/2$, $3s/4$, $3s/4$, and $s$. In the second case, all DMIs are epistatically fully recessive so that only the genotype AABB suffers a viability reduction, of relative size $s$. In our simulations, $s=0.02$, with fitness effects combining multiplicatively across DMIs. While viability selection in this model is based on epistasis across the $D$ DMI locus pairs, mate choice is based on ancestry at all $L$ loci.

If introgressed ancestry were to start at a low frequency, it would be strongly disfavored in mate choice, and sexual selection based on overall ancestry would overwhelm viability selection based on DMIs. In this case, the trajectories would resemble those for additive viability selection, as considered elsewhere in this paper. Therefore, to better isolate the role of DMIs themselves, we begin our simulations with a high fraction of introgressed ancestry (47.5%). In this case, sexual selection against introgressed ancestry is initially weak, and most selection is due to DMIs. We track the overall fraction of introgressed ancestry through time, as well as the fraction of introgressed alleles at loci where the introgressed allele is involved in an incompatibility.

Separate Preference, Trait, and Ancestry Marker Loci.

In the simulations that distinguished the loci underlying the female preference, the sexually selected male trait, and ancestry/viability fitness, we considered two architectures: 1) $P=100$ preference loci, $T=100$ trait loci, and $A=800$ ancestry loci; and 2) $P=10$ preference loci, $T=10$ trait loci, and $A=980$ ancestry loci. The strength of a female’s preference for a male was calculated as

$${\alpha }^{4\left(\frac{1}{2} - \frac{P_f}{2P}\right)\left(\frac{1}{2} - \frac{T_m}{2T}\right)},$$

where $P_f$ is the number of introgressed preference alleles carried by the female, and $T_m$ is the number of introgressed trait alleles carried by the male. Both viability fitness and the overall introgressed fraction were determined according to the fraction of introgressed alleles at ancestry loci.

Relating Ancestry Correlations between Mates to Increased Population Variance.

Since there are many more locus pairs than individual loci ($\sim L^2$ vs. $L$) and introgressed alleles initially appear in perfect cis-LD with one another, the initial population ancestry variance $V_0$ is almost entirely tied up in cis-LD between introgressed alleles, with heterozygosity at the $L$ individual loci in the genome contributing negligibly. For the same reason, the ancestry variance in generation $t$ can be written as a sum of the total cis-LD and the total trans-LD:

$$V_t = C_t+T_t.$$

We will assume that all loci are unlinked ($r=1/2$), which, for our purposes (short timescales), is approximately the case for most species (60, 61). In the construction of generation $t+1$, recombination reduces the cis-LD from generation $t$ by a factor of 1/2 and converts the trans-LD in generation $t$ to new cis-LD at a rate of 1/2. If assortative mating among generation-$t$ parents generates an ancestry correlation between mates of $\rho$, then an amount $\rho V_t/2$ of trans-LD is present in generation $t+1$ (e.g., ref. 27). Therefore,

$$V_{t+1}=C_{t+1}+T_{t+1}=\frac{C_t}{2}+\frac{T_t}{2}+\frac{\rho V_t}{2}=\frac{1+\rho }{2}{V}_t.$$ [3]

The amount of introgressed DNA purged in generation $t$ is

$$I_t-I_{t+1}=\frac{S{I}_t}{1-S{I}_t}{V}_t,$$ [4]

where $I_t$ is the introgressed fraction in generation $t$, and $S$ is the overall strength of selection against introgressed ancestry (18). In the absence of assortative mating ($\rho =0$), ${\tilde{V}}_{t+1}=\frac{1}{2}{V}_t$. The amount of introgressed DNA purged in generation $t+1$, given assortative mating in generation $t$, is $I_{t+1}-I_{t+2}=\frac{S{I}_{t+1}}{1-S{I}_{t+1}}{V}_{t+1}=\frac{S{I}_{t+1}}{1-S{I}_{t+1}}\frac{1+\rho }{2}{V}_t$, while the amount that would have been purged in the absence of the bundling effect of mate choice (same $S$ but $\rho =0$) is $I_{t+1}-{\tilde{I}}_{t+2}=\frac{S{I}_{t+1}}{1-S{I}_{t+1}}{\tilde{V}}_{t+1}=\frac{S{I}_{t+1}}{1-S{I}_{t+1}}\frac{1}{2}{V}_t$. Therefore, the additional amount of introgressed DNA that is purged in generation $t+1$ because of ancestry bundling induced by assortative mating in generation $t$ is

$$\left(I_{t+1}-I_{t+2}\right)-(I_{t+1}-{\tilde{I}}_{t+2})=\frac{S{I}_{t+1}}{1-S{I}_{t+1}}\frac{\rho }{2}{V}_t,$$

so that the bundling effect has increased purging by a proportion

$$\frac{\left(I_{t+1}-I_{t+2}\right)-(I_{t+1}-{\tilde{I}}_{t+2})}{I_{t+1}-{\tilde{I}}_{t+2}}=\rho .$$ [5]

To understand how such effects compound over generations, we assume that natural and sexual selection are weak, such that nearly all dissipation of variance is due to recombination rather than selection. We further assume that the ancestry correlation within mating pairs is a constant value $\rho$ each generation. Under these assumptions, we may iterate Eq. 3, yielding

$$V_t={\left(\frac{1+\rho }{2}\right)}^t{V}_0.$$ [6]

The amount of introgressed DNA purged in generation $t$ is

$$I_t-I_{t+1}=\frac{S{V}_t{I}_t}{1-S{I}_t}\approx S{V}_t{I}_t,$$

since $S$ is assumed to be small. The total proportion of introgressed ancestry purged up to generation $t$ can therefore be written

$$\begin{array}{c}\frac{I_0-I_t}{I_0}=\frac{\left(I_0-I_1\right)+\left(I_1-I_2\right)+\dots +\left(I_{t-2}-I_{t-1}\right)+\left(I_{t-1}-I_t\right)}{I_0}\\ =\frac{S{V}_0{I}_0+S{V}_1{I}_1+\dots +S{V}_t{I}_t}{I_0}.\end{array}$$

Since selection is assumed to be weak, $I_t$ changes slowly, so that

$$\frac{I_0-I_t}{I_0}\approx \frac{S{V}_0{I}_0+S{V}_1{I}_0+\dots +S{V}_t{I}_0}{I_0}=S\left(V_0+V_1+\dots +V_t\right).$$ [7]

In the presence of mate choice, we substitute [6] into [7] to find

$$\frac{I_0-I_t}{I_0}\approx S\left(V_0+\left(\frac{1+\rho }{2}\right)V_0+{\left(\frac{1+\rho }{2}\right)}^2{V}_0+\dots +{\left(\frac{1+\rho }{2}\right)}^t{V}_0\right)=2\text{S}{V}_0\frac{1-{\left(\frac{1+\rho }{2}\right)}^t}{1-\rho },$$ [8]

while in the absence of the bundling effect of mate choice (same $S$ but $\rho =0$), the total proportion of introgressed ancestry purged up to generation $t$ would instead be

$$\frac{I_0-I_t}{I_0}\approx 2S{V}_0\left(1-\frac{1}{2^t}\right).$$ [9]

The excess fraction of purging due to the bundling effect of mate choice is therefore given by [Eq. 8 – Eq. 9]/Eq. 9:

$$\frac{2S{V}_0\frac{1-{\left(\frac{1+\rho }{2}\right)}^t}{1-\rho } - 2S{V}_0\left(1-\frac{1}{2^t}\right)}{2S{V}_0\left(1-\frac{1}{2^t}\right)}=\frac{\rho +\frac{1}{2^t}\left[{\left(1+\rho \right)}^t-1+\rho \right]}{\left(1-\rho \right)\left(1-\frac{1}{2^t}\right)}.$$

As $t$ grows large, and assuming $\rho <1$, this expression converges to $\rho /(1-\rho )$. Thus, eventually, overall purging has been increased by a factor of

$$1+\rho /(1-\rho )=1/(1-\rho ).$$ [10]

When selection is strong, ancestry variance will decay not only because of recombination, but also because of selection; this will act to diminish the effect of assortative mating in decelerating the decay of ancestry variance.

Empirical Examples.

Hybrid zone between yellow and anubis baboons.

In this hybrid zone, rank and other social factors have been shown to play an important role in mate choice (45). In addition, Tung et al. (45) found that males with more anubis ancestry are favored in mate choice overall and that there is also ancestry-based assortative mating. If, as genomic evidence suggests (44), the minor-parent ancestry (anubis) has historically been deleterious in this population, then our calculations above imply that it is the ancestry correlation coefficient within mating pairs that determines the impact of ancestry bundling on the purging of anubis ancestry. In particular, while there is a directional mating advantage of anubis ancestry, this is part of overall direct selection on ancestry and is, therefore, included in the selection gradient along with other factors, while assortative mating increases ancestry variance (see section “The rate of purging is proportional to the population ancestry variance” above).

To calculate the correlation coefficient among putative mating pairs, we filtered the data of Tung et al. (45) to include only those male-female pairs where consortship behavior (mate guarding) was observed in a period when the female conceived—these are putative mating pairs. In cases where multiple males consorted the same female in a single conceptive period, we randomly selected one of the males and then calculated the ancestry correlation coefficient among consorting male-female pairs. Repeating this male-sampling procedure 100,000 times, we calculated an average ancestry correlation coefficient within mating pairs of $\rho =0.195$. This value is consistent with a 19.5% increase in purging of anubis ancestry in the generation after assortative mating [Eq. 5] and a ∼24.2% increase in long-term purging [Eq. 10].

Swordtail fish.

In a hybrid population between Xiphophorus birchmanni and X. cortezi, Powell et al. (47) measured genome-wide ancestry fractions in mothers and their embryos. We use these measurements to calculate the correlation coefficient between mating pairs $\rho$. Let $M$, $F$, and $O$ be minor-parent ancestry (birchmanni) fractions of a mother, father, and offspring, respectively. Then,

$$O=\frac{M+F}{2}+\epsilon ,$$ [11]

where $\epsilon$ is a noise term due to random segregation in the maternal and paternal meioses, with $\mathit{Cov}\left[M,\epsilon \right]=0$. From Eq. 11,

$$\mathit{Cov}\left(M,O\right)=\mathit{Cov}\left(M,\frac{M+F}{2}+\epsilon \right)= \frac{1}{2}\left[\mathit{Cov}\left(M,M\right)+\mathit{Cov}\left(M,F\right)\right]+\mathit{Cov}\left(M,\epsilon \right)=\frac{1}{2}\left[\mathit{Var}(M)+\mathit{Cov}\left(M,F\right)\right].$$ [12]

Using Eq. 12, the slope of the regression of offspring ancestry on maternal ancestry is

$${\beta }_{MO}=\frac{\mathit{Cov}(M,O)}{\mathit{Var}(\text{M})}=\frac{1}{2}\left[1+\frac{\mathit{Cov}\left(M,F\right)}{\mathit{Var}(M)}\right]=\frac{1}{2}\left[1+\frac{\mathit{Cov}\left(M,F\right)}{{\sigma }_M{\sigma }_F}\right]=\frac{1}{2}\left[1+\rho \right],$$ [13]

where ${\sigma }_M$ and ${\sigma }_F$ are the ancestry standard deviations of mothers and fathers, which we assume to be equal (such that $\mathit{Var}\left(M\right)={\sigma }_M{\sigma }_F$). Rearranging Eq. 13,

$$\rho =2{\beta }_{MO}-1.$$ [14]

Since some mothers have multiple embryos in the data of Powell et al. (47), we calculated ${\beta }_{MO}$ using the average ancestry fraction of each mother’s offspring, yielding an estimate of ${\beta }_{MO}=0.964$ (note that the same value would be obtained asymptotically if we averaged over iterations in which we randomly choose one embryo per mother and calculate ${\beta }_{MO}$ for the reduced dataset). From Eq. 14, our estimate of ${\beta }_{MO}=0.964$ corresponds to a correlation coefficient among mating pairs of $\rho =0.928$. This value is consistent with a 1.928-fold increase in purging of minor-parent ancestry in the generation after assortative mating [Eq. 5] and a 13.80-fold increase in long-term purging [Eq. 10].

Meta-analysis in birds.

Randler (50) carried out a meta-analysis of assortative mating in avian hybrid zones and found an average $z$-score of 0.44, which, by Fisher’s $z$-transformation, corresponds to a correlation coefficient of $\rho =0.44$ (33, 62). Randler’s (50) meta-analysis covered cases of assortative mating based on species-diagnostic phenotypes and on genetic ancestry. If the calculated correlation coefficient applies to ancestry-based assortative mating, then it is consistent with a 44% increase in purging of minor-parent ancestry in the generation after assortative mating [Eq. 5] and a 79% increase in long-term purging [Eq. 10].

Data Availability

Previously published data were used for this work(https://doi.org/10.1086/665993, https://doi.org/10.1111/evo.14337).