Evolutionary consequences of self-fertilization in plants

Abstract

The transition from outcrossing to self-fertilization is one of the most common evolutionary changes in plants, yet only about 10–15% of flowering plants are predominantly selfing. To explain this phenomenon, Stebbins proposed that selfing may be an ‘evolutionary dead end’. According to this hypothesis, transitions from outcrossing to selfing are irreversible, and selfing lineages suffer from an increased risk of extinction owing to a reduced potential for adaptation. Thus, although selfing can be advantageous in the short term, selfing lineages may be mostly short-lived owing to higher extinction rates. Here, we review recent results relevant to the ‘dead-end hypothesis’ of selfing and the maintenance of outcrossing over longer evolutionary time periods. In particular, we highlight recent results regarding diversification rates in self-incompatible and self-compatible taxa, and review evidence regarding the accumulation of deleterious mutations in selfing lineages. We conclude that while some aspects of the hypothesis of selfing as a dead end are supported by theory and empirical results, the evolutionary and ecological mechanisms remain unclear. We highlight the need for more studies on the effects of quantitative changes in outcrossing rates and on the potential for adaptation, particularly in selfing plants. In addition, there is growing evidence that transitions to selfing may themselves be drivers of speciation, and future studies of diversification and speciation should investigate this further.

1. Introduction

The transition from outcrossing to self-fertilization is considered the most common evolutionary transition in flowering plants, and it has occurred repeatedly in many independent lineages [1–3]. There are several reasons for self-fertilization (hereafter ‘selfing’) to be favoured in the short term. First, plants that undergo self-fertilization benefit from a 50 per cent transmission advantage over outcrossers, because they can contribute outcross pollen while also self-fertilizing their own ovules [4–6]. Self-fertilization can also be favoured under conditions when pollinators and/or mates are rare, because it offers reproductive assurance [7–9] and because it can result in improved colonization ability ([10]; but see [11,12]). The main force opposing this transition is generally considered to be inbreeding depression, the reduced fitness of inbred progeny relative to outcrossed progeny [13]. With prolonged selfing, purging may lead to reduced inbreeding depression [14,15]. In combination with deterioration of floral and genetic mechanisms for outcrossing, changes in sex allocation and reduced inbreeding depression makes reversions from self-fertilization to outcrossing increasingly unlikely over time [16,17].

Despite the high frequency of transitions to selfing, only 10–15% of seed plants are predominantly self-fertilizing [18]. Stebbins offered a key explanation for this apparent contradiction in 1957. In his now-classic paper, Stebbins [19] introduced the idea of self-fertilization as an evolutionary ‘dead end’. According to Stebbins' hypothesis, transitions from outcrossing to selfing are irreversible, and selfing taxa (‘selfers’) suffer an elevated risk of extinction owing to a reduced potential for adaptation. Thus, selfing evolves repeatedly owing to its short-term evolutionary advantages, but selfing lineages are rarely long-lived, and differences in net diversification rates between outcrossers and selfers explain the relatively low proportion of self-fertilizing plants overall. In support of this, Stebbins [19] noted that self-fertilizing species often possess remnants of floral structures devoted to outcrossing, and thus seem to be relatively recently derived from outcrossing ancestors. Furthermore, Stebbins noted the absence of large, long-lived clades consisting solely of self-fertilizing taxa, and interpreted this pattern to mean that self-fertilizing species do not have the same potential for diversification as outcrossing taxa.

Here, we review results relevant to the dead-end hypothesis of selfing that have accumulated since Takebayashi & Morrell's [20] excellent review of this topic. We first present recent phylogenetic studies of diversification rates, and then examine the possible causes of differences in extinction and speciation rates between selfers and outcrossers (figure 1). We give an overview of the current knowledge regarding accumulation of deleterious mutations in self-fertilizing lineages and finally, identify the potential for adaptation in self-fertilizing plants as a topic in need of further study.

Figure 1.

Processes affecting macroevolutionary diversification of outcrossers (top panel) and selfers (bottom panel), and factors that may enhance rates of transition. (a) Transitions from outcrossing to selfing are thought to be common, while the reverse is rare. (b) Selfing may often evolve as a speciation process, where the ancestral outcrossing lineage persists. (c) Speciation rates within outcrossers can be enhanced by higher rates of sexual and genomic conflicts, whereas selfers may experience accelerated rates of speciation owing to greater mating isolation and more rapid post-zygotic isolation owing to underdominant alleles. (d) Extinction rates in outcrossers can be enhanced by the lack of available mates, whereas extinction rates in selfers may be increased by reduced adaptive potential and by the accumulation of deleterious mutations.

2. Diversification rates in selfers versus outcrossers

The dead-end hypothesis of selfing posits that transitions from outcrossing to selfing are irreversible, and that selfing lineages are subject to higher rates of extinction than outcrossers. These hypotheses can be explicitly tested through macroevolutionary analyses of character evolution along reconstructed phylogenetic trees. So far, these methods have mainly been applied to test hypotheses on transitions from self-incompatibility (SI) to self-compatibility (SC), often with an underlying assumption that the transition to SC is correlated with an increase in selfing rates. Such macroevolutionary analyses can, however, be fraught with difficulties, especially for characters that may themselves affect rates of diversification, such as mating systems. Previous reports of reversions to SI [21,22] have, therefore, been criticized on methodological grounds [20,23]. One feasible approach to the problem of testing irreversibility of transitions to selfing is to infer ancestral character states based on shared ancestral polymorphism. Igić et al. [17] examined the loss of SI using the locus conferring SI (S-locus) to show that patterns at the Solanaceae S-locus are consistent with multiple losses and no gains of SI. This study also nicely demonstrated the pitfalls of some methods for testing the irreversibility hypothesis that rely on character states in extant taxa alone.

However, recent developments in this area hold the promise for more reliable tests of the dead-end hypothesis of selfing [23]. Of particular interest is a recently developed likelihood-based method that allows the joint estimation of rates of binary character change as well as state-dependent speciation and extinction rates (binary state speciation and extinction (BiSSE) [24]), given a phylogenetic tree and associated character states of terminal taxa. While large datasets are required for BiSSE [24], the method is less prone to falsely infer reversions and can, therefore, be used to test the irreversibility hypothesis [23]. It can also be used to estimate net diversification rates as well as state-dependent rates of extinction and speciation. However, as BiSSE and related methods are complex, it will be important to apply them carefully, and to explore possible biases and model specification issues using simulated data as well as empirical datasets where available. Indeed, it has already been shown that estimates of extinction rates in particular are noisy [24] and can be sensitive to model assumptions regarding rate constancy between lineages [25]. In general, conclusions are highly dependent on model assumptions about the constancy of rates of character transition, extinction and speciation across the phylogeny. Nevertheless, these methods are currently among the best for testing hypotheses related to selfing as a dead end within a macroevolutionary framework.

Using BiSSE, it was recently shown that net diversification rates for SI taxa exceed those of SC taxa in the Solanaceae family [26]. Interestingly, the rate of speciation of SC taxa was inferred to be higher than that of SI taxa, but the higher speciation rate was more than offset by an elevated extinction rate for SC taxa, yielding a lower net diversification rate for SC than SI lineages. The authors argue that species-level selection can account for the long-term maintenance of SI in the Solanaceae. Likewise, a study focusing on the effects of self-sterility (broadly defined as the lack of seed set after self-pollination) across a large set of angiosperm families found that self-sterile groups had higher species richness and net diversification rates than SC groups [27], and that this result was particularly pronounced for SI lineages.

The analysis by Ferrer & Good [27] also identified 22 families that potentially have multiple types of self-sterility, highlighting that transitions from SC to different forms of SI may not in fact approach zero, although some forms of self-sterility in this study may not be SI systems. More studies documenting the molecular basis of SI across a family are clearly needed; recent work in Leavenworthia [28] for example, suggests that the loss of the S-locus in this genus was followed by the regain of SI through paralagous copies of genes in the same gene family. The Leavenworthia study highlights the greater potential for reversibility to SI than traditionally expected, although it is difficult to completely rule out the possibility that there was a shift to a paralogous locus without a period of SC.

At present, it is unclear whether similar results showing greater diversification of outcrossing lineages will be obtained when using outcrossing rates, which may vary quantitatively over both space and time. It has been argued that the results of Goldberg et al. [26] may be largely driven by highly self-fertilizing SC species [29], however, this hypothesis has not yet been explicitly tested. Testing for an association between outcrossing rates and net diversification rates should at least in principle be possible using an extension of the BiSSE method for quantitative characters [30]. Approaches for estimating outcrossing rates using population genetic data [31], combined with high-throughput sequencing of large numbers of taxa, may allow for larger-scale quantification of mating system diversification.

With the possibility of examining the quantitative effects of outcrossing rate on diversification, a more detailed picture could emerge. In particular, although study bias may lead to overestimation of its occurrence [32], mixed mating, typically considered as outcrossing rates ranging from 0.2 to 0.8, is common [18,32], and the consequences of mixed mating for net diversification remain unknown. Whether mixed-mating systems represent a long-term evolutionarily stable strategy or a transitory phase during the shift to high rates of selfing is an area of intense study that has important implications when considering net diversification rates. If transitory, then the diversification rate of species with intermediate selfing rates may be irrelevant, because such species are in transition to high selfing rates. However, recent theoretical [33] and empirical [34,35] studies support the hypothesis that mixed mating may be stable over the long term. If intermediate selfing rates are stable, they may have very different, nonlinear effects on speciation and extinction. For example, one intriguing possibility is that mixed mating species have relatively high net diversification owing to maintaining advantages of outcrossing, while also retaining reproductive assurance conferred by SC [18].

In addition, considerations of stable-mixed-mating taxa also complicate arguments about irreversibility; the irreversibility of the transition to selfing is typically made on the theoretical grounds that inbreeding depression has been purged after high selfing [14], and that outcrossing mechanisms are complex and, therefore, difficult to re-evolve ([36]; but see discussion above). However, in mixed-mating taxa, shifts from relatively high selfing to relatively high outcrossing may occur frequently, owing to fluctuating environmental conditions (affecting inbreeding depression and/or pollinator availability). Given the lack of evidence for reduced inbreeding depression in mixed-mating populations relative to highly outcrossed taxa [34], and the possibility of standing genetic variation in floral traits affecting outcrossing, reversals may not be uncommon. Recent evidence of large-flowered, relatively outcrossing lineages with allelic variation that is a subset of that found in related small-flowered, relatively selfing lineages in Collinsia ([37,38]; figure 2) could reflect a reversal to high outcrossing rates. However, other possibilities, such as an ongoing spread of the selfing phenotype from an outcrossing lineage, need to be further tested in this case.

Figure 2.

Phylogeny of the self-compatible, mixed mating genus, Collinsia exhibits multiple shifts to high selfing from relatively high outcrossing and suggests that mating system may drive divergence in this group. Redrawn from Baldwin et al. [37], with flower sizes of the taxa indicated. Mean selfing rates associated with each species are from Kalisz et al. [35], with the exception of Collinsia rattanii, where phylogenetic and population genetic evidence places some populations of Collinsia linearis into the C. rattanii clade [37,38] indicating this clade is polymorphic for selfing rates and flower size. Asterisks (*) represent clade without posterior probability greater than 0.95.

Although less is known about the phylogenetic patterns associated with mating system evolution in animals, the results to date are also consistent with the hypothesis that selfing is typically derived and recently evolved [39,40], as has also been generally inferred in several phylogenetic analyses in other plant lineages (reviewed in [20]). Even with mixed-mating systems, high selfing appears to generally be the derived state at the tips of the phylogeny (figure 2).

3. Selfing as a speciation mechanism

Why might the rate of speciation be higher in selfing populations? One possibility is that there is a taxonomic bias towards describing more species in selfing lineages. This has probably occurred during the history of plant taxonomic description, because selfing groups typically exhibit higher levels of morphological differentiation among populations relative to outcrossers. However, there are also important biological reasons why the pattern of high diversification in selfing lineages may occur.

Research on the genetics and population genetics of species isolation provide support for the notion that selfing may promote speciation [41–44]. This is expected because the shift to selfing is associated with a suite of changes in floral morphology and reproductive investment (reviewed in [45]), and mating patterns [46] that can in turn affect the extent of hybridization in sympatry and gene flow with other populations.

Coyne & Orr [43] argued that, although selfing can indirectly promote reproductive isolating barriers, the evolution of selfing itself is not a direct isolating barrier, because selfing reduces mating among selfing individuals as much as it does to outcrossing progenitors. However, it has since been pointed out [44] that this view ignores the increased isolation experienced via pollen flow back to outcrossing progenitors. Furthermore, selfing reduces between-population gene flow via pollen; indeed, selfing populations generally show higher between-population differentiation [47,48]. Thus, selfers will effectively be more likely to be ‘allopatric’ without ongoing between-population gene flow [41], enhancing opportunities for the accumulation of reproductive isolation factors.

Similarly, rather than the evolution of selfing leading to a complete replacement of an ancestral outcrossing population, self-fertilization often evolves in peripheral populations at the edge of the geographical range, and may often be associated with a ‘budding’ process of speciation rather than more conventional lineage splitting as depicted in phylogenies [49–53]. Two of the speciation events with the shortest divergence times quantified to date are associated with a shift to selfing, including Capsella [51,52] and sea stars (Cryptasterina [54]). This suggests that mating system evolution can drive rapid speciation. Genus-wide quantification of outcrossing rates in Collinsia also suggests a recurrent pattern of speciation associated with mating system divergence in mixed-mating systems ([37]; figure 2).

Recent quantification of isolating barriers between the predominantly outcrossing Mimulus guttatus and the highly selfing Mimulus nasutus suggest that mating system shifts alone comprise a strong between-species isolating barrier [44]. Factoring out other sources of isolation, the authors estimated that mating system divergence alone caused at least a 98 per cent reduction in hybridization.

Several additional sources of isolation that were quantified separately, including reduced pollen production in the selfer and conspecific pollen precedence in the outcrosser, are also likely to be associated with mating system divergence. Thus, in this Mimulus species pair, mating system divergence appears to be a predominant factor driving speciation via prezygotic isolation.

If selfing evolves owing to its inherent short-term advantages, such as, e.g. reproductive assurance, increased premating isolation and elevated speciation rates in selfing lineages may simply result as a by-product. However, in a scenario with local adaptation, increased selfing may be selected for because it leads to reduced outbreeding depression: selfing could thus evolve through reinforcement [55,56]. This occurs because selfing reduces gene flow via heterospecific pollen and because it increases assortative mating [55]. Thus, in sympatric populations of related lineages that have hybrid inviability and/or fertility and share pollinators, there may be selection favouring higher selfing rates in one of the two lineages [55,57,58]. In Arenaria uniflora, high selfing rates occur only in regions overlapping with the distribution of Arenaria glabra [59]. Field experiments and hand pollinations showed that outcrossing A. uniflora experiences highly reduced fitness in the presence of A. glabra, primarily owing to heterospecific pollen precedence, consistent with the hypothesis that sympatric populations experience strong selection for high selfing rates to avoid hybridization. Similarly, in sympatric populations of Collinsia linearis and Collinsia rattanii, C. rattanii shows shifts in the timing of selfing, with increased rates of early selfing [60]. Furthermore, recent phylogenetic analyses suggested a consistent shift in flower size and morphology in sympatric populations of Mimulus, in accordance with the reinforcement hypothesis for the evolution of selfing [61]. Thus, selection for reinforcement provides another axis whereby mating system shifts may be important contributors to reproductive isolation and speciation.

Shifts to selfing may also promote post-pollination, pre-zygotic isolation. In SI lineages, there is a common pattern of unilateral incompatibility (UI), where hybridizations between SC species as paternal parent and SI lineages as maternal parent fail more often than the reciprocal cross owing to failure of pollen tube growth [62,63]. UI could be a factor promoting reproductive isolation between selfing and outcrossing lineages. While the UI reaction is primarily observed in SI × SC crosses in some lineages [62], in others pollen tube incompatibilities are found nearly as often in SI × SI crosses [63], and may thus also promote reproductive isolation between outcrossing lineages. However, more work is needed to quantitatively assess the contribution of UI to reproductive isolation across mating systems.

In addition, selfing may promote post-zygotic isolation [64]. First, shifts in floral morphology between selfers and outcrossers can cause aberrant reproductive phenotypes in hybrids, potentially leading to rapid post-zygotic isolation [64]. Second, because of higher homozygosity, selfers are more likely to fix underdominant alleles or chromosome rearrangements that have reduced fitness as heterozygotes [65], which can contribute to post-zygotic isolating barriers with progenitor outcrossers or with sister selfing taxa [66]. For example, large numbers of cryptic species have been identified in the predominantly selfing Draba [67], and reproductive isolation in this group seems to be caused in part by underdominant alleles [66]. Evidence for high levels of segregating post-zygotic fitness effects in Arabidopsis thaliana, both epistatic [68] and underdominant [69], even in very nearby geographical isolates, also highlights the potential for selfers to accumulate reproductive isolating factors. In addition, if highly selfing populations typically purge a considerable amount of inbreeding depression [34], it is possible that hybridization with outcrossing progenitor populations leads to reduced fitness through the reintroduction of deleterious recessive alleles that are re-exposed in homozygous form via selfing. The contribution of this effect to plant speciation remains an open empirical and theoretical question, but the high levels of standing inbreeding depression in many obligate outcrossing populations implies that selfing/outcrossing hybrids may experience strongly reduced fitness owing to this effect alone.

Other possible sources of post-zygotic isolating barriers include reduction of genomic conflicts in selfers [70]. Selfish genetic elements that have coevolved with their host genome in outcrossing populations may confer strongly reduced fitness in a ‘naive’ highly selfing genome. One possible example of this is the strong reduction in hybrid fertility between M. guttatus and M. nasutus owing to a cytoplasmic male sterility factor and a nuclear restorer allele segregating in the outcrossing species but absent in the selfing lineage [71].

Reduction of sexual conflict in selfing lineages may also lead to post-zygotic isolation between selfers and outcrossers [72]. In outcrossers, paternal alleles are expected to promote endosperm proliferation, whereas maternal alleles suppress it. When selfing evolves, the interests of maternal and paternal alleles are shared and this coevolution is lost. An expected consequence is aberrant endosperm development upon hybridization across mating systems [72]. There is some support for this hypothesis [72], and it may contribute to patterns of elevated pollen gene flow from outcrossing to selfing lineages [73,74]. However, more microevolutionary studies addressing the extent and nature of post-zygotic isolation among selfing and outcrossing populations are needed to better understand the interplay between genomic conflicts, mating systems and reproductive isolation.

If the evolution of selfing is itself frequently a speciation mechanism, this adjusts considerations about consequences for macroevolutionary diversification. In particular, the BiSSE model has commonly assumed that character transitions are within-lineage changes, rather than changes associated with a speciation event. Considering the possibility that mating system shifts may themselves be speciation mechanisms can affect inferences about speciation and extinction, and methods are currently being developed that allow for character transitions to occur at speciation events [75,76]. Goldberg & Igić [75] have recently modified their approach to analysing the Solanaceae dataset, allowing for this possibility. Interestingly, transitions to SC that include speciation events were inferred to occur 10 times more frequently than within-lineage transitions, however, the basic conclusions about high speciation and extinction rates in SC species remain intact.

Goldberg and Igić's recent results nevertheless highlight the potential for selfing to promote speciation. Indeed, almost 15 per cent of speciation processes in SI Solanaceae species involved mating system transitions. Combined with the higher rate of speciation inferred in SC species, this suggests that 67 per cent of speciation events involve the formation of new SC species. Furthermore, while selfing may generally be an evolutionary dead end, lineages that experience a high transition rate from outcrossing to selfing may generally have higher diversification rates.

4. The causes of higher extinction rates in selfers

What then could cause elevated extinction rates in selfers? Stebbins [19] suggested that selfing lineages should be unable to persist in the long term, because they suffer from a reduced potential for adaptation. Indeed, as we discuss below, selfing has numerous population genetic and ecological consequences that can affect the efficacy of both positive and purifying selection.

A key effect of selfing is that it leads to a reduction in the effective population size. First, the transition to selfing immediately leads to a twofold reduction of the effective population size, as the number of independently sampled gametes is reduced [77,78]. Second, selfing can be associated with an increased frequency of founder events, as a single selfing individual can form new populations [10]. Population structure and extinction, and recolonization dynamics [45] are also expected to lead to reduced total genetic diversity in many selfing lineages. Finally, higher homozygosity in selfers increases linkage disequilibrium, leading to greater effects of linked selection, including hitchhiking [79], background selection [80,81] and interference among weakly selected mutations [82]. The effects of such linked selection can further reduce the effective population size in selfers [83].

Reduced effective size directly affects the efficacy of selection, which depends on the product of the effective population size and the selection coefficient, N_es [84]. This can render slightly deleterious mutations effectively neutral, causing elevated rates of fixation of such mutations. Finally, in selfers, the accumulation of deleterious mutations can occur through the chance loss of mutation-free genotypes through Muller's Ratchet [85,86]. Selection on slightly deleterious mutations is, therefore, expected to be less efficient in selfing taxa [83]. This effect is expected to be detectable as an elevated ratio of non-synonymous to synonymous divergence in selfing lineages, or as an excess of high-frequency non-synonymous mutations, if the shift to selfing was recent [87].

The expectation that selfers should exhibit lower neutral genetic variation is well supported by empirical studies [51,52,88,89]. However, until recently, support for the accumulation of deleterious mutations in selfing lineages was much more limited. For instance, there is little evidence for a relaxation of selection on non-synonymous mutations in association with the shift to selfing in the Triticeae [90,91]. Early work also found little evidence for relaxation of codon usage bias or relaxed selection on non-synonymous mutations in the selfer A. thaliana in comparison with its close outcrossing relative Arabidopsis lyrata [92,93]. However, recent studies that used larger datasets and more sophisticated analysis methods have found more evidence for a relaxation of selection in selfers. For instance, Qiu et al. [94] found reduced codon bias in selfing Arabidopsis and Capsella species, and Slotte et al. [95] found a lower efficacy of purifying selection on non-synonymous mutations in the selfer A. thaliana than in the outcrosser Capsella grandiflora [95]. Population genomic data using next-generation sequencing have provided evidence for a global reduction in the efficacy of selection in the recently derived selfer Capsella rubella [96], C. rattanii [38] and in three independently derived selfers in the genus Eichhornia [97]. Subtle but significant reductions in codon bias have also been documented in selfing nematodes compared with outcrossing taxa using large-scale genome-wide analyses [39]. While it is difficult to rule out effects of other confounding variables in comparisons of pairs of taxa, these results suggests that large datasets and analysis methods that account for both positive and purifying selection using patterns of both polymorphism and divergence [98] may be required to detect relaxation of selection in association with the shift to selfing.

An outstanding question in this respect is why the effects of the transition to self-fertilization are so subtle. One frequently invoked explanation is that if the shift to selfing occurred recently, divergence-based methods to detect relaxation of selection have limited power [83,87,91]. If this is the case, polymorphism data may afford higher power to detect relaxation of selection [87]. An alternative explanation is that if a large proportion of amino acids in nuclear genes are fixed by positive selection in natural outcrossing plant populations [95], then approaches focused on testing for elevated rates of non-synonymous substitution in selfing lineages may not be very powerful for quantifying relaxed selection.

Another possible explanation for this apparent conundrum could be that most predominantly self-fertilizing plants undergo sufficient outcrossing (i.e. are mixed mating) to effectively slow the accumulation of deleterious mutations [80]. Indeed, marker-based estimates of self-fertilization rates suggest that very few if any plant species are completely selfing [18,34,99]. Genomic data support this idea and show that linkage disequilibrium in global samples of the predominantly selfing plant species A. thaliana and Medicago truncatula decays within 5–10 kb, (i.e. on a similar scale to humans [100,101]. Thus, on a global scale, linkage disequilibrium may not be sufficient to reduce the efficacy of selection in some predominantly selfing taxa. However, in local populations, linkage disequilibrium can be much more extensive [100,102] and selection efficacy within these populations may indeed be reduced [101].

When considering the effects of selfing on extinction risk, an important open question concerns our expectations for mixed-mating populations. While the rate of selfing has a linear effect on effective population size under neutrality [78], selective interference effects can sometimes be nonlinear [103,104], such that partially outcrossing populations may experience limited effects of hitchhiking on reducing effective population size. In fact, analysis by Glemin & Ronfort [104] indicates a very strong threshold effect of selfing rate on extinction risk owing to reduced adaptation, and in a number of cases intermediate selfing rates show the lowest extinction risk. On the other hand, shifts in life history may mean that elevated selfing rates may still be accompanied by increased risk of severe founder events, in which case greater reductions in effective population size and selection may be observed than expected because of demographic factors. Recent evidence for strong reductions in diversity and the efficacy of selection in the relatively selfing C. rattanii (outcrossing rate = 0.12) compared with its mixed-mating sister species C. linearis (outcrossing rate = 0.57) [38] are consistent with this latter possibility.

Furthermore, if deleterious mutations are mostly recessive, relaxed selection owing to reduced N_e may be counteracted by the increased efficacy of selection on deleterious recessive mutations in selfing lineages [87], although such purging will be less effective for more additive, weakly deleterious mutations [105]. Such a ‘homozygosity effect’ may affect patterns of polymorphism, but would have little effect on divergence [87]. In any case, given that effects of selfing on the accumulation of deleterious mutations appear to be modest, it seems fair to question whether accumulation of deleterious mutations is likely to be a major contributor to elevated rates of extinction of selfing lineages [91].

According to Stebbin's original formulation of the dead-end hypothesis, selfers should suffer from higher rates of extinction because they harbour less genetic variation and thus are unable to adapt to environmental change. This verbal model has recently been explored in a theoretical framework [104]. In this paper, the authors explore the dynamics of adaptation in outcrossers and selfers under a simple one-locus model. Interestingly, the authors also incorporate extinction as a possible consequence of lack of adaptation, and show that under frequently changing environmental conditions, selfing lineages are expected to experience increased extinction rates because they fail to adapt [104].

Coevolution with pathogens or parasites may constitute an example of such a scenario with a constantly changing environment under which selfers will fare badly. In this case, the effect of selfing on effective recombination rates could exacerbate the situation. For instance, under a Red Queen scenario, where there is coevolution between hosts and pathogens or parasites, outcrossing is beneficial because sex and recombination results in novel resistant genotypes [106]. It therefore follows that high selfing can have a negative impact on the response to pathogen pressure. As Red Queen type dynamics are likely to be very common [107], this effect could potentially have a major impact on extinction rates in selfing plant lineages. In agreement with this, there is some evidence that coevolution with pathogens may select for outcrossing in plants in the wild [108] and in a literature survey of a wide range of seed plants, outcrossing rates were positively correlated with fungal pathogen pressure [109]. While there is no similar work on plants, experimental evolution studies in Caenorhabditis have provided empirical support both for reduced potential for adaptation and deleterious mutation accumulation in selfers [110], and an increased probability of extinction of selfers under coevolution with pathogens [111]. These experiments show that theoretical expectations do seem to hold up under controlled laboratory conditions and that coevolution with pathogens may contribute to elevated extinction rates of selfing lineages.

However, there are also aspects of selfing that may counteract the effects of a reduced effective population size on adaptation. For instance, gene flow between populations is often reduced in selfers. Reduced gene flow among populations may facilitate local adaptation, and this may explain why a recent literature survey found no difference in the degree of local adaptation between selfers and outcrossers [112], because decreased between-population gene flow may roughly balance any reduced adaptive potential.

Finally, demographic factors alone could be a primary driver of greater extinction rates in selfing lineages. If selfing populations tend to persist in smaller local populations and be more often subject to extinction/ recolonization dynamics [8,113], then increased extinction risks through demographic stochasticity [114] are expected. On the other hand, the ability of selfing populations to reproduce, colonize and expand under conditions of low density and pollen limitation can also enhance their persistence, increase range size [115] and contribute to reduced extinction rate [113,116]. Direct comparisons of population dynamics of closely related selfers and outcrossers are needed to better understand the direct interplay between-populations' demographic characteristics and extinction risks.

5. Conclusions and future directions

The dead-end hypothesis of selfing posits that transitions from outcrossing to selfing are irreversible and that selfing lineages suffer from increased rates of extinction. Several aspects of this hypothesis are supported by recent phylogenetic studies on the loss of SI, but the effect of quantitative changes in outcrossing rates has not been tested. The causes of elevated extinction rates in selfers remain unclear, and while there is growing evidence for accumulation of deleterious mutations in selfing crucifer taxa, these effects are subtle and may be unlikely to contribute to elevated extinction rates. More studies of the effect of differences in adaptive potential depending on mating system should be undertaken. Further, studies addressing the role of population size and the relative demographic stability of selfing versus outcrossing taxa are needed. Finally, the transition to selfing may itself constitute an important speciation mechanism, and a better understanding of the interplay between mating system and reproductive isolation is important to better understand the macroevolutionary effects of self-fertilization.