Postzygotic reproductive isolation established in the endosperm: mechanisms, drivers and relevance

Abstract

The endosperm is a developmental innovation of angiosperms that supports embryo growth and germination. Aside from this essential reproductive function, the endosperm fuels angiosperm evolution by rapidly establishing reproductive barriers between incipient species. Specifically, the endosperm prevents hybridization of newly formed polyploids with their non-polyploid progenitors, a phenomenon termed the triploid block. Furthermore, recently diverged diploid species are frequently reproductively isolated by endosperm-based hybridization barriers. Current genetic approaches have revealed a prominent role for epigenetic processes establishing these barriers. In particular, imprinted genes, which are expressed in a parent-of-origin-specific manner, underpin the interploidy barrier in the model species Arabidopsis. We will discuss the mechanisms establishing hybridization barriers in the endosperm, the driving forces for these barriers and their impact for angiosperm evolution.

This article is part of the theme issue ‘How does epigenetics influence the course of evolution?’

1. Introduction

Flowering plants, or angiosperms, are the most recently diverged clade of vascular plants, but with more than 300 000 species, they form the dominant group of plants on our planet [1,2]. The rise of flowering plants to ecological dominance in the early to Mid-Cretaceous has been intensively discussed and connected to the evolution of novel functional and physiological traits, including flowers and fruits, xylem vessels and faster growth rates [3–9]. One major innovation of flowering plants that has been largely neglected in this discussion is the evolution of the endosperm, an embryo-nourishing tissue that develops after fertilization [10]. In this review, we will focus on the potential role of the endosperm in promoting speciation by establishing hybridization barriers and illuminate the underlying molecular mechanisms as far as they are known to date.

The endosperm is the product of a double fertilization event, where one of the two sperm cells fertilizes the central cell, while the other sperm cell fertilizes the egg cell, initiating embryo formation. The formation of the endosperm is a distinctive feature of angiosperms; embryo nourishment in gymnosperms is mediated by the large female gametophyte [10]. Most higher-order flowering plants have a homodiploid central cell and form a triploid endosperm upon fertilization; however, the ancestral state is likely a haploid central cell and a diploid endosperm, as found in Nymphaeaceae and other basal angiosperms [10,11]. Increased maternal copy number in the endosperm has been proposed to facilitate maternal control over resource allocation to the developing progeny [12]. In support of this view, families with diploid endosperm, like Nymphaeaceae and Illiciaceae, have a very rudimentary endosperm and main resource accumulation occurs in the perisperm, a nutritive tissue derived from sporophytic tissues of the ovule [11,13].

Endosperm development of most flowering plants follows the nuclear type of development, where nuclear divisions are initially not followed by cell wall formation, leading to the formation of a coenocyte [14]. After a defined number of mitotic cycles, the endosperm cellularizes, followed by the differentiation of distinct tissue types [15–17]. The transition from the coenocytic to the cellular stage of endosperm development is an important transition and essential for embryo survival, for reasons that remain to be fully explored [18,19].

Aside from the most prominent nuclear type of endosperm development, some genera such as Solanum and Mimulus follow the cellular type of endosperm development, where mitosis and cytokinesis occur after the first division of the primary endosperm nucleus [14]. A minor fraction of families like Cabombaceae, Sabiaceae and Saxifragaceae follow the helobial type of endosperm development, where after an initial division of the fertilized central cell one cell follows the nuclear type of development while the other cell either remains undivided or also follows the nuclear type of development [14,20].

Failure in endosperm development is a frequent cause of seed arrest in response to hybridizations of related plant species and species that differ in ploidy [21–23]. The phenomenon of endosperm-based hybrid seed lethality is widespread among flowering plants. It is present in diverse taxa, evolves rapidly and manifests the key role of the endosperm in establishing hybridization barriers [22,24–30]. In the following, we will discuss the underlying mechanisms establishing endosperm-based hybridization barriers and their potential drivers.

2. The endosperm is a dosage-sensitive tissue

In most flowering plants, the endosperm is a triploid tissue, having two maternal and one paternal genome copies. This particular genome dosage is essential in many, if not most flowering plants to ensure viable embryo development [31–34]. Hybridizations of plants that differ in number of chromosome sets (i.e. ploidy levels) frequently result in seed arrest, a phenomenon termed the ‘triploid block’ [24,35–37]. In species with the nuclear mode of endosperm development, interploidy hybridizations affect the timing of endosperm cellularization. Crosses of maternal plants with higher ploidy pollen donors (referred to as paternal excess) cause a delay in endosperm cellularization, while reciprocal crosses (referred to as maternal excess) cause the opposite phenotype and lead to precocious cellularization [32,38,39] (figure 1). Also, species with the cellular mode of endosperm development show non-reciprocal effects on endosperm development, differing in number and size of endosperm cells [37,40] (figure 1).

Figure 1.

Endosperm defects in response to interploidy and interspecies hybridizations. Interploidy and interspecies hybridizations cause endosperm defects leading to embryo arrest. In species forming nuclear endosperm, paternal excess crosses delay endosperm cellularization, while maternal excess crosses lead to precocious endosperm cellularization. In species with the cellular type of endosperm development, paternal excess crosses lead to the formation of fewer and enlarged cells, while maternal excess crosses cause the formation of small endosperm cells. EBN, Endosperm balance number.

Similar to interploidy hybridizations, interspecies hybridizations also cause defects in endosperm development leading to seed lethality [22,23,25–28,41–44]. Depending on which species is used as maternal plant or pollen donor, non-reciprocal endosperm defects have been observed, with some species behaving like higher ploidy plants despite being diploid [42,43,45] (figure 1). This has led to the establishment of the endosperm balance number (EBN) concept, based on which every species has an effective ploidy that potentially differs from its actual ploidy. This effective ploidy is based on test crosses with defined species and is used to assess cross-compatibility with other species [46]. The EBN must be in a 2 : 1 maternal to paternal ratio in the endosperm for viable crosses.

One implication of the EBN concept is that interspecies and interploidy crosses likely have a similar molecular basis, an idea that is supported by findings showing that increasing the ploidy of one parent allows the generation of viable interspecies hybrids [27,43,47–49] (figure 2). This phenomenon likely explains the presence of gene flow between species that have strong hybridization barriers when crossed as diploids. For example, natural tetraploid Arabidopsis lyrata is able to form viable hybrid seeds with diploid Arabidopsis arenosa, while crosses between diploid species result in inviable seeds [43].

Figure 2.

Endosperm-based postzygotic barriers shape gene flow by building interploidy and interspecies postzygotic barriers. Recent polyploidization of species A (low endosperm balance number, EBN) results in a new tetraploid population (with increased EBN) that is reproductively isolated by the triploid block from the ancestral population. Diploid populations of species A are reproductively isolated from the outcrossing species B (high EBN), while the newly formed tetraploid species A may successfully hybridize with species B as a result of increased EBN.

3. Role of genomic imprinting in establishing hybridization barriers

Interploidy and interspecies crosses both cause abnormal seed phenotypes, which are dependent on the direction of the hybridization. This cross-direction dependency raised the hypothesis that imprinted genes could be involved in establishing hybridization barriers in the endosperm [12,50,51]. Genomic imprinting is an epigenetic phenomenon that modifies the expression of genes depending on their parent-of-origin. Imprinted genes are epigenetically modified in the gametes, mainly by DNA methylation and histone modifications. The established epigenetic pattern is maintained after fertilization, leading to parent-specific gene expression. In flowering plants, genomic imprinting is mainly confined to the endosperm and affects several hundreds of genes that are preferentially expressed either maternally or paternally (MEGs and PEGs, respectively) [52–54].

Genetic support for the connection between deregulated imprinted genes and interploidy barriers came with the discovery that mutants in several PEGs could suppress the triploid block in Arabidopsis [55–61]. Imprinted expression of PEGs depends on the Polycomb Repressive Complex2 (PRC2), a chromatin-modifying complex that silences target genes by applying a repressive histone modification. The maternal alleles of PEGs are specifically targeted and silenced by the PRC2, while the paternal alleles remain active [62–64]. The activity of the paternal allele of PEGs is likely a consequence of mechanisms causing resetting of repressive epigenetic modifications in sperm, allowing transcription factors to activate the paternal alleles of PEGs after fertilization [58,65]. Loss of PRC2 function in the endosperm causes the breakdown of PEG imprinting and a phenotypic mimic of paternal excess Arabidopsis seeds, supporting a central role of deregulated PEGs in the triploid block [55,66,67].

Thus far, a role for MEGs in establishing interploidy or interspecies barriers remains to be identified. However, circumstantial evidence suggests that MEGs have a role in both types of hybridization barriers. Mutations in the MEG MEDEA, which encodes a subunit of the PRC2, normalizes seed size in maternal excess interploidy crosses in Arabidopsis [68]. Furthermore, genetic loci with maternal parent-of-origin effects underpin hybrid seed lethality in crosses between Mimulus species, suggesting that MEGs are causally involved [30].

Genomic imprinting has likely evolved as a mechanism to silence transposable elements (TEs) [69–71]; therefore, parent-of-origin-specific expression of many genes is not necessarily functionally relevant. Nevertheless, for some genes, genomic imprinting confers an advantage and maintenance of imprinted expression is likely to be under selection. This molecular scenario of TEs driving genomic imprinting can explain the rapid turnover of imprinted genes over evolutionary time and the low number of conserved imprinted genes among flowering plants [72–75]. The rapid evolution of imprinted genes provides a rationale for the rapid establishment of hybridization barriers between species, as demonstrated in Capsella, Mimulus and Solanum, where closely related sympatric species are separated by strong endosperm-based barriers [28,30,42,76,77].

4. Genetics of the interploidy barrier in Arabidopsis

The PEG PHERES1 (PHE1) encodes for a type I AGAMOUS-LIKE (AGL) MADS-box transcription factor that when mutated can suppress triploid seed inviability. PHE1 binds to the promoter region of many other PEGs, including many suppressors of the triploid block [58], suggesting that PHE1 acts upstream of the triploid block. Supporting this notion, increased dosage of PHE1 correlates with hyperactivation of suppressors of the triploid block [55,58,78]. Interestingly, the majority of suppressors that have been identified in Arabidopsis encode chromatin regulators that have functional roles in TE silencing or heterochromatin establishment [56,59,60,78–80]. This bears striking similarities to hybrid incompatibility in Drosophila, where hybrid incompatibility genes were found to encode dosage-sensitive heterochromatin-interacting proteins or components of the PIWI-interacting RNA pathway, which silences TEs [81–84]. Nevertheless, whether indeed TE derepression is causal for hybrid lethality remains to be established. In Drosophila, hybrid lethality caused by the heterochromatin-interacting proteins hybrid male rescue (Hmr) and lethal hybrid rescue (Lhr) is connected with TE derepression [81,82]; however, whether this is causal for the phenotype has been questioned [85]. Similarly, in Arabidopsis, the role of deregulated TEs in establishing the triploid block remains controversial and requires further investigation [60,78,80]. Increased dosage of the triploid block suppressor ADMETOS causes ectopic application of a heterochromatic histone modification on TEs in the endosperm of triploid Arabidopsis seeds. Genes flanking those TEs become highly overexpressed, possibly leading to triploid seed arrest [79]. Thus, dosage-sensitive chromatin-modifying complexes are causally involved in establishing postzygotic hybridization barriers in Arabidopsis and Drosophila, supporting the idea that the continuous arms race between TEs and their suppressors is a strong source for hybrid incompatibilities [86–88]. However, by which mechanism deregulated chromatin regulators cause lethality remains to be established.

5. Mechanistic similarities between interploidy and interspecies barriers

Interploidy and interspecies hybridizations cause similar developmental abnormalities of the endosperm, suggesting a common mechanistic basis. Notably, interspecies crosses resulting in paternal excess-like phenotypes in Arabidopsis, Capsella, Brassica, Solanum section Lycopersicon (wild tomatoes) and Oryza (rice) are accompanied by overexpression of several AGL Type I MADS-box genes in the developing endosperm [38,42,89–92], mimicking a pattern described for interploidy paternal excess crosses in Arabidopsis and rice [38,55,61,66,92,93]. Interestingly, deregulated AGLs are a common feature of incompatibilities between species having nuclear and cellular modes of endosperm development. The AGL PHE1 acts upstream of known suppressors of the triploid block [58], indicating that deregulated AGLs act on top of a cascade that establishes hybrid incompatibility. Furthermore, downstream pathways affecting cell-wall-modifying activities are similarly affected in Arabidopsis interploidy hybrid seeds and interspecies hybrid seeds of Arabidopsis, Capsella and wild tomatoes [42,55,90], arguing for a signalling pathway converging on similar downstream targets. This pathway likely involves auxin, since auxin signalling is similarly affected in interploidy and interspecies paternal excess seeds in Arabidopsis, as manifested by increased auxin response factor (ARF) expression levels [94,95].

Interestingly, auxin signalling is decreased in paternal excess interspecies hybrid seeds of wild tomatoes, consistent with decreased endosperm proliferation in paternal excess wild tomato seeds [44,90]. Similarly, decreased endosperm proliferation was reported for paternal excess interspecies hybridizations in Mimulus and paternal excess interploidy hybridizations in wild potato species [28,40], which like tomato have a cellular mode of endosperm development. It thus seems that in species with cellular mode of endosperm development, paternal excess interploidy and interspecies hybridizations suppress auxin signalling and reduce endosperm proliferation.

6. Role of auxin in building reproductive barriers

The Arabidopsis auxin biosynthesis genes YUC10 and TAR1 are PEGs and direct targets of PHE1, implying that increased auxin biosynthesis is a direct consequence of PHE1 overexpression [58]. Similarly, in rice, increased expression of the PHE1 orthologues MADS78 and MADS79 causes perturbed auxin homeostasis and delayed endosperm cellularization, suggesting similar regulatory circuits act in monocots [96].

Auxin biosynthesis is required for endosperm development by promoting the proliferation of nuclei [97]. Auxin levels furthermore determine the transition from the coenocytic to the cellular phase of endosperm development [95,98], a transition also defective in paternal excess interploidy and interspecies hybrid seeds [32,38,39,42]. Overexpression of auxin biosynthesis genes in the inner layer of the seed coat causes a similar paternal excess phenotype to overexpression of auxin biosynthesis genes in the endosperm, suggesting a negative feedback of auxin-induced seed coat growth on endosperm cellularization [95]. In support of this notion, the transparent testa glabra2 (ttg2) mutant has reduced integument cell elongation and precocious endosperm cellularization and acts as maternal suppressor of the triploid block [99,100]. Similarly, ttg4, defective in the enzyme chalcone synthase (CHS), is a maternal triploid block suppressor [101]. Both TTG2 and TTG4 are part of the flavonoid pathway, which produces flavonoids that, after oxidation, confer the brown colour of the seed coat in Arabidopsis and other angiosperms [102]. Flavonoids have been proposed to regulate auxin transport [103], linking flavonoids, auxin and the triploid block. Thus, altered auxin biosynthesis in the endosperm of triploid seeds causes altered auxin accumulation and growth in the seed coat, which affects endosperm cellularization. This scenario provides a possible explanation for the observed non-reciprocal effects of interploidy and interspecies crosses on seed coat development in Primula, Brassica and wild tomatoes [24,44,45,91].

7. Drivers of postzygotic barriers in the endosperm

Hybrid incompatibilities have been proposed to evolve as a consequence of interspecies divergence between selfish DNA elements and their regulators [86–88]. Thus, the genomic conflict between TEs and their repressors is considered a potent driver of postzygotic barriers [86–88]. Reduced DNA methylation in the endosperm [104–106] may render the endosperm particularly vulnerable for genomic conflict, providing an explanation for the preference of chromatin regulators among suppressors of the triploid block [56,59,60,78–80].

The conflict between maternally and paternally derived alleles (referred to as parental conflict, or kin conflict) is another potential driver of postzygotic barriers manifested in nourishing tissues of plants and animals [28,90,107–109]. Parental conflict can arise in polyandrous species because maternal and paternal parents differ in the investments of resources allocated to the offspring. Since only the maternal parent provides nutrients to the developing progeny, while there are no costs on the paternal side, genes of paternal origin are selected to increase resource allocation to the offspring. By contrast, the same or different genes when maternally inherited are under selection to equalize nutrient transfer [12,108,110]. In consequence, a co-evolutionary arms race initiates between paternally expressed loci promoting the nutrient acquisition and maternally expressed loci suppressing the growth of the progeny. If in different populations different genes have evolved to control this process, hybridizations between these populations can result in hybrid growth defects and lethality. There are several examples showing that seed size is affected by the paternal genotype and that seed size increases with the grade of outcrossing of the pollen parent [111–113]. Furthermore, several examples have shown that crosses between self-pollinating (selfers) and outcrossing plants (outcrossers) lead to seed lethality; the defects manifested in the endosperm correspond to the expected direction assuming that outcrossers behave like parents with increased ploidy or high EBN [28,42,47,114] (figures 1 and 2). This has been conceptualized in the weak inbreeder/strong outbreeder (WISO) hypothesis, which states that crosses between selfers and outcrossers cause dosage imbalance in the hybrid endosperm, resulting in seed lethality [107]. Nevertheless, there are exceptions to this rule, where outcrossers have low EBNs, which is possibly a consequence of small population size and low genetic diversity [28,90,115,116]. The parental conflict could drive the evolution of hybridization barriers by enforcing the evolution of imprinted genes with nutrient-acquiring functions as well as genes limiting nutrient acquisition. Thus, one can postulate that imprinted genes involved in establishing hybridization barriers impact endosperm growth. There is indeed supportive evidence for several PEGs having growth-promoting functions in the endosperm. Triploid seeds derived from paternal excess crosses show increased endosperm growth and delayed endosperm cellularization, connected with increased PEG expression [32,38,39,55,117]. Mutants in several PEGs can suppress endosperm overgrowth and restore endosperm cellularization in Arabidopsis paternal excess seeds, supporting a role of PEGs as growth promoters in the endosperm [55–58,60,79]. PEGs are controlled by the PRC2, and interestingly, in Arabidopsis, two subunits of this complex are encoded by MEGs [118–120], supporting the concept of MEGs having growth-suppressing functions. Similarly in maize and rice, components of the endosperm-expressed PRC2 are MEGs [121,122]. Nevertheless, further functional studies of MEGs are required to test whether this concept holds.

8. Formation of polyploids and relevance of endosperm-based hybridization barriers

There are several pathways leading to the formation of polyploids; among those, the formation via unreduced diploid gametes is considered the most frequent route to polyploidy [29,123] (figure 3). The frequency of unreduced gamete formation differs between species and was shown to increase in response to heat and cold stress, which may explain the increased occurrence of polyploids within the Arctic [124–128]. The formation of polyploids has been proposed to occur via unstable triploid intermediates: a phenomenon termed the triploid bridge [29,129–131]. This path of polyploidy formation rests on the fact that a fraction of formed triploids can survive, as reported in many species [132–134]. Furthermore, in addition to the increased incidence of unreduced gamete formation under cold conditions [124,128], lower temperatures were also shown to alleviate postzygotic endosperm barriers [135], suggesting that specific climatic conditions promote the formation of polyploids via triploid intermediates. Another mechanism that has been proposed to give rise to polyploids is polyspermy, whereby two sperm cells fertilize the egg and thus bypass the triploid block [136–139] (figure 3). Nevertheless, the reported frequency of polyspermy-induced triploids in Arabidopsis is about 100-fold lower than the frequency of unreduced male gamete formation reported in Brassicaceae [136,140]. Furthermore, unreduced gamete formation is not restricted to pollen but also occurs in the egg at comparable frequency [29,123]; therefore, the frequency of potential unreduced gametes that can give rise to triploids is likely to be higher than currently estimated. Yet, comprehensive studies are required to establish the path and frequency of triploid formation in nature.

Figure 3.

Different routes giving rise to polyploid plants. The formation of polyploids can occur via the formation of unreduced male and female gametes, leading to the formation of triploid seeds that when overcoming the triploid block give rise to triploid plants. Triploids can also arise in consequence of polyspermy, which, if only the egg cell is fertilized by two sperm cells, will give rise to a viable triploid seed with a balanced triploid endosperm and a triploid embryo. Triploids can give rise to swarms of gametes with different grades of ploidy and thus act as a bridge to the formation of stable polyploids. m, maternal; p, paternal.

While triploids suffer from meiotic problems and are mainly sterile, they nonetheless can form gametes of varying ploidy grades, among them diploid gametes which when fused with each other can give rise to stable tetraploids [134] (figure 3). Reproductive isolation of newly established tetraploids prevents generating reproductively unfit triploids by backcrossing with diploid progenitors [130]. Niche separation, local pollen and seed dispersal and the transition to selfing are important factors facilitating tetraploid establishment [130,141,142]. Selfing increases the probability of successful matings during early stages of polyploid species establishment; however, enforcement mechanisms like the triploid block are likely required to ensure that predominantly selfing progeny is produced and unstable triploids aborted. The transition to selfing is generally followed by changes in flower morphology, enforcing selfing [143]. Nevertheless, before these changes are established, additional barriers preventing hybridizations of newly emerged self-fertilizers with their outcrossing relatives are likely promoting their establishment: a hypothesis that remains to be experimentally validated.

9. Conclusion

Accumulating evidence over the last century points that endosperm-based postzygotic hybridization barriers have a strong impact as drivers of angiosperm diversification. The formation of endosperm-based hybridization barriers is propelled by different conflicts, which promote the rapid evolution of speciation genes acting in the endosperm. Important gaps in our current knowledge that remain to be closed are the nature of the genes underpinning these barriers, their evolution and mode of action establishing these barriers. Furthermore, functionally connecting interploidy and interspecies barriers and testing the concept of a shared genetic basis are interesting avenues to be explored. Finally, assessing the contribution of these barriers to species divergence and the time of their establishment are areas of research that hold much promise for important discoveries.

Data accessibility

This article has no additional data.

Authors' contributions

C.K. wrote major parts of the manuscript with the support of G.D.T.-D.L. and K.D. G.D.T.-D.L. and K.D. generated the figures.

Competing interests

We declare we have no competing interests.