Dioecy in a wind‐pollinated herb explained by disruptive selection on sex allocation via inbreeding avoidance

Summary

• The evolution of dioecy from hermaphroditism is widely thought to be a response to disruptive selection favoring males and females, driven by advantages of inbreeding avoidance, sexual specialization, or both. It has hitherto been difficult to uncouple the importance of these two hypotheses.

• We use phenotypes produced by experimental evolution to test the inbreeding avoidance hypothesis in populations from which sexual specialization can be effectively ruled out. We estimate the selfing rate and the shape of fitness gain curves under scenarios with and without inbreeding depression in experimental populations of wind‐pollinated Mercurialis annua with high variation in sex allocation.

• Our results confirm a phenotypic trade‐off between male and female allocation in M. annua. Individual selfing rates increased with pollen production. This dependence led to strong disruptive selection on sex allocation due to its interaction with the mating system under the scenario of high inbreeding depression, especially for plants of medium and large sizes.

• Taken together, our results demonstrate that inbreeding avoidance on its own can lead to disruptive selection on sex allocation, favoring the selection and maintenance of dioecy under wind pollination without associated benefits of sexual specialization.

Introduction

Why hermaphrodites should ever evolve toward dioecy has long intrigued evolutionary biologists (Darwin, 1877; Charnov et al., 1976; Bawa, 1980; Thomson & Brunet, 1990; Renner & Ricklefs, 1995; Freeman et al., 1997; Ashman, 2006; Käfer et al., 2017). Two overarching explanations have been suggested. The ‘inbreeding avoidance’ hypothesis posits that unisexuality provides a fail‐safe means of avoiding self‐fertilization and the accompanying deleterious effects of inbreeding depression (Mather, 1940; Lewis, 1941; Charlesworth & Charlesworth, 1978). By contrast, the ‘sexual specialization’ hypothesis recognizes that the separation of sexes into different individuals allows males and females to express different trait values that optimize their respective fitness (Willson, 1979; Givnish, 1980), with the resulting sexual dimorphism resolving the sexual conflict and interference that may compromise fitness in simultaneous hermaphrodites (Abbott, 2011; Schärer et al., 2015). These two explanations for dioecy are not mutually exclusive: populations may initially evolve dioecy in response to selection for inbreeding avoidance, but then subsequently evolve sexually dimorphic traits that confer on each of the two sexes benefits of specialization (Freeman et al., 1997; Charlesworth, 2018). Alternatively, dioecy may evolve gradually from hermaphroditism through divergence in sex allocation and sexual specialization jointly (Lloyd, 1980a; Lesaffre et al., 2024b). However, it has hitherto been difficult to determine the relative importance of the advantages of inbreeding avoidance vs sexual specialization for the evolution and maintenance of dioecy.

Evaluating the relative advantages of separate vs combined sexes in plants has been difficult because it requires comparing fitness among males, females, and hermaphrodites with a range of different sex‐allocation strategies in the same context, yet dioecious species typically comprise only males and females, and hermaphrodites are typically canalized in their sex allocation. While the males and/or females of some dioecious species show a degree of ‘leaky’ or inconstant sex expression, allowing some level of comparison among sex‐allocation strategies (Delph & Wolf, 2005; Ehlers & Bataillon, 2007; Käfer et al., 2022), the sex allocation of these leaky phenotypes is typically very close to that of the corresponding ‘constant’ (or pure) males or females (Cossard & Pannell, 2019). Because we lack the kind of natural variation within populations that is needed for comparisons of the fitness of phenotypes across a wide range of alternative sex‐allocation strategies, it is typically not possible to estimate the shape of ‘fitness gain curves’ on which much of the theory for the evolution of dioecy vs hermaphroditism is based, that is, the degree to which allocation to each sex function translates into fitness (Charnov et al., 1976; Charlesworth & Charlesworth, 1981; Charnov, 1982; Charlesworth & Morgan, 1991; Campbell, 2000; Zhang, 2006; West, 2009; Fromhage & Kokko, 2010; Dorken & Van Drunen, 2018; Masaka & Takada, 2023; Lesaffre et al., 2024a; but see Dorken & Barrett, 2004; Dorken & Mitchard, 2008; Perry & Dorken, 2011).

Consideration of the shape of gain curves has yielded valuable insights into when we should expect selection to favor combined vs separate sexes. Dioecy is predicted to be evolutionarily stable when pure males and females have greater fitness than any intermediate (hermaphroditic) sex‐allocation strategy such that selection on sex allocation is disruptive and the curve relating the combined fitness through both male and female functions is U‐shaped (Fig. 1); such a scenario might be revealed, for example, by a positive quadratic coefficient in a second‐order polynomial in regression analysis (Lande & Arnold, 1983). By contrast, hermaphroditism should be stable to the invasion of sexual specialists (males or females, or individuals with highly male‐biased or female‐biased sex allocation) if selection is stabilizing and the fitness gain curve is n‐shaped (potentially revealed by a negative quadratic coefficient, for instance; Fig. 1; Lande & Arnold, 1983). These ideas were applied to theory for the maintenance of dioecy vs hermaphroditism under the assumption of complete outcrossing of all phenotypes (e.g. hermaphrodites that are self‐incompatible; Charnov et al., 1976; Lloyd, 1984), but also to the case where hermaphrodites are partially self‐fertilizing, with the fitness of selfed progeny potentially diminished by inbreeding depression (e.g. Lloyd, 1975; Charlesworth & Charlesworth, 1978; Charlesworth, 1984 − the evolution of gynodioecy and androdioecy).

Fig. 1.

Conceptual diagram showing different selection schemes via total fitness on sex allocation (gender) within a population. Here, sex allocation represents how the resource of reproduction is divided into female and male functions in an individual. Unisexual female and male individuals are favored when the selection on sex allocation is disruptive (depicted by the solid line), rendering the population dioecious. By contrast, when the selection on sex allocation is stabilizing (depicted by the dashed line), hermaphroditism is favored.

Models of the effect of selfing on the evolution of dioecy have most typically assumed a parametric fixed selfing rate independent of the distribution of sex allocation (Lloyd, 1975; Charlesworth & Charlesworth, 1978; Charlesworth, 1984), yet the selfing rate might often depend on floral strategies, including relative allocation to male and female functions (Denti & Schoen, 1988; Damgaard & Abbott, 1995; Chen & Pannell, 2024). In particular, hermaphrodites allocating more to their male function might be expected to self‐fertilize more of their seeds, either because pollinators of animal‐pollinated species stay longer on plants with large floral displays (Karron et al., 2009; Christopher et al., 2021), or as a result of a simple ‘mass‐action’ process (Gregorius et al., 1987; Holsinger, 1991), whereby the probability of selfing depends on the proportion of self‐pollen carried by the pollen vector for a given plant. This is likely to be a good description of the process of pollination in wind‐pollinated species, where an individual's greater allocation to its male function may increase its outcross siring success but also its rate of self‐pollination because of the greater proportion of self‐pollen in the pollen cloud around its stigmas (Gregorius et al., 1987; Denti & Schoen, 1988; Holsinger, 1991). In self‐compatible species, the increased rate of self‐pollination should be expected to compromise female reproductive success through self‐fertilization and the potentially accompanying effects of inbreeding depression as a result of ovule or seed discounting (Lloyd, 1992; de Jong et al., 1999). Accordingly, when inbreeding depression is substantial, individuals allocating all their reproductive resources to either the male or female function should have higher fitness than those adopting a hermaphroditic strategy (Fig. 1), leading to a u‐shaped gain curve and the evolutionary stability of dioecy (de Jong et al., 1999; de Jong & Geritz, 2001).

The direction of selection on sex allocation is also predicted to depend on relative plant size (de Jong et al., 1999; de Jong & Geritz, 2001), including through effects on the selfing rate. When selfing is increased by male allocation, disruptive selection should be strongest for the relatively larger individuals in a population, which produce more pollen in absolute terms. This is because larger individuals should have higher selfing rates for a given relative male allocation, with a steeper dependence of the selfing rate on sex allocation (Fig. 2). By contrast, disruptive selection might be expected to be lowest for small individuals, which contribute relatively less to the pollen cloud around their stigmas and thus self‐fertilize less (Fig. 2). If so, small individuals might even experience stabilizing selection on their sex allocation in populations in which sex allocation in larger individuals otherwise favors the male and female extremes (de Jong & Geritz, 2001). In other words, we might expect disruptive selection in a population to dominate (because the large individuals contribute disproportionately to the next generation), even though small individuals may experience stabilizing selection. To our knowledge, this possible size dependence of the selfing rate has not been modeled formally or characterized empirically, but it seems likely in wind‐pollinated populations with mass‐action mating.

Fig. 2.

The expected effect of plant size on the selfing rates under the mass‐action assumption (see main text for details). Given the same sex allocation (gender), a large plant (shown in blue) will produce more male flowers in absolute terms compared to a small plant (shown in orange). Thus, the expected selfing rate under the mass‐action assumption, which is positively correlated with male flower production, will be higher in the larger plant.

For empirical tests of theory for the stability of dioecy, we ultimately need to compare the fitness of individuals adopting a range of alternative sex‐allocation strategies within a population, from fully male through hermaphroditism (ideally with a range of different sex allocations) to fully female (Fig. 1). However, most plant species tend to be either dioecious or hermaphroditic, and species with intermediate sexual systems, such as gynodioecy or androdioecy (where males or females, respectively, coexist with hermaphrodites), only allow comparisons between unisexuality and bisexuality for one of the sexual functions (Fritsch & Rieseberg, 1992; Weller & Sakai, 2005; Spigler & Ashman, 2012; Varga, 2021; Laugier et al., 2023). Although comparisons among related species or populations with different sexual systems can provide valuable information (Dorken et al., 2002; Steven & Waller, 2004; Sakai et al., 2006; Soza et al., 2012; Kwok & Dorken, 2022), they are often compromised by confounding trait variation and ecologies, and they do not replace comparisons among strategies within the same context. Comparisons among individuals expressing different sex‐allocation strategies within the same population can be achieved by physical manipulation of the plants, for example, through the removal of floral parts or flowers (Emms, 1993; Tomaszewski et al., 2018; Aljiboury & Friedman, 2022; Larue & Petit, 2024; Chen & Pannell, 2025), but such manipulations are often difficult to achieve realistically and may not generate phenotypes that would occur in nature. The dearth of standing variation in sex allocation for testing patterns of selection within a population has therefore been a barrier to studies of the evolution and stability of separate sexes (but see the studies using F₂ crosses between populations of different sexual systems in Sagittaria latifolia, Dorken & Barrett, 2004; Dorken & Mitchard, 2008; Perry & Dorken, 2011).

Here, we test the inbreeding avoidance hypothesis for the evolution of dioecy by studying the fitness of phenotypes of the self‐compatible, wind‐pollinated, dioecious annual plant Mercurialis annua across the full range of sex allocation, from male to female (Fig 3, Supporting Information Fig. S1), and without confounding factors of sexual specialization (dimorphism). The phenotypes used in our study were generated over several generations of experimental evolution of females from initially dioecious populations from which males were removed; these females rapidly evolved substantial male allocation through the enhanced ‘leaky’ expression of male flowers (Cossard et al., 2021; Gerchen et al., 2024; Villamil et al., 2024). Critically, because males had been removed from these populations, they lacked the strong sexual dimorphism in vegetative traits that otherwise characterize natural populations of dioecious M. annua (see supplementary analyses in Table S1), and varied largely only in their sex allocation and, as our study demonstrates, in their selfing rates. Thus, although natural populations of dioecious M. annua are sexually dimorphic for several traits (Harris & Pannell, 2008; Tonnabel et al., 2019, 2022), which likely contributes to the evolutionary stability of dioecy, our model populations here allowed us to focus attention specifically on the inbreeding avoidance hypothesis.

Fig. 3.

Plots showing the distribution of gender phenotypes in the studied populations (a), and the trade‐off between female and male flower numbers of the individuals (b), of Mercurialis annua. In (a), a value of zero in gender indicates an individual with only female flowers, whereas a value of one indicates an individual with only male flowers. In (b), the trade‐off lines for plants of different sizes were estimated by the generalized linear mixed model, with the shaded ribbons indicating the 95% confidence interval of the corresponding regression lines. Hereafter, to present the interactive effect with plant size, the regression lines for three levels of size (small, medium, and large) are shown, reflecting plants of mean size minus SD, mean size, and mean size plus SD, respectively. Note that one raw data point with an extreme number of female flowers (2108 female flowers) is not shown to avoid compression of the y‐axis (see also Supporting Information Fig. S1).

Firstly, we confirmed the existence of a sex‐allocation trade‐off between male and female functions (Gerchen et al., 2024), the fundamental assumption of sex‐allocation theory on which its predictions are based (Charnov, 1982; West, 2009). We then evaluated the merits of the inbreeding avoidance hypothesis by asking (1) how the selfing rate depends on sex allocation and individual plant size and (2) how total fitness depends on the selfing rate of individuals with different sizes and sex‐allocation strategies under contrasting scenarios of inbreeding depression. For each individual and its sex‐allocation phenotype, we estimated male reproductive success on the basis of paternity assignment for a large sample of progeny. We estimated female reproductive success for all individuals in terms of a measure of seed production per individual for scenarios of contrasting inbreeding depression. With estimates of male, female, and total reproductive success, we finally inferred the shape of the respective fitness gain curves. Our study advances an understanding of the stability of dioecy to the invasion of alternative sex‐allocation strategies on the basis of fitness comparisons over a range of physiologically realistic expressions of sex allocation that, to our knowledge, have hitherto not been realized.

Materials and Methods

Plant materials and experimental populations

Mercurialis annua L. (Euphorbiaceae) is a wind‐pollinated annual herb widely distributed around the Mediterranean Basin and throughout central and western Europe (Tutin et al., 1976; Pannell et al., 2004). Natural diploid populations are dioecious, with individuals producing either female or male unisexual flowers, though leaky sex expression in both sexes (where individuals produce a small number of unisexual flowers of the opposite sex) is not uncommon (Cossard & Pannell, 2019, 2021; Villamil et al., 2022). To establish our experimental populations, we used seeds sampled from populations of females that had been evolving for eight generations in the absence of males (Cossard et al., 2021; Gerchen et al., 2024); while these females were phenotypically similar to those found in natural populations of diploid M. annua (see Table S1 for descriptions of the ancillary traits), they had evolved substantially enhanced ‘leaky’ production of male flowers that resemble in important respects monoecious individuals from polyploid individuals (Pannell et al., 2008), but with substantially greater variation in male‐flower production.

Seeds for our study were sown in seedling trays in July 2022 and grown for 5 wk in the glasshouse of the University of Lausanne, Switzerland (Cossard et al., 2021; Gerchen et al., 2024; Methods S1). In mid‐August, we set up three common‐garden populations as independent replicates in hexagon‐shaped plots, each comprising 61 individuals (Methods S1). The populations were set up at least 150 m apart from each other on the University campus, minimizing gene flow among them (the average outcrossing mating distance within the populations was 28.5 cm; Table S2). The seedlings were potted individually in 16 cm pots with soil (Ricoter substrate 140) and slow‐release fertilizer (Hauert Tardit 6 M pellets; 5 g l⁻¹ fertilizer applied to soil) and were automatically watered throughout the period of the experiment of c. 14 wk. We harvested the aboveground parts of the plants and measured their sex allocation and other traits in mid‐October when the night temperature dropped below 5°C and the plants ceased growing.

Phenotyping of sex allocation and biomass

To quantify the sex allocation of each individual, we counted the number of female ($N_{\mathrm{F}}$) and male ($N_{\mathrm{M}}$) flowers produced on the whole plant by detailed inspection throughout all the branches at the time of harvest. For each individual i, we further calculated its functional gender (G, hereafter gender) (sensu Lloyd, 1980b) in terms of maleness as $G_i=N_{\mathrm{M},i}/\left(E\times N_{\mathrm{F},i}+N_{\mathrm{M},i}\right)$, where the equivalence factor, E, is the ratio of the number of total male flowers to total female flowers in the respective population. The inclusion of E in the formula not only guarantees that the mean gender of the population is 0.5, reflecting the fact that exactly half of all genes passed to progeny are via male and the other half via female function, but it also allows measuring male and female allocation in any units (see Harris & Pannell (2008) for an example of estimates of allocation in dioecious M. annua in terms of biomass and nitrogen content). After the phenotyping, we kept all the harvested parts in bread bags to dry at room temperature for 3 wk. We then measured their biomass and gathered and counted mature seeds that had been dispersed from capsules into the bags.

Paternity analysis and the estimate of the selfing rate

To estimate the selfing rate of each individual, we used variation at nine microsatellite markers to assign paternity to mature seeds (modified from Machado et al., 2017). Leaf samples of the parental individuals were collected upon harvest and dried in silica gel before DNA extraction. We also extracted DNA from a sample of progeny produced by each of them. Specifically, we randomly sampled 5–10 mature seeds from the total seed family of each female parent, with 5, 6, 7, 8, 9, and 10 seeds sampled for individuals with 5–100, 101–200, 201–300, 301–400, 401–500, and > 500 seeds, respectively; in the few cases in which an individual produced fewer than five seeds, we sampled all of them. In total, we extracted DNA from 181 parents and 958 seed progeny. We could assign paternity to 915 of the c. 13 800 seed progeny in the population, which amounted to a sampling effort of 6.6%. Total DNA was extracted from the leaves and seed samples using the BioSprint 96 DNA Plant Kit (Qiagen) according to the manufacturer's instructions and eluted in 100 μl of distilled water.

PCR amplification was carried out in a final volume of 10 μl, including 5 μl of 2× Multiplex PCR Master Mix (Qiagen), 0.2 μl of diluted DNA, 2.8 μl of distilled water, and 2.0 μl of multiplex containing variable primer concentrations. The nine microsatellite markers were grouped into two multiplexes modified from Machado et al. (2017). Thermal cycling was performed in a TProfessional Standard Thermocycler (Biometra GmbH, Göttingen, Germany) as follows: 95°C for 15 min; 33 cycles at a temperature of 94°C for 30 s, 90°C for 90 s, and 72°C for 90 s; and a final step at 72°C for 10 min before cooling down to 4°C. PCR products were analyzed by capillary electrophoresis on an ABI3100 Genetic Analyzer (Applied Biosystems), with an internal size standard GeneScan‐500 LIZ. Fragment length analyses and scoring were performed with GeneMapper v.6.0 (Applied Biosystems, Waltham, MA, USA).

Paternity analyses were conducted for the three populations separately to assign the most likely father to each seed for which more than five loci were genotyped. Here, we used the software Cervus v.3.0.7, assuming a relaxed confidence level of 80% and an error rate of 0.01 (Kalinowski et al., 2007). M. annua is self‐compatible, and the female selfing rate (hereafter selfing rate) of each individual was estimated by the proportion of selfed seeds to the number of total seeds whose paternity was successfully assigned.

Calculation of female, male, and total fitness

The annual life cycle of M. annua allows us explicitly to estimate lifetime fitness in terms of the number of seeds produced and sired by each individual. In this study, following the practice of most empirical studies (e.g. Karron & Mitchell, 2012; Briscoe Runquist et al., 2017; Chen & Pannell, 2023; Hou et al., 2024), we attributed the fitness gained via selfed progenies equally to female and male functions rather than attributing twice the fitness of selfed progeny to only the female function, as is typical in theoretical studies (e.g. Charlesworth & Charlesworth, 1987; Lesaffre et al., 2024a). However, note that inferences for total fitness based on the two contrasting approaches are entirely equivalent. Fitness components were estimated under two scenarios using selfing rates (S) estimated from the paternity analysis: one in which inbreeding depression (δ) was assumed to be zero; and one in which δ was assumed to equal one. The actual level of inbreeding depression in wild populations of diploid M. annua remains unknown, with previous studies showing a negligible level of inbreeding depression estimated in hexaploid dioecious populations (Eppley & Pannell, 2009; Pujol et al., 2009), and another unpublished study showing considerable inbreeding depression in diploid populations under experimental evolution (E. Le Faou & J. R. Pannell, unpublished). The scenarios assumed in our analysis reflect the two extreme levels of inbreeding depression, allowing us to explicitly evaluate the effects of inbreeding depression on the patterns of selection on sex allocation (see Fig. S2 for supplementary analyses assessing the threshold values of inbreeding depression relevant to selection on sex allocation in plants of different sizes).

We calculated the female fitness of each individual i as $W_{\mathrm{F},i}=N_{\mathrm{T},i}\times \left[S_i\left(1-\delta \right)+\left(1-S_i\right)\right]$, where $S_i$ is the estimated selfing rate of individual i. Under the scenario of δ = 0, where selfing causes no inbreeding depression, female fitness is just the number of mature seeds of the individual ($N_{\mathrm{T},i}$). When δ = 1, selfed progenies do not contribute to the next generation at all. Note that our fitness estimates assume a linear relationship between viable seed number and female fitness: Because seeds of M. annua are well dispersed by a combination of explosive capsules and, subsequently, ants (Lisci & Pacini, 1997), local‐resource competition among offspring of the same mother is probably weak and unlikely to cause saturation of the female gain curve. We thus believe that viable seed number represents a reasonable approximation for female fitness.

We calculated male fitness of each individual i using the paternity share of each dam derived from the paternity analysis as $W_{\mathrm{M},i}={\sum }_{j\in F}\frac{n_{i,j}}{N_{\mathrm{G},j}}\times N_{\mathrm{T},j}$, where F is the set of all potential dams in the population, $n_{i,j}$ is the number of seeds sired by individual i on individual j, $N_{\mathrm{G},j}$ is the number of genotyped seeds of dam j, and $N_{\mathrm{T},j}$ is the total number of mature seeds of dam j. For selfed progeny (for which i = j), fitness accrued was discounted by a factor δ. We calculated the total fitness of individual i as $W_{\mathrm{T},i}={\mathrm{W}}_{\mathrm{F},i}+W_{\mathrm{M},i}$ for each of the two scenarios of inbreeding depression.

Statistical analysis

We conducted all the analyses within the R statistical framework v.4.0.3 (R Core Team, 2021). We checked the fit of the models with the package Dharma (Hartig, 2022) and QQ plots. The detailed structures of each regression model can be found in the Methods S2. The general effects of the explanatory variables in each model were extracted using likelihood ratio tests with the drop1 function.

To evaluate the trade‐off between investment in female vs male flower numbers (revealed by a negative coefficient), we used a zero‐inflated generalized linear mixed model (glmmTMB function in package glmmTMB; Brooks et al., 2017), setting the number of female flowers (nonnegative integers) as the response variable with a negative binomial distribution. Male flower number, plant size, and population were set as the explanatory variables with two‐way and three‐way interaction terms. For model convergence, the male flower number was standardized to a mean of zero and a SD of one for each population. Aboveground biomass was log‐transformed (hereafter referred to as ‘plant size’). Note that plant size was analyzed as a continuous variable in all analyses. To examine the interactive effect with plant size, we extracted the predicted coefficients at three levels of plant size (small, medium, and large), with size thresholds defined as plants of mean size minus SD (at the 16% quantile), mean size (at the 50% quantile), and mean size plus SD (at the 84% quantile), respectively (Fig. S3).

To investigate how the selfing rate depends on sex allocation (i.e. gender) and the size of focal plants in the three populations, we constructed a generalized linear mixed model (glmer function in package lme4; Bates et al., 2015), with each seed scored as either self‐ or cross‐fertilized and thus the selfing rate treated as a binomial response variable, with seeds nested in mothers as an observation‐level random variable to account for dispersion in the residuals (Harrison, 2015). Gender, plant size, and population were set as the explanatory variables with two‐way and three‐way interaction terms. To assess how inbreeding depression affects the dependency of different components of fitness on gender and plant size under two inbreeding depression scenarios, we fitted relative female, male, and total fitness as the response variable in three separate linear mixed models (lmer function in package lme4; Bates et al., 2015).

The relative fitness of each individual was calculated by dividing the fitness by the mean of the focal population (Lande & Arnold, 1983). To detect the nonlinear dependence of fitness on gender in each model, we set both linear and quadratic terms of gender as the explanatory variables, accompanied by interactions with plant size, scenarios of inbreeding depression, and population. We weighted the variances by plant size to indicate that plants of different sizes have different variances (larger plants have larger variance in fitness; see Methods S2). We set the identity of parental individuals as a random variable to account for the fact that the fitness estimates under the two inbreeding depression scenarios were not independent (i.e. the same individual). We first extracted P values of the general effects of the explanatory factors via likelihood tests using the drop1 function. We further used the emtrends function in the emmeans package to extract from the fitted models and conducted post hoc comparisons of the linear and quadratic coefficients and their SE (Lenth, 2023), for both scenarios of contrasting inbreeding depression and different plant sizes.

Results

Variation in sex allocation

Plants in the experimental population varied greatly in their size and their male and female reproductive allocations, ranging along a continuum from pure females to pure males (Figs 3a, S1). Biomass and sex allocation were independent (r = 0.04, P > 0.05) and showed no difference among populations (Table S2). Individuals produced an average of 238 ± 282 and 316 ± 449 female and male flowers, respectively (mean ± SD; n = 180; see Table S2 for each population; Fig. 3a). Larger plants produced more female and male flowers (P < 0.001). Our data also revealed a clear sex‐allocation trade‐off, with a negative dependence of female flower number on male flower number (P < 0.001), and the strength of the trade‐off being greater for larger plants (Fig. 3b; Table S3). See Fig. S4 for an evaluation of the nonlinear sex‐allocation trade‐off.

Dependence of self‐fertilization on gender

In total, 948 seeds were successfully genotyped for at least five loci, for which paternity was assigned to a single father for 915 seeds (see also Table S2 for details of each population). Overall, the average selfing rate was 29.6% (mean over n = 172 individuals; Table S2). The selfing rate was higher for individuals with greater male allocation (and greater male gender) (P < 0.001; Fig. 4), with the dependence tending to be steeper for larger plants (Fig. 4), although there was no significant interaction among gender, size, and population (Table S4). See also Fig. S5 for the positive dependence of the selfing rate on absolute male allocation, that is, male flower number.

Fig. 4.

Plot showing the effect of gender (maleness) on the selfing rate of studied individuals of Mercurialis annua (N = 172; individuals producing no mature seed were not included). Note that although the interaction with plant size was not significant, the slope trended to be steeper in larger plants. The shaded ribbon indicates the 95% confidence interval of the regression lines. Note that there was one individual with a gender of zero that had a selfing rate > 0. This inference is likely the result of sampling error during phenotyping, where we might have overlooked male flowers on that individual.

Dependence of fitness on gender, plant size, and inbreeding depression

Patterns of mating and fitness tended to be similar across populations: because all higher‐order interaction terms involving population were nonsignificant (P > 0.05; Table S5), we dropped interactions with population in presenting the results (Table 1; see also Figs S6, S7 for a comparison of selection and effect sizes among populations). Table 1 presents the linear and quadratic coefficients of different components of fitness with gender under the two inbreeding depression scenarios for plants of different sizes.

Table 1.

Comparisons of linear and quadratic coefficients of relative female (a), male (b), and total (c) fitness on gender between the two scenarios of inbreeding depression (δ) in Mercurialis annua of different sizes.

	δ = 0	δ = 1	Comparison
(a) Relative female fitness
Small plants	Linear: 0.87 (1.86)	Linear: 0.33 (1.86)
	Quadratic: −1.03 (1.91)	Quadratic: −0.54 (1.91)
Medium plants	Linear: −2.01 (1.09)	Linear: −3.92 (1.09) ***	***
	Quadratic: 0.35 (1.1)	Quadratic: 1.97 (1.1)	***
Large plants	Linear: −4.92 (1.75) **	Linear: −8.21 (1.75) ***	***
	Quadratic: 1.74 (1.77)	Quadratic: 4.51 (1.77) *	***
(b) Relative male fitness
Small plants	Linear: 1.58 (1.78)	Linear: 1.28 (1.78)
	Quadratic: −1.54 (1.82)	Quadratic: −1.07 (1.82)
Medium plants	Linear: 0.92 (1.05)	Linear: 0.08 (1.05)	*
	Quadratic: 0.43 (1.06)	Quadratic: 2.06 (1.06)	***
Large plants	Linear: 0.27 (1.68)	Linear: −1.11 (1.68)
	Quadratic: 2.4 (1.71)	Quadratic: 5.19 (1.71) **	***
(c) Relative total fitness
Small plants	Linear: 1.23 (1.32)	Linear: 0.81 (1.32)
	Quadratic: −1.30 (1.35)	Quadratic: −0.83 (1.35)
Medium plants	Linear: −0.55 (0.77)	Linear: −1.92 (0.77) *	***
	Quadratic: 0.38 (0.78)	Quadratic: 2 (0.78) *	***
Large plants	Linear: −2.35 (1.24)	Linear: −4.68 (1.24) ***	***
	Quadratic: 2.07 (1.26)	Quadratic: 4.86 (1.26) ***	***

Standard errors of the estimates of coefficients are indicated in parenthesis. A significant comparison indicates that the coefficients under the two scenarios are statistically different, based on post hoc pairwise comparisons (see details in the main text). Significant coefficients and comparisons are noted by asterisks: *, P < 0.05; **, P < 0.01; ***, P < 0.001.

The relationship between relative fitness and gender (maleness) depended strongly on plant size and scenarios of inbreeding depression, for both female function (P < 0.001 and P < 0.01 for three‐way interaction terms involving linear and quadratic terms of gender, respectively; Table S5; Fig. 5a) and male function (P > 0.05 and P < 0.01 for three‐way interaction terms involving linear and quadratic terms of gender, respectively; Table S5; Fig. 5b). For example, in small plants, relative male and female fitness components did not depend on gender for either scenario of inbreeding depression (P > 0.05 for all coefficients; Table 1; Fig. 5a,b), whereas in large plants, relative female fitness negatively and convexly depended on gender for a scenario assuming δ = 1 (for both sexual functions, P < 0.001 and P < 0.05 for linear and quadratic terms, respectively; Table 1; Fig. 5b), but it depended only linearly on gender for a scenario assuming δ = 0 (for both sexual functions, P < 0.01 and P > 0.05 for linear and quadratic terms, respectively; Table 1; Fig. 5b).

Fig. 5.

Plots showing the interactive effects of plant size, degree of inbreeding depression, and gender on relative female (a), male (b), and total (c) fitness of Mercurialis annua. Fitness was estimated under two scenarios of inbreeding depression (δ) of zero and one, depicted by orange and green lines, respectively. Shaded ribbons indicate the 95% confidence interval of the corresponding regression lines. The interactions with population were not significant; thus, the interactive effects with population are not shown (see Table 1, Supporting Information Table S4 for the P values).

When considering total fitness under different inbreeding depression scenarios, patterns of selection on gender were different for plants of different sizes (Table S5). Relative total fitness depended disruptively on gender in large plants under a scenario of δ = 1 (P < 0.001 and P < 0.001 for linear and quadratic terms, respectively; Table 1; Fig. 5c), whereas the disruptive dependence was nonsignificant when assuming δ = 0 (P > 0.05 for both terms; Table 1; Fig. 5c). By contrast, there was no detectable selection on gender in small plants regardless of the scenarios of inbreeding depression (Table 1; Fig. 5c).

Discussion

The wide phenotypic variation in sex allocation displayed in our experimental populations of wind‐pollinated M. annua allowed us to evaluate the dependence of the selfing rate and total plant fitness on allocation to male and female functions. Three results stand out. First, we confirmed the existence of a clear trade‐off between male and female allocation (Gerchen et al., 2024), as assumed in theories of sex‐allocation and life‐history evolution (Charnov, 1982; West, 2009). Second, the selfing rate of individuals depended positively on male allocation and greatly altered the shape of female and male fitness gain curves under scenarios of strong inbreeding depression, especially for plants of medium and large sizes. And third, sexual interference caused by the interaction between sex allocation and the mating system led to consistent patterns of strong disruptive selection on sex allocation, as required for the evolution and maintenance of dioecy (de Jong et al., 1999; de Jong & Geritz, 2001).

Nonlinear trade‐off in allocation between female and male function

We found a constant negative association between the number of male and the number of female flowers for each size class in the population. This result provides evidence for a sex‐allocation trade‐off for plants that likely had a similar resource status, as assumed by sex‐allocation theory (Charnov, 1982; West, 2009). In dioecious species, the production of sons vs daughters is usually a ‘zero‐sum game’ such that sex‐allocation trade‐offs are almost axiomatic. However, it has been difficult to demonstrate such trade‐offs in hermaphroditic species, likely because critical covariates of sex allocation such as resource status have been overlooked, or because male and female functions draw on different resources (de Jong, 1993; Campbell, 2000; Johnson & Nassrullah, 2024; summarized in Ashman, 2003; Mazer et al., 2007). A negative correlation between male and female functions has been found in some monoecious species that produce a mixture of unisexual flowers, for example, Astilbe biternata (Olson & Antonovics, 2000), Pinus sylvestris (Savolainen et al., 1993), and Zea mays (Garnier et al., 1993) – but see Begonia semiovata (Agren & Schemske, 1995). Our confirmation of a trade‐off between the production of male and female flowers in M. annua under experimental evolution (Gerchen et al., 2024) adds to this evidence.

Importantly, the sex‐allocation trade‐off revealed for M. annua deviates strongly from the linear relation assumed in sex‐allocation theory (e.g. Charlesworth & Charlesworth, 1981; Charnov, 1982), with its shape being significantly concave (see also Fig. S1 for a supplementary analysis). This concavity would appear to reflect advantages of the ‘economics of scale’ for each sexual function and thus an advantage of specialization (Reekie & Avila‐Sakar, 2005; Saeki et al., 2014), possibly linked to negative physiological interference between the sexual functions of individuals producing both male and female flowers, for example brought about by hormone regulation, gene expression, and nutrient acquisition (Diggle et al., 2011; Golenberg & West, 2013; Sobral et al., 2016; Jabbour et al., 2022). Sex expression in M. annua is regulated by phytohormones at the early stage of floral development, with auxin and cytokinin inducing male and female floral buds, respectively (Louis et al., 1990). It is thus plausible that the production of a mixture of male and female flowers by monoecious individuals may partially disrupt a finely tuned regulatory network required for the production of either male or female flowers (Durand & Durand, 1991; Golenberg & West, 2013). If so, such physiological interference would reduce the efficacy in flower production for individuals with intermediate allocation, at least in our study populations that have evolved under experimental conditions, and it may generally contribute to ecological advantages of sexual specialization in allocation in wild plant populations. The specialization in sex allocation, as implied by the nonlinear resource trade‐off, likely favors dioecy in M. annua irrespective of the effects of inbreeding, because selection on sex allocation is disruptive for large plants even in the absence of inbreeding depression.

Effect of selfing and inbreeding depression on female fitness

The selfing rate of individuals in our experimental populations increased with their relative and absolute allocation to male function. A positive dependence of selfing on male allocation has been demonstrated within a population for several insect‐pollinated hermaphroditic species, both at the individual and flower levels (Damgaard & Abbott, 1995; Harder & Barrett, 1995; Karron et al., 2004; Williams, 2007; Chen & Pannell, 2024), but it has hitherto not been reported for any wind‐pollinated angiosperm (though such a pattern has been reported for a gymnosperm; Denti & Schoen, 1988). Although many wind‐pollinated angiosperms are self‐incompatible or dioecious (Renner & Ricklefs, 1995; Vogler & Kalisz, 2001; Friedman & Barrett, 2008), the selfing rate should be sensitive to male allocation in self‐compatible wind‐pollinated plants. Our study suggests that the potentially monoecious or hermaphroditic precursors of currently dioecious wind‐pollinated species, which will often have been self‐compatible (Charlesworth, 1985), may well have experienced the negative effects of self‐fertilization, and that these might have contributed to selection for unisexuality.

The effect of sex allocation on the selfing rate also depended on the size of the plants considered. This finding corresponds to the predictions of the ‘mass‐action’ model (Gregorius et al., 1987; Holsinger, 1991), in which absolute allocation to male function increases the proportion of self‐pollen in the local pollen cloud and thus increases the selfing rate of the seeds produced (see also Fig. S5). The positive dependence of selfing on sex allocation was steeper in larger plants because larger plants produced more male flowers in absolute terms than smaller plants. By contrast, male flowers produced by small plants had a much milder effect on the selfing rate because of their smaller resource budget. As a result, most of the ovules produced by the small plants, regardless of their male allocation, were outcrossed by pollen produced by the larger neighboring individuals.

We found that the effect of male allocation on the rate of self‐fertilization in M. annua causes a dependence of female fitness on sex allocation under a scenario of high inbreeding depression, especially in medium‐ and large‐sized plants. When inbreeding depression was assumed to be zero in our analyses, female fitness depended linearly and positively on the allocation to female function, implying a mostly linear female gain curve. Under high inbreeding depression, by contrast, an elevated level of ovule discounting in individuals with increased male allocation should impose substantial fitness costs on female function. As a result, females that avoid allocating to their male function should enjoy higher relative female fitness, leading to an accelerating female gain curve (de Jong et al., 1999). Our results thus provide empirical evidence for the joint effect of the mating system and inbreeding depression on the female fitness gain curves due to sexual interference, as modeled by Charlesworth & Charlesworth (1981), de Jong et al. (1999), and Lesaffre et al. (2024a); see Chen & Pannell (2024) for another recent example in an insect‐pollinated species.

Implications for the evolution of sexual systems in wind‐pollinated plants

We found that male fitness in M. annua did not level off with an increased allocation to male function, indicating a linear (nonsaturating) male fitness gain curve. Our estimates of male fitness when inbreeding depression was assumed to be zero support the general view that wind‐pollinated species should have a linear male gain curve, because wind is not easily saturated with pollen in the same way that an insect's body may be (Charnov, 1982; Lloyd & Bawa, 1984). The inferred linear male fitness gain curve for M. annua is thus similar to those for the wind‐pollinated self‐incompatible Ambrosia artemisiifolia (Nakahara et al., 2018; Aljiboury & Friedman, 2022) and self‐compatible Picea glauca (Schoen & Stewart, 1986).

Our joint estimates of female and male fitness clearly indicate that dioecy should evolve in response to selection to promote male fitness and avoid self‐fertilization in self‐compatible wind‐pollinated species (de Jong et al., 1999). Considering total fitness gained via the two sex functions under strong inbreeding depression, our study points to the action of strong disruptive selection on sex allocation in medium‐ and large‐sized plants of M. annua, favoring greater separation of the sexes in all three replicate populations (Fig. S6). Although an accelerating female gain curve due to ovule discounting may be common in both insect‐ and wind‐pollinated species (Karron et al., 2004; Williams, 2007; Chen & Pannell, 2024), the male gain curve is likely to be saturating in insect‐pollinated species (de Jong & Klinkhamer, 1994), potentially precluding disruptive selection on sex allocation and rendering the evolutionarily stable sexual system hermaphroditic or gynodioecious (de Jong et al., 1999; de Jong & Geritz, 2001). Our finding of disruptive selection on sex allocation in M. annua may thus help to explain the common association between dioecy and wind pollination in flowering plants (Renner & Ricklefs, 1995; Vamosi et al., 2003; Friedman & Barrett, 2009) and the frequent evolution of dioecy from hermaphroditism following shifts to wind‐pollination, for example, in Fraxinus (Wallander, 2008), Schiedea (Weller et al., 1995), and Thalictrum (Soza et al., 2012).

We also found that disruptive selection on sex allocation was particularly strong for large plants and almost absent for small plants (see also Fig. S2). This result would thus seem to imply that size‐dependent sex allocation ought to be selected in M. annua, with the large individuals being strictly unisexual and small individuals expressing a wider range of sex allocation (de Jong et al., 1999; de Jong & Geritz, 2001). The fact that wild individuals of M. annua are predominantly unisexual, irrespective of their size, may indicate that dioecy has been stabilized largely by the selection on mid‐ and large‐sized individuals, which produce almost all their parents' descendants; the smallest 16% of plants (i.e. plants with a size smaller than 1 SD from the mean size) produced on average only 18.7 ± 18.3 seeds, compared with the average seed production of 106 ± 160 in the populations as a whole, and it is thus plausible that they adopt a unisexual strategy because selection on their sex allocation is correspondingly weak.

Concluding remarks

Selection for inbreeding avoidance or selection for traits that promote advantages of sexual specialization has long been identified as the most likely mechanism favoring the evolution of dioecy in plants (Bawa, 1980; Thomson & Brunet, 1990; Freeman et al., 1997; Pannell & Jordan, 2022). Although separate sexes in wild populations of dioecious M. annua are likely favored by advantages of sexual specialization not linked to inbreeding depression (Eppley & Pannell, 2007; Tonnabel et al., 2019), our results now indicate that disruptive selection on sex allocation, and thus the potential evolution of separate sexes, can arise in populations through the avoidance of the negative effects of selfing when the male fitness gain curve is not too saturating, even in the absence of advantages of sexual dimorphism. Thus, while sexual specialization on ancillary traits may reinforce selection for separate sexes and the evolution of dioecy with sexual dimorphism (Lesaffre et al., 2024b), our study demonstrates that inbreeding avoidance on its own can drive the evolution of dioecy (see Fig. S8 for the negligible effects of ancillary traits on fitness from supplementary analyses). Our study thus provides support for fundamental elements of sex allocation theory that have been difficult to test or validate for lack of suitable phenotypic variation.

Competing interests

None declared.

Author contributions

K‐HC and JRP designed the project. K‐HC collected the data and analyzed the data with input from JRP. Both authors wrote and edited the manuscript.

Disclaimer

The New Phytologist Foundation remains neutral with regard to jurisdictional claims in maps and in any institutional affiliations.

Supporting information

Fig. S1 Variation of plant size and sex allocation in the study populations.

Fig. S2 Plots showing the patterns of selection on gender via total fitness under two intermediate levels of inbreeding depression in plants of different sizes.

Fig. S3 Histogram showing the size of plants in the three experimental populations.

Fig. S4 Plots showing the nonlinear trade‐off curves between female and male functions in the studied populations.

Fig. S5 Plot showing the effect of the number of male flowers on the selfing rate.

Fig. S6 Plots showing the interactive effects of plant size, degree of inbreeding depression, and gender on relative total fitness in three experimental populations.

Fig. S7 The effect sizes of explanatory factors on relative total fitness in three experimental populations.

Fig. S8 The effect sizes of eight ancillary traits, size, gender, and inbreeding depression on relative male, female, and total fitness.

Methods S1 Setup of the experimental populations.

Methods S2 Detailed structure of each regression model used in this study.

Table S1 Principal component analysis on eight ancillary traits and their correlation with gender.

Table S2 Details of sex allocation, biomass, and paternity analyses of the three experimental populations.

Table S3 Summary table of the general effects of male flower number, size, and population on female flower number.

Table S4 Summary table of the general effects of size, gender, and population on the selfing rate.

Table S5 Summary table of the general effects of size, gender, scenarios of inbreeding depression, and population on relative female, male, and total fitness.

Please note: Wiley is not responsible for the content or functionality of any Supporting Information supplied by the authors. Any queries (other than missing material) should be directed to the New Phytologist Central Office.

Data availability

All data (including the raw data of parental and offspring genotypes) and R code for analyses conducted are available in Zenodo (doi: 10.5281/zenodo.14542260).