Frequency-dependent fitness and reproductive dynamics contribute to habitat segregation in sympatric jewelflowers

Abstract

Coexistence results from a complex suite of past and contemporary processes including biogeographic history, adaptation, ecological interactions and reproductive dynamics. Here we explore drivers of local micro-parapatry in which two closely related and reproductively isolated Streptanthus species (jewelflower, Brassicaceae) inhabit continuous or adjacent habitat patches and occur within seed dispersal range, yet rarely overlap in fine-scale distribution. We find some evidence for abiotic niche partitioning and local adaptation, however differential survival across habitats cannot fully explain the scarcity of coexistence. Competition may also reduce the fitness of individuals migrating into occupied habitats, yet its effects are insufficient to drive competitive exclusion. Experimental migrants suffered reduced seed production and seed viability at sites occupied by heterospecifics, and we infer that heterospecific pollen transfer by shared pollinators contributes to wasted gametes when the two congeners come into contact. A minority disadvantage may reduce effective colonization of patches already occupied by heterospecifics, even when habitat patches are environmentally suitable. Differential adaptation and resource competition have often been evoked as primary drivers of habitat segregation in plants, yet negative reproductive interactions—including reproductive interference and decreased fecundity among low-frequency migrants—may also contribute to non-overlapping distributions of related species along local tension zones.

1. Introduction

The complex interplay among geologic, evolutionary and dispersal histories shapes species ranges at broad scales [1,2], while adaptation to environmental conditions [3,4] and random chance associated with community assembly [5,6] can shape species distributions at local scales. Resource competition among ecologically similar species can lead to local extirpation of the competitively inferior species, however by occupying different niches, plant species are able to accumulate stabilizing differences that minimize competition, and thus facilitate coexistence [7–10]. Given similarity in environmental niches and a lack of competitive dominance, regionally sympatric species might also be expected to co-occur in the same habitat patches [11,12].

In addition to abiotic resource competition, biotic interactions among sympatric species, including pollination dynamics, can profoundly influence their fine-scale distributions [13–15]. Outcrossing angiosperms may compete for pollinator services [16,17], and here the outcome of competition does not affect survival, but instead determines the availability of mating opportunities, fecundity and ultimately a species's ability to reproduce and persist at a site. Competition for pollinators can occur between any co-flowering members of the community [18], however phylogenetic conservatism in floral morphology [19] and phenology [20] results in closely related taxa that co-use shared pollinator resources more than expected by chance [21,22]. Strong competition for pollinators among co-flowering plant species can result in pollen limitation, reduced seed set and decreased fitness for the less-preferred foraging target [23,24], thus shaping community assembly [13,25]. Alternatively, given similar floral morphologies or floral rewards, pollinators may indiscriminately transfer pollen between members of the plant community [26].

Fitness-reducing interactions between related plant species can also occur via heterospecific pollen transfer (HPT) [27], leading to stigmatic interference [28], pollen loss to unreceptive stigmas [24] or ovule usurpation [29]. Reproductive interference (RIN) can be defined as any interaction among reproductively isolated heterospecifics that results in fitness losses due to either wasted mating attempts [30–32] or wasted gametes associated with hybrid failure [33,34]. The loss of ovules to incompatible pollen (i.e. reduced female fitness) may be particularly costly for co-flowering relatives that have incomplete prezygotic isolation, but produce inviable or infertile hybrids [35]. Additionally, RIN can be frequency-dependent [36,37], as rarer species in the community may be more likely to receive heterospecific pollen, and thus may experience a minority disadvantage [38]. Evidence for RIN and its effects on coexistence in plants has begun to emerge [15,36,39,40], however examples in animals still seem to outnumber those in plants [41].

The fitness costs of rarity associated with RIN directly conflict with the potential benefits of rarity under competition-based coexistence theory [7,11]. In the latter, low-frequency migrants experience a fitness advantage when invading patches of heterospecifics as abundant residents suppress themselves via strong intraspecific competition [8,42,43], whereas rare migrants suffer milder interspecific competition. Under RIN however, species do not always increase when rare, but may become increasingly rare [44–47], potentially due to the repeated loss of gametes across generations associated with unsuccessful reproduction. Recent theory shows that RIN can lead to a rarity cost, and that individual mating success can also show positive density dependence [48,49] (i.e. an Allee effect), suggesting that some species may be able to escape the detrimental effects of heterospecific interference if they occur beyond some density threshold. From an evolutionary perspective, RIN may contribute to bi-stable dynamics observed at hybrid tension zones maintained by genetic incompatibilities [50], where each species is abundant on opposite sides of a cline, effectively repelling one other along their contact zone [51]. Fine-scale spatial segregation between plant species however, has been attributed predominantly to abiotic niche differences [52,53], resource competition [54], or both [55], as opposed to reproductive dynamics.

Here we evaluate the relative importance of several ecological and evolutionary processes that might contribute to fine-scale spatial isolation in two closely related annual plants. The reproductively isolated California jewelflowers, Streptanthus breweri and Streptanthus hesperidis (Brassicaceae), are restricted to rocky serpentine outcrops and occur within metres of one another in the sympatric portion of their ranges, yet rarely coexist in intermixed patches. We assess the relative roles of niche differentiation and local adaptation, resource competition, and frequency and density dependence, in explaining patterns of micro-parapatry. Specifically, we ask the following questions. (1) Does each species occupy a distinctive abiotic niche within seemingly similar serpentine habitats? (2) Can local adaptation associated with establishment, survival, or growth explain patterns of spatial distribution at a fine scale? (3) Does immigrant inviability associated with competition, soil effects or reduced fecundity (i.e. seed production and seed viability) contribute to a lack of coexistence? and lastly (4) What factors influence seed viability in experimental migrants? We leverage previous knowledge of pollination dynamics and reproductive isolation in this system, with findings from a series of field and greenhouse experiments, to better understand how multiple ecological and evolutionary processes might contribute to, or reduce, coexistence among close relatives.

2. Material and methods

(a). Study species

The annual plants S. breweri A. Gray and S. hesperidis Jeps. are restricted to rocky, isolated serpentine outcrops in the interior Coast Range of California, where they share an area of geographic overlap [56]. Previous field surveys in an area of sympatry indicate that the two species rarely occur in intermixed stands, but instead are spatially isolated in single-species patches on the same, or closely adjacent (less than 100 m), serpentine outcrops [57]. Twenty-six of 32 previously identified patches within a 7000-acre area consisted of a single species, while only three patches of each species were intermixed [57]. Two of three intermixed sites consisted of larger S. breweri populations (greater than 250 individuals) and smaller S. hesperidis populations (less than 50 individuals) where only a few individuals of each species occurred with heterospecific neighbours. We observed only a single site consisting of hundreds of intermixed individuals of both species, suggesting that fine-scale coexistence is relatively uncommon.

Streptanthus breweri and S. hesperidis share overlapping flowering periods, and both are largely outcrossing and reliant on insect pollination for effective fruit set [57]. Field observations at sympatric sites have documented strongly overlapping pollinator communities, as approximately two-thirds of all insect visitors are shared, and the shared pollinator Bombus vosnesenskii accounts for the majority of all floral visits [57]. In mixed-species experimental arrays in the field, we have commonly observed floral visitors transitioning between species, thus S. breweri and S. hesperidis can interact via pollination when they occur in close proximity.

The species pair is isolated by strong intrinsic postzygotic reproductive isolation [58]. Hybrid crosses suffer reduced seed production (with S. breweri as maternal parents) and F₁s show reduced viability and fertility (in both directions of the cross). Heterospecific pollen can effectively fertilize both species resulting fruit and seed production [57], however HPT leads to reduced seed viability (electronic supplementary material, Experimental pollination). Both species also suffer significantly reduced seed viability (61–71% viable) when receiving 50 : 50 mixed pollen compared to when receiving pure conspecific pollen (97–100% viable; electronic supplementary material, figure S1).

(b). Study location

We conducted niche characterizations and a reciprocal transplant experiment at the University of California McLaughlin Natural Reserve (hereafter ‘McLaughlin’; 38.8711° N, 122.41930° W), the same location exhibiting strong fine-scale spatial isolation among the 32 patches described above. We also conducted several experiments in the UC Davis greenhouse and lathhouse, using seeds and soils collected from McLaughlin.

(c). Question 1: Do Streptanthus breweri and Streptanthus hesperidis occupy different microhabitats in sympatry?

We quantified multiple physical site attributes that might influence habitat affinity including: elevation, slope, aspect, soil depth, soil colour (a proxy for chemical composition and thus the relative weathering rate of serpentine parent materials), ground cover, and substrate rockiness at 32 sites at McLaughlin. We also analysed soil chemistry (NO₃, P, K, Na, Ca, Mg, boron, cation exchange capacity, organic matter, pH and saturation percentage) and soil texture (% sand, % silt, % clay) at 20 sites. We visualized niche breadth via NMDS ordination (metaMDS function in the vegan package [59] in R [60]), and tested for abiotic niche differences using a multi-dimensional permutation approach (adonis2 function in vegan). We also quantified soil moisture at sites occupied by each species by measuring gravimetric water content of soils throughout the season, and assessed difference using t-tests and Mann–Whitney tests (electronic supplementary material, Field soil moisture).

(d). Question 2: Can fine-scale local adaptation explain spatial segregation?

We conducted a reciprocal transplant experiment in the field and measured fitness components associated with establishment, survival and growth. Using adjacent population pairs replicated at three sites, we transplanted seeds approximately 10–100 m from their native sites to an adjacent serpentine patch, or to an adjacent portion of the same serpentine patch occupied by their congener. We sourced seeds from each locally adjacent population pair, lightly glued seeds to toothpicks, and planted resident (home) and migrant (away) seeds into each site. At each site (three paired sites, n = 6 total), we established ten experimental blocks (1.0 m × 0.5 m) of 50 seeds (25 of each species) planted in a checkerboard pattern (n = 3000 seeds total).

We initially transplanted seeds in the winter of 2014–2015, however the vast majority (n = 2800) of seeds failed to germinate likely due to a winter drought. Some seeds did germinate however at one paired site in the spring of 2015 (n = 200 of 1000 seeds planted here), and here we assessed germination success to determine if habitat-specific germination might prevent each species from becoming established. We modelled germination using a binomial GLMM (glmer function from the lme4 package [61] in R) with species, habitat, and a species × habitat interaction as fixed effects, with experimental block as a random effect. To further assess soil-specific germination success, we also conducted a follow-up experiment in the greenhouse using field-collected seeds (n = 240 total), and field-collected soils from all sites (electronic supplementary material, Germination).

Many seeds germinated and persisted the following year (2015–2016) and for each germinant we quantified survival, plant height at the end of the growing season and fruit production. Given the year delay between planting and emergence and the loss of some identifying toothpicks, some resident plants might have been natural germinants and not seeds we experimentally planted, however approximately 70–75% of plants retained identifying markers. We tracked all experimental migrants (n = 174), and an equal number of residents in each experimental block (n = 175 total). We quantified survival, plant growth (which reflects both the intrinsic ability of a plant to grow in a foreign habitat, as well as the potential effects of resource competition with residents occurring at ambient densities), and fruit production. We tested for local adaptation using GLMMs with species, habitat, site and a species × habitat interaction as fixed effects, with experimental block as a random effect. Here, local adaptation is demonstrated by a significant species × habitat interaction [62], in which the resident species has higher fitness in the habitat it naturally occupies. For all mixed models, we derived median and 90% confidence intervals for model predictions from 1000 bootstrap samples using the bootMer function in lme4.

(e). Question 3: Does immigrant inviability associated with (i) competition, (ii) soil effects or (iii) reduced fecundity in the presence of heterospecifics, contribute to a lack of coexistence?

(i). Competition

While direct resource competition may be a relatively weaker force in shaping the spatial dynamics of S. breweri–S. hesperidis on barren serpentine outcrops due to the low density at which native plants occur, we conducted a small experiment to roughly assess the relative effects of intra- and interspecific competition when plants occurred with 1–2 cm of one another. Here, we grew field-collected seeds in potting soil, either alone, with a single conspecific competitor, or with a single heterospecific competitor (n = 20 per species per treatment). We used plant height (which is also correlated with fruit production in these annuals; p < 0.001, Spearman ρ = 0.53 for all survivors from the field experiment) as a measure of growth, and assessed the effects of within- and between-species competition using one-way ANOVA (aov in R) and Tukey's HSD tests (HSD.test from the agricolae package [63]).

(ii). Soil effects

To determine if intrinsic soil properties influenced seed production or seed viability of migrants, we replicated our field transplant on a smaller scale in a mesh-covered lathhouse at UC Davis using field-collected soils (electronic supplementary material, Lathhouse soil transplant). Here, we grew plants (n = 152; n = 76 per species), in intact soil cores collected from the same six sites used in the field experiment. We assessed seed production and seed viability in both soil types using GLMs with species, soil type, site and a species × soil type interaction as fixed effects.

(iii). Fecundity (seed production and seed viability) in the presence of heterospecifics

In nature, S. breweri and S. hesperidis mostly occur in single-species patches, and thus are largely removed from the immediate influence of heterospecifics. In our field experiment, however, we transplanted migrant seeds into experimental blocks with resident seeds, and into habitat patches dominated by native residents occurring at ambient densities (on average four individuals m⁻²). Field sites supported hundreds of natives, and we did not remove these individuals, thus migrants were embedded into communities saturated by heterospecifics, well within pollinator flight distances. If reproductive interactions are important to fitness, we hypothesized that seed production and seed viability would be frequency-dependent, as rare migrants would receive an abundance of heterospecific pollen from common residents.

At the end of our field transplant experiment, we assessed whether surviving migrants (i.e. individuals transplanted into foreign habitats) suffered decreased seed production and seed viability compared to residents (i.e. individuals of the same species growing in their native habitats). We quantified seed production and seed viability in 120 surviving migrants by harvesting 609 fruits and 7411 seeds from six sites and 40 experimental blocks. Unfilled, shriveled or markedly shrunken seeds were deemed inviable. To minimize the effects experimental migrants may have had on immediately adjacent residents, we quantified fecundity of randomly selected residents (n = 60, n fruits = 208, n seeds = 8208) from within the habitat patches used in the transplant experiment, but from outside the immediate 1.0 m × 0.5 m experimental blocks. We used GLMMs (as described for question no. 2) to assess seed production and seed viability of individuals living in neighbourhoods consisting of conspecifics (i.e. residents), and those embedded into neighbourhoods of heterospecifics (i.e. migrants).

(f). Question 4: what factors influence seed viability in experimental migrants?

While not originally a goal of our field transplant, this experiment gave us the opportunity to evaluate a number of factors that may have influenced seed viability in experimental migrants. Under rarity disadvantage and RIN, we predicted local neighbourhoods with fewer surviving migrants (i.e. increased rarity) should have lower seed viability compared to local neighbourhoods with more survivors, due to relatively greater HPT from locally abundant residents. Additionally, microsite quality and resource availability might also reduce the seed viability of migrants. Here, we hoped to evaluate which aspects of the local environment best explained variation in rates of seed viability among migrants; thus, we employed an iterative model selection approach.

We first fit a full GLMM (glmer function) predicting seed viability of all surviving migrants (n = 119 with known fruit number) with species, site, number of conspecifics per block (to account for local density dependence), fruit number (to account for plant size and potential resource limitation), average number of fruits per resident within the block (a proxy for microsite quality), and a species × number of conspecifics interaction as fixed effects, with experimental block as a random effect. Site effects and block effects both describe aspects of environmental variation, however blocks (1.0 m × 0.5 m; scattered across sites) also capture local resident density (which we did not measure in the field) and smaller scale environmental variation. We evaluated candidate models with all combinations of fixed effects (retaining block as a random effect) using AICc (dredge function from the MuMIn package [64]). We determined the best-fitting model, as well as those predictors that were included in the most explanatory models (delta AICc ≤ 2).

3. Results

(a). Question 1: habitat partitioning

We found no significant differences in the multi-dimensional abiotic niches of the two species at McLaughlin (adonis2 permutation test, p = 0.56), however S. breweri occupied a qualitatively broader abiotic niche than S. hesperidis (figure 1a), consistent with geographical range differences, in which the smaller-ranged S. hesperidis is nested within the larger-ranged S. breweri [56]. These results were the same when we added soil texture and chemistry into our overall habitat characterizations (p = 0.49; electronic supplementary material, figure S2). Despite the overall similarity of habitats occupied by both species, soil moisture at closely adjacent sites (less than 100 m) was greater for first two-thirds of the growing season at locations occupied by S. hesperidis (p < 0.001 and p = 0.006; figure 1b). This pattern is consistent with the fact that S. hesperidis occupied sites with a qualitatively greater fraction of small (less than 2 mm) particles in the substrate (electronic supplementary material, table S1).

Figure 1.

(a) NMDS ordination of 18 physical site attributes of S. breweri (n = 17) and S. hesperidis (n = 15) sites at McLaughlin. Points represent individual sites; ellipses represent 95% confidence intervals for abiotic niche breadth. (b) Boxplots of soil moisture throughout the growing season at sites occupied by each species. (Online version in colour.)

(b). Question 2: local adaptation

Two-hundred of 1000 seeds sown at one paired site germinated in 2015, however neither species had higher germination success in their home habitat (figure 2a); to the contrary, germination success was higher for migrants (species × habitat interaction, p = 0.002; electronic supplementary material, table S2.1). Similarly, there was no evidence for a home-soil germination advantage (electronic supplementary material, figure S3; species × soil type interaction, p = 0.30; electronic supplementary material, table S2.2) when we germinated seeds in the greenhouse in unsifted field soils. In 2016, we found evidence for local adaptation with respect to survival (figure 2b; species × habitat interaction, p = 0.014; electronic supplementary material, table S2.3). This pattern was driven by S. hesperidis, which had systematically greater survival in its own habitats (99%) relative to S. breweri habitats (84%), whereas S. breweri survived equally well (95–96%) in both environments. In the field, survivors of both species grew to similar heights in both habitats (figure 2c; species × habitat interaction, p = 0.79; electronic supplementary material, table S2.4), and produced an equivalent number of fruits in both habitats (figure 2d; species × habitat interaction, p = 0.15; electronic supplementary material, table S2.5).

Figure 2.

Average fitness associated with survival and growth of residents and migrants in a field transplant experiment. (a) Germination success, (b) survival, (c) plant height at the end of the growing season and (d) fruit production. Large points represent median model predictions; error bars show 90% confidence intervals for model predictions; small points represent individual plants; asterisks show significance thresholds of species × habitat interactions. (Online version in colour.)

When replicating our field experiment in the lathhouse, growing plants in intact field soils, we did not find evidence of a home-soil survival advantage (electronic supplementary material, figure S4A; species × soil type interaction, p = 0.45; electronic supplementary material, table S2.6). Here, S. breweri qualitatively survived better in S. hesperidis soils (92% versus 70% in its own soils) which have a higher water-holding capacity (figure 1b), indicating physiologically at least, that S. breweri may be able to thrive in those soils commonly occupied by S. hesperidis. Results for plant growth (electronic supplementary material, figure S4B and table S2.7) and flower production (electronic supplementary material, figure S4C and table S2.8) were consistent with findings from the field experiment, in which there was no evidence for a home-site or a home-soil advantage.

(c). Question 3(i): competition

In the greenhouse competition experiment, in which individuals grew within 1–2 cm of one another in the same pots, patterns for plant growth were consistent with patterns from the field (figure 2b), in which both species reached similar heights in both competitive environments (electronic supplementary material, figure S5A; species × competitor interaction, p = 0.12). We found no differences in the relative effects of intraspecific compared to interspecific competition on plant growth for either species (electronic supplementary material, figures S5B and S5C). While these results must be interpreted with the caveat that plants were competing in potting soil, they indicate that under these conditions, neither species is competitively dominant over the other.

(d). Question 3(ii): intrinsic properties of the soil

We found no evidence for a home-soil advantage for seed production (figure 3a; species × soil type interaction, p = 0.08; electronic supplementary material, table S3.1) or seed viability (figure 3b; species × soil type interaction, p < 0.001, but consistent with a home-soil disadvantage; electronic supplementary material, table S3.2) when we grew both species on both soil types in the lathhouse.

Figure 3.

Average fecundity of residents and migrants in a soil transplant (lathhouse) and a field transplant experiment. (a,c) Seed production per fruit; (b,d) seed viability. Points represent median model predictions; error bars show 90% confidence intervals for model predictions (GLMs for lathhouse data; GLMMs for field data); small points represent individual plants; asterisks show significance thresholds of species × habitat or species × soil interactions. (Online version in colour.)

(e). Question 3(iii): fecundity (seed production and seed viability) in the presence of heterospecifics

In the field, experimental migrants (i.e. all surviving individuals growing at sites occupied by heterospecifics) had significantly reduced seed production (figure 3c; species × habitat interaction, p < 0.001; electronic supplementary material, table S3.3) and seed viability (figure 3d; species × habitat interaction, p < 0.001; electronic supplementary material, table S3.4) compared to native plants growing outside the immediate influence of heterospecifics. On average S. breweri produced 29.8 total seeds with 95% viability as a resident, and 9.9 total seeds with 53% viability as a migrant. Similarly, S. hesperidis produced 23.7 total seeds with 90% viability as a resident, and 8.6 total seeds with 49% viability as a migrant (figure 3c,d).

(f). Question 4: what factors influence seed viability in experimental migrants?

The best-fitting model explaining seed viability in experimental migrants included fixed effects for number of conspecifics per block and site, and a random effect for block (electronic supplementary material, table S4). Here, seed viability increased with the number of conspecifics in the local neighbourhood (p = 0.058; electronic supplementary material, table S5), and each additional migrant resulted in an average increase of 4.7% in seed viability (figure 4). In the field experiment, densely clustered migrants (n = 8 conspecifics per block) had 31% greater seed viability than migrants occurring alone (figure 4). The best-fitting model had almost twice the weight of the next most explanatory model (electronic supplementary material, table S4), which only included a single fixed effect for the number of conspecifics. In sum, five of seven of the best-fitting models (delta AICc ≤ 2) included a fixed effect for number of conspecifics, whereas only one of the top fitting models included a term encapsulating plant size and vigour (number of fruits per plant), and none included a term for microsite quality (average number of fruits per resident; electronic supplementary material, table S4).

Figure 4.

Density-dependent seed viability of experimental migrants; points represent seed viability of individual plants; horizontal lines in boxplots represent median seed viability in local neighbourhoods with varying numbers of conspecifics; trend line follows median predictions from the best-fitting model (a species term was not included in this model, so only a single line is fit). (Online version in colour.)

4. Discussion

Niche differences and the asymmetrical effects of resource competition have dominated conceptual frameworks of species coexistence [8–11], however in some cases, as here, shared ancestry might decrease both niche differences [22] and potential fitness differences [65] among close relatives. Thus, congeners may sometimes act more like nearly neutral species and may be subject to priority effects [66,67]. Negative reproductive interactions [14,35,68] can also exert strong selection pressures on sympatric but reproductively isolated relatives, potentially shaping the spatial dynamics of coexistence [69]. We tested several hypotheses that might explain the micro-parapatric distributions of S. breweri–S. hesperidis, including niche partitioning, local adaptation associated with habitat-specific establishment, survival and growth, the effects of competition, and reduced seed production and seed viability associated with co-occurrence.

We found some evidence for local adaptation as S. hesperidis suffered a 15% reduction in survival in S. breweri habitats (figure 2b), which may explain why it does not commonly occur at S. breweri sites. These effects were not absolute however, as 84% of S. hesperidis germinants survived to flowering in S. breweri habitats, and showed similar growth (figure 2c) and fruit production (figure 2d) compared to residents growing in their home habitats. Conversely, these results cannot explain why S. breweri is not found in S. hesperidis habitats, as it performed equally well in both habitats at all early life history stages (figure 2), and may even show elevated survival on S. hesperidis soils (electronic supplementary material, figure S4 and table S2.6). Together these data suggest it is unlikely that differences in performance alone explain the scarcity of occurrences of each species at sites occupied by their close relatives.

Reduced seed production (figure 3c) and seed viability (figure 3d) in migrants had a much greater overall impact on fitness than did habitat-specific establishment, survival, growth or fruit production (figure 2), or the effects of direct competition (electronic supplementary material, figure S5). Seed viability was unrelated to intrinsic soil properties (figure 3a;b), but instead varied depending whether the species co-occurred (figure 3c,d). Interestingly, data in which we experimentally created mixed-species patches, closely match seed viability patterns from the single large mixed-species patch that we observed in the field. Here, S. hesperidis produced 58% viable seeds and S. breweri produced 71% viable seeds (based on data from 2270 seeds in 76 fruits from 20 individuals), supporting the idea that occurring in close proximity to heterospecifics, per se, is related to reduced fitness. Data from experimental pollinations, in which plants receiving mixed pollen produced 61–71% viable seeds (electronic supplementary material, figure S1), compared to 90–95% seed viability of residents occurring in mono-specific patches (figure 3d), further support the idea that pollination dynamics among co-occurring relatives is related to this fitness reduction.

We cannot rule out every potential contributor to fine-scale spatial distributions, such as seed dispersal or pollen limitation; however, of the factors we assessed, reduced fecundity in the presence of heterospecifics had the largest effects on plant fitness. We infer that when the two species come into close contact and exchange pollen, RIN results in wasted ovules and a subsequent minority disadvantage [31,33], potentially preventing migrants from successfully colonizing occupied habitats. Here, S. breweri–S. hesperidis represent an active tension zone, in which two species occur in essentially alternative states—either abundant or absent—and repel one another along a micro-parapatric interface [70]. Our findings suggest that reproductive interactions, priority effects, and frequency-dependent fitness costs imposed by HPT may play an underappreciated role in explaining fine-scale distribution patterns among sympatric relatives.

(a). Habitat differentiation

Empirical evidence for RIN in plants is mounting [14,39,71], however habitat divergence sometimes occurs in concert with RIN [36,40], thus it is not always possible to link the outcome of RIN to fine-scale distribution patterns. One difficulty stems from cryptic habitat differentiation, where competitive interactions might be minimized by subtle, yet stabilizing abiotic niche differences. For example, Lasthenia gracilis and L. californica occur within metres of one another on serpentine ridges, but are differentially adapted to changing soil conditions along an environmental gradient [72]. Similarly, the serpentine endemics Monardella stebbinsii and M. follettii are broadly sympatric, but generally occupy microhabitats that differ in soil texture and chemistry, however both can occur in intermediate habitats [73]. Peterson et al. [52] in Mimulus and Eckhart et al. [74] in Clarkia documented niche partitioning of substrates in close relatives, but also discovered founder effects in which intermediate habitats were subject to stochastic colonization by either co-occurring congener. In the present study, soil moisture availability may impose an environmental filter on S. hesperidis (figure 1b); however, S. breweri survives equally well in both slightly wetter and slightly drier habitats (figure 2b), thus occupation of wetter sites by S. breweri and not S. hesperidis, might result more from chance than deterministic niche differences. In contrast to the above and other recent studies in plants [40,53,75,76], there is less evidence for strong habitat differentiation in S. breweri–S. hesperidis, suggesting that RIN, as opposed to niche-based processes, is one primary mechanism responsible for fine-scale spatial segregation.

(b). Density dependence, Allee effects and escape from minority disadvantage

In competition coexistence theory, rare colonists benefit because abundant residents suppress themselves via intraspecific competition [7,8,11], however a growing body of theoretical work [48,49,51] suggests that rarity can also entail fitness costs. Our empirical work documents this, as seed viability of rare migrants was approximately 45% less than in residents (figure 3d), mirroring frequency and density-dependent trends for both Limnanthes douglasii ssp. rosea [14] and Taraxacum japonicum [77]. Together these findings corroborate reproductive interactions as potential obstacles to coexistence, but also suggest positive density dependence and a possible escape from the costs of rarity once individuals occur beyond a minimum threshold. In our field experiment, migrants showed seed viability levels approaching those of residents once they occurred in local neighbourhoods that were relatively dense with conspecifics (n = 8 in figure 4). Given the average density of native plants (on average four individuals m⁻²), it follows that obtaining a majority of conspecifics within a local neighbourhood, and associated propagule pressure, might offset the cost of migrating into habitats occupied by heterospecifics.

Another common outcome of RIN is its asymmetrical effects, with one species suffering relatively less when rare compared to another [15,39,47,78]. Such asymmetrical effects in plants may stem from to one species's superiority in attracting pollinators, and thus facilitating assortative mating [14]. Here, three of seven of the best-fitting models explaining variation in seed viability in migrants included a significant species effect (electronic supplementary material, table S4); however qualitatively, some S. breweri occurring alone (n = 1 in figure 4) still produced seeds with high viability. This suggests that species-specific pollinator preference, pollinator constancy or conspecific pollen precedence might promote assortative mating in mixed-species patches. On the other hand, low-density S. hesperidis individuals may lack equally well-developed mechanisms enabling high reproductive fitness when rare. In controlled crosses, S. hesperidis produced roughly the same number of total seeds regardless of whether pollen was conspecific, heterospecific, or mixed (electronic supplementary material, figure S1); however, in the field, migrants produced many fewer seeds than residents (figure 3c). This suggests that S. hesperidis migrants may suffer from both pollen limitation and reduced seed viability associated with frequency- and density-dependent RIN.

(c). Reproductive exclusion, and the role of RIN in habitat segregation

Waters et al. [79] proposed a ‘founder takes all’ idea in which a high density of established founders is able to block secondary colonizers by preventing access to resources. In the case of S. breweri–S. hesperidis on rocky serpentine outcrops, this resource may be shared pollinators. As pollinators commonly move between adjacent plants, low-frequency migrants are subject to fitness losses associated with intrinsic postzygotic reproductive isolation and HPT. Considering the cumulative effects of seed production (figure 3c) and seed viability (figure 3d), migrants had less than 20% the relative fitness of residents, but still produced on average 4.2 (S. hesperidis) to 5.2 (S. breweri) viable seeds per fruit. Initially this may seem too high to attribute a scarcity of local co-occurrence entirely to the effects of frequency-dependent RIN, however our estimates of fecundity in migrants may be overestimates of natural conditions. We established ten experimental blocks at each site, each with up to eight surviving migrants (i.e. conspecifics), a much higher density than we would expect given natural seed dispersal. Thus, conspecific pollen transfer among migrants may have inflated seed production and seed viability compared to if only one or a few seeds dispersed onto a site occupied by heterospecifics. Additionally, we only measured fecundity in a single year, and did not take into account the potentially detrimental effects of inbreeding depression, which would further reduce the fitness of marooned migrants if all of their offspring were produced via insect-facilitated self-fertilization.

5. Conclusion

Myriad factors can facilitate or preclude local coexistence, including adaptation, chance, competition and reproductive dynamics. Long-standing theory [38] predicts that reduced fitness resulting from interspecific matings can exacerbate rarity, and shape species distributions at fine spatial scales. In addition to earlier work documenting RIN in plants [80,81], a resurgence of recent empirical work [15,36,39,40] and theoretical advances [49] support the idea that reproductive dynamics and RIN can be important mechanisms shaping species distributions. It is well appreciated that rarity can confer a fitness advantage by allowing invaders to escape intense intraspecific competition [8,11], however rarity can also entail fitness costs. In much the same way that a frequency-dependent mating disadvantage leads to minority cytotype exclusion in polyploids [44,82,83], RIN can exclude rare outcrossers from becoming established in habitats occupied by a more abundant relative with shared pollinators. Evidence for RIN has now been found in a diversity of organisms including copepods [46], insects [84], amphibians [85] and flowering plants. This work suggests that reproductive dynamics, together with other processes such as niche partitioning and competition, contribute to fine-scale spatial isolation and bi-stable dynamics of close relatives along local contact zones.

Data accessibility

Data and code used to conduct analyses and create figures are archived in the Dryad Digital Repository: https://doi.org/10.25338/B8NG87 [86].

Authors' contributions

K.C. and S.Y.S. co-designed the study; K.C. gathered and analysed the data; K.C. and S.Y.S. wrote the manuscript.

Competing interests

We declare we have no competing interests.

Funding

NSF-1120387 to S.Y.S., NSF-1601186 to K.C. and S.Y.S., a Mildred E. Mathias Graduate Student Research Grant, a UC Davis Natural Reserve System Grant, and the UC Agricultural Experiment Station Supported this research.