Ecological and Mutation‐Order Speciation in Senecio

ABSTRACT

Natural selection shapes how new species arise, yet the mechanisms that generate reproductive barriers remain actively debated. Although ecological divergence in contrasting environments and mutation‐order processes in similar environments are often viewed as distinct speciation mechanisms, we show they can occur simultaneously and act as part of a continuum of selective pressures. In the Senecio lautus species complex, Dune and Headland ecotypes have evolved repeatedly along the Australian coastline. Through crossing experiments and field studies, we find that divergent natural selection promotes strong reproductive isolation between the Dune and Headland ecotypes. While uniform selection maintains reproductive compatibility among ecologically similar Dune populations, geographically distant Headland populations have evolved reproductive barriers despite their convergent prostrate phenotypes, likely driven by adaptation to subtle environmental differences between each Headland location. To understand how this habitat heterogeneity contributes to patterns of reproductive isolation, we extend previous theoretical work on the accumulation of hybrid incompatibilities to account for environmental gradients and polygenic adaptation. We show that the probability of reproductive isolation depends on three factors: how similar the environments are, how complex the genetic architecture is and how selection coefficients are distributed among beneficial mutations. These theoretical findings explain how reproductive isolation arises in systems like Senecio, where multiple forms of selection jointly drive speciation.

1. Introduction

Speciation, the process by which new species arise, is a fundamental driver of biodiversity. However, demonstrating the direct role of natural selection in speciation remains a challenge. Systems exhibiting parallel evolution—where populations independently acquire similar phenotypes under comparable selective pressures—provide a powerful framework for investigating the role of selection in speciation (Schluter 2001; Langerhans and Riesch 2013). These natural replicates of the evolutionary process (Schluter and Nagel 1995; Schluter 2000) enable a thorough examination of the mechanisms that drive reproductive isolation. Evolutionary theory distinguishes two modes of speciation by natural selection: parallel ecological speciation, in which divergent selection in contrasting environments creates reproductive isolation between populations, and mutation‐order speciation, where reproductive isolation arises from the stochastic fixation of different beneficial alleles under similar selection regimes (Schluter 2009; Schluter and Conte 2009; Nosil and Flaxman 2011; Nosil 2012; Crespi and Nosil 2013; but see Sobel et al. 2010; Langerhans and Riesch 2013).

Parallel ecological speciation repeatedly generates reproductive isolation between populations adapting to contrasting environments, while maintaining reproductive compatibility among populations experiencing similar selective pressures (Figure 1A; Schluter and Nagel 1995; Ostevik et al. 2012). The resulting reproductive barriers can manifest through environment‐dependent isolation, where immigrants and hybrids show reduced fitness in parental habitats, and environment‐independent isolation, characterised by intrinsic hybrid incompatibilities regardless of ecological context (Rundle and Nosil 2005; Nosil 2012). The threespine stickleback ( Gasterosteus aculeatus ) system exemplifies this process, where the repeated colonisation of freshwater habitats by marine ancestors has driven parallel evolution of adaptive traits, including reduced body armour and modified feeding morphology (Colosimo et al. 2005; Wund et al. 2008). Reproductive isolation between marine and freshwater forms has evolved primarily through extrinsic mechanisms, while geographically separated freshwater populations maintain reproductive compatibility (Rundle et al. 2000; McKinnon and Rundle 2002; McKinnon et al. 2004). The fact that ecological divergence (marine vs. freshwater) predicts reproductive isolation more reliably than geographic distance demonstrates that natural selection, rather than neutral processes, drives speciation in this system. Similar patterns of parallel ecological speciation have been documented in other animal taxa, including Littorina snails (Johannesson et al. 2010, 2024) and Timema walking stick insects (Nosil et al. 2002; Soria‐Carrasco et al. 2014), although comparable evidence in plants remains limited (see Ostevik et al. 2012; James, Brodribb, et al. 2023; James, Allsopp, et al. 2023).

FIGURE 1.

Schematic diagram representing the differences between parallel ecological speciation and mutation‐order speciation. Circles represent populations, whereas colours represent different environmental conditions. Ancestral populations are in grey. (A) In ecological speciation, intrinsic and extrinsic reproductive isolation (RI) evolves during the adaptation of populations to different environments (i.e., during divergent natural selection). (B) In contrast, during mutation‐order speciation, intrinsic reproductive isolation evolves during the adaptation of populations to similar environments (i.e., during uniform natural selection).

On the other hand, mutation‐order speciation creates reproductive isolation among populations evolving under uniform natural selection (Figure 1B). Although natural selection favours the same beneficial alleles across populations, reproductive incompatibilities arise as different beneficial alleles randomly arise and fix in each population. These alleles are incompatible when combined in hybrid genomes due to negative epistatic interactions, and as a result, can cause reproductive isolation between taxa (Mani and Clarke 1990; Schluter 2009; Nosil and Flaxman 2011; Nosil 2012). The mechanistic basis of mutation‐order speciation differs fundamentally from ecological speciation: rather than arising from maladaptation to parental environments, reproductive barriers emerge from deleterious interactions between independently acquired beneficial mutations. Because these mutations experience positive selection, their fixation and subsequent incompatibilities can accumulate more rapidly than under neutral divergence through genetic drift (Mani and Clarke 1990).

The most compelling examples of mutation‐order speciation in nature come from cases of intragenomic conflict, such as meiotic drive and selfish genetic elements (Presgraves 2010; Crespi and Nosil 2013; Fishman and Sweigart 2018). Experimental studies also provide evidence for mutation‐order speciation, demonstrating the evolution of reproductive barriers between populations adapting to identical environments such as in Drosophila (Hsu et al. 2024) and through extrinsic environmental selection in Saccharomyces (Ono et al. 2017). Yet, convincing evidence for mutation‐order speciation driven by extrinsic selection in natural systems is lacking. This gap likely exists because documenting mutation‐order speciation in nature is challenging: determining whether reproductive barriers arose from the stochastic fixation of beneficial mutations or subtle ecological differences can make it difficult to identify which process predominates in natural systems. Addressing this challenge requires systematic investigation of speciation mechanisms across diverse taxa with broad geographic ranges (Anderson and Weir 2022), particularly in understudied groups such as plants.

The Senecio lautus species complex (Ali 1964; Radford et al. 2004; Thompson 2005) provides an exceptional system for examining the relationship between natural selection and speciation. Two phenotypically distinct coastal ecotypes have repeatedly evolved in the species complex: an ancestral erect Dune ecotype occupying sheltered sand dunes and a derived prostrate Headland ecotype inhabiting exposed rocky outcrops (Figure 2A). These ecotypes form multiple Dune‐Headland parapatric pairs along the Australian coastline (Figure 2B), which can be considered natural replicates of the evolutionary process (Roda, Ambrose, et al. 2013; James, Arenas‐Castro, et al. 2021). Each ecotype is locally adapted to a distinct habitat type (Roda, Liu, et al. 2013; Walter et al. 2016; Walter, Aguirre, et al. 2018; James, Wilkinson, et al. 2021), yet Headland sites exhibit fine‐scale heterogeneity in soil chemistry, microclimate and biotic interactions, which generates a complex mosaic of selective pressures between Headland localities (Roda, Liu, et al. 2013; Walter et al. 2016). This environmental variation has implications for the genetic architecture of adaptation: while Headland populations exhibit convergent prostrate phenotypes, they achieve this convergence through distinct combinations of small‐effect alleles, revealing the polygenic nature of adaptive evolution in the system (Roda, Liu, et al. 2013; James, Wilkinson, et al. 2021; James, Allsopp, et al. 2023; Kaur 2024).

FIGURE 2.

Parallel evolution of Senecio lautus Dune and Headland populations. (A) Illustrations of the Dune and Headland environments and ecotypes. (B) Geographic distribution in Australia of the 16 populations in the study (blue Dune ecotype, orange Headland ecotype, pink Alpine ecotype, green Inland ecotype). (C) Bayesian phylogeny constructed with 13 neutral markers implemented in *BEAST. Numbers on nodes are credible posterior probabilities. (D) Bayesian assignment of individuals to genetic clusters within STRUCTURE for $K=2$. Each individual is depicted as a bar, with colours representing ancestry proportions to each $K$ cluster.

Research in Senecio has revealed complex interactions between environment‐dependent and intrinsic reproductive barriers. Reciprocal transplant experiments demonstrate extrinsic isolation between Dune and Headland ecotypes, indicating that divergent selection maintains reproductive boundaries through adaptation to contrasting environments (Melo et al. 2014; Richards and Ortiz‐Barrientos 2016; Richards et al. 2016; Walter et al. 2016; Walter, Wilkinson, et al. 2018; Wilkinson et al. 2021). While F ₁ hybrids between eastern Dune and Headland coastal populations are largely reproductively compatible (Melo et al. 2014; Richards and Ortiz‐Barrientos 2016; Walter et al. 2016), F ₂ and subsequent generations display strong intrinsic reproductive isolation (Walter et al. 2020; Wilkinson et al. 2021). Finally, the identification of a shared genetic basis between gravitropism—a key ecological trait differentiating the Dune and Headland ecotypes—and hybrid sterility in Senecio demonstrates a direct pathway through which natural selection on ecological traits promotes speciation (Wilkinson et al. 2021).

In this work, we use empirical analyses to test fundamental predictions of parallel ecological and mutation‐order speciation in the Senecio lautus species complex. We investigate how environmental heterogeneity shapes the accumulation of reproductive barriers and create a mathematical framework to explicitly incorporate environmental gradients and polygenic architectures, providing insights into speciation across heterogeneous landscapes. Our work shows that deterministic and stochastic processes interact in complex ways to generate reproductive isolation during speciation.

2. Methods

2.1. Phylogenetic Independence of Replicate Populations

2.1.1. Samples

We studied parallel speciation in six parapatric Dune‐Headland Senecio lautus population pairs from the eastern and southern coasts of Australia (Figure 2A, Table S1). For each population, we used previously collected leaf samples from 12 individuals (Roda, Ambrose, et al. 2013). To strengthen our phylogenetic analysis, we included two additional ecotypes (two Inland and two Alpine populations; n _pop = 11) and the closely related African species S. madagascariensis (see Roda, Ambrose, et al. 2013; Roda, Liu, et al. 2013) as an outgroup (n = 4; Table S1). We extracted DNA from each sample using a modified CTAB protocol (Clarke 2009), purified samples using Promega's Wizard SV Gel and PCR Clean‐Up System and standardised each sample to 30 ng/μL. We undertook targeted re‐sequencing of nuclear genomic regions (primarily intronic and intergenic regions), resulting in 13 neutral markers across all populations (see Material 1: Data S1 for details on primers, library preparation, sequencing, bioinformatics and neutrality tests; Tables S2 and S3). We note that this genetic work was undertaken before high‐throughput sequencing was available for non‐model organisms. Recent research using genotyping‐by‐sequencing in S. lautus has demonstrated phylogenetic independence of five out of the six Dune‐Headland population pairs in the current study (James, Arenas‐Castro, et al. 2021) – we primarily use this small subset of 13 loci to confirm the independence of all six population pairs. For a detailed taxonomy of the S. lautus species complex, please see supplementary tables in Roda, Ambrose, et al. (2013).

2.1.2. Phylogeny

To examine the phylogenetic independence of the S. lautus population pairs, we conducted a Bayesian phylogenetic analysis using the 13 neutral loci with *BEAST v1.7.5 (Heled and Drummond 2010), an extension of BEAST (Drummond et al. 2012) designed for multilocus and multi‐individual species tree estimation. We generated the XML file for *BEAST using BEAUTi v1.7.5 (Drummond et al. 2012) and ran the analysis with a chain length of 300,000,000 under a strict molecular clock. Using ITS as our reference locus (mutation rate = 4.13 × 10⁻⁹ subs/site/year for herbaceous plants; Kay et al. 2006), we estimated relative mutation rates for all other loci. The HKY model (Hasegawa et al. 1985) best fit our 13 loci based on the Bayesian Information Criterion implemented in jModelTest v2.1.1 (Posada and Crandall 1998). We used a Yules speciation process for species tree estimation, which assumes that lineages split at a constant rate. We generated the maximum clade credibility tree in TreeAnnotator v1.7.5 (Drummond et al. 2012) with a burn‐in of 10,000 steps and visualised the tree in FigTree v1.4.4 using S. madagascariensis as the outgroup.

2.1.3. Population Structure

To assess the major population structure in the data, we identified the most likely number of genetic clusters (K) across all populations using STRUCTURE v2.3.4, a Bayesian Markov Chain Monte Carlo (MCMC) approach (Pritchard et al. 2000). We performed analyses with the variable sites of the 13 neutral loci using an admixture model and the correlated allele frequency model (Falush et al. 2003). Following guidelines from Gilbert et al. (2012) and Janes et al. (2017), we tested K values from 1 to 16, running 20 iterations per K with a burn‐in of 100,000 and MCMC run length of 100,000. We confirmed convergence of model parameters through visual inspection of MCMC summary statistics. The most likely K value was evaluated using methods from Pritchard et al. (2000) and Evanno et al. (2005), implemented in STRUCTURE HARVESTER v0.6.93 (Earl and vonHoldt 2012). Since both methods tend to overestimate K, and higher K values provided no additional clustering information, we selected the smallest K that captured the major population structure in the data to assess whether populations cluster by geography or ecology. We aligned cluster assignments across iterations using CLUMPP v1.1 (Jakobsson and Rosenberg 2007) with the complete search algorithm and visualised the results using DISTRUCT v1.1 (Rosenberg 2004).

2.2. Intrinsic Reproductive Isolation Between and Within Ecotypes

2.2.1. Samples

To investigate patterns of reproductive isolation, we selected four Dune‐Headland population pairs: two from the eastern clade (LH and CH) and two from the southeastern clade (VC and SA) of the phylogeny (Figure 2; Table S1). From each population, we collected seeds from 30 individuals in the field spaced at least 10 m apart. We stored seeds in dry conditions at 4°C at The University of Queensland. To create seed stocks for each population as well as to eliminate maternal effects (Bischoff and Müller‐Schärer 2010), we first germinated and grew plants for one generation. To induce germination, we scarified seeds by trimming 1 mm off the micropyle side and placed them on moist filter paper in Petri dishes in a controlled growth room at 25°C. We kept seeds in darkness for 3 days to promote root elongation and then transferred them to a 12 h:12 h light: dark cycle for 7 days to encourage vegetative growth. We then transplanted seedlings into 0.25 L pots containing a soil mix (70% pine bark, 30% coco peat) supplemented with slow‐release Osmocote fertiliser (5 kg/m³) and Suscon Maxi insecticide (830 g/m³). After 2 months, we conducted controlled crosses by repeatedly rubbing mature flower heads together over 3–5 days. This ensured bi‐directional pollen transfer and maximum fertilisation opportunity. The resulting seeds were stored at 4°C.

2.2.2. Reproductive Isolation Measures

We quantified two components of intrinsic reproductive isolation: F ₁ seed set and F ₁ viability (germination). To assess these barriers, we compared the success of crosses between populations of the same ecotype (D × D and H × H population crosses) with those of different ecotypes (D × H). This was undertaken both within and between the eastern and southern phylogenetic clades to account for phylogenetic divergence time (Coyne and Orr 1997). We also conducted intra‐population crosses as controls. Using the protocol outlined above, we grew up to 14 families from each of the eight populations from the eastern and southeastern clades of the phylogeny (Table S1). When the plants flowered, we performed intra‐ and inter‐population crosses twice daily, completing 260 crosses in total (see Table S4).

To measure reproductive isolation at the F ₁ seed set stage, we counted the proportion of fertilised seeds per flower head, identifying unfertilised seeds by their thin size and pale colour. We acknowledge that our approach is not able to disentangle gametic isolation from early‐acting F ₁ incompatibilities, as embryo development studies would be needed to reveal the precise developmental mechanisms through which reproductive isolation is generated. Nevertheless, we treat F ₁ seed set as a composite measure of pre‐zygotic and very early post‐zygotic barriers, which collectively represent the earliest stages of reproductive isolation between populations. We quantified the strength of reproductive isolation (RI) for F ₁ seed set between populations using the linear formula in Sobel and Chen (2014):

$$\mathrm{RI}=1-2\times \left(\frac{H}{H+C}\right)$$

where H represents the proportion of fertilised seeds from the interspecific crosses and C is the average proportion of fertilised seeds from the two parental intraspecific crosses.

To assess reproductive isolation at the F ₁ viability stage, we measured germination rates from the crosses defined above. For each of the 260 crosses, we germinated five seeds by placing them onto moist filter paper in Petri dishes (one cross per Petri dish). We used the same germination conditions as above but without scarification to test the intrinsic ability of embryos to emerge from the seed coat. We specifically chose not to scarify these seeds to capture potential developmental incompatibilities during germination, as Australian Senecio populations typically exhibit minimal dormancy (> 90% germination without scarification in field conditions). The strength of reproductive isolation for F ₁ viability was calculated using the formula above, where H represents the proportion of germinated seeds from the interspecific crosses and C is the average proportion of germinated seeds from the two parental intraspecific crosses.

2.2.3. Statistical Analyses

We analysed patterns of intrinsic reproductive isolation using linear mixed‐effect models in R v4.1.0 (R Core Team 2021) with the lmerTest package (Kuznetsova et al. 2017):

$$\mathrm{RI}=\text{Clade}+\text{Ecotype}+\left(1|\text{CrossType}\right)$$

where RI is the reproductive isolation values for either F ₁ seed set or F ₁ viability, Clade indicates whether crosses occurred within or between phylogenetic clades, Ecotype denotes whether crosses were between the same or different ecotypes and CrossType is a random effect with a random intercept model, representing the population comparison. For both F ₁ seed set and viability models, we initially tested for an interaction between Clade and Ecotype but removed it due to non‐significance. To examine specific cross‐type effects, we ran additional linear models:

$$\mathrm{RI}=\text{Clade}+\text{EcotypeCross}$$

where EcotypeCross specifies the exact cross comparison (D × D, H × H, D × H or H × D). Again, we removed the interaction terms as they were non‐significant. Finally, we used one‐sided t‐tests to determine whether reproductive isolation values for D × D and H × H crosses (both within and between clades) were significantly greater than zero.

2.3. Extrinsic Reproductive Isolation Within Ecotypes

2.3.1. Samples

We tested for the prerequisite conditions of mutation‐order speciation by examining extrinsic reproductive isolation within ecotypes. If environments are sufficiently uniform to allow mutation‐order speciation in Senecio, populations of the same ecotype should exhibit similar performance across different localities within the same environment type, indicating consistent selection pressures. Throughout the remainder of the manuscript, we define extrinsic reproductive isolation as reduced fitness of immigrants or hybrids that is directly caused by divergent natural selection between environments (Schluter 2000), rather than using the broader definition that includes any environmentally dependent fitness effects (Thompson et al. 2023). We assessed extrinsic reproductive isolation by reanalyzing data from a previous field transplant experiment by Walter et al. (2016). Briefly, in this study seeds of six populations (three Dune‐Headland replicate pairs) were generated under common garden conditions and then transplanted (n = 150–180 seeds per population in each environment; Table S1) into the dune and headland environments of the Lennox Head (LH) population pair. Seeds were planted in fully randomised grids with six replicate blocks in each environment, seedling survival was monitored over time and seedling establishment was recorded when plants had grown 10 true leaves. For further details, see Walter et al. (2016).

2.3.2. Statistical Analyses

We compared local population performance (LH Dune seeds planted in the LH dune habitat; LH Headland seeds planted in LH headland habitat) against non‐local populations (i.e., Stradbroke Island and Cabarita Beach Dune and Headland seeds in the LH dune and headland environments, respectively). Our analyses examined two fitness components: survival and establishment.

Using the R package coxme (Therneau 2024), we modelled survival separately for dune and headland environments:

$$\text{Survival}=\text{Population}+\left(1|\text{Block}\right)$$

where Survival is days survived (censored at day 320), Population indicates seed source location and Block is a random effect for replicate experimental blocks within each environment.

We then tested whether local populations showed similar probabilities of reaching seedling establishment compared to non‐local populations using a generalised linear mixed‐effects model in the R package lme4 (Bates et al. 2015). The model used the binomial family with a logit link function to model the probability of establishment success. We analysed dune and headland environments separately with the model:

$$\text{Establishment}=\text{Population}+\left(1|\text{Block}\right)$$

where Establishment is a binary variable indicating whether a plant reached 10 leaves and Block is a random effect for replicate experimental blocks within each environment.

In both analyses, we tested whether non‐local populations performed similarly to the local population. To do this, we explicitly set the local population as the reference level (intercept) for the Population factor. This parameterisation allows coefficient estimates to represent deviations of non‐local populations from local population performance, where significant negative coefficients indicate local adaptation.

2.4. Connecting Ecology and Phenotype With Intrinsic Reproductive Isolation

To further assess the role of mutation‐order in driving speciation in Senecio, we examined the relationship between intrinsic reproductive isolation and environmental or phenotypic distance. For mutation‐order speciation to hold true, we expect no associations because populations are expected to fix advantageous mutations randomly rather than in response to ecological or phenotypic differences (Schluter 2009).

2.4.1. Intrinsic Reproductive Isolation and Environmental Distance

To test the role of ecological differences to intrinsic isolation, we explored the association between intrinsic reproductive isolation (F ₁ seed set and F ₁ viability, as measured above) and environmental variance among Headland populations. We used previously published soil data from Roda, Liu, et al. (2013) to quantify environmental variance, where 38 soil variables (nutrients, salts and metals) were measured for each population. We used R for statistical analyses and scaled each soil variable to a mean of zero and a standard deviation of one. We used the vegan package (Blanchet et al. 2018) in R to create a distance matrix of the soil variables for each population. We performed multidimensional scaling to calculate the Euclidean distance between populations in environmental space using the first five principal component axes, which explained more than 95% of the variance. To estimate genetic distance, we used the ape package (Paradis et al. 2004) in R to calculate pairwise distances between populations based on branch lengths from the phylogeny constructed above.

To assess the relationship between reproductive isolation and environmental distances, we performed linear models:

$$\mathrm{RI}=\text{GeneticDistance}+\text{EnvironmentalDistance}$$

where RI is the reproductive isolation value for either F ₁ seed set or F ₁ viability.

2.4.2. Intrinsic Reproductive Isolation and Phenotypic Distance

We examined the association between intrinsic reproductive isolation (F ₁ seed set and F ₁ viability) with phenotypic variance between Headland populations. We used a combination of previously published phenotypic data from Walter, Aguirre, et al. (2018) and unpublished data for additional populations measured at the same time, where phenotypes were measured in controlled glasshouse conditions. This dataset consists of four plant architecture traits (vegetative height, stem length/width, number of branches and stem diameter) and six leaf traits (area, perimeter²/area², circularity, number of indents, indent width and indent depth), measured for up to 17 individuals per population (Table S1); see Walter, Aguirre, et al. (2018) for specific details on growing conditions and trait measurements. We scaled each phenotypic variable to have a mean of zero and a standard deviation of one, calculated the distance matrix of the phenotypic traits for each population and performed multidimensional scaling to calculate the Euclidian distance between populations in environmental space using the first seven principal component axes, which explained more than 95% of the variance. We used the genetic distances calculated above in ape. We performed linear models to assess the relationship between reproductive isolation and phenotypic distances, again averaging the reciprocal crosses:

$$\mathrm{RI}=\text{GeneticDistance}+\text{PhenotypicDistance}$$

where RI is the reproductive isolation value for either the F ₁ seed set or F ₁ viability for each population comparison.

2.5. Mathematical Analysis of Speciation

To further understand how various forms of natural selection drive patterns of reproductive isolation in Senecio, we created a mathematical framework to explore how both ecological and mutation‐order processes can jointly underlie patterns of reproductive isolation. While existing speciation models typically treat these as distinct mechanisms, our framework extends the Unckless–Orr framework of how Dobzhansky–Muller Incompatibilities (DMIs) accumulate in identical environments (Unckless and Orr 2009) to incorporate a continuum of environmental similarity and genetic complexity. By introducing an environmental symmetry parameter ($\phi$) and considering polygenic architectures, our model provides theoretical predictions for how reproductive isolation might emerge under different scenarios of environmental heterogeneity and genetic complexity. This mathematical approach complements our empirical investigations in Senecio by providing a theoretical foundation to interpret the complex patterns of reproductive isolation that can emerge during speciation.

2.5.1. Foundations

The foundation of our analysis builds upon the classic Unckless–Orr model examining DMI formation between two populations adapting to identical environments (Unckless and Orr 2009). This model considers two interacting loci A and B, each with ancestral (A ₀, B ₀) and derived (A ₁, B ₁) alleles. The ancestral genotype (A ₀, B ₀) has a fitness of 1, while genotypes with single derived alleles A ₁ B ₀ and A ₀ B ₁ have fitness values of $1+s_{\mathrm{A}}$ and $1+s_{\mathrm{B}}$ respectively. The combination of both derived alleles (A ₁ B ₁) creates an incompatibility with fitness $1-t$, where $t$ quantifies the severity of the DMI (Orr 1995). When $t=1$, the hybrid combination is completely inviable/sterile, while smaller values of $t$ represent partial reproductive isolation. The inclusion of $t$ in the model helps capture the concept that while individual‐derived alleles can be beneficial in their genetic backgrounds, their combination in hybrids can be deleterious $\left(t>0\right)$. This is a key feature of the Dobzhansky‐Muller model of speciation—incompatibilities arise not from individual mutations being deleterious, but from their negative epistatic interactions when brought together in hybrids.

Under strong selection $\left(\mathrm{Ns}>>1\right)$ and weak mutation $\left(\mathrm{N\mu }<<1\right)$, where adaptive trajectories retain stochastic elements despite directional selection, the probability of DMI formation depends on selection coefficients through the Unckless–Orr model:

$$P_{\mathrm{DM},\mathrm{U}}=\frac{2s_{\mathrm{A}}{s}_{\mathrm{B}}}{{\left(s_{\mathrm{A}}+s_{\mathrm{B}}\right)}^2}$$

We use the subscript U to refer to the Unckless–Orr model. This equation reveals that during adaptation to identical environments, DMIs are most probable when selection coefficients are similar ($s_{\mathrm{A}}\approx s_{\mathrm{B}}$), as populations experience maximum stochasticity in the order of mutation fixation. When selection coefficients are equal, either beneficial mutation is equally likely to fix first in each population, maximising the probability that populations will fix different alleles. However, when selection coefficients differ substantially, both populations are likely to fix the mutation with the stronger selective advantage first, reducing the probability of DMI formation.

2.5.2. Extensions

We extend this framework by modelling how environmental differences create asymmetric selection on alleles between populations. For any locus, we assume its selection coefficient in its ‘home’ environment $\left(s_{\mathrm{i}}\right)$ is reduced by a factor $\phi$ when it occurs in the alternative environment $\left(\phi s_{\mathrm{i}}\right)$, that is, $\phi =\frac{s_{\text{foreign}}}{s_{\text{home}}}$. Thus, this parameter $\phi$ ∈ [0, 1] captures the degree to which selection pressures transfer between environments, with $\phi$ = 1 indicating identical selection and $\phi$ = 0 indicating complete asymmetry where alleles that are beneficial in one environment are neutral in alternative environments. Because fixation time scales as $\frac{1}{s}$, $\phi$ also rescales the relative rate of adaptive substitutions between environments.

Under these assumptions, the probability of DMI formation becomes:

$$P_{\mathrm{DM},\mathrm{E}}=\frac{s_{\mathrm{A}}{s}_{\mathrm{B}}\left(1+{\phi }^2\right)}{\left(s_{\mathrm{A}}+\phi s_{\mathrm{B}}\right)\left(\phi s_{\mathrm{A}}+s_{\mathrm{B}}\right)}$$ (Equation 1)

When environments are identical $\left(\phi =1\right)$, the equation reduces to the classic Unckless–Orr model. As environmental differences increase (decreasing $\phi$), selection becomes increasingly asymmetric between populations, making it more likely they will fix different alleles. In the extreme case of complete asymmetry $\left(\phi =0\right)$, DMI formation becomes inevitable because alleles strongly favoured in one environment are neutral in the other.

For polygenic traits involving $n$ interacting loci with selection coefficients $s_{\mathrm{i}}$, the probability of at least one DMI forming is:

$$P_{\mathrm{DM},\mathrm{P}}=1-\frac{\phi }{{\sigma }_1{\sigma }_2}\sum _{i=1}^n{s}_{\mathrm{i}}^2$$ (Equation 2)

where ${\sigma }_1$ and ${\sigma }_2$ represent the sums of selection coefficients in each population. When environments are identical $\left(\phi =1\right)$, this reduces to an $n$‐locus extension of the two‐locus Unckless and Orr (2009) model, while complete asymmetry ($\phi$ = 0) guarantees DMI formation. For equal selection coefficients $\left(s_{\mathrm{i}}=s\right)$, the formula simplifies to:

(Equation 3)

This shows how genetic complexity amplifies the effects of selection asymmetry through combinatorial growth in potential incompatibilities (see Material 2: Data S1 for detailed derivations of all equations).

2.5.3. Implementation

We implemented these models using Python v3.10.9 and NumPy v1.23.5 (Harris et al. 2020) for numerical calculations and Matplotlib v3.9.2 (Hunter 2007) for visualisation. Our analysis explored selection coefficient space $\left(s_{\mathrm{A}}\mathrm{vs}{s}_{\mathrm{B}}\right)$, environmental similarity $\left(\phi \right)$ and the impact of interacting loci number $\left(n\right)$ on DMI probability, providing a theoretical explanation for the observed patterns of reproductive isolation in Senecio.

3. Results and Discussion

3.1. Empirical Patterns of Parallel Ecological and Mutation‐Order Speciation

3.1.1. Phylogenetic Independence Reveals Multiple Origins of Ecotypes

To investigate speciation in Senecio, we first confirmed the independent origins of Dune and Headland populations in this study (also see Roda, Ambrose, et al. 2013; James, Arenas‐Castro, et al. 2021). Using 13 neutral markers, we found that populations cluster into two main clades (eastern and southeastern; Figure 2C,D), that align with their coastal geographic distribution (Figure 2B). As expected, the phylogenetic relationships support the parallel evolution of ecotypes: Dune‐Headland population pairs form sister taxa, with both ecotypes present within each clade (Figure 2C). This result confirms previous independent datasets in the Senecio system where populations cluster by geography and not ecology (Roda, Ambrose, et al. 2013; James, Arenas‐Castro, et al. 2021). Although gene flow can create false signals of parallel evolution (Endler 1977; Barton and Hewitt 1985; Coyne and Orr 2004; Bierne et al. 2013), previous work in Senecio showed that gene flow among Senecio populations, both within and between ecotypes, is negligible (James, Arenas‐Castro, et al. 2021). These findings indicate that the observed phylogenetic patterns reflect the independent adaptive divergence of Dune‐Headland pairs along the Australian coastline.

3.1.2. Divergent Selection Drives Strong Reproductive Isolation Between Ecotypes

Parallel ecological speciation generates specific predictions about patterns of reproductive isolation: populations adapting to contrasting environments should develop reproductive incompatibilities, while those inhabiting similar environments should maintain reproductive compatibility. In contrast, mutation‐order processes predict that populations evolve reproductive barriers even when adapting to similar environments through the independent fixation of different beneficial alleles (Schluter and Nagel 1995; Ostevik et al. 2012). To evaluate these contrasting predictions in Senecio, we quantified intrinsic reproductive isolation between populations from the same environments (within Dune or within Headland ecotypes) and different environments (between Dune and Headland ecotypes). Our analysis focused on two quantitative components of reproductive isolation, measured relative to parental averages: F ₁ hybrid seed production and F ₁ hybrid viability during germination (Table S5).

Consistent with the role of divergent selection, populations from similar environments showed higher reproductive compatibility than those from different environments. This pattern held for both seed set ($F_{\mathrm{1,15.34}}=11.11,p=0.004,R^2=0.62$ after taking into account the effect of clade $F_{\mathrm{1,15.23}}=15.40,p=0.001$) and viability ($F_{\mathrm{1,13.91}}=8.33,p=0.012,R^2=0.59$ after taking into account the impact of clade $F_{\mathrm{1,13.80}}=8.34,p=0.012$; Figure 3A). Crosses within clades were generally viable, with crosses between the same ecotypes showing some hybrid vigour (reproductive isolation values < 0), suggesting complementary genetic interactions. The most substantial (almost complete) reproductive isolation occurred in Dune‐Headland crosses between the two clades (Figure 3A). These results indicate that natural selection drives speciation in Senecio, with reproductive isolation increasing as genetic divergence accumulates between ecotypes. This pattern mirrors findings in other systems, including stickleback fish (Hendry et al. 2009; Stuart et al. 2017), Littorina snails (Johannesson et al. 2024) and cichlid fishes (Weber et al. 2021).

FIGURE 3.

Strength of intrinsic reproductive isolation (RI) of coastal Dune (D) and Headland (H) Senecio lautus populations for F ₁ seed set (left graphs) and F ₁ viability. Data points represent the mean of multiple crosses for each population comparison (F ₁ seed set: N _total = 260 crosses; F ₁ viability: N _total = 259 crosses). Crosses were undertaken within and between the two clades defined in the phylogeny of Figure 1. Reproductive isolation was examined (A) between the same ecotype (D × D and H × H) or between different ecotypes (D × H and H × D), as well as (B) for each separate cross type (D × D, H × H, D × H and H × D). Positive values imply that hybrids perform worse than parents, and negative values imply that hybrids perform better than parents.

3.1.3. Unexpected Reproductive Barriers Emerge Among Convergent Headland Populations

To establish whether the observed patterns of intrinsic reproductive isolation represent parallel ecological speciation in Senecio, we must demonstrate reproductive compatibility within each ecotype, for both hybrid viability and fertility. Cross type (D × D, H × H, D × H or H × D) significantly affected reproductive isolation for both F ₁ seed set ($F_{3,29}=6.12,p=0.002,R^2=0.53$ after taking into account the effect of clade $F_{1,29}=14.53,p=0.0007$) and F ₁ viability ($F_{3,29}=4.41,p=0.01,R^2=0.40$ after taking into account the effect of clade $F_{1,29}=6.40,p=0.017$; Figure 3B). Dune populations showed the expected pattern of parallel ecological speciation, maintaining reproductive compatibility within and between clades. Their reproductive isolation values were not significantly greater than zero for either F ₁ seed set (within clades $t_5=-2.02,p=0.85$, between clades $t_5=-0.05,p=0.52$) or F ₁ viability (within clades $t_5=-4.68,p=0.93$, between clades $t_5=-1.33,p=0.88$). Headland populations were similarly compatible within clades for both F ₁ seed set $\left(t_3=-2.01,p=0.93\right)$ and F ₁ viability $\left(t_3=-1.0,p=0.80\right)$ and between clades for viability $\left(t_5=1.14,p=0.15\right)$.

Our analyses revealed an unexpected pattern of reproductive incompatibility among Headland populations from different clades, manifested as significantly reduced F ₁ seed set (t ₅ = 6.93, p = 0.0005; Figure 3B). While this pattern contradicts predictions of ecological speciation, it aligns with the expectation that mutation‐order processes contribute to speciation. The reproductive isolation between Headland populations suggests parallel evolution of prostrate phenotypes occurred through distinct adaptive trajectories, resulting in different combinations of beneficial alleles in each population that create intrinsically unfit hybrids between populations. This interpretation is supported by previous genetic analyses in Senecio, which demonstrated that convergent Headland morphology evolved through the selection of distinct alleles and genes across populations (Roda, Liu, et al. 2013; James, Wilkinson, et al. 2021). To further explore the role of mutation‐order effects on speciation in Senecio, we tested additional predictions of speciation theory.

3.1.4. Ecological Differences Do Not Explain Reproductive Isolation in Headland Populations

Previous work in Senecio revealed contrasting patterns of environmental variation. Each ecotype occupies a distinct soil type, where variation within ecotypes is lower than between ecotypes (Roda, Liu, et al. 2013; Walter et al. 2016). The soils of different Headland populations share fundamental characteristics of being shallow and nutrient‐rich but vary in specific mineral composition based on local geology. This habitat heterogeneity contrasts with the Dune populations, which occupy remarkably similar habitats even when separated by thousands of kilometres. To determine whether ecological variance reflects fitness variation within each ecotype, we reanalyzed reciprocal transplant experiments from Walter et al. (2016). In these experiments, three eastern‐coast Dune populations were transplanted into the dune habitat at Lennox Head, while three eastern‐coast Headland populations were transplanted into the rocky headland. Dune populations showed no signal of local adaptation: non‐local individuals performed equally well or better than local Lennox Head individuals in both survival (Figure 4A) and establishment (Figure 4B). In contrast, the local Lennox Head Headland population consistently outperformed non‐local populations in both fitness components (Figure 4).

FIGURE 4.

Strength of extrinsic reproductive isolation of coastal Dune and Headland Senecio lautus populations. Fitness of three replicate populations (LH—Lennox Head, SI—Stradbroke Island; CB—Cabarita Beach) from the Dune and Headland ecotypes transplanted into the dune and headland environments, respectively, at Lennox Head (Dune: N _total = 479 individuals; Headland: N _total = 480 crosses). Fitness was measured by (A) survival curves, where the local population is denoted in colour and dashed lines are 95% confidence intervals, and (B) the probability of seedling establishment (production of 10 leaves), where asterisks denote significant differences from the local population.

Given the fitness differences among Headland populations at Lennox Head, we tested whether environmental variation (measured by soil ecology) explains patterns of intrinsic reproductive isolation between Headland localities. If ecological divergence in soil composition drives reproductive barriers, we expect to find a correlation between measured environmental differences and intrinsic reproductive isolation across Headland populations. After controlling for genetic distance, we found no significant relationship between soil‐based ecological divergence and reproductive isolation for F ₁ seed set $\left(F_{1,2}=3.03,p=0.224\right)$ or F ₁ viability $\left(F_{1,2}=0.065,p=0.822\right)$. The absence of an environment‐reproductive isolation correlation for soil composition is consistent with mutation‐order effects on speciation, where populations fix advantageous mutations randomly rather than in response to ecological differences (Schluter 2009). However, we cannot disregard the potential role of ecological differences in generating reproductive incompatibilities for two main reasons: other non‐measured ecological variables may be the main selective force driving the association, and the relatively small sample size of soil data reduces our power to detect an effect.

3.1.5. Phenotypic Divergence Does Not Predict Reproductive Isolation in Headland Populations

We next asked whether subtle phenotypic variation among Headland populations could explain their reproductive isolation patterns. If divergent rather than uniform selection drives intrinsic reproductive isolation between Headlands, populations with greater phenotypic differences should show stronger reproductive barriers. However, after controlling for genetic distance, we found no significant relationship between phenotypic divergence and reproductive isolation for either F ₁ seed set $\left(F_{1,2}=3.91,p=0.187\right)$ or F ₁ viability $\left(F_{1,2}=8.63,p=0.099\right)$. The absence of a correlation between phenotypic divergence and intrinsic reproductive isolation further supports mutation‐order effects on speciation, where genetic incompatibilities independently arise and contribute to the same adaptive phenotypes across populations (Schluter 2009). However, we again cannot disregard a possible association between phenotypic differences and reproductive incompatibilities due to potential unmeasured ecological variables and relatively small sample sizes for this analysis.

3.2. Multiple Modes of Speciation in Senecio

In the Senecio lautus species complex, patterns of reproductive isolation reveal a more nuanced speciation process than the traditional dichotomy between ecological and mutation‐order mechanisms (Figure 5). While divergent natural selection in contrasting coastal environments has repeatedly driven the evolution of reproductive barriers between Dune and Headland ecotypes (Melo et al. 2014; Richards and Ortiz‐Barrientos 2016; Richards et al. 2016; Walter et al. 2016; Walter, Wilkinson, et al. 2018; Wilkinson et al. 2021), the pattern within each ecotype reveals an unexpected asymmetry. In Dunes, populations experience relatively uniform selection due to consistent environmental pressures (Roda, Liu, et al. 2013; Walter et al. 2016), allowing for reproductive compatibility despite geographic separation of thousands of kilometres and ancient divergence. Crosses between distant Dune populations even exhibit enhanced fitness through hybrid vigour. In contrast, Headland populations have evolved reproductive barriers in crosses between the phylogenetic clades, despite convergence on similar prostrate phenotypes. This suggests that the repeated evolution of similar phenotypes may obscure profound differences in how populations traverse adaptive landscapes. These incompatibilities between Headland populations do not correlate with environmental or morphological differences, suggesting that mutation‐order processes are at play. Thus, it appears there is mutation‐order speciation of the Headland ecotype between clades.

FIGURE 5.

The dynamics of speciation in Senecio lautus. (A) Patterns of intrinsic reproductive isolation in Senecio. Strong intrinsic reproductive isolation has evolved between ecotypes. In contrast, weaker intrinsic reproductive isolation has evolved between Headland (H) populations but not the Dune (D) populations, as shown by the thickness of the reproductive isolation lines. These patterns of intrinsic isolation are apparent for the most divergent crosses (i.e., crosses between clades). Reproductive isolation values are means for the crosses for F ₁ seed set. (B) Schematic diagram representing fitness landscapes of the Dune (blue) and Headland (orange) ecotypes. Dune populations share one optimum, driven by uniform selective pressures, whereas the heterogeneous headland environments likely lead to several optima for the Headland ecotype.

Notably, our findings in Senecio provide evidence for mutation‐order speciation driven by extrinsic selection rather than intragenomic conflict, which has been the source of most documented cases of mutation‐order speciation in nature (Presgraves 2010; Crespi and Nosil 2013; Fishman and Sweigart 2018). The accumulation of reproductive barriers between phenotypically convergent Headland populations demonstrates how responding to similar selective environments can drive genetic incompatibilities through alternative adaptive trajectories, expanding our understanding of how mutation‐order processes can operate in natural systems. However, the environmental heterogeneity (Roda, Liu, et al. 2013; Walter et al. 2016) and localised selective pressures among Headland populations indicate that patterns of reproductive isolation may not strictly align with mutation‐order speciation.

In Senecio, the combination of divergent, uniform and similar selection forms a continuum of selective pressures where reproductive isolation can arise through multiple mechanisms (also see Langerhans and Riesch 2013). While theoretical frameworks have traditionally treated mutation‐order and ecological speciation as distinct processes (Schluter 2009; Nosil and Flaxman 2011), empirical evidence suggests natural systems operate along a selection continuum (Johannesson et al. 2024). This complexity is particularly evident in plants, such as the Senecio system, where fine‐scale environmental heterogeneity creates intricate selective mosaics that do not neatly fit into the binary classification of uniform versus divergent selection (Lowry et al. 2019; James, Brodribb, et al. 2023).

Our work illustrates how different forms of natural selection are simultaneously driving multiple modes of speciation within a single system, where ecological and mutation‐order processes represent the extremes of this continuum.

Our empirical findings in Senecio create several theoretical challenges. Our discovery of reproductive compatibility among Dune populations contrasted with incompatibility between distant Headland populations raises fundamental questions: How do systems of parallel ecotypic evolution evolve different forms of reproductive isolation within and between ecotypes? How can populations that evolved nearly identical phenotypes accumulate reproductive barriers? And what is the effect of genetic architecture and environmental variance in creating patterns of reproductive isolation during speciation? The complex patterns we observed in Senecio cannot be adequately explained by existing theoretical frameworks that often treat ecological and mutation‐order speciation as discrete processes. Although theory has addressed components of these problems (e.g., Orr 1995; Barton 2001; Orr and Turelli 2001; Porter and Johnson 2002; Gavrilets 2004; Palmer and Feldman 2009; Fierst and Hansen 2010; Nosil and Flaxman 2011; Bank et al. 2012; Chevin et al. 2014; Fraïsse et al. 2014; Thompson et al. 2019, 2024; Yamaguchi and Otto 2020), many models oversimplify the genetic architectures underlying adaptation. Additionally, current frameworks inadequately account for how genetic architecture interacts with environmental heterogeneity to drive the accumulation of reproductive barriers—a critical factor in Senecio where Headland environments show heterogeneity between localities while Dune environments remain relatively uniform. To address these empirical patterns, we developed a theoretical framework that explains speciation under a range of environmental similarities and genetic architectures.

3.3. Theoretical Predictions Under Variable Selection Asymmetry

To place our empirical findings in Senecio within a broader, predictive framework and to disentangle the relative roles of environment and genetic complexity in speciation, we developed a mathematical extension of the Unckless–Orr model. During speciation, reproductive isolation can emerge through the stochastic fixation of alternative beneficial alleles under similar selective pressures (as with mutation‐order speciation), the deterministic fixation of alternative alleles under divergent selective pressures (as with ecological speciation) or a combination of both. Our mathematical framework demonstrates how these processes represent a continuum governed by three key parameters: the distribution of selection coefficients (s), environmental symmetry ($\phi$) and the genetic architecture (n) of the interacting loci (summarised in Table 1).

TABLE 1.

Theoretical framework for the evolution of reproductive isolation under variable selection asymmetry.

Component	Mathematical form	Key parameters
Classic Unckless–Orr Model	$P_{\mathrm{DM},\mathrm{U}}=\left(2s_{\mathrm{A}}{s}_{\mathrm{B}}\right)/{\left(s_{\mathrm{A}}+s_{\mathrm{B}}\right)}^2$	$s_{\mathrm{A}}$, $s_{\mathrm{B}}$: selection coefficients
Environmental effects model	$P_{\mathrm{DM},\mathrm{E}}=\frac{s_{\mathrm{A}}{s}_{\mathrm{B}}\left(1+{\phi }^2\right)}{\left(s_{\mathrm{A}}+\phi s_{\mathrm{B}}\right)\left(\phi s_{\mathrm{A}}+s_{\mathrm{B}}\right)}$	$\phi$: environmental symmetry (0–1)
Polygenic architecture model	$P_{\mathrm{DM},\mathrm{P}}=1-\frac{\phi }{{\sigma }_1{\sigma }_2}\sum _{i=1}^n{s}_i^2$	$n$: number of interacting loci

Note: Dobzhansky‐Muller Incompatibilities (DMIs) accumulate between populations through asymmetric selection effects and genetic architecture. The framework extends classic models to incorporate environmental heterogeneity and polygenic adaptation, providing a quantitative foundation for understanding speciation.

The classic Unckless–Orr model (Equation 1), $\left(P_{\mathrm{DM},\mathrm{U}}\right)$ demonstrates that under identical environments, the probability of DMI formation reaches a maximum when selection coefficients are equal (Table 2). This upper bound reflects maximum stochasticity in adaptive trajectories: when mutations confer equal fitness benefits ($s_{\mathrm{A}}$ = $s_{\mathrm{B}}$), either allele is equally likely to fix first in separate populations, maximising the probability of incompatible combinations. Our extensions to this model allow us to generate quantitative expectations across untested parameter regimes, unify mutation‐order and ecological speciation as a single continuum and identify what features of the data can—or cannot—be explained by this theoretical extension. We hope that this work will not only illuminate our discoveries in Senecio but also stimulate novel interpretations in systems of speciation under variable selection asymmetry.

TABLE 2.

Unification of speciation models through a continuous framework.

Model component	Environmental symmetry $\left(\phi \right)$	Selection coefficients	Population structure	Reproductive isolation	Example systems
Parallel ecological speciation	$\phi =0$ between environments $\phi =1$ within environments	$s_{\mathrm{A}}\gg s_{\mathrm{B}}$ (highly asymmetric)	Discrete habitat transitions	Strong between environments; absent within environments	Threespine stickleback marine‐freshwater pairs (Rundle et al. 2000)
Mutation‐order speciation	$\phi =1$ (symmetric selection)	$S_{\mathrm{A}}\approx S_{\mathrm{B}}$ (similar magnitudes)	Independent populations in similar habitats	Present despite selection symmetry	Experimental Drosophila lineages (Hsu et al. 2024)
Environmental similarity model	Continuous: $0\le \phi \le 1$	Variable relationship between $s_{\mathrm{A}}$ and $s_{\mathrm{B}}$	Spatially structured populations across environmental mosaics	Varies with $\Phi$ and selection coefficient relationships	Senecio lautus complex

Note: Environmental symmetry $\left(\phi \right)$ and selection coefficients $\left(s\right)$ interact to produce diverse patterns of reproductive isolation across heterogeneous landscapes. Classic parallel ecological and mutation‐order speciation represent extreme cases of a gradient of environmental similarity, where environmental heterogeneity and genetic architecture jointly determine the probability of speciation. $\phi$ refers to how selection coefficients transfer between environments. For instance, high symmetry means an allele experiences similar selection pressure in both populations. Also see Table 1.

3.3.1. Environmental Symmetry Determines the Probability of Incompatibility Formation

We extended the Unckless–Orr model by incorporating environmental symmetry $\left(\phi \right)$, to reveal how environmental heterogeneity systematically modifies DMI formation probability through its effects on relative selection pressures (Figure 6A). This environmental similarity parameter enables us to model speciation scenarios along a continuum from identical to contrasting environments. Recall that, because $\phi =s_{\text{foreign}}/s_{\text{home}}$ and waiting times to fixation ∝ 1/s, lower $\phi$ both signifies stronger divergence in selection and predicts a faster build‐up of DMIs in the foreign lineage. Our extended model (Equation 2, $P_{\mathrm{DM},\mathrm{E}}$) reveals three key insights: First, when environments are identical between populations $\left(\phi =1\right)$, the model ($P_{\mathrm{DM},\mathrm{E}}$) reduces to the original Unckless–Orr formulation ($P_{\mathrm{DM},\mathrm{U}}$), recovering classic predictions about the relationship between selection coefficients and reproductive isolation. This is akin to the Senecio Dune populations, where populations remain reproductively compatible due to nearly identical selective pressures (Table 2). Second, as environments become increasingly different between populations $\left(\phi \to 0\right)$, DMI probability approaches unity regardless of selection coefficient ratios, demonstrating how environmental differences can override stochastic selection effects, as seen between the Dune and Headland ecotypes. Third, for intermediate symmetry $\left(0<\phi <1\right)$, DMI probability increases monotonically with decreasing $\phi$ while becoming progressively less dependent on the ratio $s_{\mathrm{B}}$/$s_{\mathrm{A}}$. This is similar to the Headland populations, which experience some degree of environmental heterogeneity between localities (Table 2).

FIGURE 6.

Environmental and polygenic effects on Dobzhansky‐Muller incompatibility (DMI) formation probability. (A) Environmental effects model ($P_{\mathrm{DM},\mathrm{E}}$), showing how the probability of incompatibility formation varies with environmental similarity $\phi$ under different selection ratios S _A/ S _B. Lower $\phi$ (greater environmental difference) increases DMI probability, particularly when selection is more asymmetric (larger S _A/ S _B). In the boundary case $\phi =1$ and $s_{\mathrm{A}}=s_{\mathrm{B}}$, this single‐locus model reduces to the Unckless–Orr result of $P_{\mathrm{DM},\mathrm{U}}=0.5$ (purple line). (B) Polygenic architecture model ($P_{\mathrm{DM},\mathrm{P}}$), illustrating how the probability of DMI increases with the number of contributing loci $n$. Genetic complexity amplifies the impact of selection on reproductive isolation, especially under strongly asymmetric selection regimes.

When $\phi$ ≠ 1, our model predicts that stochastic fixation of different alleles can also lead to DMIs, even under partial environmental overlap. As φ quantifies the ratio of foreign to home selection, a value of φ = 0.4 implies alleles retain 40% of their selective advantage when moved between habitats. Since the rate of adaptive substitutions is inversely proportional to selection strength (1/s), populations in the foreign environment will accumulate adaptive substitutions at approximately 40% of the rate of populations in the home environment. In this way φ captures both the strength of divergent selection and, as a corollary, its relative contribution to reproductive isolation. In contexts of parallel phenotypic evolution, our model predicts parallel molecular evolution at parallel trait loci when selection coefficients are highly asymmetric $\left(s_A\gg s_B\right)$ and environmental symmetry is minimal $\left(\phi \to 1\right)$, a pattern observed in sticklebacks (Jones et al. 2012) and mimetic butterflies (Van Belleghem et al. 2017).

3.3.2. Complex Genetic Architectures Enhance the Probability of Reproductive Isolation

Our extension of the model to polygenic traits (Equation 3, $P_{\mathrm{DM},\mathrm{P}}$) reveals that genetic complexity amplifies the probability of reproductive isolation (Figure 6B). By accounting for combinatorial growth in potential interactions with increasing numbers of loci, our framework shows that polygenic adaptation amplifies the effects of selection asymmetry, particularly when multiple loci with varying selection coefficients contribute to adaptation. This finding explains why polygenic traits readily accumulate DMIs despite weak selection asymmetry. These results align with research showing how pleiotropic mutations, which affect multiple traits, are likely to generate incompatibilities under conditions of rapid adaptation (Yamaguchi and Otto 2020), and with findings that show complex trait architectures modulate hybrid incompatibilities through the distribution of adaptive alleles across multi‐trait space (Schneemann et al. 2020). The Senecio system exemplifies these findings: Headland populations achieve phenotypic convergence through distinct combinations of alleles at gravitropism and architectural loci, facilitated by conditional neutrality and asymmetric selection ($\phi <1$; James, Allsopp, et al. 2023; Kaur 2024).

Our mathematical framework provides insights into how populations move along the adaptive landscape. When $\phi$ approaches 1, populations traverse similar adaptive landscapes, minimising opportunities for incompatibility evolution. As $\phi$ decreases, increasing landscape complexity creates multiple possible adaptive trajectories, facilitating the accumulation of reproductive barriers even between phenotypically convergent populations or arriving to different adaptive peaks. These theoretical predictions align with empirical observations in several systems, including Anolis lizards (Mahler et al. 2013), where fine‐scale habitat differences create multiple adaptive solutions despite broad‐scale ecological similarities, or in bacteria (Nahum et al. 2015) where environmental heterogeneity can drive populations toward distinct adaptive peaks despite experiencing similar selective pressures. However, as with any proposed mechanistic model, loci contributing to adaptation will not contribute to intrinsic reproductive isolation unless they directly form or become integrated into DMI systems that require conditional neutrality to create a genetic correlation between adaptation and speciation. See Material 3: Data S1 for a mechanistic model that connects patterns of reproductive compatibility to parallel phenotypic evolution in Senecio.

Overall, our mathematical framework demonstrates how uniform, similar and divergent selection can simultaneously shape patterns of reproductive isolation during parallel speciation. The framework unifies ecological and mutation‐order mechanisms along a continuum of selective pressures through a single parameter $\left(\phi \right)$ while revealing how genetic architecture modulates the translation of selection asymmetry into reproductive barriers. However, it is important to acknowledge several simplifying assumptions in our model. Following Unckless and Orr (2009), we assume haploid populations, which begin without standing genetic variation, with adaptations arising through new mutations. This simplification may not fully capture the complexity of diploid systems like Senecio, where standing genetic variation likely contributes to adaptation. In particular, the probability of populations sharing standing genetic variants may decrease with geographic distance, potentially contributing to the reproductive isolation between northern and southern Headland populations. Our model also makes simplifying assumptions about mutation rates and population sizes and treats adaptive alleles as having consistent, context‐independent effects rather than accounting for epistatic interactions or pleiotropy that likely characterise complex adaptive traits in Senecio. Despite these limitations, our results provide a quantitative foundation for understanding the emergence of biodiversity across heterogeneous landscapes in systems like Senecio, where environmental variation creates complex mosaics of selection pressures. Future research should extend the model to include additional complexities, such as standing genetic variation, recombination dynamics, diploid populations, dominance effects and temporal fluctuations in selection pressures.

3.4. Synthesis and Broader Implications

3.4.1. Mechanisms Generating Genetic Incompatibilities During Parallel Speciation

Previous work in Senecio has found that hormone signalling pathways, particularly auxin regulation, appear to coordinate adaptive traits (gravitropism, architecture; Wilkinson et al. 2021; James, Allsopp, et al. 2023; Broad et al. 2024) and reproductive processes (fertilisation success; Wilkinson et al. 2021). Because hormone pathways involve multiple interacting components (McGlothlin and Ketterson 2008) with potentially similar selection coefficients, they provide opportunities for selection to generate different genetic solutions to the same challenges, as seen in other systems of parallel evolution (e.g., Rokas and Carroll 2008; Reid et al. 2016; Birkeland et al. 2020). This integration of deterministic and stochastic processes might explain how Senecio populations achieve similar phenotypes through different genetic architectures while accumulating reproductive barriers. These empirical findings suggest a broader mechanism where genetic incompatibilities arise through the divergence of pleiotropic developmental pathways that affect both adaptive traits and reproductive compatibility. Such patterns align with predictions from our theoretical framework, where populations experiencing similar selection coefficients but with complex genetic architectures can evolve different solutions to similar adaptive challenges, ultimately generating reproductive isolation despite phenotypic convergence.

Reproductive isolation can also accumulate between populations via processes traditionally considered neutral (True and Haag 2001; Gavrilets 2004; Palmer and Feldman 2009). For instance, system drift is a form of genetic drift where populations traverse a “fitness ridge” of functionally equivalent genetic networks. As populations move along this ridge via stochastic processes, they may develop genetic incompatibilities while maintaining their phenotype and fitness (True and Haag 2001; Schiffman and Ralph 2021; James et al. 2022). However, genetic variation is unlikely to be perfectly smooth along such ridges—drift likely perturbs populations from optimal states, with selection bringing them back to different ridge positions, making such scenarios a product of both selection and drift. Our data provide limited support for a major role of system drift Senecio: reproductive isolation shows consistent ecotype dependence rather than random accumulation, effective population sizes remain large across ecotypes (James, Arenas‐Castro, et al. 2021), and Headland populations exhibit clear signatures of local adaptation rather than drift. Furthermore, most candidate speciation genes in plants (Rieseberg and Blackman 2010) reflect deterministic evolution due to selection, though specific cases, such as certain S‐RNase self‐incompatibility genes (Murfett et al. 1996; Hancock et al. 2003) and cytoplasmic male sterility loci (Bentolila et al. 2002; Wang et al. 2006), could represent drift‐driven incompatibilities, particularly in populations with small effective sizes. Even so, these examples are exceptions rather than the rule, suggesting that drift plays a limited role in the evolution of reproductive isolation in plant systems.

Genetic conflict –a form of mutation‐order speciation– and sexual selection are key mechanisms known to generate intrinsic incompatibilities in plants (Moore and Pannell 2011; Crespi and Nosil 2013; Tonnabel et al. 2021; Coughlan 2023). This can arise through divergent evolution of mating systems (e.g., Martin and Willis 2007), parent‐of‐origin effects on hybrid fitness (e.g., Coughlan et al. 2020) or antagonistic coevolution between male and female reproductive traits (e.g., Arnqvist et al. 2000). Recent work on conspecific pollen precedence in Senecio revealed that local adaptation influences competitive gametic interactions controlled by females (Arenas‐Castro 2022), suggesting sexual selection might contribute to reproductive isolation. However, we found no significant asymmetries in hybrid viability or fertility that would indicate strong parent‐of‐origin effects. The absence of pronounced reciprocal cross differences suggests that while sexual selection may fine‐tune reproductive barriers, it is unlikely to be the primary driver of isolation in this system.

3.4.2. Release From Trade‐Offs and the Evolution of Reproductive Isolation

Trade‐offs arise when a set of alleles or traits that confer advantages in one environment impose costs in another (Stearns 1989). In the extended Unckless–Orr model, the parameter $\phi$ captures how colonisation of new habitats can relax these constraints. When $\phi$ is significantly less than 1, ancestral constraints no longer fully operate, enabling novel sets of mutations to fix without previous fitness penalties. This release from ancestral trade‐offs expands the genetic “toolbox” available for adaptation, allowing different populations to fix different beneficial alleles for seemingly identical ecological contexts, a phenomenon seen particularly in Headland environments of Senecio. Despite convergent prostrate morphologies, Headland populations have accumulated genetic differences that reduce hybrid fitness. These patterns arise when new habitats only partially resemble ancestral ones (intermediate $\phi$ values), providing sufficient shared selective pressures for similar phenotypes to evolve while simultaneously allowing distinct adaptive trajectories, ultimately yielding a mosaic of reproductive barriers across the range. In this way, the relaxation of trade‐offs not only promotes phenotypic innovation but can also generate the conditions necessary for the formation of DMIs under different modes of divergence by natural selection.

3.4.3. Effect of Gene Flow on the Likelihood of Parallel Speciation

Population connectivity further complicates the dynamics of speciation because gene flow between populations homogenises allele frequencies and reduces the likelihood of independent fixation of alternative beneficial alleles (Bank et al. 2012). For example, in populations adapted to similar environments, mutation‐order speciation is challenging under moderate gene flow (Schluter 2009; Nosil and Flaxman 2011). This is because advantageous mutations are likely to spread and fix in all populations, thereby preventing the accumulation of incompatibilities. Consequently, mutation‐order speciation typically requires very low or absent gene flow (Nosil and Flaxman 2011; Nosil 2012). In Senecio, the negligible gene flow in the system (James, Arenas‐Castro, et al. 2021) has created ideal conditions for intrinsic incompatibilities to accumulate among Headland populations.

During ecological speciation, where alternative alleles are favoured in different environments, there is an antagonism between gene flow and divergent natural selection (Felsenstein 1981). As with mutation‐order speciation, gene flow constrains the evolution of reproductive barriers during adaptation, yet speciation remains possible if mechanisms arise to resolve this conflict (also see Porter and Johnson 2002). If locally adaptive alleles reside in genomic regions of reduced recombination, such as chromosomal inversions, population divergence and speciation can proceed despite ongoing gene flow (Noor et al. 2001; Rieseberg 2001; Butlin 2005; Kirkpatrick and Barton 2006; Ortiz‐Barrientos et al. 2016). The role of chromosomal rearrangements in mitigating the antagonism between gene flow and selection has been demonstrated across numerous systems including sticklebacks (Samuk et al. 2017), Atlantic cod (Berg et al. 2017) Heliconius butterflies (Martin et al. 2019), Mimulus monkeyflowers (Lowry and Willis 2010), Littorina snails (Le Moan et al. 2024) and Drosophila (Poikela et al. 2024). In the Senecio system, the low levels of contemporary gene flow between the Dune and Headland ecotypes likely facilitated the evolution of incompatibilities, yet ancient gene flow may also have shaped their divergence (James, Arenas‐Castro, et al. 2021). Future work will explore the role of chromosomal inversions during parallel phenotypic evolution in Senecio, shedding light on their contributions to speciation dynamics.

3.4.4. Patterns of Intrinsic Reproductive Isolation Across Traits and Development

Reproductive isolation can manifest differently across traits and developmental stages due to varying selective pressures and genetic complexities (Cutter 2023). For instance, while natural selection may act divergently at the population level, individual traits may experience selection that ranges from uniform to highly divergent (Langerhans and Riesch 2013). Additionally, the type and strength of selection upon traits may change over the course of the development of the organism, such as when trade‐offs exist where traits favoured early in development are not beneficial in later life (Stearns 1989; Garland et al. 2022). Finally, shared genetic architectures among traits can impose pleiotropic constraints, shaping how reproductive barriers evolve and influencing the potential for independent adaptation of specific traits (Mauro and Ghalambor 2020; Zhang 2023).

In Senecio, we found that reproductive barriers manifest differently in seed set versus viability. Crosses between Headland populations from different clades show reduced F₁ seed set, supporting our theoretical predictions about DMI formation under similar selection pressures. However, hybrid viability does not follow this pattern, suggesting that different components of reproductive isolation may evolve through distinct genetic architectures. This asymmetry in reproductive barriers aligns with our mathematical framework: seed set may be governed by a small number of loci with large effect sizes, such as those involved in specific pollen‐pistil recognition and hormone signalling pathways, making parallel selection more likely to generate consistent reproductive barriers. In contrast, hybrid viability likely involves more loci with smaller effects across diffuse genetic networks, creating different outcomes across population pairs. These differences between seed set and viability warrant further exploration to determine whether different components of isolation can evolve through distinct mechanistic pathways despite similar selective pressures.

3.4.5. A Continuous Framework for Speciation

The speciation framework we propose positions ecological and mutation‐order speciation as endpoints of a continuous spectrum rather than discrete processes (Langerhans and Riesch 2013). Our approach complements other speciation perspectives such as the traditional “speciation continuum” model (Seehausen et al. 2014; Stankowski and Ravinet 2021) and the “speciation hypercube” framework (Bolnick et al. 2023; Johannesson et al. 2024) which emphasises the multidimensionality of speciation, integrating genetic, phenotypic, ecological and temporal factors. While pure mutation‐order speciation can theoretically occur under identical environments, it becomes increasingly improbable with increasing genetic complexity, which may explain why the most convincing cases of mutation‐order speciation primarily arise in experimental systems (Ono et al. 2017; Hsu et al. 2024). Understanding the relative contributions of mutation‐order versus ecological adaptation in natural systems requires identifying the specific loci underlying reproductive barriers and their link to local adaptation (Wu and Ting 2004; Lowry et al. 2008; Nosil and Schluter 2011). This molecular characterisation would help clarify the extent to which apparent cases of mutation‐order speciation in nature truly reflect stochastic processes versus environmental heterogeneity. We thus urge researchers to move beyond the categorical classification of ecotypes and incorporate within‐ecotype environmental variation to better understand the relative contributions of deterministic versus stochastic processes during the evolution of reproductive incompatibilities.

4. Conclusions

Our study demonstrates how multiple forms of selection jointly drive the evolution of reproductive isolation in Senecio lautus. By integrating field experiments and genetic analyses, we reveal that divergent selection between contrasting environments, uniform selection in identical dune environments and similar selection in headland environments together shape speciation dynamics. Our findings challenge the traditional dichotomy between ecological and mutation‐order speciation, highlighting their complementary roles in generating biodiversity. Our mathematical framework incorporating polygenic adaptation and environmental symmetry demonstrates that genetic complexity increases the probability of speciation, even without stark ecological contrasts, explaining how phenotypically convergent populations can obscure genetic incompatibilities. Future work integrating genomic and ecological data across will further illuminate how divergent and stochastic processes jointly drive the origins of species across heterogeneous environments.

Author Contributions

Conceptualization: Daniel Ortiz‐Barrientos. Formal Analysis: Maddie James, Maria Melo and Gregory Walter. Mathematical Model: Daniel Ortiz‐Barrientos and Jan Engelstädter. Experiments: Maria Melo, Maddie James, Federico Roda, Diana Bernal‐Franco, Melanie Wilkinson and Huanle Liu. Data Curation: Maddie James, Maria Melo. Visualisation: Maddie James, Gregory Walter and Daniel Ortiz‐Barrientos. Writing: Maddie James and Daniel Ortiz‐Barrientos, with contributions from Gregory Walter, Melanie Wilkinson, Maria Melo, Federico Roda, Diana Bernal‐Franco, Huanle Liu and Jan Engelstädter. Supervision: Daniel Ortiz‐Barrientos. Funding Acquisition: Daniel Ortiz‐Barrientos and Jan Engelstädter.

Conflicts of Interest

The authors declare no conflicts of interest.

Supporting information

Data S1: mec70090‐sup‐0001‐Supinfo.pdf.

Table S1: mec70090‐sup‐0002‐Tables.xlsx.

Data Availability Statement

Final data files, protocols and code are found on GitHub (https://github.com/MaddieEJames/James_et_al._2025_MolecularEcology). Sequence data have been uploaded to GenBank, accession numbers PX139708‐PX146707.