Local negative frequency dependence can decrease global coexistence in fragmented populations

Significance

As human activities increasingly fragment once-continuous habitats, it becomes ever more critical to understand how these changes impact diversity. Negative frequency-dependent (NFD) selection, a key mechanism promoting coexistence, is well known for preserving genetic and species diversity in well-mixed and spatially continuous populations. Using a theoretical model bridging population genetics and community ecology, we show that NFD selection can actually accelerate extinctions and erode biodiversity, if the population is fragmented. This counterintuitive result arises because habitat fragmentation disrupts the alignment between the local scale of ecological interactions and the broader landscape structure of the habitat. Our findings suggest that, in fragmented environments, biodiversity may be more easily preserved under neutral dynamics than under NFD selection.

Abstract

Most biological populations are rich in diversity, and negative frequency-dependent (NFD) selection is a well-known mechanism thought to underlie this stable coexistence of multiple variants. Recent studies confirm its widespread presence at local spatial scales, however it remains unclear whether these local-scale dynamics are sufficient to maintain diversity across larger, landscape-level scales. While prior theoretical work has found that local NFD selection can indeed promote global coexistence, these studies only analyzed contiguous landscapes. In contrast, many ecosystems are not contiguous, but rather spatially fragmented or exhibit spatial variation in the local carrying capacity. Using a population-genetic model based on the classic island framework, we show that, in fragmented populations, NFD selection can paradoxically reduce coexistence and shorten fixation times, relative to neutrality. Fragmentation also generates a nonmonotonic relationship between fixation time and population size, with diversity minimized at intermediate scales, contrasting with the classical species–area relationship. Extending the model to a multispecies context reveals that these effects also shape species-level diversity. This allows us to develop a test for the influence of NFD selection in fragmented systems, based on the predicted nonmonotonic species–area relationship. Applying this test to an avian diversity dataset from the Ryukyu Islands, we find a signature suggesting that NFD selection and habitat fragmentation may interact to suppress biodiversity in this system. Together, our findings suggest that fragmentation can undermine the stabilizing effects of NFD selection, calling into question its generality as a mechanism for maintaining biodiversity in heterogeneous landscapes.

The maintenance of diversity across natural systems has long been a central question in ecology and evolutionary biology. From genes to species to ecosystems, biological systems exhibit a striking degree of variation that persists despite the homogenizing forces of directional selection, competitive exclusion, genetic or ecological drift. Explaining the persistence of this variation has motivated extensive theoretical (1–3) and empirical research (4–6). While population geneticists have traditionally examined the maintenance of genetic diversity within species, and ecologists have emphasized species-level diversity, both fields have converged on similar conceptual frameworks (7).

Across levels of organization, from genotypes within populations to species within communities, stable coexistence often requires some form of direct or indirect negative frequency dependence (NFD): when rare types have higher growth rates than common types, extinction of alleles/species is unlikely to occur and genetic/species diversity is preserved (8). In population genetics, balancing selection represents a canonical example, encompassing mechanisms such as overdominance, temporally fluctuating selection, and storage effects (9–14). Despite their mechanistic diversity, most forms of balancing selection ultimately operate through direct or indirect negative frequency dependence, a general principle strongly supported by empirical evidence across taxa (15–17).

Ecological theory also emphasizes the role of NFD for maintaining diversity. For example, in Modern Coexistence Theory (MCT), coexistence is framed as the outcome of two opposing forces: niche differentiation, which promotes coexistence by reducing interspecific competition relative to intraspecific competition, and fitness differences, which diminish coexistence by promoting competitive exclusion (1, 2). Crucially, the stabilizing force of MCT, niche differentiation, maintains coexistence precisely because it generates negative frequency-dependent dynamics (7). Empirical applications across ecosystems consistently validate the explanatory power of MCT in accounting for observed patterns of species coexistence (18–20).

Separately, in the absence of any selective forces, neutral theory offers an alternative minimal model of diversity maintenance. In this framework, diverse communities emerge when random extinctions are balanced by the introduction of novel types through mutation or speciation (21, 22). Unlike the deterministic, infinite population size models frequently used to study NFD selection, in which coexistence can persist indefinitely, finite-population neutral models predict eventual fixation of a single variant. More generally, in finite-population models, fixation (or equivalently, persistence) time is a natural proxy for coexistence (23, 24). Comparative analyses demonstrate that coexistence times and other biodiversity measures are typically far greater under NFD selection than under neutrality (11, 25, 26).

These classical approaches, however, assume well-mixed populations, whereas in spatially structured systems, NFD selection acts at the scale of local interactions and global diversity does not necessarily emerge from local stabilizing dynamics (27). For example, a species might be rare, and gain a fitness advantage, within a local patch, even if it is globally abundant (28). Although local NFD selection has been documented across diverse ecosystems, including tropical and temperate forests, grasslands, coral reefs, microbial communities, and vertebrate populations (5, 29–35), such evidence alone does not establish that local NFD dynamics translate into the maintenance of global diversity. Bridging this gap requires theoretical models that explicitly connect local frequency-dependent processes to large-scale coexistence outcomes.

While, traditionally, theoretical frameworks used to study coexistence and NFD selection have been developed independently from the frameworks used to study the effects of spatial structure, such as metacommunity theory (22) or evolutionary graph theory (36–38), recent theoretical work has begun to address this gap by analyzing models incorporating contiguous, regular spatial landscapes. At a big picture level, these studies find that local NFD selection does, in fact, continue to maintain high levels of global coexistence (39), but also that even modest interspecific asymmetries can quickly overwhelm the stabilizing effects of frequency dependence and dramatically reduce coexistence. For example, discrete lattice and continuous planar Lotka–Volterra models have found that spatial structure shrinks the coexistence region of the parameter space, even when the classical condition, intraspecific competition larger than interspecific competition, holds (40, 41). Stochastic multispecies models in continuous planar (39) and discrete lattice-structured (42, 43) spatial representations have further shown that interspecific differences in the strength of NFD selection can sharply reduce biodiversity proxies like fixation times, species richness, and evenness. Similarly, lattice-based models have demonstrated that fitness asymmetries (44, 45) and differences in resource uptake rates (46) can also significantly diminish diversity relative to well-mixed environments.

Most prior work has focused on regular, contiguous landscapes, whereas many natural systems are fragmented or patchy due to spatial heterogeneity in local carrying capacity (47) or connectivity loss (48). Fragmentation is likely to further decouple local dynamics from global outcomes, potentially weakening the stabilizing effects of NFD. Although population genetic studies focused on times to fixation have examined fragmented systems (e.g., island or network-structured populations) (36–38, 49–51), they have primarily examined neutrality or constant selection, with little exploration of frequency dependence beyond cooperative games (52, 53). We lack a general theory for how NFD selection shapes patterns of coexistence in fragmented spatial systems and how properties of population spatial arrangement modulate the relative strength of NFD selection versus neutrality in maintenance of diversity.

Here, we address this gap by developing a stochastic population-genetic model of local frequency-dependent selection in an island-structured population (54, 55) and ask how fragmentation shapes coexistence under NFD compared to neutrality. To quantify coexistence, we first analyze a tractable two-variant model and study the average conditional fixation time (conditional on fixation) of a new variant invading a wild-type population, using simulations and analytic approximations. We find that in fragmented populations, for small enough migration rates, coexistence times can be shorter under NFD selection than under neutrality, in contrast to predictions of previous models. Moreover, fixation times exhibit a nonmonotonic dependence on deme size.

While our simple two-variant model facilitates analytic tractability, to generalize these findings, we extend our analysis to a multi-species model, using species richness as a proxy for coexistence. Our aim is to treat NFD in its most general form, so that our model captures a mechanism that can manifest at multiple biological scales. We find that NFD selection again leads to lower species richness than neutrality, consistent with analytical predictions from the two-variant case. This framework further allows us to propose an empirical test for identifying whether natural populations operate in a regime where NFD reduces biodiversity relative to neutral expectations, using only species presence–absence data. Applying this test to a dataset of avian species in the Ryukyu Islands (56), we find evidence that NFD selection, and by implication, niche effects, is operating and contributes to coexistence in this fragmented ecosystem.

Together, our results demonstrate that spatial fragmentation can fundamentally alter the role of negative frequency dependence in maintaining biodiversity. Under fragmentation, local stabilizing dynamics may reduce, rather than enhance, global diversity, in stark contrast to predictions from well-mixed or contiguous models. This finding suggests that the explanatory power of niche differentiation may break down in fragmented systems, helping to explain why functional traits, which are indicative of niche differentiation, often fail to predict coexistence in natural communities (57–59).

Results

A Model of Local Frequency-Dependent Dynamics in Fragmented Populations.

To understand the interaction between spatial structure and frequency-dependent selection, we first make use of an extension of Wright’s classic island model and consider a haploid population composed of two different types of individuals: wild-type and mutant variants. The population of size $N_T=\mathit{ND}$ is structured into $D$ demes of $N$ individuals each, with probability of migration $m$ between any two demes. This means that, at every generation, an offspring’s parents come from within the deme with probability $(1-m)$ and from the overall metapopulation with probability $m$. The case when migration $m=1$ represents a well-mixed metapopulation.

We model neutral and frequency-dependent dynamics by assuming that reproduction within a deme is biased by local frequency-dependent selection, such that the wild-type fitness is given by $\frac{1}{2}-s(x_i-\frac{1}{2})$ and the mutant fitness by $\frac{1}{2}+s(x_i-\frac{1}{2})$, where $x_i$ is the mutant frequency in deme $i$ and $s$ is a frequency-dependent selection coefficient (60). Notice that, if $s$ is not zero, this mapping leads to a mutant equilibrium frequency of exactly 0.5 in the well-mixed population regime (see InsertFig. 1B), although eventually drift will push the mutant to fixation or extinction. This flexible model, although simple, allows us to capture both structured ($0<m<1$) and well-mixed populations ($m=1$), as well as neutral ($s=0$), negative frequency-dependent ($s<0$) and positive frequency-dependent ($s>0$) selection (Fig. 1A).

Fig. 1.

(A) Illustration of the four regimes we compare using our model. (B) Ten mutant trajectories for each of the four regimes considered. The total population size here is $N_T=\mathit{ND}=2,000$ with $N=100$, $D=20$, and $m={10}^{-6}$ for the fragmented populations. Under neutrality, $s=0$ and under NFD, $s=-0.0625$. One representative trajectory for each case is shown in bold.

Let us consider the average mutant conditional fixation time (conditional on mutant fixation) as a proxy for coexistence in this model. In what follows, we will use “fixation time” to refer to this expected conditional fixation time, and we will explicitly specify when the unconditional fixation time is considered instead.

While neutral dynamics are often invoked as a highly effective minimal model to explain patterns of coexistence (21, 22), the amount of biodiversity observed under neutrality, in the presence of genetic drift or demographic stochasticity, is much smaller compared to what can be observed under classic mechanisms of coexistence that give rise to NFD selection, such as stabilizing niche differences. We visualize these phenomena in Fig. 1B, where we simulate trajectories of a novel mutant variant in the population, under neutral ($s=0$, $m=1$) and negative frequency-dependent selection ($s<0$, $m=1$) in a well-mixed population (gray and orange trajectories, respectively). While under neutrality the mutant fixes after $\sim$2$N_T$ ($\sim$10³) generations, there is still no fixation even after ${10}^7$ generations under negative frequency-dependent selection.

In fragmented populations, even under neutral dynamics, spatial structure can significantly prolong coexistence ($\sim \frac{D}{m}$ generations, see ref. 50) compared to the well-mixed model, by restricting gene or species flow (gray ($s=0$, $m=1$) and blue ($s=0$, $0<m<1$) trajectories in Fig. 1B) (50). However, when both mechanisms are present (that is, in fragmented populations with local negative frequency-dependent selection, $s<0,0<m<1$) we find, counterintuitively, that the mutant tends to fix faster ($\sim$10⁶ generations) than under either mechanism alone.

In Fig. 2, we study the amount of population fragmentation that can lead to this observed interaction effect. We first show the difference in fixation times between the frequency-dependent and the neutral regimes for a well-mixed population. As expected, increasing the strength of NFD (negative values of $s$) increases fixation times, while increasing the strength of PFD (positive values of $s$) has the opposite effect, leading to a decrease in the fixation times (Fig. 2A). As population fragmentation decreases, the result is maintained (Fig. 2B), however, for small enough migration rates $m$ (high enough levels of population fragmentation), the opposite result starts to be observed: negative frequency dependence decreases fixation times compared to neutrality, while positive frequency dependence increases fixation times and coexistence of the two variants in the population (Fig. 2C).

Fig. 2.

Fixation times and critical migration rates. We show the difference in conditional fixation times relative to neutrality as a function of $s$ in a well-mixed population in (A), and spatially structured populations with $m=0.005$ and $m=0.00001$ in (B and C), respectively. Here, $N=20$ and $D=50$. Dots represent averages over ${10}^6$ simulation runs, and solid lines show analytic approximations Eq. 3. (D) shows the derivative of the conditional fixation time with respect to the frequency-dependent selection coefficient $s$, as a function of migration rate $m$. Total population size is fixed to $N_T=1,000$, with deme size $N$ and number of demes $D$ varied as in the legend. Dots represent averages over ${10}^6$ simulation runs, while the black line represents Eq. 4.

To quantitatively understand this result, we derive analytic approximations for the fixation time, under an assumption of weak selection ($|s|\ll 1$), weak migration ($m\ll 1$) (see more in SI Appendix, section 1). We first write analytic expressions for the mean per generation change in the mutant frequency, $\mathbb{E}(\mathrm{\Delta }x)$,

$$\begin{array}{c}\hfill \mathbb{E}(\mathrm{\Delta }x)\approx 8N^2{m}^{2}sx(1-x)(x-\frac{1}{2})\end{array}$$ [1]

and the variance in the mutant frequency change per generation, $\text{Var}(\mathrm{\Delta }x)$,

$$\begin{array}{c}\hfill \text{Var}(\mathrm{\Delta }x)\approx 2D^{-1}mx(1-x)(1-\frac{2}{3}\mathit{Ns}),\end{array}$$ [2]

(SI Appendix, section 1). For $s<0$ (the NFD regime), the sign of $\mathbb{E}(\mathrm{\Delta }x)$ is determined by the sign of $(x-\frac{1}{2})$ and increasing the strength of negative frequency dependence increases the magnitude of the selective force pushing the population toward the equilibrium frequency of $0.5$, which effectively acts to increase the fixation time. In contrast, the expression for $\text{Var}(\mathrm{\Delta }x)$ shows that increasing the strength of NFD increases the strength of random fluctuations in the mutant frequency, which in turn acts to decrease the fixation time. More specifically, the effect of NFD on $\text{Var}(\mathrm{\Delta }x)$ comes from the $(1-\frac{2}{3}\mathit{Ns})$ term, which becomes larger as $s$ becomes more negative.

Since NFD simultaneously increases fixation time through $\mathbb{E}(\mathrm{\Delta }x)$ and decreases fixation time through $\text{Var}(\mathrm{\Delta }x)$, the overall effect of NFD on the fixation time depends on the balance between these two effects. This balance is dictated by the strength of migration, $m$. Because $\mathbb{E}(\mathrm{\Delta }x)$ scales with $\mathcal{O}(m^2)$ and $\text{Var}(\mathrm{\Delta }x)$ scales with $\mathcal{O}(m)$, for large enough migration rate $m$, both the mean and variance in the per-generation mutant frequency change are of comparable magnitude. Under increasingly small migration however, $\mathbb{E}(\mathrm{\Delta }x)$ vanishes faster than $\text{Var}(\mathrm{\Delta }x)$ and, since $\mathbb{E}(\mathrm{\Delta }x)\to 0$, the population effectively evolves neutrally at a global spatial scale (although being under NFD over local spatial scales), while also experiencing stochastic fluctuations that are larger than what is observed under neutrality.

Using a diffusion approximation (50), we write the average fixation time of the mutant

$$\begin{array}{c}\hfill T\approx \frac{D}{m}+\frac{2}{3}\mathit{NDs}(\frac{1}{m}-\frac{1}{3}\mathit{ND})\end{array}$$ [3]

(SI Appendix, section 1) and show this approximation, together with simulation results, in Fig. 2 B and C. For migration rate $m=0.005$, although NFD still increases fixation time relative to neutrality, we find that this increase is linear compared to the superlinear increase observed in a well-mixed population (Fig. 2B). Hence, some degree of population subdivision acts to dampen the effects of NFD. However, in a regime of weaker migration ($m=0.00001$), we find that the relationship between $s$ and fixation time is reversed (Fig. 2C).

Note that the effect of frequency dependence on fixation time can essentially be captured by the sign of the derivative of the fixation time at neutrality, $s=0$, which can be written as

$$\begin{array}{c}\hfill \frac{\partial T}{\partial s}\approx \frac{2}{3}\mathit{ND}(\frac{1}{m}-\frac{1}{3}\mathit{ND}).\end{array}$$ [4]

The role of NFD is reversed ($\frac{\partial T}{\partial s}$ is positive) when $\mathit{NDm}\lesssim 1$. Therefore we can define a critical migration rate $m^{\ast }$ as

$$\begin{array}{c}\hfill m^{\ast }\approx \frac{1}{\mathit{ND}}=\frac{1}{N_T}.\end{array}$$ [5]

This expression shows that, independent of how the population is divided up into demes, for a given total population size $N_T=\mathit{ND}$, it is the migration rate that critically governs this transition. This analytic prediction is verified in Fig. 2D where we show the derivative of the fixation time at neutrality $\frac{\partial T}{\partial s}{|}_{s=0}$ for a populations with a fixed total population of $N_T=1,000$, but with different values for $N$ and $D$. The analytic approximation given by Eq. 4 (black solid line) overestimates the critical migration rate when the number of demes is very small ($D=2$), since the diffusion approximation we employ assumes a large enough number of demes ($D\gg 1$).

This phenomenon becomes more intuitive in the limit where migration is sufficiently rare so that individual demes reach fixation between migration events. In this regime, the fixation time of the whole metapopulation is largely determined by the number of invasion attempts required for migrants to take over individual demes. Under neutrality, this requires on the order of $N$ attempts, since the probability of fixation for a single migrant is $\frac{1}{N}$. By contrast, under NFD selection the number of required attempts is reduced ($<N$), because a migrant of the alternative type enjoys a fitness advantage when rare.

However, NFD selection can still reduce coexistence relative to neutrality even at higher migration rates, when demes are not close to fixation between migration events. To see this, recall the condition derived earlier, $\mathit{NDm}\lesssim 1$. Rearranging, we find that the fixation time is lower under NFD selection compared to neutrality when the expected number of migrants entering each deme per generation ($\mathit{Nm}$) is less than $D^{-1}$ ($\mathit{Nm}\lesssim \frac{1}{D}$). When a migrant enters a deme, successful fixation requires $\sim N$ generations, and therefore, our condition is satisfied as long as demes receive no more than $\sim \frac{N}{D}$ mutants in-between fixation events. Therefore, when $N\gtrsim D$, this upper bound ($\sim \frac{N}{D}$) can exceed one and it is possible for the phenomenon we describe to occur, even when demes are not at fixation in-between migration events.

Although NFD selection can decrease conditional fixation time in fragmented populations, it is important to note that it can simultaneously promote the establishment of rare mutants by increasing their probability of fixation (see, for example, the probability of fixation derived in SI Appendix, Eq. S15). To account for this trade-off, we derive an analytic approximation for the unconditional fixation time in SI Appendix, section 2. By analyzing the derivative of the unconditional fixation time, $T_{\mathit{uc}}$, at neutrality (SI Appendix, Fig. S1), we find that the unconditional fixation time can also be lower under NFD selection, compared to neutrality. The critical migration rate for this effect ($m^{\ast }\approx \frac{ln(D)}{N^{2}D}$) is smaller by a factor of $\frac{ln(D)}{N}$ (SI Appendix, section 2). This occurs because NFD increases fixation probability, while trajectories that do fix tend to take longer, thereby elevating the unconditional fixation time.

Finally, we show that these results extend beyond fully symmetric populations. Even when only a subset of demes are connected, our analytic predictions remain valid, and simulations confirm the robustness of our findings for less regular deme topologies (SI Appendix, Fig. S2).

Nonmonotonic Dependence of Fixation Time on Deme Size.

Another prediction that arises from our analytic results is a nonmonotonic relationship between the deme size $N$ and the fixation time $T$. Computing the derivative $\frac{\partial T}{\partial N}$,

$$\begin{array}{c}\hfill \frac{\partial T}{\partial N}\approx \frac{2}{3}\frac{D}{m}s-\frac{4}{9}{\mathit{ND}}^{2}s,\end{array}$$ [6]

we find that the fixation time is minimized at intermediate deme size $N^{\ast }\approx \frac{3}{2\mathit{Dm}}$.

This expression is similar to that of $m^{\ast }$ derived earlier, which means that the nonmonotonic pattern occurs in the same regime in which the fixation time is smaller under NFD than neutrality. We show this theoretical prediction of fixation time as a function of deme size under neutrality and NFD in the fragmented (red and blue solid lines, respectively) and the well-mixed limit (gray and orange solid lines respectively) in the Insert in Fig. 3. Specially, the fixation time under NFD is minimized when $N^{\ast }Dm\approx \frac{3}{2}$ and intersects with the fixation time under neutrality when $N^{\ast }Dm\approx 3$. Beyond this intersection point, NFD promotes coexistence relative to neutrality. Note that under neutrality, theory predicts that the fixation time does not depend on deme size, since migration between demes occurs much slower than fixation within demes. Results from simulations are shown in Fig. 3 and although the theoretical approximations underestimate the fixation time (with increasing $N$ there are increasingly more mutants entering each deme and increasingly stronger effective selection $\mathit{Ns}$), the prediction of a nonmonotonic pattern in the fixation time under NFD remains.

Fig. 3.

Nonmonotonic dependence of conditional fixation time on deme size. Dots represent conditional fixation times estimated from ${10}^6$ simulation runs, as a function of deme size. We use number of demes $D=50$ and migration rate $m=0.0001$. Blue points show conditional fixation times under neutrality ($s=0$) and red points under NFD ($s=-0.001$), in the fragmented regime where $\mathit{NDm}\lesssim 1$. Gray dots show conditional fixation times under neutrality ($s=0$) and orange points under NFD ($s=-0.001$) in the well-mixed regime where $\mathit{NDm}\gtrsim 1$. Solid lines show simulation results fitted to a quadratic function. The Insert shows the corresponding analytic approximations from Eq. 3, exhibiting similar quantitative behaviors.

This nonmonotonic pattern can be understood by again considering the regime where demes are at fixation between migration events. Recall that successful fixation of a neutral migrant (within a deme) occurs with probability $\frac{1}{N}$, and since each deme receives a migrant with probability $\sim \mathit{Nm}$ per generation, the overall rate of successful neutral invasions is $\sim \frac{1}{N}·\mathit{Nm}=m$, per deme, per generation. Under frequency dependence, migrants fix with probability $\sim (\frac{1}{N}-\mathcal{O}(s))$ (see the within-deme probability of fixation SI Appendix, Eq. S28), and the rate of successful invasions becomes $\sim m(1-\mathcal{O}(\mathit{Ns}))$. Therefore under NFD selection ($s<0$), initial increases in deme size $N$ will increase the rate of invasions and accelerate the fixation of the overall population. However, for sufficiently large N, demes will no longer fix between invasions, and increases in deme size will instead translate into larger fixation times at the global scale. Therefore global fixation times are minimized for intermediate deme sizes because, at the local scale, they strike an optimal balance between large fixation probability and small fixation time (within the demes).

Multispecies Coexistence.

Thus far we have considered a two type model and studied average conditional mutant fixation times as a proxy for their coexistence. While this model captures the essence of the interplay between fragmentation and frequency-dependent selection and allows for analytical treatment, it cannot capture other complex dynamics that can occur in fragmented landscapes. A landscape can have demes of different sizes, more complex patterns of connectivity and migration, or multiple species competing in the population.

In this section, to verify that the results from our simple two-type island model can be extrapolated to more complex ecosystems, we consider a multi-species extension of the model, with demes that can vary in size. We use species richness as the proxy for coexistence, with each species assumed to experience negative frequency-dependent selection as a result of ecological niche differentiation. Since we assume no speciation or migration of new species from an external pool, we analyze the decay of species richness over time, due to loss from random drift.

To construct a metapopulation with variability in deme size, we utilize a dataset of islands that form the Ryukyu islands ecosystem (56). The Ryukyu island system consists of $68$ individual islands with areas ranging from $0.041$ to $1,183$ square kilometers. We assume that the area of an island is directly proportional to its population size $N$ and build an island model in which each deme $i$ has size $N_i=100A_i$, where $A_i$ is the area of island $i$ in square kilometers.

In line with expectation, in the “high migration regime,” with $N_{T}m\approx 4.5>1$, we find that the species richness decays faster over time under neutrality compared to NFD selection (gray and orange lines in Fig. 4A respectively). Similarly, the distribution of species richnesses after $1e6$ generations has a lower mean under neutrality compared to NFD selection (Fig. 4B). In contrast, in the “low migration” regime ($N_{T}m\approx 0.23<1$), the species richness decays faster under NFD compared to neutrality (red and blue lines in Fig. 4C respectively, and corresponding richness distributions in Fig. 4C). Hence, the results from our simpler model, and the associated theory, can be extrapolated to more complex ecosystems and different proxies of coexistence.

Fig. 4.

Multispecies model. Each simulation is initialized with a unique species in each deme, such that the initial species richness is 68. (A) shows 100 independent trajectories of the decay in species richness in the high migration regime where $m=1e-5,\mathit{NDm}\approx 4.5$. Trajectories under neutrality ($s=0$) are shown in gray and those under NFD ($s=-0.01$) in orange. A representative trajectory under both conditions is shown in bold. (B) shows histograms of species richness outcomes after $1e6$ generations. (C) similarly shows species richness trajectories in the low migration regime where $m=5e-7,\mathit{NDm}\approx 0.23$, and (D) the associated species richness outcomes after 1e6 generations. Total population size is fixed to $N_T\approx \text{5e5}$.

Thus far, we have not considered mutation/speciation or immigration from an external population/species pool. This is important, since our earlier results show that NFD selection also promotes the establishment of novel, rare mutants. To capture this, we extend our model by allowing a new species to enter the population each generation with probability $\mu$, creating an equilibrium species richness determined by a mutation/speciation-extinction balance. Even under this extended model, we find that coexistence in fragmented populations remains lower under NFD selection than under neutrality, across a broad range of mutation rates (SI Appendix, Fig. S3).

Observing a Signature of Negative Frequency Dependence in Data.

We have shown that coexistence in a population increases monotonically with average deme size in three of the studied scenarios: 1) neutrality and high migration, 2) neutrality and low migration, and 3) NFD and high migration. On the other hand, in the case of negative frequency-dependent dynamics, coexistence is minimized in fragmented metapopulations with intermediate deme sizes (Fig. 3). This nonlinear relationship between conditional fixation time and deme size points to a possible test for inferring when coexistence in a metapopulation is maintained by the presence of local negative frequency dependence, over the alternative hypothesis of neutral dynamics.

Using a presence–absence dataset of $99$ avian species across the Ryukyu set of islands (data from ref. 56, visualization in Fig. 5), we provide evidence suggesting that avian biodiversity in this fragmented ecosystem is maintained by NFD at a lower level than would be expected under neutrality. Let us first notice that, plotting the species richness values among individual islands, we observe a linear trend with species richness increasing with island area (Fig. 5).

Fig. 5.

Visualization of data from ref. 56. Presence–absence dataset of $99$ avian species across the Ryukyu set of islands.

Since in this metapopulation every island (deme) has a different land mass (and population size), direct application of our previous result is not feasible. To understand the relationship between species richness and deme size, let us therefore consider the total species richness in subsets of demes of similar size that belong to the larger metapopulation. Here, island subsets effectively represent fragmented populations coupled to an external population via immigration. In order to sample island subsets, we first rank islands by size and then select subsets using a sliding window of size $20$. This way of constructing island subsets ensures that we have the broadest range of average island sizes possible while minimizing variation of island sizes within subsets. We demonstrate that our results are robust to the selected window size of 20 islands in SI Appendix, Fig. S4.

We find that the total predicted species richness after $1e6$ generations, as a function of average deme size, under 1) neutrality and high migration, 2) neutrality and low migration, and 3) NFD and high migration (Fig. 6A–C respectively) does indeed increases with deme size, as predicted by our initial simpler model. Here, the solid lines show averages over $100$ independent runs of the model, with the shaded areas representing $0.99$ CIs. However, for the case of NFD in the fragmented population regime, we again observe the nonmonotonic relationship between average deme size and species richness (Fig. 6D). The black dotted lines across the panels indicate species richness in the island subset having the smallest average size in order to highlight that biodiversity initially decreases with increasing average island size only in the regime presented in Fig. 6D. This theoretical result points to the fact that such a nonlinear pattern could be a signature of a population that is evolving under negative frequency-dependent dynamics, given a fragmented population, in the low migration regime. It is also important to notice how, while, in the fragmented regime, population-wide species richness is higher under NFD selection compared to neutrality, this, again, does not hold at the island level, and does also not necessarily hold at the population subset level (Fig. 6D).

Fig. 6.

A test for detecting NFD dynamics in fragmented data. In (A–D), we show the total species richness using metapopulation deme subsets of size 20 after $1e6$ generations, as a function of average deme size under 1) $s=0$ and $m=5e-7$, 2) $s=0$ and $m=1e-5$, 3) $s=-0.01$ and $m=1e-5$, and 4) $s=-0.01$ and $m=5e-7$ respectively. The solid lines show the average over 100 independent runs of the model and the shaded areas represent $0.99$ CIs. In (E), we show the empirical avian species richness values as a function of average island size using ranked island subsets of size 20. In order to sample fragmented island subsets, we first rank all islands by size and then select subsets of islands of similar size, using a sliding window of size 20 (that is, each fragmented population has exactly 20 islands), and plot the average size in this fragmented island system on the $x$-axis.

Finally, we perform the same analysis using the true species richness values recorded in the dataset of avian species across the Ryukyu islands (56) and observe a similar nonmonotonic pattern in species richness as a function of average island area (Fig. 6E). This suggests that local negative frequency-dependent dynamics could be decreasing the species richness of the avian population in these populations, relative to neutrality, since it is only in this regime that such a nonmonotonic pattern in species richness is expected. We demonstrate that our results are robust to the selected window size of 20 islands in SI Appendix, Fig. S5. The intuition for this nonmonotonic pattern follows from the argument made in Nonmonotonic Dependence of Fixation Time on Deme Size. Specifically, intermediate size demes are more susceptible to successful invasion by migrants, compared to small demes, while still reaching fixation between migration events. Migrant fixation tends to homogenize subsets of demes when the population as a whole is dominated by relatively few species, because individual demes become more likely to independently fix migrants of the same species. Therefore this increases the overlap in species compositions between these islands. We also note that the linear trend observed in Fig. 5 highlights that the nonlinear pattern in species richness is not driven by the individual island level. This makes it less likely that the small islands have greater species richness than medium-sized islands due to their proximity to much larger islands.

Discussion

To understand how spatial fragmentation modulates the effects of frequency-dependent selection, we introduce a model of population dynamics driven by direct frequency-dependent selection in fragmented populations. Unlike earlier models of frequency-dependent selection in spatially structured systems (61), our framework explicitly incorporates drift and demographic stochasticity. In this setting, coexistence is not a binary outcome but is quantified through the average fixation time of a novel type invading a wild-type population (40, 41). Using population-genetic methods to derive a diffusion approximation for fixation time, we find that under sufficient population fragmentation, local NFD can paradoxically reduce global coexistence relative to neutrality, contrasting with results in well-mixed systems, where NFD typically promotes diversity.

Although our analysis is rooted in population genetics, our goal was to treat NFD in its most general form so that the framework highlights commonalities across biological scales. Extending the model to a multispecies regime, we find consistent results when coexistence is measured via species richness. In particular, a metapopulation model based on 68 islands from the Ryukyu archipelago shows that species richness under low migration is lower with NFD selection than under neutrality, mirroring the two-type case. This allows us to find an empirical signature of this regime by extending the classic species–area relationship (62). While species richness usually increases monotonically with island area in contiguous populations, our model predicts that under NFD with limited migration, richness follows a nonmonotonic pattern, minimized at intermediate total area, in fragmented ones. Presence–absence data for 99 avian species across the Ryukyu Islands (56) reveal precisely this pattern, consistent with our theoretical predictions.

Our results contribute to a growing body of work revisiting the conditions under which NFD selection promotes coexistence. While prior studies have shown that partial niche differentiation can yield lower coexistence than neutrality (63), our model goes further: even a fully niche-differentiated community (where each species’ fitness is dependent on its abundance and occupies its own niche) can be outperformed by a neutral system in maintaining diversity, once spatial fragmentation is introduced. Separately, unlike earlier findings in contiguous systems, where coexistence loss arises from interspecific differences in fitness or selection strength (39–46, 64), here, biodiversity loss emerges solely from the interaction of NFD selection with spatial structure, with no variation in selection coefficients ($s$ is fixed for all types). This theoretical result resonates with empirical findings showing that niche-based interactions often fail to predict species coexistence in natural systems (65).

This work also aligns with a broader set of ecological and evolutionary models that demonstrate how spatial structure can invert expected outcomes. For instance, positive frequency-dependent selection, typically destabilizing in well-mixed populations, has been shown to prolong coexistence relative to neutrality in lattice structured populations (66). This occurs because strong positive frequency dependence generates spatial banding patterns in which species cluster together, thereby decreasing interspecific competition. In follow-up work, it was shown that spatial patchiness can further elevate coexistence under positive frequency dependence (60). Other processes such as sexual selection (47), Allee effects (67), heterozygote disadvantage (68), and interspecific competition (69) have similarly shown spatial reversals of their classical effects. Although not the primary focus of this study, our model recapitulates this result. Specifically, we observe that for $s>0$, when frequency dependence is positive, conditional fixation times can exceed neutrality, under low migration. This happens because positive frequency dependence decreases the probability of invasion of migrants within individual demes, thereby slowing down the population-wide fixation process in the weak migration regime.

Previous work has also shown that local NFD selection can produce effectively neutral global dynamics in spatially explicit Lotka–Volterra models (40, 64, 70). Our model reproduces this outcome: as shown in (Eq. 1), the mean per-generation change in mutant fitness analytically approaches neutrality in the limit of weak migration. However, we go further in showing that coexistence can actually fall below neutrality: as (Eq. 2) demonstrates, NFD amplifies fluctuations beyond neutral expectations, shortening coexistence despite appearing neutral on average.

These results also offer insights into the long-standing neutral-versus-niche debate. Specifically, while neutral theory offers a minimal mechanistic model that accurately reproduces key patterns of biodiversity, such as species abundance distributions, it has been criticized for being overly simplistic, since it assumes no niche-based interactions between individuals (71), despite evidence that such interactions are widespread (72). Our analysis reveals a possible explanation: in fragmented systems, populations subject to NFD selection can appear effectively neutral at the global scale. Neutral theory has also been criticized due to its inability to reproduce certain other key characteristics of natural populations. For instance, neutral theory significantly overestimates fixation times and underestimates the magnitude of abundance fluctuations (73). Our model of fragmented populations also offers a solution to these limitations, since it predicts that fixation times may be shorter and the variance in per-generation change in the mutant frequency can be larger under NFD selection, compared to neutrality.

Separately, prior work has uncovered another counterintuitive effect of population subdivision interacting with balancing selection (including direct NFD selection): coexistence can increase as migration rates increase. By contrast, under neutrality coexistence always decreases with increasing migration. This phenomenon was first described in diploid populations with gametophytic self-incompatibility, where the equilibrium number of alleles was minimized at intermediate migration rates (74). Subsequent work has generalized these findings to other forms of balancing selection (75) and provided analytic results (76, 77). More recently, this phenomenon has also been demonstrated in haploid subdivided populations under NFD selection (78). Together, this body of work shows that, under balancing selection, increasing migration can paradoxically enhance coexistence. In contrast, our results reveal a different and, to our knowledge, previously undescribed phenomenon: increasing the strength of NFD selection can paradoxically reduce coexistence in subdivided populations. Although our analysis focuses on haploid populations under direct NFD, we expect the effect to extend more broadly to other forms of balancing selection, such as heterozygote advantage in diploid systems, or systems where the storage effect is at play. Ultimately, all forms of balancing selection rely on some form of direct or indirect negative frequency dependence (8), and further studies are needed to examine whether the result qualitatively holds for other mechanisms.

Finally, our models are intentionally simple. This simplicity is both a strength, allowing us to demonstrate broad principles across evolutionary and ecological contexts, and a limitation, as many complexities are omitted. Considerations of diploidy, dominance, and recombination are central to population genetics, whereas niche differentiation is a core ecological concept. Shared processes such as, competition, mutation/migration, fitness variation, temporal fluctuations, can play different roles depending on the scale (7). In addition, our assumption of uniform migration rates and fully connected demes ignores the realistic heterogeneity of networks shaped by geography and matrix quality (79). We also assume no interspecific variation in selection strength ($s$) and no intrinsic fitness differences, both of which are known to impact coexistence outcomes (43). These extensions are important for a more complete theory of how NFD shapes coexistence across biological scales.

Model and Methods

Frequency-Dependent Dynamics in a Fragmented Population: Two-Variant Model.

To a model a spatially structured population of species competing under frequency-dependent selection, we consider an extension of Wright’s classic island model and assume a population of wild-type and mutant variants that are structured into $D$ demes of $N$ individuals each linked by migration occurring with probability $m$. This means that each generation the offspring’s parents come from within the deme with probability $(1-m)$ and from the overall population with probability $m$. The population composed of two different types of individuals: wild-type and mutant variants, with further mutation neglected.

Following ref. 60, we assume that reproduction within demes is biased by local frequency-dependent selection, such that the mutant fitness in deme $i$ is given by

$$\begin{array}{c}\hfill f_i=\frac{1}{2}+s(x_i-\frac{1}{2}),\end{array}$$ [7]

where $x_i$ is the mutant frequency in deme $i$ and $s$ is a frequency-dependent selection coefficient. The wild-type fitness in deme $i$ follows the same form, where we replace $x_i$ with $(1-x_i)$ and get $\frac{1}{2}-s(x_i-\frac{1}{2})$, which allows us to write the relative mutant fitness ${\widehat{f}}_i$ as

$$\begin{array}{c}\hfill {\widehat{f}}_i=\frac{\frac{1}{2}+s(x_i-\frac{1}{2})}{\frac{1}{2}-s(x_i-\frac{1}{2})}.\end{array}$$ [8]

For each deme, to create the next generation, individuals are sampled from within the deme with probability $(1-m)$ and from the entire population with probability $m$, followed by local frequency-dependent selection. Therefore, under neutrality, an individual in deme $i$ is of the mutant type with a probability given by

$$\begin{array}{c}\hfill \widetilde{p_i}=(1-m)x_i+m\overline{x},\end{array}$$ [9]

where $x_i$ is the mutant frequency in deme $i$ and $\overline{x}$ is the mean population-wide mutant frequency in the previous generation. When the sampling of alleles is further biased by local frequency-dependent selection according to Eq. 7, the mutant sampling probability becomes

$$\begin{array}{c}\hfill p_i=\frac{(1+{\widehat{s}}_i)\widetilde{p_i}}{1+{\widehat{s}}_i\widetilde{p_i}},\end{array}$$ [10]

where ${\widehat{s}}_i$ is the relative mutant selection coefficient (50). To determine ${\widehat{s}}_i$ for our model, using a Taylor expansion around $s=0$, we can write the relative fitness of the mutant allele in deme $i$ as

$$\begin{array}{c}\hfill {\widehat{f}}_i=\frac{\frac{1}{2}+s(\widetilde{p_i}-\frac{1}{2})}{\frac{1}{2}-s(\widetilde{p_i}-\frac{1}{2})}\approx 1+4s(\widetilde{p_i}-\frac{1}{2})\end{array}$$ [11]

and therefore ${\widehat{s}}_i\approx 4s(\widetilde{p_i}-\frac{1}{2})$.

Frequency-Dependent Dynamics in a Fragmented Population Based on the Ryukyu Island Ecosystem: Multispecies Model.

To verify that our results can be extrapolated to more complex ecosystems, we consider a multispecies extension of the two-variant island model. Specifically, we relax two assumptions: 1) there may be more than two species and 2) deme sizes may vary.

Let us assume that the fitness of species $\sigma$ in deme $i$ is given by

$$\begin{array}{c}\hfill f_{\sigma ,i}=\frac{1}{2}+s(x_{\sigma ,i}-\frac{1}{2}),\end{array}$$ [12]

where $x_{\sigma ,i}$ is the frequency of the species in deme $i$, similar to Eq. 7. Under this fitness function, the equilibrium state of a well-mixed population still occurs when all the species have equal frequencies because this is the condition under which they all attain the same value of $f_{\sigma ,i}$. Reproduction and migration are implemented in the same way as the two-variant model above. In this multispecies model, we focus on species richness as the main proxy for coexistence. Since we assume no speciation events that can create new species or migration of new species from an external pool, the species richness will only decay over time, because of loss of species from random drift. We initialize the population with a unique species in each island and analyze this decay of species richness over time.

In order to understand the maintenance of biodiversity in an island ecosystem with varying island sizes, we model the metapopulation after the structure of the Ryukyu island ecosystem. The Ryukyu islands are made up of $68$ individual islands that vary in area ranging from $0.041$ square kilometers to $1,183$ square kilometers. We make the assumption that the area of an island is directly proportional to the population size on that island, $N$. Specifically, we represent the Ryukyu island population using an island model in which each deme $i$ has size $N_i=100·A_i$, where $A_i$ is the area of island $i$ in square kilometers. We select a scaling factor of $100$ for $N_i$, because this is the minimum value at which all islands have a size that is greater than $1$.

Supplementary Material

Appendix 01 (PDF)

Data, Materials, and Software Availability

Custom scripts were used for the simulations and data analyses. All C++ and Python simulation code is available on Github at https://github.com/anushdevv/NFD_fragmented (80). All packages used for analysis and visualization are open-source. All other data are included in the manuscript and/or SI Appendix.