A spatial approach to jointly estimate Wright's neighborhood size and long-term effective population size

Abstract

Spatially continuous patterns of genetic differentiation, which are common in nature, are often poorly described by existing population genetic theory or methods that assume either panmixia or discrete, clearly definable populations. There is therefore a need for statistical approaches in population genetics that can accommodate continuous geographic structure, and that ideally use georeferenced individuals as the unit of analysis, rather than populations or subpopulations. In addition, researchers are often interested in describing the diversity of a population distributed continuously in space; this diversity is intimately linked to both the dispersal potential and the population density of the organism. A statistical model that leverages information from patterns of isolation by distance to jointly infer parameters that control local demography (such as Wright's neighborhood size), and the long-term effective size (N_e) of a population would be useful. Here, we introduce such a model that uses individual-level pairwise genetic and geographic distances to infer Wright's neighborhood size and long-term N_e. We demonstrate the utility of our model by applying it to complex, forward-time demographic simulations as well as an empirical dataset of the two-form bumblebee (Bombus bifarius). The model performed well on simulated data relative to alternative approaches and produced reasonable empirical results given the natural history of bumblebees. The resulting inferences provide important insights into the population genetic dynamics of spatially structured populations.

Introduction

In many species, dispersal is geographically limited, leading to spatial structure. This spatial structure can, in turn, give rise to a pattern of isolation by distance (Wright 1943, 1946; Meirmans 2012), in which a focal individual is, on average, more closely related to an individual sampled nearby than it is to another individual sampled farther away. Much of early population genetic theory was derived under the assumption of random mating, which is significantly more tractable mathematically, and may also have adequately described the model organism populations common in empirical population genetic studies of the time (e.g. Drosophila in vials; Merrell 1953; Prout 1954; Dobzhansky and Spassky 1962) but is less well-suited to describing spatially continuous genetic structure.

Several theoretical approaches have relaxed these assumptions by modeling a population as partitioned into demes with some constant rate of migration between them [e.g. Wright's (1931) “island model” and Kimura and Weiss’ (1964) “stepping-stone model”]. The island and stepping-stone models inspired a series of statistical approaches that rely on partitioning samples into discrete populations with some level of genetic differentiation between them (e.g. Wright 1951; Pritchard et al. 2000; Pickrell and Pritchard 2012; Peter 2016). This genetic differentiation is estimated via F_ST, defined as the variance in allele frequencies within subpopulations relative to the total population. These approaches have been expanded to estimate the degree of admixture between these discrete populations (e.g. ADMIXTURE—Alexander et al. 2009), where individuals inferred to have ancestry proportions in multiple inferred clusters are described as “admixed.” However, these models (and the statistical approaches they inspired) still maintain panmixia within demes, and hence do not effectively capture population genetic dynamics in continuous space. For example, when patterns of ancestry are truly continuous, the inferred K clusters in a method like STRUCTURE or individual admixture proportions in ADMIXTURE are mere artifacts of the sampling scheme (Frantz et al. 2009; Bradburd et al. 2018). An example of this phenomenon in action can be found in studies that group human genetic variation by continent, a procedure that often generates an apparent pattern of discrete clusters (Rosenberg et al. 2002; Li et al. 2008); however, when individuals are the unit of investigation, human genetic ancestry is continuous and defies simple continental or population groupings (Ramachandran et al. 2005; Novembre et al. 2008; Carlson et al. 2022; Lewis et al. 2022).

Wright (1943) and Malécot (1948) were among the first to consider population models in which individuals were continuously distributed across one- and two dimensions in geographic space. Wright (1943, 1946) introduced the concept of “neighborhood size” ($\mathcal{N}$) as a statistic to describe natural populations; $\mathcal{N}$ was meant to capture the number of potential parents within a given radius of a focal individual, and was defined as half the inverse of the probability that two alleles sampled in nearby individuals were derived from the same ancestral allele in the previous generation. Wright (1946) showed that, for populations occupying an infinitely large range, in one-dimension $\mathcal{N}$ = $2\sqrt{\pi }\sigma \rho$ and in two-dimensions $\mathcal{N}=4\pi {\sigma }^2\rho$, where π is the mathematical constant, $\mathrm{\sigma }$ is the dispersal distance along one axis defined as the standard deviation of a normal distribution with mean zero, and ρ is the local population density. The early theoretical work of Wright and Malécot has been further expanded by Kimura and Weiss (1964); Nagylaki (1975, 1978); Felsenstein (1976); Barton et al. (2002); Barton et al. (2013), among others.

Despite these theoretical advances, there are far fewer statistical methods for examining populations in continuous space than as discrete entities. Wright's neighborhood size ($\mathcal{N}$) provides important information about the dispersal potential and the rate of genetic drift in continuously distributed populations at a localized level, and is therefore a useful quantity to know, both for conservation purposes and for a general understanding of the evolutionary context of a particular population or species.

Rousset (1997, 2000) introduced a method for the estimation of Wright's neighborhood size as the inverse of the slope of a regression between pairwise F_ST/(1 – F_ST) and the logarithm of pairwise geographic distance. While this method is limited in that it assumes a constant population density across space and time, it has the useful benefit of providing a single estimate of $\mathcal{N}$ for the entire population (Shirk and Cushman 2014). Several popular programs implement this expected relationship to enable researchers to estimate $\mathcal{N}$(e.g. SPAGeDI—Hardy and Vekemans 2002; Rousset and Leblois 2012 and GENEPOP—Rousset 2007). Importantly, the decision of which estimator of F_ST to use is non-trivial and can produce dramatically different results (Pearse and Crandall 2004; Bhatia et al. 2013). Furthermore, researchers must decide whether to estimate F_ST between individuals or between designated subpopulations. The former is very sensitive to individual measures of genetic diversity, which can be particularly noisy and impacted by bioinformatic decisions that affect the presence of missing data and rare variants (Bhatia et al. 2013); the latter, lumping individuals into subpopulations, can smooth this noisiness, but this “lumping” approach is not ideal when sampling covers a large geographic area and there is a continuous pattern of isolation by distance (Pearse and Crandall 2004).

Another quantity that, like Wright's neighborhood size, is useful for understanding and conserving species is the effective population size (N_e). While $\mathcal{N}$ is a measure of the local rate of drift and dispersal (analogous to the number of migrants in Wright's island model in two dimensions; Wright 1931), N_e is a global measure, capturing the long-term rate of coalescence. In a conservation setting, N_e may serve as a rough proxy for census size and the adaptive potential of threatened populations (Theodoridis et al. 2021; Exposito-Alonso et al. 2022). Others have used N_e to investigate its relationship with range size or dispersal ability (De Kort et al. 2021; Leigh et al. 2021). Estimation of the coalescent effective size, N_e, which describes the expected rate of coalescence in a population (Kingman 1982; Wang and Caballero 1999; Sjödin et al. 2005; Charlesworth 2009), often relies on its relationship with genetic diversity, which, when the mutation rate is μ and the population is at mutation-drift equilibrium, is given by 4N_eμ (Kimura and Crow 1964). While often presented as a single N_e value, the coalescent effective size is more accurately represented as the inverse of the rate of coalescence at time t, N_e(t), and hence technically has no single value. One estimator of the population mutation rate is Wu and Watterson's θ_w, which is S/a_n where S is the number of segregating sites in a sample of n sequences and $a_n={\sum }_{i=1}^{n-1}\frac{1}{i}$ (Watterson 1975). However, Wu and Watterson's θ_w is naive to the spatial structure of the sample and is known to be upwardly biased relative to random mating expectations when neighborhood sizes are very low (Wilkins 2004; Battey et al. 2020). Another estimator of the population mutation rate, also biased in the presence of spatial structure, is $\hat{\pi }$, which is $\hat{\pi }=\frac{n}{n-1}{\sum }_{ij}{x}_i{x}_j{\pi }_{ij}$, where n is the number of samples, x_i and x_j are the frequencies of the ith and jth sequence, and ${\pi }_{ij}$ is the proportion of nucleotide differences between the ith and jth sequence (Nei and Tajima 1981). At mutation-drift equilibrium, and assuming no selection, θ_w = $\hat{\pi }$ = 4N_eμ.

An important issue emerges in these estimators of N_e in the presence of population structure with respect to sampling scheme. Often, sampling is under-dispersed due to the vagaries of field collection techniques that prioritize easy-to-sample locations and are therefore often spatially under-dispersed relative to the population's range. When sampling is spatially under-dispersed and populations are spatially structured, common estimators of N_e are downwardly biased by the inclusion of nearby individuals who are more closely related than expected by chance. Hence, studies that are interested in comparing estimates of N_e across species may be significantly underestimating long-term N_e in the presence of spatial structure when these estimates rely on summary statistics like $\hat{\pi }$. However, in general, F_ST is low, and in many systems, this issue may not play a major role in shaping genetic diversity.

In continuously distributed populations at migration-drift equilibrium, inbreeding N_e is not independent of $\mathcal{N}$, as each is impacted by dispersal (Supplementary Fig. 1; Barton et al. 2002; Wilkins 2004). When dispersal is high, N_e converges to the population census size of reproductive adults (N_c), but it tends to be much larger when dispersal is low (assuming constant density). Similarly, because higher dispersal leads to more potential parents, it increases $\mathcal{N}$. An ideal model of spatial population structure would thus be individual-based so that researchers would not need to arbitrarily group individuals into subpopulations, and would co-estimate Wright's neighborhood size and inbreeding N_e while explicitly accounting for the shared influence of dispersal across timescales. In addition, estimators of N_e within the model would be robust to sampling scheme and, ideally, not force researchers to make arbitrary decisions about which individuals are “too related” to be included within the dataset.

In this paper, we take steps toward this goal by introducing a model that jointly estimates $\mathcal{N}$ and long-term diversity (which is related to N_e) from data on pairwise sequence divergence and geographic distance between individuals. We validate the model's behavior using individual-based forward-time simulations and compare its performance against Rousset's (1997). Finally, we apply our model to an empirical dataset of Bombus bifarius, the two-form bumblebee, to evaluate its utility in practical applications.

Methods

Model intuition

A pedigree is shaped by an organism's life-history, including its dispersal potential, generational structure, and mating strategies. Because mutations occur along the branches of the genetic genealogies embedded within the pedigree, the shape of the pedigree fundamentally determines the diversity of a sample as well as a whole host of additional summary statistics (Fig. 1). The field of statistical population genetics is predicated on the idea that information about processes shaping the pedigree (e.g. selection and demography) leave their imprint in patterns of genetic diversity and divergence observable in a modern-day sample.

Fig. 1.

Relationship between the separation-of-timescales of the coalescent and the isolation-by-distance curve in continuous space. a) A single representative gene tree showing the relationships between sampled individuals (colored circles) across a continuous landscape with dimensionality (x, y). Relative to a focal sample (dotted circle), the transition to the collecting phase occurs as the rate of coalescence converges to a neutral Kingman's coalescence. b) An isolation-by-distance plot relative to the focal individual. The transition to the collecting phase occurs when geographic distance is no longer predictive of genetic distance (i.e. pairwise heterozygosity). The red dotted line denotes s (or 1 – π_c), which is the estimated mean minimum relatedness between individuals in the population.

For populations that are spatially structured, such that there is correlation between geographic location and genetic ancestry, the set of genetic genealogies contained within the pedigree (the Ancestral Recombination Graph, or “ARG;” Lewanski et al. 2024) becomes distorted relative to random-mating expectations (e.g. Anderson-Trocmé et al. 2023). In two-dimensions, the coalescent process can be considered as occurring in two distinct phases: the scattering phase and the collecting phase (Wright 1943; Wakeley 1999; Wilkins and Wakeley 2002; Wilkins 2004). Going backward in time from the present, the scattering phase occurs first, and is characterized by lineages that are geographically near one another coalescing, on average, more rapidly than expected under random mating (i.e. with a probability greater than 1/2N_e, Wilkins and Wakeley 2002; Wilkins 2004). This signature is especially strong when dispersal is low, as nearby individuals are more likely to be more closely related than a pair of individuals selected from the population at random (with respect to geography). The parameter that governs the rate of coalescence in this phase of the ARG is Wright's neighborhood size, which, in two-dimensions, is defined as $\mathcal{N}=4\pi \rho {\sigma }^2$, where π is the mathematical constant, ρ is population density, and σ is the standard deviation of the effective dispersal distance along an axis defined by a normal distribution with mean zero (Wright 1940, 1943, 1946, 1949). Forwards in time, $\mathcal{N}$ describes the number of potential mates within a circle of radius 2σ, within which breeding occurs approximately at random with respect to geographic position. Backwards-in-time, Wright's neighborhood size can be thought of as the “pool of possible parents” of a focal individual (i.e. the number of reproductively mature individuals within a circle of radius 2σ centered on the focal individual). In Fig. 1a, we show a single representative gene tree that depicts coalescence happening more quickly between nearby individuals than distant individuals (although we note that the pattern of relatedness depicted in Fig. 1b is generated by the aggregation of many gene trees that adhere to this geographical pattern).

Further in the past—exactly how far depends on the rate of dispersal and the habitat geometry (Wilkins 2004)—the coalescent process shifts to the second phase, known as the collecting phase, in which the rate of coalescence is independent of the geographic distribution of the modern-day sample of individuals. This independence arises because, following a focal individual's pedigree backward through time and across space, the geographic distribution of its genetic ancestors expands until it ceases to be correlated with that focal individual's location (Fig. 1b, Wilkins 2004; Bradburd and Ralph 2019). In the collecting phase, the rate of coalescence in a simple model (e.g. the island model, the stepping-stone model, or in continuous, two-dimensional space) is well-represented by a neutral coalescent process (Kingman 1982), in which the rate of coalescence t generations in the past is 1/2N_et and the average time to the most recent common ancestor is 4N_e generations. Importantly, N_e is not independent of σ and will, in general, be larger at lower σ.

This two-phase distortion of the shape of the ARG relative to random-mating expectations affects estimates of the genetic diversity of the population. In a spatially structured population at migration-drift equilibrium, dispersal limitations shrink the depth of the coalescent tree locally (i.e. individuals nearby are more related on average than expected under panmixia) while expanding it globally (farther away, individuals are more distantly related than expected). As a result, estimates of $\hat{\pi }$ or $\hat{\theta }$ are dependent on the spatial scale of sampling; the genetic diversity estimated from a sample of nearby (and therefore likely more related than a sample of individuals taken at random from across the species’ range) individuals is likely lower than that of the entire population or species. This effect is particularly strong when dispersal is very low relative to the length of the range and the geographic area encompassed by the genotyped samples is small (e.g. Exposito-Alonso et al. 2022).

Our model (explained below) relies on the relationship between gene genealogies and the pattern of isolation by distance (Fig. 1b, 2). At short geographic distances, there is strong spatial autocorrelation of relatedness—this captures the scattering phase of the ARG, and it decays rapidly (Fig. 1b). As the curve flattens, geographic distance ceases to explain relatedness and the population approaches expectations under panmixia—this is capturing the transition to the collecting phase. The shape of the decay of relatedness over short spatial scales carries information about $\mathcal{N}$, whereas the inferred asymptote of relatedness over large geographic distances represents a diversity equilibrium (what we term “collecting phase π”, ${\pi }_c$), and is most informative about long-term inbreeding N_e. We define the quantity ${\pi }_c$ as an estimate of the expected maximum heterozygosity between a pair of individuals in the population. Because a random (or even an exhaustive) sample of a population will contain many pairs of closely related individuals, expected heterozygosity itself should be lower than ${\pi }_c$; the latter effectively ignores relationships in the scattering phase and hence behaves as if samples were collected in a spatially over-dispersed way (Supplementary Fig. 2).

Fig. 2.

Expected relatedness decay curves of ${\mathrm{\Omega }}_{i,j}$ with distance given s = 0.95 for various values of $\mathcal{N}$ (Equation 5 in the text).

Model

We first introduce the model of isolation by distance (IBD) that we use—both its form and its assumptions—and then describe how we fit this model to observed genomic data. Briefly, our model is closely related to previous theoretical models of genetic differentiation in continuous space (e.g. Wright 1943; Malécot 1948; Barton et al. 2002, 2010, 2013; Ringbauer et al. 2017), and describes the decay in pairwise homozygosity with the geographic distance between samples assuming a homogeneous landscape with isotropic dispersal. We implement this model in a Bayesian framework to estimate the posterior distribution of model parameters conditioned on observed pairwise sample homozygosity and pairwise geographic distance between samples.

Our model seeks to capture two important components of spatially structured populations: (1) that samples covary in their allele frequencies, with a covariance that decays with geographic distance during the scattering phase, and (2) that there exists an equilibrium level of maximum divergence between individuals that is established during the collecting phase (Fig. 1). Furthermore, our model assumes that populations exist in continuous space in two-dimensions and that dispersal is random and diffusive. In a single dimension, tracking diffusive dispersal backwards-in-time, two lineages will eventually exist in the same location at the same time at some point in the past. However, in two-dimensions, lineages diffusing via Brownian motion will never arrive in the same place at the same time (e.g. Nagylaki 1978; Barton et al. 2002). Modeling a spatial coalescent process in two dimensions is therefore tricky. This issue has been circumvented in the past (Wright 1943; Malécot 1948) by assuming that individuals need not be in the exact same location at the same time, but merely within a given radius of one another. Within this radius, individuals are assumed to interact in a way that is independent of the geographic distance between them. The rate of coalescence between individuals within this radius is determined by the effective population density, ρ_e (Barton et al. 2002), which defines the probability of identity by descent 1/(2ρ_e) for nearby genes in some previous time slice, integrated over all separations within that “nearby” distance. For habitats that are relatively homogenous, such that geographic distance is the primary explanation of covariance, with constant ρ and σ through time and across space, the probability of samples i and j being identical by descent ($F_{ij}$) can be estimated by

$$F_{ij}={\displaystyle \frac{1}{4\pi \rho {\sigma }^2}{K}_0\left(\sqrt{2\mu }{\displaystyle \frac{d_{ij}}{\sigma }}\right)}$$ (1)

where K₀ is a modified Bessel function of the second kind of order 0, d_ij is the geographic distance between samples i and j, and π is the mathematical constant (Wright 1943, Malécot 1948; Barton et al. 2002). Barton et al. (2002) note that this approximation diverges as $d_{ij}\to 0$; to account for this, following Ringbauer et al. (2017), we designate a short distance, κ, within which the rate of coalescence becomes a constant γ that describes the mean probability of being identical by descent between all pairs of individuals closer than $\mathrm{\kappa }$ to each other. There is no optimal value for $\mathrm{\kappa }$ as it depends on the local population structure, but is generally of order σ (Barton et al. 2002).

The classic Wright-Malécot formula (Eqn. 1) describes the theoretical probability of identity-by-descent, which, in an infinite population, decays to zero as the distance between a pair of sampled individuals goes to infinity (see also Maruyama 1972). Our model breaks the assumptions of the Wright–Malécot model in two important ways. First, we assume that populations are finite, meaning that all individuals are identical by descent at some point in the past. Second, we choose to model identity-by-state, rather than identity-by-descent, as we assume more empiricists will have access to identity-by-state information than identity-by-descent (particularly in non-model organisms). Therefore, we must incorporate into our model a background rate of genetic similarity at which all individuals in the population are identical-by-state. The expected homozygosity of a pair of samples, i and j, is thus

$${\hat{H}}_{ij}=F_{ij}+(1-F_{ij})s$$ (2)

where $s=\frac{1}{L}{\sum }_{\ell =1}^L{\hat{p}}_{\ell }^2+(1-{\hat{p}}_{\ell }{)}^2$ and ${\hat{p}}_{\ell }$ is the frequency of a particular allele in a spatially over-dispersed sample of individuals at the ℓth of L biallelic loci (Ringbauer et al. 2018). The quantity s represents the “background” rate of sequence similarity (the mean minimum sequence similarity between any pair of individuals) and can be thought of as the complement of the amount of genetic diversity in a population at equilibrium during the collecting phase. We can therefore define a quantity “collecting phase π”: $1-s=\frac{1}{L}{\sum }_{\ell =1}^{L}2{\hat{p}}_{\ell }(1-{\hat{p}}_{\ell })={\pi }_c$. Collecting phase π can be used as an estimator of long-term inbreeding N_e, independent of the scattering phase. Relative to $\hat{\pi }$ (mean pairwise genetic distance in a population, Nei and Tajima 1981), estimates of ${\pi }_c$ (defined by the asymptote of the IBD curve, which showcases the transition to the collecting phase; Fig. 1) should be less sensitive to the size of a sampled area.

Inference and implementation

We assume users’ data consist of allele frequencies taken across L unlinked, biallelic single nucleotide polymorphisms (SNPs) genotyped across a set of N samples. Each sample may consist of a single individual or a group of individuals collected at a single location. Allele frequencies may be estimated from genotype data (e.g. the frequency of an allele at the ℓ th locus in the nth sample is simply the number of times that allele is observed divided by the total number of genotyped haplotypes in that sample at that locus) or from pooled sequencing data. From these data, we compute the sample pairwise homozygosity, which is the complement of the pairwise diversity between the samples (i.e. $1-D_{xy}$). We note that users working with low-coverage sequence data may wish to generate estimates of D_xy without conditioning on allele frequencies (e.g. Buerkle and Gompert 2012; Ellegren 2014). We assume that SNPs are ascertained without bias; as with other approaches for statistical inference from population genetic data, ascertainment bias may strongly impact results, although in ways that are difficult to predict, depending on the nature of the bias. Pairwise homozygosity between samples i and j gives the probability that, at a locus chosen at random, a pair of alleles sampled at random from i and j, respectively, are the same. We calculate it as:

$${\hat{H}}_{ij}=1-{\displaystyle \frac{1}{L}{\displaystyle \sum _{\ell =1}^L}{\hat{f}}_{i,\ell }(1-{\hat{f}}_{j,\ell })+{\hat{f}}_{j,\ell }(1-{\hat{f}}_{i,\ell })}$$ (3)

where, ${\hat{H}}_{ij}$ gives the sample homozygosity between samples i and j calculated across all L loci, and ${\hat{f}}_{i,\ell }$ gives the sample allele frequency in the ith sample at the ℓth locus. Pairwise homozygosity is a measure of absolute genetic similarity, so it is not sensitive to the sampling configuration. Additionally, $\hat{H}$ is proportional to the allelic diversity defined by Bradburd et al. (2018), so we proceed by assuming it can be reasonably modeled as Wishart-distributed, and the framework we use for statistical inference is similar to that of Bradburd et al. (2018); see also Ringbauer et al. (2018). Note that, in doing so, we are assuming that allele frequencies are multivariate normally distributed across samples and are independent between loci (i.e. not in linkage disequilibrium). Linkage disequilibrium (LD) will have the effect of decreasing the number of independent observations we have, thereby decreasing the actual number of degrees of freedom of the Wishart distribution.

To infer parameter values, we construct a parametric expected homozygosity matrix using a modified version of the Wright-Malécot model of isolation by distance introduced in Equation 2 and calculate the likelihood of the sample homozygosity as a draw from a Wishart distribution parameterized by the parametric homozygosity. Because the individual parameters of the Wright–Malécot model (in particular, σ and ρ as well as σ and μ) are partially non-identifiable, we instead implement a model that employs the compound parameters [following Ringbauer et al. (2018)] $\mathcal{N}$, which is neighborhood size, and is defined as $4\pi \rho {\sigma }^2$, and m, which is defined as $\frac{2\mu }{{\sigma }^2}$ (Barton et al. 2002; Ringbauer et al. 2018). Concretely, we write the probability of identity by descent between samples i and j, $F_{ij}$, as

$$F_{ij}=\{\begin{array}{c}\gamma if{d}_{ij}\le \kappa \\ {\displaystyle \frac{K_0\left(\sqrt{m}{d}_{ij}\right)}{\mathcal{N}}if{d}_{ij}>\kappa }\end{array}$$ (4)

where γ is the probability of being identical by descent at distances $d_{ij}\le \kappa$ (i.e. short enough that mating might reasonably be considered panmictic), and $F_{ij}$ for $d_{ij}>\kappa$ is given by the Wright-Malécot function introduced in Equation 2. We then construct our parametric homozygosity between individuals i and j, ${\mathrm{\Omega }}_{ij}$, as

$${\mathrm{\Omega }}_{ij}=F_{ij}+(1-F_{ij})s+{\delta }_{ij}{\eta }_i$$ (5)

where $(1-F_{ij})s$ is the rate of identity by state not due to identity by descent, ${\delta }_{ij}$ is the Kronecker δ, and ${\eta }_i$ is a statistical parameter we include in our model to describe inbreeding specific to the ith individual. We then calculate our likelihood as

$$P(\mathrm{\Omega })=W(\mathrm{\Omega },L)$$ (6)

where L is the number of independent genomic loci used in the calculation of $\hat{H}$.

We take a Bayesian approach to infer the parameters of this model. The posterior probability density of our parameters is given by

$$P(\hat{H})\propto P(\mathrm{\Omega }(\gamma ,m,\mathcal{N},s,\overrightarrow{\eta }))P(\gamma )P(m)P(\mathcal{N})P(s)P(\overrightarrow{\eta })$$ (7)

where $\mathrm{\Omega }(\gamma ,m,\mathcal{N},s,\overrightarrow{\eta })$ denotes the dependence of Ω on its constituent parameters $(\gamma ,m,\mathcal{N},s,\overrightarrow{\eta })$ , and $P(\theta )$ denotes the prior probability of a given parameter θ. The parameters of our model are therefore $\mathcal{N}$, m, and γ (which determine F as defined in Equation 4), as well as s and η (which, along with F, determine $\mathrm{\Omega }$, as defined in Equation 5). Note that $\mathcal{N}$ and m are the two mechanistic (ontological) parameters of our model and entirely determine the expected probability of identity-by-descent between all pairs of individuals found greater than distance $\mathrm{\kappa }$ from each other. The parameters γ and η do not appear in the Wright–Malécot model of isolation by distance (Equation 1) and are included as phenomenological parameters here to facilitate statistical inference. However, we note that γ, which is intended to capture variation in relatedness over very small spatial scales, and η, which is intended to capture variation in inbreeding specific to the individual (i.e. above and beyond that found between individuals that occur very close to one another) are closely related to the quantities F_ST and F_IS, respectively. The value of $\mathrm{\kappa }$ is specified by the user, a modeling choice we discuss further below. Supplementary Table 1 describes the prior probability distributions we implement for each parameter in our model. We implement this model in Rstan (Stan Development Team 2023) and use STAN's Hamiltonian Monte Carlo algorithm (a type of Markov chain Monte Carlo, or MCMC, approach; Betancourt and Girolami 2015) to characterize the posterior distribution of the parameters.

Because pairwise homozygosity often varies over a very small absolute range (e.g. 0.99–0.999), we take several steps to facilitate inference on the parameters of the model. First, we estimate the parameters m, γ, and $\overrightarrow{\eta }$ in log space, which should help chains mix over the posterior density. Second, we scale the sample homozygosity $\hat{\mathrm{\Omega }}$ so that it varies between 0 and 1 (we apply the same scaling to the parametric homozygosity):

$$\begin{array}{c}a=\min (\hat{H})\\ \\ b=\max (\hat{H}-a)\\ {\hat{H}}^{\mathrm{\prime }}={\displaystyle \frac{\hat{H}-a}{b}}\\ {\mathrm{\Omega }}^{\mathrm{\prime }}={\displaystyle \frac{\mathrm{\Omega }-a}{b}}\end{array}$$ (8)

This scaling is not without drawbacks, as the variance of a Wishart distribution parameterized by Ω is not the same as that parameterized by ${\mathrm{\Omega }}^{\mathrm{\prime }}$. However, we feel that the benefits it offers outweigh the costs, particularly because, to our knowledge, there is no theoretically motivated “correct” variance for the expected homozygosity as a function of geographic distance.

Simulations

We evaluated model performance using individual-based forward-time simulations implemented in SLiM v3.6 (Haller and Messer 2019). Our simulations had non-overlapping generations, diploid and hermaphroditic individuals with haploid genomes of 100 Mbp, and a uniform recombination rate of 10⁻⁹ per base-pair per generation. Mate choice, spatial competition, and dispersal are controlled by a single parameter, σ. Distance units in SLiM are arbitrary; for convenience, we will refer to them as kilometers. Individuals were simulated on a continuous, two-dimensional 25 × 25 km landscape with reflecting boundaries. Total population density was regulated by an enforced local carrying-capacity, K, to avoid spatial clumping (Felsenstein 1975). This was achieved by first computing the spatial distance, d, between a focal individual and all individuals within distance 3σ. The strength of each interaction is determined by a Gaussian density, g(d) = exp(−d²/2σ²)/(2σ²), and the total competitive interactions felt by individual i is $c={\sum }_{j}g(d_{ij})$, where d_ij is the distance between individual i and j (Battey et al. 2020). Local population density was then regulated by scaling the fitness of individual i as fitness_i = K/c. Fitness was implemented via mortality, which was evaluated before mating occurred within the simulation. To reduce the impact of edge effects, we then divided this strength by the integral of the interaction after clipping by the bounds of the specified landscape (Haller and Messer 2022), which has the desired effect of rescaling competition relative to the occupiable area. Edge-effects and spatial competition collectively reduced the census population size by ∼35% relative to Kw², where w is the width of the square simulated landscape (w = 25 km). We also simulated scenarios in which the edge width was set to 50 km, as well as scenarios in which there were no edges (i.e. on a torus) for comparison (Supplementary Fig. 3–5). We implemented spatial mating dynamics in a way similar to spatial competition: mates were chosen within a maximum distance of 3σ, with each potential mate assigned a weight that was an inverse function of its geographic distance from a focal individual. These distances were converted to weights using a Gaussian function with a max 1/2πσ²; the lower the total probability of all mates, the higher the probability that a focal individual might potentially not choose any mate. The number of offspring produced per mating event was drawn from a Poisson distribution with shape parameter λ = 2. Finally, offspring dispersal was drawn from a normal distribution with mean 0, a standard deviation σ, and a maximum of 3σ.

We performed simulations across a range of values of K (2, 5, 10, 25) and σ (0.5, 0.75, 1.0, 1.25, 1.5, 2.0), where σ is the distance moved along a single axis, which varied theoretical neighborhood size from a minimum of 6.25 to a maximum of 861.81, and total census size from ∼800–10,000 (note these values are scaled by the observed density instead of the input K). We performed 10 simulation replicates for each combination of parameter values, for a total of 240 simulations. Each simulation was run for 100,000 generations to ensure time for dispersal to shape patterns of genetic diversity. The output from SLiM were tree-sequences, which were parsed in Python using the package pyslim v.1.0.1 (Kelleher et al. 2018). For trees in which multiple roots existed (i.e. coalescence had not yet occurred during the SLiM run), we performed “recapitation,” which simulates a neutral coalescent process among remaining, uncoalesced lineages using the parameter-combination specific census population size (Haller et al. 2018). Only 25% of simulations (n = 60) had not coalesced and required recapitation; of these, 59 had two roots, and a single simulation had 3 roots. As such, our decision about which population size we used for recapitation had minimal impact on overall results (Supplementary Fig. 6). Mutations were then simulated onto the tree-sequences using msprime v.1.2.0 (Kelleher et al. 2016) at a rate of 10⁻⁷ per base-pair per generation. We randomly sampled 100 individuals alive in the final generation and calculated pairwise π (the average genetic divergence between a pair of samples) between each pair of individuals using tskit v.0.5.3 (Kelleher et al. 2018). We output the pairwise individual π matrix and geographic coordinate matrix, the latter of which was converted into a distance matrix in R (R Core Team 2022) using the function rdist in the package fields v.2.9–1 (Nychka et al. 2021). These two matrices were then used as input to our statistical model, which we used to perform inference on all model parameters ($\mathcal{N}$, s, λ, and m, and $\overrightarrow{\eta }$). These values have theoretical expectations given the parameters we chose for each individual simulation, but are not explicitly defined because we were not simulating directly under the inference model.

To examine how recent population fluctuations might impact our model inference, we also investigated simple contraction and expansion scenarios. In each scenario, the value of $\mathrm{\sigma }$ was set to 1. The expansion scenario was initialized with K = 5 (census population size of ∼2200), the contraction scenario with K = 10 (census population size of ∼4400). For each, after 100,000 generations of the initial population size, we instituted an instantaneous size change, either a doubling (expansion) or halving (contraction) in size, and sampled individuals after a number of generations equal to 0.1, 1, and 10% of the post-change census population size. We simulated 10 replicates of each scenario and followed the same procedure as the constant-size simulations, sampling 100 individuals, re-capitating (using the initial population size) if necessary, and, for each simulation, outputting the pairwise individual π matrix and geographic coordinate matrix. Because $\mathcal{N}$ is expected to be a reflection of the dynamics of the scattering-phase, we expected that it should largely reflect the new population census size, whereas ${\pi }_c$ should approximately reflect diversity prior to the size change, as it captures deeper-time dynamics.

We ran the implementation of our model on each simulated dataset, running four independent MCMC chains of 4,000 steps each. We pruned the first 2,000 steps in each chain as burn-in and thinned the remaining steps by sampling every eighth iteration, for a total of 250 sampled post burn-in iterations per chain. Because some chains displayed poor mixing, we visually verified that the chains had achieved convergence by inspecting the trace plots of the posterior probability and parameter values of interest, then selected the (well-behaved) chain with the highest mean posterior probability as the one from which to report results. For purposes of testing the model against simulations, we set κ = 0.25 km.

Simulations and model estimation were performed on the Advanced Research Computing (ARC) Great Lakes High Performance Computing Cluster at the University of Michigan. Code, including the ARC HPCC SLURM workflow, SLiM recipes, and Python scripts, can be found at https://github.com/zachbhancock/WM_model.

Finally, we compared our estimated values to the true $\mathcal{N}$, which we estimate as $4\pi {\rho }_e{\sigma }_e^2.$ Here, ${\rho }_e$ is calculated as $2N_p/A$, where $N_p$ is the number of parents in the generation preceding sampling, A is the area of the simulation, and ${\sigma }_e$ is the maximum likelihood estimate of the variance of parent-offspring dispersal (the average squared parent-offspring displacement in each dimension); both ${\rho }_e$ and ${\sigma }_e$ are calculated for each simulation. We include a comparison of $\mathcal{N}$ as calculated using the parameter values we input into SLiM to this effective $\mathcal{N}$ in Supplementary Fig. 7. To evaluate our performance in estimating ${\pi }_c$, we compared our model estimate with mean pairwise π calculated between all sampled pairs of individuals that were at least 20 km away; the number of pairs compared varied by simulation because samples were taken randomly with respect to space. This distance was chosen because it is the minimal distance that ensured in all simulations, irrespective of the value of σ, there was no longer spatial signal in the isolation-by-distance curves, indicating that the populations had transitioned to the collecting phase. The distance at which the isolation-by-distance curve levels off varies between simulations is a function of the value of σ used to simulate the data; we pick a single cutoff to apply across all simulations for ease of comparison. We explored how the choice of this cutoff distance value affected estimates of ${\pi }_c$ by also using cutoffs of 15 and 25 km. We found that, when we used a distance cutoff of 15 km, we included significantly more related individuals, leading to an overestimation of ${\pi }_c$ relative to the sample, while at distances >25 km, there was no discernable difference in estimated diversity compared to samples >20 km (Supplementary Fig. 8).

Empirical dataset

We also tested the performance of our method on a published empirical spatial population genetic dataset of 383 individuals of the two-form bumblebee Bombus bifarius (Jackson et al. 2018a). Bumblebees have been widely studied due to their ecological and commercial importance as pollinators (Garibaldi et al. 2013). Like many hymenopterans, bumblebees are eusocial, and hence each nest represents a single reproductive female. Furthermore, there is evidence that some bumblebee species (including B. bifarius) tend to avoid mating with nestlings (Foster 1992); the number of nests within a given dispersal distance can thus be a reasonable proxy for $\mathcal{N}$ in these species.

We analyzed a published RADseq dataset for B. bifarius generated by Jackson et al. (2018a), in which the full molecular methods used to generate the data can be found. Briefly, sequence reads were generated from thoracic muscle tissue using RADseq and restriction enzyme PstI and sequenced single-end in multiplexed libraries (1 × 100 bp, Illumina HiSeq 2,000 or 4000). We downloaded the raw (demultiplexed) sequence reads for each of the 383 B. bifarius individuals archived in the National Center for Biotechnology Information (BioProject PRJNA473221, SRP149031; Jackson et al. 2018b) using the fasterq-dump function in the SRA Toolkit (V2.10.7, Kodama, Shumway, and Leinonen 2012). We dropped reads with uncalled bases and for which the mean Phred score was < 15 (sliding window 15% of read length; Stacks2 process_radtags module V2.54—Rochette, Rivera-Colón, and Catchen 2019). We confirmed that there were no over-represented sequences (sequences present in > 5% of reads per individual, averaged across all individuals) and trimmed all reads to a uniform length (80 bp). Next, we assembled these reads de novo using Stacks2 and nine combinations of assembly parameters (v.2.54, Rochette, Rivera-Colón, and Catchen 2019). We used the R80 rule (Paris, Stevens, and Catchen 2017) to select an optimal assembly (in this case, when parameters were: m = 3, M = 3, and n = 3) and dropped loci that were scored in <50% of individuals. This resulted in 58,970 loci that were used for the subsequent analyses. We then calculated pairwise $\pi$ and pairwise geographic distance between each pair of individuals in the dataset. Geographic distance here is measured as great circle distance (rdist.earth function of the fields R package; v.14.1—Nychka et al. 2021). Lastly, we chose $\kappa =1$ km for our empirical dataset; as discussed above, this theoretically represents the distance within which mating is panmictic (less than the per-generation dispersal distance of reproducing individuals). The scale of distance in this dataset is kilometers, and thus individuals are modeled as equally likely to be related if they are within 1 km of one another, which covers the scale of sample sites in Jackson et al. (2018a) but is much less than the distance between localities.

For comparison, we also estimated neighborhood size using Rousset's method. To do so, we compared two measures of F_ST: a SNP-based estimation from Jackson et al. (2018a), in which individuals are subsetted into “populations” (sample locations); and an individual-based estimation using the pairwise $\pi$ matrix used as input for our model. In this second, individual-based approach, we estimated F_ST as $1-2({\pi }_i+{\pi }_j)/({\pi }_i+2{\pi }_{ij}+{\pi }_j)$, where ${\pi }_i$ is individual diversity, and ${\pi }_{ij}$ is the divergence between i and j. We regressed F_ST against both geographic distance and log(geographic distance) and estimated $\mathcal{N}$ as 1/ β, where β was the slope of that regression.

Results

Our model performed well on both the simulated and empirical datasets (Supplementary Fig. 9–10). In the former, the model converged on the theoretical expectations for both $\mathcal{N}$ and ${\pi }_c$, which can be used to estimate N_e. Furthermore, it performed favorably relative to Rousset's method, which was unbiased, but had much higher variance than our model, especially at lower $\mathcal{N}$. Finally, as applied to the empirical dataset, our model produced reasonable results given the known natural history of bumblebees.

Simulation results

The individual-based simulations were performed by varying both dispersal distance (σ) and local carrying-capacity (K), which generated a range of $\mathcal{N}$ from 6.25 to 861.81. Given the simulated geographic range area, this range of values encompasses extreme/strong spatial structure at the lower end and almost panmictic populations at the upper end.

At almost all simulated values of σ and K, the 95% equal-tailed credible interval of $\mathcal{N}$ consistently included the theoretical $\mathcal{N}$; however, at larger σ and K, it did so with high variance, with the median falling below the theoretical value (Fig. 3a). This is likely due to the reliance of the model on detectable signatures of covariance between geography and ancestry, which decays at large $\mathcal{N}$ (Wilkins 2004) as illustrated in Fig. 2 (see also Supplementary Fig. 11). At small dispersal distances, the variance in our estimates of $\mathcal{N}$ was relatively low even at higher K. However, at larger dispersal distances, the variance subsequently increased regardless of K, indicating that σ likely has the largest impact on model precision. Notably, our model estimated a value of $\mathcal{N}$ greater than the true value in the simulation only at the lowest K and σ combination. In this region of parameter space, the Wright–Malécot model we have implemented is a poor approximation of relatedness. For datasets with small $\mathcal{N}$, it would be more appropriate to use the small $\mathcal{N}$ approximation to the Wright–Malécot model (Eqn 15, Barton et al. 2002). We discuss this further below. Although we found that Rousset's method is an unbiased estimator, the variance in its estimates across replicates is much larger at lower K and σ pairs than that of our model; Rousset's method can even generate negative values, the interpretation of which is murky (Fig. 4).

Fig. 3.

Model accuracy and precision. a) Estimates of Wright's neighborhood size for each value of K and σ (gray header); black circles are medians and error bars are the 95% quantile. Blue circles are the theoretical $\mathcal{N}$. b) As in (a), but for estimates of π_c. Blue circles represent π when d_ij > 20.

Fig. 4.

Results of Rousset's method for estimating $\mathcal{N}$ across datasets simulated with different values of K (population density in SLiM; x-axis) and σ (dispersal; gray header). As in Fig. 3a, black dots and bars are the estimated values with the median and 95% quantile; blue dots are the true $\mathcal{N}$ for each simulation. Notice the axes differ greatly between Fig. 3a and this figure, demonstrating the wide variance in Rousset's estimator. Supplementary Figs. 15–17 show Rousset's estimator conditioned by various maximum distance thresholds.

On simulated scenarios with recent population size changes, the model performed reasonably well (Supplementary Fig. 12–13). We slightly underestimated $\mathcal{N}$ in the expansion scenario (relative to the theoretical value of the post-expansion conditions); this underestimation was most pronounced at the most recent sampled time point after the population size had changed (0.1% N_c generations; Supplementary Fig. 13). Similarly, in the contraction scenario, we accurately estimated the new $\mathcal{N}$ at all times except at 0.1% N_c generations, when it was slightly overestimated. Our model accurately estimated the expected deep-time levels of diversity (${\pi }_c$), which is the diversity prior to the population size change (Supplementary Fig. 13). Collectively, these results indicate that our estimation of $\mathcal{N}$ reasonably tracks very recent population size changes, whereas ${\pi }_c$ consistently is influenced by deeper-time dynamics and is insensitive to recent demography.

The model also performed well at estimating ${\pi }_c$ (with the true value set as ${\pi }_{d_{ij}>20}$ or ${\pi }_{d_{ij}>25}$, see Supplementary Fig. 8), and did so with little variance across simulations. In the simulations with the lowest $\mathrm{\sigma }$, there was a slight overestimation of ${\pi }_c$, which may indicate that the empirical IBD curve still maintains a subtle signal at d_ij > 20; this would cause the model to extrapolate that the IBD curve hits its asymptote at greater distances, possibly beyond the range edge. Interestingly, we also find that the levels of ${\pi }_c$ are largely affected by values of K rather than values of $\mathrm{\sigma }$ (with the exception of the lowest σ). Theory predicts that the total amount of genetic diversity in a finite, spatially structured population should increase with the degree of spatial structure, and therefore decrease with the scale of dispersal (Wilkins 2004). The deviation from theory that we observed likely results from the fact that, in our models, K changes by a larger magnitude than $\mathrm{\sigma }$ (with respect to the total simulated area) across simulation parameter values.

Empirical results

For the empirical dataset of Bombus bifarius (two-form bumblebee), we performed five independent MCMC analyses, each with 5,000 iterations. The model mixed well (Supplementary Fig. 14), but the joint marginal plots showed that several parameters were interacting: m and s were positively correlated, and γ was negatively correlated with m and s (Supplementary Fig. 15). Furthermore, there was a weak but significant (R² = 0.02512; P = 1.056e-8) negative relationship between $\mathcal{N}$ and m. Similarly, there was a very weak but significant (R² = 0.0028; P = 0.0318) relationship between ${\pi }_c$ and $\mathcal{N}$. These relationships reflect relationships between parameters in our model, despite which we were nonetheless able to infer parameter values well.

We estimated that $\hat{\mathcal{N}}=137.18$ (±0.9) and ${\hat{\pi }}_c=0.002$ (±2e–6) for this two-form bumblebee dataset (Fig. 5). While the true $\mathcal{N}$ is unknown for most species, given the performance of the model on simulated data and the natural history of bumblebees, we think this is a reasonable estimate. We explore the biological implications (and caveats) of these results further in the Discussion.

Fig. 5.

Empirical application of the method. a) Sample sites of B. bifarius from Jackson et al. (2018a), size of circles represent the number of individuals sampled, and dark gray shading is the known range extent. b) Density of $\mathcal{N}$ estimates across all five chains. c) Density of N_e estimates, where N_e = ${\hat{\pi }}_c/4\mu$, and μ is the empirically estimated mutation rate for bumblebees (Liu et al. 2017).

Discussion

An extensive literature exists on the biases that spatial structure introduces to commonly employed population genetic summary statistics (e.g. Miermans 2012; Bradburd and Ralph 2019; Battey et al. 2020). However, many studies still use measures of effective population size (such as Wu and Watterson's θ_w and π) that are naïve to population structure (and the geographic sampling of individuals) in empirical systems that are spatially structured. Other statistical population genetic approaches attempt to account for population structure by discretizing the habitat into demes defined by sampling region or violation of Hardy–Weinberg (e.g. STRUCTURE—Pritchard et al. 2000; Montana and Hoggart 2007), but in reality, many populations display a continuous pattern of isolation by distance that cannot be adequately reduced to a discrete stepping-stone model (Kimura and Weiss 1964). Indeed, even in a fine-scaled lattice population, Battey et al. (2020) showed that biases in estimates of Wu and Watterson's θ and Tajima's D emerge as the sample size per deme approaches the deme size.

To investigate populations with continuous patterns of isolation by distance, we suggest that a single summary diversity statistic often does not capture the dynamics of interest. For example, $\hat{\pi }$ is dragged down by rapid coalescence in the scattering phase but inflated by the influence of low dispersal in the collecting phase. Hence, $\hat{\pi }$ ceases to adequately reflect either process—it cannot tell us about local demography because it is upwardly inflated by deep-time coalescence, and it cannot tell us about ancient events because it is downwardly biased by local demography. Indeed, dispersal acts in opposing ways in these two phases: low dispersal causes local individuals to be more related on average and increases the length of the scattering phase, but over large distances and deeper timescales it inflates $\hat{\pi }$ (Wilkins 2004). Our model generates estimates of a diversity statistic (${\pi }_c$) that is insensitive to the geography of sampling, and simultaneously provides estimates of Wright's neighborhood size, thereby better capturing the spatial dynamics of the population.

Simulations

Our model performs well over a wide range of true $\mathcal{N}$, though the variance of our estimates of $\mathcal{N}$ increases as the true $\mathcal{N}$ grows large. In contrast, Rousset's (1997) estimator consistently had extremely large variances and would often take negative values (Fig. 4), particularly when dispersal was high (resulting in low differentiation across the sampled range). At small $\mathcal{N}$, our model overestimates $\mathcal{N}$ slightly; this is likely due to the fact that we have implemented the “large $\mathcal{N}$ ” approximation of the Wright–Malécot model, which must be modified for small $\mathcal{N}$ as shown in Equation 15 of Barton et al. (2002). However, the departure between the truth and our model estimates is relatively small, so we feel like this shortcoming is outweighed by the convenience to the empirical user of being able to implement a single inference model, rather than one that might vary based on the specifics of their system (and their choice of κ).

In addition, in our simulations, the spatial scale of dispersal, mate choice, and competition were all set to be the same (controlled by a single parameter, σ). This will obviously not be the case for most empirical systems; however, we believe that even in such systems, our method will still be able to accurately infer the standard deviation of effective dispersal.

Empirical application

Few empirical systems have reliable estimates of effective population density and dispersal distance such that estimates of Wright's neighborhood size made using genetic data can be directly compared against a known biological quantity. However, because bumblebees are an important commercial pollinator, their populations are better characterized than many natural systems, and we can use this natural history knowledge to inform the interpretation of our model-based estimates. Goulson et al. (2010) examined bumblebee nest density across a 10 × 20 km area for B. lapidarius and B. pascuorum, two species that are closely related to our model species (Bombus bifarius). On the scale of their sampling, they found significant F_ST between, but not within, sites, where each “site” is a 200 × 10-meter strip within the 10 × 20 km censused area, indicating that the distance between sampling site locations might reasonably be expected to exceed the average dispersal distance in a generation. Within sites, they estimated a mean of 114 nests per site for B. lapidarius and 87 nests per site for B. pascuorum. Because bumblebees are eusocial, each nest represents a single mating female, and queens tend to avoid mating with nestlings (Foster 1992). If we therefore assume that the mean number of observed nests per site is roughly the same as the pool of possible parents within two dispersal distances of a focal individual in B. lapidarius and B. pascuorum, our estimate of $\mathcal{N}$ in B. bifarius ($\hat{\mathcal{N}}=137.18$ (±0.9)) is quite close to the observed values of $\mathcal{N}$ in two closely related species. One thing to note is that the uncertainty associated with this estimate (as well as other estimates output by the model) is a function of the number of degrees of freedom specified for the Wishart distribution, which, in this model, is the number of specified loci. Because that number can be very large, it can lead to unrealistically low uncertainty in parameter estimates, which should be interpreted with caution.

We applied Rousset's method to the SNP dataset from Jackson et al. 2018a both assuming a two-dimensional population (i.e. with geographic distance logged) and one-dimensional (raw geographic distance). The former produced an estimate of $\hat{\mathcal{N}}=34141\pm 1821494$ (Supplementary Fig. 16), with the latter estimating $\mathcal{N}$ as 66979 (± 6887) (Supplementary Fig. 17; see also Supplementary Fig. 18–20). In light of the extremely high variance in Rousset's estimator's results from both our simulations and in this empirical example, it is difficult to evaluate the accuracy of these results. We also do not know the extent to which edge effects are impacting this system; however, given that our model performs well in the presence of edges, we do not think these will systematically bias our estimates.

We are making a number of assumptions regarding the comparability of these census values taken from B. lapidarius and B. pascuorum and our estimate of $\mathcal{N}$ in B. bifarius. First, we are assuming that belonging to the same genus is predictive of similarity in $\mathcal{N}$ between species, which seems reasonable, but for which, to our knowledge, there is little empirical support. Second, we are assuming that each “site” in Goulson et al.'s (2010) study represents an area approximately equal to that of a circle of radius 2σ, such that the number of nests within a site can be thought of as the pool of possible parents within two dispersal distances of a focal individual. Third, and perhaps most crucially, we are assuming that there is a good concordance between these censused quantities and the effective $\mathcal{N}$ that can be estimated from genetic data. There are many reasons why this assumption may be violated, most notably, if there is either a high variance in reproductive output between queens, such that the effective density differs from the census density, or if there is a relationship between dispersal distance and fitness, such that effective σ differs systematically from the standard deviation of the forward-time parent-offspring dispersal distance distribution. The validity of these assumptions is difficult to evaluate without more data taken across generations, but it is nonetheless reassuring that our estimate of $\mathcal{N}$ in B. bifarius is of the correct order of magnitude compared to the census neighborhood sizes in two closely related species.

Model assumptions and shortcomings

While our model performs well on simulated data and the empirical dataset presented here, it makes several important assumptions that may often be violated in natural systems. First, we assume that dispersal is random (within a specified dispersal kernel) and non-directional; thus, our model may poorly approximate patterns of isolation by distance in organisms that have directional movement, such as some marine planktonic species that may be carried by ocean currents or wind-dispersed pollen grains. Second, our model assumes that the habitat is relatively homogenous such that there are no major barriers to gene flow—i.e. patterns of isolation-by-distance are generated solely by the traversable Euclidean distance between two individuals. Strong physical or environmentally mediated barriers to dispersal may affect model performance (Wang and Bradburd 2014; Bradburd et al. 2013; Ringbauer et al. 2018). In a similar vein, our model ignores confounding factors such as local adaptation that may drive clinal patterns of relatedness (e.g. Pruisscher et al. 2018; Jofre and Rosenthal 2021).

Another important consideration is that the Wright–Malécot model relies on assumptions that are mutually incompatible (independent dispersal and homogeneous population density). In SLiM, we modeled populations with local density dependence to overcome Felsenstein's “pain in the torus” (Felsenstein 1975) and maintain a stochastically homogenous distribution of individuals across space. In this way, comparing the theoretical expectations from Wright–Malécot with a population model in which dispersal and density are not strictly independent is inexact. However, despite this discrepancy between theory and the simulated population, our model reasonably captured the true simulation neighborhood size, indicating that the Wright–Malécot formulation that underpins our model is robust to this violation.

Finally, our model relies on a user-defined κ, interpreted as the minimum distance between two individuals in which the Wright–Malécot formulation breaks down and relatedness converges on 1/2ρ (Barton et al. 2002; Ringbauer et al. 2018). Preliminary exploration of the model indicates that model performance is poor at arbitrarily high κ, but not at low κ (Supplementary Fig. 21). This is due to the fact that, at higher κ, less of the scattering phase is being captured by the model, leading to poor mixing. Ideally, κ would be estimated like the other parameters of the model; however, an attempt to implement such a model led to dramatic increases in computation time, and thus for the current work we opted to set a constant κ.

Conclusions

For many organisms, geographical distance influences dispersal, competition, and mate-choice, leading to patterns of continuous spatial structure. An important parameter governing the strength of isolation by distance is Wright's neighborhood size ($\mathcal{N}$), a theoretical quantity that describes the number of potential breeding individuals within a given dispersal radius. Previously, empirical researchers interested in estimating $\mathcal{N}$ relied upon Rousset's (1997; 2000) method, but this method requires either subsetting individuals into pseudo-populations (arbitrarily discretizing a potentially continuous reality) and calculating F_ST between them or estimating pairwise F_ST between individuals. The latter approach introduces a significant amount of noise into patterns of isolation by distance, and, in our simulation study, demonstrates poor performance due to high variance in results.

Here, we have presented a model that jointly estimates $\mathcal{N}$ and long-term N_e using an individual-based approach (i.e. it does not require arbitrary discretization via the lumping of samples). Unlike Rousset's estimator, our method shows good performance across values of $\mathcal{N}$, and offers the additional benefit of generating an estimator of the long-term effective population size (N_e) that is insensitive to recent demographic shifts. The introduced model produced reasonable estimates of the theoretical expectations from simulated data and performed well on an empirical dataset of two-form bumblebees. Future work will aim to develop an R package for ease of use for researchers (though code for performing the presented model is available in the Github link).

Data Availability

Scripts for running the Wright–Malécot model and SLiM recipes are available at https://github.com/zachbhancock/WM_model (permanent doi:10.5281/zenodo.11508544).

Supplemental material available at GENETICS online.

Funding

This work was supported in part by computational resources and services provided by the Institute for Cyber-Enabled Research at Michigan State University and the Great Lakes High-Performance Computing Center at the University of Michigan. Research reported in this publication was supported by the National Institute of General Medical Sciences of the National Institutes of Health under Award Number R35GM137919 (awarded to G.S.B.). The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.