Estimating Effective Number of Breeders Under the Heterozygote‐Excess Equilibrium

ABSTRACT

Estimating the effective number of breeders per reproductive cycle or cohort ($N_b$) can contribute to understanding the viability of wild populations. However, previous $N_b$ estimators based on heterozygote‐excess assume that parental genotypes are in accordance with Hardy–Weinberg equilibrium (HWE), and can only be applied to diploids. When these criteria are not fulfilled, estimates of expected heterozygosity may thus be biased. We extended a previous $N_b$ estimator to autopolyploids and account for parental genotypes either in HWE or heterozygote‐excess equilibrium (HEE). Results include the following: the distributions of genotypes converge to HEE within nine generations (with a relative error of ${10}^{-3}$), the corrected estimators can asymptotically estimate $N_b$ without bias, and estimations for autopolyploids are slightly better than those for diploids. The relationships between ${\widehat{N}}_b$ and $N_b$ and the influence of variable effective breeding numbers across generations on ${\widehat{N}}_b$ were clarified. We find ${\widehat{N}}_b$ mainly reflects $N_b$ in the previous generation in diploids but is a weighted harmonic mean of $N_b$ in previous generations in polyploids. Our methods are implemented in a software package, polygene v1.7, which is freely available at https://github.com/huangkang1987/polygene.

1. Introduction

The effective population size, $N_e$, is an important parameter in population genetics in both theoretical and practical applications to animal and plant breeding, ecology, evolutionary biology and conservation biology (Falconer and Mackay 1996) and evolutionary theory (Charlesworth and Charlesworth 2010). Effective population size reveals the dynamics of genetic drift, selection, and gene flow and provides insights into the evolutionary history of species and populations, improving the understanding of complex patterns of speciation and adaptation. The effective number of breeders (Pudovkin et al. 1996) in one reproductive cycle ($N_b$), is a more informative metric to use than $N_e$ in some cases, such as for populations with clearly defined age structure or with overlapping generations. Because $N_b$ only requires data for a single breeding season rather than an entire life span, the term is easier to estimate than $N_e$ (Schwartz et al. 2007).

Several methods have been developed to indirectly estimate $N_e$ or $N_b$ from genetic data, which can be broadly classified into two main categories: temporal and single‐sample methods. Temporal methods use genetic data from two or more temporally spaced samples obtained from the same population (Nei and Tajima 1981; Waples 1989), which is difficult for long‐lived species. In contrast, the single‐sample methods utilise genetic data from a single sample collected at a specific point in time to estimate the $N_e$ or $N_b$ of a population at or near to that time. Single‐sample methods require less time, cost and sampling effort than temporal methods, and have therefore recently increased in popularity (Waples 2006; Tallmon et al. 2008; Waples and Do 2008, 2010; Wang 2009, 2016; Huang et al. 2022). For discrete generation models, estimates from single‐sample and temporal methods differ in the timeframes to which they apply. The single‐sample methods estimate the inbreeding $N_e$, whereas the temporal methods operate based on the variance $N_e$ (Waples 2005).

The heterozygote‐excess method is a single sample method and assumes the breeders are randomly sampled from a wider population. When there are few breeders, the binomial sampling error results in a heterozygote‐excess in the offspring (Pudovkin et al. 1996). There are several deficiencies in current heterozygote‐excess methods: (i) existing methods assume unrealistic conditions that parental genotypes are in accordance with HWE. However, HWE is not an equilibrium state in finite populations with a dioecious mating system and the distributions of genotypes will eventually converge to some new equilibrium states; (ii) the estimation of expected heterozygosity, and Selander's (1970) heterozygote‐excess index $D,$ are biased. Although Pudovkin et al. (2009) applied a Gaussian correction to obtain an unbiased expected heterozygosity, the influence of a correlation between allele copies was not considered; (iii) current methods are developed for diploids and cannot be directly applied to autopolyploids. However, polyploidy has occurred in every ancestral plant lineage (Otto 2007), and is frequently observed in extant species (Burow et al. 2001; Comai 2005). Polyploidy plays a significant role in the formation and evolution of plant species, and has received much recent attention in evolutionary ecology research (Selmecki et al. 2015; Avni et al. 2017; Burns et al. 2021; Lovell et al. 2021).

Here, we aim to (i) extend the definitions of both $N_b$ and $N_e$ to account for autopolyploids, and quantify the relationship between these two metrics; (ii) develop a $N_b$ estimator without the need to assume unrealistic conditions and correct any bias from expected heterozygosities; (iii) extend original and corrected estimators to the case of autopolyploids. We also use symbolic and numerical calculations to study the convergency of heterozygote‐excess index $D$ and to compare the statistical performance of different estimators. By comparing the results of Pudovkin et al.'s (1996) estimator ${\widehat{N}}_{b,\mathrm{Pu}}$ and our corrected estimator ${\widehat{N}}_{b,-\infty }$ (that is the estimator in which the genotypes of $-\infty$ generation are assumed to follow HWE) by polygene v1.7 in two empirical datasets, we find that our estimator is more stable than ${\widehat{N}}_{b,\mathrm{Pu}}.$

2. Methods

2.1. $N_e$ and $N_b$ for Autopolyploids

In the dioecious with random paring (DR) mating system (Weir and Hill 1980), in which, each offspring is reproduced by a new pairing, and there are $F$ breeding females and $M$ breeding males in the population for each generation, the effective number of breeders is defined as

$$N_b\stackrel{\scriptscriptstyle\mathrm{def}}{=}\frac{4\mathit{FM}}{F+M}.$$

This definition is compatible with autopolyploids, and the value of $N_b$ is close to the effective population size $N_e$ in diploids.

In polyploids, the effective population size has several definitions. Some studies define $N_e$ of an ideal polyploid population with $N$ individuals as $N$ (e.g., Pollak and Sabran 1996; Lynch et al. 2016); others define $N_e$ as $\mathit{Nv}/2$, with $v$ being the ploidy level (Hamilton 2009) because polyploids have a lower rate of genetic drift.

Here, we define $N_e$ as the size of an ideal population with a haplotype sampling (HS) mating system (Huang et al. 2022) that has the same rate of decrease in heterozygosity $\left(\lambda \right)$ at equilibrium as the real population. The HS mating system assumes that each individual is the product of randomly sampling $v$ haplotypes with replacement from the previous generation, and is reduced to the DR mating system in haploid and diploid Wright–Fisher populations. Because the rate of decrease in heterozygosity may vary among generations, we use this at equilibrium state in order to define $N_e$:

$$N_e\stackrel{\scriptscriptstyle\mathrm{def}}{=}\frac{1}{\left(1-\lambda \right)v}.$$

We use this definition of $N_e$ because: (i) the expression of $\lambda$ of a polyploid ideal population with HS mating system is simple (Equation (2)); (ii) $\lambda$ of a polyploid ideal population with $N$ individuals is identical to that of a haploid Wright–Fisher population with $\mathit{Nv}$ individuals, or a diploid Wright–Fisher populations with $\mathit{Nv}/2$ individuals; (iii) the value of $N_e$ of a real polyploid population is still close to $N_b$.

2.2. The Rate of Decrease in Heterozygosity

According to the above definition of $N_e$, we need to derive the decay of $H_O$ and $H_E$ under two mating systems. The recurrence of heterozygosity can be expressed by matrix multiplication:

$$\text{for}\mathrm{HS}:\left[\begin{array}{c}\text{E}\left(H_O^{\prime }\right)\\ \text{E}\left(H_E^{\prime }\right)\end{array}\right]=\left[\begin{array}{cc}\frac{v-1}{N_{e}v}& \frac{N_e-1}{N_e}\\ \frac{v-1}{N_{e}v}& \frac{N_e-1}{N_e}\end{array}\right]\left[\begin{array}{c}{H}_O\\ H_E\end{array}\right]={\mathbf{T}}_{\mathrm{HS}}\left[\begin{array}{c}{H}_O\\ H_E\end{array}\right],$$

$$\text{for}\mathrm{DR}:\left[\begin{array}{c}\text{E}\left(H_O^{\prime }\right)\\ \text{E}\left(H_E^{\prime }\right)\end{array}\right]=\left[\begin{array}{cc}\frac{v-2}{2v-2}& \frac{v}{2v-2}\\ \frac{v-1}{N_{b}v}& \frac{N_b-1}{N_b}\end{array}\right]\left[\begin{array}{c}{H}_O\\ H_E\end{array}\right]={\mathbf{T}}_{\mathrm{DR}}\left[\begin{array}{c}{H}_O\\ H_E\end{array}\right].$$

Therefore, given initial heterozygosities, the observed and expected heterozygosities at generation $t$ can be expressed by

$$\left[\begin{array}{c}\text{E}\left(H_{O,t},H_{O,0}|H_{E,0}\right)\\ \text{E}\left(H_{E,t},H_{O,0}|H_{E,0}\right)\end{array}\right]={\mathbf{T}}^t\left[\begin{array}{c}{H}_{O,0}\\ H_{E,0}\end{array}\right].$$ (1)

The derivations are provided in Appendix A.

By performing eigen‐value decomposition for the coefficient matrix $\mathbf{T}$, we obtain the values of $\lambda$ under different mating systems:

$${\lambda }_{\mathrm{HS}}=1-\frac{1}{v{N}_e},$$

$${\lambda }_{\mathrm{DR}}=\frac{c_1+c_2}{4N_b\left(v-1\right)},$$ (2)

where

$$c_1=\sqrt{8N_b+4-4\left(3N_b+2\right)v+{\left(N_b+2\right)}^2{v}^2},$$

$$c_2=2-2v+3N_{b}v-4N_b.$$

The corresponding eigen‐vectors for ${\lambda }_{\mathrm{HS}}$ and ${\lambda }_{\mathrm{DR}}$ are

$${\mathbf{U}}_{\mathrm{HS}}=\left[\begin{array}{c}1\\ 1\end{array}\right],{\mathbf{U}}_{\mathrm{DR}}=\left[\begin{array}{c}{D}_{\mathrm{eq}}+1\\ 1\end{array}\right],$$

respectively, where $D_{\mathrm{eq}}$ is the Selander's (1970) heterozygote‐excess index at heterozygote‐excess equilibrium, which is

$$D_{\mathrm{eq}}=\frac{4v-4}{N_b{v}^2+2v^2-6v+c_{1}v+4}.$$ (3)

The value of $N_e$ for a polyploid population with a DR mating system can be expressed as a function of $N_b$ by equating ${\lambda }_{\mathrm{HS}}={\lambda }_{\mathrm{DR}}$:

$$N_e=\frac{2v-2+N_{b}v+c_1}{2v}.$$

By expanding the zeroth‐order Taylor series for the above equation at $N_b\to \infty$, we find that $N_e$ is approximately equal to $N_b$ plus a constant:

$$N_e\approx N_b+\frac{2{\left(v-1\right)}^2}{v^2}.$$

The constant for $v\in \left\{2,4,6,8,10\right\}$ is $\left\{\frac{1}{2},\frac{9}{8},\frac{25}{18},\frac{49}{32},\frac{81}{50}\right\}$ and $N_e$ is at most $N_b+2$ at $v\to \infty$.

2.3. Parents in Accordance With HWE

Previous studies (Pudovkin et al. 1996, 2009; Huang et al. 2019) have derived and estimated $N_b$ assuming that parental genotypes are in accordance with HWE (abbreviated hereafter to ${\text{HWE}}_0$). The model assumes a dioecious ideal population, with each generation having an infinite number of individuals. In each generation, $M$ males and $F$ females are randomly sampled in order to produce the next generation, each individual generates an infinite number of gametes and these gametes randomly unite to generate the offspring.

According to Equation (1), we derive expected heterozygosities under ${\text{HWE}}_0$:

$$\text{E}\left(H_{O,1}|{\text{HWE}}_0\right)=\left[\mathrm{1\; 0}\right]{\mathbf{T}}_{\mathrm{DR}}{\left[\mathrm{1\; 1}\right]}^T{H}_{E,0},$$

$$\text{E}\left(H_{E,1}|{\text{HWE}}_0\right)=\left[\mathrm{0\; 1}\right]{\mathbf{T}}_{\mathrm{DR}}{\left[\mathrm{1\; 1}\right]}^T{H}_{E,0}.$$ (4)

Therefore, Selander's heterozygote‐excess index is

$$\text{E}\left(D_1|{\text{HWE}}_0\right)=\frac{\text{E}\left(H_{O,1}|{\text{HWE}}_0\right)}{\text{E}\left(H_{E,1}|{\text{HWE}}_0\right)}-1=\frac{1}{N_{b}v-1}.$$

By replacing $\text{E}\left(D_1|{\text{HWE}}_0\right)$ with $\widehat{D}$ and $N_b$ with ${\widehat{N}}_b$, we obtain an estimator of $N_b$ assuming ${\text{HWE}}_0$:

$${\widehat{N}}_{b,0}=\frac{1}{\widehat{D}v}+\frac{1}{v}.$$

The special cases for $2\le v\le 10$ are listed in Table 1.

TABLE 1.

The estimators of $N_b$.

$v$	${\widehat{N}}_{b,\mathrm{Pu}}$ a	${\widehat{N}}_{b,0}$ a	${\widehat{N}}_{b,-\infty }$ a
2	$\frac{1}{2\widehat{D}}+\frac{1}{2\left(1+\widehat{D}\right)}$	$\frac{1}{2\widehat{D}}+\frac{1}{2}$	$\frac{1}{2\widehat{D}}-\frac{1}{2}\widehat{D}$
4	$\frac{1}{4\widehat{D}}-\frac{1}{8}-\frac{3}{16}\widehat{D}+\frac{87}{32}{\widehat{D}}^2$	$\frac{1}{4\widehat{D}}+\frac{1}{4}$	$\frac{3}{8\widehat{D}}-\frac{3}{4}-\frac{9}{8}\widehat{D}$
6	$\frac{1}{6\widehat{D}}-\frac{7}{18}+\frac{55}{54}\widehat{D}+\frac{1085}{162}{\widehat{D}}^2$	$\frac{1}{6\widehat{D}}+\frac{1}{6}$	$\frac{5}{18\widehat{D}}-\frac{10}{9}-\frac{25}{18}\widehat{D}$
8	$\frac{1}{8\widehat{D}}-\frac{17}{32}+\frac{329}{128}\widehat{D}+\frac{5887}{512}{\widehat{D}}^2$	$\frac{1}{8\widehat{D}}+\frac{1}{8}$	$\frac{7}{32\widehat{D}}-\frac{21}{16}-\frac{49}{32}\widehat{D}$
10	$\frac{1}{10\widehat{D}}-\frac{31}{50}+\frac{1071}{250}\widehat{D}+\frac{20889}{1250}{\widehat{D}}^2$	$\frac{1}{10\widehat{D}}+\frac{1}{10}$	$\frac{9}{50\widehat{D}}-\frac{36}{25}-\frac{81}{50}\widehat{D}$

^a ${\widehat{N}}_{b,\mathrm{Pu}}$ denotes the original Pudovkin et al. (1996) $N_b$ estimator. ${\widehat{N}}_{b,0}$ and ${\widehat{N}}_{b,-\infty }$ are corrected estimators assuming ${\text{HWE}}_0$ and ${\text{HWE}}_{-\infty }$, respectively. The expressions of ${\widehat{N}}_{b,\mathrm{Pu}}$ (with $v>2$), were expanded to second‐order Taylor series at $\widehat{D}=0$.

2.4. Generation $t$ in Accordance With HWE

Equation (4) can be extended to the scenario that the genotypes of generation $t$ are in accordance with HWE (abbreviated hereafter to ${\mathrm{HWE}}_t$):

$$\text{E}\left(H_{O,1}|{\text{HWE}}_t\right)=\left[\mathrm{1\; 0}\right]{\mathbf{T}}_{\mathrm{DR}}^{1-t}{\left[\mathrm{1\; 1}\right]}^T{H}_{E,t},$$

$$\text{E}\left(H_{E,1}|{\text{HWE}}_t\right)=\left[\mathrm{0\; 1}\right]{\mathbf{T}}_{\mathrm{DR}}^{1-t}{\left[\mathrm{1\; 1}\right]}^T{H}_{E,t}.$$

In this case, Selander's heterozygote‐excess index is

$$\text{E}\left(D_1|{\text{HWE}}_t\right)=\frac{4v-4}{2v^2-6v+4+N_b{v}^2+c_{1}v+2c_{1}v⁄\left[{\left(\frac{c_2+c_1}{c_2-c_1}\right)}^{1-t}-1\right]}.$$ (5)

For $t\to -\infty$, $\mathrm{E}\left(D_1|{\mathrm{HWE}}_t\right)$ converges to $D_{\mathrm{eq}}$ when $N_b\ge 2$, thus $\mathrm{E}\left(D_1|{\mathrm{HWE}}_{-\infty }\right)=D_{\mathrm{eq}}$ and we can estimate $N_b$ under the assumption of ${\mathrm{HWE}}_{-\infty }$:

$${\widehat{N}}_{b,-\infty }=\frac{2\left(v-1\right)}{v^2\widehat{D}}-\left(\frac{4}{v^2}-\frac{6}{v}+2\right)-\left(\frac{2}{v^2}-\frac{4}{v}+2\right)\widehat{D}.$$ (6)

The special cases for $2\le v\le 10$ are listed in Table 1, with the computer derivations of ${\widehat{N}}_{b,0}$ and ${\widehat{N}}_{b,-\infty }$ provided in File S1.

2.5. Pudovkin's (1996) Estimator

Pudovkin et al. (1996) original estimator can only be used for diploids and also assumes ${\text{HWE}}_0$. Here, we use this method to expand the estimator to account for polyploids. The observed heterozygosity in generation 1 can be expressed by:

$$H_{O,1}={\left(\genfrac{}{}{0pt}{}{v}{2}\right)}^{-1}\left\{\left(\genfrac{}{}{0pt}{}{v/2}{2}\right)\left(H_{F,0}+H_{M,0}\right)+{\left(\frac{v}{2}\right)}^2\left(p_{F,0}+p_{M,0}-2p_{F,0}{p}_{M,0}\right)\right\}$$

$$={\left(\genfrac{}{}{0pt}{}{v}{2}\right)}^{-1}\left\{\left(\genfrac{}{}{0pt}{}{v/2}{2}\right)\left(H_{F,0}+H_{M,0}\right)+{\left(\frac{v}{2}\right)}^2\left(H_{E,1}+\frac{1}{2}{d}_0^2\right)\right\},$$ (7)

here $p_{F,0}$ and $p_{M,0}$ denote the allele frequency of the breeding females and males in generation $0$, respectively, and $d_0$ is equal to $p_{M,0}-p_{F,0}$. By taking the expectation of both sides of the above equation (Appendix B), and using the approximation

$$\text{E}\left(H_{E,1}\right)\approx \lambda H_{E,0}.$$

The expectation of $H_{O,1}$ is approximately

Therefore, the expectation of Selander's heterozygote‐excess index is

$$\text{E}\left(D|{\text{HWE}}_0\right)=\frac{\text{E}\left(H_{O,1}|{\text{HWE}}_0\right)}{\text{E}\left(H_{E,1}\right)}-1\approx \frac{1-N_b\left(v-2\right)\left(\lambda -1\right)}{2N_b\left(v-1\right)\lambda }.$$

Substituting the expression of $\lambda$ (Equation (2)) into the above equation, and using $\text{E}\left(D\right)$ as known and $N_b$ as the unknown, the expression of $N_b$ as a function of $\text{E}\left(D\right)$ can be solved. Then replacing $N_b$ with ${\widehat{N}}_b$ and $\text{E}\left(D\right)$ with $\widehat{D}$ (which can be calculated from the samples), we obtained the estimator of $N_b$.

For $v=2$, the result is identical to Pudovkin et al. (1996):

$${\widehat{N}}_{b,\mathrm{Pu}}=\frac{1}{2\widehat{D}}+\frac{1}{2\left(1+\widehat{D}\right)}.$$

For $v>2$, the expression is complex, for example, in autotetraploids,

$${\widehat{N}}_{b,\mathrm{Pu}}=\frac{1-8\widehat{D}-15{\widehat{D}}^2+\sqrt{{\left(1+3\widehat{D}\right)}^2\left(1+2\widehat{D}+25{\widehat{D}}^2\right)}}{8\widehat{D}-12{\widehat{D}}^2}.$$

For simplicity, we expand the second‐order Taylor series at $\widehat{D}=0$, and list the estimators of $N_b$ for $2\le v\le 10$ in Table 1. The computer derivations of ${\widehat{N}}_{b,\mathrm{Pu}}$ are provided in File S2.

2.6. Unbiased Estimation of $H_E$

The ‘traditional’ estimator of $H_E$ is ${\widehat{H}}_E=1-{\sum }_k{\widehat{p}}_k^2$, which is the probability of sampling two non‐identical‐by‐state (non‐IBS) allele copies with replacement. However, the pairs for the same allele copy are always IBS, which results in an underestimation of $H_E$. We use $B_{\mathit{kia}}$ to derive the expectation of ${\widehat{H}}_E$, and $B_{\mathit{kia}}$ is one if the a‐th allele copy in individual $i$ is allele $k$, otherwise zero. Therefore,

$${\widehat{H}}_E=1-{\sum }_k{\widehat{p}}_k^2=1-\frac{1}{n^2{v}^2}{\sum }_k{\sum }_{i,a,i^{\prime },a^{\prime }}{B}_{kia}{B}_{k{i}^{\prime }{a}^{\prime }}$$

$$=1-\frac{1}{n^2{v}^2}\left({\sum }_{k,i,a}{B}_{kia}+{\sum }_{k,i,a\ne a^{\prime }}{B}_{kia}{B}_{ki{a}^{\prime }}+{\sum }_{\begin{array}{c}i\ne i^{\prime }\\ k,a,a^{\prime }\end{array}}{B}_{kia}{B}_{k{i}^{\prime }{a}^{\prime }}\right).$$

Its expectation is

$$\text{E}\left({\widehat{H}}_E\right)=1-\frac{1}{nv}\left[{\sum }_k\text{E}\left(B_{kia}\right)+\left(v-1\right){\sum }_k\text{E}\left(B_{kia}{B}_{ki{a}^{\prime }}\right)+\right.\left.\left(n-1\right)v{\sum }_k\text{E}\left(B_{kia}{B}_{k{i}^{\prime }{a}^{\prime }}\right)\right],$$

where $n$ is the sample size, $B_{\mathit{kia}}{B}_{\mathit{kia}\prime }$ and $B_{kia}{B}_{k{i}^{\prime }{a}^{\prime }}$ are the products of indicators for allele pairs within the individual and between individuals, respectively, and it can be inferred that $\text{E}\left(B_{kia}\right)=p_k$, $\text{E}\left(B_{kia}{B}_{ki{a}^{\prime }}\right)=\mathrm{Cov}\left(B_{kia},B_{ki{a}^{\prime }}\right)+p_k^2$, $\text{E}\left(B_{kia}{B}_{k{i}^{\prime }{a}^{\prime }}\right)=\mathrm{Cov}\left(B_{kia},B_{k{i}^{\prime }{a}^{\prime }}\right)+p_k^2$.

The method of Pudovkin et al. (1996) ignores the correlation between alleles, and assumes ${\sum }_k\text{E}\left(B_{kia}{B}_{ki{a}^{\prime }}\right)={\sum }_k\text{E}\left(B_{kia}{B}_{k{i}^{\prime }{a}^{\prime }}\right)=1-H_E$, then the above expectation can be simplified as

$$\text{E}\left({\widehat{H}}_E\right)=1-\frac{1}{nv}\left[1+\left(nv-1\right)\left(1-H_E\right)\right],$$

Then, by substituting $\text{E}\left({\widehat{H}}_E\right)$ with ${\widehat{H}}_E$ and $H_E$ with ${\tilde{H}}_E$, the corrected $H_E$ estimator is given by

$${\tilde{H}}_E=\frac{\mathit{nv}}{\mathit{nv}-1}{\widehat{H}}_E.$$

We consider the influence of the correlation between alleles and only use the allele pairs between individuals to estimate $H_E$. In this case, $\text{E}\left(B_{kia}\right)=p_k$, ${\sum }_k\mathrm{E}\left(B_{kia}{B}_{ki{a}^{\prime }}\right)=1-H_O$, ${\sum }_k\text{E}\left(B_{kia}{B}_{k{i}^{\prime }{a}^{\prime }}\right)=1-H_E$. Then the expectation of $H_E$ is revised as

$$\text{E}\left({\widehat{H}}_E\right)=1-\frac{1}{nv}\left[1+\left(v-1\right)\left(1-H_O\right)+\left(n-1\right)v\left(1-H_E\right)\right].$$

Then substituting $\text{E}\left({\widehat{H}}_E\right)$ with ${\widehat{H}}_E$ and $H_E$ with ${\tilde{H}}_E^{\prime }$, the corrected $H_E$ estimator is given by

$${\tilde{H}}_E^{\prime }=\frac{{\widehat{H}}_E\mathit{nv}-{\widehat{H}}_O\left(v-1\right)}{\mathit{nv}-v}.$$

We use $\widehat{D}$ and ${\widehat{D}}^{\prime }$ to distinguish the two heterozygote‐excess indices calculated from different $H_E$ estimator:

$$\widehat{D}=\frac{{\widehat{H}}_O}{{\tilde{H}}_E}-1,{\widehat{D}}^{\prime }=\frac{{\widehat{H}}_O}{{\tilde{H}}_E^{\prime }}-1.$$ (8)

2.7. Simulation Method

Monte‐Carlo simulations were conducted to investigate the changes of $D$ and $N_b$ across generations and assess the statistical performance of various $N_b$ estimators. A population with DR mating system was simulated, comprising an initial parental generation of $M$ breeding males and $F$ breeding females. Parental genotypes at $L$ unlinked loci were randomly drawn according to either HWE or HEE (i.e., ${\text{HWE}}_0$ or ${\text{HEE}}_0$). To exclude the influence of locus polymorphism on the estimation, we generated offspring data with $\text{E}\left(H_O\right)=0.32$ at diallelic markers (Appendix C). We used $H_O$ to evaluate locus polymorphism, because $H_E$ is sensitive to deviations from the HWE (e.g., Wahlund effect).

The parents then mate randomly to produce the offspring. We simulated $R$ populations and calculated the harmonic mean, the root‐mean‐squared‐error (RMSE) of ${\widehat{N}}_b$ and 1/${\widehat{N}}_b$ for each parameter combination. For each population, $n$ individuals were sampled from the offspring generation and were genotyped at all $L$ loci. The values of $\widehat{D}$ and ${\widehat{D}}^{\prime }$ were calculated by summing the heterozygosites across loci. For example,

$$\widehat{D}=\frac{{\sum }_l{\widehat{H}}_{O,l}}{{\sum }_l{\tilde{H}}_{E,l}}-1,$$

where ${\widehat{H}}_{O,l}$ and ${\tilde{H}}_{E,l}$ denotes the estimated observed and expected heterozygosities at locus $l$.

When $\widehat{D}<0$, ${\widehat{N}}_b$ is negative. The main reason for the negative value of the $N_b$ estimate is that for the heterozygote excess method, when there is inbreeding or substructure in the population, there will be insufficient heterozygotes. Thus, negative $D$ and $N_b$ estimates will be obtained. Moreover, when $\widehat{D}$ is close to zero, ${\widehat{N}}_b$ can be very large (e.g., ${10}^{100}$). Although negative $N_b$ estimates present conceptual challenges, these values still reflect significant information about genetic drift, thus the recommended approach involves calculating the harmonic mean across all estimates including negative ones as the measure of central tendency. Moreover, systematically ignoring negative or infinite estimates leads to estimation bias of $\widehat{N}$ (Waples 2024, 2025). We carried out five simulations.

Simulation 1 compared the changes of theoretical and estimated $D$ across generations to validate our derivations. The founders were under HWE and the population was reproduced for 12 generations, each generation had exactly $N_b$ individuals with a sex ratio of 0.5. For each generation, all individuals were sampled to estimate $D$ using Equation (8), while the theoretical $D$ was calculated using Equation (5). Each allele copy in the founder generation is unique so as to offset genetic drift. The key parameters and their settings were as follows: the ploidy level $v$ was set to $\left\{2,4,6\right\}$; the effective population size $N_b$ to $\left\{2,6,10\right\}$; the number of alleles per locus K (here set as $N_{b}v$) to the corresponding products; the number of loci L to 2000; the sample size to $n=N_b$; and the number of simulation replicates $R$ to 2000. $\widehat{D}$ and ${\widehat{D}}^{\prime }$ were calculated by summing the heterozygosities across 2000 populations and 2000 loci, for example, the value of $\widehat{D}$ at generation $t$ was calculated by

$${\widehat{D}}_t=\frac{{\sum }_{\mathit{ptl}}{\widehat{H}}_{O,\mathit{ptl}}}{{\sum }_{\mathit{ptl}}{\tilde{H}}_{E,\mathit{ptl}}}-1,$$

where ${\widehat{H}}_{O,\mathit{ptl}}$ and ${\tilde{H}}_{E,\mathit{ptl}}$ denote the estimated observed and expected heterozygosities of population $p$ in generation $t$ at locus $l$. The Jackknife method was not applied for this simulation. Further, to study the convergence speed of $D$, we also use symbolic calculation to obtain the number of generations $N_{\mathrm{eq}}$ required to reach a predefined relative error.

Simulation 2 studied the behaviour of ${\widehat{N}}_{b,-\infty }$ under population expansion and decline. The $N_b$ before generation 1 was 2 (or 1024), hence we generated genotypes for generation 1 under HEE. After generation 1, the $N_b$ value of each subsequent generation is twice or half that of the previous generation. Each generation had 1024 individuals and generation 10 is the final generation. The parameters were $v\in \left\{2,4,6\right\}$, $N_{b,t}=2^{t+1}$ (or $2^{10-t}$), $K=2$, $L=1000$, $n=1024$ and $R=10000$. The Jackknife method was not applied for this simulation.

Simulation 3 studied the distribution of ${\widehat{N}}_b$ to show asymptotically that ${\widehat{N}}_b$ is unbiased, where $v\in \left\{2,4,6\right\}$, $N_b\in \left\{10,30,100\right\}$, $K=2$, $L=2000$, $n=1000$ and $R=2000$.

Simulation 4 evaluated the harmonic mean of ${\widehat{N}}_b$, the RMSE of ${\widehat{N}}_b$ and $1/{\widehat{N}}_b$ as the number of loci $L$ changes, where $v\in \left\{2,4,6\right\}$, $N_b\in \left\{\mathrm{10,30,100}\right\}$, $n=500$, $L\in \left\{200,400,\cdots ,2000\right\}$ and $R=\mathrm{10,000}$.

Simulation 5 evaluated the harmonic mean of ${\widehat{N}}_b$, the RMSE of ${\widehat{N}}_b$ and $1/{\widehat{N}}_b$ as the sample size $n$ changes, where $v\in \left\{2,4,6\right\}$, $N_b\in \left\{\mathrm{10,30,100}\right\}$, $L=1000$, $n=\left\{\mathrm{100,200},\cdots ,1000\right\}$ and $R=\mathrm{10,000}$.

Two additional estimators implemented in polygene v1.7, Nomura's (2008) estimator ${\widehat{N}}_{b,\mathrm{No}}$ and Huang et al.'s (2022) estimator ${\widehat{N}}_{b,\mathrm{Hu}}$ were also compared in Simulations 4 and 5. However, they operate under different assumptions and may perform suboptimally when their assumptions are violated. To ensure a fair comparison, we generated offspring data under their respective assumptions: ${\widehat{N}}_{b,\mathrm{Pu}}$ and ${\widehat{N}}_{b,0}$ assume ${\text{HWE}}_0$, ${\widehat{N}}_{b,-\infty }$ assumes ${\text{HEE}}_0$, ${\widehat{N}}_{b,\mathrm{No}}$ does not rely on assumptions regarding the distribution of genotypes of parental generation and utilises ${\text{HEE}}_0$, while ${\widehat{N}}_{b,\mathrm{Hu}}$ assumes equilibrium for the squared LD coefficient, $r_{\mathrm{\Delta }}^2$. This equilibrium was simulated by generating a founder generation under HEE and reproduced over approximately 6 generations using the equation $-4.21/\log \left(1-c\right)\approx 6$.

The C++ source code is provided in File S3, the simulation programme, parameter files and batch files are provided in File S4, and the results of processing scripts are provided in File S5.

2.8. The Number of Loci Required

We also derived the number of loci required to reach a predefined coefficient of variation (CV) under ${\text{HEE}}_0$. We used CV instead of variance or RMSE, because these statistics are influenced by $\mathrm{E}\left({\widehat{N}}_b\right)$, and a smaller $\mathrm{E}\left({\widehat{N}}_b\right)$ usually has a smaller $\mathrm{Var}\left({\widehat{N}}_b\right)$.

We first derived the moments of heterozygosities of given parental allele frequencies $p_f$ and $p_m$ (as detailed in Appendix D, Tables S1 and S2, File S6), then derived the variance of ${\widehat{N}}_{b,-\infty }$ (as detailed in Appendix E, File S7). These procedures are briefly described as follows: (i) using the model described in Appendix C, we derived the moments of heterozygosities of given allele frequency $p$ and $\alpha$, where $\alpha$ is used to generate a heterozygote‐excess sample; (ii) by assuming a uniform a priori allele frequency distribution (Kimura 1954), $p$ is eliminated from the conditions of these moments; (iii) according to Equation (6), the variance of ${\widehat{N}}_{b,-\infty }$ at $L$ loci $\mathrm{Var}\left({\widehat{N}}_{b,-\infty }|L\right)$ is approximately $\frac{4{\left(v-1\right)}^2}{v^4}·\mathrm{Var}\left(1/{\widehat{D}}^{\prime }|L\right)$, where $\mathrm{Var}\left(1/{\widehat{D}}^{\prime }|L\right)$ is approximated by the variance of ratios (Stuart and Ord 1998).

By solving the equation $\mathrm{CV}=\sqrt{\mathrm{Var}\left({\widehat{N}}_{b,-\infty }|L\right)}/N_b$ (see File S8), we calculated the number of loci required to reach a predefined $\mathrm{CV}\in \left\{10\%,15\%,20\%\right\}$ for a population with $N_b\in \left\{10,30,100,300,1000,3000\right\}$ and $n\in \left\{50,100,\cdots ,3000\right\}$.

2.9. Empirical Data

Kamath et al. (2015) analysed 729 Yellowstone grizzly bears ( Ursus arctos ) from an isolated and well‐documented population in the Greater Yellowstone Ecosystem. These bears, born between 1962 and 2010, were sampled and genotyped at 20 microsatellite loci. Herein we analyse the genotype data of the sampled individuals born in the periods 1983–2008. We used the sampling method in Kamath et al. (2015), that is, to ensure sufficient sample sizes and account for intermittent breeding, bear samples were grouped into 3‐year overlapping cohorts based on birth year from 1983 to 2008. We calculated the estimates and 95% confidence intervals of ${\widehat{N}}_{b,\mathrm{Pu}}$ and ${\widehat{N}}_{b,-\infty }$ by polygene v1.7 in this Yellowstone grizzly bear population. The 95% confidence intervals are estimated by the Jackknife method.

We also used another empirical genotypic dataset to evaluate the performance of ${\widehat{N}}_{b,\mathrm{Pu}}$ and ${\widehat{N}}_{b,-\infty }$ in handling SNPs. The brook trout ( Salvelinus fontinalis ) dataset (Anne‐Laure et al. 2020) comprised genotype data from 1416 individuals across 50 populations, analysed at 14,779 high‐quality filtered SNPs.

3. Results

Simulations were performed using different combinations of the following population parameters: generation ($t$), ploidy level ($v$), number of loci ($L$), number of alleles per locus ($K$) and other parameters.

3.1. Simulation 1

The results of simulation 1 are shown in Figure 1. It can be found that $D$, $\widehat{D}$ and ${\widehat{D}}^{\prime }$ converge to $D_{\mathrm{eq}}$. $D_{\mathrm{eq}}$ is smaller in higher $N_b$ and ploidy, which is approximately $\frac{2\left(v-1\right)}{N_b{v}^2}$. ${\widehat{D}}^{\prime }$ seems to be an unbiased estimator of $D$, while $\widehat{D}$ is downwardly biased. The absolute bias of $\widehat{D}$ decreases as $N_b$ increases and can be reduced to a negligible level for $N_b=10$, but increases as $v$ increases. This shows that the amplitude of the curves of $D$ decreases as the number of generations increase (damped vibration) around $D_{\mathrm{eq}}$ at $v=2$ or $N_b=2$.

FIGURE 1.

The values of $D$, $\widehat{D}$ and ${\widehat{D}}^{\prime }$ as a function of the number of generations reproduced. The subfigures in different columns are for different levels of ploidy. Empty and filled dots denote $\widehat{D}$ and ${\widehat{D}}^{\prime }$, respectively, and the solid curves denote $D$. Different colours denote different $N_b$, where red, blue and magenta objects denote $N_b$ = 2, 6 and 10, respectively.

The numbers of generations $N_{\mathrm{eq}}$ required to reach a predefined relative error are shown in Table 2. For polyploids, $N_{\mathrm{eq}}$ increases as $N_b$ increases, whilst in diploids, $N_{\mathrm{eq}}$ decreases as $N_b$ increases.

TABLE 2.

The number of generations to allow the relative error $\frac{\left|D-D_{\mathrm{eq}}\right|}{D_{\mathrm{eq}}}<{10}^{-2},{10}^{-4}$ and ${10}^{-6}$.

$N_b$	Diploids	Tetraploids	Hexaploids	Octoploids	Decaploids
2	5/10/15	3/4/6 a	2/4/5	2/3/5	2/3/4
5	3/5/7	3/6/9	4/7/10	4/7/11	4/7/11
10	2/4/5	4/7/11	5/9/13	5/9/13	5/9/14
100	1/2/3	5/9/13	5/10/15	6/11/16	6/12/17

^a Cell values (e.g., 3/4/6) indicate the number of generations required for $N_b=2$ and tetraploids to achieve relative error thresholds of heterozygote excess index $\left(D\right)$ below ${10}^{-2},{10}^{-4}$ and ${10}^{-6}$, respectively.

3.2. Simulation 2

The $N_b$ for the preceding generation, the mean ${\widehat{N}}_{b,-\infty }$, the mean ${\widehat{N}}_{b,-\infty }$ predicted by Equation (9) and by Equations (10) and (11) are shown in Figure S7. It can be found that: (i) ${\widehat{N}}_{b,-\infty }$ does not estimate the $N_b$ of the previous generation, but rather estimates some mean $N_b$ across several preceding generations; (ii) in diploids, the $N_b$ of the prior generation contributes more to the mean $N_b$, whereas its contribution is relatively less significant in polyploids; (iii) Equation (9) can precisely predict ${\widehat{N}}_{b,-\infty }$ under any conditions, while Equations (10) and (11) exhibit decreased precision, particularly at lower $N_b$. This is because Equations (10) and (11) are first‐order Taylor series expansions at $D_0=0$ and $D_{\mathrm{eq},0}=0$.

3.3. Simulation 3

We used violin plots to show the distribution of ${\widehat{N}}_b$ (Figure 2). We found that ${\widehat{N}}_{b,\mathrm{Pu}}$ is nearly unbiased when its core assumptions are met (diploids and ${\text{HWE}}_0$), but is downwardly biased for polyploids (either ${\text{HWE}}_0$ or ${\text{HEE}}_0$) and upwardly biased for diploids with ${\text{HEE}}_0$.

FIGURE 2.

Violin plots for ${\widehat{N}}_b$. The subfigures in different columns show cases for ${\text{HWE}}_0$ or ${\text{HEE}}_0$, and the subfigures in different rows show the results for different values of $N_b$. The blue, red and black symbols represent the probability density functions of ${\widehat{N}}_{b,\mathrm{Pu}}$, ${\widehat{N}}_{b,0}$ and ${\widehat{N}}_{b,-\infty }$, respectively. The grey bars denote the mean estimate and the error bars denote the quartiles. Di, Tetra and Hexa denote results for diploids, autotetraploids and autohexaploids, respectively.

${\widehat{N}}_{b,0}$ and ${\widehat{N}}_{b,-\infty }$ are nearly unbiased under their own assumptions, but become biased when these assumptions are not met. Under ${\text{HEE}}_0$, ${\widehat{N}}_{b,0}$ becomes downwardly biased in polyploids and upwardly biased in diploids. Under ${\text{HWE}}_0$, ${\widehat{N}}_{b,-\infty }$ becomes upwardly biased in polyploids and downwardly biased in diploids.

For larger values of $N_b$, although some estimators are still unbiased, variance tends to increase.

3.4. Simulation 4

The harmonic means of the estimates, along with the RMSE of the five $N_b$ estimators and that of their reciprocals, are shown as functions of the number of loci $L$ in Figure 3, Figures S1 and S2, respectively. The proportions of positive estimates of ${\widehat{N}}_{b,\mathrm{Pu}}$, ${\widehat{N}}_{b,0}$ and ${\widehat{N}}_{b,-\infty }$ under different numbers of loci are present in Figure S3.

FIGURE 3.

The harmonic means of the five ${\widehat{N}}_b$ as functions of the number of loci $L$ under their own assumptions. The different columns show the harmonic means for different levels of ploidy. The different rows show the harmonic means for different values of $N_b$. The blue empty dot, red filled dot, black line, magenta line and green line are the estimators ${\widehat{N}}_{b,\mathrm{Pu}}$, ${\widehat{N}}_{b,0}$, ${\widehat{N}}_{b,-\infty }$, ${\widehat{N}}_{b,\mathrm{Hu}}$ and ${\widehat{N}}_{b,\mathrm{No}}$, respectively. The black dotted line denotes the baseline which equals to the true $N_b$. Di, Tetra and Hexa denote results for diploids, autotetraploids and autohexaploids, respectively.

We found that ${\widehat{N}}_{b,\mathrm{Pu}}$, ${\widehat{N}}_{b,0}$ and ${\widehat{N}}_{b,-\infty }$ are asymptotically unbiased under their respective assumptions, but as $N_b$ increases, for example when $N_b=100$, achieving asymptotic unbiasedness in the estimator becomes progressively more challenging. The harmonic mean of ${\widehat{N}}_{b,\mathrm{No}}$ quickly converges with more loci and closely matches the true value for larger $N_b$. The harmonic mean of ${\widehat{N}}_{b,\mathrm{Hu}}$ exhibits little to no change with the number of loci, and increases with the level of ploidy. For all five estimators, the reciprocals yield a dramatically reduced RMSE compared to the raw estimates.

The ploidy level slightly affects the harmonic means, the RMSE of $N_b$ estimators and the RMSE of the reciprocals of these estimators: (i) for diploids, the harmonic means of these estimators are closer to the true value, in comparison, their RMSE values (and those of their reciprocals) are slightly lower in diploids than in polyploids; (ii) diploids require fewer loci than polyploids for these estimators to approach unbiasedness. In contrast, $N_b$ has a more pronounced influence: (i) as $N_b$ increases, more loci are required for these estimators to become nearly unbiased and achieving asymptotic unbiasedness becomes increasingly difficult; (ii) the RMSE of ${\widehat{N}}_{b,-\infty }$ rises sharply from 0.3 to about 60 when the loci number $L>1500$ as $N_b$ increases from 10 to 100, and the situation deteriorates further when fewer loci are used. In all cases, the RMSE is significantly lower for the reciprocals of these estimators compared to the estimators themselves.

3.5. Simulation 5

The harmonic means of the estimates, along with the RMSE of the five $N_b$ estimators and that of their reciprocals, are shown as functions of sample size in Figure 4, Figures S4 and S5, respectively. The proportions of positive estimates of ${\widehat{N}}_{b,\mathrm{Pu}}$, ${\widehat{N}}_{b,0}$ and ${\widehat{N}}_{b,-\infty }$ across a range of sample sizes are present in Figure S6. In accordance with the current parameter settings, the harmonic means of the four estimators (${\widehat{N}}_{b,\mathrm{Pu}}$, ${\widehat{N}}_{b,0}$, ${\widehat{N}}_{b,\mathrm{Hu}}$ and ${\widehat{N}}_{b,-\infty }$) exhibit highly similar response patterns to variations in both sample size and number of loci, with largely consistent convergence behaviour and asymptotic values.

FIGURE 4.

The harmonic means of the five ${\widehat{N}}_b$ as functions of the number of samples $n$ under their own assumptions. The different columns show the harmonic means for different levels of ploidy. The different rows show the harmonic means for different values of $N_b$. The blue empty dot, red filled dot, black line, magenta line and green line are the estimators ${\widehat{N}}_{b,\mathrm{Pu}}$, ${\widehat{N}}_{b,0}$, ${\widehat{N}}_{b,-\infty }$, ${\widehat{N}}_{b,\mathrm{Hu}}$ and ${\widehat{N}}_{b,\mathrm{No}}$, respectively. The black dotted line denotes the baseline which equals to the true $N_b$. Di, Tetra and Hexa denote results for diploids, autotetraploids and autohexaploids, respectively.

However, for the estimator ${\widehat{N}}_{b,\mathrm{No}}$, a pronounced discrepancy was observed in the response pattern of the harmonic mean to changes in sample size versus changes in the number of loci. The harmonic mean of ${\widehat{N}}_{b,\mathrm{No}}$ shows minimal variation with sample size and increases with the level of ploidy for $N_b\le 30$. When $N_b=100$ with small sample size, the harmonic mean of ${\widehat{N}}_{b,\mathrm{No}}$ remains close to the true value. However, it decreases rapidly as the sample size increases.

3.6. The Number of Loci Required

The numbers of loci $L_{\mathrm{CV}}$ required to reach a predefined CV as a function of $n$ are shown in Figure 5. We found that $L_{\mathrm{CV}}$ decreases as $n$, $v$ and CV increase, while $L_{\mathrm{CV}}$ increases as $N_b$ increases. A polyploid sample with a large $n$ and small $N_b$ will result in the most precision in estimating $N_b$. When $N_b\ge 100$, the curves of $L_{\mathrm{CV}}$ for different values of $N_b$ are isometrically distributed.

FIGURE 5.

The number of loci required to reach a predefined coefficient of variation. The subfigures in different columns show the results from different predefined values of CV. The subfigures in different rows show the results for different levels of ploidy. Di, Tetra and Hexa denote results for diploids, autotetraploids and autohexaploids, respectively. In each subfigure, the different curves indicate various values of $N_b$.

3.7. Empirical Data

Tables S4 and S5 present the effective breeder number estimates ${\widehat{N}}_{b,\mathrm{Pu}}$ and ${\widehat{N}}_{b,-\infty }$ (with 95% CIs) for grizzly bears ( Ursus arctos ; SSR data) and brook trout ( Salvelinus fontinalis ; SNP data), respectively. For grizzly bears, the harmonic means calculated across all 3‐years cohorts are 185.49 for ${\widehat{N}}_{b,\mathrm{Pu}}$ and 149.05 for ${\widehat{N}}_{b,-\infty }$. While in the case of brook trout data, the harmonic means across populations yield 87.92 for ${\widehat{N}}_{b,\mathrm{Pu}}$ and 70.65 for ${\widehat{N}}_{b,-\infty }$.

Our analysis reveals three key patterns: (i) in both datasets, for the vast majority of populations or years with positive estimates, ${\widehat{N}}_{b,-\infty }$ produces consistently lower values than ${\widehat{N}}_{b,\mathrm{Pu}}$, despite following a similar overall trend, and this systematic reduction is further supported by the final harmonic mean results; (ii) ${\widehat{N}}_{b,-\infty }$ systematically exhibits lower variance yet wider 95% confidence intervals relative to ${\widehat{N}}_{b,\mathrm{Pu}}$; and (iii) the rates of positive values vary across different marker datasets (SSR: 50% vs. SNP: 62%).

4. Discussion

4.1. Dynamics of $D$

We found from Equation (5) that the damped vibration of $D$ at $v=2$ or $N_b=2$ in Figure 1 is due to the sign of $\frac{c_2+c_1}{c_2-c_1}$. If negative, the curve of $D$ exhibits damped vibration. By solving the inequality $\frac{c_2+c_1}{c_2-c_1}<0$, we found that it holds in two scenarios: (i) $v=2$ and $N_b>0$, or (ii) $v>2$ and $0<N_b<\frac{2v-3}{v-2}$. This is consistent with our simulation (Figure 1).

According to numerical simulations (data not shown), we found $D$ quickly converges to $D_{\mathrm{eq}}$, and reaches a relative error of $\left\{{10}^{-2},{10}^{-3},{10}^{-4},{10}^{-5},{10}^{-6},{10}^{-7}\right\}$ with at most $\left\{6,9,\mathrm{12,14,17,20}\right\}$ generations for $2\le v\le 10$. This suggests that as $N_b$ changes, for example, due to population expansion or a bottleneck effect, $D$ will quickly converge to a new $D_{\mathrm{eq}}$ within several generations. Therefore, the values of $D$ are mainly influenced by recent values of $N_b$, implying that our $N_b$ estimator reflects recent values of $N_b$.

4.2. The Relationship Between ${\widehat{N}}_b$ and $N_b$

For a given sample of $n$ individuals, the pedigree has been determined. In this case, the expectation of ${\widehat{H}}_O$ is still equal to $\frac{v-2}{2v-2}{H}_{O,0}+\frac{v}{2v-2}{H}_{E,0}$, because the ratio of the probabilities that two allele copies within an individual are from the same gamete or different gametes is not influenced by the pedigree, and is always equal to $v-2:v$. The expectation of ${\tilde{H}}_E^{\prime }$ for a given sample varies around $\frac{v-1}{N_{b}v}{H}_{O,0}+\frac{N_b-1}{N_b}{H}_{E,0}$, because the probability that two allele copies between individuals are from the same parents depends on the pedigree. We derived the expectation of ${\tilde{H}}_E^{\prime }$ in Appendix F, and found that this is equal to $\frac{v-1}{N_{p}v}{H}_{O,0}+\frac{N_p-1}{N_p}{H}_{E,0}$. Here, the term $N_p=\frac{4}{P_{\mathit{sf}}+P_{\mathit{sm}}}$ represents the effective number of parents contributing to the sampled cohort, where $P_{\mathit{sf}}$ (or $P_{\mathit{sm}}$) is the probability of two allele copies in eggs (or sperms) that form the sampled individuals (or breeding individuals) are from the same parent.

The expectation of $N_p$ was derived in Appendix F, and can be calculated as

$$\text{E}\left(N_p|r\right)\approx N_b+\frac{N_b\left(N_b-4+8r-8r^2\right)}{2n\left(n-1\right)},$$

where $r$ is the proportion of males in the breeding individuals in the previous generation. Without a priori information, we assume $r$ is uniformly distributed and take the integral of $\text{E}\left(N_p\right)$ over $r\in \left(0,1\right)$ to eliminate $r$:

$$\text{E}\left(N_p\right)\approx N_b+\frac{N_b\left(N_b-8⁄3\right)}{2n\left(n-1\right)}.$$

The computer derivations are provided in File S9.

We therefore conclude that ${\widehat{N}}_b$ actually estimates $N_p$, ${\widehat{N}}_b$ converges to $N_p$ as $L\to \infty$, $N_p$ is greater than $N_b$ if $N_b>8/3$, and $N_p$ converges to $N_b$ as $n\to \infty$.

4.3. ${\widehat{N}}_b$ With Fluctuating $N_b$

For variation of $N_b$ or $N_p$ across generations, we consider the transition from an initial state $D_0$ to the new equilibrium state $D_{\mathrm{eq}}$ for the non‐vibrated $D$ (polyploids and $N_b>2$). According to Equation (1), the expectation of $D$ in generation $t$ is:

(9)

Expanding the first‐order Taylor series for $\text{E}\left(D_1\right)$ at $D_0=0$ and $D_{\mathrm{eq},0}=0$, and eliminating the remainder of the Taylor series expansion, we obtained

$$\text{E}\left(D_1\right)\approx \frac{v-2}{2v-2}{D}_0+\frac{v}{2v-2}{D}_{\mathrm{eq},0}.$$

Therefore, $\text{E}\left(D_1\right)$ can be approximated by a linear combination of $D_0$ and $D_{\mathrm{eq},0}$, then assuming the population reproduces for several more generations the expectation of $D$ in generation $t$ is

$$\text{E}\left(D_t\right)\approx {\left(\frac{v-2}{2v-2}\right)}^t{D}_0+\frac{v}{2v-2}·{\sum }_{i=1}^t{\left(\frac{v-2}{2v-2}\right)}^{t-i}·D_{\text{eq},i-1}.$$

Therefore, $\text{E}\left(D_t\right)$ is a linear combination of $D_{\mathrm{eq}}$ in previous generations. For example, in autotetraploids and autohexaploids,

$$\text{E}\left(D_{t,v=4}\right)\approx \frac{2}{3}·D_{\text{eq},t-1}+\frac{2}{9}·D_{\text{eq},t-2}+\frac{2}{27}·D_{\text{eq},t-3}+\cdots ,$$

$$\text{E}\left(D_{t,v=6}\right)\approx \frac{6}{10}·D_{\text{eq},t-1}+\frac{24}{100}·D_{\text{eq},t-2}+\frac{96}{1000}·D_{\text{eq},t-3}+\cdots .$$

When $N_b$ is large, $D_{\mathrm{eq}}\approx \frac{2\left(v-1\right)}{N_b{v}^2}$ according to Equation (3). Assuming there are infinite numbers of both loci and samples, then $\widehat{D}$ can be estimated as previously shown, that is, $\widehat{D}=\text{E}\left(D_t\right)$. Moreover, according to Equation (6), $\text{E}\left(D_t\right)\approx \frac{2\left(v-1\right)}{v^2{\widehat{N}}_b}$. Replacing $D_{\text{eq},t}$ with $\frac{2\left(v-1\right)}{N_{b,t}{v}^2}$ and $\text{E}\left(D_t\right)$ with $\frac{2\left(v-1\right)}{v^2{\widehat{N}}_b}$ in the approximated expressions of $\text{E}\left(D_{t,v=4}\right)$ and $\text{E}\left(D_{t,v=6}\right)$, we obtained

$$\frac{1}{{\widehat{N}}_{b,v=4}}\approx \frac{2}{3}·\frac{1}{N_{b,t-1}}+\frac{2}{9}·\frac{1}{N_{b,t-2}}+\frac{2}{27}·\frac{1}{N_{b,t-3}}+\cdots ,$$

$$\frac{1}{{\widehat{N}}_{b,v=6}}\approx \frac{6}{10}·\frac{1}{N_{b,t-1}}+\frac{24}{100}·\frac{1}{N_{b,t-2}}+\frac{96}{1000}·\frac{1}{N_{b,t-3}}+\cdots .$$ (10)

This suggests that when ${\widehat{N}}_b$ is previously estimated, $1/{\widehat{N}}_b$ would be the weighted sum of $1/N_b$ in previous generations with the weights forming a geometric progression.

For finite samples, $N_{b,t}$ in Equation (10) should be replaced by $N_p$. Clearly, the coefficients are geometric sequences and the first term $N_{b,t-1}$ are more influential for autotetraploids than for higher levels of ploidy. This also explains why $D$ converges more slowly at higher ploidy levels when $N_b>2$ (Figure 1, Table 2). It is noteworthy that because we expanded the Taylor series for $D_1$ at $D_0=0$ and $D_{\mathrm{eq},0}=0$, the above equations are applicable for a high value of $N_b$ but not when $N_b=2$.

We next discuss the scenario for diploids, according to Equations (1) and (3):

$$\text{E}\left(D_{1,v=2}\right)=\frac{\left(1-D_0\right)D_{\text{eq},0}}{1-D_{\text{eq},0}+D_0{D}_{\text{eq},0}-D_{\text{eq},0}^2}.$$

Similarly, by expanding the second‐order Taylor series at $D_0=0$ and $D_{\mathrm{eq},0}=0$ and eliminating the remainder, we obtained

$$\text{E}\left(D_{1,v=2}\right)\approx D_{\text{eq},0}-D_0{D}_{\text{eq},0}+D_{\text{eq},0}^2.$$

The above equation can be expanded to $\text{E}\left(D_{t,v=2}\right)\approx D_{\text{eq},t-1}-\text{E}\left(D_{t-1}\right)D_{\text{eq},t-1}+D_{\text{eq},t-1}^2$. Replacing $\text{E}\left(D_{t-1}\right)$ with $D_{\text{eq},t-2}-\text{E}\left(D_{t-2}\right)D_{\text{eq},t-2}+D_{\text{eq},t-2}^2$ and eliminating the third‐order terms, we obtained:

Replacing $D_{\text{eq},t}$ with $\frac{2\left(v-1\right)}{N_{b,t}{v}^2}$ and $\text{E}\left(D_{t,v=2}\right)$ with $\frac{2\left(v-1\right)}{{\widehat{N}}_b{v}^2}$,

$$\frac{1}{{\widehat{N}}_{b,v=2}}\approx \frac{1}{N_{b,t-1}}\left(1+\frac{1}{N_{b,t-1}}-\frac{1}{N_{b,t-2}}\right).$$ (11)

Equation (11) implies that ${\widehat{N}}_b$ mainly reflects the value of $N_b$ in the previous generation in diploids. This also explains why diploids converged fastest when $N_b>2$ (Table 2, Figure S7).

It can also be inferred from Equations (10) and (11) that values of ${\widehat{N}}_b$ at lower‐levels of ploidy reflect more recent values of $N_b$, especially when $v=2$. In addition, because $D$ converges very fast, ${\widehat{N}}_b$ is less likely to be influenced by the mutations.

4.4. Bias of Pudovkin's (1996) Estimator

There are two potential causes of bias in Pudovkin et al. (1996) estimator under its core assumption (${\text{HWE}}_0$): (i) the biased estimation of $D$, and (ii) the use of $\text{E}\left(H_{E,t}\right)\approx {\lambda }^t{H}_{E,0}$ to approximate expected heterozygosities.

Biased estimation of $D$ is a more serious problem for smaller values of $N_b$, which can cause an underestimation of $D$ (Figure 1) and overestimation of $N_b$ under ${\text{HWE}}_0$ (Table 1). However, we used $N_b\ge 10$ in Figure 2 so any effects of a biased estimation of $D$ are negligible. Hence, the use of $\text{E}\left(H_{E,t}\right)\approx {\lambda }^t{H}_{E,0}$ to approximate expected heterozygosities is the main reason causing the bias under ${\text{HWE}}_0$, and we use Equation (1) to correct this bias and obtain an asymptoticly unbiased estimator ${\widehat{N}}_{b,0}$. By comparing the expressions presented in Table 1, it can be shown that in diploids ${\widehat{N}}_{b,\mathrm{Pu}}\approx {\widehat{N}}_{b,0}$, while ${\widehat{N}}_{b,0}$ is approximately ${\widehat{N}}_{b,\mathrm{Pu}}$ plus $\frac{3}{8}$ and $\frac{10}{18}$ in autotetraploids and autohexaploids, respectively, if they use the same $D$ estimator. This explains the downward bias of ${\widehat{N}}_{b,\mathrm{Pu}}$ under ${\text{HWE}}_0$ (Figure 2). When values for $N_b$ are high, this bias still exists but is concealed by the higher values of $N_b$ and increased variance of ${\widehat{N}}_{b,\mathrm{Pu}}$.

Under ${\text{HEE}}_0$, both ${\widehat{N}}_{b,\mathrm{Pu}}$ and ${\widehat{N}}_{b,0}$ are upwardly biased for diploids but downwardly biased for polyploids (Figure 2). These biases are caused by the difference between $\text{E}\left(D_1|{\text{HWE}}_0\right)$ and $D_{\mathrm{eq}}$ (Figure 1). ${\widehat{N}}_{b,\mathrm{Pu}}$ and ${\widehat{N}}_{b,0}$ assume ${\text{HWE}}_0$, under which the expected heterozygote‐excess index is $\text{E}\left(D_1|{\text{HWE}}_0\right)$. However, it is decreased (or increased) to $D_{\mathrm{eq}}$ in diploids (or polyploids) under ${\text{HEE}}_0$. Given the reciprocal relationship between ${\widehat{N}}_b$ and $\widehat{D}$, ${\widehat{N}}_{b,\mathrm{Pu}}$ and ${\widehat{N}}_{b,0}$ are upwardly (or downwardly) biased in diploids (or polyploids) under ${\text{HEE}}_0$. Moreover, the degrees of absolute bias of ${\widehat{N}}_{b,0}$ and ${\widehat{N}}_{b,\mathrm{Pu}}$ under ${\text{HEE}}_0$ also differ; this is lower in diploids than polyploids and is caused by the difference between $D_{\mathrm{eq}}$ and $D_1$, which is slight in diploids but clear in polyploids ($N_b=10$, Figure 1).

4.5. Bias and Variance of Corrected Estimators

For a single locus $l$, ${\widehat{H}}_{O,l}$ and ${\tilde{H}}_{E,l}$ are correlated. However, ${\widehat{D}}^{\prime }$ is calculated by summing the numerators and denominators across loci: ${\widehat{D}}^{\prime }=\frac{{\sum }_l{\widehat{H}}_{O,l}}{{\sum }_l{\tilde{H}}_{E,l}^{\prime }}-1$ according to Equation (8). Because the loci are assumed to be unlinked, the summation reduces the correlations between the numerators and the denominators $.$ Thus, ${\widehat{D}}^{\prime }$ is an asymptotically unbiased estimator of $D$, and the bias can be reduced to a negligible level when $L$ is high (e.g., 100). Besides, the inverse $1/{\widehat{D}}^{\prime }$ can be expressed as $\frac{1}{{\widehat{D}}^{\prime }}=\frac{{\sum }_l{\tilde{H}}_{E,l}^{\prime }}{{\sum }_l\left({\widehat{H}}_{O,l}-{\tilde{H}}_{E,l}^{\prime }\right)}.$ Similarly, the summation reduces the correlations between the numerator and the denominator, and $1/{\widehat{D}}^{\prime }$ is also an asymptoticly unbiased estimator of $1/D$. This suggests that our ${\widehat{N}}_{b,0}$ and ${\widehat{N}}_{b,-\infty }$ are asymptoticly unbiased, which is consistent with our simulations (Figure 3).

In polyploids, $N_b$ can still be unbiased, as estimated by ${\widehat{N}}_{b,0}$ or ${\widehat{N}}_{b,-\infty }$ under their core assumptions, while variance may be either increased or decreased with increased level of ploidy. Two factors cause this phenomenon: (i) under the same conditions, polyploids have more within and between individual allele pairs than diploids. The observed and expected heterozygosities are therefore estimated with increased accuracy, as well as estimates for $D$ and $1/D$, which reduces the variance of ${\widehat{N}}_b$; (ii) because ${\widehat{N}}_b\approx a/\widehat{D}$ (Table 1), the coefficient $a$ is higher in polyploids and increases the variance of ${\widehat{N}}_b$.

For variance, the increase in accuracy and precision by sampling one more individual is approximately equal to using two additional loci (Figures S2 and S5). Due to the cost of sequencing one extra individual usually being higher than using two additional loci, especially in polyploids, using more loci is probably in most cases more practical because this is feasible due to the current capabilities of modern genomic technologies, which enable the simultaneous detection of millions of genetic loci. But the number of individuals and loci should be balanced based on the study's goals, population characteristics, and available resources. In medical and agricultural research, larger sample sizes are common, while conservation focused ecological studies often use fewer individuals. For example, in our simulations for diploids with $N_b=10$, we achieve a 10% CV with 50 individuals and 1262 loci, and the same with 1000 individuals and 333 loci, which corresponds to more individuals and fewer loci. However, increasing the number of individuals further does not reduce the number of loci needed to maintain the same CV (Figure 5 and Table S3). From the isometric curves, $\log L_{\mathrm{CV}}$ increases by approximately 2 as $\log N_b$ increases by 1, such that the relationship between $N_b$ and $L_{\mathrm{CV}}$ can be roughly described by $L_{\mathrm{CV}}\propto N_b^2$ when $N_b>100$.

4.6. Application Scope

The extended definitions of $N_e$ and $N_b$ to polyploid species, along with the corrected Pudovkin et al. (1996) $N_b$ estimator under ${\text{HWE}}_0$ and ${\text{HEE}}_0$, provide new opportunities for accurate estimation of population parameters both in polyploid and diploid organisms.

Our corrected estimator ${\widehat{N}}_{b,-\infty }$ is appropriate for DR mating system, and performs well for small to medium sized populations with $N_b\le 100$ (Figures 3 and 4, Figures S1 and S4), which are in need of monitoring to prevent extirpation. Moreover, it can be easily adapted for use in both diploid and polyploid species, and also performs well in polyploids. Regarding marker types, our estimator is applicable to both single nucleotide polymorphisms (SNPs) and simple sequence repeats (SSRs). Compared with ${\widehat{N}}_{b,\mathrm{Pu}}$ and ${\widehat{N}}_{b,0}$, we recommend use of ${\widehat{N}}_{b,-\infty }$ in practice because it assumes more realistic conditions.

While the true $N_b$ of the empirical dataset remains unknown, our assessments show that ${\widehat{N}}_{b,-\infty }$ offers greater robustness than ${\widehat{N}}_{b,\mathrm{Pu}}$, with wider 95% CIs (Tables S4 and S5). For high‐risk decisions, this reliability is paramount, conservative estimation presents a much lower risk than the severe consequences of overestimation.

It is crucial to emphasise that our method relies on the assumption of heterozygote excess. However, in many populations relevant to conservation biology, factors such as inbreeding and population subdivision—which often lead to heterozygote deficiency—are frequently observed. When applied to such populations, our method may yield negative estimates, thereby limiting its applicability. In such cases, simply discarding negative values may introduce bias. To enhance the applicability of the method in complex practical scenarios, the following adjustment strategies may be considered: (i) for inbred populations, if high‐resolution genotyping techniques such as resequencing are used, individuals whose Runs of Homozygosity (ROH) cover a specific locus can be excluded when calculating heterozygosity for that locus, thereby reducing the influence of inbreeding on heterozygosity estimation; (ii) for populations with substructure, subpopulations can first be identified using methods such as principal coordinate analysis (PCoA), K‐means clustering, or hierarchical clustering. $N_b$ can then be estimated separately for each genetically homogeneous subpopulation, or the harmonic mean of $N_b$ across subpopulations can be calculated to more accurately reflect the reproductive characteristics of the population.

It should be noted that the specific implementation of the above strategies is beyond the scope of this study. This research develops bias‐corrected, robust $N_b$ estimator and extends them to autopolyploids, validated through simulations. We advise users to confirm heterozygosity excess before applying the method and to preprocess data or interpret results with the above strategies in mind. These limitations also highlight avenues for future methodological and applied research.

5. Conclusion

We extended the definition of $N_e$ and $N_b$ to polyploids, corrected Pudovkin et al. (1996) $N_b$ estimator under ${\text{HWE}}_0$ and ${\text{HEE}}_0$, and extended and corrected Pudovkin et al. (1996) estimator to account for polyploids. From computer simulations, we found that genotypes can rapidly reach HEE, our corrected estimators can unbiasedly estimate $N_b$, and our estimates for polyploids are slightly better than those for diploids. We also discussed the relationship between ${\widehat{N}}_b$ and $N_b$, and ${\widehat{N}}_b$ in variable $N_b$ across generations.

Author Contributions

B.Y., K.H., and B.L. conceptualised, designed and performed the study; K.H. and B.Y. developed methods; B.Y. and K.H. wrote the original draft paper, B.Y., K.H., D.W., Y.S., W.L, H.W., R.W., R.P., S.Z, Z.X. and D.W.D. reviewed and edited the paper; B.L. supervised the process of writing; K.H. and B.L. funded the study.

Funding

This work was supported by the National Natural Science Foundation of China (32570568, 32170515, 32371563, 32070453, 12171391) and the National Key R&D Program of China (2024YFF1307302).

Disclosure

Benefit‐sharing section: This study contains no new empirical data. The Yellowstone grizzly bears dataset and the brook trout dataset are retrieved from Dryad with https://doi.org/10.5061/dryad.s6764 (Kamath et al. 2015) and https://doi.org/10.5061/dryad.n02v6wwv6 (Anne‐Laure et al. 2020), respectively. Benefits from this research accrue from the sharing of our software and source code on public databases as described above.

Conflicts of Interest

The authors declare no conflicts of interest.

Supporting information

Data S1: Supporting Information.

File S1: S1.nb, derivation of the corrected estimator (*.nb is the mathematica v9.0 notebook file).

File S2: S2.nb, derivation of Pudovkin et al. (1996) estimator.

File S3: S3.7z, C++ source code of the simulation programme.

File S4: S4.7z, Simulation programme (64‐bit Windows PE executable, compiled by Visual Studio 2019), parameters files (Sim1.txt to Sim5.txt) and batch files to perform the Monte‐Carlo simulations (Sim1.bat to Sim5.bat).

File S5: S5.7z, matlab v8.2 scripts to process the simulation results (Sim1.m to Sim6.m) and resulting figures in PDF format.

File S6: S6.nb, derivation of moments of homozygosities.

File S7: S7.nb, derivation of bias and variance of ${\widehat{N}}_{b,-\infty }$.

File S8: S8.nb, calculation of the number of loci required to reach a CV.

File S9: S9.nb, derivation of the expectation of $N_p$.

Yang, B. , Wang D., Shen Y., et al. 2026. “Estimating Effective Number of Breeders Under the Heterozygote‐Excess Equilibrium.” Molecular Ecology Resources 26, no. 2: e70107. 10.1111/1755-0998.70107.

Data Availability Statement

The derivations of original and corrected estimators, and the source code and binary executables of the simulation programme are available in the Supporting Information of this article. Our methods have been implemented in a software package, polygene v1.7, which is freely available at https://github.com/huangkang1987/polygene.