An Unbiased Estimator of Gene Diversity with Improved Variance for Samples Containing Related and Inbred Individuals of any Ploidy

Abstract

Gene diversity, or expected heterozygosity (H), is a common statistic for assessing genetic variation within populations. Estimation of this statistic decreases in accuracy and precision when individuals are related or inbred, due to increased dependence among allele copies in the sample. The original unbiased estimator of expected heterozygosity underestimates true population diversity in samples containing relatives, as it only accounts for sample size. More recently, a general unbiased estimator of expected heterozygosity was developed that explicitly accounts for related and inbred individuals in samples. Though unbiased, this estimator’s variance is greater than that of the original estimator. To address this issue, we introduce a general unbiased estimator of gene diversity for samples containing related or inbred individuals, which employs the best linear unbiased estimator of allele frequencies, rather than the commonly used sample proportion. We examine the properties of this estimator, ${\stackrel{\sim }{H}}_{\text{BLUE}},$ relative to alternative estimators using simulations and theoretical predictions, and show that it predominantly has the smallest mean squared error relative to others. Further, we empirically assess the performance of ${\stackrel{\sim }{H}}_{\text{BLUE}}$ on a global human microsatellite dataset of 5795 individuals, from 267 populations, genotyped at 645 loci. Additionally, we show that the improved variance of ${\stackrel{\sim }{H}}_{\text{BLUE}}$ leads to improved estimates of the population differentiation statistic, $F_{\text{ST}},$ which employs measures of gene diversity within its calculation. Finally, we provide an R script, BestHet, to compute this estimator from genomic and pedigree data.

The gene diversity of a locus, also known as its expected heterozygosity (H), is a fundamental measure of genetic variation in a population, and describes the proportion of heterozygous genotypes expected under Hardy-Weinberg equilibrium (Nei 1973). Formally, gene diversity is the probability that a pair of randomly sampled allele copies from a population are different, and is computed as

$$H=1-{\displaystyle \sum _{i=1}^I{p}_i^2},$$ (1)

where I is the number of distinct alleles at a locus, and $p_i$ ($i=1,2,\dots ,I$) is the frequency of allele i in the population.

For a sample without related or inbred individuals composed of n allele copies, an unbiased estimator of expected heterozygosity is (Nei and Roychoudhury 1974)

$$\widehat{H}=\frac{n}{n-1}\left(1-{\displaystyle \sum _{i=1}^I{\widehat{p}}_i^2}\right),$$ (2)

where ${\widehat{p}}_i$ is the sample proportion of allele i. $\widehat{H}$ is a biased estimator when inbred or related individuals are included in the sample (DeGiorgio and Rosenberg 2009). This result is based on the idea that, as the proportion of related individuals in the sample increases, the number of independent allele observations decreases.

When two alleles are drawn from a sample, one each from a pair of related individuals, there is a nonzero probability that they will be identical by descent (IBD), rather than just identical by state (Lange 2002). This IBD probability is known as the kinship coefficient, and is denoted by ${\Phi }_{jk}$ for a pair of individuals j and k. Thus, the observed diversity will be lower than the true value because a greater proportion of identical alleles are observed than for a sample in which there are no related individuals. DeGiorgio et al. (2010) developed an estimator of expected heterozygosity,

$$\stackrel{\sim }{H}=\frac{1}{1-{\overline{\Phi }}_2}\left(1-{\displaystyle \sum _{i=1}^I{\widehat{p}}_i^2}\right),$$ (3)

which is unbiased for samples containing related and inbred individuals of any ploidy, and employs a weighted mean kinship coefficient ${\overline{\Phi }}_2$ as a bias correction factor. ${\overline{\Phi }}_2$ is the average of all kinship coefficients ${\Phi }_{jk}$ for every pair of individuals within the sample (see Methods). Further, DeGiorgio et al. (2010) derived the theoretical variance of $\stackrel{\sim }{H},$ as well as its approximate value for samples wherein individuals are related to no more than one other sampled individual.

As an alternative to the sample proportion (${\widehat{p}}_i$), McPeek et al. (2004) introduced the best linear unbiased estimator (BLUE, denoted as ${\stackrel{\sim }{p}}_i$) of population allele frequency, which is an unbiased linear estimator with smaller variance than the unbiased linear estimator ${\widehat{p}}_i.$ The BLUE incorporates the relatedness of individuals in the sample as a covariance matrix to define the weight of each observation. Simulations and analytical evaluation corroborating their result suggest that the mean squared error (MSE) of ${\stackrel{\sim }{p}}_i$ is always smaller than that of ${\widehat{p}}_i,$ and this difference is especially evident for samples with complex pedigrees.

Because ${\stackrel{\sim }{p}}_i$ has the smallest variance of any unbiased linear estimator of allele frequencies, we expect its low variance to translate to smaller variance of gene diversity statistics that use ${\stackrel{\sim }{p}}_i.$ We developed such a statistic, termed ${\stackrel{\sim }{H}}_{\text{BLUE}},$ that is an unbiased estimator of expected heterozygosity in samples containing related and inbred individuals of arbitrary ploidy. Through simulations, analytical predictions, and empirical assessments, we compare the performance of ${\stackrel{\sim }{H}}_{\text{BLUE}}$ to that of $\stackrel{\sim }{H}$ and $\widehat{H}$ for samples containing related individuals of various types across different ploidy and inbreeding status. Additionally, we derive the variance of any measure of expected heterozygosity that uses unbiased linear estimators of allele frequencies. We find that the increased precision of allele frequency estimates transfers to our unbiased estimator, yielding values for MSE invariably equal to or smaller than those of $\stackrel{\sim }{H},$ while occasionally exceeding the precision of $\widehat{H}.$ The improved properties of ${\stackrel{\sim }{H}}_{\text{BLUE}}$ translate to its applications as well, which we demonstrate in the calculation of the population differentiation statistic, $F_{\text{ST}}$ (Wright 1951). $F_{\text{ST}}$ can be written in terms of intrapopulation and interpopulation gene diversity as (Hudson et al. 1992)

$$F_{\text{ST}}=\frac{H_{12}-\frac{1}{2}\left(H_1+H_2\right)}{H_{12}},$$ (4)

where $H_1$ and $H_2$ are the values of expected heterozygosity within each of two compared populations, and $H_{12}$ is the expected heterozygosity between them.

Methods

Consider a locus with I distinct alleles in a sample of n individuals. Let $X_k^{\left(i\right)}$ denote the fraction of alleles at the locus in individual k that are of type i, $i=1,2,\dots ,I.$ An unbiased linear estimator of population allele frequencies $p_i,$ denoted by ${\breve{p}}_i,$ is defined as

$${\breve{p}}_i={\displaystyle \sum _{k=1}^n{w}_k{X}_k^{\left(i\right)}},$$ (5)

where $w_k,$ $0\le w_k\le 1,$ is the weight of individual k, $k=1,2,\dots ,n,$ and ${\displaystyle {\sum }_{k=1}^n{w}_k}=1.$ Formally, we have that

$$X_k^{\left(i\right)}=\frac{1}{m_k}{\displaystyle \sum _{t=1}^{m_k}{A}_{kt}^{\left(i\right)}},$$

where $A_{kt}^{\left(i\right)}$ is an indicator random variable whose value is 1 if allele t of individual k is of type i, and zero otherwise, and where $m_k$ is the ploidy of individual k. As an example, if individual k were diploid at the locus, then $m_k=2.$ Taking the expectation of ${\breve{p}}_i,$

$$\begin{array}{c}\mathbb{E}\left[{\breve{p}}_i\right]={\displaystyle \sum _{k=1}^n\frac{w_k}{m_k}}{\displaystyle \sum _{t=1}^{m_k}\mathbb{E}\left[A_{kt}^{\left(i\right)}\right]}\\ ={\displaystyle \sum _{k=1}^n\frac{w_k}{m_k}}{\displaystyle \sum _{t=1}^{m_k}{p}_i}\\ =p_i,\end{array}$$

shows that it is an unbiased estimator of $p_i.$

Unbiased estimation of gene diversity using unbiased linear estimators of allele frequencies

In this section, we construct an unbiased estimator, $\breve{H},$ of expected heterozygosity that uses a general unbiased linear estimator, ${\breve{p}}_i,$ of allele frequency $p_i$ (Proposition 1). We then show that the unbiased estimator, $\stackrel{\sim }{H},$ of DeGiorgio et al. (2010) follows as a corollary, assuming that ${\breve{p}}_i={\widehat{p}}_i,$ the sample proportion allele frequency estimator (Corollary 2). We then derive a new estimator, ${\stackrel{\sim }{H}}_{\text{BLUE}},$ also as a corollary, assuming that ${\breve{p}}_i={\stackrel{\sim }{p}}_i,$ the BLUE of allele frequency (Corollary 3).

Proposition 1:

Consider a locus with I distinct alleles and parametric allele frequencies $p_i\in \left[0,1\right],$ $i=1,2,\dots ,I,$ and ${\displaystyle {\sum }_{i=1}^I{p}_i}=1.$ For a sample of size n individuals of any ploidy, inbreeding status, and relatedness,

$$\breve{H}=\frac{1}{1-{\rho }_2}\left(1-{\displaystyle \sum _{i=1}^I{\breve{p}}_i^2}\right)$$ (6)

is an unbiased estimator of expected heterozygosity, where

$${\rho }_2={\displaystyle \sum _{j=1}^n{\displaystyle \sum _{k=1}^n{w}_j{w}_k{\Phi }_{jk}}}$$

is a weighted mean kinship coefficient of the sample for all pairs of individuals in the sample, and where $w_k,$ $k=1,2,\dots ,n,$ is the weight for individual k. The proof of Proposition 1 is found in the Appendix.

From ${\breve{p}}_i,$ the sample proportion estimator ${\widehat{p}}_i$ of allele frequency i, $i=1,2,\dots ,I,$ is recovered when $w_k=m_k/{\displaystyle {\sum }_{j=1}^n{m}_j}$ for individual k, $k=1,2,\dots ,n,$ leading to

$${\widehat{p}}_i={\displaystyle \sum _{k=1}^n\frac{m_k}{{\displaystyle {\sum }_{j=1}^n{m}_j}}{X}_k^{\left(i\right)}}.$$

Here, each individual is weighted by its contribution to the number of allele copies in the sample.

Corollary 2:

Consider a locus with I distinct alleles and parametric allele frequencies $p_i\in \left[0,1\right],$ $i=1,2,\dots ,I,$ and ${\displaystyle {\sum }_{i=1}^I{p}_i}=1.$ For a sample of size n individuals of any ploidy, inbreeding status, and relatedness,

$$\stackrel{\sim }{H}=\frac{1}{1-{\overline{\Phi }}_2}\left(1-{\displaystyle \sum _{i=1}^I{\widehat{p}}_i^2}\right)$$ (7)

is an unbiased estimator of expected heterozygosity, where

$${\widehat{p}}_i={\displaystyle \sum _{k=1}^n\frac{m_k}{{\displaystyle {\sum }_{j=1}^n{m}_j}}{X}_k^{\left(i\right)}}$$

is the sample proportion estimator of allele frequency i, where

$${\overline{\Phi }}_2={\displaystyle \sum _{j=1}^n{\displaystyle \sum _{k=1}^n\frac{m_j}{{\displaystyle {\sum }_{x=1}^n{m}_x}}\frac{m_k}{{\displaystyle {\sum }_{y=1}^n{m}_y}}{\Phi }_{jk}}}$$

is a weighted mean kinship coefficient of the sample for all pairs of individuals, and where $m_k,$ $k=1,2,\dots ,n,$ is the ploidy for individual k. The proof of Corollary 2 is found in the Appendix.

It may be beneficial to apply an unbiased linear estimator of allele frequencies that has minimum variance. McPeek et al. (2004) introduced the BLUE of allele frequencies, which we formally define here. We will use the BLUE of allele frequencies to construct a new unbiased estimator of gene diversity that would ideally have improved variance over other estimators. Let $K$ be an $n\times n$ symmetric matrix of kinship coefficients, with $K_{jk}={\Phi }_{jk}.$ The BLUE (${\stackrel{\sim }{p}}_i$) of allele frequency is obtained when $w_k=\frac{{\displaystyle {\sum }_{j=1}^n{\left(K^{-1}\right)}_{jk}}}{{\mathbf{1}}^T{K}^{-1}\mathbf{1}},$ yielding

$${\stackrel{\sim }{p}}_i={\displaystyle \sum _{k=1}^n\frac{{\displaystyle {\sum }_{j=1}^n{\left(K^{-1}\right)}_{jk}}}{{\mathbf{1}}^T{K}^{-1}\mathbf{1}}{X}_k^{\left(i\right)}},$$

where $K^{-1}$ denotes the inverse matrix of $K,$ 1 is a column vector of n elements with all entries equal to 1, and ${\mathbf{1}}^T$ is the transpose of 1.

Corollary 3:

Consider a locus with I distinct alleles, and parametric allele frequencies $p_i\in \left[0,1\right],$ $i=1,2,\dots ,I,$ and ${\displaystyle {\sum }_{i=1}^I{p}_i}=1.$ For a sample of size n individuals of any ploidy, inbreeding status, and relatedness,

$${\stackrel{\sim }{H}}_{\text{BLUE}}=\frac{1}{1-{\kappa }_2}\left(1-{\displaystyle \sum _{i=1}^I{\stackrel{\sim }{p}}_i^2}\right)$$ (8)

is an unbiased estimator of expected heterozygosity, where

$${\stackrel{\sim }{p}}_i={\displaystyle \sum _{k=1}^n\frac{{\displaystyle {\sum }_{j=1}^n{\left(K^{-1}\right)}_{jk}}}{{\mathbf{1}}^T{K}^{-1}\mathbf{1}}{X}_k^{\left(i\right)}}$$

is the BLUE of allele frequencies, and where

$${\kappa }_2={\displaystyle \sum _{j=1}^n{\displaystyle \sum _{k=1}^n\frac{{\displaystyle {\sum }_{x=1}^n{\left(K^{-1}\right)}_{xj}}}{{\mathbf{1}}^T{K}^{-1}\mathbf{1}}\frac{{\displaystyle {\sum }_{y=1}^n{\left(K^{-1}\right)}_{yk}}}{{\mathbf{1}}^T{K}^{-1}\mathbf{1}}{\Phi }_{jk}}}$$

is a weighted mean kinship coefficient of the sample for all pairs of individuals. The proof of Corollary 3 is found in the Appendix.

Variance of H estimators using unbiased linear estimators of allele frequencies

We now derive the equation (Proposition 4) describing the variance of the unbiased estimator $\breve{H},$ which takes ${\breve{p}}_i$ as the unbiased linear estimate of population allele frequency $p_i.$ This value depends on the weighted mean kinship coefficients of the sample for all pairs, trios, quartets, and pairs of pairs of individuals in the sample, defined as

$$\begin{array}{l}{\rho }_2={\displaystyle \sum _{j=1}^n{\displaystyle \sum _{k=1}^n{w}_j{w}_k{\Phi }_{jk}}}\\ {\rho }_3={\displaystyle \sum _{j=1}^n{\displaystyle \sum _{k=1}^n{\displaystyle \sum _{j^{\prime }=1}^n{w}_j{w}_k{w}_{j^{\prime }}{\Phi }_{jk{j}^{\prime }}}}}\\ {\rho }_4={\displaystyle \sum _{j=1}^n{\displaystyle \sum _{k=1}^n{\displaystyle \sum _{j^{\prime }=1}^n{\displaystyle \sum _{k^{\prime }=1}^n{w}_j{w}_k{w}_{j^{\prime }}{w}_{k^{\prime }}{\Phi }_{jk{j}^{\prime }{k}^{\prime }}}}}}\\ {\rho }_{2,2}={\displaystyle \sum _{j=1}^n{\displaystyle \sum _{k=1}^n{\displaystyle \sum _{j^{\prime }=1}^n{\displaystyle \sum _{k^{\prime }=1}^n{w}_j{w}_k{w}_{j^{\prime }}{w}_{k^{\prime }}{\Phi }_{jk,j^{\prime }{k}^{\prime }}}}}}.\end{array}$$

Here, ${\Phi }_{jk{j}^{\prime }}$ is the probability that three randomly sampled alleles, one each from individuals j, k, and $j^{\prime },$ are IBD. ${\Phi }_{jk{j}^{\prime }{k}^{\prime }}$ is the probability that four randomly sampled alleles, one each from individuals j, k, $j^{\prime },$ and $k^{\prime },$ are IBD. Finally, ${\Phi }_{jk,j^{\prime }{k}^{\prime }}$ is the joint probability that two randomly sampled alleles, one each from individuals j and k are IBD, and two randomly sampled alleles, one each from individuals $j^{\prime }$ and $k^{\prime },$ are IBD. Note that individuals j, k, $j^{\prime },$ and $k^{\prime }$ are not necessarily distinct. The variances of $\stackrel{\sim }{H}$ and of ${\stackrel{\sim }{H}}_{\text{BLUE}}$ follow as Corollaries 7 and 8, once again differing only in the weight of a sampled individual in the mean kinship coefficient calculation.

Proposition 4:

Consider a locus with I distinct alleles and parametric allele frequencies $p_i\in \left[0,1\right],$ $i=1,2,\dots ,I,$ and ${\displaystyle {\sum }_{i=1}^I{p}_i}=1.$ For a sample of size n individuals of any ploidy, inbreeding status, and relatedness,

$$\text{Var}\left[\breve{H}\right]=\frac{1}{{\left(1-{\rho }_2\right)}^2}\text{Var}\left[1-{\displaystyle \sum _{i=1}^I{\breve{p}}_i^2}\right]$$ (9)

is the variance of the unbiased estimator of expected heterozygosity $\breve{H},$ where ${\rho }_2={\displaystyle {\sum }_{j=1}^n{\displaystyle {\sum }_{k=1}^n{w}_j{w}_k{\Phi }_{jk}}}$ is a weighted mean kinship coefficient of the sample, and where $w_k$ for $k=1,2,\dots ,n$ is the weight of individual k. Further, we have

$$\begin{array}{c}\text{Var}\left[1-{\displaystyle \sum _{i=1}^I{\breve{p}}_i^2}\right]={\rho }_{2,2}-{\rho }_2^2+2\left({\rho }_2^2-{\rho }_4\right){\displaystyle \sum _{i=1}^I{p}_i^2}+4\left(2{\rho }_4+{\rho }_2-2{\rho }_3-{\rho }_{2,2}\right){\displaystyle \sum _{i=1}^I{p}_i^3}\\ +\left(3{\rho }_{2,2}+8{\rho }_3-6{\rho }_4-4{\rho }_2-{\rho }_2^2\right){\left({\displaystyle \sum _{i=1}^I{p}_i^2}\right)}^2.\end{array}$$ (10)

The proof of Equation 10 is presented for the specific case of $\text{Var}\left[1-{\displaystyle {\sum }_{i=1}^I{\widehat{p}}_i^2}\right]$ in Appendix B of DeGiorgio et al. (2010), where ${\widehat{p}}_i$ is substituted for ${\breve{p}}_i,$ and ${\overline{\Phi }}_2,$ ${\overline{\Phi }}_3,$ and ${\overline{\Phi }}_4,$ and ${\overline{\Phi }}_{2,2}$ coefficients are substituted for ${\rho }_2,$ ${\rho }_3,$ ${\rho }_4,$ and ${\rho }_{2,2}$ coefficients, respectively. We provide an abbreviated version of this proof for the general case in the Appendix. Further, the approximate value of Equation 10 for samples wherein no individual is related to more than one other is

$$\text{Var}\left[1-{\displaystyle \sum _{i=1}^I{\breve{p}}_i^2}\right]\approx 4{\rho }_2\left[{\displaystyle \sum _{i=1}^I{p}_i^3}-{\left({\displaystyle \sum _{i=1}^I{p}_i^2}\right)}^2\right].$$ (11)

For this simplifying case, the terms ${\rho }_3,$ ${\rho }_4,$ ${\rho }_{2,2},$ and ${\rho }_2^2$ are negligible compared to ${\rho }_2.$

In the Appendix, we reintroduce the definition of $\text{Var}\left[\stackrel{\sim }{H}\right]$ from DeGiorgio et al. (2010) (Corollary 7), and then define $\text{Var}\left[{\stackrel{\sim }{H}}_{\text{BLUE}}\right]$ (Corollary 8), both of which take the form illustrated in Proposition 4. As demonstrated by DeGiorgio et al. (2010), the mean kinship coefficients composing Equation 10 derive from the relationship between the 15 identity states available to four alleles (Gillois 1965; Cockerham 1971), and the coefficients of kinship between pairs, trios, quartets, and pairs of pairs of alleles within those four.

Bias of $\widehat{\text{H}}$ for samples containing related or inbred individuals

Here, we briefly derive an equation (Equation 12) within Proposition 5 that describes the bias of $\widehat{H},$ which we display in the left panels of Supplemental Material, Figure S1A and Figure S2A. We include Corollaries 9 and 10 to Proposition 5 within the Appendix for specific cases of bias derived from ${\widehat{p}}_i$-based and ${\stackrel{\sim }{p}}_i$-based estimations, respectively. We also note that Equation A10 of Corollary 9 represents the form of the bias typically encountered in applications of $\widehat{H},$ as well as in all of our experimental scenarios.

Proposition 5:

Consider a locus with I distinct alleles and parametric allele frequencies $p_i\in \left[0,1\right],$ $i=1,2,\dots ,I,$ and ${\displaystyle {\sum }_{i=1}^I{p}_i}=1.$ For a sample of n possibly related or inbred individuals, the bias of the estimator of expected heterozygosity $\widehat{H}$ changes with the true locus expected heterozygosity such that

$$\text{Bias}\left[\widehat{H}\left({\breve{p}}_i\right)\right]=\frac{1-n{\rho }_2}{n-1}H,$$ (12)

where

$$\widehat{H}\left({\breve{p}}_i\right)=\frac{n}{n-1}\left(1-{\displaystyle \sum _{i=1}^I{\breve{p}}_i^2}\right).$$ (13)

Proof:

We begin by substituting Equation 6 into Equation 13 such that

$$\widehat{H}\left({\breve{p}}_i\right)=\frac{n\left(1-{\rho }_2\right)}{n-1}\breve{H},$$

and

$$\mathbb{E}\left[\widehat{H}\left({\breve{p}}_i\right)\right]=\frac{n\left(1-{\rho }_2\right)}{n-1}H.$$

From the definition of bias,

$$\begin{array}{c}\text{Bias}\left[\widehat{H}\left({\breve{p}}_i\right)\right]=\mathbb{E}\left[\widehat{H}\left({\breve{p}}_i\right)\right]-H\\ =\frac{1-n{\rho }_2}{n-1}H.\end{array}$$ □

Variance of ${\text{F}}_{\text{ST}}$ estimators using unbiased linear estimators of allele frequencies

Because the population differentiation statistic $F_{\text{ST}}$ (Wright 1951) can be defined in terms of expected heterozygosities, it is possible to theoretically evaluate its approximate variance. A general estimator of $F_{\text{ST}}$ can be written as

$${\breve{F}}_{\text{ST}}=\frac{{\breve{H}}_{12}-\frac{1}{2}\left({\breve{H}}_1+{\breve{H}}_2\right)}{{\breve{H}}_{12}},$$ (14)

where ${\breve{H}}_{12}$ is an unbiased estimator for the expected heterozygosity between a pair of sampled populations, numbered 1 and 2, defined as ${\breve{H}}_{12}=1-{\displaystyle {\sum }_{i=1}^I{\breve{p}}_i{\breve{q}}_i}$ (where ${\breve{q}}_i$ is a linear unbiased estimator of the frequency of allele i in population 2, analogous to ${\breve{p}}_i$ in population 1), while ${\breve{H}}_1$ and ${\breve{H}}_2$ are the within-population expected heterozygosities for populations 1 and 2, respectively. Referring to the numerator as x, and the denominator as y, we can write the expression for an approximation of the variance of a ratio as

$$\text{Var}\left[\frac{x}{y}\right]\approx \frac{{\left(\mathbb{E}\left[x\right]\right)}^2}{{\left(\mathbb{E}\left[y\right]\right)}^2}\left[\frac{\text{Var}\left[x\right]}{{\left(\mathbb{E}\left[x\right]\right)}^2}+\frac{\text{Var}\left[y\right]}{{\left(\mathbb{E}\left[y\right]\right)}^2}-2\frac{\text{Cov}\left[x,y\right]}{\mathbb{E}\left[x\right]\mathbb{E}\left[y\right]}\right],$$ (15)

following the definition for the approximate variance of a ratio (Wolter 2007).

Proposition 6:

Consider a locus with I distinct alleles across two populations and parametric allele frequencies $p_i\in \left[0,1\right],$ $i=1,2,\dots ,I,$ and ${\displaystyle {\sum }_{i=1}^I{p}_i}=1$ for population 1, and $q_i\in \left[0,1\right],$ $i=1,2,\dots ,I,$ and ${\displaystyle {\sum }_{i=1}^I{q}_i}=1$ for population 2. For samples of size $n_1$ and $n_2$ individuals from populations 1 and 2, respectively, each with individuals of any ploidy, inbreeding status, and relatedness, the variance of the population differentiation statistic calculated from their respective expected heterozygosities is approximated as

$$\text{Var}\left[{\breve{F}}_{\text{ST}}\right]\approx \frac{{\left[H_{12}-\frac{1}{2}\left(H_1+H_2\right)\right]}^2}{H_{12}^2}\times \left[\frac{\text{Var}\left[{\breve{H}}_{12}-\frac{1}{2}\left({\breve{H}}_1+{\breve{H}}_2\right)\right]}{{\left[H_{12}-\frac{1}{2}\left(H_1+H_2\right)\right]}^2}+\frac{\text{Var}\left[{\breve{H}}_{12}\right]}{H_{12}^2}-2\frac{\text{Cov}\left[{\breve{H}}_{12}-\frac{1}{2}\left({\breve{H}}_1+{\breve{H}}_2\right),{\breve{H}}_{12}\right]}{\left[H_{12}-\frac{1}{2}\left(H_1+H_2\right)\right]H_{12}}\right],$$ (16)

where

$$\text{Var}\left[{\breve{H}}_{12}-\frac{1}{2}\left({\breve{H}}_1+{\breve{H}}_2\right)\right]=\text{Var}\left[{\breve{H}}_{12}\right]+\frac{1}{4}\text{Var}\left[{\breve{H}}_1\right]+\frac{1}{4}\text{Var}\left[{\breve{H}}_2\right]-\left(\text{Cov}\left[{\breve{H}}_{12},{\breve{H}}_1\right]+\text{Cov}\left[{\breve{H}}_{12},{\breve{H}}_2\right]\right).$$ (17)

In the Appendix, we provide a derivation of the variance and covariance components of Equations 16 and 17. For each of these equations, the result and proof are fairly long, and do not simplify when arranged into Equation 16.

Data availability

The authors state that all data necessary for confirming the conclusions presented in the article are represented fully within the article.

Results

Analytical validation of ${\stackrel{\sim }{\text{H}}}_{\text{BLUE}}$

We tested the performance of ${\stackrel{\sim }{H}}_{\text{BLUE}}$ using both theory and simulations against that of the unbiased estimator $\stackrel{\sim }{H}$ (DeGiorgio et al. 2010), and of $\widehat{H}$ (Nei and Roychoudhury 1974). Here, we applied the estimators to samples of individuals wherein each individual was related to exactly one other. Thus, for samples of size n individuals, the number of relative pairs was n/2. When inbred or closely related individuals are included in a sample, $\widehat{H}$ is a biased estimator of gene diversity for which we use the symbol ${\widehat{H}}_{\text{full}}.$ To construct an unbiased estimator with $\widehat{H},$ we also applied $\widehat{H}$ to a reduced sample in which one member of each relative pair was removed randomly for samples containing only diploid individuals, and the haploid member was removed for each haploid-diploid (i.e., male-female) pair (reduced sample size of n/2), and we denote this estimator by ${\widehat{H}}_{\text{red}}.$ To evaluate the performance of the four estimators (${\widehat{H}}_{\text{full}},$ ${\widehat{H}}_{\text{red}},$ $\stackrel{\sim }{H},$ and ${\stackrel{\sim }{H}}_{\text{BLUE}}$), we modified the factors upon which their variance depends: true locus expected heterozygosity (H), sample size n, and relatedness of individuals within the sample ($\Phi$).

Effect of true locus expected heterozygosity, H, on estimators

We first evaluated the theoretical bias, variance, and mean squared error (MSE) of each estimator across the 645 human microsatellite loci from across the genome in the composite dataset MS5795 of Pemberton et al. (2013), where MSE is the sum of the squared bias and variance. The data used in our analyses is freely available online within File S1 of Pemberton et al. (2013) (http://www.g3journal.org/content/early/2013/03/27/g3.113.005728/suppl/DC1). We took the sample allele frequencies calculated from all individuals in the MS5795 dataset as the true population allele frequencies for the variance calculations, and, from these, determined the true expected heterozygosity at each locus using Equation 1 (see File S1; incorporated into Equation A10). Here, each sample contained 60 diploid individuals composed of 10 inbred full-sibling, 10 outbred full-sibling, and 10 outbred avuncular pairs. Each point in Figure 1 and Figure S1 represents a single analytical computation for a sample of 60 (or 30 for ${\widehat{H}}_{\text{red}}$) individuals at a microsatellite locus. We report the approximate variance and MSE because each individual is related to exactly one other in the sample, satisfying the assumption of Equation 11. Further, under this scenario DeGiorgio et al. (2010) showed that this was a reasonable approximation of the exact variance.

Figure 1.

Theoretical difference in MSE between the unbiased estimator ${\widehat{H}}_{\text{red}}$ (left), $\stackrel{\sim }{H}$ (center), or ${\stackrel{\sim }{H}}_{\text{BLUE}}$ (right), and the biased estimator ${\widehat{H}}_{\text{full}}$ calculated at each of 645 microsatellite loci ($0.5212\le H\le 0.9301$) in the MS5795 dataset for samples of 60 diploid individuals containing some inbred relative pairs. Each sampled individual was related to exactly one other, and samples contained 10 pairs of inbred full-siblings ($\Phi =3/8$), 10 pairs of outbred full-siblings ($\Phi =1/4$), and 10 outbred avuncular pairs ($\Phi =1/8$). Dotted lines in each plot correspond to a difference in MSE of zero with ${\widehat{H}}_{\text{full}}.$ See File S1 for the true expected heterozygosity values incorporated into analytical calculations.

We begin by demonstrating the relative performance of the unbiased estimators ${\widehat{H}}_{\text{red}},$ $\stackrel{\sim }{H},$ and ${\stackrel{\sim }{H}}_{\text{BLUE}},$ measured in terms of MSE, against the biased estimator ${\widehat{H}}_{\text{full}}$ (Figure 1). While the variance of ${\widehat{H}}_{\text{full}}$ is invariably smaller than that of the other estimators, and the MSE and variance of each estimator decrease with increasing locus expected heterozygosity ($0.5212\le H\le 0.9301$), ${\widehat{H}}_{\text{full}}$ accumulates bias quadratically with increasing H, and thus yields an increasingly unreliable estimate with increasing site diversity (Figure S1A, left). However, the effect of this trend differs for each comparison. The MSE of ${\widehat{H}}_{\text{red}}$ always exceeds that of ${\widehat{H}}_{\text{full}},$ because the removal of relatives to create the reduced sample causes a substantial increase in estimator variance, though, for high diversity markers, the MSE values of ${\widehat{H}}_{\text{full}}$ and ${\widehat{H}}_{\text{red}}$ converge (Figure 1, left). In contrast, $\stackrel{\sim }{H}$ outperforms ${\widehat{H}}_{\text{full}}$ for most loci, demonstrating that the rate of decrease in MSE with increasing H is greater for $\stackrel{\sim }{H}$ than for ${\widehat{H}}_{\text{full}}$ (Figure 1, center). Interestingly, the comparison of ${\stackrel{\sim }{H}}_{\text{BLUE}}$ with ${\widehat{H}}_{\text{full}}$ shows an opposite trend to the preceding two. Despite the impact of bias, the decrease in variance of ${\widehat{H}}_{\text{full}}$ over the analyzed range outpaces that of ${\stackrel{\sim }{H}}_{\text{BLUE}}.$ Even so, ${\stackrel{\sim }{H}}_{\text{BLUE}}$ uniformly yields a smaller MSE for the analyzed diploid samples (which contain a proportion of inbred individuals) across all loci (Figure 1, right).

To validate these theoretical predictions, we simulated 30 independent genotypes for each locus, and, for each independent genotype, simulated a single relative’s genotype (inbred full-sibling, outbred full-sibling, or avuncular). Briefly, we generated the independent genotypes by sampling alleles uniformly at random from the distribution of allele frequencies at each microsatellite locus, and generated relatives by copying zero, one, or two alleles from the relative according to the probability the pair would share zero, one, or two alleles IBD [see Lange (2002), Chapter 5]. The patterns observed for the simulated data accord with those of the theoretical predictions (Figure S2, each point is based on ${10}^4$ simulations). It is clear from these results that locus expected heterozygosity is heavily influential on estimator MSE. However, we also find that the observed value of expected heterozygosity for a locus normalized to its range of expected heterozygosity values has an impact on estimator MSE. The maximum and minimum values of expected heterozygosity for a locus depend on the number of distinct alleles (I), and the frequency of the most frequent allele (M), at that locus [see Theorem 2 of Reddy and Rosenberg (2012)]. We quantify proximity of H for a locus to its maximum possible value as $B=D/R,$ where D is the observed value of expected heterozygosity for a locus minus its minimum possible value given I and M, and R is the maximum minus the minimum value of expected heterozygosity, given I and M, such that $B\in \left[0,1\right].$ Loci with a smaller value of B yield a smaller MSE for all estimators (Figure S3).

Effect of sample size, n, on estimators

We next examined the properties of each estimator as a function of sample size. All estimators perform increasingly well for samples of increasing size. We demonstrate this property by measuring estimator MSE for samples containing 2–100 relative pairs of various type and ploidy at the D3S2427 locus, selected to highlight the improved performance of ${\stackrel{\sim }{H}}_{\text{BLUE}}$ as the bias of ${\widehat{H}}_{\text{full}}$ increases ($H=0.9301;$ Figure 2). For these tests, we considered only a single relative pair type at a time. The unbiased estimators $\stackrel{\sim }{H}$ and ${\stackrel{\sim }{H}}_{\text{BLUE}}$ perform identically for diploid samples of first- and second-degree relative pairs regardless of inbreeding (Figure 2, A–D). Additionally, estimator MSE is uniformly smaller for samples containing only second-degree relative pairs than it is for samples containing only first-degree pairs (cf. Figure 2, A and B, and Figure 2, C and D; see also, Figure S4A). However, ${\stackrel{\sim }{H}}_{\text{BLUE}}$ unambiguously outperforms the other estimators with relative pairs of varying ploidy (in this case, male-female full-sibling pairs at an X-linked locus). In this scenario, ${\widehat{H}}_{\text{red}}$ provides a more accurate estimate of expected heterozygosity than does $\stackrel{\sim }{H}$ when the reduced set is created by removing only males from the original while retaining females (Figure 2E). When all females are removed instead, and males retained (Figure 2F), the MSE of ${\widehat{H}}_{\text{red}}$ is markedly the largest of the four estimators because 2/3 of the alleles in the sample are discarded, rather than 1/3. For samples with inbred full-siblings whose parents are brother and sister (Figure 2, C and D), the trend of MSE with sample size mirrors that of outbred diploid samples (Figure 2, A and B), but with larger MSE. However, the relative performance of ${\widehat{H}}_{\text{full}}$ is notably worse for samples containing inbred diploid avuncular pairs (Figure 2D) than for samples containing outbred diploid avuncular pairs (Figure 2B). That is, its MSE remains greater than, or equal to, that of the other estimators over the range of sample sizes considered for the inbred diploid avuncular pair scenario (Figure 2D), but consistently has smaller MSE than ${\widehat{H}}_{\text{red}}$ for the outbred diploid avuncular pair scenario (Figure 2B). Generally, increasing the sample size is most effective for samples of <20 individuals, and it is over this range that the difference in performance of the estimators is most apparent.

Figure 2.

Theoretical MSE as a function of sample size for samples of outbred diploid full-siblings (A), outbred diploid avuncular pairs (B), inbred diploid full-siblings (C), inbred diploid avuncular pairs (D), male-female full siblings at an X-linked locus with the reduced set omitting males and retaining females (E), and male-female full siblings at an X-linked locus with the reduced set omitting females and retaining males (F). The samples were evaluated for the D3S2427 locus ($H=0.9301$), and sample size was always twice the number of relative pairs included in the sample for samples containing 2–100 relative pairs. Each individual in the sample was related to exactly one other.

Effect of varying sample relative pair composition on estimators

Finally, we calculated the MSE of each estimator for all 1326 combinations of one to three relative pair types for samples of 100 individuals fixed at 50 relative pairs, which we represent as triangular heat maps, across samples containing outbred diploids, males and females at an X-linked locus, or inbred diploids (each individual related to exactly one other; Figure 3, Figure S4, Figure S5, Figure S6, Figure S7, and Figure S8). The kinship coefficients ($\Phi$) for each relative pair type considered across our tests are defined in Lange (2002, Chapter 5) and DeGiorgio et al. (2010, see Table 2), and modeled on the D3S2427 locus ($H=0.9301$).

Figure 3.

Theoretical difference in MSE between ${\widehat{H}}_{\text{full}}$ (left), ${\widehat{H}}_{\text{red}}$ (center), or $\stackrel{\sim }{H}$ (right), and ${\stackrel{\sim }{H}}_{\text{BLUE}},$ for samples of 100 (A) outbred diploid individuals, (B) male and female individuals at an X-linked locus, or (C) diploid individuals wherein some full siblings are inbred with brother-sister parents. The samples and MSE values considered for each subtraction were modeled on the D3S2427 locus ($H=0.9301$). Each sample contained 50 relative pairs, such that each individual was related to exactly one other. Each sample configuration is a single point in the space of a heat map defined by three coordinates (each representing the count of a relative pair type). For each configuration, the MSE of ${\stackrel{\sim }{H}}_{\text{BLUE}}$ is subtracted from that of the other estimators, yielding a value >0. Samples were composed of one to three relative pair types where the vertex of each heat map represents a sample with only a single relative pair type. The relative pair types were (A) parent-offspring (PO), second-degree avuncular (AV), and full-sibling (FS), (B) male-male (MM), male-female (MF), and female-female (FF) full-sibling such that the number of males and females in each sample is not fixed, or (C) inbred full-sibling (FSi), second-degree avuncular (AV), and outbred full-sibling (FSo). Blue and black points indicate the smallest and largest values, respectively, on each map. Threshold values for coloration are indicated in the scales to the right of each heat map, with smaller values colored lighter. Note that the scales are not identical across heat maps. The values upon which these subtractions are based are represented as heat maps in (A) Figure S4A, (B) Figure S4B, or (C) Figure S4C.

Table 2. Wilcoxon signed-rank test for weighted mean across all loci of ${\widehat{F}}_{\text{ST},\text{red}}$ with ${\widehat{F}}_{\text{ST}}$ and ${\stackrel{\sim }{F}}_{\text{ST},\text{BLUE}}$ for the French population with the 92 other populations whose samples contained related individuals.

Comparison	P-Value for Wilcoxon Signed-Rank Test
${\widehat{F}}_{\text{ST},\text{red}}$ with ${\widehat{F}}_{\text{ST}}$	$5.25\times {10}^{-15}$
${\widehat{F}}_{\text{ST},\text{red}}$ with ${\stackrel{\sim }{F}}_{\text{ST},\text{BLUE}}$	0.967

The outbred diploid samples included parent-offspring ($\Phi =1/4$), avuncular ($\Phi =1/8$), and full-sibling ($\Phi =1/4$) relative pairs. Because parent-offspring and full-sibling pairs have the same kinship coefficient, the heat maps in Figure 3, Figure S4A, Figure S5A, Figure S6A, and Figure S7A are symmetrical with parent-offspring and full-sibling pairs on the bottom vertices, and avuncular pairs on the top vertex. ${\widehat{H}}_{\text{red}}$ yielded the largest MSE of the four estimators, and this value was constant throughout the space of the heat map (Figure S4A, second triangle), because all reduced sets are identical for outbred diploid samples. ${\stackrel{\sim }{H}}_{\text{BLUE}}$ consistently yielded the smallest MSE across configurations (Figure S4A, fourth triangle). As was the case in Figure 2, the MSE of the estimators ${\widehat{H}}_{\text{full}},$ $\stackrel{\sim }{H},$ and ${\stackrel{\sim }{H}}_{\text{BLUE}}$ was smallest for samples with only avuncular pairs, because these contain fewer dependent allele observations on average. We observed these features in simulated data as well (Figure S8A).

Although ${\stackrel{\sim }{H}}_{\text{BLUE}}$ performed best overall for samples including outbred diploid relative pairs at D3S2427, the estimator with the smallest variance in all situations is the biased estimator ${\widehat{H}}_{\text{full}}$ (Figure S6A). However, because its squared bias increases with the number of first-degree pairs (Figure S5A), its relative performance declines compared to ${\stackrel{\sim }{H}}_{\text{BLUE}}$ as more of these pairs are sampled (Figure 3A, left triangle). The relative performance of ${\widehat{H}}_{\text{red}}$ is highest when the number of first degree pairs is maximized, but this is due to the decreasing performance of ${\stackrel{\sim }{H}}_{\text{BLUE}}$ as more dependent observations are included (Figure 3A, center triangle). While the difference in MSE between $\stackrel{\sim }{H}$ and ${\stackrel{\sim }{H}}_{\text{BLUE}}$ is always slight for samples of noninbred diploids, these values diverge as the complexity of the sample increases (Figure 3A, right triangle). That is, as the numbers of first- and second-degree pairs approach each other, ${\stackrel{\sim }{H}}_{\text{BLUE}}$ emerges decisively as the more accurate estimator, with the maximum value of this difference reached at 23 second-degree and 27 first-degree pairs. Thus, while the performance of the estimators for a sample containing relatives follows the same general trend, ${\stackrel{\sim }{H}}_{\text{BLUE}}$ provides the greatest accuracy for heterogeneous samples of outbred diploid individuals.

We also considered the relative performance of each estimator when using either the BLUE (${\stackrel{\sim }{p}}_i$) or the sample proportion (${\widehat{p}}_i$) to estimate allele frequencies. Notably, all estimators perform best when the BLUE (${\stackrel{\sim }{p}}_i$) of allele frequency rather than the sample proportion (${\widehat{p}}_i$) is used to infer population allele frequencies. We calculated the theoretical MSE for each estimator once with ${\widehat{p}}_i,$ and once with ${\stackrel{\sim }{p}}_i,$ across all combinations of relative pairs for diploid individuals at the D3S2427 locus and mapped its value for the estimate with ${\widehat{p}}_i$ minus the estimate with ${\stackrel{\sim }{p}}_i$ (Figure S7A). Because both frequency estimations yield the same values in samples of unrelated individuals, ${\widehat{H}}_{\text{red}}$ performs identically for ${\widehat{p}}_i$ and ${\stackrel{\sim }{p}}_i,$ and is not included. The MSE of an estimator calculated with ${\stackrel{\sim }{p}}_i$ is always smaller than that of the estimator calculated with ${\widehat{p}}_i,$ and the pattern of divergence between their MSEs follows a similar trend across all estimators, resembling the rightmost panel in Figure 3A. This result suggests that the difference in MSE between $\stackrel{\sim }{H}$ and ${\stackrel{\sim }{H}}_{\text{BLUE}}$ is driven primarily by the difference in performance between ${\widehat{p}}_i$ and ${\stackrel{\sim }{p}}_i.$ Both the ${\widehat{p}}_i$ and ${\stackrel{\sim }{p}}_i$ estimators yield the same value at the vertices of the triangles, and the difference in their MSEs reaches a maximum at 22 second-degree pairs for ${\widehat{H}}_{\text{full}}$ and 24 second-degree pairs for $\stackrel{\sim }{H}$ and ${\stackrel{\sim }{H}}_{\text{BLUE}}$ (Figure S7A, center and right triangles). The MSE of ${\stackrel{\sim }{H}}_{\text{BLUE}}$ calculated with ${\widehat{p}}_i$ is, at most, on the order of ${10}^{-9}$ greater than that of ${\stackrel{\sim }{H}}_{\text{BLUE}}$ calculated with ${\stackrel{\sim }{p}}_i,$ indicating its robustness to variance in allele frequency determination (Figure S7A, right triangle). In contrast, the other estimators return a maximum difference in MSE on the order of ${10}^{-7}.$ The estimation of expected heterozygosity with ${\widehat{H}}_{\text{full}},$ $\stackrel{\sim }{H},$ or ${\stackrel{\sim }{H}}_{\text{BLUE}}$ will always yield a smaller MSE for samples of outbred, diploid individuals when ${\stackrel{\sim }{p}}_i$ rather than ${\widehat{p}}_i$ is taken as the estimator of population allele frequency.

We repeated these tests in samples of mixed ploidy (Figure 3B, Figure S4B, Figure S5B, Figure S6B, Figure S7B, and Figure S8B), and ${\stackrel{\sim }{H}}_{\text{BLUE}}$ emerged similarly superior to the other estimators, once again yielding the smallest MSE. We analyzed the D3S2427 locus as X-linked for these tests, counting males as haploid and females as diploid, and observed full-sibling pairs [similarly to DeGiorgio et al. (2010), $\Phi =1/2$ for male-male pairs, $\Phi =1/4$ for male-female pairs, and $\Phi =3/8$ for female-female pairs] for samples of 100 individuals and 50 relative pairs. All estimators reach their maximum MSE in samples containing only male-male pairs (Figure S4B). This is because the number of independent observations (indicated by a larger mean kinship coefficient) is smallest when there are no females in the sample. Correspondingly, the estimators yield smaller MSE values with increasing incorporation of male-female pairs. The minimum MSE of ${\widehat{H}}_{\text{full}}$ is reached at 50 male-female pairs, as with $\stackrel{\sim }{H}$ and ${\stackrel{\sim }{H}}_{\text{BLUE}}$ because its squared bias (Figure S5B) decreases with increasing male-female pairs, though its variance is smallest at 50 female-female pairs, due to the greater number of alleles in the sample (Figure S6B). To create the reduced sets, males were removed from male-female pairs to minimize the subsequent increase in MSE. That is, the removal of males removes 1/3 of the allele copies from the sample, rather than 2/3 if females are removed, or 1/2 for a pair of same-ploidy individuals, and so ${\widehat{H}}_{\text{red}}$ has the same value across samples with the same number of male-male pairs (Figure S4B, second triangle).

The direct comparison of ${\stackrel{\sim }{H}}_{\text{BLUE}}$ with the other estimators once again yielded different signatures for each subtraction for mixed-ploidy samples (Figure 3B). The point of greatest difference in MSE between ${\widehat{H}}_{\text{full}}$ and ${\stackrel{\sim }{H}}_{\text{BLUE}}$ occurs when all relative pairs are male-male, while the point of least difference occurs for samples of only male-female pairs (Figure 3B, left triangle). This pattern broadly resembles the squared bias of ${\widehat{H}}_{\text{full}}$ (Figure S5B, first triangle), underscoring the effect of bias on estimator performance. The pattern of difference in performance between ${\widehat{H}}_{\text{red}}$ and ${\stackrel{\sim }{H}}_{\text{BLUE}}$ differs markedly, and the two estimators perform most similarly as the number of male-male pairs decreases, reaching a minimum at 33 male-female pairs plus 17 female-female pairs (Figure 3B, middle triangle). $\stackrel{\sim }{H}$ yields the closest MSE to that of ${\stackrel{\sim }{H}}_{\text{BLUE}}$ for all relative pair configurations, and their difference is, at most, on the order of ${10}^{-6}$ (Figure 3B, right triangle). The pattern here mainly reflects the difference in performance between ${\widehat{p}}_i$ and ${\stackrel{\sim }{p}}_i$ estimates of population allele frequency, as in Figure S7B, where ${\stackrel{\sim }{p}}_i$ estimators yield increasingly smaller comparative MSE values as the numbers of relative pairs in the sample approach each other.

We repeated the preceding tests once more for a sample in which full-siblings resulting from a brother-sister mating were included alongside second-degree and outbred full-sibling pairs (Figure 3C, Figure S4C, Figure S5C, Figure S6C, Figure S7C, and Figure S8C). Here, the kinship of inbred individuals with each other was 3/8 rather than 1/4. For all estimators, the inclusion of inbred full-siblings increased the MSE of the estimator, with a maximum MSE at 50 inbred full-sibling pairs, and a minimum at 50 second-degree pairs. For ${\widehat{H}}_{\text{red}},$ this minimum was also reached for any sample in which there were no inbred individuals, because the reduced sample is identical for these (Figure S4C, second triangle). Again, ${\stackrel{\sim }{H}}_{\text{BLUE}}$ was the least errant estimator across the space of sample configurations (Figure S4C, fourth triangle), and its advantage over the other estimators differs for each estimator (Figure 3C). Because the bias of ${\widehat{H}}_{\text{full}}$ is largest at 50 inbred full-sibling pairs, the greatest difference in performance between it and ${\stackrel{\sim }{H}}_{\text{BLUE}}$ is at this point (Figure 3C, left triangle). Meanwhile, the largest differences in MSE between ${\widehat{H}}_{\text{red}}$ and ${\stackrel{\sim }{H}}_{\text{BLUE}}$ are near the top vertex, where second-degree relative pairs predominate, while the smallest are toward the bottom vertices (Figure 3C, center triangle). The difference in MSE between $\stackrel{\sim }{H}$ and ${\stackrel{\sim }{H}}_{\text{BLUE}}$ is at least an order of magnitude less than for the other comparisons, and increases for increasing sample complexity, but reaches its maximum for samples of 28 inbred full-sibling plus 22 second-degree pairs (Figure 3C, right triangle). This pattern reflects the decreased MSE for the estimators when calculated with ${\stackrel{\sim }{p}}_i$ compared to their calculation with ${\widehat{p}}_i$ (Figure S7C). Ultimately, the performance of the estimators of expected heterozygosity across varying sample compositions depends on the estimator of allele frequency incorporated into the expected heterozygosity calculation. No matter the sample type, estimators based on ${\stackrel{\sim }{p}}_i$ outperform estimators based on ${\widehat{p}}_i,$ and ${\stackrel{\sim }{H}}_{\text{BLUE}}$ outperforms ${\widehat{H}}_{\text{full}},$ ${\widehat{H}}_{\text{red}},$ and $\stackrel{\sim }{H}.$

Tests of ${\stackrel{\sim }{\text{H}}}_{\text{BLUE}}$ on single-nucleotide polymorphism (SNP) loci

Because SNP datasets are more common in recent studies, we performed analyses equivalent to our microsatellite analyses for 50 hypothetical SNP loci. These loci were biallelic with minor allele frequency (MAF) between 0.01 and 0.5, with increments of 0.01, corresponding to expected heterozygosity values ranging from 0.0198 to 0.5. We first measured the difference in MSE of ${\widehat{H}}_{\text{full}}$ with that of ${\widehat{H}}_{\text{red}},$ $\stackrel{\sim }{H},$ or ${\stackrel{\sim }{H}}_{\text{BLUE}}$ as a function of true locus expected heterozygosity (H), as we did in Figure 1 (Figure S9). For each locus, the MSE of ${\stackrel{\sim }{H}}_{\text{BLUE}}$ was smallest, while that of ${\widehat{H}}_{\text{full}}$ was generally second-smallest, following the trend for microsatellite loci visible in Figure 1, wherein less diverse loci yielded a smaller MSE for ${\widehat{H}}_{\text{full}}$ than for $\stackrel{\sim }{H}.$ However, unlike for microsatellite loci, estimator MSE peaks midway through the range of evaluated SNP loci, such that the smallest MSE values lie at either extreme of the range and the largest MSE value, as well as the largest difference in MSE values for all comparisons, is at the locus with MAF$=0.15$ ($H=0.255$). Additionally, ${\widehat{H}}_{\text{full}}$ performs comparatively better than ${\widehat{H}}_{\text{red}}$ (Figure S9, left) and $\stackrel{\sim }{H}$ (Figure S9, center) as H approaches 0.255, but is outperformed by these unbiased estimators as H approaches 0.5. Once more, the trend is opposite for the comparison between ${\widehat{H}}_{\text{full}}$ and ${\stackrel{\sim }{H}}_{\text{BLUE}},$ showing the greatest comparative performance by ${\stackrel{\sim }{H}}_{\text{BLUE}}$ at the same locus (MAF $=0.15,$ $H=0.255$). Thus, considering the results presented in Figure 1 and Figure S9, the greatest relative performance of ${\stackrel{\sim }{H}}_{\text{BLUE}}$ for inbred samples is achieved at loci for which estimator MSE is largest.

We next examined the effect of sample size on estimator performance for hypothetical samples of outbred diploid, inbred diploid, and outbred male-female relative pairs at the simulated locus with MAF $=0.05$ ($H=0.095$). As we varied the sample size from two relative pairs to 100 (each individual related to exactly one other, one relative pair type per sample), we found that ${\stackrel{\sim }{H}}_{\text{BLUE}}$ yielded the smallest MSE of all estimators only for samples containing male-female full-sibling pairs modeled at an X-linked locus (Figure S10, E and F). This observation mirrors the trend seen in Figure 2, wherein ${\stackrel{\sim }{H}}_{\text{BLUE}}$ outperformed the other estimators across all sample sizes. However, ${\widehat{H}}_{\text{full}}$ yielded the smallest MSE across all sample sizes for outbred and inbred diploid full-siblings and avuncular pairs (Figure S10, A–D). This result is because the samples modeled here are minimally complex, with only one relative pair type, and modeled for a highly homozygous marker—two conditions under which the low bias and variance of ${\widehat{H}}_{\text{full}}$ result in favorable performance.

Finally, we analyzed estimator performance once more for the locus with MAF $=0.05$ ($H=0.095$), for a sample of 50 individuals across changing outbred diploid, inbred diploid, and male-female full-sibling relative pair compositions (Figure S11, A–C). We display these results as heat maps, and find that our results here are broadly concordant with those for the D3S2427 human microsatellite locus ($H=0.9301$). As with the experiments displayed in Figure S10, the least complex samples yielded a smaller MSE for ${\widehat{H}}_{\text{full}}$ estimates than for ${\stackrel{\sim }{H}}_{\text{BLUE}}$ estimates. Correspondingly, samples whose relative pair compositions resulted in fewer independent allele observations were more accurately and precisely evaluated with ${\stackrel{\sim }{H}}_{\text{BLUE}}.$ Thus, while sampling lower-diversity markers may occasionally favor the use of ${\widehat{H}}_{\text{full}},$ the inclusion of two or more relative pair types in the sample is likely to bias ${\widehat{H}}_{\text{full}}$, and require the use of ${\stackrel{\sim }{H}}_{\text{BLUE}}$ to yield accurate inferences.

Empirical application of ${\stackrel{\sim }{\text{H}}}_{\text{BLUE}}$

To conclude our investigation into the performance of ${\stackrel{\sim }{H}}_{\text{BLUE}},$ we applied it to empirical data from the MS5795 dataset. We retrieved human microsatellite data from 5795 individuals (11,590 allele copies) across 645 autosomal loci sampled genome wide. We assumed the mean value across loci for ${\widehat{H}}_{\text{red}}$ in each of 267 populations to be the true expected heterozygosity value for these populations, as it is an unbiased estimate. We additionally chose to compare the other estimators with ${\widehat{H}}_{\text{red}},$ because an important basis for their evaluation is their agreement with this unbiased estimator, irrespective of the data to which they are applied.

To emphasize this, we performed three Wilcoxon signed-rank tests to compare the ranking of populations by their mean expected heterozygosity across all loci calculated with ${\widehat{H}}_{\text{red}},$ and either ${\widehat{H}}_{\text{full}},$ $\stackrel{\sim }{H},$ or ${\stackrel{\sim }{H}}_{\text{BLUE}}$ (Table 1). At the $\alpha <0.01$ significance level, the comparisons showed that the inclusion of relatives for ${\widehat{H}}_{\text{full}}$ was highly significant on the rankings it yielded, indicating that not correcting for relatedness among samples can significantly alter the estimates of expected heterozygosity. However, both $\stackrel{\sim }{H}$ and, especially, ${\stackrel{\sim }{H}}_{\text{BLUE}},$ yielded P-values greater than the threshold for the test against ${\widehat{H}}_{\text{red}}.$ These results indicate that the estimates of expected heterozygosity are not significantly affected by the inclusion of related individuals in the sample when relatedness is taken into account. Furthermore, a test between $\stackrel{\sim }{H}$ and ${\stackrel{\sim }{H}}_{\text{BLUE}}$ yielded a P-value of $3.44\times {10}^{-2},$ suggesting no significant difference in the ranking of populations by mean expected heterozygosity with these two estimators.

Table 1. Wilcoxon signed-rank test for mean across loci of ${\widehat{H}}_{\text{red}}$ with ${\widehat{H}}_{\text{full}},$ $\stackrel{\sim }{H},$ and ${\stackrel{\sim }{H}}_{\text{BLUE}}$ for the 93 populations whose samples contained related individuals.

Comparison	P-Value for Wilcoxon Signed-Rank Test
${\widehat{H}}_{\text{red}}$ with ${\widehat{H}}_{\text{full}}$	$4.39\times {10}^{-15}$
${\widehat{H}}_{\text{red}}$ with $\stackrel{\sim }{H}$	$1.00\times {10}^{-2}$
${\widehat{H}}_{\text{red}}$ with ${\stackrel{\sim }{H}}_{\text{BLUE}}$	0.255

Although the unbiased estimators $\stackrel{\sim }{H}$ and ${\stackrel{\sim }{H}}_{\text{BLUE}}$ have smaller MSE than ${\widehat{H}}_{\text{full}}$ for samples with related individuals, their variance tends to be larger than that of ${\widehat{H}}_{\text{full}}.$ DeGiorgio et al. (2010) previously showed that the difference in SD of $\stackrel{\sim }{H}$ with ${\widehat{H}}_{\text{full}}$ was small, while the mean values of $\stackrel{\sim }{H}$ and ${\widehat{H}}_{\text{red}}$ were much more similar to each other than either of them was to the mean of ${\widehat{H}}_{\text{full}}.$ We again show this to be the case, and find as well that ${\stackrel{\sim }{H}}_{\text{BLUE}}$ not only repeats, or improves upon, the concordance of $\stackrel{\sim }{H}$ with ${\widehat{H}}_{\text{red}},$ but, in some cases, ${\stackrel{\sim }{H}}_{\text{BLUE}}$ has a smaller SD than does ${\widehat{H}}_{\text{full}}$ (Figure 4, left and center panels). A direct comparison of the performance of $\stackrel{\sim }{H}$ against that of ${\stackrel{\sim }{H}}_{\text{BLUE}}$ (Figure 4, right panel) shows that ${\stackrel{\sim }{H}}_{\text{BLUE}}$ has a generally improved SD, and similarity to the ${\widehat{H}}_{\text{red}}$ estimate over $\stackrel{\sim }{H}.$ For some samples (primarily those from the Americas), this is not the case, possibly because all close relatives were not identified in the original dataset, resulting in an incorrect kinship matrix for calculation of the statistic.

Figure 4.

Application of the estimators to dataset MS5795. Here, we show a comparison of two estimators at a time (${\widehat{H}}_{\text{full}},$ $\stackrel{\sim }{H},$ or ${\stackrel{\sim }{H}}_{\text{BLUE}}$) by the difference in their mean with that of ${\widehat{H}}_{\text{red}}$ across the 645 sampled microsatellite loci of MS5795 (vertical axis), and by their SDs (horizontal axis). The horizontal dotted line corresponds to no difference between the mean of the estimator and the mean of the unbiased estimator ${\widehat{H}}_{\text{red}}.$ Solid lines connect calculations made for the same population with different estimators. Points are colored by geographic division defined in the dataset. Only the 93 populations with relatives in their samples were included because ${\widehat{H}}_{\text{full}},$ $\stackrel{\sim }{H},$ and ${\stackrel{\sim }{H}}_{\text{BLUE}}$ return the same value for samples of unrelated individuals. In the leftmost plot, open points are estimates for ${\widehat{H}}_{\text{full}},$ while closed points are for $\stackrel{\sim }{H}.$ In the center plot, open points are estimates for ${\widehat{H}}_{\text{full}},$ while closed points are for ${\stackrel{\sim }{H}}_{\text{BLUE}}.$ In the rightmost plot, open points are estimates for $\stackrel{\sim }{H},$ while closed points are for ${\stackrel{\sim }{H}}_{\text{BLUE}}.$

Improving estimates of ${\text{F}}_{\text{ST}}$ by application of ${\stackrel{\sim }{\text{H}}}_{\text{BLUE}}$

We predicted that the smaller MSE of ${\stackrel{\sim }{H}}_{\text{BLUE}}$ would translate to improved accuracy for estimators that are summaries of expected heterozygosity when samples contain related individuals. To test this hypothesis, we calculated the population differentiation statistic, $F_{\text{ST}}$ (Equation 4), for pairs of populations whose samples in the MS5795 dataset contained related individuals. Our intent was to compare the MSE and bias of the commonly used $F_{\text{ST}}$ estimator of Reynolds et al. (1983), which is based on ${\widehat{H}}_{\text{full}},$ and which we label as ${\widehat{F}}_{\text{ST}},$ to an estimate of $F_{\text{ST}}$ calculated from ${\stackrel{\sim }{H}}_{\text{BLUE}},$ which we label ${\stackrel{\sim }{F}}_{\text{ST},\text{BLUE}}.$ The formulas for these estimators follow the form of the general estimator of $F_{\text{ST}}$ (Equation 14). We first measured the MSE of both methods (and an estimate using $\stackrel{\sim }{H},$ ${\stackrel{\sim }{F}}_{\text{ST}}$) on simulated data, where the $F_{\text{ST}}$ of pairs of populations with samples of size 60 diploids each (30 relative pairs, 10 inbred full-sibling, 10 outbred full-sibling, and 10 avuncular pairs; Figure 5) was averaged across ${10}^4$ simulated replicates. The calculations included here were performed for simulated Gujarati and Maya (left), Gujarati and Japanese (center), or Gujarati and Hadza (right) samples for the least diverse (TCTA015M_22), median diverse (D10S2327), and most diverse (D3S2427) loci of the MS5795 dataset, following their allele frequency distribution in MS5795. ${\stackrel{\sim }{F}}_{\text{ST},\text{BLUE}}$ consistently has a smaller MSE than the others, and the MSE of all estimators of $F_{\text{ST}}$ decreases with increasing locus diversity, as the MSE of the estimator of expected heterozygosity decreases.

Figure 5.

Application of the estimators ${\widehat{H}}_{\text{full}},$ $\stackrel{\sim }{H},$ and ${\stackrel{\sim }{H}}_{\text{BLUE}}$ to the calculation of $F_{\text{ST}}$ as ${\widehat{F}}_{\text{ST}},$ ${\stackrel{\sim }{F}}_{\text{ST}},$ and ${\stackrel{\sim }{F}}_{\text{ST},\text{BLUE}},$ respectively, using simulated data for the Gujarati sample, with either the Maya (left), Japanese (center), or Hadza (right) samples, showing MSE on the vertical axis. The Reynolds et al. (1983) estimator is equivalent to the application of ${\widehat{H}}_{\text{full}}$ in calculating population differentiation. The simulated samples contained 60 individuals and 30 relative pairs, of which 10 were inbred full-siblings, 10 were outbred full-siblings, and 10 were outbred avuncular pairs. Each individual was related to exactly one other, and the data were simulated following the same probabilistic method as employed to generate Figure S2. The three loci displayed on the horizontal axis are the least diverse, median diverse, and most diverse loci of the 645 MS5795 human microsatellites.

We additionally find that ${\widehat{F}}_{\text{ST}}$ has an upward bias compared with ${\widehat{F}}_{\text{ST},\text{red}}$ (calculated with ${\widehat{H}}_{\text{red}}$), as well as a larger SD in general than ${\stackrel{\sim }{F}}_{\text{ST},\text{BLUE}}$ (Figure 6). Furthermore, all values of ${\stackrel{\sim }{F}}_{\text{ST},\text{BLUE}}$ are smaller than the paired value of ${\widehat{F}}_{\text{ST}}$ calculated for the same population. The difference in the mean of ${\widehat{F}}_{\text{ST}}$ and of ${\stackrel{\sim }{F}}_{\text{ST},\text{BLUE}}$ across all loci with the mean of ${\widehat{F}}_{\text{ST},\text{red}},$ an estimator which serves as a proxy for the true value of $F_{\text{ST}},$ is displayed on the vertical axis, while the horizontal axis measures the SD of ${\widehat{F}}_{\text{ST}}$ and of ${\stackrel{\sim }{F}}_{\text{ST},\text{BLUE}}$ (Figure 6). Supporting our observations indicating the improved accuracy of ${\stackrel{\sim }{F}}_{\text{ST},\text{BLUE}}$ over ${\widehat{F}}_{\text{ST}},$ Wilcoxon signed-rank tests (Table 2) between ${\widehat{F}}_{\text{ST},\text{red}}$ and either ${\widehat{F}}_{\text{ST}}$ or ${\stackrel{\sim }{F}}_{\text{ST},\text{BLUE}}$ indicate that the inclusion of relatives significantly affects the estimate of population differentiation at the $\alpha <0.01$ significance level. Meanwhile, ${\widehat{F}}_{\text{ST},\text{red}}$ and ${\stackrel{\sim }{F}}_{\text{ST},\text{BLUE}}$ are not significantly different in their estimates. These results suggest that the improved properties of ${\stackrel{\sim }{H}}_{\text{BLUE}}$ transfer to the summaries that include it in their calculations.

Figure 6.

Application of the estimators ${\stackrel{\sim }{H}}_{\text{BLUE}}$ and ${\widehat{H}}_{\text{full}}$ to the estimation of $F_{\text{ST}}$ as ${\widehat{F}}_{\text{ST}}$ and ${\stackrel{\sim }{F}}_{\text{ST},\text{BLUE}},$ respectively, from empirical data. Similarly to Figure 4, the difference between the mean of the estimator of $F_{\text{ST}}$ (either derived from ${\stackrel{\sim }{H}}_{\text{BLUE}}$ or ${\widehat{H}}_{\text{full}}$) and an unbiased estimator (derived from ${\widehat{H}}_{\text{red}}$), is displayed on the vertical axis, while the SD of the estimator is displayed on the horizontal axis. The empty circles represent the Reynolds et al. (1983) estimator (identical to the ${\widehat{H}}_{\text{full}}$-derived estimation), while the filled circles represent the estimation derived from ${\stackrel{\sim }{H}}_{\text{BLUE}}.$ Here, the $F_{\text{ST}}$ values for the French sample with each of the 92 other samples containing related individuals in the dataset MS5795 are plotted, colored by the region of the changing sample.

Discussion

We have introduced ${\stackrel{\sim }{H}}_{\text{BLUE}},$ an extension to the estimator ($\stackrel{\sim }{H}$) of expected heterozygosity developed by DeGiorgio et al. (2010) that yields a smaller mean squared error in samples containing related individuals, while maintaining unbiasedness. Conveniently, the derivations of ${\stackrel{\sim }{H}}_{\text{BLUE}},$ and its variance, are parallel in form to those of $\stackrel{\sim }{H},$ and we were therefore able to analytically evaluate the performance of the new estimator simultaneously with that of its predecessor. Our updated estimator, ${\stackrel{\sim }{H}}_{\text{BLUE}},$ is based on results from McPeek et al. (2004), who characterized the BLUE (${\stackrel{\sim }{p}}_i$) of allele frequency. The BLUE improves the precision of allele frequency estimation in complex pedigrees, for which the sample proportion (${\widehat{p}}_i,$ the estimator of allele frequency used in $\widehat{H}$ and $\stackrel{\sim }{H}$) is unbiased, but increases in variance with inclusion of related and inbred individuals. Because the properties of the estimator of allele frequency transfer to the estimator of expected heterozygosity, ${\stackrel{\sim }{H}}_{\text{BLUE}}$ is likely to outperform $\stackrel{\sim }{H}$ in situations where ${\stackrel{\sim }{p}}_i$ has a smaller variance than ${\widehat{p}}_i.$ This trend is true for genome-wide data as well (Figure 4 and Table 1).

Overall, ${\stackrel{\sim }{H}}_{\text{BLUE}}$ yields identical results to $\stackrel{\sim }{H}$ in samples containing only one relative pair type, but the two diverge in performance as sample complexity increases (see heat maps in Figure 3, Figure S4, Figure S5, Figure S6, Figure S7, and Figure S8). While both estimators are unbiased, $\stackrel{\sim }{H}$ experiences a larger increase in variance for each additional relative pair type introduced into a sample after the first. This holds true for all sample types regardless of ploidy and inbreeding, suggesting that ${\stackrel{\sim }{H}}_{\text{BLUE}}$ will outperform $\stackrel{\sim }{H}$ in practice, where datasets are often complex. Furthermore, the results of our empirical analysis provide an equally important complement to this observation. Of the 93 populations from the MS5795 dataset we considered that contained relative pairs in their samples, each contained sampled individuals that were not related to any other in the sample. Thus, these samples were more complex than those in which each individual was part of a relative pair of the same type. For most of these cases, except for some American populations (discussed below), ${\stackrel{\sim }{H}}_{\text{BLUE}}$ outperformed $\stackrel{\sim }{H}.$ This is corroborated by the Wilcoxon signed-rank test (Table 1). We expect therefore that any scenario in which there is heterogeneity in relative pair type among sampled individuals, as is observed in many human population-genetic datasets (Pemberton et al. 2010, 2013), should favor the application of ${\stackrel{\sim }{H}}_{\text{BLUE}}$ over other estimators.

In addition, random sampling of small isolated populations yields an increased chance that related individuals will be included with large enough sample sizes. Further, inbreeding may confound estimates of diversity, and mislead ${\widehat{H}}_{\text{full}}$ to underreport true population expected heterozygosity. Populations of interest that may display these attributes include geographically isolated human settlements in remote alpine (Coia et al. 2012; Capocasa et al. 2013), South American rainforest (Wang et al. 2007), and Siberian taiga and steppe habitats (Dulik et al. 2012), and groups such as the Old Order Amish (Van Hout et al. 2010), Hutterites (Abney et al. 2002; Chong et al. 2011), and Mennonites (Payne et al. 2011). Further, though our analysis did not directly consider polyploid organisms, the applicability of ${\stackrel{\sim }{H}}_{\text{BLUE}}$ to samples containing individuals of any, and varying, ploidy highlights its usefulness for such data. Prominently, analysis on polyploid organisms such as plants including tetraploid Arabidopsis thaliana (Hollister et al. 2012), and hexaploid bread wheat (Nielsen et al. 2014), both of which self-fertilize, and may therefore be inbred, as well as commercially and ecologically significant Hymenopteran insects, including honeybees (Solignac et al. 2003; Harpur et al. 2014), bumblebees (Lye et al. 2011), and ants (Butler et al. 2014), whose males are haploid at all loci, while females are diploid, is likely to benefit from the improved accuracy and precision of ${\stackrel{\sim }{H}}_{\text{BLUE}}.$

We additionally believe that continued investigations into the diversity at single sites in organisms as diverse as dogs (Sutter et al. 2007), gray wolves (Zhang et al. 2014), humans living at high altitude (Simonson et al. 2010; Huerta-Sánchez et al. 2013), and rice (Huang et al. 2012), in addition to host-microbiome studies (Blekhman et al. 2015), will benefit from the advances provided by ${\stackrel{\sim }{H}}_{\text{BLUE}}.$ These studies, as well as many others, have performed scans for positive selection using genomic outliers of population differentiation-based statistics (e.g., $F_{\text{ST}},$ locus-specific branch length, and the population branch statistic), where the calculation is performed per-site, rather than averaged across a large number of sites. Such studies would benefit from estimators of genetic diversity, such as ${\stackrel{\sim }{H}}_{\text{BLUE}}$ and ${\stackrel{\sim }{F}}_{\text{ST},\text{BLUE}},$ with improved variance.

It is pertinent at this point to revisit a pair of potential limitations in our method and examine their implications. First, in Figure 4 (rightmost panel), the mean of $\stackrel{\sim }{H}$ is either closer to that of ${\widehat{H}}_{\text{red}}$ than to ${\stackrel{\sim }{H}}_{\text{BLUE}},$ has smaller SD than ${\stackrel{\sim }{H}}_{\text{BLUE}},$ or both for certain samples (predominantly from the Americas). These observations indicate that the accuracy and precision of ${\stackrel{\sim }{H}}_{\text{BLUE}}$ may be impacted by the accuracy of the kinship information incorporated into the calculation. The pedigrees of smaller, more remotely located, populations may be more complex compared to those of larger groups. Further, with a greater proportion of relative pairs in each sample, the effect of relative pair type misidentification may be larger. For RELPAIR (Epstein et al. 2000), which was the software chosen to identify relative pairs in MS5795 samples, second-degree pairs cannot be identified as confidently as first-degree pairs (Pemberton et al. 2010). Even so, although $\stackrel{\sim }{H}$ may exhibit a somewhat greater robustness to relative pair misclassification, it is still generally outperformed by ${\stackrel{\sim }{H}}_{\text{BLUE}}.$

The second point we address is the smaller MSE of ${\widehat{H}}_{\text{full}}$ at less diverse loci in the dataset, especially for samples with fewer relative pairs. While the variance of ${\widehat{H}}_{\text{full}}$ is always smaller than that of the other estimators, its bias increases with increasing locus allelic diversity. It is for this reason that the unbiasedness of ${\stackrel{\sim }{H}}_{\text{BLUE}}$ is its most desirable property. In practice, the mean of expected heterozygosity is often taken across loci. Based on such an approach, ${\stackrel{\sim }{H}}_{\text{BLUE}}$ (and $\stackrel{\sim }{H}$ as well) will return the mean expected heterozygosity, and the variance of this estimation (as with all estimators taking the mean across loci) approaches zero as more loci are sampled. An interesting property of all estimators is that their variance (and therefore MSE) is larger for loci whose value for B is closer to 1, where $B=D/R$ ($B\in \left[0,1\right];$ see Results and Figure S3). Because this effect is greatest for loci with lower true values of H, we expect ${\widehat{H}}_{\text{full}}$ to have the smallest MSE of all estimators at less diverse loci that are close to their maximum expected heterozygosity, and for which the sample mean kinship coefficient is insufficiently large to appreciably bias the estimator (Equation 12). It is thus important to note that no estimator is uniformly superior to the others. Accordingly, the unique limitation of ${\stackrel{\sim }{H}}_{\text{BLUE}}$ is that the sample kinship matrix must be invertible for the calculation to proceed.

${\stackrel{\sim }{H}}_{\text{BLUE}}$ additionally confers its improved MSE over ${\widehat{H}}_{\text{full}}$ downstream to calculations that incorporate estimates of expected heterozygosity. To illustrate this point, we computed $F_{\text{ST}}$ as a function of three estimators: ${\widehat{H}}_{\text{full}},$ $\stackrel{\sim }{H},$ and ${\stackrel{\sim }{H}}_{\text{BLUE}}.$ For simulated data, we found that ${\stackrel{\sim }{F}}_{\text{ST},\text{BLUE}},$ yielded an estimate with smaller MSE for the three tested loci than did ${\widehat{F}}_{\text{ST}}$ (Figure 5) or ${\stackrel{\sim }{F}}_{\text{ST}},$ and a much smaller mean distance from the true $F_{\text{ST}}$ value than ${\widehat{F}}_{\text{ST}}.$ For empirical data (Figure 6), we observed a consistent upward bias for ${\widehat{F}}_{\text{ST}}$ compared to ${\widehat{F}}_{\text{ST},\text{red}}$ in samples containing relatives that followed much the same pattern as the downward bias of ${\widehat{H}}_{\text{full}}$ for such samples. This trend is clear when we consider the formula for $F_{\text{ST}},$ which can be written as $1-\left(H_1+H_2\right)/\left(2H_{12}\right).$ Taking ${\widehat{H}}_{1,\text{full}}$ and ${\widehat{H}}_{2,\text{full}}$ as $H_1$ and $H_2,$ this expression yields a larger value than if ${\widehat{H}}_{1,\text{red}}$ and ${\widehat{H}}_{2,\text{red}}$ were used, because the ratio $\left(H_1+H_2\right)/\left(2H_{12}\right)$ is smaller for downwardly biased estimators. Interestingly, the SD of ${\stackrel{\sim }{F}}_{\text{ST},\text{BLUE}}$ is, in most cases, smaller than that of ${\widehat{F}}_{\text{ST}}$ for the dataset, while the SD of ${\stackrel{\sim }{H}}_{\text{BLUE}}$ was frequently (though not consistently) larger than that of ${\widehat{H}}_{\text{full}}$ (Figure 4, center panel).

It is thus noteworthy to consider that the performance of ${\stackrel{\sim }{H}}_{\text{BLUE}}$ and ${\widehat{H}}_{\text{full}}$ may diverge further in their applications, where any improvement in MSE for ${\stackrel{\sim }{H}}_{\text{BLUE}}$ may be magnified downstream. This is highlighted by the increased concordance between ${\stackrel{\sim }{F}}_{\text{ST},\text{BLUE}}$ and ${\widehat{F}}_{\text{ST},\text{red}}$ compared to ${\stackrel{\sim }{H}}_{\text{BLUE}}$ and ${\widehat{H}}_{\text{red}}$ (cf. P-values between Table 1 and Table 2). With this in mind, applications of ${\stackrel{\sim }{F}}_{\text{ST},\text{BLUE}}$ can also be considered. Two such examples are the locus-specific branch length (LSBL; Shriver et al. 2004) and the similar population branch statistic (PBS; Yi et al. 2010). These statistics incorporate $F_{\text{ST}}$ values between three populations as measures of branch length to detect positive selection at a locus. Loci for which the unrooted three-taxon tree indicates a significantly longer branch length in a particular lineage may represent regions possibly under selection. To allow for the easy application of ${\stackrel{\sim }{H}}_{\text{BLUE}},$ we have written an R script, BestHet, that computes ${\stackrel{\sim }{H}}_{\text{BLUE}},$ ${\stackrel{\sim }{F}}_{\text{ST},\text{BLUE}},$ and $LSB{L}_{\text{BLUE}},$ given matrices of genotype and kinship data for a sample (download available at http://www.personal.psu.edu/mxd60/best_het.html).

Supplementary Material

Supplemental material is available online at www.g3journal.org/lookup/suppl/doi:10.1534/g3.116.037168/-/DC1.

Appendix

Derivations of unbiased estimators of expected heterozygosity

In this section, we derive the general unbiased estimator of expected heterozygosity $\breve{H}$ for any unbiased linear estimator of population allele frequencies, defined in Proposition 1, and show how the formulas for $\stackrel{\sim }{H}$ (DeGiorgio et al. 2010), and ${\stackrel{\sim }{H}}_{\text{BLUE}}$ (Corollaries 2 and 3), emerge from specific cases of $\breve{H}.$

Proof of Proposition 1:

We need to show that $\mathbb{E}\left[\breve{H}\right]=H.$ Note that

$$\begin{array}{c}{\breve{p}}_i^2={\displaystyle \sum _{j=1}^n{\displaystyle \sum _{k=1}^n{w}_j{w}_k{X}_j^{\left(i\right)}{X}_k^{\left(i\right)}}}\\ ={\displaystyle \sum _{j=1}^n{\displaystyle \sum _{k=1}^n\frac{w_j{w}_k}{m_j{m}_k}{\displaystyle \sum _{\ell =1}^{m_j}{\displaystyle \sum _{t=1}^{m_k}{A}_{j\ell }^{\left(i\right)}{A}_{kt}^{\left(i\right)}}}}}.\end{array}$$

Taking the expectation, we obtain

$$\begin{array}{c}\mathbb{E}\left[{\breve{p}}_i^2\right]={\displaystyle \sum _{j=1}^n{\displaystyle \sum _{k=1}^n\frac{w_j{w}_k}{m_j{m}_k}}}{\displaystyle \sum _{\ell =1}^{m_j}{\displaystyle \sum _{t=1}^{m_k}\mathbb{E}\left[A_{j\ell }^{\left(i\right)}{A}_{kt}^{\left(i\right)}\right]}}\\ ={\displaystyle \sum _{j=1}^n{\displaystyle \sum _{k=1}^n\frac{w_j{w}_k}{m_j{m}_k}{\displaystyle \sum _{\ell =1}^{m_j}{\displaystyle \sum _{t=1}^{m_k}\mathrm{\mathbb{P}}\left[A_{j\ell }^{\left(i\right)}=1,A_{kt}^{\left(i\right)}=1\right]}}}}\\ ={\displaystyle \sum _{j=1}^n{\displaystyle \sum _{k=1}^n\frac{w_j{w}_k}{m_j{m}_k}{\displaystyle \sum _{\ell =1}^{m_j}{\displaystyle \sum _{t=1}^{m_k}\left[\left(1-{\Phi }_{jk}\right)p_i^2+{\Phi }_{jk}{p}_i\right]}}}}\\ =p_i^2+{\rho }_2{p}_i\left(1-p_i\right).\end{array}$$ (A1)

Therefore

$$\begin{array}{c}\mathbb{E}\left[\breve{H}\right]=\frac{1}{1-{\rho }_2}\left(1-{\displaystyle \sum _{i=1}^I\mathbb{E}\left[{\breve{p}}_i^2\right]}\right)\\ =\frac{1}{1-{\rho }_2}\left(1-{\displaystyle \sum _{i=1}^I\left[p_i^2+{\rho }_2{p}_i\left(1-p_i\right)\right]}\right)\\ =1-{\displaystyle \sum _{i=1}^I{p}_i^2}\\ =H.\end{array}$$ □

Proof of Corollary 2:

We show that defining the weight of each individual in the calculation of ${\rho }_2$ in terms of an individual’s relative allele copy contribution yields $\stackrel{\sim }{H}$ from $\breve{H}.$ Letting $w_k=m_k/{\displaystyle {\sum }_{x=1}^n{m}_x},$ we have that

$$\begin{array}{c}{\breve{p}}_i={\displaystyle \sum _{k=1}^n\frac{m_k}{{\displaystyle {\sum }_{j=1}^n{m}_j}}{X}_k^{\left(i\right)}}\\ ={\widehat{p}}_i\end{array}$$

and

$$\begin{array}{c}{\rho }_2={\displaystyle \sum _{j=1}^n{\displaystyle \sum _{k=1}^n\frac{m_j}{{\displaystyle {\sum }_{x=1}^n{m}_x}}\frac{m_k}{{\displaystyle {\sum }_{y=1}^n{m}_y}}{\Phi }_{jk}}}\\ ={\overline{\Phi }}_2.\end{array}$$

Plugging in yields

$$\breve{H}=\frac{1}{1-{\overline{\Phi }}_2}\left(1-{\displaystyle \sum _{i=1}^I{\widehat{p}}_i^2}\right)=\stackrel{\sim }{H}.$$ □

Proof of Corollary 3:

We show that defining the weight each individual according to their relative contribution to the inverted kinship matrix of the sample yields ${\stackrel{\sim }{H}}_{\text{BLUE}}$ from $\breve{H}.$ Letting $w_k={{{\displaystyle \sum }}^{\text{}}}_{j=1}^n{\left(K^{-1}\right)}_{jk}/{\mathbf{1}}^T{K}^{-1}\mathbf{1},$ we have that

$$\begin{array}{c}{\breve{p}}_i={\displaystyle \sum _{k=1}^n\frac{{\displaystyle {\sum }_{j=1}^n{\left(K^{-1}\right)}_{jk}}}{{\mathbf{1}}^T{K}^{-1}\mathbf{1}}{X}_k^{\left(i\right)}}\\ ={\stackrel{\sim }{p}}_i\end{array}$$

and

$$\begin{array}{c}{\rho }_2={\displaystyle \sum _{j=1}^n{\displaystyle \sum _{k=1}^n\frac{{\displaystyle {\sum }_{x=1}^n{\left(K^{-1}\right)}_{xj}}}{{\mathbf{1}}^T{K}^{-1}\mathbf{1}}\frac{{\displaystyle {\sum }_{y=1}^n{\left(K^{-1}\right)}_{yk}}}{{\mathbf{1}}^T{K}^{-1}\mathbf{1}}{\Phi }_{jk}}}\\ ={\kappa }_2.\end{array}$$

Plugging in yields

$$\begin{array}{c}\breve{H}=\frac{1}{1-{\kappa }_2}\left(1-{\displaystyle \sum _{i=1}^I{\stackrel{\sim }{p}}_i^2}\right)\\ ={\stackrel{\sim }{H}}_{\text{BLUE}}.\end{array}$$ □

Derivations of variances of expected heterozygosity estimators

In this section, we summarize the procedure by which DeGiorgio et al. (2010) derived the equation for the variance of $\stackrel{\sim }{H},$ illustrating the variance of the general case, $\breve{H}.$ For the full derivation, see Appendix B of DeGiorgio et al. (2010). We then provide the specific formulation for the variance of $\stackrel{\sim }{H}$ (Corollary 7) and ${\stackrel{\sim }{H}}_{\text{BLUE}}$ (Corollary 8).

Abbreviated proof of Proposition 4:

The variance of $\breve{H}$ (Equation 9) is defined as

$$\text{Var}\left[\breve{H}\right]=\frac{1}{{\left(1-{\rho }_2\right)}^2}\text{Var}\left[1-{\displaystyle \sum _{i=1}^I{\breve{p}}_i^2}\right].$$

By definition of variance, we get

$$\text{Var}\left[1-{\displaystyle \sum _{i=1}^I{\breve{p}}_i^2}\right]={\displaystyle \sum _{i=1}^I\text{Var}\left[{\breve{p}}_i^2\right]}+2{\displaystyle \sum _{i=1}^{I-1}{\displaystyle \sum _{i^{\prime }=i+1}^I\text{Cov}\left[{\breve{p}}_i^2,{\breve{p}}_{i^{\prime }}^2\right]}},$$

with

$$\text{Var}\left[{\breve{p}}_i^2\right]=\mathbb{E}\left[{\breve{p}}_i^4\right]-{\left(\mathbb{E}\left[{\breve{p}}_i^2\right]\right)}^2$$

and

$$\text{Cov}\left[{\breve{p}}_i^2,{\breve{p}}_{i^{\prime }}^2\right]=\mathbb{E}\left[{\breve{p}}_i^2{\breve{p}}_{i^{\prime }}^2\right]-\mathbb{E}\left[{\breve{p}}_i^2\right]\mathbb{E}\left[{\breve{p}}_{i^{\prime }}^2\right].$$

Recalling that ${\breve{p}}_i={\displaystyle {\sum }_{j=1}^n{\displaystyle {\sum }_{\ell =1}^{m_j}\frac{w_j}{m_j}{A}_{j\ell }^{\left(i\right)}}}$ for the $\ell$th allele copy of individual j, whose ploidy is $m_j,$ we have that

$$\mathbb{E}\left[{\breve{p}}_i^4\right]={\displaystyle \sum _{j=1}^n{\displaystyle \sum _{k=1}^n{\displaystyle \sum _{j^{\prime }=1}^n{\displaystyle \sum _{k^{\prime }=1}^n{\displaystyle \sum _{\ell =1}^{m_j}{\displaystyle \sum _{t=1}^{m_k}{\displaystyle \sum _{{\ell }^{\prime }=1}^{m_{j^{\prime }}}{\displaystyle \sum _{t^{\prime }=1}^{m_{k^{\prime }}}\frac{w_j{w}_k{w}_{j\prime }{w}_{k\prime }}{m_j{m}_k{m}_{j\prime }{m}_{k\prime }}\mathbb{E}\left[A_{j\ell }^{\left(i\right)}{A}_{kt}^{\left(i\right)}{A}_{j\prime \ell \prime }^{\left(i\right)}{A}_{k\prime t\prime }^{\left(i\right)}\right]}}}}}}}},$$

and

$$\mathbb{E}\left[{\breve{p}}_i^2{\breve{p}}_{i\prime }^2\right]={\displaystyle \sum _{j=1}^n{\displaystyle \sum _{k=1}^n{\displaystyle \sum _{j^{\prime }=1}^n{\displaystyle \sum _{k^{\prime }=1}^n{\displaystyle \sum _{\ell =1}^{m_j}{\displaystyle \sum _{t=1}^{m_k}{\displaystyle \sum _{{\ell }^{\prime }=1}^{m_{j^{\prime }}}{\displaystyle \sum _{t^{\prime }=1}^{m_{k^{\prime }}}\frac{w_j{w}_k{w}_{j^{\prime }}{w}_{k^{\prime }}}{m_j{m}_k{m}_{j^{\prime }}{m}_{k^{\prime }}}\mathbb{E}\left[A_{j\ell }^{\left(i\right)}{A}_{kt}^{\left(i\right)}{A}_{j^{\prime }{\ell }^{\prime }}^{\left(i^{\prime }\right)}{A}_{k^{\prime }{t}^{\prime }}^{\left(i^{\prime }\right)}\right]}}}}}}}}$$

for the case $i\ne i^{\prime }.$ We have previously shown in Equation A1 that $\mathbb{E}\left[{\breve{p}}_i^2\right]=p_i^2+{\rho }_2{p}_i\left(1-p_i\right),$ and the value of $\mathbb{E}\left[{\breve{p}}_{i^{\prime }}^2\right]$ similarly follows. Thus, we need to calculate $\mathbb{E}\left[A_{j\ell }^{\left(i\right)}{A}_{kt}^{\left(i\right)}{A}_{j^{\prime }{\ell }^{\prime }}^{\left(i\right)}{A}_{k^{\prime }{t}^{\prime }}^{\left(i\right)}\right]$ and $\mathbb{E}\left[A_{j\ell }^{\left(i\right)}{A}_{kt}^{\left(i\right)}{A}_{j^{\prime }{\ell }^{\prime }}^{\left(i^{\prime }\right)}{A}_{k^{\prime }{t}^{\prime }}^{\left(i^{\prime }\right)}\right]$ for the $i\ne i^{\prime }$ case. These are

$$\begin{array}{c}\mathbb{E}\left[A_{j\ell }^{\left(i\right)}{A}_{kt}^{\left(i\right)}{A}_{j^{\prime }{\ell }^{\prime }}^{\left(i\right)}{A}_{k^{\prime }{t}^{\prime }}^{\left(i\right)}\right]={\Phi }_{jk{j}^{\prime }{k}^{\prime }}{p}_i+\left[{\Phi }_{jk{j}^{\prime }}+{\Phi }_{jk{k}^{\prime }}+{\Phi }_{j{j}^{\prime }{k}^{\prime }}+{\Phi }_{k{j}^{\prime }{k}^{\prime }}+{\Phi }_{jk,j^{\prime }{k}^{\prime }}+{\Phi }_{j{j}^{\prime },k{k}^{\prime }}+{\Phi }_{j{k}^{\prime },k{j}^{\prime }}-7{\Phi }_{jk{j}^{\prime }{k}^{\prime }}\right]p_i^2\\ +[12{\Phi }_{jk{j}^{\prime }{k}^{\prime }}+{\Phi }_{jk}+{\Phi }_{j{j}^{\prime }}+{\Phi }_{j{k}^{\prime }}+{\Phi }_{k{j}^{\prime }}+{\Phi }_{k{k}^{\prime }}+{\Phi }_{j^{\prime }{k}^{\prime }}-3\left({\Phi }_{jk{j}^{\prime }}+{\Phi }_{jk{k}^{\prime }}+{\Phi }_{j{j}^{\prime }{k}^{\prime }}+{\Phi }_{k{j}^{\prime }{k}^{\prime }}\right)\\ -2\left({\Phi }_{jk,j^{\prime }{k}^{\prime }}+{\Phi }_{jj,k{k}^{\prime }}+{\Phi }_{jk,k{j}^{\prime }}\right)]p_i^3+[1+{\Phi }_{jk,j^{\prime }{k}^{\prime }}+{\Phi }_{j{j}^{\prime },k{k}^{\prime }}+{\Phi }_{j{k}^{\prime },k{j}^{\prime }}\\ +2\left({\Phi }_{jk{j}^{\prime }}+{\Phi }_{jk{k}^{\prime }}+{\Phi }_{j{j}^{\prime }{k}^{\prime }}+{\Phi }_{k{j}^{\prime }{k}^{\prime }}\right)-6{\Phi }_{jk{j}^{\prime }{k}^{\prime }}-\left({\Phi }_{jk}+{\Phi }_{j{j}^{\prime }}+{\Phi }_{j{k}^{\prime }}+{\Phi }_{k{j}^{\prime }}+{\Phi }_{k{k}^{\prime }}+{\Phi }_{j^{\prime }{k}^{\prime }}\right)]p_i^4\end{array}$$ (A2)

and

$$\begin{array}{c}\mathbb{E}\left[A_{j\ell }^{\left(i\right)}{A}_{kt}^{\left(i\right)}{A}_{j^{\prime }{\ell }^{\prime }}^{\left(i^{\prime }\right)}{A}_{k^{\prime }{t}^{\prime }}^{\left(i^{\prime }\right)}\right]=\left[{\Phi }_{jk,j^{\prime }{k}^{\prime }}-{\Phi }_{jk{j}^{\prime }{k}^{\prime }}\right]p_i{p}_{i^{\prime }}+\left[2{\Phi }_{jk{j}^{\prime }{k}^{\prime }}+{\Phi }_{jk}-\left({\Phi }_{jk{j}^{\prime }}+{\Phi }_{jk{k}^{\prime }}\right)-{\Phi }_{jk,j^{\prime }{k}^{\prime }}\right]p_i{p}_{i^{\prime }}^2\\ +\left[2{\Phi }_{jk{j}^{\prime }{k}^{\prime }}+{\Phi }_{j^{\prime }{k}^{\prime }}-\left({\Phi }_{j{j}^{\prime }{k}^{\prime }}+{\Phi }_{k{j}^{\prime }{k}^{\prime }}\right)-{\Phi }_{jk,j^{\prime }{k}^{\prime }}\right]p_i^2{p}_{i^{\prime }}+[1+{\Phi }_{jk,j^{\prime }{k}^{\prime }}+{\Phi }_{j{j}^{\prime },k{k}^{\prime }}+{\Phi }_{j{k}^{\prime },k{j}^{\prime }}\\ +2\left({\Phi }_{jk{j}^{\prime }}+{\Phi }_{jk{k}^{\prime }}+{\Phi }_{j{j}^{\prime }{k}^{\prime }}+{\Phi }_{k{j}^{\prime }{k}^{\prime }}\right)-6{\Phi }_{jk{j}^{\prime }{k}^{\prime }}-\left({\Phi }_{jk}+{\Phi }_{j{j}^{\prime }}+{\Phi }_{j{k}^{\prime }}+{\Phi }_{k{j}^{\prime }}+{\Phi }_{k{k}^{\prime }}+{\Phi }_{j^{\prime }{k}^{\prime }}\right)]p_i^2{p}_{i^{\prime }}^2.\end{array}$$ (A3)

Substituting Equation A2 into $\mathbb{E}\left[{\breve{p}}_i^4\right]$ and solving for $\text{Var}\left[{\breve{p}}_i^2\right],$ we obtain

$$\text{Var}\left[{\breve{p}}_i^2\right]={\rho }_4{p}_i+\left(4{\rho }_3+3{\rho }_{2,2}-7{\rho }_4-{\rho }_2^2\right)p_i^2+\left(12{\rho }_4+4{\rho }_2+2{\rho }_2^2-12{\rho }_3-6{\rho }_{2,2}\right)p_i^3+\left(3{\rho }_{2,2}+8{\rho }_3-6{\rho }_4-4{\rho }_2-{\rho }_2^2\right)p_i^4,$$

and, substituting Equation A3 into $\mathbb{E}\left[{\breve{p}}_i^2{\breve{p}}_{i\prime }^2\right],$ and solving for $\text{Cov}\left[{\breve{p}}_i^2,{\breve{p}}_{i^{\prime }}^2\right],$ we obtain

$$\text{Cov}\left[{\breve{p}}_i^2,{\breve{p}}_{i^{\prime }}^2\right]=\left({\rho }_{2,2}-{\rho }_4-{\rho }_2^2\right)p_i{p}_{i^{\prime }}+\left(2{\rho }_4+{\rho }_2^2-2{\rho }_3-{\rho }_{2,2}\right)p_i{p}_{i^{\prime }}^2+\left(2{\rho }_4+{\rho }_2^2-2{\rho }_3-{\rho }_{2,2}\right)p_i^2{p}_{i^{\prime }}+\left(3{\rho }_{2,2}+8{\rho }_3-6{\rho }_4-4{\rho }_2-{\rho }_2^2\right)p_i^2{p}_{i^{\prime }}^2.$$

Thus, substituting the values for variance and covariance into the definition of variance, we have

$$\begin{array}{l}\text{Var}\left[\breve{H}\right]=\frac{1}{{\left(1-{\rho }_2\right)}^2}[{\rho }_{2,2}-{\rho }_2^2+2\left({\rho }_2^2-{\rho }_4\right){\displaystyle \sum _{i=1}^I{p}_i^2}+4\left(2{\rho }_4+{\rho }_2-2{\rho }_3-{\rho }_{2,2}\right){\displaystyle \sum _{i=1}^I{p}_i^3}\\ +\left(3{\rho }_{2,2}+8{\rho }_3-6{\rho }_4-4{\rho }_2-{\rho }_2^2\right){\left({\displaystyle \sum _{i=1}^I{p}_i^2}\right)}^2].\end{array}$$ □

Corollary 7:

Consider a locus with I distinct alleles, and parametric allele frequencies $p_i\in \left[0,1\right],$ $i=1,2,\dots ,I,$ and ${\displaystyle {\sum }_{i=1}^I{p}_i}=1.$ For a sample of size n individuals of any ploidy, inbreeding status, and relatedness,

$$\text{Var}\left[\stackrel{\sim }{H}\right]=\frac{1}{{\left(1-{\overline{\Phi }}_2\right)}^2}\text{Var}\left[1-{\displaystyle \sum _{i=1}^I{\widehat{p}}_i^2}\right]$$ (A4)

and

$$\begin{array}{c}\text{Var}\left[1-{\displaystyle \sum _{i=1}^I{\widehat{p}}_i^2}\right]={\overline{\Phi }}_{2,2}-{\overline{\Phi }}_2^2+2\left({\overline{\Phi }}_2^2-{\overline{\Phi }}_4\right){\displaystyle \sum _{i=1}^I{p}_i^2}+4\left(2{\overline{\Phi }}_4+{\overline{\Phi }}_2-2{\overline{\Phi }}_3-{\overline{\Phi }}_{2,2}\right){\displaystyle \sum _{i=1}^I{p}_i^3}\\ +\left(3{\overline{\Phi }}_{2,2}+8{\overline{\Phi }}_3-6{\overline{\Phi }}_4-4{\overline{\Phi }}_2-{\overline{\Phi }}_2^2\right){\left({\displaystyle \sum _{i=1}^I{p}_i^2}\right)}^2,\end{array}$$ (A5)

where ${\overline{\Phi }}_2,$ ${\overline{\Phi }}_3,$ ${\overline{\Phi }}_4,$ and ${\overline{\Phi }}_{2,2}$ are mean kinship coefficients, weighted by the contribution of individuals to the number of allele copies in the sample, with subscripts corresponding to the number of individuals considered for the calculation. Additionally,

$$\text{Var}\left[\stackrel{\sim }{H}\right]\approx 4{\overline{\Phi }}_2\left[{\displaystyle \sum _{i=1}^I{p}_i^3}-{\left({\displaystyle \sum _{i=1}^I{p}_i^2}\right)}^2\right].$$ (A6)

The proof of Corollary 7 follows from the proof of Proposition 4, where ${\widehat{p}}_i$ is substituted for ${\breve{p}}_i,$ and ${\overline{\Phi }}_2,$ ${\overline{\Phi }}_3,$ ${\overline{\Phi }}_4,$ and ${\overline{\Phi }}_{2,2}$ are substituted for ${\rho }_2,$ ${\rho }_3,$ ${\rho }_4,$ and ${\rho }_{2,2},$ respectively.

Corollary 8:

Consider a locus with I distinct alleles and parametric allele frequencies $p_i\in \left[0,1\right],$ $i=1,2,\dots ,I,$ and ${\displaystyle {\sum }_{i=1}^I{p}_i}=1.$ For a sample of size n individuals of any ploidy, inbreeding status, and relatedness,

$$\text{Var}\left[{\stackrel{\sim }{H}}_{\text{BLUE}}\right]=\frac{1}{{\left(1-{\kappa }_2\right)}^2}\text{Var}\left[1-{\displaystyle \sum _{i=1}^I{\stackrel{\sim }{p}}_i^2}\right]$$ (A7)

and

$$\begin{array}{c}\text{Var}\left[1-{\displaystyle \sum _{i=1}^I{\stackrel{\sim }{p}}_i^2}\right]={\kappa }_{2,2}-{\kappa }_2^2+2\left({\kappa }_2^2-{\kappa }_4\right){\displaystyle \sum _{i=1}^I{p}_i^2}+4\left(2{\kappa }_4+{\kappa }_2-2{\kappa }_3-{\kappa }_{2,2}\right){\displaystyle \sum _{i=1}^I{p}_i^3}\\ +\left(3{\kappa }_{2,2}+8{\kappa }_3-6{\kappa }_4-4{\kappa }_2-{\kappa }_2^2\right){\left({\displaystyle \sum _{i=1}^I{p}_i^2}\right)}^2,\end{array}$$ (A8)

where ${\kappa }_2,$ ${\kappa }_3,$ ${\kappa }_4,$ and ${\kappa }_{2,2}$ are mean kinship coefficients, weighted by the contribution of individuals to the inverted kinship matrix, with subscripts corresponding to the number of individuals considered for the calculation. Additionally,

$$\text{Var}\left[{\stackrel{\sim }{H}}_{\text{BLUE}}\right]\approx 4{\kappa }_2\left[{\displaystyle \sum _{i=1}^I{p}_i^3}-{\left({\displaystyle \sum _{i=1}^I{p}_i^2}\right)}^2\right].$$ (A9)

The proof of Corollary 8 follows from the proof of Proposition 4, where ${\stackrel{\sim }{p}}_i$ is substituted for ${\breve{p}}_i,$ and ${\kappa }_2,$ ${\kappa }_3,$ ${\kappa }_4,$ and ${\kappa }_{2,2}$ are substituted for ${\rho }_2,$ ${\rho }_3,$ ${\rho }_4,$ and ${\rho }_{2,2},$ respectively.

Derivations of bias measurements in the application of $\widehat{\text{H}}$

For samples containing related and inbred individuals, $\widehat{H}$ has a downward bias, which is defined in Equation 12 for the general estimator of population allele frequency ${\breve{p}}_i.$ Here, we present Corollaries 9 and 10 for the specific estimators of population allele frequency ${\widehat{p}}_i$ and ${\stackrel{\sim }{p}}_i,$ respectively.

Corollary 9:

Consider a locus with I distinct alleles and parametric allele frequencies $p_i\in \left[0,1\right],$ $i=1,2,\dots ,I,$ and ${\displaystyle {\sum }_{i=1}^I{p}_i}=1.$ For a sample of size n possibly related or inbred individuals, the bias of the estimator of expected heterozygosity $\widehat{H}$ changes with the true locus expected heterozygosity such that

$$\text{Bias}\left[\widehat{H}\left({\widehat{p}}_i\right)\right]=\frac{1-n{\overline{\Phi }}_2}{n-1}H,$$ (A10)

where

$$\widehat{H}\left({\widehat{p}}_i\right)=\frac{n}{n-1}\left(1-{\displaystyle \sum _{i=1}^I{\widehat{p}}_i^2}\right).$$

As this is the standard application of $\widehat{H}$ (Equation 2), Equation A10 describes the bias of $\widehat{H}$ in the Results. However, $\widehat{H}$ is biased with any unbiased linear estimator of allele frequency for samples containing related or inbred individuals. The proof of Corollary 9 follows from the proof of Proposition 5, where ${\overline{\Phi }}_2$ is substituted for ${\rho }_2.$

Corollary 10:

Consider a locus with I distinct alleles and parametric allele frequencies $p_i\in \left[0,1\right],$ $i=1,2,\dots ,I,$ and ${\displaystyle {\sum }_{i=1}^I{p}_i}=1.$ For a sample of size n possibly related or inbred individuals, the bias of the estimator of expected heterozygosity $\widehat{H}$ changes with the true locus expected heterozygosity such that

$$\text{Bias}\left[\widehat{H}\left({\stackrel{\sim }{p}}_i\right)\right]=\frac{1-n{\kappa }_2}{n-1}H,$$ (A11)

where

$$\widehat{H}\left({\stackrel{\sim }{p}}_i\right)=\frac{n}{n-1}\left(1-{\displaystyle \sum _{i=1}^I{\stackrel{\sim }{p}}_i^2}\right).$$ (A12)

The proof of Corollary 10 follows from the proof of Proposition 5, where ${\kappa }_2$ is substituted for ${\rho }_2.$

Derivations of components for the variance of ${\text{F}}_{\text{ST}}$ estimators

In this final section of the Appendix, we provide derivations for the components of Equations 16 and 17, which describe the variance of ${\breve{F}}_{\text{ST}}.$ We derive the variance of ${\breve{H}}_{12},$ as well as the covariances of ${\breve{H}}_{12}$ with ${\breve{H}}_1$ (and interchangeably, ${\breve{H}}_{12}$ with ${\breve{H}}_2$), and of $\left[{\breve{H}}_{12}-\frac{1}{2}{\breve{H}}_1-\frac{1}{2}{\breve{H}}_2\right]$ with ${\breve{H}}_{12}.$ Because the complete expression for $\text{Var}\left[{\breve{F}}_{\text{ST}}\right]$ is unwieldy, we stop at the derivation of the final component.

Lemma 11:

Consider a locus with I distinct alleles across two independent populations and parametric allele frequencies $p_i\in \left[0,1\right],$ $i=1,2,\dots ,I,$ and ${\displaystyle {\sum }_{i=1}^I{p}_i}=1$ for population 1, and $q_i\in \left[0,1\right],$ $i=1,2,\dots ,I,$ and ${\displaystyle {\sum }_{i=1}^I{q}_i}=1$ for population 2. For two samples of size $n_1$ and $n_2,$ individuals from populations 1 and 2, respectively, each with individuals of any ploidy, inbreeding status, and relatedness,

$$\text{Var}\left[{\breve{H}}_{12}\right]={\rho }_2^{\left(1\right)}\left(1-{\rho }_2^{\left(2\right)}\right){\displaystyle \sum _{i=1}^I{p}_i{q}_i^2}+{\rho }_2^{\left(2\right)}\left(1-{\rho }_2^{\left(1\right)}\right){\displaystyle \sum _{i=1}^I{p}_i^2{q}_i}+{\rho }_2^{\left(1\right)}{\rho }_2^{\left(2\right)}{\displaystyle \sum _{i=1}^I{p}_i{q}_i}+\left({\rho }_2^{\left(1\right)}{\rho }_2^{\left(2\right)}-{\rho }_2^{\left(1\right)}-{\rho }_2^{\left(2\right)}\right){\left({\displaystyle \sum _{i=1}^I{p}_i{q}_i}\right)}^2,$$ (A13)

where the superscript of the mean kinship coefficient ${\rho }_2$ corresponds to the population for which it is calculated. The equations for the variance of ${\stackrel{\sim }{H}}_{12}$ and ${\stackrel{\sim }{H}}_{12,\text{BLUE}}$ are obtained by substituting ${\overline{\Phi }}_2$ and ${\kappa }_2,$ respectively, into Equation A13 as the mean kinship coefficients in place of ${\rho }_2.$

Proof:

By definition of variance,

$$\text{Var}\left[{\breve{H}}_{12}\right]={\displaystyle \sum _{i=1}^I\text{Var}\left[{\breve{p}}_i{\breve{q}}_i\right]}+2{\displaystyle \sum _{i=1}^{I-1}{\displaystyle \sum _{i^{\prime }=i+1}^I\text{Cov}\left[{\breve{p}}_i{\breve{q}}_i,{\breve{p}}_{i^{\prime }}{\breve{q}}_{i^{\prime }}\right]}}$$

where

$$\text{Var}\left[{\breve{p}}_i{\breve{q}}_i\right]=\mathbb{E}\left[{\breve{p}}_i^2{\breve{q}}_i^2\right]-{\left(\mathbb{E}\left[{\breve{p}}_i{\breve{q}}_i\right]\right)}^2$$

and

$$\text{Cov}\left[{\breve{p}}_i{\breve{q}}_i,{\breve{p}}_{i^{\prime }}{\breve{q}}_{i^{\prime }}\right]=\mathbb{E}\left[{\breve{p}}_i{\breve{q}}_i{\breve{p}}_{i^{\prime }}{\breve{q}}_{i^{\prime }}\right]-\mathbb{E}\left[{\breve{p}}_i{\breve{q}}_i\right]\mathbb{E}\left[{\breve{p}}_{i^{\prime }}{\breve{q}}_{i^{\prime }}\right].$$

Because ${\breve{p}}_i$ and ${\breve{q}}_i$ are unbiased estimators of population allele frequency, and populations 1 and 2 are independent,

$$\mathbb{E}\left[{\breve{p}}_i{\breve{q}}_i\right]=p_i{q}_i$$

Similarly, $\mathbb{E}\left[{\breve{p}}_{i^{\prime }}{\breve{q}}_{i^{\prime }}\right]=p_{i^{\prime }}{q}_{i^{\prime }}.$ Next, we have

$$\begin{array}{c}\mathbb{E}\left[{\breve{p}}_i^2{\breve{q}}_i^2\right]=\mathbb{E}\left[{\breve{p}}_i^2\right]\mathbb{E}\left[{\breve{q}}_i^2\right]\\ =\left[p_i^2+{\rho }_2^{\left(1\right)}{p}_i\left(1-p_i\right)\right]\left[q_i^2+{\rho }_2^{\left(2\right)}{q}_i\left(1-q_i\right)\right]\\ =p_i{q}_i\left[p_i+{\rho }_2^{\left(1\right)}\left(1-p_i\right)\right]\left[q_i+{\rho }_2^{\left(2\right)}\left(1-q_i\right)\right],\end{array}$$ (A14)

where $\mathbb{E}\left[{\breve{q}}_i^2\right]$ takes the same form as $\mathbb{E}\left[{\breve{p}}_i^2\right]$ (Equation A1), except that the resulting weighted mean kinship coefficient ${\rho }_2$ is for population 2, indicated by the superscript. By substituting Equation A14 into $\text{Var}\left[{\breve{p}}_i{\breve{q}}_i\right],$ we have

$$\begin{array}{c}\text{Var}\left[{\breve{p}}_i{\breve{q}}_i\right]=p_i{q}_i\left[p_i+{\rho }_2^{\left(1\right)}\left(1-p_i\right)\right]\left[q_i+{\rho }_2^{\left(2\right)}\left(1-q_i\right)\right]-{\left(p_i{q}_i\right)}^2\\ =p_i{q}_i\left\{\left[p_i+{\rho }_2^{\left(1\right)}\left(1-p_i\right)\right]\left[q_i+{\rho }_2^{\left(2\right)}\left(1-q_i\right)\right]-p_i{q}_i\right\}\\ =p_i{q}_i\left[{\rho }_2^{\left(1\right)}\left(1-p_i\right)q_i+{\rho }_2^{\left(2\right)}{p}_i\left(1-q_i\right)+{\rho }_2^{\left(1\right)}{\rho }_2^{\left(2\right)}\left(1-p_i\right)\left(1-q_i\right)\right].\end{array}$$ (A15)

We now derive an expression for $\text{Cov}\left[{\breve{p}}_i{\breve{q}}_i,{\breve{p}}_{i^{\prime }}{\breve{q}}_{i^{\prime }}\right].$ Let $B_{kt}^{\left(i\right)}$ be an indicator random variable in population 2 analogous to the indicator random variable $A_{j\ell }^{\left(i\right)},$ which we have previously defined for population 1.

$$\begin{array}{c}\mathbb{E}\left[{\breve{p}}_i{\breve{q}}_i{\breve{p}}_{i^{\prime }}{\breve{q}}_{i^{\prime }}\right]=\mathbb{E}\left[{\breve{p}}_i{\breve{p}}_{i^{\prime }}\right]\mathbb{E}\left[{\breve{q}}_i{\breve{q}}_{i^{\prime }}\right]\\ =\left({\displaystyle \sum _{j=1}^{n_1}{\displaystyle \sum _{j^{\prime }=1}^{n_1}{\displaystyle \sum _{\ell =1}^{m_j}{\displaystyle \sum _{{\ell }^{\prime }=1}^{m_{j^{\prime }}}\frac{w_j{w}_{j^{\prime }}}{m_j{m}_{j^{\prime }}}}}}}\mathbb{E}\left[A_{j\ell }^{\left(i\right)}{A}_{j^{\prime }{\ell }^{\prime }}^{\left(i^{\prime }\right)}\right]\right)\left({\displaystyle \sum _{k=1}^{n_2}{\displaystyle \sum _{k^{\prime }=1}^{n_2}{\displaystyle \sum _{t=1}^{m_k}{\displaystyle \sum _{t^{\prime }=1}^{m_{k^{\prime }}}}}}}\frac{w_k{w}_{k^{\prime }}}{m_k{m}_{k^{\prime }}}\mathbb{E}\left[B_{kt}^{\left(i\right)}{B}_{k^{\prime }{t}^{\prime }}^{\left(i^{\prime }\right)}\right]\right),\end{array}$$

where

$$\mathbb{E}\left[A_{j\ell }^{\left(i\right)}{A}_{j^{\prime }{\ell }^{\prime }}^{\left(i^{\prime }\right)}\right]=\mathrm{\mathbb{P}}\left[A_{j\ell }^{\left(i\right)}=1,A_{j^{\prime }{\ell }^{\prime }}^{\left(i^{\prime }\right)}=1\right],$$

and

$$\mathbb{E}\left[B_{kt}^{\left(i\right)}{B}_{k^{\prime }{t}^{\prime }}^{\left(i\prime \right)}\right]=\mathrm{\mathbb{P}}\left[B_{kt}^{\left(i\right)}=1,B_{k^{\prime }{t}^{\prime }}^{\left(i^{\prime }\right)}=1\right].$$

Consider a scenario in which we have two allele copies. Let $s_1$ be the identity state with probability ${\Delta }_1,$ in which two randomly drawn alleles are not IBD, and $s_2$ be the identity state occurring with probability ${\Delta }_2=1-{\Delta }_1,$ in which the two alleles are IBD.

$$\begin{array}{c}\mathrm{\mathbb{P}}\left[A_{j\ell }^{\left(i\right)}=1,A_{j^{\prime }{\ell }^{\prime }}^{\left(i^{\prime }\right)}=1\right]=\mathrm{\mathbb{P}}\left[A_{j\ell }^{\left(i\right)}=1,A_{j^{\prime }{\ell }^{\prime }}^{\left(i^{\prime }\right)}=1|s_1\right]\mathrm{\mathbb{P}}\left[s_1\right]+\mathrm{\mathbb{P}}\left[A_{j\ell }^{\left(i\right)}=1,A_{j^{\prime }{\ell }^{\prime }}^{\left(i^{\prime }\right)}=1|s_2\right]\mathrm{\mathbb{P}}\left[s_2\right]\\ =p_i{p}_{i^{\prime }}{\Delta }_1+0\times {\Delta }_2\\ ={\Delta }_1{p}_i{p}_{i^{\prime }}.\end{array}$$

Note that, because ${\Delta }_1+{\Delta }_2=1$ and ${\Phi }_{j{j}^{\prime }}^{\left(1\right)}={\Delta }_2$ (same with ${\Phi }_{k{k}^{\prime }}^{\left(2\right)}$), we have ${\Delta }_1=1-{\Phi }_{j{j}^{\prime }}^{\left(1\right)}.$ Thus,

$$\mathrm{\mathbb{P}}\left[A_{j\ell }^{\left(i\right)}=1,A_{j^{\prime }{\ell }^{\prime }}^{\left(i^{\prime }\right)}=1\right]=\left(1-{\Phi }_{j{j}^{\prime }}^{\left(1\right)}\right)p_i{p}_{i^{\prime }}$$

and

$$\mathrm{\mathbb{P}}\left[B_{kt}^{\left(i\right)}=1,B_{k^{\prime }{t}^{\prime }}^{\left(i^{\prime }\right)}=1\right]=\left(1-{\Phi }_{k{k}^{\prime }}^{\left(2\right)}\right)q_i{q}_{i^{\prime }}.$$

Substituting, we now have

$$\mathbb{E}\left[{\breve{p}}_i{\breve{p}}_{i^{\prime }}{\breve{q}}_i{\breve{q}}_{i^{\prime }}\right]=\left({\displaystyle \sum _{j=1}^{n_1}{\displaystyle \sum _{j^{\prime }=1}^{n_1}{\displaystyle \sum _{\ell =1}^{m_j}{\displaystyle \sum _{{\ell }^{\prime }=1}^{m_{j^{\prime }}}\frac{w_j{w}_{j^{\prime }}}{m_j{m}_{j^{\prime }}}}}}}\left(1-{\Phi }_{j{j}^{\prime }}^{\left(1\right)}\right)p_i{p}_{i^{\prime }}\right)\left({\displaystyle \sum _{k=1}^{n_2}{\displaystyle \sum _{k^{\prime }=1}^{n_2}{\displaystyle \sum _{t=1}^{m_k}{\displaystyle \sum _{t^{\prime }=1}^{m_{k^{\prime }}}\frac{w_k{w}_{k^{\prime }}}{m_k{m}_{k^{\prime }}}\left(1-{\Phi }_{k{k}^{\prime }}^{\left(2\right)}\right)q_i{q}_{i^{\prime }}}}}}\right)=\left(1-{\rho }_2^{\left(1\right)}\right)\left(1-{\rho }_2^{\left(2\right)}\right)p_i{p}_{i^{\prime }}{q}_i{q}_{i^{\prime }},$$ (A16)

and substituting Equation A16 into $\text{Cov}\left[{\breve{p}}_i{\breve{q}}_i,{\breve{p}}_{i^{\prime }}{\breve{q}}_{i^{\prime }}\right]$ yields

$$\begin{array}{c}\text{Cov}\left[{\breve{p}}_i{\breve{q}}_i,{\breve{p}}_{i^{\prime }}{\breve{q}}_{i^{\prime }}\right]=\left(1-{\rho }_2^{\left(1\right)}\right)\left(1-{\rho }_2^{\left(2\right)}\right)p_i{p}_{i^{\prime }}{q}_i{q}_{i^{\prime }}-p_i{q}_i{p}_{i^{\prime }}{q}_{i^{\prime }}.\\ =\left({\rho }_2^{\left(1\right)}{\rho }_2^{\left(2\right)}-{\rho }_2^{\left(1\right)}-{\rho }_2^{\left(2\right)}\right)p_i{p}_{i^{\prime }}{q}_i{q}_{i^{\prime }}.\end{array}$$ (A17)

Therefore, using Equations A15 and A17,

$$\text{Var}\left[{\breve{H}}_{12}\right]={\displaystyle \sum _{i=1}^I{p}_i{q}_i\left[{\rho }_2^{\left(1\right)}\left(1-p_i\right)q_i+{\rho }_2^{\left(2\right)}{p}_i\left(1-q_i\right)+{\rho }_2^{\left(1\right)}{\rho }_2^{\left(2\right)}\left(1-p_i\right)\left(1-q_i\right)\right]}+2{\displaystyle \sum _{i=1}^{I-1}{\displaystyle \sum _{i^{\prime }=i+1}^I\left({\rho }_2^{\left(1\right)}{\rho }_2^{\left(2\right)}-{\rho }_2^{\left(1\right)}-{\rho }_2^{\left(2\right)}\right)p_i{p}_{i^{\prime }}{q}_i{q}_{i^{\prime }}}}={\rho }_2^{\left(1\right)}{\displaystyle \sum _{i=1}^I{p}_i\left(1-p_i\right)q_i^2}+{\rho }_2^{\left(2\right)}{\displaystyle \sum _{i=1}^I{p}_i^2{q}_i\left(1-q_i\right)}+{\rho }_2^{\left(1\right)}{\rho }_2^{\left(2\right)}{\displaystyle \sum _{i=1}^I{p}_i\left(1-p_i\right)q_i\left(1-q_i\right)}+2{\displaystyle \sum _{i=1}^{I-1}{\displaystyle \sum _{i^{\prime }=i+1}^I\left({\rho }_2^{\left(1\right)}{\rho }_2^{\left(2\right)}-{\rho }_2^{\left(1\right)}-{\rho }_2^{\left(2\right)}\right)p_i{p}_{i^{\prime }}{q}_i{q}_{i^{\prime }}}}={\rho }_2^{\left(1\right)}\left(1-{\rho }_2^{\left(2\right)}\right){\displaystyle \sum _{i=1}^I{p}_i{q}_i^2}+{\rho }_2^{\left(2\right)}\left(1-{\rho }_2^{\left(1\right)}\right){\displaystyle \sum _{i=1}^I{p}_i^2{q}_i}+\left({\rho }_2^{\left(1\right)}{\rho }_2^{\left(2\right)}-{\rho }_2^{\left(1\right)}-{\rho }_2^{\left(2\right)}\right){\displaystyle \sum _{i=1}^I{p}_i^2{q}_i^2}+{\rho }_2^{\left(1\right)}{\rho }_2^{\left(2\right)}{\displaystyle \sum _{i=1}^I{p}_i{q}_i}+2\left({\rho }_2^{\left(1\right)}{\rho }_2^{\left(2\right)}-{\rho }_2^{\left(1\right)}-{\rho }_2^{\left(2\right)}\right){\displaystyle \sum _{i=1}^I{\displaystyle \sum _{i^{\prime }=1}^I{p}_i{p}_{i^{\prime }}{q}_i{q}_{i^{\prime }}}}={\rho }_2^{\left(1\right)}\left(1-{\rho }_2^{\left(2\right)}\right){\displaystyle \sum _{i=1}^I{p}_i{q}_i^2}+{\rho }_2^{\left(2\right)}\left(1-{\rho }_2^{\left(1\right)}\right){\displaystyle \sum _{i=1}^I{p}_i^2{q}_i}+{\rho }_2^{\left(1\right)}{\rho }_2^{\left(2\right)}{\displaystyle \sum _{i=1}^I{p}_i{q}_i}+\left({\rho }_2^{\left(1\right)}{\rho }_2^{\left(2\right)}-{\rho }_2^{\left(1\right)}-{\rho }_2^{\left(2\right)}\right){\left({\displaystyle \sum _{i=1}^I{p}_i{q}_i}\right)}^2.$$ □

Lemma 12:

Consider a locus with I distinct alleles across two independent populations and parametric allele frequencies $p_i\in \left[0,1\right],$ $i=1,2,\dots ,I,$ and ${\displaystyle {\sum }_{i=1}^I{p}_i}=1$ for population 1, or $q_i\in \left[0,1\right],$ $i=1,2,\dots ,I,$ and ${\displaystyle {\sum }_{i=1}^I{q}_i}=1$ for population 2. For two samples of size $n_1$ and $n_2$ individuals from populations 1 and 2, respectively, each with individuals of any ploidy, inbreeding status, and relatedness,

$$\text{Cov}\left[{\breve{H}}_{12},{\breve{H}}_1\right]=\frac{1}{1-{\rho }_2^{\left(1\right)}}\left[2\left({\rho }_3^{\left(1\right)}-{\rho }_2^{\left(1\right)}\right){\displaystyle \sum _{i=1}^I{p}_i^3{q}_i}+\left(2{\rho }_2^{\left(1\right)}-3{\rho }_3^{\left(1\right)}\right){\displaystyle \sum _{i=1}^I{p}_i^2{q}_i}+{\rho }_3^{\left(1\right)}{\displaystyle \sum _{i=1}^I{p}_i{q}_i}\right]$$ (A18)

and

$$\text{Cov}\left[{\breve{H}}_{12},{\breve{H}}_2\right]=\frac{1}{1-{\rho }_2^{\left(2\right)}}\left[2\left({\rho }_3^{\left(2\right)}-{\rho }_2^{\left(2\right)}\right){\displaystyle \sum _{i=1}^I{p}_i{q}_i^3}+\left(2{\rho }_2^{\left(2\right)}-3{\rho }_3^{\left(2\right)}\right){\displaystyle \sum _{i=1}^I{p}_i{q}_i^2}+{\rho }_3^{\left(2\right)}{\displaystyle \sum _{i=1}^I{p}_i{q}_i}\right],$$ (A19)

where the superscript of the mean kinship coefficients ${\rho }_2$ and ${\rho }_3$ corresponds to the population for which these are calculated. The formulas for $\text{Cov}\left[{\stackrel{\sim }{H}}_{12},{\stackrel{\sim }{H}}_1\right],$ $\text{Cov}\left[{\stackrel{\sim }{H}}_{12},{\stackrel{\sim }{H}}_2\right],$ $\text{Cov}\left[{\stackrel{\sim }{H}}_{12,\text{BLUE}},{\stackrel{\sim }{H}}_{1,\text{BLUE}}\right],$ and $\text{Cov}\left[{\stackrel{\sim }{H}}_{12,\text{BLUE}},{\stackrel{\sim }{H}}_{2,\text{BLUE}}\right]$ are obtained by substituting ${\overline{\Phi }}_2$ and ${\overline{\Phi }}_3$ (for $\stackrel{\sim }{H}$), or ${\kappa }_2$ and ${\kappa }_3$ (for ${\stackrel{\sim }{H}}_{\text{BLUE}}$) into Equations A18 and A19, respectively.

Proof:

The covariance between ${\breve{H}}_{12}$ and ${\breve{H}}_1$ is

$$\begin{array}{c}\text{Cov}\left[{\breve{H}}_{12},{\breve{H}}_1\right]=\frac{1}{1-{\rho }_2^{\left(1\right)}}\text{Cov}\left[\left(1-{\displaystyle \sum _{i=1}^I{\breve{p}}_i{\breve{q}}_i}\right),\left(1-{\displaystyle \sum _{i=1}^I{\breve{p}}_i^2}\right)\right]\\ =\frac{1}{1-{\rho }_2^{\left(1\right)}}{\displaystyle \sum _{i=1}^I{\displaystyle \sum _{i^{\prime }=1}^I\text{Cov}\left[{\breve{p}}_i{\breve{q}}_i,{\breve{p}}_{i^{\prime }}^2\right]}}\\ =\frac{1}{1-{\rho }_2^{\left(1\right)}}\left({\displaystyle \sum _{i=1}^I\text{Cov}\left[{\breve{p}}_i{\breve{q}}_i,{\breve{p}}_i^2\right]}+{\displaystyle \sum _{i=1}^I{\displaystyle \sum _{\begin{array}{l}{i}^{\prime }=1\\ i^{\prime }\ne i\end{array}}^I\text{Cov}\left[{\breve{p}}_i{\breve{q}}_i,{\breve{p}}_{i^{\prime }}^2\right]}}\right).\end{array}$$

The value of the covariance calculated for the case where $i=i^{\prime }$ can be written as

$$\text{Cov}\left[{\breve{p}}_i{\breve{q}}_i,{\breve{p}}_i^2\right]=\mathbb{E}\left[{\breve{p}}_i^3{\breve{q}}_i\right]-\mathbb{E}\left[{\breve{p}}_i{\breve{q}}_i\right]\mathbb{E}\left[{\breve{p}}_i^2\right].$$

From the proof of Lemma 11, we have derived the value of $\mathbb{E}\left[{\breve{p}}_i{\breve{q}}_i\right],$ and, from the proof of Proposition 1 we know the value for $\mathbb{E}\left[{\breve{p}}_i^2\right].$ We therefore only need to compute

$$\mathbb{E}\left[{\breve{p}}_i^3{\breve{q}}_i\right]=\mathbb{E}\left[{\breve{p}}_i^3\right]\mathbb{E}\left[{\breve{q}}_i\right],$$

where $\mathbb{E}\left[{\breve{q}}_i\right]=q_i.$ Solving for $\mathbb{E}\left[{\breve{p}}_i^3\right],$ we have

$$\begin{array}{c}\mathbb{E}\left[{\breve{p}}_i^3\right]={\displaystyle \sum _{j=1}^{n_1}{\displaystyle \sum _{v=1}^{n_1}{\displaystyle \sum _{v^{\prime }=1}^{n_1}{\displaystyle \sum _{\ell =1}^{m_j}{\displaystyle \sum _{z=1}^{m_v}{\displaystyle \sum _{z^{\prime }=1}^{m_{v^{\prime }}}\frac{w_j{w}_v{w}_{v^{\prime }}}{m_j{m}_v{m}_{v^{\prime }}}}}}}}}\mathbb{E}\left[A_{j\ell }^{\left(i\right)}{A}_{vz}^{\left(i\right)}{A}_{v^{\prime }{z}^{\prime }}^{\left(i\right)}\right]\\ ={\displaystyle \sum _{j=1}^{n_1}{\displaystyle \sum _{v=1}^{n_1}{\displaystyle \sum _{v^{\prime }=1}^{n_1}{\displaystyle \sum _{\ell =1}^{m_j}{\displaystyle \sum _{z=1}^{m_v}{\displaystyle \sum _{z^{\prime }=1}^{m_{v^{\prime }}}\frac{w_j{w}_v{w}_{v^{\prime }}}{m_j{m}_v{m}_{v^{\prime }}}}}}}}}\mathrm{\mathbb{P}}\left[A_{j\ell }^{\left(i\right)}=1,A_{vz}^{\left(i\right)}=1,A_{v^{\prime }{z}^{\prime }}^{\left(i\right)}=1\right].\end{array}$$

The value of $\mathrm{\mathbb{P}}\left[A_{j\ell }^{\left(i\right)}=1,A_{vz}^{\left(i\right)}=1,A_{v\prime z\prime }^{\left(i\right)}=1\right]$ depends on the probabilities of distinct identity states in which three alleles are drawn from the sample (one each from individuals j, v, and $v^{\prime }$). We define state 1 as no IBD alleles drawn (probability ${\delta }_1$), state 2 as IBD alleles drawn from j and v (probability ${\delta }_2$), state 3 as IBD alleles drawn from v and $v^{\prime }$ IBD (probability ${\delta }_3$), state 4 as IBD alleles drawn from j and $v^{\prime }$ IBD (probability ${\delta }_4$), and state 5 as all three IBD (probability ${\delta }_5$), with ${\displaystyle {\sum }_{s=1}^5{\delta }_s}=1.$ Thus, the probabilities for the relevant kinship coefficients are

$${\Phi }_{jv{v}^{\prime }}^{\left(1\right)}={\delta }_5$$

$${\Phi }_{jv}^{\left(1\right)}={\delta }_5+{\delta }_2$$

$${\Phi }_{v{v}^{\prime }}^{\left(1\right)}={\delta }_5+{\delta }_3$$

$${\Phi }_{j{v}^{\prime }}^{\left(1\right)}={\delta }_5+{\delta }_4,$$

which yields

$$\begin{array}{l}\mathrm{\mathbb{P}}\left[A_{j\ell }^{\left(i\right)}=1,A_{vz}^{\left(i\right)}=1,A_{v^{\prime }{z}^{\prime }}^{\left(i\right)}=1\right]={\delta }_5{p}_i+\left({\delta }_2+{\delta }_3+{\delta }_4\right)p_i^2+{\delta }_1{p}_i^3\\ ={\Phi }_{jv{v}^{\prime }}^{\left(1\right)}{p}_i+\left({\Phi }_{jv}^{\left(1\right)}+{\Phi }_{v{v}^{\prime }}^{\left(1\right)}+{\Phi }_{j{v}^{\prime }}^{\left(1\right)}-3{\Phi }_{jv{v}^{\prime }}^{\left(1\right)}\right)p_i^2+\left(1+2{\Phi }_{jv{v}^{\prime }}^{\left(1\right)}-{\Phi }_{j{v}^{\prime }}^{\left(1\right)}-{\Phi }_{v{v}^{\prime }}^{\left(1\right)}-{\Phi }_{jv}^{\left(1\right)}\right)p_i^3.\end{array}$$

Thus, $\mathbb{E}\left[{\breve{p}}_i^3{\breve{q}}_i\right]$ is

$$\mathbb{E}\left[{\breve{p}}_i^3{\breve{q}}_i\right]={\rho }_3^{\left(1\right)}{p}_i{q}_i+3\left({\rho }_2^{\left(1\right)}-{\rho }_3^{\left(1\right)}\right)p_i^2{q}_i+\left(1+2{\rho }_3^{\left(1\right)}-3{\rho }_2^{\left(1\right)}\right)p_i^3{q}_i,$$ (A20)

and from Equations A20 and A1, and the definition of $\mathbb{E}\left[{\breve{p}}_i{\breve{q}}_i\right],$

$$\begin{array}{c}\text{Cov}\left[{\breve{p}}_i{\breve{q}}_i,{\breve{p}}_i^2\right]={\rho }_3^{\left(1\right)}{p}_i{q}_i+3\left({\rho }_2^{\left(1\right)}-{\rho }_3^{\left(1\right)}\right)p_i^2{q}_i+\left(1+2{\rho }_3^{\left(1\right)}-3{\rho }_2^{\left(1\right)}\right)p_i^3{q}_i-p_i{q}_i\left[p_i^2+{\rho }_2^{\left(1\right)}{p}_i\left(1-p_i\right)\right]\\ =2\left({\rho }_3^{\left(1\right)}-{\rho }_2^{\left(1\right)}\right)p_i^3{q}_i+\left(2{\rho }_2^{\left(1\right)}-3{\rho }_3^{\left(1\right)}\right)p_i^2{q}_i+{\rho }_3^{\left(1\right)}{p}_i{q}_i.\end{array}$$ (A21)

Meanwhile, for the $\text{Cov}\left[{\breve{p}}_i{\breve{q}}_i,{\breve{p}}_{i^{\prime }}^2\right]$ case of $i\ne i^{\prime },$ $\text{Cov}\left[{\breve{p}}_i{\breve{q}}_i,{\breve{p}}_{i^{\prime }}^2\right]=0.$ This is intuitively sensible because the products ${\breve{p}}_i{\breve{q}}_i$ and ${\breve{p}}_{i^{\prime }}^2$ are independent, describing different alleles, and should not covary.

Finally, we can see that, when the two populations considered are independent from one another, the value of $\text{Cov}\left[{\breve{H}}_{12},{\breve{H}}_1\right]$ (or equivalently of $\text{Cov}\left[{\breve{H}}_{12},{\breve{H}}_2\right]$) is driven entirely by the case in which $i=i^{\prime },$ such that

$$\begin{array}{c}\mathrm{Cov}\left[{\breve{H}}_{12},{\breve{H}}_1\right]=\frac{1}{1-{\rho }_2^{\left(1\right)}}{\displaystyle \sum _{i=1}^I\text{Cov}\left[{\breve{p}}_i{\breve{q}}_i,{\breve{p}}_i^2\right]}\\ =\frac{1}{1-{\rho }_2^{\left(1\right)}}\left[2\left({\rho }_3^{\left(1\right)}-{\rho }_2^{\left(1\right)}\right){\displaystyle \sum _{i=1}^I{p}_i^3{q}_i}+\left(2{\rho }_2^{\left(1\right)}-3{\rho }_3^{\left(1\right)}\right){\displaystyle \sum _{i=1}^I{p}_i^2{q}_i}+{\rho }_3^{\left(1\right)}{\displaystyle \sum _{i=1}^I{p}_i{q}_i}\right].\end{array}$$ □

We now need to derive $\text{Cov}\left[{\breve{H}}_{12}-\frac{1}{2}\left({\breve{H}}_1+{\breve{H}}_2\right),{\breve{H}}_{12}\right],$ the final term required to compute $\mathrm{Var}\left[{\breve{F}}_{\text{ST}}\right].$

Lemma 13:

Consider a locus with I distinct alleles across two independent populations and parametric allele frequencies $p_i\in \left[0,1\right],$ $i=1,2,\dots ,I,$ and ${\displaystyle {\sum }_{i=1}^I{p}_i}=1$ for population 1, or $q_i\in \left[0,1\right],$ $i=1,2,\dots ,I,$ and ${\displaystyle {\sum }_{i=1}^I{q}_i}=1$ for population 2. For two samples of size $n_1$ and $n_2$ individuals from populations 1 and 2, respectively, each with individuals of any ploidy, inbreeding status, and relatedness,

$$\begin{array}{c}\text{Cov}\left[{\breve{H}}_{12}-\frac{1}{2}{\breve{H}}_1-\frac{1}{2}{\breve{H}}_2,{\breve{H}}_{12}\right]=\left[{\rho }_2^{\left(1\right)}\left(1-{\rho }_2^{\left(2\right)}\right)+\frac{3{\rho }_3^{\left(2\right)}-2{\rho }_2^{\left(2\right)}}{2\left(1-{\rho }_2^{\left(2\right)}\right)}\right]{\displaystyle \sum _{i=1}^I{p}_i{q}_i^2}+\left[{\rho }_2^{\left(2\right)}\left(1-{\rho }_2^{\left(1\right)}\right)+\frac{3{\rho }_3^{\left(1\right)}-2{\rho }_2^{\left(1\right)}}{2\left(1-{\rho }_2^{\left(1\right)}\right)}\right]\stackrel{I}{{\displaystyle \sum }_{i=1}}{p}_i^2{q}_i\\ +\left[{\rho }_2^{\left(1\right)}{\rho }_2^{\left(2\right)}-\frac{{\rho }_3^{\left(1\right)}}{2\left(1-{\rho }_2^{\left(1\right)}\right)}-\frac{{\rho }_3^{\left(2\right)}}{2\left(1-{\rho }_2^{\left(2\right)}\right)}\right]{\displaystyle \sum _{i=1}^I{p}_i{q}_i}+\left({\rho }_2^{\left(1\right)}{\rho }_2^{\left(2\right)}-{\rho }_2^{\left(1\right)}-{\rho }_2^{\left(2\right)}\right){\left({\displaystyle \sum _{i=1}^I{p}_i{q}_i}\right)}^2\\ +\frac{{\rho }_2^{\left(1\right)}-{\rho }_3^{\left(1\right)}}{1-{\rho }_2^{\left(1\right)}}{\displaystyle \sum _{i=1}^I{p}_i^3{q}_i}+\frac{{\rho }_2^{\left(2\right)}-{\rho }_3^{\left(2\right)}}{1-{\rho }_2^{\left(2\right)}}{\displaystyle \sum _{i=1}^I{p}_i{q}_i^3},\end{array}$$ (A22)

where the superscript of the mean kinship coefficients ${\rho }_2$ and ${\rho }_3$ corresponds to the population for which these quantities are calculated. The formulas for $\text{Cov}\left[{\stackrel{\sim }{H}}_{12}-\left(1/2\right){\stackrel{\sim }{H}}_1-\left(1/2\right){\stackrel{\sim }{H}}_2,{\stackrel{\sim }{H}}_{12}\right]$ and $\mathrm{Cov}\left[{\stackrel{\sim }{H}}_{12,\text{BLUE}}-\left(1/2\right){\stackrel{\sim }{H}}_{1,\text{BLUE}}-\left(1/2\right){\stackrel{\sim }{H}}_{2,\text{BLUE}},{\stackrel{\sim }{H}}_{12,\text{BLUE}}\right]$ are obtained by substituting ${\overline{\Phi }}_2$ and ${\overline{\Phi }}_3$ (for $\stackrel{\sim }{H}$), or ${\kappa }_2$ and ${\kappa }_3$ (for ${\stackrel{\sim }{H}}_{\text{BLUE}}$) into Equation A22.

Proof:

We begin by breaking up the covariance into its components,

$$\mathrm{Cov}\left[{\breve{H}}_{12}-\frac{1}{2}{\breve{H}}_1-\frac{1}{2}{\breve{H}}_2,{\breve{H}}_{12}\right]=\mathrm{Var}\left[{\breve{H}}_{12}\right]-\frac{1}{2}\mathrm{Cov}\left[{\breve{H}}_1,{\breve{H}}_{12}\right]-\frac{1}{2}\mathrm{Cov}\left[{\breve{H}}_2,{\breve{H}}_{12}\right].$$

This equation is composed of terms that we previously derived (Equations A13, A18, and A19). Therefore,

$$\begin{array}{c}\text{Cov}\left[{\breve{H}}_{12}-\frac{1}{2}{\breve{H}}_1-\frac{1}{2}{\breve{H}}_2,{\breve{H}}_{12}\right]={\rho }_2^{\left(1\right)}\left(1-{\rho }_2^{\left(2\right)}\right){\displaystyle \sum _{i=1}^I{p}_i{q}_i^2}+{\rho }_2^{\left(2\right)}\left(1-{\rho }_2^{\left(1\right)}\right){\displaystyle \sum _{i=1}^I{p}_i^2{q}_i}+{\rho }_2^{\left(1\right)}{\rho }_2^{\left(2\right)}{\displaystyle \sum _{i=1}^I{p}_i{q}_i}\\ +\left({\rho }_2^{\left(1\right)}{\rho }_2^{\left(2\right)}-{\rho }_2^{\left(1\right)}-{\rho }_2^{\left(2\right)}\right){\left({\displaystyle \sum _{i=1}^I{p}_i{q}_i}\right)}^2\\ -\frac{1}{2\left(1-{\rho }_2^{\left(1\right)}\right)}\left[2\left({\rho }_3^{\left(1\right)}-{\rho }_2^{\left(1\right)}\right){\displaystyle \sum _{i=1}^I{p}_i^3{q}_i}+\left(2{\rho }_2^{\left(1\right)}-3{\rho }_3^{\left(1\right)}\right){\displaystyle \sum _{i=1}^I{p}_i^2{q}_i}+{\rho }_3^{\left(1\right)}{\displaystyle \sum _{i=1}^I{p}_i{q}_i}\right]\\ -\frac{1}{2\left(1-{\rho }_2^{\left(2\right)}\right)}\left[2\left({\rho }_3^{\left(2\right)}-{\rho }_2^{\left(2\right)}\right){\displaystyle \sum _{i=1}^I{p}_i{q}_i^3}+\left(2{\rho }_2^{\left(2\right)}-3{\rho }_3^{\left(2\right)}\right){\displaystyle \sum _{i=1}^I{p}_i{q}_i^2}+{\rho }_3^{\left(2\right)}{\displaystyle \sum _{i=1}^I{p}_i{q}_i}\right]\\ =\left[{\rho }_2^{\left(1\right)}\left(1-{\rho }_2^{\left(2\right)}\right)+\frac{3{\rho }_3^{\left(2\right)}-2{\rho }_2^{\left(2\right)}}{2\left(1-{\rho }_2^{\left(2\right)}\right)}\right]{\displaystyle \sum _{i=1}^I{p}_i{q}_i^2}+\left[{\rho }_2^{\left(2\right)}\left(1-{\rho }_2^{\left(1\right)}\right)+\frac{3{\rho }_3^{\left(1\right)}-2{\rho }_2^{\left(1\right)}}{2\left(1-{\rho }_2^{\left(1\right)}\right)}\right]{\displaystyle \sum _{i=1}^I{p}_i^2{q}_i}\\ +\left[{\rho }_2^{\left(1\right)}{\rho }_2^{\left(2\right)}-\frac{{\rho }_3^{\left(1\right)}}{2\left(1-{\rho }_2^{\left(1\right)}\right)}-\frac{{\rho }_3^{\left(2\right)}}{2\left(1-{\rho }_2^{\left(2\right)}\right)}\right]{\displaystyle \sum _{i=1}^I{p}_i{q}_i}+\left({\rho }_2^{\left(1\right)}{\rho }_2^{\left(2\right)}-{\rho }_2^{\left(1\right)}-{\rho }_2^{\left(2\right)}\right){\left({\displaystyle \sum _{i=1}^I{p}_i{q}_i}\right)}^2\\ +\frac{{\rho }_2^{\left(1\right)}-{\rho }_3^{\left(1\right)}}{1-{\rho }_2^{\left(1\right)}}{\displaystyle \sum _{i=1}^I{p}_i^3{q}_i}+\frac{{\rho }_2^{\left(2\right)}-{\rho }_3^{\left(2\right)}}{1-{\rho }_2^{\left(2\right)}}{\displaystyle \sum _{i=1}^I{p}_i{q}_i^3}.\end{array}$$ □