Fig 1. Accuracy of F_ST and kinship estimators: Overview of models and results.

Our analysis is based on the generalized F_ST definition (section The generalized F_ST for arbitrary population structures) and two parallel models: the “Coancestry Model” for individual-specific allele frequencies (π_ij), and the “Kinship Model” for genotypes (x_ij). The “Coancestry in Terms of Kinship” panel connects kinship (${\phi }_{jk}^T$, $f_j^T$) and coancestry (${\theta }_{jk}^T$) parameters (section The kinship and coancestry models). We use these models to study the accuracy of F_ST and kinship method-of-moment estimators under arbitrary population structures. The “Indep. Subpop. F_ST Estimator” panel shows the bias resulting from the misapplication of F_ST estimators for independent subpopulations (${\widehat{F}}_{\text{ST}}^{\text{indep}}$) to arbitrary structures (section F_ST estimation based on the independent subpopulations model), as calculated under the coancestry model. The “Existing Kinship Estimator” panel shows the bias in the standard kinship model estimator (${\widehat{\phi }}_{jk}^{T,\text{std}}$) and its resulting plug-in F_ST estimator (${\widehat{F}}_{\text{ST}}^{\text{std}}$; section Characterizing a kinship estimator and its relationship to F_ST), as calculated under the kinship model. The “New Kinship Estimator” panel presents a new statistic A_jk that estimates kinship with a uniform bias, which together with a consistent estimator of its minimum value (${\widehat{A}}_{\text{min}}$) results in our new kinship (${\widehat{\phi }}_{jk}^{T,\text{new}}$) and F_ST (${\widehat{F}}_{\text{ST}}^{\text{new}}$) estimators, which are consistent under arbitrary population structure (section A new approach for kinship and F_ST estimation).