Why are rare variants hard to impute? Coalescent models reveal theoretical limits in existing algorithms

Abstract

Genotype imputation is an indispensable step in human genetic studies. Large reference panels with deeply sequenced genomes now allow interrogating variants with minor allele frequency < 1% without sequencing. Although it is critical to consider limits of this approach, imputation methods for rare variants have only done so empirically; the theoretical basis of their imputation accuracy has not been explored. To provide theoretical consideration of imputation accuracy under the current imputation framework, we develop a coalescent model of imputing rare variants, leveraging the joint genealogy of the sample to be imputed and reference individuals. We show that broadly used imputation algorithms include model misspecifications about this joint genealogy that limit the ability to correctly impute rare variants. We develop closed-form solutions for the probability distribution of this joint genealogy and quantify the inevitable error rate resulting from the model misspecification across a range of allele frequencies and reference sample sizes. We show that the probability of a falsely imputed minor allele decreases with reference sample size, but the proportion of falsely imputed minor alleles mostly depends on the allele count in the reference sample. We summarize the impact of this error on genotype imputation on association tests by calculating the r² between imputed and true genotype and show that even when modeling other sources of error, the impact of the model misspecification has a significant impact on the r² of rare variants. To evaluate these predictions in practice, we compare the imputation of the same dataset across imputation panels of different sizes. Although this empirical imputation accuracy is substantially lower than our theoretical prediction, modeling misspecification seems to further decrease imputation accuracy for variants with low allele counts in the reference. These results provide a framework for developing new imputation algorithms and for interpreting rare variant association analyses.

Introduction

Genome sequencing studies with deep coverage allow directly assessing rare genetic variants (Lee et al.2014) to elucidate their impact on the etiology of complex diseases (Timpson et al. 2018). However, such studies are still expensive. As an alternative, array-based technologies can be employed at a substantially lower cost. Commercial genotyping arrays cover a pre-selected set of common variants, and in some cases low-frequency variants known to be of interest from previous studies. To recover high-resolution genetic information, genotype imputation compares assayed genotypes to a sequenced reference panel, thus leveraging the shared genealogy between genotyped (target) individuals and reference panel to infer unobserved genotypes (Servin and Stephens 2007).

Modern imputation methods combined with large reference panels have achieved high accuracy among even low-frequency variants. For example, using the TOPMed data as reference, the average imputation quality (r²) of variants with frequency 0.1% is over 0.90 for both African and European ancestry genomes (Taliun et al. 2021). Such high resolution improves the power of genome-wide association studies for low frequency and rare variants, and enables joint analysis across studies with different sets of genotyped variants (Das et al. 2016): Recently, the TOPMed consortium identified a new risk variant for breast cancer by imputing rare variants with minor allele frequency (MAF) < 0.5% into the UK Biobank (Taliun et al. 2021). The advent of affordable whole-genome sequencing generating large collections of reference haplotypes combined with efficient imputation algorithms will power more of such discoveries.

Most of these modern imputation methods are based on the Li and Stephens’ (2003) model. They leverage that haplotypes from unrelated individuals share chromosome segments from a common ancestor. These segments are more similar to each other if their common ancestor is more recent. Among haplotypes in a large reference panel, the haplotype that has the most recent common ancestor (MRCA) with the target haplotype can thus provide information about the unobserved genotypes of the target. Using a Hidden Markov Model (HMM), the Li and Stephens’ algorithm models each target haplotype in the study as an imperfect mosaic of haplotypes from the reference panel. The haplotypes making up this mosaic are inferred to be the most closely related to the target, and thus provide information for unobserved genotype information. The HMM framework is computationally tractable and naturally approximates recombination and mutation in its transition and emission probabilities.

Simulation studies have shown that imputation quality depends on the imputed variants’ allele frequency, genomic context, the size of the reference panel, and population demographics (Huang et al. 2013; Das et al. 2016). Such studies illustrate that, despite significant improvements, imputation of rare variants still has high uncertainty: The average squared correlation (r²) for variants with MAF 0.01% is below 0.5 even with the largest reference from the same continental population (Taliun et al. 2021). But, simulation studies only provide a limited opportunity to understand these limits. Although they have the flexibility to mimic a particular imputation setting, they are often computationally expensive and hard to generalize. Alternatively, probabilistic models capturing the basic properties of the imputation process can make predictions before data collection and are easily generalizable.

The key in such a probabilistic model is the relatedness between reference and target haplotypes, which explains how the reference informs unobserved genotypes in target haplotypes. Kingman’s (1982) coalescent provides a suitable theoretical framework for modeling the shared genealogy of these haplotypes. The coalescent traces the genealogy back in time, modeling a sequence of events where individuals find their common ancestors. Mutations resulting in polymorphisms can be mapped to branches on the coalescence tree at each locus, with all its descendants carrying the derived allele (Figure 1). The coalescent time (the time it takes for two or more haplotypes to find their MRCA and TMRCA) thus gives a measure of expected genetic dissimilarity, since only mutations occurring more recent than the coalescent time can result in different alleles among those haplotypes.

Figure 1.

Connection between genotype sequences and coalescence tree. On the left is the genotype matrix of eight haplotypes (a–g) as reference plus a target haplotype at five SNPs (1–5). Ancestral alleles are colored black, derived (mutated) alleles are colored red. Question marks denote sites where the target is not genotyped. The Li and Stephens algorithm models the target haplotype as the observation from a HMM where reference haplotypes a–g constitute the state space. By identifying which of the reference haplotypes is the hidden state of the target haplotype, the algorithm can identify which allele to infer for the unknown sites. Note that haplotypes c, d, and e all have the same alleles as the target at the genotyped loci, making them equally likely to be the hidden state without further information. On the right is the corresponding coalescence tree assuming no recombination. Mutations are labeled on the branch, with all the descendants (leaves) carrying the derived allele. Template haplotypes c, d, or e being the hidden state is equivalent to the target haplotype coalescing at one of the yellow arrows. As the Li and Stephens algorithm assumes a single best template, it does not consider the possibility that the target may fist coalesce with the branch pointed by the blue arrow. All indicated coalescence events are compatible with the observed genotypes but give different imputation results.

Some aspects of imputation accuracy have already been explored using coalescent theory. Jewett et al. (2012) studied the scenario where the target and reference are sampled from two populations diverged in the past, and derived expected imputation error rate as a function of reference size and divergence time. Huang et al. (2013) further included mutation rate and marker density as factors; and analyzed the potential gain in accuracy by choosing the reference panel from a more closely related population.

Here, we develop a coalescent approach to understand imputation accuracy within population, focused on rare variants. It is useful to recognize that imputation error has two types of sources: (1) failure to identify haplotypes most closely related to the target as the template and (2) true differences between the template and the target haplotype due to recent mutation events. Although improving imputation algorithms may reduce error of the first type, error of the second type is a result of the underlying genealogical relation between the target and the reference sample and how this relationship is modeled. Here, we focus on modeling the second type of error, where we determine the error that is immanent to the Li and Stephens’ model.

For this purpose, we consider one target haplotype with missing genotype and n fully sequenced haplotypes as references. These n + 1 observed lineages form the leaves of a binary tree with their (unobserved) ancestors as internal nodes. Intuitively, if the target first finds its MRCA with a set of reference haplotypes, their genotypes are the most similar to the target thus the most informative for imputation. The Li and Stephens’ algorithm assumes that there is exactly one such most closely related reference haplotype, but we show that this assumption is wrong with probability 1/3.

We compute how often this misspecification leads to an ambiguous or wrongly imputed genotype, assuming the imputation algorithm correctly identifies exactly those haplotypes that are most closely related to the target sample in the reference. We then provide the probability of generating a particular imputed dosage conditional on allele frequency of the variant and the size of reference panel. We also quantify the imputation accuracy in terms of the r² between the imputed dosage and the true genotype, and show that, as a result of this misspecification, the r² largely depends on the allele count in the reference panel, improving only marginally with increased reference size. We assess the impact of population history on these results and use coalescent simulation to confirm our analytic results. Taken together, our approach provides the minimum size of the reference panel necessary to achieve the desired imputation accuracy for a given allele frequency.

We evaluate the upper bounds predicted with this model where the only source of error is model misspecification by imputing a sample of 56,984 individuals using reference panels of different sizes and evaluating the empirical imputation accuracy > 60,000 variants. We observe substantially higher imputation error in the empirical results than the theoretical bound in our model. Moreover, when conditioning on minor allele count (MAC), imputation error is larger in larger reference panels. We also observe that for MAC 2–10, imputation error increases much faster with decreasing MAC, consistent with our result that especially for these MACs, model misspecification reduces imputation accuracy.

The model we develop here can also be leveraged to improve current imputation algorithms. For example, most imputation algorithms assume that the distance between switching template haplotypes reflects recombination events and is exponentially distributed (Paul et al. 2011). We derive the length distribution of contiguous segments without observable recombination breakpoints, and show that it differs substantially from the exponential assumption by having thicker tails for both very short and long segments.

Methods

We consider imputing a single-target variant for one target haplotype with a reference panel consisting of n reference haplotypes. For simplicity, we only discuss biallelic sites denoting the ancestral allele as 0 and derived allele as 1. At each SNP site, all reference haplotypes and the target form a coalescent tree with n + 1 leaves. This tree is unknown but represents the complete information that imputation can possibly use (Figure 1). If the target’s MRCA with the reference sample occurs on a branch that is ancestral to $u\ge 1$ reference haplotypes, all u present day descendants of that branch are the most closely related reference haplotypes. We assume that the imputation algorithm identifies these most closely related reference haplotypes, then assigns the mean genotype of them to the target haplotype as the imputation dosage.

Under this assumption, we consider three scenarios: (1) The mutation generating the target variant is ancestral to the TMRCA of the target haplotype and all its most closely related reference haplotypes. In this case, it will be imputed correctly with dosage 1 (Figure 2a). (2) The mutation occurred more recently than the TMRCA and the mutation occurred on the branch to the target haplotype. Then the target variant is not polymorphic in the reference sample and will always be falsely imputed to be the ancestral allele (Figure 2b). (3) The mutation occurred more recently than the TMRCA and the mutation occurred on the branches to the reference haplotype. Then some or all of the most closely related reference haplotypes carry the derived allele while the template haplotype carries the ancestral allele and the reference sample will be imputed to carry the derived allele with some dosage > 0 (Figure 2, c and d).

Figure 2.

Possible scenarios when imputing one target haplotype (yellow lineage) using a reference panel with n haplotypes. Red lines are haplotypes carrying a mutation that occurs on the blue branch. Gray horizontal dashed line indicate the time intervals where an event of interest occurs. (a) The derived allele arises ancestral to the MRCA of the target with the reference haplotypes and is shared between the target (g = 1) and all most closely related haplotypes. Thus it will be correctly imputed ($\widehat{g}\approx 1$). (b) The derived allele arises on the branch of the target (g = 1) and is thus absent from the reference. It will be falsely imputed to be ancestral ($\widehat{g}=0$). (c) The derived allele arises after the MRCA of the closest reference haplotypes before coalescing with the target haplotype. The target haplotype does not carry the derived allele (g = 0), but all closest reference haplotypes do. The target will be imputed to carry the derived allele ($\widehat{g}\approx 1$). (d) The derived allele arises on a branch ancestral to j = 2 of the closest reference haplotypes before the MRCA of all u = 4 closest haplotypes. The target haplotype does not carry the derived allele (g = 0). The target’s derived allele dosage will be imputed to reflect that j of u closest haplotypes carry the derived allele ($\widehat{g}\approx j/u=0.5$).

We focus on the third scenario where j of the u most closely related templates carry the derived allele while the target does not. The Li and Stephens model assumes a single closest haplotype, and in the HMM implementation the imputed fractional genotype (dosage) is a weighted average of multiple reference haplotypes based on their posterior probabilities of being that right template. When the u is equivalently close to the target, they are expected to have equal posterior probabilities. Thus the imputed dosage $\widehat{g}$ is $\widehat{g}=j/u\in (0,1]$ while the true genotype g is g = 0 (“false positive”), resulting in loss of information in downstream analysis.

In the following sections, we derive the probability for all possible (j, u) configuration, conditional on observing the derived allele count (DAC)j in the whole reference. For simplicity, we distinguish the one target haplotype from the rest n reference ones, making the whole tree size n + 1, although they are exchangeable under the assumption of homogeneity and random mating. We will always consider time backward with t = 0 being the current generation. Notations are listed in Table 1.

Table 1.

Notation for key quantities

Symbol	Meaning
g	True genotype at the imputed locus
$\widehat{g}$	Estimated genotype at the imputed locus
n	Number of haplotypes in the reference panel.
u	Number of most closely related haplotypes (MCRH) in the reference panel
j	Number of the rare allele in the reference panel
k	Number of lines remaining on the tree just after all MCRH have reached their MRCA
d	Number of lines remaining on the tree when the target haplotype coalesces with one ancestor of the reference haplotypes
l	Length of an internal branch
m	Number of mutation events on a particular branch

Number of most closely related templates

We first give the probability of having u equally good templates at any random position for a target haplotype: $P(u;n)$. This probability depends only the topology of the genealogy, independent from mutation events. We leverage that the probability that a set of k lines coalesce before they coalesce with any line among the rest n—k (Wiuf and Donnelly 1999) is:

$$q_{k,n-k}=\frac{2(k-1)!(n-k)!}{(k+1)(n-1)!}.$$ (1)

The generative process can then be imagined as three steps: u templates coalesce first before their MRCA meets the target; then the resulting branch of size u + 1 meets the rest of the tree. Finally, we sum over all possible sets of u templates:

$$\begin{array}{cc}P(u;n)\hfill & =q_{u+1,n-u}·q_{u,1}·\left(\begin{array}{c}n\\ u\end{array}\right)\hspace{1em}(u<n)\hfill \\ \hfill & =\frac{2u!(n-u)!}{(u+2)n!}\frac{2(u-1)!}{(u+1)u!}\frac{n!}{(n-u)!u!}\hfill \\ \hfill & =\frac{4}{u(u+1)(u+2)}\hspace{1em}\hspace{1em}(1\le u\le n-1)\hfill \\ P(u=n;n)\hfill & =q_{n,1}=\frac{2}{n(n+1)}\hfill \end{array}.$$ (2)

The expected number of best templates is close to 2 when the reference is large:

$$\begin{array}{c}E[u;n]=\sum _{u=1}^{n}uP(u;n)\hfill \\ =\sum _{u=1}^{n-1}u\frac{4}{u(u+1)(u+2)}+n·q_{n,1}\hfill \\ =\frac{2n}{n+1}\approx 2\hfill \end{array}.$$ (3)

High certainty error

We now derive the probability of imputing the target to carry the mutation with dosage $\widehat{g}=1$ while the truth is g = 0, conditional on the observed DAC in the reference panel: $P(\widehat{g}=1,g=0|j;n)$. This happens when the target haplotype first coalesces with a branch (l) of size j carrying a mutation (Figure 2C).

Throughout the following derivation, we use the number of ancestral lines k for the current day sample to keep track of coalescent time and to connect topology and branch lengths. We first introduce some quantities useful for our derivation:

Let $P(j,k;n)$ be the probability for j lines to reach their MRCA at the coalescent event that reduces the overall number of ancestral lines from k + 1 to k, without any of the j lines coalescing with any of the n−j other lines first. Let $P_0(n,d)$ be the probability for one line to encounter no coalescent event for (at least) the first n−d events. Let $P(m\ge 1|l)$ be the probability of having at least one mutation event on a branch with length l, and let $P(L=l|k,d;n)$ be the probability density function for the length of one internal branch starting when there are k lines left and ending when there are d lines left, in a tree of size n. Using these terms, we can now calculate:

$$P(\widehat{g}=1,g=0 | j;n)=\frac{P(\widehat{g}=1,g=0,j;n)}{P(j;n)}.$$

We calculate the joint probability $P(\widehat{g}=1,g=0,j;n)$ in the following three steps by conditioning on d and k (Equation 4 and Figure 2C). We calculate $P(j;n)$ by adapting the second step (Equation 5).

1. To coalesce in the $(n-d+1)$th coalescent event, when d lines remain on the tree, the target haplotype cannot coalesce in the first n−d events. By definition of P₀, this is $P_0(n+1,d+1)$.

2. A branch of size j ancestral to all most closely related reference haplotypes arises in the reference at the $(n-k)$th event ($P(j,k;n)$) and encounters a mutation before it coalesces with the target branch in event $n-d+1$ ($g(k,d;n)$). Here, the probability for a mutation to occur on that particular branch is computed by integrating over all possible branch length l given k and d: $g(k,d;n)={\int }_{l=0}^{l=\infty }P(m\ge 1|l)P(L=l|k+1,d;n+1)dl$.

3. The size-j branch does not coalesce till the $(n-d)$th event and then coalesces with the target: $P_0(k,d)·\frac{1}{\left(\begin{array}{c}d+1\\ 2\end{array}\right)}$.

Multiplying the probabilities of these sequential events and summing over all possible values of d, k give the joint probability:

$$\begin{array}{cc}\hfill & P(\widehat{g}=1,g=0,j;n)\hfill \\ =\hfill & \sum _{k=2}^{n-j+1}\sum _{d=2}^k{P}_0(n+1,d+1)·P(j,k;n)·g(k,d;n)·P_0(k,d)\frac{1}{\left(\begin{array}{c}d+1\\ 2\end{array}\right)}\hfill \\ \hfill & g(k,d;n)={\int }_{l=0}^{l=\infty }P(m\ge 1|l)P(L=l|k+1,d;n+1)dl\hfill \end{array}$$ (4)

$$\begin{array}{cc}\hfill & P(j;n)\hfill \\ =\hfill & \sum _{k=2}^{n-j+1}\sum _{d=1}^{k-1}P(j,k;n)·g(k-1,d;n+1)·P_0(k,d+1)\frac{d}{\left(\begin{array}{c}d+1\\ 2\end{array}\right)}\hfill \\ \hfill & g(k-1,d;n+1)={\int }_{l=0}^{l=\infty }P(m\ge 1|l)P(L=l|k,d;n)dl\hfill \end{array}.$$ (5)

To compute $P(j,k;n)$, define $H(n,k):=\left(\begin{array}{c}n\\ 2\end{array}\right)\left(\begin{array}{c}n-1\\ 2\end{array}\right)\cdots \left(\begin{array}{c}k+1\\ 2\end{array}\right)$, the number of possible configurations of the topological history for n lines to coalesce till there are k ancestral lines left. Then:

$$\begin{array}{cc}P(j,k;n)\hfill & =\frac{\left(\begin{array}{c}n\\ j\end{array}\right)H(j,1)H(n-j,k-1)\left(\begin{array}{c}n-k-1\\ j-2\end{array}\right)}{H(n,k)}\hfill \\ \hfill & =\frac{\left(\begin{array}{c}n-k-1\\ j-2\end{array}\right)}{\left(\begin{array}{c}n-1\\ j\end{array}\right)}\frac{(k-1)k}{j}.\hfill \end{array}$$ (6)

Next we consider the integral corresponding to the mutation event $g(k,d;n)$ (Equation 7). Let, $E[T_i;n]$ denote the expected time for the number of ancestral lines to go from i to i−1 in the coalescence process for a sample of n current haplotypes, which depends on the population history model. Thus, $g(k,d;n)$ depends on mutation rate μ and expected coalescent time intervals $E[T_k;n]$, which can be numerically computed (Polanski et al. 2017) or approximated by Monte Carlo.

$$\begin{array}{cc}\hfill & g(k,d;n)={\int }_{l=0}^{l=\infty }P(m\ge 1|l)P(L=l|k,d;n)dl\hfill \\ \hfill & ={\int }_l(1-e^{-l\mu })P(L=l|k,d;n)dl\hfill \\ \hfill & =E[1-e^{-l\mu }|k,d;n]\hfill \\ \hfill & \approx \mu E[l|k,d;n]\hfill \\ \hfill & =\mu \sum _{i=d+1}^{i=k}E[T_k;n]\hfill \end{array}.$$ (7)

Substituting Equations (6), (7), and $P_0(k,d)=\frac{d(d-1)}{k(k-1)}$ into Equations (4) and (7), we now have the conditional probability fully defined:

$$\begin{array}{cc}\hfill & P(\widehat{g}=1,g=0|j;n)\hfill \\ =\hfill & \frac{\frac{1}{n(n+1)}{\sum }_{k=2}^{n-j+1}P(j,k;n)\frac{1}{(k-1)k}{\sum }_{d=2}^{k}d(d-1)\hspace{0.17em}g(k+1,d;n+1)}{{\sum }_{k=2}^{n-j+1}P(j,k;n)\frac{1}{(k-1)k}{\sum }_{d=1}^{k-1}d\hspace{0.17em}g(k,d;n)}\hfill \end{array}.$$ (8)

Following similar logic as above, we then consider the conditional probability distribution for fractional dosages. We derive $P(\widehat{g}=j/u,g=0|j;n)$, the probability of having u equally closely related best templates, j of which carry the mutation, for a target actually carrying the ancestral allele, conditional on observing the DAC as j in a reference of size n (Figure 2D). Details of derivation are in the Appendix.

Modeling r² as a function of misidentification proportion

The above derivation fully characterizes the distribution of the imputed dosages assuming we perfectly identify the optimal templates. Although it is difficult to model all the possible sources of error in the real imputation process, we can use the total weight (q) attributed to haplotypes outside the set of most closely related templates to measure the identification error. Here, we derive its influence on the square of correlation coefficient (r²) between the true genotype and imputed dosage.

Let g be the true genotype, 1 for the derived allele and 0 for the ancestral allele; let f be the derived allele frequency (DAF), so $E[g]=f$. Let $\tilde{g}$ and $\widehat{g}$ be the imputed dosage with and without misidentification, respectively.

Let z be the random variable representing the proportion of carriers of the minor allele among the suboptimal templates contributing to the imputed dosage, so that $\tilde{g}$ can be modeled as a mixture $\tilde{g}=(1-q)\widehat{g}+qz$. If we assume that the suboptimal templates are sampled at random from the population and each of the suboptimal templates is given equal weight, z is the average genotype of m random haplotypes, so E(z) = f and $V(z)=V(g)/m=f(1-f)/m$.

When the optimal templates for a target sample provide the perfect information $(\widehat{g}=g)$, misidentified templates may be the alternate allele and therefore increase imputation error. However, where the target sample carries the ancestral allele but some of its optimal templates carry the derived allele, suboptimal templates attenuate the resulting error. Nevertheless, here we show that in expectation imputation error will always reduce the correlation between the true and the imputed genotype. Let’s express $r_{g,\tilde{g}}^2$ in terms of quantities regarding the theoretical imputed dosage $\widehat{g}$.

$$\begin{array}{cc}\tilde{g}\hfill & =(1-q)\widehat{g}+qz\hfill \\ \Rightarrow {\tilde{r}}_{g,\tilde{g}}^2\hfill & =\frac{\mathit{Cov}{(g,\tilde{g})}^2}{V(g)V(\tilde{g})}=\frac{{((1-q)\mathit{Cov}(g,\widehat{g})+\mathit{qCov}(g,z))}^2}{V(g)V((1-q)\widehat{g}+qz)}\hfill \\ \hfill & =\frac{\mathit{Cov}{(g,\widehat{g})}^2+\frac{2q}{1-q}Cov(g,\widehat{g})\mathit{Cov}(g,z)+\frac{q^2}{{(1-q)}^2}Cov{(g,z)}^2}{V(g)V(\widehat{g})+\frac{2q}{1-q}V(g)\mathit{Cov}(\widehat{g},z)+\frac{q^2}{{(1-q)}^2}V(g)V(z)}\hfill \end{array}.$$ (9)

Comparing to the theoretical $r_{g,\widehat{g}}^2=\frac{\mathit{Cov}{(g,\widehat{g})}^2}{V(g)V(\widehat{g})}$:

$$\frac{{\tilde{r}}_{g,\tilde{g}}^2}{r_{g,\widehat{g}}^2}=\frac{{(1-q)}^2+2q(1-q)\frac{\mathit{Cov}(g,z)}{\mathit{Cov}(g,\widehat{g})}+q^2{\left(\frac{\mathit{Cov}(g,z)}{\mathit{Cov}(g,\widehat{g})}\right)}^2}{{(1-q)}^2+2q(1-q)\frac{\mathit{Cov}(\widehat{g},z)}{V(\widehat{g})}+q^2\frac{V(z)}{V(\widehat{g})}}.$$ (10)

Since $\widehat{g}$ is based on a few reference haplotypes and equal to g with large probability, we can reasonably assume $\mathit{Cov}(g,z)\approx \mathit{Cov}(\widehat{g},z),\hspace{0.17em}\mathit{Cov}(g,\widehat{g})\approx V(g)$ and $\mathit{Cov}(g,z)<\mathit{Cov}(g,\widehat{g})$ (misidentified haplotypes are more distant). Therefore the middle terms in the numerator and denominator of Equation (10) are approximately equal; the difference between the r²’s is governed by the last terms, which have a ratio strictly smaller than one: $\frac{\mathit{Cov}{(g,z)}^2}{\mathit{Cov}{(g,\widehat{g})}^2}\frac{V(\widehat{g})}{V(z)}=\frac{\mathit{Cor}{(g,z)}^2}{\mathit{Cor}{(g,\widehat{g})}^2}<1$.

If misidentified haplotypes are close to random draws from all reference haplotypes ($\mathit{Cov}(g,z)\approx 0$), the ratio simplifies to:

$$\frac{{\tilde{r}}_{g,\tilde{g}}^2}{r_{g,\widehat{g}}^2}\approx \frac{{(1-q)}^2}{{(1-q)}^2+q^2\frac{V(z)}{V(\widehat{g})}}<1.$$ (11)

In practice, the suboptimal templates contributing to the imputed dosage are more likely to be from lineages closer to the target, so their effect on imputation accuracy would be smaller than that from the above assumptions.

Length of haplotype before the next recombination breakpoint

Next we aim to derive the distance between consecutive recombination events on an external branch. For the purpose of modeling imputation, these distances represent the lengths of segments that are copied from the same haplotype. Let X be the genetic distance to the next observed recombination event on the target, and T be the length of its corresponding external branch. We calculate $f_X(x)=\int f_{X|T}(x)f_T(t)dt$ (we will use $f(·)$ for continuous and $P(·)$ for discrete distributions). Similar to above, we compute the probability density function of T by considering it first coalesces at the ($n+1-k$)th event. $T|k$ is the sum of $n+1-k$ coalescent time intervals, thus following a convolution distribution as the sum of $n+1-k$ exponential with different rates. Conditional on T, X follows an exponential distribution with rate 2T since a recombination event on either the current template or the target haplotype results in a switch of template:

$$\begin{array}{cc}{f}_{X|T}(x)\hfill & =2T{e}^{-2Tx}\hfill \\ f_{T|k}(t)\hfill & =f_{T_{n+1}+\cdots +T_{k+1}}(t)\hfill \\ \Rightarrow f_{X|K}(x|k)\hfill & ={\int }_{t=0}^{\infty }{f}_{T_{n+1}+\cdots +T_{k+1}}(t)f_{X|T}(x|t)dt\hfill \\ \hfill & =E_{T|k}[2T{e}^{-2Tx}]\hspace{1em}\hspace{1em}(T|k\sim f_{T_{n+1}+\cdots +T_{k+1}})\hfill \end{array}.$$ (12)

When population size is constant, $E_{T|k}[2T{e}^{-2Tx}]=\frac{\mathit{dMG}{F}_T(y)}{dy}{|}_{y=-2x}$, where MGF_T is the moment generating function of the hypoexponential distribution $T|k$. When population size changes through time so time intervals are correlated, we can approximate the expected value with $2E[T|k]e^{-2E[T|k]x}$ and compute $E[T|k]$ with Monte Carlo.

The probability for a haplotype to first coalesce at the ($n+1-k$)th event is $P_0(n+1,k+1)\frac{k}{\left(\begin{array}{c}k+1\\ 2\end{array}\right)}$, with $P_0(n+1,k+1)$ being the probability of not coalescing with any lineage in the first (n−k) events (starting from total sample size n + 1). Therefore, we have the distribution of X is:

$$f_X(x)=\sum _{k=1}^n{P}_0(n+1,k+1)\frac{k}{\left(\begin{array}{c}k+1\\ 2\end{array}\right)}{f}_{X|K}(x|k)$$ (13)

$$=\frac{2}{n(n+1)}\sum _{k=1}^{n}k·f_{X|K}(x|k).$$ (14)

Simulation

We performed a standard Kingman coalescent simulation under population history models suggested by recent studies (detailed in ‘Results’). We simulate size-(n + 1) trees with n varying from 500 to 50k. For each generated tree, we keep track for all subtree configurations ($1+u(j)$) (one external branch first coalesces with a branch of size u, which contains a branch of size j) of interests, and record the length of this size j branch. The sum of all such relevant branch lengths in the tree divided by the number of present haplotypes n + 1 is one realization of the joint probability $P(\widehat{g}=j/u,\hspace{0.17em}g=0,\hspace{0.17em}j;n)$, which will then be approximated by averaging such realizations over all simulated trees. To approximate the probability of observing j derived alleles in the reference panel, we simulated another independent set of trees of size n and summed over the lengths of all size j branches to get sample frequency spectrum. We present results from 50k independent simulations under each parameter setting throughout this article.

Empirical evaluation

To evaluate the predictions of our model, we imputed genotypes from the Michigan Genomics Initiative (MGI; Fritsche et al. 2018) using different imputation panels. To assess imputation quality, we masked some genotypes and compared the imputed genotypes to the genotypes generated by the array. MGI is collected from the patient population of the University of Michigan Hospitals and thus mostly ($>90\%$) consists of individuals of European decent. The MGI individuals analyzed here are genotyped on a Illumina Infinium CoreExome chip with $\sim 60,000$ custom markers, providing sufficient low-frequency variants to assess imputation performance for rare variants.

We considered five reference panels: (1) The Haplotype Reference Consortium (HRC) r1.1 reference panel (n = 32,470 individuals of mostly European descent, $\approx 39M$ variants; McCarthy et al. 2016), (2) the 1000 Genomes (1KG) Project variant calls from low-coverage sequencing (n = 2548, $\approx 78M$ variants; Lowy-Gallego et al. 2019), (3) the 1KG variant calls from high-coverage sequencing (n = 2504, $\approx 100M$ variants; Byrska-Bishop et al. 2021), (4 and 5) subsets of the high- and low-coverage 1KG variant calls containing only samples from the European super population (n = 503). Note that the HRC reference panel only provides variants with MAC$\ge$ 5 (McCarthy et al. 2016). We subsetted all 1KG variant reference panels to the set of overlapping individuals and markers and filtered out monomorphic and singleton sites, resulting in 2503 samples and $\approx 41M$ variants for panels (2) and (3) and 503 samples and $\approx 13M$ variants for panels (4) and (5). We also re-phased all 1KG panels using Eagle (v2.4.1; Loh et al. 2016) without a reference panel to avoid confounding due to differing phasing quality.

Using each reference panel, we imputed all autosomal variants by running Minimac4 (v1.0.0; Howie et al. 2012) on a computing cluster maintained by the Center for Statistical Genetics at the University of Michigan in Ann Arbor. We set Minimac4 parameters to use 16 cpus and output data in genotype, estimated alternate allele dosage, and estimated haploid alternate allele dosage formats. For all rare variants genotyped in MGI, we estimated imputation quality using the squared Pearson correlation coefficient between known and imputed genotypes.

Data availability

1000 Genomes low-coverage sequence data were downloaded from ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/data_collections/1000_genomes_project/release/20190312_biallelic_SNV_and_INDEL/. 1000 Genomes high-coverage sequence data were downloaded from ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/data_collections/1000G_2504_high_coverage/.

Results

We derived an analytical approach to calculate the impact of model misspecification of the current imputation framework with our coalescent model on rare variants (see ‘Methods’). Assuming a Li and Stephens (2003) model-based imputation algorithm correctly identifies all and only those haplotypes in the reference that are most closely related to the target sample (“closest templates”), we calculated the theoretical error rate as a function of reference size n and the DAC. Using coalescent simulations, we confirm the analytical results, and incorporate general population models. We compared our theoretical predictions with empirical imputation accuracy observed in the MGI, with the 1000 Genomes (Abecasis et al. 2010) or the Haplotye Reference Consortium (McCarthy et al. 2016) as reference.

The number of closest template haplotypes

In the coalescence context, identifying a single-reference haplotype that is most similar to the target haplotype is equivalent to selecting the reference haplotype that has the MRCA with the target haplotype or, equivalently, whose lineage is the first to coalesce with the target. However, by the time this lineage coalesces with the target, it may be ancestral to multiple reference haplotypes (Figure 1). In this case, those reference haplotypes are equally closely related with the target, each of them in expectation providing the same amount of information for imputing the target sample. Thus, the model assuming one single best template is misspecified, and imputation is more likely to be ambiguous when the number of closest templates is larger.

The probability for the target haplotype to first coalesce with a lineage having u descendants in the reference is:

$$P(u)=\frac{4}{u(u+1)(u+2)},(1\le u\le n-1).$$ (15)

Note that the derivations of this probability do not condition on the presence of a variable site. Thus, this probability depends only on the topology of the coalescent tree, independent of population history (see ‘Methods’). Interestingly, the probability distribution of the number of best matches does not depend on the size of the reference panel (n), except for the extreme case u = n, where the entire reference has equal genetic distance to the target. In that case, $P(u=n)=\frac{2}{n(n+1)}$. The expected number of closest templates $E(u)=2-\frac{2}{n+1}$, approaching 2 as the size of the reference panel increases.

From Equation (15), we see that with probability $\frac{2}{3}$, the target haplotype first coalesces with an external branch (u = 1) and the reference contains exactly one best match. Thus, the Li and Stephens model of exactly one best template is misspecified with probability $\frac{1}{3}$. The probability of u equally good templates drops rapidly with increasing u (Table 2). With probability $\frac{1}{6}$, the target haplotype first coalesces with an internal branch with two descendants in the reference (u = 2), while the probability is 0.0108 that the target haplotype first coalesces with an internal branch with more than 10 descendants in the reference (u > 10).

Table 2.

Probability of multiple reference haplotypes (“No. templates”) being most closely related to the target haplotype

No. templates	1	2	3	4	5	6	7	8	9	10
Probability	0.6667	0.1667	0.0667	0.0333	0.0190	0.0119	0.0079	0.0056	0.0040	0.0030

Impact on imputation accuracy

As we have just demonstrated, the model assumption that each target haplotype has a single most closely related template haplotype is a model misspecification for 1/3 of the genome where there are multiple closest templates. This model misspecification contributes to imputation error beyond the typical source of error created by failing to identify the most closely related haplotype. To isolate this additional error, we now evaluate the imputation of a missing genotype assuming that all closest templates are correctly identified but analyzed under a model that assumes a single best haplotype. In this scenario, imputation algorithms will correctly impute all non-singleton variants if the target haplotype has a single most closely related template haplotype. Imputation errors can only occur if the target haplotype has more than one closest template, i.e. a model misspecification. In this case, imputation algorithms consider all these template haplotypes to be equally likely to be the“best” template. Accordingly, it will interpret identifying multiple equally close templates as uncertainty in identifying the best template. For variants that differ between these template haplotypes, the imputed genotype is then usually the average of the DAC of those templates, a fractional genotype (dosage). Thus, for one given target haplotype carrying the ancestral allele to be imputed with a nonzero DAC dosage (“false positive”), some of its templates have to carry the derived allele.

Although the probability of this misspecification is independent of population history, the probability of a mutation event causing an imputation error depends on the history of the population template and target haplotypes are sampled from. In our primary analysis, we consider a population history model approximating European population history (Terhorst et al. 2017; Speidel et al. 2019). Starting from an ancestral population with effective population size 10⁴, it undergoes a bottleneck with $N_e=2·{10}^3$ for 100k years (∼3450 generations), then grows with an accelerated rate (faster than exponential growth;Reppell et al. 2014) to $N_e={10}^7$ during the most recent 10k years. The effect of these model parameters on the following results is marginal within a model space reasonable for human population (Appendix).

We first consider the special case where all the closest templates for a target carry the derived allele, while the target haplotype carries the ancestral allele. In this case, the imputed dosage of the derived allele is 1 when the truth is 0 (Figure 2A). By integrating over all possible tree shapes and branching times (see ‘Methods’), we calculated the probability for this configuration for reference sample sizes of 500, 5000, 20,000, and 50,000 individuals. We verified all results using computer simulations. Across all considered reference sizes, the probability of this completely wrong imputation for one imputed haplotype is small ranging from ${10}^{-3}$ to $<{10}^{-6}$ (Figure 3A andTable 3). Holding reference size constant, this probability decreases with rising DAC. Similarly, holding DAC constant, this probability of imputation error decreases with increasing number of templates.

Figure 3.

Upper panel: False discovery rate among noncarriers. Given a target haplotype carrying the ancestral allele (true dosage 0), the probability (y-axis, in log scale) of having (a) all the closest templates in the reference panel carrying the derived allele thus an estimated dosage 1; or (b) having nonzero estimated dosage for the derived allele. Lower panel: proportion of false positives among all imputed carriers. For a target sample, the proportion (y-axis) of haplotypes with the ancestral allele among (c) individuals with estimated dosage 1, or (d) haplotypes with nonzero dosage. Both results are conditional on the DAF in the reference (x-axis); color indicates the size of reference panel, in the number of individuals. Results for reference size below 20,000 are from analytical calculation while those for 20,000 and 50,000 are from coalescence simulations.

Table 3.

Expected number of imputation errors

Reference size (individuals)	DAC in the reference panel
	2			3			5			10			20
	Imputed dosage for the derived allele
500	2145	1122	848	1802	1006	653	1361	815	420	840	541	153	435	292	57
5000	214	109	80	179	97	60	136	78	41	84	52	23	49	32	13
20,000	53	27	18	44	24	15	34	19	11	21	13	7	12	7	4
50,000	21	10	7	17	9	6	13	8	4	8	5	3	5	3	2

Consider imputing a sample dataset with one million individuals using a reference panel containing 500–50,000 individuals from the same population as the sample, the expected number of individuals who are homozygous for the ancestral allele (g = 0) but have an imputed genotype dosage $\widehat{g}$ as $\widehat{g}>0,\hspace{0.17em}\widehat{g}>0.5$ or $\widehat{g}=1$. Each combined column represents one DAC in the reference. Results are from 50k independent coalescence simulations, each data point is an average of over 10⁷ loci.

To put these error rates in relationship with the number of true carriers of the derived allele, we assume that the DAF in the target population is the same as that in the reference population. For instance, consider a site with two copies of the derived allele in a reference of 500 individuals (MAF $=0.002$). For this site, we expect to impute one individual as false carrier for every 1210 individuals (Figure 3). Among these 1210 individuals, we also expect 4.84 true carriers of this variant assuming the same MAF in the target population. Thus, for variants that are doubletons in a reference sample of 500 individuals, about one in six imputed rare alleles will be a false positive (Table 3). More generally, for a fixed DAC in the reference, the probability of falsely imputing the derived allele decreases with increasing reference size while the number of true carriers of the derived allele also decreases. As a result, the proportion of falsely imputed derived alleles decreases only moderately with increasing reference size. For example, for variants that are doubletons in the reference, the proportion of false positive-derived alleles decreases only from 17.5% for a reference size of 500 to 14.7% for a reference size of 50,000.

If we now consider the more general case where at least one of the closest templates carry the derived allele while the target carries the ancestral allele (dosage 0). In this case the imputed dosage $\widehat{g}>0$ (Figure 3). Across reference sizes of 500, 5000, 20,000, and 50,000 individuals, we calculated the probability of $\widehat{g}>0$ for each imputed haplotype. This probability is notably larger than the probability of falsely imputing dosage 1, ranging from $\sim {10}^{-3}$ to $\sim {10}^{-5}$ for doubletons (with DAC 2;Figure 3B). This probability decreases with rising DAC and increasing reference size.

From these probabilities, we calculate the expected number of haplotypes carrying the ancestral allele (g = 0) that are falsely imputed to have either a nonzero derived allele dosage ($\widehat{g}>0$), a higher dosage $\widehat{g}>0.5$ or a dosage of 1 for every million target individuals (Table 3). Haplotypes with $\widehat{g}>0.5$ represent cases where a“best-guess” imputation algorithm would infer the alternate allele. Beyond the previously described impact of DAC and reference size, these results show that for DAC < 5, about half of all falsely imputed derived alleles have a dosage > 0.5. This observation can be explained by the fact that for a given DAC, observing a dosage < 0.5 requires a larger number of equally good templates than observing a larger dosage. As large numbers of equally good templates are rare (Table 2), the proportion of higher dosage among all falsely imputed nonzero dosages increases with the DAC.

To summarize the impact of this error on association tests, we calculated the squared correlation coefficient (r²) between the imputed dosages and the true genotypes using simulations, both under the assumption that model misspecification is the only source of error and under the assumption that some reference haplotypes are falsely identified as being most closely related. r² is a commonly used measure for imputation quality, as it is directly related to statistical power in downstream association tests (Guan and Stephens 2008; Huang et al. 2009; Wojcik et al. 2018).

Still assuming that closest templates are identified perfectly, we generated the distribution of expected r² for a range of MACs and reference sizes using simulations (see ‘Methods’). The expected r² increases monotonically with DAF in the reference and reference size (Figure 4). If we consider the increase with DAF across reference sizes, we observe that the curves of expected r² values look very similar for each reference panel size, only shifted by a factor of 1/panel size. In other words, r² of variants with the same MAC remains almost the same across all considered reference sizes. For example, the average r² among doubletons is 0.822 in a reference size of 500 and 0.831 in a reference sample size of 50,000, where its frequency is only 1/100 of the former. For variants observed 10 or more times in the reference, the expected r² is > 0.97 regardless of the size of the reference, as model misspecification do not play a major role for the imputation quality of these variants, but for variants observed less often, r² decreases rapidly with decreasing allele count.

Figure 4.

(A) The upper bound of squared correlation (r²) (y-axis) between imputed and true genotype dosages, conditional on the DAF in the reference (x-axis). Color indicates the size of reference panel. (B) The r² (top) and MSE (bottom) when a certain proportion of weight (x-axis) is attributed to suboptimal templates. The impact depends on the absolute number of suboptimal templates, we show examples of 1, 5 in the left and right columns. Here we fix the reference size to 20k. Results are from 50k independent coalescence simulations, each data point is an average of over 10⁷ loci.

To include other sources of error in the imputation process, we model the false identification of reference haplotypes that are more distantly related to the target as templates. Those distantly related haplotypes are less likely to carry the same allele as the target haplotype and thus introduce an additional source of error. We parameterize this identification error as the sum of all probabilities assigned to falsely identified templates, denoted as q (see ‘Methods’). In practice, this imputation error depends on the choice of imputation algorithm, marker density, quality of genotyping and statistical phasing; detailed modeling of these factors is beyond the scope of this article.

As expected, imputation error of the variant increases with increasing q shown as the mean squared error (MSE) of the imputed genotype (Figure 4B). As the imputed genotype for all individuals converges to the allele frequency f as q goes to 1, the MSE increases faster with higher f. Similarly, the r² between the true genotype and the imputed genotype decreases with increasing q. However, including some suboptimal haplotypes only has a small effect for rare variants. If we assume a single suboptimal template, $q<10\%$ only has marginal effect on r², when q increases beyond this threshold the decrease of r² becomes almost linear. This threshold depends on the number of suboptimal templates: with a larger number of templates, the variance of the imputed genotype decreases and the effect of including suboptimal templates is more deterministic (see ‘Methods’). Accordingly, if the number of suboptimal templates is larger, the threshold where r² starts to decrease with q is larger while the MSE is lower (Figure 4B). For example, consider imputing a variant that is a doubleton in the reference with 20k haplotypes. Assuming no identification error, $r_0^{\mathrm{2}}=0.83$. If we identify one suboptimal templates with posterior probability 0.3 as well as all the closest templates, the expected r² is 0.70 and the MSE is $1.80\times {10}^{-5}$. If we instead identify 5 suboptimal templates with total posterior probability 0.3 as well as all the closest templates, the expected r² increases to 0.80 while the MSE decreases to $1.44\times {10}^{-5}$.

For variants with DAC < 5 in the reference, the error caused by model misspecification dominates for a wide range of q; while for variants of DAC over 20 the error caused by model misspecification is negligible and the $r_0^{\mathrm{2}}$ with perfectly identified optimal templates is above 0.99 regardless of the reference size (Table 4), thus the identification error dominates the empirical imputation error.

Table 4.

The maximal squared correlation can be achieved

Reference size (individuals)	DAC in the reference
	2	3	4	5	10	20
500	0.822 (0.089)	0.885 (0.067)	0.919 (0.051)	0.940 (0.041)	0.979 (0.017)	0.995 (0.0055)
2000	0.823 (0.089)	0.886 (0.066)	0.920 (0.051)	0.940 (0.040)	0.979 (0.016)	0.994 (0.0054)
5000	0.826 (0.088)	0.887 (0.065)	0.921 (0.050)	0.941 (0.040)	0.979 (0.016)	0.994 (0.0055)
20,000	0.828 (0.087)	0.889 (0.065)	0.922 (0.050)	0.942 (0.040)	0.980 (0.016)	0.994 (0.0053)
50,000	0.831 (0.087)	0.891 (0.065)	0.923 (0.050)	0.943 (0.039)	0.980 (0.016)	0.994 (0.0053)

Numbers in the parentheses are SD summarized over 10⁷ loci, representing the variation among variants in a population sample. Results are from 50 k independent coalescence simulations.

Comparison with empirical imputation accuracy

We imputed genotypes of 56,984 participants in MGI using two standard reference panels, the 1KG (2504 individuals) and the HRC (32,470 individuals). For variants present in the MGI dataset, we calculated r² between observed genotypes and imputed genotypes and stratified these by the MAC of the variant in the reference panel (Figure 5).

Figure 5.

Imputation accuracy empirically evaluated in MGI using different reference panels. Y-axis is average r² evaluated at genotyped sites, x-axis is (A) MAC or (B) MAF in the reference panel, with MAF plotted in log scale. Rarest variants are highlighted in (B) for comparison. Color coding in (A) and (B) is the same.

We recapitulate previous results (e.g.Das et al. 2016) that panel size had a notable effect on imputation accuracy. For a given MAC, r² for variants imputed with the largest panel (HRC) is lower than that for variants imputed with the full 1KG panel by $\approx 0.10$, the later is in turn lower than that for variants imputed with the European ancestry individuals from 1KG by $\approx 0.04$ (difference relatively stable across MAC). However, when conditioning on MAF, the order of panels flips, as the same MAC reflects much smaller MAF in larger panels. Here, r² for variants imputed with HRC is higher than that for variants imputed with the full 1KG panel by $\approx 0.05$, the later is in turn higher than that for variants imputed with the European ancestry individuals from 1KG by $\approx 0.10$.

Sequencing depth of the reference panels on the other hand had only a very small effect. The use of deep coverage 1KG panels increased imputation accuracy typically by < 0.01, compared with the low-coverage panel. Note that markers called only in the deep coverage panel are not included in this comparison.

Comparing these empirical result with the theoretical upper bound from our model (Figure 4), we observe that the empirical r² is much lower than what is predicted under the assumption that only model misspecification affects imputation accuracy. For doubletons imputed with the whole 1KG, the average r² is 0.32; for doubletons imputed with the European subset (504 individuals) the average r² is 0.41, while the theoretical upper bound is above 0.82 for both sample sizes. Although empirical imputation accuracy of rare variants decreases with decreasing MAC across all reference panels, it decreases more rapidly for MAC < 10. This more rapid decrease is mirrored by the rapid decrease of the theoretical upper bound for MAC < 10 observed in the theoretical prediction.

Length between template switches

We now consider another potential source of error in current imputation model: the switch between templates. When we model the target haplotype as a mosaic of templates from the reference, switching between templates can be interpreted as a historical recombination event that breaks the genealogical bond between the target and its current template, i.e. the branches connecting the two leaves in a coalescent tree. The two haplotypes will become practically independent beyond the recombination break point.

Conditional on a known local genealogy, the length to the next recombination break point follows an exponential distribution, with the rate proportional to twice the TMRCA between the target and the template. In practice, the genealogy is unknown, and the length to the next recombination break point is a combination of the distribution of the conditional length and the time to the TMRCA (detailed in ‘Methods’). Here we compare this mixture distribution with the exponential distribution that is typically assumed in current imputation methods (Li and Stephens 2003; Browning et al. 2018).

Comparing the mixture distribution to an exponential distribution with the same mean shows that with genealogy and population history aware modeling, the distribution of length between switches is very similar for small reference samples (n = 500; Figure 6). For larger reference samples (n = 20,000) the mixture distribution has larger variance, with higher density in both extremely short and long intervals but lower density for intermediate ones. If the excess of short no-recombination intervals is not well captured due to model misspecification, we expect to have more suboptimal templates thus higher imputation error rate. When the reference size increases, switches are less frequent as it is more likely to find a template sharing a very recent MRCA with the target.

Figure 6.

Distribution of the distance before the next recombination event breaking the relation between a pair of haplotypes. The x-axis shows the length in genetic distance (cM). Solid line is predicted by our coalescent model, the dashed line is an exponential distribution with the same mean as we predicted. Colors represent different reference sizes (n) in the number of individuals. The average lengths with reference being n = 500, 20,000, 50,000 are 0.26, 1.59, and 3.41 cM, respectively.

Discussion

The empirical performance of imputation methods has been extensively studied (Chan et al. 2016; Das et al. 2016; Shi et al. 2018; Taliun et al. 2021), while the theoretical behavior and limit of the underlying framework has not been well characterized, especially for rare variants. We formulate this problem of imputation accuracy into a coalescent model, considering imputing the missing genotypes on one target haplotype by copying from a set of reference haplotypes that share the MRCA with the target. Our approach identifies two model misspecifications in modern imputation algorithms and explores their impact on our ability to impute rare variants.

First, most imputation algorithms model a single best template as the hidden variable to infer. We show that in 1/3 of the cases, multiple haplotypes are equally good templates for imputing one target in any given reference panel, independent of reference sample size or population history. The resulting model misspecification leads to imputation error when genetic variants are shared by some of the optimal templates but not with the target. We develop analytical expressions for the imputation error resulting from this model misspecification as a function of the DAF and reference size.

For this purpose, we assume that the imputation algorithm correctly identifies the most closely related haplotypes so that this model misspecification is the only source of error. In this idealized scenario, we observe that for variants observed five or less times in the reference panel, $>8\%$ of variants with nonzero dosage are noncarriers; up to MAC 10, the mean r² between the imputed dosage and true genotypes < 0.98. Conditional on the DAC, this effect is broadly independent of reference panel size: although the probability of falsely imputing carriers of the rare allele decreases as the reference panel size becomes large, the number of true carriers also decreases and the proportion of false carriers over true carriers stays about the same. Including other sources of error that occur in practice further decreases imputation accuracy. Thus the expected r² we present here designate the upper bound of achievable mean imputation accuracy for rare variants in current imputation framework.

Such imputation error substantially reduces the power for detecting novel risk variants in an association study based on imputed genotypes. Two scenarios can be considered here: First, single-marker tests of imputed rare variants can be powerful if the case-control dataset is much larger than the imputation panel (Taliun et al. 2021). In this scenario, inaccurately inferred genotypes at a disease-related locus attenuate the allele frequency difference between cases and controls, and r² is directly related to this loss of information that compromises the statistical power in association tests. Imputation error rate of 2–6% leads to 10–60%increase in required sample size in a single-marker test (Huang et al. 2009). As a second scenario, imputed rare variants can be aggregated into a single-test statistic (Lee et al. 2014). In this study design, poorly imputed variants will have an attenuated signal, potentially diluting the signal from better imputed variants. This loss of power can be limited by focusing on imputed variants where misspecification will not impede imputation accuracy, e.g. variants that occur more than five times in the reference.

We consider a second model misspecification: the use of exponential distributions to model the length of contiguous haplotypes without template switches. The true distribution of this length is driven both by the recombination rate and the relationship between the template haplotype and the target haplotype. For smaller reference such as the 1000 Genomes (Abecasiset al. 2010; 2504 genomes) this model misspecification has a negligible effect, but for reference samples of the scale of TopMed (Zachary ; 53,831 genomes), gnomAD (Karczewski et al. 2020; 15,708 genomes), or the Haplotye Reference Consortium (McCarthy et al. 2016; 38,821 genomes), the length distribution of shared segments between the target and a single template has a much heavier tails than modeled. This misspecification penalizes both extremely short and extremely long switching intervals, decreasing the probability of finding the optimal templates. This in turn reduces imputation accuracy, especially for low-frequency variants beyond the effect of the first model misspecification described earlier.

The theoretical results we present here are broadly robust to assumptions about population size history in a range reasonable for major human populations. We assume that the imputed region is evolutionary neutral; for loci under selection the impact of the described model misspecifications would likely depend both on the selection model and on the population history. Further, we assume that the reference is from the same homogeneous population as the target sample. If we instead modeled a diverse reference sample, results would depend on the frequency distribution of the imputed variant among the reference samples. For rare alleles, which are typically private to a single population, only members of that single population among the reference would affect imputation accuracy. Our results can be extended to including migration, admixture, or selection, as the mathematical derivation and simulation scheme in this work are general for coalescence at a single locus.

In our empirical evaluation of imputation quality using two commonly used reference panel, the accuracy of rare variants is well below theoretical upper bound, suggesting that we have not reached the limit of the Li and Stephen’s framework. Conditional on MAC, imputation becomes less accurate with increasing reference size, suggesting that failure to identify the genealogically best templates is likely the biggest source of error. Identifying these best templates may not always be possible with available marker data, whose resolution is limited by the observed polymorphic sites and local variation of recombination rate. Other factors that are likely to reduce imputation accuracy include error in statistical phasing in the reference or the target haplotypes and departures from the infinite sites assumption (parallel mutations or back mutations). However, as our model predicts, for all reference panels, empirical imputation accuracy decreases much faster for MAC < 10, suggesting that for these low counts the model misspecification modeled here further reduces the ability to correctly impute rare variants.

Our results suggest that potential improvement of the imputation framework may lie in more detailed modeling of the underlying genealogy, especially for extremely rare variants where only a small subset of the reference contributes information about the imputed genotype. For example, suppose the ancestral recombination graph (ARG) including both the target and the reference haplotypes is constructed, the scenarios resulting in fractional dosage could be avoided (up to uncertainty in ARG). However, alleles absent from the reference would still be missed, and there would be uncertainty when all and only the closest reference haplotypes carry an allele: we can only probabilistically decide whether the mutation event or the coalescence event with the target haplotype happens first.

Overall, we identify that the model misspecifications in imputation algorithms limit our ability for imputing rare variants. Such inherent errors reduce the power of single-variant tests as well as aggregation tests in studies that impute genotypes in large cohorts, especially if they focus on alleles that are observed only a few times in the reference panel. Beyond improving our understanding of the performance of imputation algorithms, these results point to potential new imputation strategies that help identify new risk variants.

Software availability

Codes used to perform analytical calculation and coalescent simulation are available from https://github.com/Yichen-Si/ImputationBound.

Appendix

Probability distribution of fractional dosages

In the ‘Methods’ section, we derived the probability of having even the best possible evidence suggesting a wrong allele type. Here we will generalize it to the cases where the most closely related templates have different allele types, resulting in a fractional estimated dosage (Figure 2D).

Let $P(\widehat{g}=j/u,g=0|j;n)$ be the probability of having u equally good optimal templates, j of which carrying the mutation, for a target actually carrying the ancestral allele; conditional on observing the DAC as j. Let $(k\prime ,d\prime )$ denote the numbers of ancestral lines within the size u subtree when the branch carrying the mutation starts and ends; $(k″,d″)$ denote the numbers of lines in the whole tree at the corresponding time point.

We will outline the calculation of the joint probability (the numerator), while the denominator is the same unfolded frequency spectrum as before. We proceed by the following steps, each subsequent probability is conditional on the previous one, in parallel with that for the special case in ‘Methods’.

1. The target haplotype does not coalesce in the first $(n-d)$ events: $P_0(n+1,d+1)$.

2. A branch of size u arises in the reference at the $(n-k)$th event, then remains alone till the $(n-d)$th event to coalesce with that external branch: $P(u,k,d;n)P_0(k,d+1)\frac{1}{\left(\begin{array}{c}d+1\\ 2\end{array}\right)}$.

3. The subtree of size u contains a branch of size j: $P(j|u;n)=\sum _{(k\prime ,d\prime )}P(j,k\prime ,d\prime |u;n)$ (this step only concerns the topology within the subtree).

4. The size-j branch encounters a mutation: $P(m\ge 1|j,k\prime ,d\prime ,u,k,n)=\sum _{(k″,d″)}P(m\ge 1|k″,d″;n+1)P(k″,d″|k\prime ,d\prime ,u,k,n)$(here, we need to put the subtree back to the whole size-(n + 1) tree to get the branch length and introduce mutation).

Steps (1) and (2) are similar to the three steps in the previous section except for not involving mutation events, we combine them to $P(\mathit{ext},u,k,d;n)$: an external branch first coalesces at the $(n-d)$th event with an internal branch of size u which starts at the $(n-k)$th event. In step (3), $P(j,k\prime ,d\prime |u;n)$ is similar to $P(u,k,d;n)$ in step (2), as $P(j,k\prime ,d\prime |u;n)=P(j,k\prime ,d\prime ;u)$: the probability for a branch of size j to start at the $(u-k\prime )$th event and end at the $(u-d\prime )$th event.

$$\begin{array}{cc}\hfill & P(\widehat{g}=j/u,\hspace{0.17em}g=0,\hspace{0.17em}j;n)\hfill \\ =\hfill & {\displaystyle \sum _{(k,d)}}P(\mathit{ext},u,k,d;n)P(m\ge 1,j|u,k,d;n)\hfill \\ =\hfill & {\displaystyle \sum _{(k,d)}}P(\mathit{ext},u,k,d;n)\left\{{\displaystyle \sum _{(k\prime ,d\prime )}}\left[P(j,k\prime ,d\prime |u,k;n){\displaystyle \sum _{(k″,d″)}}P(m\ge 1|k″,d″;n+1)P(k″,d″|k\prime ,d\prime ,u,k,n)\right]\right\}\hfill \\ \hfill & (\sum _{(k,d)}=\sum _{k=2}^{n-u+1}\hspace{0.17em}\sum _{d=1}^k;\hspace{1em}\sum _{(k\prime ,d\prime )}=\sum _{k\prime =2}^{u-j+1}\hspace{0.17em}\sum _{d\prime =1}^{k\prime -1};\hspace{1em}\sum _{(k″,d″)}=\sum _{k″=k\prime +1}^{n-u+k\prime }\hspace{0.17em}\sum _{d″=d\prime +k-1}^{k″-1}.)\hfill \end{array}$$ (16)

The only component left is $P(k″,d″|k\prime ,d\prime ,u,k,n)$, the link between the topology within the subtree and the branch lengths, which is relative to the whole sample and involves the population size history model. Since coalescent time intervals are measured for the whole tree, $(k″,d″)$ tells us when the branch of size j starts and ends, which leads to the probability for a mutation to occur.

We divide the coalescent process among the whole reference from n lines to k lines into three parts: $n\to k″\to d″\to k$. There are $(n-k″-1)+(k″-d″-1)+(d″-k-1)$ unfixed events (since we are conditioning on the three time points, $(k\prime ,k″),\hspace{0.17em}(d\prime ,d″)$ and (k)). Within each part, we consider the number of events that has to happen inside the subtree: $(u-k\prime -1),(k\prime -d\prime -1),(d\prime -1-1)$. With the number of lines fixed, $P(k″,d″|k\prime ,d\prime ,u,k,n)$ is calculated by considering the possible ways to arrange those events in the three time intervals:

$$\begin{array}{c}P(k″,d″|k\prime ,d\prime ,u,k,n)=\frac{\left(\begin{array}{c}n-k″-1\\ u-k\prime -1\end{array}\right)\left(\begin{array}{c}k″-d″-1\\ k\prime -d\prime -1\end{array}\right)\left(\begin{array}{c}d″-k-1\\ d\prime -2\end{array}\right)}{\left(\begin{array}{c}n-k-1\\ u-2\end{array}\right)}\hfill \end{array}.$$ (17)

The denominator in Equation (10) comes from removing the constrains introduced by the mutation event. Conditional on (u, k), there are in total $(n-k-1)$ unfixed events in the whole tree, including $(u-2)$ in the subtree. Since $(k″,d″)$ defines the coalescent time in the whole size-(n + 1) tree, we can calculate the length of the branch carrying mutation by Equation (6): $P(m\ge 1|k″,d″;n+1)=\mu \sum _{i=d″+2}^{k″+1}E[T_i;n+1]$.

Comparison of population growth models (Figures A1 and A2)

Genotyping and sample quality control

DNA samples from the blood of MGI participants were processed on one of two production batches of a customized Illumina Infinium CoreExome-24 bead array. Genotype calls were produced with the Illumina GenomeStudio 2.0 software operating the Genotyping Module v2.0.4 and the Gentrain clustering algorithm v3.0. Variant- and sample-level quality control (QC) is detailed as following.

Figure A1.

Effect of population growth model on false discovery rate among noncarriers, from analytical calculation. Given a target haplotype carrying the ancestral allele, the probability (y-axis, in log scale) of having all the most closely related reference haplotype in the reference panel carrying the derived allele thus an estimated dosage 1, on the DAF in the reference (x-axis). Each sub-figure represents one reference size (in individual); each color represents a population growth model. FTE(EXP)_10M_2K: faster-than-exponential (exponential) growth with current day effective population size 10 million, and a bottleneck with effective population size 2 thousands.

Figure A2.

Effect of population growth model on imputation accuracy upper bound (r²), from coalescent simulation. Y-axis is the average r² between imputed dosages and true genotypes without identification error; error bars represent 1 SD. Each sub-figure represents one reference size (in individual); each color represents a population growth model. Please provide expansion for FTE in first occurrence if necessary.]FTE(EXP)_10M_2K: faster-than-exponential (exponential) growth with current day effective population size 10 million, and a bottleneck with effective population size 2 thousands.

Bead array probe mapping and genotype calling were performed according to the Genome Reference Consortium Human Build 37 (GRCh37), 502,255 variants passed variant-level QC. Variants were excluded if (1) bead array probe sequences did not perfectly and uniquely map to the reference (BLAT v.351; Kent 2002), (2) variant-level call-rate was below 99%, (3) variant fail Hardy–Weinberg equilibrium exact test with P < 10⁻⁶ in unrelated European samples (PLINK v1.90; Purcell et al. 2007), (4) GenomeStudio GenTrain score < 0.15 or Cluster Separation score < 0.3, or (5) allele frequency differed between bead array production batches (P < 10⁻³), Fisher’s exact test). The genetic ancestry of MGI participants was inferred by projecting MGI samples onto the space created by the first two principal components (PCs) of 938 unrelated samples of the Human Genome Diversity Project (HGDP) reference panel (Li et al. 2008; PLINK). MGI samples were inferred to belong to a HGDP reference population if they fell within a circle drawn around that population in a plot of the PCs. We also lift it over to GRCh38, where 501,607 variants remained.

Only consented individuals were included, and 56,984 samples passed sample-level QC. Samples were excluded in QC if (1) genotype-inferred sex is abnormal or did not match the self-reported gender of the participant or the self-reported gender was missing, (2) sample shared a kinship coefficient > 0.45 with another sample with a different identification tag (KING v2.1.3; Manichaikul et al. 2010), (3) sample-level call-rate was below 99% or any chromosome had a call-rate below 1/5 of the average, or (4) estimated contamination level exceeded 2.5% (VICES; Zajac et al. 2019).