Rare-variant association studies: When are aggregation tests more powerful than single-variant tests?

Summary

Because single-variant tests are not as powerful for identifying associations with rare variants as for common variants, aggregation tests pooling information from multiple rare variants within genes or other genomic regions were developed. While single-variant tests generally have yielded more associations, recent large-scale biobank studies have uncovered numerous significant findings through aggregation tests. We investigate the range of genetic models for which aggregation tests are expected to be more powerful than single-variant tests for rare-variant association studies. We consider a normally distributed trait following an additive genetic model with $c$ causal out of $v$ total rare variants in an autosomal gene/region with region heritability $h^2$, measured in $n$ independent study participants. Analytic calculations assuming independent variants, for which we developed a user-friendly online tool, show that power depends on $n{h}^2,c$, and $v$. These analytic calculations and simulations based on 378,215 unrelated UK Biobank participants revealed that aggregation tests are more powerful than single-variant tests only when a substantial proportion of variants are causal and that power is strongly dependent on the underlying genetic model and set of rare variants aggregated. For example, if we aggregate all rare protein-truncating variants (PTVs) and deleterious missense variants, aggregation tests are more powerful than single-variant tests for >55% of genes when PTVs, deleterious missense variants, and other missense variants have 80%, 50%, and 1% probabilities of being causal, with $n=100,000$ and $h^2=0.1\%$. With continued use of single-variant and aggregation tests in rapidly growing studies, our investigation sheds light on the situations favoring each test.

Aggregation tests were developed to increase power to detect associations with rare variants. Bose et al. investigate a range of genetic models and sample sizes and demonstrate how sample size × heritability, proportion of causal variants, and choice of mask determine when aggregation tests are more powerful than single-variant tests.

Introduction

Genome-wide association studies (GWASs) have identified hundreds of thousands of associations with thousands of human diseases and traits (https://www.ebi.ac.uk/gwas/). Most of these associations have been identified using single-variant tests on common genetic variants (minor allele frequency [MAF] > 1%).

To increase power to detect associations for rare variants (MAF < 1%), aggregation tests (reviewed in Lee et al.,¹ Nicolae,² and Goswami et al.³) pool association evidence across multiple rare variants in a gene or other genomic region. The simplest of these, the burden test, calculates a (weighted) sum of the minor allele counts (MACs) of the rare variants in the gene or region for each study participant and then regresses the trait on this sum. Other classes of aggregation tests include general and adaptive burden tests,^{4,5,6,7,8,9,10,11,12,13} variance-component tests,^14,15 combinations of burden and variance-component tests,^16,17,18 GWAS p-value combination tests,^19,20,21 and tests^22,23,24 based on the concept of higher criticism.²⁵

Aggregation tests require the selection of a mask to specify which rare variants in the gene or region to include, with the aim to include causal variants and exclude neutral ones. Masks typically focus on likely high-impact variants, such as protein-truncating variants (PTVs) and/or putatively deleterious missense variants.

Despite the goal of aggregation tests to increase power to identify rare-variant associations, in most studies to date that have employed both single-variant and aggregation tests, single-variant tests have yielded many more associations, even when attention has been restricted to rare variants.^{26,27,28,29,30,31} Previous studies have compared the performances of a wide range of aggregation tests in different settings,^{30,32,33,34,35,36,37} with a few also comparing results to those for single-variant tests.^38,39,40 For example, Konigorski et al.³⁹ found that effect sizes, sample size, and proportion of causal variants were key factors in determining whether aggregation tests performed better than single-variant tests. Others noted that aggregation tests were sensitive to deviations from their assumptions of the relationship between the trait and the rare variants in terms of distributions and directions of effect sizes and proportion of causal variants³⁴ and that they would perform better with more extensive functional annotation.³⁸ In fact, a more recent study has shown that with hundreds of thousands of UK Biobank exomes, aggregation tests revealed thousands of associations, many of which were undetected by single-variant tests.⁴¹

With rapidly growing biobanks over the past decade, investigating the relative performance of these tests on large sample sizes is important. This prompted us to compare the powers of single-variant and aggregation tests over a range of genetic models, sample sizes, and masks to identify the circumstances for which aggregation tests are more powerful than single-variant tests for rare-variant association studies. To address this question, we first calculate analytically the non-centrality parameters (NCPs) of the single-variant, burden test, and sequence kernel association test (SKAT) statistics under the assumption of independent variants. Since these NCP formulae are difficult to interpret, we then restrict attention to variants with equal MAFs and effect sizes. We then consider the more realistic situation of dependent variants with unequal MAFs and effect sizes by using computer simulations based on data from the UK Biobank⁴² to compare power for single-variant, burden, SKAT, and SKAT-O tests. For simplicity, we focused on genes in the exome; we expect our comparisons between tests to be valid for other regions of the genome as well.

Methods

Model and notation

Assume $n$ independent study participants genotyped for $v$ bi-allelic rare variants in an autosomal gene or other genomic region (henceforth gene), $0\le c\le v$ of which are causal and the remainder neutral, with $C\subseteq V$ being the corresponding variant sets. Let $p_j$ and ${\beta }_j$ denote the MAF and effect size of variant $j$, and let $G_{ij}$ = 0, 1, 2 denote the MAC or the number of rare alleles in participant $i$ at variant $j$. We consider a quantitative trait $Y$ that follows the additive linear model $Y_i=$ $\alpha +{\sum }_{j\in C}{\beta }_j{G}_{ij}+{ϵ}_i,i=1,2,\dots ,n$. The ${ϵ}_i$ are independent normal variables with mean 0 and variance $1-h^2$, i.e., $N\left(0,1-h^2\right)$, where $h^2=2{\sum }_{j\in C}{\beta }_j^2{p}_j\left(1-p_j\right)={\sum }_{j\in C}{h}_j^2$ is the heritability due to rare variants in the gene and $h_j^2$ is the heritability due to variant $j$.

For aggregation tests, one chooses which rare variants to include in the mask used to perform the test. Masks typically combine a MAF threshold (e.g., MAF < 1%) and annotation requirement (e.g., protein-truncating or missense variants annotated as deleterious by methods such as those of Kumar et al.,⁴³ Chun and Fay,⁴⁴ Adzhubei et al.,⁴⁵ and Schwarz et al.⁴⁶). Let $M\subseteq V$ be the set of $m$ variants in the gene included in the mask with aggregation test weights $w_j>0,j\in M$. $M=C$ is the (unknown) “perfect mask” comprising all causal and no neutral variants.

Test statistics and significance thresholds

Because single-variant and aggregation tests have different units of investigation (variant versus gene), we choose gene as our region of interest to conduct a fair comparison when testing the null hypothesis of no causal variants in the gene against the alternative that at least one variant is causal. We define the score statistic for variant $j$ as $S_j=\sum _{i=1}^n{G}_{ij}\left(Y_i-\overline{Y}\right)=\sum _{i=1}^n{Y}_i\left(G_{ij}-{\overline{G}}_j\right)$, where ${\overline{G}}_j=\frac{1}{n}\sum _{i=1}^n{G}_{ij}$ is twice the MAF for variant $j$, and $\overline{Y}=\frac{1}{n}\sum _{i=1}^n{Y}_i$ is the trait sample mean. We compare the following test statistics:

• (1)most significant single-variant test, $T=\underset{j\in V}{\max }{T}_j=\underset{j\in V}{\max }\frac{S_j^2}{Var\left(S_j\right)}$, where $Var\left(S_j\right)$ denotes the variance of $S_j$;
• (2)burden test, $B=\frac{{\left(\sum _{j\in M}{w}_j{S}_j\right)}^2}{\sum _{j\in M}{w}_j^{2}Var\left(S_j\right)}$;
• (3)SKAT, $Q=\sum _{j\in M}{w}_j{S}_j^2$; and
• (4)SKAT-O, $Q_{\rho }=\rho {\left(\sum _{j\in M}{w}_j{S}_j\right)}^2+\left(1-\rho \right){\sum _{j\in M}w}_j{S}_j^2$, where $0<\rho <1$.

For single-variant tests, we consider both the standard genome-wide significance threshold of $5\times {10}^{-8}$ based on Bonferroni correction for 1 million independent common variant tests,^47,48,49,50 and the more stringent threshold of $5\times {10}^{-9}$ to account for the larger number of tests when including rare variants.⁵¹ For aggregation tests, we apply the standard significance threshold of $2.5\times {10}^{-6}$ based on Bonferroni correction for 20,000 independent genes, and a more stringent threshold of $1.12\times {10}^{-6}$ that accounts for the fact that we often consider multiple masks that result in overlapping sets of dependent tests (see “simulations”).

Analytic calculations: Power to detect association given independent variants

First, we analytically calculate the power of burden, SKAT, and single-variant tests assuming that all $v$ genetic variants are independent. Because a closed-form expression of the distribution of the SKAT-O statistic is not available and p values are computed using numerical integration, we did not attempt analytic calculations for SKAT-O.

Distribution of single-variant score statistics

The trait values $Y_i$ are independent and identically distributed as $N\left(\alpha +{\sum }_{j\in C}{\beta }_j{G}_{ij},1-h^2\right)$, and the score statistics $S_j=\sum _{i=1}^n{Y}_i\left(G_{ij}-{\overline{G}}_j\right)$ are linear functions of the trait values. Assuming the variants are independent, each single-variant score statistic $S_j\sim N\left({\mu }_j,{\sigma }_j^2\right)$, a normal distribution with mean

$$E\left(S_j\right)={\mu }_j=\{\begin{array}{c}{\beta }_j\sum _{i=1}^n{\left(G_{ij}-{\overline{G}}_j\right)}^2={\beta }_j{V}_j,ifj\in C\\ 0,ifj\notin C\end{array}$$

and variance

$$Var\left(S_j\right)={\sigma }_j^2=\sum _{i=1}^n{\left(G_{ij}-{\overline{G}}_j\right)}^{2}Var\left(Y_i\right)=\left(1-h^2\right)V_j,\text{where}{V}_j=\sum _{i=1}^n{\left(G_{ij}-{\overline{G}}_j\right)}^2$$

Replacing $G_{ij}$ and $G_{ij}^2$ by their expected values yields the approximations $V_j\approx V_j^E=2n{p}_j\left(1-p_j\right)$ and, hence, ${\mu }_j\approx 2n{\beta }_j{p}_j\left(1-p_j\right)$ and ${\sigma }_j^2\approx 2n\left(1-h^2\right)p_j\left(1-p_j\right)$.

Power of single-variant test

The single-variant test statistic $T_j$ follows a non-central ${\chi }_1^2$ distribution with non-centrality parameter $NC{P}_{SV;j}=\frac{E^2\left(S_j\right)}{Var\left(S_j\right)}=\frac{n{h}_j^2}{1-h^2}$, if $j\in C$. Under the usual single-variant null hypothesis that the variant is neutral, $T_j$ follows a central ${\chi }_1^2$ distribution. The power of the single-variant test is the probability that at least one of these $v$ tests is significant at level ${\alpha }_{SV}$ (e.g., ${\alpha }_{SV}=5\times {10}^{-8})$. Thus,

$$Powe{r}_{SV}=1-\left[\prod _{j\in C}{F}_j\left({\chi }_{1;{\alpha }_{SV}}^2\right)\right]{\left[F_0\left({\chi }_{1;{\alpha }_{SV}}^2\right)\right]}^{v-c}$$ (Equation 1)

where $F_j$ is the cumulative distribution function (cdf) of a ${\chi }_1^2$ distribution with NCP $={NCP}_{SV;j},F_0$ is the cdf of a central ${\chi }_1^2$ distribution, and ${\chi }_{1;{\alpha }_{SV}}^2$ is the ${\left\{1-{\alpha }_{SV}\right\}}^{\text{th}}$ quantile of the central ${\chi }_1^2$ distribution.

Power of burden test

The square root of the numerator of burden test statistic $B$ (i.e., ${\sum }_{j\in M}{w}_j{S}_j$) is distributed as $N\left({\sum }_{j\in M}{w}_j{\mu }_j,{\sum }_{j\in M}{w}_j^2{\sigma }_j^2\right)\text{.}$ $B$ follows a non-central ${\chi }_1^2$ distribution with non-centrality parameter $NC{P}_{Burden}=\frac{{2n\left({\sum }_{j\in M}{w}_j{\beta }_j{p}_j\left(1-p_j\right)\right)}^2}{\left(1-h^2\right)\left({\sum }_{j\in M}{w}_j^2{p}_j\left(1-p_j\right)\right)}$. Under the null hypothesis of no causal variants, $B$ follows a central ${\chi }_1^2$ distribution. The power of the burden test at level ${\alpha }_A$ (e.g., ${\alpha }_A=2.5\times {10}^{-6}$) is

$$Powe{r}_{Burden}=P\left(T>{\chi }_{1;{\alpha }_A}^2\right)=1-F_{Burden}\left({\chi }_{1;{\alpha }_A}^2\right),$$ (Equation 2)

where $F_{Burden}$ is the cdf of a ${\chi }_1^2$ distribution with NCP $=NC{P}_{Burden}$. Since ${\beta }_j=0$ for $j\notin C$, including neutral variants in the mask does not change the numerator of $NC{P}_{Burden}$ but increases the denominator and so reduces power. Excluding causal variants from the mask will typically reduce power, especially if the variants being excluded have moderate to large effect sizes and affect the trait in the same direction as the majority of the causal variants (but see discussion).

Power of SKAT

The SKAT statistic $Q={\sum }_{j\in M}{w}_j{S}_j^2={\sum }_{j\in M}{w}_j{\sigma }_j^2{T}_j$.¹⁵ Since $T_j$ follows a non-central ${\chi }_1^2$ distribution with non-centrality parameter $NC{P}_{SV;j}$, Q is distributed as

$$\sum _{j\in M}{w}_j{\sigma }_j^2{\chi }_1^2\left(NC{P}_{SV;j}\right)=\sum _{j\in C\cap M}{w}_j{\sigma }_j^2{\chi }_1^2\left(\frac{n{h}_j^2}{1-h^2}\right)+\sum _{j\in C^c\cap M}{w}_j{\sigma }_j^2{\chi }_1^2$$ (Equation 3)

and is under the null hypothesis of no causal variants, as ${\sum }_{j\in M}{w}_j{\sigma }_j^2{\chi }_1^2$. The power of SKAT at level ${\alpha }_A$ cannot be obtained exactly but can be approximated using Davies’ method.^15,52

Derkach et al.⁵³ previously studied the NCPs of linear (e.g., burden) and quadratic (e.g., SKAT) aggregation test statistics to assess the effects of weights and genetic model parameters on power to detect association. Our NCP formulae generalize those of Derkach et al.⁵³ by allowing for all three genotypes at a bi-allelic site rather than ignoring the rare homozygote.

Power to detect association given independent variants of equal MAFs and effect sizes

While these close-formed power formulae are interesting, it is not obvious from them how each model parameter influences the power of the aggregation tests. To obtain more interpretable expressions, we now further assume all $v$ variants have the same MAF, all $c$ causal variants have the same effect size and effect direction, and all $m$ variants in the mask have the same weight. With these assumptions, the NCPs for the single-variant and burden test statistics simplify to

$$NC{P}_{SV;j}=\{\begin{array}{c}\frac{n{h}^2}{c\left(1-h^2\right)},\text{if}j\in C\\ 0,\text{if}j\notin C\end{array}$$

$$NC{P}_{Burden}=\frac{n{h}^2{c}_m^2}{\left(1-h^2\right)mc}\text{,}$$

where 0 ≤ $c_m$ ≤ c is the number of causal variants in the mask. Under these same assumptions, the SKAT statistic $Q$ is distributed as ${\sum }_{j\in C\cap M}{\sigma }^2{\chi }_1^2\left(\frac{n{h}^2}{c\left(1-h^2\right)}\right)+{\sum }_{j\in C^c\cap M}{\sigma }^2{\chi }_1^2$, which is equal in distribution to ${\sigma }^2{\chi }_m^2\left(\frac{n{h}^2{c}_m}{c\left(1-h^2\right)}\right)$, where ${\sigma }^2=2n\left(1-h^2\right)p\left(1-p\right)$.

Thus, for fixed region heritability $h^2$ and fixed sample size $n$, the power of the single-variant test decreases as the number of causal variants in the gene increases, since then each causal variant has smaller effect size and explains a smaller proportion of heritability. Power for the burden and SKAT tests increases as the proportion of causal variants included in the mask increases.

If we further assume all variants in the gene are in the mask ($V=M$), the burden test statistic B is then distributed as ${\chi }_1^2\left(\frac{n{h}^{2}c}{\left(1-h^2\right)v}\right)$ and the SKAT statistic $Q$ as ${\sigma }^2{\chi }_v^2\left(\frac{n{h}^2}{1-h^2}\right)$. Under these assumptions, the power of SKAT is independent of the number of causal variants. Because SKAT tests for the presence of heritability, the power of SKAT changes with a change in heritability, so fixing the heritability and allowing different numbers of causal variants to explain equal proportions of this heritability does not have any additional effect on the power. Further, in the usual case when the heritability for the single gene $h^2$ is near zero, $1-h^2\approx 1$, and the NCPs and, thus, the powers for all three tests are (essentially) functions of $n{h}^2,c$, and $v$. Fixing $n{h}^2=20,50,100$ and $v=10,50,100$, we plotted power as a function of the proportion of causal variants ($c/v$) for each of the three tests in this simple case.

Because NCPs and, hence, power for all tests are computable without these further assumptions, we also fixed $n=50,000$ and $h^2=0.1\%$ and chose MAFs of variants from two genes that we use in our simulations: DNAJC5G, whose 49 rare variants have a wide range of MAFs ($1.32\times {10}^{-6}$ to $7.62\times {10}^{-3}$), and LINC01305, whose 20 variants have very similar MAFs ($2.64\times {10}^{-6}$ to $3.17\times {10}^{-5}$). We considered two inverse MAF-effect size models to assign larger effects to rarer variants:

• (1)effect size proportional to $-{\log }_{10}MAF$^15,16; and
• (2)effect sizes $\sim \{\begin{array}{c}N\left(\mu ,{\sigma }^2\right),\text{if}{10}^{-3}<MAF<{10}^{-2}\\ N\left(\mu +1,{\sigma }^2\right),\text{if}{10}^{-4}<MAF<{10}^{-3}\\ N\left(\mu +2,{\sigma }^2\right),\text{if}{10}^{-5}<MAF<{10}^{-4}\\ N\left(\mu +3,{\sigma }^2\right),\text{if}MAF<{10}^{-5}\end{array}$.

• We took $\left(\mu ,\sigma \right)=\left(1,0.25\right)$ to allow effect sizes from rarer variant bins to be larger with very small (2.3%) distributional overlap between adjacent bins, and $\left(\mu ,\sigma \right)=\left(2,0.5\right)$ to allow moderate (15.9%) distributional overlap between adjacent bins.

We then plotted power as a function of the proportion of causal variants by randomly adding one variant in the gene at a time to the causal set. For both models, effect sizes are scaled so that the total heritability was $h^2$. We included all variants in the gene in the mask and assumed Beta(1,25) weights^15,16 for performing burden and SKAT tests. Exact computation of power for single-variant and burden tests is possible using Equations 1 and 2. Since the SKAT statistic is a mixture of scaled chi-square distributions (Equation 3), we calculated statistics based on $2\times {10}^6$ random draws from the component chi-square distributions to approximate the mixture distribution.

UK Biobank data

To conduct a more realistic analysis, we performed simulations based on data from 166,891 White British ancestry participants in the UK Biobank, where whole-exome sequence powered imputation was used to increase the sample size from 166,891 to 408,511.⁴² Informed consent was obtained by the UK Biobank for all participants. We used KING⁵⁴ to exclude participants of second-degree or closer relationships, resulting in an analysis set of 378,215 nearly unrelated participants.

Variants and aggregation test masks

Our UK Biobank exome data on the 378,215 participants included 6,877,794 bi-allelic variants, 6,700,700 (97.4%) of which were rare (MAF < 1%). We used the Ensembl variant effect predictor (VEP)⁵⁵ to annotate all variants, and we used SIFT,⁴³ LRT,⁴⁴ PolyPhen2 HumDiv, PolyPhen2 HumVar,⁴⁵ and MutationTaster⁴⁶ to predict deleteriousness of missense variants (Table S1).

To perform aggregation tests for each gene, we investigated three nested rare-variant masks²⁷ within each gene (Table S2) and included for analysis genes with at least one rare variant in the corresponding mask.

• (1)protein-truncating variants (PTVs): frameshift, splice acceptor, splice donor, stop lost, stop gained, or start lost (128,324 variants in 15,734 genes)
• (2)PTV+Missense(Deleterious): PTVs and the subset of missense variants predicted deleterious by all five algorithms (451,213 variants in 16,927 genes)
• (3)PTV+Missense(All): PTVs and all missense variants (1,688,697 variants in 17,245 genes)

The PTV mask typically included only a modest number of variants per gene: 54% of the genes had ≤6 qualifying variants, and PTVs were generally much rarer than other missense variants. In contrast, the PTV+Missense(All) mask contained many variants per gene: 52% of the genes had >70 qualifying variants, and the variants were generally less rare. The PTV+Missense(Deleterious) mask was intermediate (Table S3).

Simulations

For each gene, we assigned variants’ probabilities of being causal depending on their functional annotation (PTV, deleterious missense, or other missense). Missense variants that were predicted to be deleterious by all five algorithms listed above we classified as “deleterious missense” while the remainder we classified as “other missense.” We assumed PTVs to be more likely causal than deleterious missense variants, which in turn we assumed to be more likely causal than other missense variants; we chose a range of plausible values for these probabilities. There is evidence of an inverse relationship between MAF and effect size^56,57; to model this relationship, we assigned causal variant effect sizes proportional to $-{\log }_{10}MAF$ as suggested by Wu, Lee, and colleagues.^15,16 For most simulations, we assumed that all causal variants affected the trait in the same direction. We used the observed genotypes $G=\left\{0,1,2\right\}$ for the $v$ variants in the gene of $n$ participants randomly chosen from the 378,215 UK Biobank participants and simulated the trait ($Y$) for each chosen participant as ${\sum }_{j\in C}{G}_j{\beta }_j+\sqrt{1-h^2}X$, where $X$ is a standard normal random number. We performed single-variant tests for all $v$ variants in the gene. We then created the three masks for the aggregation tests based on the observed variant MAF and annotations. We performed burden, SKAT, and SKAT-O tests with each mask using the R library, SKAT, with Beta(1,25) weights^15,16 so that rarer variants have larger weights in the analyses. We repeated a subset of the simulations (1) analyzing the data using equal variant weights in the aggregation tests, (2) simulating the data with 50% of the causal variants with a positive direction of effect and 50% negative direction of effect, and (3) simulating the data under the step function-effect size model described in the analytic calculations subsection with $\left(\mu ,\sigma \right)=\left(1,0.25\right)$ and $\left(\mu ,\sigma \right)=\left(2,0.5\right)$.

For each simulation setting, we estimated the power of each test to detect an association of the trait with the gene of interest as the proportion of 1,000 simulation replicates with significant p values using the significance thresholds listed in analytic calculations (see above). The more stringent threshold of $1.12\times {10}^{-6}$ for aggregation tests is based on Bonferroni correction for 44,668 unique tests being conducted for the combination of the three masks described above for the UK Biobank data we employed. We summarize our simulations in Figure S1, and Table 1 lists the genetic models we investigated. We chose $n=100,000$ and $h^2=0.1\%$ as the basis for most of the simulations. We initially investigated a dense grid of causal probabilities between 0 and 1, assuming PTVs were more likely to be causal than deleterious missense variants, which in turn were more likely to be causal than other missense variants. We included a subset of these models to illustrate situations in which aggregation tests were more powerful for >50% of genes, even if causal probabilities are high for some of these situations (Table 1, first three sections). We repeated a subset of the simulations with $n=30,000,100,000$, or $300,000$ and $h^2=0.1\%$ or $0.01\%$ to investigate the impact of changing sample size and heritability on test performance (Table 1, fourth and fifth sections). Under the simplifying assumptions used in the analytic calculations, we showed that power for all tests remains the same for change in $n$ and $h^2$ such that $n{h}^2$ is fixed. We repeated a subset of our simulations with $\left(n,h^2\right)=\left(5,000,2\%\right),\left(10,000,1\%\right),\text{and}\left(50,000,0.2\%\right)$ to investigate how the tests performed with a different sample size and region heritability but the same $n{h}^2$ (Table 1, sixth section) under more realistic settings.

Table 1.

Genetic models in our simulations

$\mathbf{n}$	${\mathbf{h}}^{\mathbf{2}}$	p(causal ∣ variant category)
		PTV	Deleterious missense	Other missense
100,000	0.10%	0.60	0.30	0.01
100,000	0.10%	0.80	0.30	0.01
100,000	0.10%	1.00	0.30	0.01
100,000	0.10%	0.80	0.10	0.01
100,000	0.10%	0.80	0.30	0.01
100,000	0.10%	0.80	0.50	0.01
100,000	0.10%	0.80	0.30	0.00
100,000	0.10%	0.80	0.30	0.01
100,000	0.10%	0.80	0.30	0.10
30,000	0.10%	0.80	0.30	0.01
100,000	0.10%	0.80	0.30	0.01
300,000	0.10%	0.80	0.30	0.01
30,000	0.01%	0.80	0.30	0.01
100,000	0.01%	0.80	0.30	0.01
300,000	0.01%	0.80	0.30	0.01
5,000	2.00%	0.80	0.50	0.10
10,000	1.00%	0.80	0.50	0.10
50,000	0.20%	0.80	0.50	0.10
100,000	0.10%	0.80	0.50	0.10

The first three sections address the impact of changing p(causal) values for the three variant categories; the fourth and fifth sections address the impact of varying $n$ and $h^2$; and the sixth section addresses the impact of varying $n$ and $h^2$ while keeping $n{h}^2$ fixed.

All four tests we consider are commonly used and have been shown to have well-controlled type I error rates (e.g., Wu et al.¹⁵ and Lee et al.¹⁶) given sufficiently large sample sizes. Therefore, we did not re-demonstrate that these tests were well calibrated, given the heavy computational burden to accurately estimate significance levels of $2.5\times {10}^{-6}$ to $5\times {10}^{-9}$. To check our implementations of these tests, we did demonstrate accurate calibration at significance levels on the order of ${10}^{-3}$ (data not shown).

To limit computational cost, we restricted our analyses to the 1,060 genes in chromosome 2 having at least one qualifying variant in the PTV mask in our analysis subset of 378,215 participants from the UK Biobank. These genes are representative of the 17,245 genes genome wide in number of variants and sum of MAFs of the rare variants for each of the three masks (Table S3). To assess the accuracy of our assumption of independent variants in our analytic calculations, we pruned the rare variants in chromosome 2 with a linkage disequilibrium (LD) $r^2$ threshold of 0.2. This resulted in a loss of <1.4% of the variants, implying that most rare variants are in very weak LD with each other and that the assumption of independence in the analytic calculations is not too unrealistic.

We performed all simulations and analyses in R version 4.2.2 or higher.

Results

Introduction

We first compared powers of burden and SKAT tests with the power of single-variant tests using analytic calculations in the simplest case of independent variants with equal MAFs and effect sizes. We next repeated these calculations for the DNAJC5G and LINC01305 genes only with the assumption of independent variants. We then performed simulations with the UK Biobank data to relax these assumptions. We describe results for significance levels of $5\times {10}^{-8}$ for single-variant tests and $2.5\times {10}^{-6}$ for aggregation tests except where noted otherwise.

Analytic calculations: Power to detect association given independent variants with equal MAFs and effect sizes and all variants included in the mask

We first examined the power of each test as a function of the number of causal variants $c$ in the gene. We fixed the product of sample size and region heritability $n{h}^2$ at $50$ and the total number of variants $v$ at 50. Under these circumstances, we found that power for single-variant, burden, and SKAT tests decreased, increased, and was unaffected by an increase in the number of causal variants, respectively. Further, burden and SKAT tests required at least 16% and 6% of variants to be causal for the aggregation test to have greater power than the single-variant test. SKAT had greater power than the burden test for a smaller proportion of causal variants, but the burden test became more powerful than SKAT as proportion of causal variants increased (Figure 1).

Figure 1.

Power for burden, SKAT, and single-variant tests as a function of number of causal variants $c$ for $n{h}^2=50$ and total number of variants $v=50$

Blue, burden; red, SKAT; green, single variant.

Next, we studied the effects of varying $n{h}^2$ and $v$ on the performance of each test. As expected, the power for each test increased with an increase in either sample size $n$ or region heritability $h^2$ (Figure 2). Further, we observed that with a fixed number of variants $v$, an increase in $n{h}^2$ had little impact on the proportion of variants needed for burden and SKAT tests to be more powerful than single-variant tests. Moreover, with sample size $n$ and region heritability $h^2$ fixed, for a larger number of variants $v$ in the gene, we required a lower proportion of causal variants $c/v$ for aggregation tests to be more powerful than single-variant tests. The power for SKAT decreased as the number of variants in the gene increased, due to the increase in the degrees of freedom of the SKAT statistic. We created a user-friendly R Shiny app to evaluate power of each test as a function of the different parameters considered (https://debrajbose.shinyapps.io/analytic_calculations/).

Figure 2.

Power for burden, SKAT, and single-variant tests as a function of proportion of causal variants $c/v$ for $n{h}^2=20,50,\text{and}100$, and total number of variants $v=10,50,\text{and}100$

Blue, burden; red, SKAT; green, single variant.

Analytic calculations: Power to detect association given independent variants and all variants being included in the mask

We computed the power of all three tests for the DNAJC5G and LINC01305 genes in chromosome 2 to investigate their dependence on MAF and effect sizes. The 20 rare variants in LINC01305 have very similar MAFs ($2.64\times {10}^{-6}$ to $3.17\times {10}^{-5}$); power curves for all three tests (Figure S2, second row) were similar to those for the case of equal MAF and effect sizes (Figure 2). In contrast, DNAJC5G has one variant that is less rare (MAF = $7.62\times {10}^{-3}$) than the other 48 variants ($1.32\times {10}^{-6}$ to $2.62\times {10}^{-4}$), and inclusion of this variant in the causal set led to a substantial increase in power for all tests (Figure S2, first row). Power curves were less smooth for the step function-effect size models than the negative logarithmic MAF-effect size model (Figure S2), due to the randomness introduced by sampling effect sizes from a normal distribution. Our R Shiny app (https://debrajbose.shinyapps.io/analytic_calculations/) can be used to generate power curves like these for any specified set of model parameters.

Simulations

We next performed simulations based on observed and imputed exome sequence data for 378,215 nearly unrelated UK Biobank White British participants to address the impact of variability in variant MAFs and effect sizes and LD between variants.

Effect of changing sample size and heritability while keeping their product constant

The analytic calculations initially assumed only that all variants in the gene were independent. In that situation, NCPs depended on sample size, MAFs, effect sizes, and aggregation test weights. When we further assumed that all variants had the same MAF, causal variants had the same effect sizes, and all variants had equal aggregation test weights, all NCPs and power were (nearly) constant as a function of $n{h}^2$. Therefore, we chose to investigate how power was affected by change in sample size and heritability while keeping their product constant when all these further assumptions are violated. Power plots over four different $\left(n,h^2\right)$ combinations showed similar shapes for burden and SKAT-O tests (Figure 3) for analysis with PTV+Missense(Deleterious) and PTV+Missense(All) masks. In contrast, for SKAT and single-variant tests, we found a clear increase in power as region heritability $h^2$ increased and sample size $n$ decreased with $n{h}^2$ constant. Since SKAT is a variance component test that explicitly tests for the presence of heritability, a larger value of heritability may have had greater impact than decreasing sample size. Also, effect sizes are generally larger for larger $h^2$, making it easier for single-variant tests to detect true associations. Boxplots for difference in power (aggregation test minus single-variant test) were similar in shape for each of the tests across the four different ($n,h^2$) combinations, especially for sample sizes of 50,000 and higher (Figure S3).

Figure 3.

Power for burden, SKAT-O, SKAT, and maximum single-variant tests for the 1,060 chromosome 2 genes having at least one PTV with MAF < 1%: Different $\left(n,h^2\right)$ combinations chosen so that $n{h}^2=100$

PTVs, deleterious missense variants, and other missense variants are assumed to have probabilities of 0.8, 0.5, and 0.1 of being causal, respectively.

Effect of changing sample size and heritability

Now, we consider region heritabilities $h^2$ of both 0.1% and 0.01%, and sample sizes of $n$ = 30,000, 100,000, and 300,000, with causal probabilities of 0.8, 0.3, and 0.01 for PTVs, deleterious missense variants, and other missense variants, respectively. With region heritability of 0.01%, both aggregation and single-variant tests had essentially no power at sample sizes ≤100,000 (i.e., $n{h}^2<10$), but power for aggregation tests increased more rapidly than single-variant tests as sample size reached 300,000 (i.e., until $n{h}^2<30$). Aggregation tests were more powerful than single-variant tests for >50% of the 1,060 chromosome 2 genes having at least one qualifying variant in the PTV mask for sample size and heritability combinations ($n,h^2$) of (30,000, 0.1%) and (300,000, 0.01%) with the PTV and PTV+Missense(Deleterious) mask, and boxplots for these two combinations were very similar because their $n{h}^2$ was the same. With region heritability of 0.1%, power for single-variant tests increased more rapidly than that of aggregation tests as sample size increased from 30,000 to 300,000 (i.e., $30<n{h}^2<300$). Overall, if either sample size or region heritability was sufficiently high, aggregation tests were more powerful at lower values of $n{h}^2$ while single-variant tests were more powerful at higher values of $n{h}^2$ (Figures S4 and S5).

Effect of causal probability of variants and choice of masks

Figures 4, 5, and 6 demonstrate that power for aggregation tests was greater than power for single-variant tests only when a large enough proportion of variants in the gene $c/v$ were causal. Choice of masks was also a key factor, as including more causal variants and excluding neutral ones generally increased the power of aggregation tests.

Figure 4.

Differences in power for burden, SKAT-O, and SKAT tests for the 1,060 chromosome 2 genes having at least one PTV with MAF < 1% for varying probability of being causal for PTVs

Difference in power is defined as aggregation test power minus single-variant test power. Probability of being causal for PTVs is allowed to vary as 0.6, 0.8, and 1 with $n=100,000$ and $h^2=0.1\%$. Numbers on top of each boxplot denote the proportion of genes for which the corresponding aggregation test is more powerful than single-variant tests. If the median of a boxplot corresponds with the dashed line at zero, aggregation tests perform better than single-variant tests for 50% of the genes considered.

Figure 5.

Differences in power for burden, SKAT-O, and SKAT tests for the 1,060 chromosome 2 genes having at least one PTV with MAF < 1% for varying probability of being causal for deleterious missense variants

Difference in power is defined as aggregation test power minus single-variant test power. Probability of being causal for deleterious missense variants is allowed to vary as 0.1, 0.3, and 0.5 with $n=100,000$ and $h^2=0.1\%$. Numbers on top of each boxplot denote the proportion of genes for which the corresponding aggregation test is more powerful than single-variant tests. If the median of a boxplot corresponds with the dashed line at zero, aggregation tests perform better than single-variant tests for 50% of the genes considered.

Figure 6.

Differences in power for burden, SKAT-O, and SKAT tests for the 1,060 chromosome 2 genes having at least one PTV with MAF < 1% for varying probability of being causal for other missense variants

Difference in power is defined as aggregation test power minus single-variant test power. Probability of being causal for other missense variants is allowed to vary as 0, 0.01, and 0.1 with $n=100,000$ and $h^2=0.1\%$. Numbers on top of each boxplot denote the proportion of genes for which the corresponding aggregation test is more powerful than single-variant tests. If the median of a boxplot corresponds with the dashed line at zero, aggregation tests perform better than single-variant tests for 50% of the genes considered.

We found that single-variant tests had greater power than aggregation tests for more than half of the 1,060 chromosome 2 genes for most of the scenarios over the range of causal probabilities considered in Figures 4, 5, and 6. However, there were some scenarios where aggregation tests were more powerful. For the PTV mask, burden and SKAT-O tests were more powerful than single-variant tests for 60% of the 1,060 genes in chromosome 2 having at least one qualifying variant in the PTV mask only when all PTVs in the gene were causal and deleterious and other missense variants had less than 30% and 1% probabilities of being causal (Figure 4). For the PTV+Missense(Deleterious) mask, burden, SKAT, and SKAT-O tests were more powerful than single-variant tests for 66%, 55%, and 70% genes, respectively, when PTVs, deleterious missense variants, and other missense variants had 80%, 50%, and 1% probabilities of being causal (Figure 5) and for 59%, 50%, and 63% of genes, respectively, when PTVs, deleterious missense variants, and other missense variants had 100%, 30%, and 1% probabilities of being causal (Figure 4). Only SKAT-O performed better than single-variant tests for >50% of genes when PTVs, deleterious missense variants, and other missense variants had 80%, 30%, and 0% probabilities of being causal (Figure 6). For the PTV+Missense(All) mask, we found no situations in Figures 4, 5, and 6 where aggregation tests performed better than single-variant tests for >50% of genes. As an exercise, we examined higher causal probabilities for all categories of variants to find the minimum causal probability of other missense variants for which aggregation tests are more powerful than single-variant tests for >50% of genes and found this to be the case when PTVs, deleterious missense variants, and other missense variants had probabilities of at least 80%, 70%, and 20%, respectively, of being causal, a scenario that is plausible for genes with a very small number of missense variants.

In general, when we increased the causal probability of variants from a particular functional annotation category, aggregation tests with masks containing those variants exhibited an increase in power and masks not containing those variants suffered a loss in power. Figures 4, 5, and 6 demonstrate that aggregation tests had greater power only when the mask chosen had an adequate coverage of the causal variants. For example, for the PTV+Missense(Deleterious) mask, the boxplots show that we would require >80% PTVs, >30% deleterious missense variants, and <1% other missense variants to be causal for aggregation tests to have greater power than single-variant tests for >50% of genes.

SKAT-O power resembled that of the burden test in most scenarios for analysis with the PTV and PTV+Missense(Deleterious) masks, as the burden test has greater power than SKAT when variants have the same effect directions. However, for analysis with the PTV+Missense(All) mask, which contains a significantly larger number of variants per gene due to genes having a much larger number of other missense variants compared to PTVs and deleterious missense variants, we assumed a smaller causal probability for other missense variants, resulting in a lower proportion of causal variants. As a result, despite our assumption of variants having the same effect directions, SKAT-O power resembled that of SKAT, as SKAT typically has greater power than the burden test when a lower proportion of variants are causal.

Effect of more stringent significance thresholds

We next investigated whether the relative performance of the tests changed when we used stricter levels of significance: ${\alpha }_{SV}=5\times {10}^{-9}$ instead of $5\times {10}^{-8}$ and ${\alpha }_A=1.12\times {10}^{-6}$ instead of $2.5\times {10}^{-6}$. As expected, power decreased for all tests using these stricter levels of significance. However, the general conclusions remained the same at all four combinations of significance levels (Figure S6).

Effect of equal weights of mask variants in aggregation tests

We repeated simulations with causal probabilities of 0.8, 0.3, and 0.01 for PTVs, deleterious missense variants, and other missense variants with equal weights for all association tests instead of Beta(1,25) weights. For most genes, aggregation test powers decreased very slightly when we used equal weights, with tests for masks with more variants showing slightly larger decreases in power than those with fewer variants. However, the average decrease in power for all three aggregation tests was <1%, and boxplots for difference in power between aggregation tests and single-variant tests exhibited essentially no discernible differences from their Beta(1,25) weight counterparts (Figure S7).

Effect of causal variants having opposite effect directions

We repeated the same set of simulations, this time randomly assigning a negative effect direction to half the causal variants. As expected, there was a drastic loss of power for burden tests, while power for SKAT remained nearly the same as when all variants had the same effect direction. Power for SKAT-O now resembled that for SKAT, since SKAT-O was designed to be an optimal combination of SKAT and burden tests, and SKAT had greater power (Figure S8).

Effect of different effect size models

We repeated the same set of simulations with two step function-effect size models. Boxplots for difference in power with the $-{\log }_{10}MAF$ model were very similar to the ones with the step function model with $\left(\mu ,\sigma \right)=\left(1,0.25\right)$. Aggregation tests were slightly more powerful under the step function-effect size model with $\left(\mu ,\sigma \right)=\left(2,0.5\right)$ (Figure S9), suggesting that results did not vary substantially under different inverse MAF-effect size models.

Discussion

Aggregation tests aim to increase power to detect rare-variant associations by pooling information across sets of rare variants in a gene or genomic region. Here, we investigated a range of genetic models and sample sizes to determine when aggregation tests are of greater power than single-variant tests for association analysis of quantitative traits. We found in our analytic calculations that the proportion of causal variants needed for aggregation tests to have greater power than single-variant tests decreased with an increase in the number of variants in the gene and was almost unaffected by an increase in the product of sample size and heritability. Our simulations, which accounted for a wider range of genetic models and for LD, found that (1) single-variant tests had greater power than aggregation tests for most of the scenarios in the wide range of causal probabilities considered, and (2) lower values of the product of sample size and heritability favor aggregation tests while higher values favor single-variant tests. We demonstrated the importance of mask selection across different causal probabilities of variant functional categories. Our analysis showed that the combination of PTVs and deleterious missense variants represents the most powerful mask for the range of causal probabilities under consideration.

Previous studies^38,39,40 have performed limited simulations to compare powers of single-variant and aggregation tests. Our work offers the following strengths. First, we performed analytic calculations to compute the power of tests under the assumption of independent variants and provide an online tool (https://debrajbose.shinyapps.io/analytic_calculations/) to allow investigators to investigate how the different variables (sample size, heritability, MAFs, effect sizes, and proportion of causal variants) affect the power of each test. Second, our simulations are based on genotypes on a large dataset of 378,215 individuals. Third, we take functional annotations of variants into account and investigate the power of each test under a wide range of causal probabilities of variants with different functional annotations. Fourth, we analyze the performance of aggregation tests by grouping variants based on functional annotations and MAFs into three commonly used masks.

We might have anticipated that the “perfect” mask that includes all causal variants and excludes all neutral variants would result in the most powerful aggregation test. However, this is not always the case, particularly given one or a few rare variants of large effect and/or many variants of very modest effect. For example, with $n=100,000$ and $h^2=0.1\%$, suppose there are ten causal variants in a gene, all with MAF $=0.001$, and one with effect size $\beta =0.5$ and the others with effect size $\beta =0.05$. The mask that contains only the large-effect variant will result in a more powerful burden test (power $=0.99$) than the mask containing all causal variants (power $=0.32$), consistent with the single-variant test being substantially more powerful than the aggregation test. Burden tests with the PTV and the PTV+Missense(Deleterious) mask are more likely to get results, since other missense variants that are causal are expected to have lower effect sizes than PTVs and deleterious missense variants due to other missense variants being less rare on average, and analysis with the PTV+Missense(All) mask would decrease power.

Our study also has some limitations. First, we chose to conduct this study on quantitative traits; it would be interesting to conduct a similar study for binary traits. Second, we have limited our attention to three commonly used aggregation tests that span a substantial range for such tests; many other aggregation tests could have been considered. One such test is the aggregated Cauchy association test (ACAT),²¹ which was developed to increase power when there is a lower proportion of causal variants in the mask. We repeated a subset of simulations also performing ACAT and found that ACAT was more powerful than the burden, SKAT, and SKAT-O tests for the PTV+Missense(All) mask which had the smallest proportion of causal variants and was less or similarly powerful for the other two smaller masks (Figure S10). Third, we focused on three variant masks and a single MAF threshold; these masks and thresholds are commonly used, but others are possible. Fourth, while we have categorized the variants according to VEP⁵⁵ annotations, we acknowledge that annotation accuracy often is imperfect. Fifth, we assumed two inverse MAF-effect size models, a negative logarithmic model and a step function model; obviously, other models are possible, but results were consistent across both models. Sixth, to save computation, we restricted our simulations to genes on chromosome 2; however, we expect our results to generalize, since chromosome 2 genes are representative of all autosomal genes in the number of rare variants and sum of rare-variant MAFs.

In summary, we investigated and compared the powers of single-variant and aggregation tests under different genetic models, variant masks, and sample sizes. We found that we require a substantial proportion of variants in a gene to be causal along with the mask having adequate coverage of these causal variants for aggregation tests to be more powerful than single-variant tests. Our online tool (https://debrajbose.shinyapps.io/analytic_calculations/) provides investigators the opportunity to explore situations of interest to them. With rapid growth of large-scale biobanks and the use of both single-variant and aggregation tests in rare-variant association analysis, our study provides insight into the kind of genetic models favoring each test.

Data and code availability

We provide code to perform our analytic calculations and simulations in our GitHub repository (https://github.com/bosedebraj96/Aggregation_vs_SV). Our R Shiny app for the analytic calculations can be found at https://debrajbose.shinyapps.io/analytic_calculations/.

Web resources

NHGRI-EBI GWAS Catalog, https://www.ebi.ac.uk/gwas/

R Shiny app for analytic calculations, https://debrajbose.shinyapps.io/analytic_calculations/

Acknowledgments

We thank all the participants and investigators of the UK Biobank study. This work was supported by the National Institutes of Health (NIH) under award R01 HG009976 (to M.B.).

Declaration of interests

The authors declare no competing interests.

Published: July 29, 2025