A simple new approach to variable selection in regression, with application to genetic fine mapping

Abstract

We introduce a simple new approach to variable selection in linear regression, with a particular focus on quantifying uncertainty in which variables should be selected. The approach is based on a new model — the “Sum of Single Effects” (SuSiE) model — which comes from writing the sparse vector of regression coefficients as a sum of “single-effect” vectors, each with one non-zero element. We also introduce a corresponding new fitting procedure — Iterative Bayesian Stepwise Selection (IBSS) — which is a Bayesian analogue of stepwise selection methods. IBSS shares the computational simplicity and speed of traditional stepwise methods, but instead of selecting a single variable at each step, IBSS computes a distribution on variables that captures uncertainty in which variable to select. We provide a formal justification of this intuitive algorithm by showing that it optimizes a variational approximation to the posterior distribution under the SuSiE model. Further, this approximate posterior distribution naturally yields convenient novel summaries of uncertainty in variable selection, providing a Credible Set of variables for each selection. Our methods are particularly well-suited to settings where variables are highly correlated and detectable effects are sparse, both of which are characteristics of genetic fine-mapping applications. We demonstrate through numerical experiments that our methods outperform existing methods for this task, and illustrate their application to fine-mapping genetic variants influencing alternative splicing in human cell-lines. We also discuss the potential and challenges for applying these methods to generic variable selection problems.

1. INTRODUCTION

The need to identify, or “select”, relevant variables in regression models arises in a diverse range of applications, and has spurred development of a correspondingly diverse range of methods (e.g., see O’Hara and Sillanpää 2009; Fan and Lv, 2010; Desboulets, 2018; George and McCulloch, 1997, for reviews). However, variable selection is a complex problem, and so despite considerable work in this area there remain important issues that existing methods do not fully address. One such issue is assessing uncertainty in which variables should be selected, particularly in settings involving very highly correlated variables. Here we introduce a simple and computationally scalable approach to variable selection that helps address this issue.

Highly correlated variables pose an obvious challenge to variable selection methods, simply because they are hard to distinguish from one another. Indeed, in an extreme case where two variables (say, x₁ and x₂) are completely correlated, it is impossible to claim, based on a regression analysis, that one variable should be selected as relevant rather than the other. In some applications such ambiguity causes few practical problems. Specifically, in some applications variable selection is used only to help build an accurate predictor, in which case it suffices to arbitrarily select one of the two identical variables (or both); prediction accuracy is unaffected by this choice. However, in other scientific applications, variable selection is used as a means to help learn something about the world, and in those applications the ambiguity created by highly correlated variables is more problematic because scientific conclusions depend on which variables are selected. In these applications, it is crucial to acknowledge uncertainty in which variables should be selected. This requires methods that can draw conclusions such as “either x₁ or x₂ is relevant and we cannot decide which” rather than methods that arbitrarily select one of the variables and ignore the other. While this may seem a simple goal, in practice most existing variable selection methods do not satisfactorily address this problem (see Section 2 for further discussion). These shortcomings motivate our work here.

One particular application where these issues arise is genetic fine-mapping (e.g., Veyrieras et al., 2008; Maller et al., 2012; Spain and Barrett, 2015; Huang et al., 2017; Schaid et al., 2018). The goal of fine-mapping is to identify the genetic variants that causally affect some traits of interest (e.g., low density lipoprotein cholesterol in blood, or gene expression in cells). In other words, the main goal of fine-mapping is to learn something about the world, rather than build a better predictor. (This is not to say that predicting traits from genetic variants is not important; indeed, there is also a lot of work on prediction of genetic traits, but this is not the main goal of fine-mapping.) The most successful current approaches to fine-mapping frame the problem as a variable selection problem, building a regression model in which the regression outcome is the trait of interest, and the candidate predictor variables are the available genetic variants (Sillanpää and Bhattacharjee, 2005). Performing variable selection in a regression model identifies variants that may causally affect the trait. Fine-mapping is challenging because the variables (genetic variants) can be very highly correlated, due to a phenomenon called linkage disequilibrium (Ott, 1999). Indeed, typical studies contain many pairs of genetic variants with sample correlations exceeding 0.99, or even equaling 1.

Our approach builds on previous work on Bayesian variable selection in regression (BVSR) (Mitchell and Beauchamp, 1988; George and McCulloch, 1997), which has already been widely applied to genetic fine-mapping and related applications (e.g., Meuwissen et al., 2001; Sillanpää and Bhattacharjee, 2005; Servin and Stephens, 2007; Hoggart et al., 2008; Stephens and Balding, 2009; Logsdon et al., 2010; Guan and Stephens, 2011; Bottolo et al., 2011; Maller et al., 2012; Carbonetto and Stephens, 2012; Zhou et al., 2013; Hormozdiari et al., 2014; Chen et al., 2015; Wallace et al., 2015; Moser et al., 2015; Wen et al., 2016; Lee et al., 2018). BVSR is an attractive approach to these problems because it can, in principle, assess uncertainty in which variables to select, even when the variables are highly correlated. However, applying BVSR in practice remains difficult for at least two reasons. First, BVSR is computationally challenging, often requiring implementation of sophisticated Markov chain Monte Carlo or stochastic search algorithms (e.g., Bottolo and Richardson, 2010; Bottolo et al., 2011; Guan and Stephens, 2011; Wallace et al., 2015; Benner et al., 2016; Wen et al., 2016; Lee et al., 2018). Second, and perhaps more importantly, the output from BVSR methods is typically a complex posterior distribution — or samples approximating the posterior distribution — and this can be difficult to distill into results that are easily interpretable.

Our work addresses these shortcomings of BVSR through several innovations. We introduce a new formulation of BVSR, which we call the “Sum of Single Effects” (SuSiE) model. This model, while similar to existing BVSR models, has a different structure that naturally leads to a simple, intuitive, and fast procedure for model fitting — Iterative Bayesian Stepwise Selection (IBSS) — which is a Bayesian analogue of traditional stepwise selection methods (and which enjoys important advantages over these traditional selection methods, as we explain below). We provide a principled justification for this intuitive algorithm by showing that it optimizes a variational approximation to the posterior distribution under the SuSiE model. Although variational approaches to BVSR already exist (Logsdon et al., 2010; Carbonetto and Stephens, 2012), our new approach introduces a different family of approximating distribution that provides much more accurate inferences in settings with highly correlated variables.

A key feature of our method, which distinguishes it from most existing BVSR methods, is that it produces “Credible Sets” of variables that quantify uncertainty in which variable should be selected when multiple, highly correlated variables compete with one another. These Credible Sets are designed to be as small as possible while still each capturing a relevant variable. Arguably, this is exactly the kind of posterior summary that one would like to obtain from MCMC-based or stochastic search BVSR methods, but doing so would require non-trivial post-processing of their output. In contrast, our method provides this posterior summary directly, and with little extra computational effort.

The structure of this paper is as follows. Section 2 provides further motivation for our work, and brief background on BVSR. Section 3 describes the new SuSiE model and fitting procedure. Section 4 uses simulations, designed to mimic realistic genetic fine-mapping studies, to demonstrate the effectiveness of our approach compared with existing methods. Section 5 illustrates the application of our methods to fine-mapping of genetic variants affecting splicing, and Section 6 briefly highlights the promise (and limitations) of our methods for other applications such as change-point problems. We end with a discussion highlighting avenues for further work.

2. BACKGROUND

2.1. A motivating toy example

Suppose the relationship between an n-vector y and an n × p matrix X = (x₁, …, x_p), is modeled as a multiple regression:

$$\begin{gathered} y=Xb+\mathit{e} \\ \mathit{e}~N_n\left(0,{\sigma }^2{I}_n\right), \end{gathered}$$ (2.1)

where b is a p-vector of regression coefficients, e is an n-vector of error terms, σ² > 0 is the residual variance, I_n is the n × n identity matrix, and N_r (μ, Σ) denotes the r-variate normal distribution with mean μ and variance Σ. For brevity, we will refer to variables j with non-zero effects (b_j ≠ 0) as “effect variables”.

Assume now that exactly two variables are effect variables — variables 1 and 4, say — and that these two effect variables are each completely correlated with another non-effect variable, say x₁ = x₂ and x₃ = x₄. Further suppose that no other pairs of variables are correlated. Here, because the effect variables are completely correlated with other variables, it is impossible to confidently select the correct variables, even when n is very large. However, given sufficient data it should be possible to conclude that there are (at least) two effect variables, and that:

$$(b_1\ne 0\text{\hspace{0.17em}or\hspace{0.17em}}{b}_2\ne 0)\text{\hspace{0.17em}and\hspace{0.17em}}(b_3\ne 0\text{\hspace{0.17em}or\hspace{0.17em}}{b}_4\ne 0).$$ (2.2)

Our goal, in short, is to provide methods that directly produce this kind of inferential statement. Although this example is simplistic, it mimics the kind of structure that occurs in, for example, genetic fine-mapping applications, where it often happens that an association can be narrowed down to a small set of highly correlated genetic variants, but not down to an individual variant.

Most existing approaches to sparse regression do not provide statements like (2.2), nor do they attempt to do so. For example, methods that maximize a penalized likelihood, such as the lasso (Tibshirani, 1996) or elastic net (EN; Zou and Hastie, 2005), select a single “best” combination of variables, and make no attempt to assess whether other combinations are also plausible. In our toy example, EN selects all four variables, implying b₁ ≠ 0, b₂ ≠ 0, b₃ ≠ 0 and b₄ ≠ 0, which is quite different from (2.2). Recently-developed selective inference approaches (Taylor and Tibshirani, 2015) do not solve this problem, because they do not assess uncertainty in which variables should be selected; instead they assess uncertainty in the coefficients of the selected variables within the selected model. In our toy motivating example, selective inference methods sometimes selects the wrong variables (inevitably, due to the complete correlation with other variables) and then assigns them highly significant p values (see Wang et al., 2020b, for an explicit example accompanied by code). The p values are significant because, even though the wrong variables are selected, their coefficients — within the (wrong) selected model — can be estimated precisely. An alternative approach, which does address uncertainty in variable selection, is to control the false discovery rate (FDR) among selected variables — for example, using stability selection (Meinshausen and Bühlmann, 2010) or the knockoff filter (Barber and Candés, 2015). However, in examples with very highly correlated variables no individual variable can be confidently declared an effect variable, and so controlling the FDR among selected variables results in no discoveries, and not inferences like (2.2).

One approach to producing inferences like (2.2) is to reframe the problem, and focus on selecting groups of variables, rather than individual variables. A simple version of this idea might first cluster the variables into groups of highly correlated variables, and then perform some kind of “group selection” (Huang et al., 2012). However, while this could work in our toy example, in general this approach requires ad hoc decisions about which variables to group together, and how many groups to create — an unattractive feature we seek to avoid. A more sophisticated version of this idea is to use hierarchical testing (Meinshausen, 2008; Yekutieli, 2008; Mandozzi and Bühlmann, 2016; Renaux et al., 2020), which requires specification of a hierarchy on the variables, but avoids an a priori decision on where to draw group boundaries. However, in applications where variables are not precisely arranged in a known hierarchy — which includes genetic fine-mapping — this approach is also not entirely satisfactory. In numerical assessments shown later (Section 4), we find that this approach can considerably overstate the uncertainty in which variables should be selected.

Another approach that could yield statements like (2.2), at least in principle, is the Bayesian approach to variable selection (BVSR; see Introduction for references). BVSR methods introduce a prior distribution on b that favours sparse models (few effect variables), and then compute a posterior distribution assessing relative support for each combination of variables. In our toy example, the posterior distribution would roughly have equal mass (≈ 0.25) on each of the four equivalent combinations {1, 3} {1, 4}, {2, 3} and {2, 4}. This posterior distribution contains exactly the information necessary to infer (2.2). Likewise, in more complex settings, the posterior distribution contains information that could, in principle, be translated to simple statements analogous to (2.2). This translation is, however, highly non-trivial in general. Consequently, most implementations of BVSR do not provide statements like (2.2), but rather summarize the posterior distribution with a simpler but less informative quantity: the marginal posterior inclusion probability (PIP) of each variable,

$${\text{PIP}}_j≔\text{Pr}\left(b_j\ne 0\mid X,y\right).$$ (2.3)

In our example, PIP₁ = PIP₂ = PIP₃ = PIP₄ ≈ 0.5. While not inaccurate, the PIPs do not contain the information in (2.2). In Wang et al. (2020b), we illustrate inference of Credible Sets in two additional toy examples in which the variables are correlated in more complicated ways.

2.2. Credible Sets

To define our main goal more formally, we introduce the concept of a Credible Set (CS) of variables:

Definition 1. In the context of a multiple regression model, a level-ρ Credible Set is defined to be a subset of variables that has probability ⩾ ρ of containing at least one effect variable (i.e., a variable with non-zero regression coefficient). Equivalently, the probability that all variables in the Credible Set have zero regression coefficients is ⩽ 1 − ρ.

Our use of the term Credible Set here indicates that we have in mind a Bayesian inference approach, in which the probability statements in the definition are statements about uncertainty in which variables are selected given the available data and modelling assumptions. One could analogously define a Confidence Set by interpreting the probability statements as referring to the set, considered random.

Although the term Credible Set has been used in fine-mapping applications before, most previous uses either assumed there was a single effect variable (Maller et al., 2012), or defined a CS as a set that contains all effect variables (Hormozdiari et al., 2014), which is a very different definition (and, we argue, both less informative and less attainable; see further discussion below). Our definition here is closer to the “signal clusters” from Lee et al. (2018), and related to the idea of “minimal true detection” in Mandozzi and Bühlmann (2016).

With Definition 1 in place, our primary aim can be restated: we wish to report as many CSs as the data support, each with as few variables as possible. For example, to convey (2.2) we would report two CSs, {1, 2} and {3, 4}. As a secondary goal, we would also like to prioritize the variables within each CS, assigning each a probability that reflects the strength of the evidence for that variable being an effect variable. Our methods achieve both of these goals.

It is important to note that, if a variable is not included in any CS produced by our method, this does not imply that it is not an effect variable. This is analogous to the fact that, in hypothesis testing applications, a non-significant p value does not imply that the null hypothesis is true. In practice no variable selection method can guarantee identifying every effect variable unless it simply selects all variables, because finite data cannot rule out that every variable has a (possibly tiny) effect. This is why the CS definition of Hormozdiari et al. (2014) is unattainable, at least without strong assumptions on sparsity. It also explains why attempting to form confidence or credible sets for identifying the true model (i.e., the exact combination of effect variables) leads to very large sets of models; see Ferrari and Yang (2015) for example.

2.3. The single effect regression model

We now describe the building block for our approach, the “Single Effect Regression” (SER) model, which we define as a multiple regression model in which exactly one of the p explanatory variables has a non-zero regression coefficient. This idea was introduced in Servin and Stephens (2007) to fine-map genetic associations, and consequently has been adopted and extended by others, including Veyrieras et al. (2008) and Pickrell (2014). Although of very narrow applicability, the SER model is trivial to fit. Furthermore, when its assumptions hold, the SER provides exactly the inferences we desire, including CSs. For example, if we simplify our motivating example (Section 2.1) to have a single effect variable — variable 1, for example — then the SER model would, given sufficient data, infer a 95% CS containing both of the correlated variables, 1 and 2, with PIPs of approximately 0.5 each. This CS tells us that we can be confident that one of the two variables has a non-zero coefficient, but we do not know which one.

Specifically, we consider the following SER model, with hyperparameters for the residual variance, σ², the prior variance of the non-zero effect, ${\sigma }_0^2$, and the prior inclusion probabilities, π = (π₁, …, π_p), in which π_j gives the prior probability that variable j is the effect variable:

$$y=Xb+e$$ (2.4)

$$\mathit{e}~N_n\left(0,{\sigma }^2{I}_n\right)$$ (2.5)

$$\mathit{b}=b\mathit{\gamma }$$ (2.6)

$$\mathit{\gamma }~\text{Mult}(1,\mathit{\pi })$$ (2.7)

$$b~N_1\left(0,{\sigma }_0^2\right).$$ (2.8)

Here, y is the n-vector of response data; X = (x₁, …, x_p) is an n × p matrix containing n observations of p explanatory variables; b is a scalar representing the “single effect”; γ ∈ {0, 1}^p is a p-vector of indicator variables; b is the p-vector of regression coefficients; e is an n-vector of independent error terms; and Mult(m, π) denotes the multinomial distribution on class counts obtained when m samples are drawn with class probabilities given by π. We assume that y and the columns of X have been centered to have mean zero, which avoids the need for an intercept term (Chipman et al., 2001).

Under the SER model (2.4–2.8), the effect vector b has exactly one non-zero element (equal to b), so we refer to b as a “single effect vector”. The element of b that is non-zero is determined by the binary vector γ, which also has exactly one non-zero entry. The probability vector π determines the prior probability distribution on which of the p variables is the effect variable. In the simplest case, π = (1/p, …, 1/p); we assume this uniform prior here for simplicity, but our methods require only that π is fixed and known (so in fine-mapping one could incorporate different priors based on genetic annotations; e.g., Veyrieras et al., 2008). To lighten notation, we henceforth make conditioning on π implicit.

2.3.1. Posterior under SER model

Given σ² and ${\sigma }_0^2$, the posterior distribution on b = γb is easily computed:

$$\gamma \mid X,y,{\sigma }^2,{\sigma }_0^2~\text{Mult}(1,\alpha )$$ (2.9)

$$b\mid X,y,{\sigma }^2,{\sigma }_0^2,{\gamma }_j=1~N\left({\mu }_{1j},{\sigma }_{1j}^2\right),$$ (2.10)

where α = (α₁, ⋯, α_p) is the vector of PIPs, with ${\alpha }_j≔\text{Pr}\left({\gamma }_j=1\mid X,y,{\sigma }^2,{\sigma }_0^2\right)$, and μ_1j, ${\sigma }_{1j}^2$ are the posterior mean and variance of b given γ_j = 1. Calculating these quantities simply involves performing the p univariate regressions of y on columns x_j of X, for j = 1, …, p, as shown in Appendix A. From α, it is also straightforward to compute a level-ρ CS (Definition 1), CS(α; ρ), as described in Maller et al. (2012), and detailed in Appendix A. In brief, this involves sorting variables by decreasing α_j, then including variables in the CS until their cumulative probability exceeds ρ.

For later convenience, we introduce a function, SER, that returns the posterior distribution for b under the SER model. Since this posterior distribution is uniquely determined by the values of α, μ₁ ≔ (μ₁₁, …, μ_1p) and ${\sigma }_1^2≔\left({\sigma }_{11}^2,\dots ,{\sigma }_{1p}^2\right)$ in (2.9–2.10), we can write

$$\mathit{\text{SER}}\left(X,y;{\sigma }^2,{\sigma }_0^2\right)≔\left(\alpha ,{\mu }_1,{\sigma }_1^2\right).$$ (2.11)

2.3.2. Empirical Bayes for SER model

Although most previous treatments of the SER model assume ${\sigma }_0^2$ and σ² are fixed and known, we note here the possibility of estimating ${\sigma }_0^2$ and/or σ² by maximum-likelihood before computing the posterior distribution of b. This is effectively an Empirical Bayes approach. The log-likelihood for ${\sigma }_0^2$ and σ² under the SER,

$${\ell }_{\text{SER}}\left(y;{\sigma }_0^2,{\sigma }^2\right)≔\text{log\hspace{0.17em}}p\left(y\mid X,{\sigma }_0^2,{\sigma }^2\right),$$ (2.12)

is available in closed form, and can be maximized over one or both parameters (Appendix A).

3. THE SUM OF SINGLE EFFECTS REGRESSION MODEL

We now introduce a new approach to variable selection in multiple regression. Our approach is motivated by the observation that the SER model provides simple inference if there is indeed exactly one effect variable; it is thus desirable to extend the SER to allow for multiple variables. The conventional approach to doing this in BVSR is to introduce a prior on b that allows for multiple non-zero entries (e.g., using a “spike-and-slab” prior; Mitchell and Beauchamp, 1988). However, this approach no longer enjoys the convenient analytic properties of the SER model; posterior distributions become difficult to compute accurately, and computing CSs is even harder.

Here we introduce a different approach which better preserves the desirable features of the SER model. The key idea is simple: introduce multiple single-effect vectors, b₁, …, b_L, and construct the overall effect vector, b, as the sum of these single effects. We call this the “Sum of Single Effects” (SuSiE) regression model:

$$y=Xb+e$$ (3.1)

$$\mathit{e}~N_n\left(0,{\sigma }^2{I}_n\right)$$ (3.2)

$$b=\sum _{l=1}^L{b}_l$$ (3.3)

$$b_l={\mathit{\gamma }}_l{b}_I$$ (3.4)

$${\mathit{\gamma }}_l~\text{Mult}(1,\pi )$$ (3.5)

$$b_l~N_1\left(0,{\sigma }_{0l}^2\right).$$ (3.6)

For generality, we have allowed the variance of each effect, ${\sigma }_{0l}^2$, to vary among the components, l = 1, …, L. The special case in which L = 1 recovers the SER model. For simplicity, we initially assume σ² and ${\sigma }_0^2=\left({\sigma }_{01}^2,\dots ,{\sigma }_{0L}^2\right)$ are given, and defer estimation of these hyperparameters to Section 3.1.3.

Note that if L ≪ p then the SuSiE model is approximately equivalent to a standard BVSR model in which L randomly chosen variables have non-zero coefficients (see Proposition A2 in Appendix C for a formal statement). The main difference is that with some (small) probability some of the single effects b_l in the SuSiE model have the same non-zero co-ordinates, and so the number of non-zero elements in b has some (small) probability of being less than L. Thus, at most L variables have non-zero coefficients in this model. We discuss the choice of L in Section 3.3.

Although the SuSiE model is approximately equivalent to a standard BVSR model, its novel structure has two major advantages. First, it leads to a simple, iterative and deterministic algorithm for computing approximate posterior distributions. Second, it yields a simple way to calculate the CSs. In essence, because each b_l captures only one effect, the posterior distribution on each γ_l can be used to compute a CS that has a high probability of containing an effect variable. The remainder of this section describes both these advantages, and other issues that may arise in fitting the model.

3.1. Fitting SuSiE: Iterative Bayesian stepwise selection

A key motivation for the SuSiE model (3.1–3.6) is that, given b₁, …, b_L−1, estimating b_L involves simply fitting an SER model, which is analytically tractable. This immediately suggests an iterative approach to fitting this model: at each iteration use the SER model to estimate b_l given current estimates of b_l′, for l′ ≠ l; see Algorithm 1. This algorithm is simple and computationally scalable, with computational complexity O(npL) per outer-loop iteration.

We call Algorithm 1 “Iterative Bayesian Stepwise Selection” (IBSS) because it can be viewed as a Bayesian version of stepwise selection approaches. For example, we can compare it with an approach referred to as “forward stagewise” (FS) selection in Hastie et al. 2009, Section 3.3.3 (although subsequent literature often uses this term to mean something slightly different), also known as “matching pursuit” (Mallat and Zhang, 1993). In brief, FS first selects the single “best” variable among p candidates by comparing the results of the p univariate regressions. It then computes the residuals from the univariate regression on this selected variable, then selects the next “best” variable by comparing the results of univariate regression of the residuals on each variable. This process repeats, selecting one variable each iteration, until some stopping criterion is reached.

IBSS is similar in structure to FS, but instead of selecting a single “best” variable at each step, it computes a distribution on which variable to select by fitting the Bayesian SER model. Similar to FS, this distribution is based on the results of the p univariate regressions; consequently each selection step in IBSS has the same computational complexity as in FS, O(np). However, by computing a distribution on variables — rather than choosing a single best variable — IBSS captures uncertainty about which variable should be selected at each step. This uncertainty is taken into account when computing residuals by using a model-averaged (posterior mean) estimate for the regression coefficients. In IBSS, we use an iterative procedure, whereby early selections are re-evaluated in light of the later selections (as in “backfitting”; Friedman and Stuetzle, 1981). The final output of IBSS is L distributions on variables, parameterized by $\left({\alpha }_l,{\mu }_{1l},{\sigma }_{1l}^2\right)$, for l = 1, …, L, in place of the L variables selected by FS. Each distribution is easily summarized, for example, by a 95% CS for each selection.

To illustrate, consider our motivating example (Section 2.1) with x₁ = x₂, x₃ = x₄, and with variables 1 and 4 having non-zero effects. To simplify the example, suppose that the effect of variable 1 is substantially larger than the effect of variable 4. Then FS would first (arbitrarily) select either variable 1 or 2, and then select (again arbitrarily) variable 3 or 4. In contrast, given enough data, the first IBSS update would select variables 1 and 2; that is, it would assign approximately equal weights of 0.5 to variables 1 and 2, and small weights to other variables. The second IBSS update would similarly select variables 3 and 4 (again, with equal weights of approximately 0.5). Summarizing these results would yield two CSs, {1, 2} and {3, 4}, and the inference (2.2) is achieved. This simple example is intended only to sharpen intuition; later numerical experiments demonstrate that IBSS also works well in more realistic settings.

3.1.1. IBSS computes a variational approximation to the SuSiE posterior distribution

The analogy between the IBSS algorithm and the simple FS procedure emphasizes the intuitive and computational simplicity of IBSS, but of course does not give it any formal support. We now provide a formal justification for IBSS. Specifically, we show that it is a coordinate ascent algorithm for optimizing a variational approximation (VA) to the posterior distribution for b₁, …, b_L under the SuSiE model (3.1–3.6). This result also suggests a method for estimating the hyperparameters σ² and ${\sigma }_0^2$.

The idea behind VA methods for Bayesian models (e.g., Jordan et al., 1999; Blei et al., 2017) is to find an approximation q(b₁, …, b_L) to the posterior distribution $p_{\text{post }}≔p\left(b_1,\dots ,b_L\mid X,y,{\sigma }^2,{\sigma }_0^2\right)$ by minimizing the Kullback-Leibler (KL) divergence from q to p_post, written as D_KL(q, p_post), subject to constraints on q that make the problem tractable. Although D_KL(q, p_post) itself is hard to compute, it can be formulated in terms of an easier-to-compute function, F, known as the “evidence lower bound” (ELBO):

$$D_{\text{KL}}\left(q,p_{\text{post }}\right)=\text{log\hspace{0.17em}}p\left(y\mid {\sigma }^2,{\sigma }_0^2\right)-F\left(q;{\sigma }^2,{\sigma }_0^2\right).$$

Because log $p\left(y\mid {\sigma }^2,{\sigma }_0^2\right)$ does not depend on q, minimizing D_KL over q is equivalent to maximizing F; and since F is easier to compute, this is how the problem is usually framed. See Appendix B.1 for further details. (Note that the ELBO also depends on the data, X and y, but we make this dependence implicit to lighten notation.)

We seek an approximate posterior, q, that factorizes as

$$q\left(b_1,\dots ,b_L\right)=\prod _{l=1}^L{q}_l\left(b_l\right).$$ (3.7)

Under this approximation, b₁, …, b_L are independent a posteriori. We make no assumptions on the form of q_l; in particular, we do not require that each q_l factorizes over the p elements of b_l. This is a crucial difference from previous VA approaches for BVSR (e.g., Logsdon et al., 2010; Carbonetto and Stephens, 2012), and it means that q_l can accurately capture strong dependencies among the elements of b_l under the assumption that exactly one element of b_l is non-zero. Intuitively, each factor q_l captures one effect variable, and provides inferences of the form that “we need one of variables {A, B, C}, and we are unsure about which one to select.” By extension, the approximation (3.7) provides inferences of the form “we need to select one variable among the set {A, B, C}, one variable among the set {D, E, F, G}, and so on.”

Under the assumption that the VA factorizes as (3.7), finding the optimal q reduces to the following problem:

$$\underset{q_1,\dots ,q_L}{\text{maximize\hspace{0.17em}}}F\left(q_1,\dots ,q_L;{\sigma }^2,{\sigma }_0^2\right).$$ (3.8)

Although jointly optimizing F over q₁, …, q_L is hard, optimizing an individual factor, q_l, is straightforward, and in fact reduces to fitting an SER model, as formalized in the following proposition.

Proposition 1.

$$\underset{q_l}{\text{argmax\hspace{0.17em}}}F\left(q_1,\dots ,q_L;{\sigma }^2,{\sigma }_0^2\right)=\mathit{\text{SER}}\left(X,{\overline{r}}_l;{\sigma }^2,{\sigma }_{0l}^2\right),$$ (3.9)

where $\overline{r_l}$ denotes the expected value of the residuals obtained by removing the estimated effects other than l,

$${\overline{r}}_l≔y-X\sum _{l^{\prime }\ne l}{\overline{b}}_{l^{\prime }},$$ (3.10)

and where ${\overline{b}}_l$ denotes the expected value of b_l with respect to the distribution q_l.

For intuition, note that computing the posterior distribution for b_l under (3.1–3.6), given the other effects b_l′ for l′ ≠ l, involves fitting a SER to the residuals y – X ∑_l′≠l b_l′. Now consider computing an (approximate) posterior distribution for b_l when b_l′ are not known, and we have approximations q_l′ to their posterior distributions. Proposition 1 states that we can solve for argmax_ql F (q₁, …, q_L) using a similar procedure, except that each b_l′ is replaced with the (approximate) posterior mean ${\overline{b}}_{l^{\prime }}$.

The following is an immediate consequence of Proposition 1:

Corollary 1. IBSS (Algorithm 1) is a coordinate ascent algorithm for maximizing the ELBO, F, over q satisfying (3.7). Equivalently, it is a coordinate ascent algorithm for minimizing the KL divergence D_KL(q, p_post) over q satisfying (3.7), where p_post is the true posterior distribution under the SuSiE model.

Further, as a consequence of being a coordinate ascent algorithm, IBSS converges to a stationary point of F under conditions that are easily satisfied:

Proposition 2. Provided that 0 < σ, σ₀ < ∞ and π_{j >} 0 for all j = 1, …, p, the sequence of iterates q generated by the IBSS method (parameterized by α₁, μ₁₁, ${\sigma }_{11}^2$, …, _L, μ_1L, ${\sigma }_{1L}^2$) converges to a limit point that is a stationary point of F.

The proof of Propositions 1 and 2 and Corollary 1 are given in Appendix B.

3.1.2. Contrast to previous variational approximations

A critical point is that the VA being computed by IBSS is different from previous “fully factorized” VAs for BVSR (e.g., Logsdon et al., 2010; Carbonetto and Stephens, 2012). In settings with highly correlated variables, the new VA produces results that are not only quantitatively different, but also qualitatively different from the fully factorized VA. For example, in our motivating example (Section 2.1), the new VA provides statements like (2.2), whereas the fully factorized VAs do not. Rather, a fully factorized VA often selects at most one of two identical variables without adequately capturing uncertainty in which variable should be selected (Carbonetto and Stephens, 2012). This feature makes the fully factorized VA unsuitable for applications where it is important to assess uncertainty in which variables are selected.

More generally, the new VA computed by IBSS satisfies the following intuitive condition: when two variables are identical, inferences drawn about their coefficients are identical (assuming the priors on their coefficients are the same). Despite the simplicity of this condition, it is not satisfied by existing VAs, nor by point estimates from penalized likelihood approaches with L₀ or L₁ penalty terms. (In fact, Zou and Hastie 2005 use this condition as motivation for the elastic net method, which does ensure that point estimates for coefficients of identical variables are equal.) This property is formalized in the following proposition.

Proposition 3. Consider applying the IBSS algorithm (Algorithm 1) to a data set in which two columns of X are identical; that is, x_j = x_k for some j ≠ k. Further suppose that the prior distributions on selecting these two variables are equal (π_j = π_k). Then the approximate posterior computed by IBSS will be exchangeable in j, k; that is, if ${\omega }_{jk}:{ℝ}^p\to {ℝ}^p$ denotes the function that permutes elements j and k of a p-vector, and q denotes the approximate posterior obtained from the IBSS algorithm, then

$$q\left({\omega }_{jk}\left(b_1\right),\dots ,{\omega }_{jk}\left(b_L\right)\right)=q\left(b_1,\dots ,b_L\right).$$ (3.11)

Proof. Since $q\left(b_1,\dots ,b_L\right)={\prod }_{l=1}^L{q}_l\left(b_l\right)$, it suffices to show that each q_l is exchangeable in j, k; i.e, q_l (ω_jk (b_l)) = q_l (b_l) for all l = 1, …, L. This exchangeability is satisfied after every iteration of the IBSS algorithm because the algorithm computes q_l (parameterized by α_l, μ_1l, ${\sigma }_{1l}^2$) as the exact posterior distribution under an SER model (Step 5 of Algorithm 1), and this posterior is exchangeable in j, k because both the prior and likelihood are exchangeable. □

Because the exchangeability is satisfied after every iteration of IBSS, and not just at convergence, the result is not sensitive to stopping criteria. By contrast, the corresponding EN property (Zou and Hastie, 2005) holds only at convergence — for example, in numerical implementations of the EN method (e.g., the glmnet R package), the coefficient estimates for identical variables can differ substantially. Similarly, MCMC-based implementations of BVSR may satisfy this exchangeability property only asymptotically.

3.1.3. Estimating σ², ${\sigma }_0^2$

Algorithm 1 can be extended to estimate the hyperparameters σ² and ${\sigma }_0^2$ by adding steps to maximize F (q₁, …, q_L; σ², ${\sigma }_0^2$) over σ² and/or ${\sigma }_0^2$. Estimating the hyperparameters by maximizing the ELBO can be viewed as an EM algorithm (Dempster et al., 1977) in which the E-step is approximate (Heskes et al., 2004; Neal and Hinton, 1998).

Optimizing F over σ² involves computing the expected residual sum of squares under the VA, which is straightforward; see Appendix B for details.

Optimizing F over ${\sigma }_0^2=\left({\sigma }_{0l}^2,\dots ,{\sigma }_{0L}^2\right)$ can be achieved by modifying the step that computes the posterior distribution for b_l under the SER model to first estimate the hyperparameter ${\sigma }_{0l}^2$ in the SER model by maximum likelihood; that is, by maximizing the SER likelihood (2.12) over ${\sigma }_0^2$, keeping σ² fixed (Step 5 of Algorithm A3). This is a one-dimensional optimization which is easily performed numerically (we used the R function optim).

Algorithm A3 in Appendix B extends Algorithm 1 to include both these steps.

3.2. Posterior inference: posterior inclusion probabilities and Credible Sets

Algorithm 1 provides an approximation to the posterior distribution of b under the SuSiE model, parameterized by (α₁, μ₁₁, ${\sigma }_{11}^2$), …, (α_L, μ_1L, ${\sigma }_{1L}^2$). From these results it is straightforward to compute approximations to various posterior quantities of interest, including PIPs and CSs.

3.2.1. Posterior inclusion probabilities

Under the SuSiE model, the effect of explanatory variable j is $b^{(j)}≔{\sum }_{l=1}^L{b}_{lj}$, which is zero if and only if b_lj = 0 for all l = 1, …, L. Under our VA the b_lj are independent across l, and therefore

$${\text{PIP}}_j≔\text{Pr}\left(b^{(j)}\ne 0\mid X,y\right)\approx 1-\prod _{l\in ℒ}\left(1-{\alpha }_{lj}\right).$$ (3.12)

Here, we set $ℒ≔\left\{l:{\sigma }_{0l}^2>0\right\}$ to treat the case where some ${\sigma }_{0l}^2$ are zero, which can happen if ${\sigma }_0^2$ is estimated.

3.2.2. Credible Sets

Computing the sets CS(α_l; ρ) (A.4), for l = 1, …, L immediately yields L CSs that satisfy Definition 1 under the VA to the posterior.

If L exceeds the number of detectable effects in the data, then in practice many of the L CSs are large, often containing the majority of variables. The intuition is that once all the detectable signals have been accounted for, the IBSS algorithm becomes very uncertain about which variable to include at each step, and so the distributions α_l become very diffuse. CSs that contain very many uncorrelated variables are of essentially no inferential value — whether or not they contain an effect variable — and so in practice it makes sense to ignore them. To automate this, in this paper we discard CSs with “purity” less than 0.5, where we define purity as the smallest absolute correlation among all pairs of variables within the CS. (To reduce computation for CSs containing over 100 variables, we sampled 100 variables at random to estimate the purity.) The purity threshold of 0.5 was chosen primarily for comparing with Lee et al. (2018), who use a similar threshold in a related context. While any choice of threshold is somewhat arbitrary, in practice we observed that most CSs are either very pure (> 0.95) or very impure (< 0.05), with intermediate cases being rare (Figure S2), so most results are robust to this choice of threshold.

3.3. Choice of L

It may seem that SuSiE would be sensitive to the choice of L. In practice, however, key inferences are often robust to overstating L; for example, in our simulations below, the simulated number of effects was between 1 and 5, whereas we still obtain good results with L = 10. This is because, when L is larger than necessary, the method is very uncertain about where to place the extra effects — consequently, it distributes them broadly among many variables, and therefore they are too diffuse to impact key inferences. For example, setting L to be larger than necessary inflates the PIPs of many variables, but only slightly, and the extra components result in CSs with low purity.

While inferences are generally robust to overstating L, we also note that the Empirical Bayes version of our method, which estimates ${\sigma }_0^2$, also effectively estimates the number of effects: when L is greater than the number of signals in the data, the maximum likelihood estimate of ${\sigma }_{0l}^2$ will be zero or close to zero for many l, which in turn forces b_l to zero. This is closely related to the idea behind “automatic relevance determination” (Neal, 1996).

3.4. Identifiability and label-switching

The parameter vectors b₁, …, b_l introduced in the SuSiE model are technically non-identifiable, in that the likelihood p(y | X, σ², b₁, …, b_l) is unchanged by permutation of the labels 1, …, L. As a result, the posterior distribution p_post is symmetric with respect to permutations of the labels (assuming the prior is also symmetric) — that is, for any permutation, ν : {1, …, L} → {1, …, L}, we have p(b₁, …, b_l | X, y, σ²) = p(b_v(1), …, b_v(L) | X, y, σ²). A similar non-identifiability also occurs in mixture models, where it is known as the “label-switching problem” (Stephens, 2000).

In principle, non-identifiability due to label-switching does not complicate Bayesian inference; the posterior distribution is well-defined, and correctly reflects uncertainty in the parameters. In practice, however, complications can arise. Specifically, the label-switching typically causes the posterior distribution to be multi-modal, with L! symmetric modes corresponding to the L! different labellings (ν above). Care is then needed when summarizing this posterior distribution. For example, the posterior mean will not be a sensible estimate for b₁, …, b_l because it averages over the L! modes (Stephens, 2000).

Fortunately, our variational approximation (Section 3.1.1) avoids these potential complications of label-switching. This is due to the way that variational approximations behave when approximating the posterior distribution of a mixture model; they typically produce a good approximation to one of the permutations, effectively ignoring the others (Wang and Titterington, 2006; Blei et al., 2017; Pati et al., 2018). See also the discussion of “spontaneous symmetry-breaking” in Wainwright and Jordan (2007). Consequently, our posterior approximation q(b₁, …, b_l) approximates just one of the L! symmetric modes of the true posterior, avoiding the issues with label-switching that can occur when summarizing the true posterior distribution.

Formally, this non-identifiability causes the objective F optimized by the IBSS algorithm to be invariant to relabeling — that is,

$$F\left(q_{v(1)},\dots ,q_{v(L)};{\sigma }^2,{\sigma }_{0v(1)}^2,\dots ,{\sigma }_{0v(L)}^2\right)$$

is the same for all permutations ν — and therefore every solution $\widehat{q},\widehat{\sigma },{\widehat{\sigma }}_0^2$ returned by our IBSS algorithm has L! equivalent solutions that achieve the same value of the ELBO, F, each corresponding to a different labeling (and a different mode of the true posterior). These L! solutions are inferentially equivalent; they all imply the same distribution for the unordered set {b₁, …, b_l}, the same distribution for the sum $b={\sum }_{l=1}^L{b}_l$ (which does not depend on the labeling), and they all produce the same PIPs and the same CSs. Thus, it does not matter which mode is used.

4. NUMERICAL COMPARISONS

We performed numerical comparisons on data generated to closely mimic our main motivating application: genetic fine-mapping. Specifically, we generated data for fine-mapping of expression quantitative trait loci (eQTLs), which are genetic variants associated with gene expression. We used these simulations to assess our methods, and compare with state-of-the-art BVSR methods that were specifically developed for this problem. We also compared against a (frequentist) hierarchical testing method (Mandozzi and Bühlmann, 2016; Renaux et al., 2020).

In genetic fine-mapping, X is a matrix of genotype data, in which each row corresponds to an individual, and each column corresponds to a genetic variant, typically a single nucleotide polymorphism (SNP). In our simulations, we used the real human genotype data from n = 574 genotype samples collected as part of the Genotype-Tissue Expression (GTEx) project (GTEx Consortium, 2017). To simulate fine-mapping of locally-acting variants associated with gene expression (cis eQTLs), we randomly selected 150 genes out of the > 20, 000 genes on chromosomes 1–22, then assigned X to be the genotypes for genetic variants nearby the transcribed region of the selected gene. For a given gene, between p = 1,000 and p = 12,000 SNPs were included in the fine-mapping analysis; for more details on how SNPs were selected, see Appendix D.

These real genotype matrices, X, exhibit complex patterns of correlations; see Figure A1 for example. Furthermore, many variables are strongly correlated with other variables: for a randomly chosen variable, the median number of other variables with which its correlation exceeds 0.9 is 8; and the median number of other variables with which its correlation exceeds 0.99 is 1. Corresponding means are even larger — 26 and 8 other variables, respectively — because some variables are strongly correlated with hundreds of other variables. Thus these genotype matrices lead to challenging, but realistic, variable selection problems.

We generated synthetic outcomes y under the multiple regression model (2.1), with assumptions on b specified by two parameters: S, the number of effect variables; and ϕ, the proportion of variance in y explained by X (abbreviated as “PVE”). Given S and ϕ, we simulated b and y as follows:

1. Sample the indices of the S effect variables, 𝒮, uniformly at random from {1, …, p}.

2. For each j ∈ 𝒮, independently draw b_j ~ N (0, 0.6²), and for all j ∉ 𝒮, set b_j = 0.

3. Set σ² to achieve the desired PVE, ϕ; specifically, we solve for σ² in $\varphi =\frac{\text{Var}(Xb)}{{\sigma }^2+\text{Var}(Xb)}$, where Var(·) denotes sample variance.

4. For each i = 1, …, n, draw y_i ~ N (x_i1b₁ + ⋯ + x_ip b_p, σ²).

We generated data sets under two simulation scenarios. In the first scenario, each data set has p = 1, 000 SNPs. We generated data sets using all pairwise combinations of S ∈ {1, …, 5} and ϕ ∈ {0.05, 0.1, 0.2, 0.4}. These settings were chosen to span typical expected values for eQTL studies. We simulated two replicates for each gene and for each combination of S and ϕ. Therefore, in total we generated 2 × 150 × 5 × 4 = 6,000 data sets for the first simulation scenario.

In the second simulation scenario, we generated data sets with all cis SNPs (defined as SNPs within 1 Megabase radius from transcription start site of a gene), ranging from 3,000 to 12,000 SNPs, and to generate the outcomes y, we set S = 10 and ϕ = 0.3. We generated two replicates for each gene, resulting in a total of 2 × 150 = 300 data sets in the second simulation scenario.

4.1. Illustrative example

We begin with an example to illustrate that the IBSS algorithm (Algorithm 1) can perform well in a challenging fine-mapping setting. This example is summarized in Figure 1.

FIGURE 1. Fine-mapping example to illustrate that IBSS algorithm can deal with a challenging case.

Results are from a simulated data set with p = 1, 000 variables (SNPs). Some of these variables are very strongly correlated (Figure A1). Two out of the 1,000 variables are effect variables (red points, labeled “SNP 1” and “SNP 2” in the left-hand panel). We chose this example from our simulations because the strongest marginal association (SMA) is a non-effect variable (yellow point, labeled “SMA” in the left-hand panel). After 1 iteration (middle panel), IBSS incorrectly identifies a CS containing the SMA and no effect variable (orange points). However, after 10 iterations (and also at convergence) the IBSS algorithm has corrected itself (right-panel), finding two 95% CSs — dark blue and green open circles — each containing a true effect variable. Additionally, neither CS contains the SMA variable. One CS (blue) contains only 3 SNPs (purity of 0.85), whereas the other CS (green) contains 37 very highly correlated variables (purity of 0.97). In the latter CS, the individual PIPs are small, but the inclusion of the 37 variables in this CS indicates, correctly, high confidence in at least one effect variable among them.

We draw this example from one of our simulations in which the variable with the strongest marginal association (SMA) with y is not one of the actual effect variables (in this example, there are two effect variables). This situation occurs because the SMA variable has moderate correlation with both effect variables, and these effects combine to make its marginal association stronger than the marginal associations of the individual effect variables. Standard forward selection in this case would select the wrong (SMA) variable in the first step; indeed, after one iteration, IBSS also yields a CS that includes the SMA variable (Figure 1, middle panel). However, as the IBSS algorithm proceeds, it recognizes that, once other variables are accounted for, the SMA variable is no longer required. After 10 iterations (at which point the IBSS solution is close to convergence) IBSS yields two high-purity CSs, neither containing the SMA, and each containing one of the effect variables (Figure 1, right panel). Our manuscript resource repository includes an animation showing the iteration-by-iteration progress of the IBSS algorithm (Wang et al., 2020a).

This example, where the SMA variable does not appear in a CS, also illustrates that multiple regression can sometimes result in very different conclusions than a marginal association analysis.

4.2. Posterior inclusion probabilities

Next, we seek to assess the effectiveness of our methods more quantitatively. We focus initially on one of the simpler tasks in BVSR: computing posterior inclusion probabilities (PIPs). Most implementations of BVSR compute PIPs, making it possible to compare results across several implementations. Here we compare our methods (henceforth SuSiE, implemented in R package susieR, version 0.4.29) with three other software implementations specifically developed for genetic fine-mapping applications: CAVIAR (Hormozdiari et al., 2014, version 2.2), FINEMAP (Benner et al., 2016, version 1.1) and DAP-G (Wen et al., 2016; Lee et al., 2018, installed using source code from the git repository, commit id ef11b26). These methods are all implemented as C++ programs. They implement similar BVSR models, and differ in the algorithms used to fit these models and the priors on the effect sizes. CAVIAR exhaustively evaluates all possible combinations of up to L non-zero effects among the p variables. FINEMAP and DAP-G approximate this exhaustive approach by heuristics that target the best combinations. Another important difference among methods is that FINEMAP and CAVIAR perform inference using summary statistics computed from each data set — specifically, the marginal association Z scores and the p × p correlation matrix for all variables — whereas, as we apply them here, DAP-G and SuSiE use the full data. The summary statistic approach can be viewed as approximating inferences from the full data; see Lee et al. (2018) for discussion.

For SuSiE, we set L = 10 for all data sets generated in the first simulation scenario, and L = 20 for the second scenario. We assessed performance when both estimating the hyperparameters σ², ${\sigma }_0^2$, and when fixing one or both of these hyperparameters. Overall performance of these different approaches were similar, and here we show results when σ² was estimated, and when ${\sigma }_{0l}^2$ was fixed to 0.1Var(y) (consistent with data applications in Section 5); other results are in Supplementary Data (Figure S4 and Figure S5). Parameter settings for other methods are given in Appendix D. We ran CAVIAR and FINEMAP only on simulations with S ⩽ 3 since these methods are computationally more intensive than the others (particularly for larger S).

Since these methods differ in their modelling assumptions, one should not expect their PIPs to be equal. Nonetheless, we found generally reasonably good agreement (Figure 2A). For S = 1, the PIPs from all four methods agree closely. For S > 1, the PIPs from different methods are also highly correlated; correlations between PIPs from SuSiE and other methods vary from 0.94 to 1 across individual data sets, and the number of PIPs differing by more than 0.1 is always small — the proportions vary from 0.013% to 0.2%. In the scatterplots, this agreement appears less strong because the eye is drawn to the small proportion of points that lie away from the diagonal, but the vast majority of points lie on or near the origin. In addition, all four methods produce reasonably well-calibrated PIPs (Figure S1).

FIGURE 2. Evaluation of posterior inclusion probabilities (PIPs).

Scatterplots in Panel A compare PIPs computed by SuSiE against PIPs computed using other methods (DAP-G, CAVIAR, FINEMAP). Each point depicts a single variable in one of the simulations: dark red points represent true effect variables, whereas light gray points represent variables with no effect. The scatterplot in Panel B combine results across the first set of simulations. Panel B summarizes power versus FDR from the first simulation scenario of. These curves are obtained by independently varying the PIP threshold for each method. The open circles in the left-hand plot highlight power versus FDR at PIP thresholds of 0.9 and 0.95). These quantities are calculated as $\text{FDR}≔\frac{\text{FP}}{\text{TP}+\text{FP}}$ (also known as the “false discovery proportion”) and $\text{ power }≔\frac{\text{TP}}{\text{TP}+\text{FN}}$, where FP, TP, FN and TN denote the number of False Positives, True Positives, False Negatives and True Negatives, respectively. (This plot is the same as a precision-recall curve after reversing the x-axis, because $\text{ precision }=\frac{\text{TP}}{\text{TP}+\text{FP}}=1-\text{FDR}$, and recall = power.) Note that CAVIAR and FINEMAP were run only on data sets with 1 − 3 effect variables.

The general agreement of PIPs from four different methods suggests that: (i) all four methods are mostly accurate for computing PIPs for the data set sizes explored in our numerical comparisons; and (ii) the PIPs themselves are usually robust to details of the modelling assumptions. Nonetheless, some non-trivial differences in PIPs are clearly visible from Figure 2A. Visual inspection of these differences suggests that the SuSiE PIPs may better distinguish effect variables from non-effect variables, in that there appears a higher ratio of red-gray points below the diagonal than above the diagonal. This is confirmed in our analysis of power versus False Discovery Rate (FDR), obtained by varying the PIP threshold independently for each method; at a given FDR, the SuSiE PIPs always yield higher power (Figure 2B).

Notably, even though SuSiE is implemented in R, its computations are much faster than the other methods implemented in C++: for example, in the data sets simulated with S = 3, SuSiE is, on average, roughly 4 times faster than DAP-G, 30 times faster than FINEMAP, and 4,000 times faster than CAVIAR (Table 1).

TABLE 1:

Runtimes from data sets simulated with S = 3 (all runtimes are in seconds)

method	mean	min.	max.
SuSiE	0.64	0.34	2.28
DAP-G	2.87	2.23	8.87
FINEMAP	23.01	10.99	48.16
CAVIAR	2907.51	2637.34	3018.52

Because SuSiE computations scale linearly with data size (computational complexity O(npL) per iteration) it can easily handle data sets much larger than the ones in these simulations. To illustrate, running SuSiE (L = 10) on two larger simulated data sets — one with n = 100, 000, p = 500; another with n = 1, 000, p = 50, 000; each with 4 effect variables — took 25s and 43s on a modern Linux workstation (see Appendix D.2 for details). This is competitive with lasso, implemented in the glmnet R package, version 2.0.18, which with 10-fold cross-validation (and other parameters at their defaults) took 82s for each data set.

In summary, in the settings considered here, SuSiE produces PIPs that are as or more reliable than existing BVSR methods, and does so at a fraction of the computational effort.

4.3. Credible Sets

Comparison with DAP-G

A key feature of SuSiE is that it yields multiple Credible Sets (CSs), each aimed at capturing an effect variable (Definition 1). The only other BVSR method that attempts something similar, as far as we are aware, is DAP-G, which outputs signal clusters defined by heuristic rules (Lee et al., 2018). Although the authors do not refer to their signal clusters as CSs, and they do not give a formal definition of signal cluster, the intent of these signal clusters is similar to our CSs, and so for brevity we henceforth refer to them as CSs.

We compared the level 95% CSs produced by SuSiE and DAP-G in several ways. First we assessed their empirical (frequentist) coverage levels; that is, the proportion of CSs that contain an effect variable. Since our CSs are Bayesian Credible Sets, 95% CSs are not designed, or guaranteed, to have frequentist coverage of 0.95 (Fraser, 2011). Indeed, coverage will inevitably depend on simulation scenario; for example, in completely null simulations, in which the data are simulated with b = 0, every CS would necessarily contain no effect variable, and so the coverage would be zero Nonetheless, under reasonable circumstances that include effect variables, one might hope that the Bayesian CSs would have coverage near the nominal levels. And, indeed, we confirmed this was the case: in the simulations, CSs from both methods typically had coverage slightly below 0.95, and in most cases above 0.90 (Figure 3; see Figure S3 for additional results).

FIGURE 3. Comparison of 95% credible sets (CS) from SuSiE and DAP-G.

Panels show A) coverage, B) power, C) median size and D) average squared correlation of the variables in each CS. These statistics are taken as mean over all CSs computed in all data sets; error bars in Panel A show 2 × standard error. Simulations with 1–5 effect variables are from the first simulation scenario, and simulations with 10 effect variables are from the second scenario.

Having established that the methods produce CSs with similar coverage, we compared them by three other criteria: (i) power (overall proportion of simulated effect variables included in a CS); (ii) average size (median number of variables included in a CS) and (iii) purity (here, measured as the average squared correlation of variables in a CS since this statistic is provided by DAP-G). By all three metrics, the CSs from SuSiE are consistently an improvement over DAP-G — they achieve higher power, smaller size, and higher purity (Figure 3).

Although the way that we construct CSs in SuSiE does not require that they be disjoint, we note that the CSs rarely overlapped (after filtering out low purity CSs; see Section 3.2.2). Indeed, across the thousands of simulations, there was only one example of two CSs overlapping.

Comparison with hierarchical testing

Finally, we compared our CSs with results produced by the R package hierinf (Renaux et al., 2020) (version 1.3.1), which implements a frequentist approach to identifying significant clusters of variables based on hierarchical testing (Meinshausen, 2008; Mandozzi and Bühlmann, 2016). In brief, this approach starts by assuming that the variables are organized in a given hierarchy. Then, starting from the top of the hierarchy, it proceeds to test whether groups of variables (clades in the hierarchy) contain at least one non-zero effect. Each time a group is deemed significant, the method proceeds to test clades in the next level of the hierarchy. The procedure ultimately reports the smallest significant clades detected, where the significance criteria are designed to control the overall family-wise error rate (FWER) at a pre-specified level, α We note that FWER control is not guaranteed when p > n and variables are highly correlated (Mandozzi and Bühlmann, 2016), which is the situation in our simulations.

Although the theory for controlling FWER in hierarchical testing is elegant, genetic variants do not come in a natural hierarchy, and so for fine-mapping the need to specify a hierarchy is a drawback. Here we use the cluster_var function from hierinf, which infers a hierarchical clustering. There is no simple correspondence between the level α and (frequentist) coverage rates of the significant clusters, so selecting a suitable α is non-trivial; in our simulations, we found that empirical coverage was typically close to 0.95 when α = 0.1, so we report results for α = 0.1.

The results (Table 2) show that the hierinf clusters are substantially larger, and have lower purity than the CSs from SuSiE, as well as DAP-G. For example, in simulations with 5 effect variables, the SuSiE CSs have a median size of 7 variables with an average r² of 0.97, whereas the hierinf clusters have a median size of 54 variables with an average r² of 0.56. Further, SuSiE and DAP-G achieved greater power — that is, they identified more credible sets containing true signals — than the significant clusters from hierinf.

TABLE 2:

Comparison of CSs from SuSiE and DAP-G to significant clusters from hierarchical inference (hierinf software, with FWER level α = 0.1). Results are averages across all data sets in the first simulation scenario.

#effects	power			coverage			median size			average r2
	SuSiE	DAP-G	hierinf	SuSiE	DAP-G	hierinf	SuSiE	DAP-G	hierinf	SuSiE	DAP-G	hierinf
1	0.99	0.89	0.97	0.98	0.99	0.94	3	3	8	0.99	0.99	0.82
2	0.67	0.60	0.55	0.95	0.92	0.96	4	5	20	0.99	0.97	0.71
3	0.52	0.49	0.39	0.93	0.91	0.95	6	7	34	0.98	0.96	0.64
4	0.45	0.40	0.29	0.92	0.89	0.95	6	8	37	0.98	0.95	0.60
5	0.37	0.32	0.24	0.90	0.87	0.98	7	9	54	0.97	0.95	0.56

We believe that the much larger number of variables included in the hierinf clusters partly reflects a fundamental limitation of the hierarchical approach to this problem. Specifically, by assuming a hierarchy that does not truly exist, the method artificially limits the clusters of variables it can report. This will sometimes force it to report clusters that are larger than necessary. For example, with 3 variables, if variables 2 and 3 are grouped together at the bottom of the hierarchy, then the method could never report a cluster {1, 2}, representing the statement “either variable 1 or 2 is an effect variable, but we cannot tell which,” even if the data support such an inference. Instead, it would have to report the larger cluster, {1, 2, 3}.

While our work here was under peer review, and available as a preprint (Wang et al., 2019), we became aware of new related work in a preprint by Sesia et al. (2020). Similar to hierinf this new method tests groups of variables at multiple resolutions in a hierarchy; but it improves on hierinf by controlling the false discovery rate of selected groups (rather than type I error), and with statistical guarantees that hold even in the presence of highly correlated variables. Comparisons with our method find that their significant groups are typically larger than ours (Sesia et al., 2020, Figure 4), presumably in part due to the fundamental limitation with the hierarchical approach (discussed above).

FIGURE 4. Illustration of SuSiE applied to two change point problems.

The top panel shows a simulated example with seven change points (the vertical black lines). The blue horizontal lines show the mean function inferred by the segment method from the DNAcopy R package (version 1.56.0). The inference is reasonably accurate — all change points except the left-most one are nearly exactly recovered — but provides no indication of uncertainty in the locations of the change points. The red regions depict the 95% CSs for change point locations inferred by SuSiE; in this example, every CS contains a true change point. The bottom panel shows a simulated example with two change points in quick succession. This example is intended to illustrate convergence of the IBSS algorithm to a (poor) local optimum. The black line shows the fit from the IBSS algorithm when it is initialized to a null model in which there are no change points; this fit results in no change points being detected. The red line also shows the result of running IBSS, but this time the fitting algorithm is initialized to the true model with two change points. The latter accurately recovers both change points, and attains a higher value of the objective function (−148.2 versus −181.8).

5. APPLICATION TO FINE-MAPPING SPLICING QTLS

To illustrate SuSiE for a real fine-mapping problem, we analyzed data from Li et al. (2016) aimed at detecting genetic variants (SNPs) that influence splicing (known as “splicing QTLs”, sQTLs). These authors quantified alternative splicing by estimating, at each intron in each sample, a ratio capturing how often the intron is used relative to other introns in the same “cluster” (roughly, gene). The data involve 77,345 intron ratios measured on lymphoblastoid cell lines from 87 Yoruban individuals, together with genotypes of these individuals. Following Li et al. (2016), we preprocessed the intron ratios by regressing out the first 3 principle components of the matrix of intron ratios; the intent is to control for unmeasured confounders (Leek and Storey, 2007). For each intron ratio, we fine-mapped SNPs within 100 kb of the intron, which is approximately 600 SNPs on average. In short, we ran SuSiE on 77,345 data sets with n = 87 and p ≈ 600.

To specify the prior variance ${\sigma }_{0l}^2$, we first estimated typical effect sizes from the data on all introns. Specifically, we performed univariate (SNP-by-SNP) regression analysis at every intron, and estimated the PVE of the top (strongest associated) SNP. The mean PVE of the top SNP across all introns was 0.096, so we applied SuSiE with ${\sigma }_{0l}^2=0.096\text{Var}(y)$, and with the columns of X standardized to have unit variance. The residual variance parameter σ² was estimated by IBSS.

We then ran SuSiE to fine-map sQTLs at all 77,345 introns. After filtering for purity, this yielded a total of 2,652 CSs (level 0.95) spread across 2,496 intron units. These numbers are broadly in line with the original study, which reported 2,893 significant introns at 10% FDR. Of the 2,652 CSs identified, 457 contain exactly one SNP, representing strong candidates for being the causal variants that affect splicing. Another 239 CSs contain exactly two SNPs. The median size of a CS was 7, and the median purity was 0.94.

The vast majority of intron units with a CS had exactly one CS (2,357 of 2,496). Thus, SuSiE could detect at most one sQTL for most introns. Of the remainder, 129 introns yielded 2 CSs, 5 introns yielded 3 CSs, 3 introns yielded 4 CSs, and 2 introns yielded 5 CSs. This represents a total of 129 + 10 + 9 + 8 = 156 additional (“secondary”) signals that would be missed in conventional analyses that report only one signal per intron. Both primary and secondary signals were enriched in regulatory regions (Appendix E), lending some independent support that SuSiE is detecting real signals. Although these data show relatively few secondary signals, this is a small study (n = 87); in larger studies, the ability of SuSiE to detect secondary signals will likely be greater.

6. AN EXAMPLE BEYOND FINE-MAPPING: CHANGE POINT DETECTION

Although our methods were motivated by genetic fine-mapping, they are also applicable to other sparse regression problems. Here we apply SuSiE to an example quite different from fine-mapping: change point detection. This application also demonstrates that the IBSS algorithm can sometimes produce a poor fit — due to getting stuck in a local optimum — which was seldom observed in our fine-mapping simulations. We believe that examples where algorithms fail are just as important as examples where they succeed — perhaps more so — and that this example could motivate improvements.

We consider a simple change point model

$$y_t={\mu }_t+e_t,t=1,\dots ,T,$$ (6.1)

where t indexes a dimension such as space or time, and the errors e_t are independently normal with zero mean and variance σ². The mean vector μ ≔ (μ₁, …, μ_T) is assumed to be piecewise constant; the indices t where changes to μ occur, μ_t ≠ μ_t+1, are called the “change points.”

To capture change points being rare, we formulate the change point model as a sparse multiple regression (2.1) in which X hasT − 1 columns, and the tth column is a step function with a step at location t; that is, x_st = 0 for s ⩽ t, and x_st = 1 for all s > t). The tth element of b then determines the change in the mean at position t, μ_t+1 − μ_t. Therefore, the non-zero regression coefficients in this multiple regression model correspond to change points in μ.

The design matrix X in this setting has a very special structure, and quite different from fine-mapping applications; the (T − 1) × (T – 1) correlation matrix decays systematically and very slowly away from the diagonal. By exploiting this special structure of X, SuSiE computations can be made O(TL) rather than the O(T²L) of a naive implementation; for example, the matrix-vector product X^T y, naively an O(T²) computation, can be computed as the cumulative sum of the elements of the reverse of y, which is an O(T) computation.

Change point detection has a wide range of potential applications, such as segmentation of genomes into regions with different numbers of copies of the genome. Software packages in R that can be used for detecting change points include changepoint (Killick and Eckley, 2014), DNAcopy (Seshan and Olshen, 2018; Olshen et al., 2004), bcp (Erdman and Emerson, 2007) and genlasso (Tibshirani, 2014; Arnold and Tibshirani, 2016); see Killick and Eckley (2014) for a longer list. Of these, only bcp, which implements a Bayesian method, quantifies uncertainty in estimated change point locations, and bcp provides only PIPs, not CSs for change point locations. Therefore, the ability of SuSiE to provide CSs is unusual, and perhaps unique, among existing change point detection methods.

To illustrate its potential for change point estimation, we applied SuSiE to a simulated example included with the DNAcopy R package. In this example, all settings for running SuSiE remain unchanged from the fine-mapping simulations (Section 4). The top label of Figure 4 shows results of applying SuSiE and DNAcopy to the data set. Both methods provide accurate estimates of the change points; indeed all change point locations except the left-most one are recovered nearly exactly. However, only SuSiE provides 95% CSs for each estimate of a change point location. And, indeed, SuSiE is most uncertain about the left-most change point. All the true change points in this example are contained in a SuSiE CS, and every CS contains a true change point. This occurs even though we set L = 10 to be greater than the number of true change points (7); the three extra CSs were filtered out because they contained variables that were very uncorrelated. (To be precise, SuSiE reported 8 CSs after filtering, but two of the CSs overlapped and contained the same change point; this observation of overlapping of CSs contrasts with the fine-mapping simulations in Section 4 where overlapping CSs occurred very rarely.)

To demonstrate that IBSS can converge to a poor local optimum, consider the simulated example shown in the bottom panel of Figure 4, which consists of two change points in quick succession that cancel each other out (the means before and after the change points are the same). This example was created specifically to illustrate a limitation of the IBSS procedure: IBSS can only introduce or update one change point at a time, and every update is guaranteed to increase the objective, whereas in this example introducing one change point will make the fit worse. Consequently, when SuSiE is run from a null initialization, IBSS finds no change points, and reports no CSs.

This poor outcome represents a limitation of the IBSS algorithm, not a limitation of the SuSiE model or the variational approximation. To show this, we re-ran the IBSS algorithm, but initializing at a solution that contained the two true change points. This yielded a fit with two CSs, each containing the one of the correct change points. This also resulted in a much improved value of the objective function (−148.2 versus −181.8). Better algorithms for fitting SuSiE models, or more careful initializations of IBSS, will be needed to address this shortcoming,

7. DISCUSSION

We have presented a simple new approach to variable selection in regression. Compared with existing methods, the main benefits of our approach are its computational efficiency, and its ability to provide CSs summarizing uncertainty in which variables should be selected. Our numerical comparisons demonstrate that for genetic fine-mapping our methods outperform existing methods at a fraction of the computational cost.

Although our methods apply generally to variable selection in linear regression, further work may be required to improve performance in difficult settings. In particular, while the IBSS algorithm worked well in our fine-mapping experiments, for change point problems we showed that IBSS may converge to poor local optima. We have also seen convergence problems in experiments with many effect variables (e.g. 200 effect variables out of 1,000). Such problems may be alleviated by better initialization, for example using fits from convex objective functions (e.g., lasso) or from more sophisticated algorithms for non-convex problems (Bertsimas et al., 2016; Hazimeh and Mazumder, 2018). More ambitiously, one could attempt to develop better algorithms to reliably optimize the SuSiE variational objective function in difficult cases. For example, taking smaller steps each iteration, rather than full coordinate ascent, may help.

At its core, the SuSiE model is based on adding up simple models (SERs) to create more flexible models (sparse multiple regression). This additive structure is the key to our variational approximations, and indeed our methods apply generally to adding up any simple models for which exact Bayesian calculations are tractable, not only SER models (Appendix B; Algorithm A2). These observations suggest connections with both additive models and boosting (e.g., Friedman et al., 2000; Freund et al., 2017). However, our methods differ from most work on boosting in that each “weak learner” (here, SER model) itself yields a model-averaged predictor. Other differences include our use of backfitting, the potential to estimate hyper-parameters by maximizing an objective function rather than cross-validation, and the interpretation of our algorithm as a variational approximation to a Bayesian posterior. Although we did not focus on prediction accuracy here, the generally good predictive performance of methods based on model averaging and boosting suggest that SuSiE should work well for prediction as well as variable selection.

It would be natural to extend our methods to generalized linear models (GLMs), particularly logistic regression. In genetic studies with small effects, Gaussian models are often adequate to model binary outcomes (e.g. Pirinen et al., 2013; Zhou et al., 2013). However, in other settings this extension may be more important. One strategy would be to directly modify the IBSS algorithm, replacing the SER fitting procedure with a logistic or GLM equivalent. This strategy is appealing in its simplicity, although it is not obvious what objective function the resulting algorithm is optimizing. Alternatively, for logistic regression one could use the variational approximations developed by Jaakkola and Jordan (2000).

For genetic fine-mapping, it would also be useful to modify our methods to deal with settings where only summary data are available (e.g. the p univariate regression results). Many recent fine-mapping methods deal with this (e.g., Chen et al., 2015; Benner et al., 2016; Newcombe et al., 2016) and ideas used by these methods can also be applied to SuSiE. Indeed, our software already includes preliminary implementations for this problem.

Beyond genetic fine-mapping, one could consider applying SuSiE to related tasks, such as genetic prediction of complex traits and heritability estimation (Yang et al., 2011). However, we do not expect SuSiE to provide substantial improvements over existing methods for these tasks. This is because, in general, the best existing approaches to these problems do not make strict sparsity assumptions on the effect variables; they allow for models in which many (or all) genetic variants affect the outcome (Meuwissen et al., 2001; Moser et al., 2015; Speed and Balding, 2014; Vilhjálmsson et al., 2015; Zhou et al., 2013). Nonetheless, it is possible that the ideas introduced here for sparse modelling could be combined with existing methods allowing non-sparse effects to improve prediction and heritability estimation, similar to Zhou et al. (2013).

Finally, we are particularly interested in extending these methods to select variables simultaneously for multiple outcomes (multivariate regression and multi-task learning). Joint analysis of multiple outcomes should greatly enhance power and precision to identify relevant variables (e.g., Stephens, 2013). The computational simplicity of our approach makes it particularly appealing for this complex task, and we are currently pursuing this direction by combining our methods with those from Urbut et al. (2019).

Appendices

A. DETAILS OF POSTERIOR COMPUTATIONS FOR THE SER MODEL

A.1. Bayesian simple linear regression

To derive posterior computations for the SER model (2.4–2.8), it helps to start with an even simpler (univariate) linear regression model:

$$y=xb+e$$

$$\mathit{e}~N_n\left(0,{\sigma }^2{I}_n\right)$$

$$b~N_1\left(0,{\sigma }_0^2\right).$$

Here, y is an n-vector of response data (centered to have mean zero), x is an n-vector containing values of a single explanatory variable (similarly centered), e is an n-vector of independent error terms with variance σ², b is the scalar regression coefficient to be estimated, ${\sigma }_0^2$ is the prior variance of b, and I_n is the n × n identity matrix.

Given σ² and ${\sigma }_0^2$, the posterior computations for this model are very simple; they can be conveniently written in terms of the usual least-squares estimate of b, $\widehat{b}≔{\left(x^{\top }x\right)}^{-1}{x}^{\top }y$, its variance ${\mathit{s}}^2≔\frac{{\sigma }^2}{{\mathit{x}}^{\top }x}$, and the corresponding z score, $z≔\widehat{b}/\mathit{s}$. The posterior distribution for b is

$$b\mid y,{\sigma }^2,{\sigma }_0^2~N_1\left({\mu }_1,{\sigma }_1^2\right),$$

where

$${\sigma }_1^2\left(x,y;{\sigma }^2,{\sigma }_0^2\right)≔\frac{1}{1/{\mathit{s}}^2+1/{\sigma }_0^2}$$ (A.1)

$${\mu }_1\left(x,y;{\sigma }^2,{\sigma }_0^2\right)≔\frac{{\sigma }_1^2}{{\mathit{s}}^2}\times \widehat{b},$$ (A.2)

and the Bayes Factor (BF) for comparing this model with the null model (b = 0) is

$$\begin{gathered} \text{BF}\left(x,y;{\sigma }^2,{\sigma }_0^2\right)≔\frac{p\left(y\mid x,{\sigma }^2,{\sigma }_0^2\right)}{p\left(y\mid x;{\sigma }^2,b=0\right)} \\ =\sqrt{\frac{{\mathit{s}}^2}{{\sigma }_0^2+{\mathit{s}}^2}}\text{\hspace{0.17em}exp}\left(\frac{z^2}{2}\times \frac{{\sigma }_0^2}{{\sigma }_0^2+{\mathit{s}}^2}\right). \end{gathered}$$ (A.3)

This expression matches the “asymptotic BF” of Wakefield (2009), but here, because we consider linear regression given σ², it is an exact expression for the BF, not just asymptotic.

A.2. The single effect regression model

Under the SER model (2.4–2.8), the posterior distribution of (b₁, …, b_p) = (bγ₁, …, bγ_p) conditioned on σ², ${\sigma }_0^2$, π is given in the main text (eqs. 2.9 and 2.10), and is reproduced here for convenience:

$$\gamma \mid X,y,{\sigma }^2,{\sigma }_0^2~\text{Mult}(1,\alpha )$$

$$b\mid X,y,{\sigma }^2,{\sigma }_0^2,{\gamma }_j=1~N_1\left({\mu }_{1j},{\sigma }_{1j}^2\right),$$

where the vector of posterior inclusion probabilities (PIPs), α = (α₁, …α_p), can be expressed in terms of the simple linear regression BFs (A.3),

$${\alpha }_j=\text{Pr}\left({\gamma }_j=1\mid X,y,{\sigma }^2,{\sigma }_0^2\right)=\frac{{\pi }_j\text{BF}\left(x_j,y;{\sigma }^2,{\sigma }_0^2\right)}{{\sum }_{j^{\prime }=1}^p{\pi }_{j^{\prime }}\text{BF}\left(x_{j^{\prime }},y;{\sigma }^2,{\sigma }_0^2\right)},$$

where μ_1j and ${\sigma }_{1j}^2$ are the posterior mean (A.2) and variance (A.1) from the simple regression model of y on x_j :

$${\mu }_{1j}={\mu }_1\left(x_j,y;{\sigma }^2,{\sigma }_0^2\right)$$

$${\sigma }_{1j}={\sigma }_1\left(x_j,y;{\sigma }^2,{\sigma }_0^2\right).$$

Our algorithm requires the first and second moments of this posterior distribution, which are

$$\text{E}\left[b_j\mid X,y,{\sigma }^2,{\sigma }_0^2\right]={\alpha }_j{\mu }_{1j}$$

$$\text{E}\left[b_j^2\mid X,y,{\sigma }^2,{\sigma }_0^2\right]={\alpha }_j\left({\sigma }_{1j}^2+{\mu }_{1j}^2\right).$$

A.3. Computing Credible Sets

As noted in the main text, under the SER model it is straightforward to compute a level-ρ CS (Definition 1), CS(α; ρ). The procedure is given in Maller et al. (2012), and for convenience we describe it here as well.

Given, let r = (r₁, …, r_p) denote the indices of the variables ranked in order of decreasing α_j, so that ${\alpha }_{r_1}>{\alpha }_{r_2}>\cdots >{\alpha }_{r_p}$, and let S_k denote the cumulative sum of the k largest PIPs:

$$S_k≔\sum _{j=1}^k{\alpha }_{r_j}.$$

Now take

$$CS(\alpha ;\rho )≔\left\{r_1,\dots ,r_{k_0}\right\},$$ (A.4)

where k₀ = mint{k : S_k ⩾ ρ}. This choice of k₀ ensures that the CS is as small as possible while satisfying the requirement that it is a level-ρ CS.

A.4. Estimating hyperparameters

As noted in the main text, it is possible to take an empirical Bayes approach to estimating the hyperparameters σ², ${\sigma }_0^2$. The likelihood is

$${\ell }_{\text{SER}}\left(y;{\sigma }_0^2,{\sigma }^2\right)≔p\left(y\mid X,{\sigma }_0^2,{\sigma }^2\right)=p_0\left(y\mid {\sigma }^2\right)\sum _{j=1}^p{\pi }_j\text{BF}\left(x_j,y;{\sigma }^2,{\sigma }_0^2\right),$$ (A.5)

where p₀ denotes the distribution of y under the “null” that b = 0 (i.e. N_n(0, σ²I_n)), and BF(x, y; σ², ${\sigma }_0^2$) is given in eq. A.3. The likelihood (A.5) can be maximized over one or both parameters using available numerical algorithms.

B. DERIVATION OF VARIATIONAL ALGORITHMS

B.1. Background: Empirical Bayes and variational approximation

Here we introduce some notation and elementary results which are later applied to our specific application.

B.1.1. Empirical Bayes as a single optimization problem

Consider the following generic model:

$$y~p(y\mid b,\theta )$$

$$b~g(b),$$

where y represents a vector of observed data, b represents a vector of unobserved (latent) variables of interest, g ∈ 𝒢 represents a prior distribution for b (which in the empirical Bayes paradigm is treated as an unknown to be estimated), and θ ∈ ⊝ represents an additional set of parameters to be estimated. This formulation also includes as a special case situations where g is pre-specified rather than estimated simply by making 𝒢 contain a single distribution.

Fitting this model by empirical Bayes typically involves the following two steps:

1. Obtain estimates $(\widehat{g},\widehat{\theta })$ of (g, θ) by maximizing the log-likelihood:

   $$(\widehat{g},\widehat{\theta })≔\underset{g\in 𝒢,\theta \in \Theta }{\text{argmax}}\ell (g,\theta ;y),$$

   where

   $$\ell (y;g,\theta )≔\text{log\hspace{0.17em}}\int p(y\mid b,\theta )g(b)db.$$
2. Given these estimates, $\widehat{g}$ and $\widehat{\theta }$, compute the posterior distribution for b,

   $${\widehat{p}}_{\text{post }}(b)≔p_{\text{post }}(b;y,g,\theta )=p(b\mid y,g,\theta )\propto p(y\mid b,\theta )g(b).$$

This two-step procedure can be conveniently expressed as a single optimization problem:

$$\left({\widehat{p}}_{\text{post }},\widehat{g},\widehat{\theta }\right)=\underset{g\in 𝒢,\theta \in \Theta ,q}{\text{argmax}}F(q,g,\theta ;y),$$ (B.1)

with

$$F(q,g,\theta ;y)≔\ell (g,\theta ;y)-D_{\text{KL}}\left(q\Vert {\widehat{p}}_{\text{post}}\right),$$ (B.2)

and where

$$D_{\text{KL}}(q\Vert p)≔\int q(b)\text{\hspace{0.17em}log}\frac{q(b)}{p(b)}db$$

is the Kullback-Leibler (KL) divergence from q to p, and the optimization of q in (B.1) is over all possible distributions on b. The function F (B.2) is often called the “evidence lower bound”, or ELBO, because it is a lower bound for the “evidence” (the marginal log-likelihood). (This follows from the fact that KL divergence is always non-negative.)

This optimization problem (B.1) is equivalent to the usual two-step EB procedure. This equivalence follows from two observations:

1. Since the marginal log-likelihood, ℓ, does not depend on q, we have

   $$\underset{q}{\text{argmax\hspace{0.17em}}}F(q,g,\theta ;y)=\underset{q}{\text{argmin\hspace{0.17em}}}{D}_{\text{KL}}\left(q\Vert {\widehat{p}}_{\text{post }}\right)={\widehat{p}}_{\text{post }}.$$
2. Since the minimum of D_KL with respect to q is zero for any (θ, g), we have that max_q F (q, g, θ; y) = ℓ(y; g, θ), and as a result

   $$(\widehat{g},\widehat{\theta })=\underset{g\in 𝒢,\theta \in \Theta }{\text{argmax}}\ell (y;g,\theta )=\underset{g\in 𝒢,\theta \in \Theta ,q}{\text{argmax}}\underset{q}{\text{max\hspace{0.17em}}}F(q,g,\theta ;y).$$

B.1.2. Variational approximation

The optimization problem (B.1) is often intractable. The idea of variational approximation is to adjust the problem to make it tractable, simply by restricting the optimization over all possible distributions on b to q ∈ 𝒬, where 𝒬 denotes a suitably chosen class of distributions. Therefore, we seek to solve B.1 subject to the additional constraint that q ∈ 𝒬:

$$\left({\widehat{p}}_{\text{post }},\widehat{g},\widehat{\theta }\right)=\underset{g\in 𝒢,\theta \in \Theta ,q\in 𝒬}{\text{argmax}}F(q,g,\theta ;y).$$ (B.3)

From the definition of F, it follows that optimizing F over q ∈ 𝒬 (for a given g and θ) corresponds to minimizing the KL divergence from q to the posterior distribution, and so can be interpreted as finding the “best” approximation to the posterior distribution for b among distributions in the class 𝒬. And the optimization of F over (g, θ) can be thought of as replacing the optimization of the log-likelihood with optimization of a lower bound to the log-likelihood (the ELBO).

We refer to solutions of the general problem (B.1), in which q is unrestricted, as “empirical Bayes (EB) solutions,” and we refer to solutions of the restricted problem (B.3) as “variational empirical Bayes (VEB) solutions.”

B.1.3. Form of ELBO

It is helpful to note that, by simple algebraic manipulations, the ELBO (B.2) can be decomposed as

$$\begin{gathered} F(q,g,\theta ;y)={\text{E}}_q\left[\text{log}\frac{p(y,b\mid g,\theta )}{q(b)}\right] \\ ={\text{E}}_q[\text{log\hspace{0.17em}}p(y\mid b,\theta )]+{\text{E}}_q\left[\text{log}\frac{g(b)}{q(b)}\right]. \end{gathered}$$ (B.4)

B.2. The additive effects model

We now apply the above results to fitting an additive model, ℳ, that includes the SuSiE model (3.1–3.6) as a special case:

$$y=\sum _{l=1}^L{\mu }_l+e$$

$$\mathit{e}~N_n\left(0,{\sigma }^2{I}_n\right)$$

$${\mathit{\mu }}_l~g_l,\text{\hspace{1em}\hspace{1em}\hspace{1em}independently\hspace{0.17em}for\hspace{0.17em}}l=1,\dots ,L,$$

where y = (y₁, …, y_n), e = (e₁, …, e_n), ${\mu }_I=\left({\mu }_{l1},\dots ,{\mu }_{ln}\right)\in {ℝ}^n$. We let ℳ_l denote the simpler model that is derived from ℳ by setting μ_l′ = 0 for all l′ ≠ l (i.e., ℳ_l is the model that includes only the lth additive term), and we use ℓ_l to denote the marginal log-likelihood for this simpler model:

$${\ell }_l\left(y;g_l,{\sigma }^2\right)≔\text{log\hspace{0.17em}}p\left(y\mid {ℳ}_l,g_l,{\sigma }^2\right).$$ (B.5)

The SuSiE model corresponds to the special case of ℳ where μ_l = Xb_l, for l = 1, …, L, and each g_l is the “single effect prior” in (2.6–2.8). Further, in this special case each ℳ_l is a “single effect regression” (SER) model (2.4–2.8).

The key idea introduced in this section is that we can fit ℳ by variational empirical Bayes (VEB) provided we can fit each simpler model ℳ_l by EB. To expand on this, consider fitting the model ℳ by VEB, where the restricted family 𝒬 is the class of distributions on (μ₁, …, μ_L) that factorize over μ₁, …, μ_L; that is, for any q ∈ 𝒬,

$$q\left({\mu }_1,\dots ,{\mu }_L\right)=\prod _{l=1}^L{q}_l\left({\mu }_l\right).$$

For q ∈ 𝒬, using expression (B.4), we obtain the following expression for the ELBO, F:

$$F\left(q,g,{\sigma }^2;y\right)=-\frac{n}{2}\text{log}\left(2\pi {\sigma }^2\right)-\frac{1}{2{\sigma }^2}{\text{E}}_q\left[{\Vert y-{\sum }_{l=1}^L{\mu }_l\Vert }^2\right]+\sum _{l=1}^L{\text{E}}_{q_l}\left[\text{log}\frac{g_l\left({\mu }_l\right)}{q_l\left({\mu }_l\right)}\right],$$ (B.6)

in which ∥ · ∥ denotes the Euclidean norm, and g denotes the collection of priors (g₁, …, g_L). The expected value in the second term of (B.6) is the expected residual sum of squares (ERSS) under the variational approximation q, and depends on q only through its first and second moments. Indeed, if we denote the posterior first and second moments by

$${\overline{\mu }}_{li}≔{\text{E}}_{q_l}\left[{\mu }_{li}\right]$$ (B.7)

$$\overline{{\mu }_{li}^2}≔{\text{E}}_{q_l}\left[{\mu }_{li}^2\right],$$ (B.8)

and we define ${\overline{\mu }}_l≔\left({\overline{\mu }}_{l1},\dots ,{\overline{\mu }}_{ln}\right)$, $\overline{{\mu }_l^2}:=\left(\overline{{\mu }_{l1}^2},\dots ,\overline{{\mu }_{ln}^2}\right)$, $\overline{\mu }≔\left({\overline{\mu }}_1,\dots ,{\overline{\mu }}_L\right)$, $\overline{{\mu }^2}≔\left(\overline{{\mu }_1^2},\dots ,\overline{{\mu }_L^2}\right)$, then we have that

$$\text{ERSS}\left(y,\overline{\mu },\overline{{\mu }^2}\right)={\text{E}}_q\left[{\Vert y-{\sum }_{l=1}^L{\mu }_l\Vert }^2\right]={\Vert y-{\sum }_{l=1}^L{\overline{\mu }}_l\Vert }^2+\sum _{l=1}^L\sum _{i=1}^n\text{Var}\left[{\mu }_{li}\right],$$ (B.9)

where $\text{Var}\left[{\mu }_{li}\right]=\overline{{\mu }_{li}^2}-{\overline{\mu }}_{li}^2$. This expression follows from the definition of the expected residual sum of squares, and from independence across l = 1, …, L, after some algebraic manipulation; see Section B.7.

Fitting ℳ by VEB involves optimizing F in (B.6) over q, g, σ². Our strategy is to update each (q_l, g_l) for l = 1, …, L while keeping σ² and other elements of q, g fixed, and with a separate optimization step for σ² with q, g fixed. This strategy is summarized in Algorithm A1.

The update for σ² in Algorithm A1 is easily obtained by taking partial derivative of (B.6), setting to zero, and solving for σ², giving

$${\widehat{\sigma }}^2≔\frac{\text{ERSS}\left(y,\overline{\mu },\overline{{\mu }^2}\right)}{n}.$$ (B.10)

The update for q_l, g_l corresponds to finding the EB solution for the simpler (single effect) model ℳ_l in which the data y are replaced with the expected residuals,

$${\overline{r}}_l≔{\text{E}}_q\left[r_l\right]≔{\text{E}}_q\left[y-{\sum }_{l^{\prime }\ne l}{\mu }_{l^{\prime }}\right]=y-{\sum }_{l^{\prime }\ne l}{\overline{\mu }}_{l^{\prime }}.$$

The proof of this result is given below in Proposition A1.

Substituting these ideas into Algorithm A1 yields Algorithm A2, which generalizes the IBSS algorithm (Algorithm 1) given in the main text.

B.3. Special case of SuSiE model

The SuSiE model is a special case of the above additive effects model when μ_l = Xb_l. In this case, ℳ_l is the SER model, and the first and second moments of μ_l are easily found from the first and second moments of b_l:

$$\text{E}\left[{\mu }_{li}\right]=\text{E}\left[{\sum }_{j=1}^p{x}_{ij}{b}_{lj}\right]={\sum }_{j=1}^p{x}_{ij}\text{E}\left[b_{lj}\right]$$

$$\text{E}\left[{\mu }_{li}^2\right]=\text{E}\left[{\left({\sum }_{j=1}^p{x}_{ij}{b}_{lj}\right)}^2\right]={\sum }_{j=1}^p{x}_{ij}^2\text{E}\left[b_{lj}^2\right].$$

The expression for the second moment simplifies because only one element of b_l is non-zero under the SER model, and so b_ljb_lj′ = 0 for any j ≠ j′. Because of this, we can easily formulate ERSS(y, $\overline{\mu }$, $\overline{{\mu }^2}$) as a function of the first and second moments of b_l — denoting this as ERSS(y, $\overline{b}$, $\overline{b^2}$) — and Algorithm A2 can be implemented using posterior distributions of b instead of posterior distributions of μ.

For completeness, we give this algorithm, which is Algorithm A3. This algorithm is the same as the IBSS algorithm in the main text (Algorithm 1), with additional steps for fitting the hyperparameters σ² and ${\sigma }_0^2$. This is the algorithm implemented in the susieR software. The step to update ${\sigma }_{0l}^2$ is a one-dimensional optimization problem; we implemented this step using the R function optim, which finds a stationary point of the likelihood surface with respect to ${\sigma }_{0l}^2$. The algorithm terminates when the increase in the ELBO between successive iterations is smaller than a small non-negative number, δ (set to 0.001 unless otherwise stated). This is a commonly used stopping criterion in algorithms for fitting variational approximations.

B.4. Update for q_l, g_l in additive effects model is EB solution for simpler model, ℳ_l

Here we establish that the update to q_l, g_l in Algorithm A1 can be implemented as the EB solution for ℳ_l (Steps 5 and 6 in Algorithm A2). This result is formalized in the following proposition, which generalizes Proposition 1 in the main text.

Proposition A1. The q_l, g_l that maximizes F in (B.6), the ELBO for the additive model, ℳ, can be found by maximizing the ELBO for the simpler model, ℳ_l, in which the observed responses y are replaced by the expected residuals, ${\overline{r}}_l$:

$$\underset{q_l,g_l}{\text{argmax\hspace{0.17em}}}F\left(q,g,{\sigma }^2;y\right)=\underset{q_l,g_l}{\text{argmax\hspace{0.17em}}}{F}_l\left(q_l,g_l,{\sigma }^2;{\overline{r}}_l\right),$$

where ${\overline{\mu }}_l$ is the vector of posterior mean effects defined above (see eq. B.7), and we define

$$F_l\left(q_l,g_l,{\sigma }^2;y\right)=-\frac{n}{2}\text{log}\left(2\pi {\sigma }^2\right)-\frac{1}{2{\sigma }^2}{\text{E}}_{q_l}\left[{\Vert y-{\mu }_l\Vert }^2\right]+{\text{E}}_{q_l}\left[\text{log}\frac{g_l\left({\mu }_l\right)}{q_l\left({\mu }_l\right)}\right].$$ (B.11)

Proof. Omitting terms in the expression for F (from eq. B.6) that do not depend on q_l, g_l (these terms are captured by “const”), we have

$$\begin{gathered} F\left(q,g,{\sigma }^2;y\right)=-\frac{1}{2{\sigma }^2}{\text{E}}_q\left[{\left(r_l-{\mu }_l\right)}^{\top }\left(r_l-{\mu }_l\right)\right]+{\text{E}}_{q_l}\left[\text{log}\frac{g_l\left({\mu }_l\right)}{q_l\left({\mu }_l\right)}\right]+\text{ const } \\ =-\frac{1}{2{\sigma }^2}{\text{E}}_q\left[-2r_l^{\top }{\mu }_l+{\mu }_l^{\top }{\mu }_l\right]+{\text{E}}_{q_l}\left[\text{log}\frac{g_l\left({\mu }_l\right)}{q_l\left({\mu }_l\right)}\right]+\text{ const } \\ =-\frac{1}{2{\sigma }^2}{\text{E}}_{q_l}\left[-2{\overline{r}}_l^{\top }{\mu }_l+{\mu }_l^{\top }{\mu }_l\right]+{\text{E}}_{q_l}\left[\text{log}\frac{g_l\left({\mu }_l\right)}{q_l\left({\mu }_l\right)}\right]+\text{ const } \\ =F_I\left(q_l,g_l,{\sigma }^2;{\overline{r}}_l\right)+\text{ const}. \end{gathered}$$

□

Further note that the optimization of F_l does not restrict q_l, so the maximum yields the exact EB solution for M_l (refer to Section B.1.1); that is $q_l\left({\mu }_l\right)=p\left({\mu }_l\mid {\overline{r}}_l,{ℳ}_l,g_l,{\sigma }^2\right)\propto p\left({\overline{r}}_l\mid {ℳ}_l,g_l,{\sigma }^2\right)g_l\left({\mu }_l\right)$ at the maximum.

B.5. Convergence of IBSS algorithm

B.5.1. Proof of Corollary 1

Proof. Step 5 of Algorithm 1 is simply computing the right-hand side of (3.9), in which the posterior distribution is determined by parameters α_l, μ_1l, ${\sigma }_{1l}^2$. Therefore, by Proposition 1, it is a coordinate ascent step for optimizing the lth coordinate of F (q₁, …, q_L; σ², ${\sigma }_0^2$) in which q_l is determined by the parameters α_l, μ_1l, ${\sigma }_{1l}^2$. □

B.5.2. Proof of Proposition 2

Proof. By Proposition 2.7.1 of Bertsekas (1999), the sequence of iterates q converges to a stationary point of F provided that argmax_ql,gl F_l (q_l, g_l, σ²; ${\overline{r}}_l$) is uniquely attained for each l. When ℳ_l is the SER model and μ_l = Xb_l, the lower bound F_l (B.11) is

$$F_l\left(q_l,g_l,{\sigma }^2;y\right)=-\frac{n}{2}\text{log}\left(2\pi {\sigma }^2\right)-\frac{\Vert y-X\overline{b}{\Vert }^2}{2{\sigma }^2}+\frac{\Vert X\overline{b}{\Vert }^2}{2{\sigma }^2}-\frac{1}{2{\sigma }^2}\sum _{j=1}^p{x}_j^{\top }{x}_j{\alpha }_j\left({\mu }_{1j}^2+{\sigma }_{1j}^2\right)+\sum _{j=1}^p\frac{{\alpha }_j}{2}\left[1+\text{log}\frac{{\sigma }_{1j}^2}{{\sigma }_0^2}-\frac{{\mu }_{1j}^2+{\sigma }_{1j}^2}{{\sigma }_0^2}\right]+\sum _{j=1}^p{\alpha }_j\text{\hspace{0.17em}log}\frac{{\pi }_j}{{\alpha }_j},$$

To lighten notation in the above expression, the l subscript was omitted from the quantities α = (α₁, …,α_p), μ₁ = (μ₁₁, …, μ_1p) and ${\sigma }_1^2=\left({\sigma }_{11}^2,\dots ,{\sigma }_{1p}^2\right)$ specifying the SER approximate posterior, q_l, and likewise for the vector of posterior means, $\overline{b}≔{\overline{b}}_l$ with elements ${\overline{b}}_j={\alpha }_j{\mu }_{1j}$. Taking partial derivatives of this expression with respect to the parameters α, μ₁ and ${\sigma }_1^2$, the maximum can be expressed as the solution to the following system of equations:

$${\alpha }_j\left[\frac{1}{{\sigma }_{1j}^2}-\left(\frac{x_j^T{x}_j}{{\sigma }^2}+\frac{1}{{\sigma }_0^2}\right)\right]=0$$ (B.12)

$${\alpha }_j\left[\frac{{\mu }_{1j}}{{\sigma }_{1j}^2}-\frac{\left(X^{T}y\right)}{{\sigma }^2}\right]=0$$ (B.13)

$$\text{log}\frac{{\alpha }_j}{{\pi }_j}-\text{log}\frac{{\sigma }_{1j}}{{\sigma }_0}-\frac{{\mu }_{1j}^2}{2{\sigma }_{1j}^2}+\lambda =0,$$ (B.14)

where $\lambda \in ℝ$ is an additional unknown, set so that α₁ + ⋯ + α_p = 1 is satisfied. The solution to this set of equations is finite and unique if 0 < σ, σ₀ < ∞ and π_j > 0 for all j = 1, …, p. Also note that the solution to (B.12–B.14) recovers the posterior expressions for the SER model. □

B.6. Computing the evidence lower bound

Although not strictly needed to implement Algorithms A2 and A3, it can be helpful to compute the objective function, F (e.g., to monitor the algorithm’s progress, or to compare solutions). Here we outline a practical approach to computing F for the SuSiE model.

Refer to the expression for the ELBO, F, given in (B.6). Computing the first term is straightforward. The second term is the ERSS (B.9). The third term can be computed from the marginal log-likelihoods ℓ_l in (B.5), and computing this is straightforward for the SER model, involving a sum over the p possible single effects (see eq. A.5). This is shown by the following lemma:

Lemma A1. Let ${\widehat{q}}_I≔{\text{argmax}}_q{F}_l\left(q_l,g_l,{\sigma }^2;{\overline{r}}_l\right)$. Then

$${\text{E}}_{{\widehat{q}}_l}\left[\text{log}\frac{g_l\left({\mu }_l\right)}{{\widehat{q}}_l\left({\mu }_l\right)}\right]={\ell }_l\left({\overline{r}}_l;g_l,{\sigma }^2\right)+\frac{n}{2}\text{log}\left(2\pi {\sigma }^2\right)+\frac{1}{2{\sigma }^2}{\text{E}}_{{\widehat{q}}_l}{\Vert {\overline{r}}_l-{\mu }_l\Vert }^2.$$ (B.15)

Proof. Rearranging (B.11), and replacing y with ${\overline{r}}_l$, we have

$${\text{E}}_{q_l}\left[\text{log}\frac{g_l\left({\mu }_I\right)}{q_l\left({\mu }_l\right)}\right]=F_l\left(q_l,g_l,{\sigma }^2;{\overline{r}}_l\right)+\frac{n}{2}\text{log}\left(2\pi {\sigma }^2\right)+\frac{1}{2{\sigma }^2}{\text{E}}_{q_l}{\Vert {\overline{r}}_l-{\mu }_l\Vert }^2.$$ (B.16)

The result then follows from observing that F_l is equal to ℓ_l at the maximum, $q_l={\widehat{q}}_l$; that is, $F_l\left({\widehat{q}}_l,g_l,{\sigma }^2;{\overline{r}}_l\right)={\ell }_l\left({\overline{r}}_l;g_l,{\sigma }^2\right)$. □

B.7. Expression for the expected residual sum of squares (ERSS)

The expression (B.9) is derived as follows:

$$\begin{gathered} \text{ERSS}\left(y,\overline{\mu },\overline{{\mu }^2}\right)={\text{E}}_q\left[{\Vert y-{\sum }_{l=1}^L{\mu }_l\Vert }^2\right] \\ =y^{\top }y-2y^{\top }\sum _{l=1}^L{\overline{\mu }}_l+\sum _{l=1}^L\sum _{l^{\prime }=1}^L{\overline{\mu }}_l^{\top }{\overline{\mu }}_{l^{\prime }}-\sum _{l=1}^L{\overline{\mu }}_l^{\top }{\overline{\mu }}_l+\sum _{l=1}^L{\text{E}}_{q_l}\left[{\mu }_l^{\top }{\mu }_l\right] \\ ={\Vert y-{\sum }_{l=1}^L{\overline{\mu }}_l\Vert }^2+\sum _{l=1}^L\sum _{i=1}^n\text{Var}\left[{\mu }_{li}\right], \end{gathered}$$

where $\text{Var}\left[{\mu }_{li}\right]=\overline{{\mu }_{li}^2}-{\overline{\mu }}_{li}^2$.

C. CONNECTING SUSIE TO STANDARD BVSR

When L ≪ p, the SuSiE model (3.1–3.6) is closely related to a standard BVSR model in which a subset of L regression coefficients are randomly chosen to have non-zero effects.

To make this precise, consider the following regression model:

$$y=Xb+e$$

$$\mathit{e}~N_n\left(0,{\sigma }^2{I}_n\right)$$

with n observations and p variables, so that b is a p-vector. Let ${\Pi }_{L,p}^{\text{standard }}\left(\cdot ;{\sigma }_0^2\right)$ denote the prior distribution on b that first randomly selects a subset S ⊂ {1, …, p} uniformly among all $\left(\begin{array}{l}p\hfill \\ L\hfill \end{array}\right)$ subsets of cardinality |S| = L, and then randomly samples the non-zero values b_S – {b_j : j ∈ S} independently from $N_1\left(0,{\sigma }_0^2\right)$, setting the other values $b_{\overline{S}}≔\left\{b_j:j\notin S\right\}$ to 0. (This is a version of the prior considered by Castillo et al. 2015, with |S| = L.) Further, let ${\Pi }_{L,p}^{\text{susie }}\left(\cdot ;{\sigma }_0^2\right)$ denote the prior distribution on b induced by the SuSiE model (3.1–3.6) with identical prior variances, ${\sigma }_{l0}^2={\sigma }_0^2$, for all l = 1, …, L.

Proposition A2. With L fixed, letting p → ∞, the SuSiE prior is equivalent to the standard prior. Specifically, for any event A,

$$\underset{p\to \infty }{\text{lim}}\left({\Pi }_{L,p}^{\text{susie }}\left(A;{\sigma }_0^2\right)-{\Pi }_{L,p}^{\text{standard }}\left(A;{\sigma }_0^2\right)\right)=0.$$

Proof. Fix L and p, and let B denote the event that the L vectors γ₁, …, γ_L in the SuSiE model are distinct from one another. Conditional on B, it is clear from symmetry that the SuSiE prior exactly matches the standard prior; that is, ${\Pi }_{L,p}^{\text{susie }}(A\mid B)={\Pi }_{L,p}^{\text{standard }}(A)$, dropping notational dependence on ${\sigma }_0^2$ for simplicity. Thus,

$$\begin{gathered} {\Pi }_{L,p}^{\text{susie }}(A)-{\Pi }_{L,p}^{\text{standard }}(A)={\Pi }_{L,p}^{\text{susie }}(A)-{\Pi }_{L,p}^{\text{susie }}(A\mid B) \\ ={\Pi }_{L,p}^{\text{susie }}(A\mid B){\text{Pr}}_{L,p}(B)+{\Pi }_{L,p}^{\text{susie }}(A\mid \overline{B}){\text{Pr}}_{L,p}(\overline{B})-{\Pi }_{L,p}^{\text{susie }}(A\mid B), \end{gathered}$$

where the last line follows from the law of total probability. The result then follows from the fact that the probability Pr_L,p(B) → 1 as p → ∞:

$${\text{Pr}}_{L,p}(B)=[p/p][(p-1)/p][(p-2)/p]\cdots [(p-L+1)/p]\to 1\text{ as }p\to \infty .$$

D. SIMULATION DETAILS

D.1. Simulated data

For the numerical simulations of eQTL fine mapping in Section 4, we used n = 574 human genotypes collected as part of the Genotype-Tissue Expression (GTEx) project (GTEx Consortium, 2017). Specifically, we obtained genotype data from whole-genome sequencing, with imputed genotypes, under dbGaP accession phs000424.v7.p2. In our analyses, we only included SNPs with minor allele frequencies 1% or greater. All reported SNP base-pair positions were based on Genome Reference Consortium (GRC) human genome assembly 38.

To select SNPs nearby each gene, we considered two SNP selection schemes in our simulations: (i) in the first scheme, we included all SNPs within 1 Megabase (Mb) of the gene’s transcription start site (TSS); (ii) in the second, we used the p = 1, 000 SNPs closest to the TSS. Since the GTEx data contain a very large number of SNPs, the 1,000 closest SNPs are never more than 0.4 Mb away from the TSS. Selection scheme (i) yields genotype matrices X with at least p = 3,022 SNPs and at most p = 11,999 SNPs, and an average of 7,217 SNPs.

For illustration, correlations among the SNPs for one of the data sets are shown in Fig. A1 (see also Fig. 1).

FIGURE A1. Correlations among variables (SNPs) in an example data set used in the fine mapping comparisons.

Left-hand panel shows correlations among variables shown at positions 100–200 in Fig. 1; right-hand panel shows correlations among variables shown at positions 350–450. For more details on this example data set, see Section 4.1 in the main text.

D.2. Software and hardware specifications for numerical comparisons study

In CAVIAR, we set all prior inclusion probabilities to 1/p to match the default settings used in other methods. In CAVIAR and FINEMAP, we set the maximum number of effect variables to the value of S that was used to simulate the gene expression data. The maximum number of iterations in FINEMAP was set to 100,000 (this is the FINEMAP default). We estimate σ² in SuSiE for all simulations.

All computations were performed on Linux systems with Intel Xeon E5–2680 v4 (2.40 GHz) processors. We ran SuSiE in R 3.5.1, with optimized matrix operations provided by the dynamically linked OpenBLAS libraries (version 0.3.5). DAP-G and CAVIAR were compiled from source using GCC 4.9.2, and pre-compiled binary executables, available from the author’s website, were used to run FINEMAP.

E. FUNCTIONAL ENRICHMENT OF SPLICE QTL FINE MAPPING

To strengthen results of Section 5, here we provide evidence that splice QTLs identified by SuSiE are enriched in functional genomic regions, thus likely to contain true causal effects. To perform this analysis, we labelled one CS at each intron the “primary CS.” We chose the CS with highest purity at each intron as the primary CS; any additional CSs at each intron were labelled as “secondary CSs.” We then tested both primary and secondary CSs for enrichment of biological annotations by comparing the SNPs inside these CSs (those with PIP > 0.2) against random “control” SNPs outside all primary and secondary CSs.

We tested for enrichment of the same generic variant annotations used in Li et al. (2016). These include LCL-specific histone marks (H3K27ac, H3K27me3, H3K36me3, H3K4me1, H3K4me2, H3K4me3, H3K79me2, H3K9ac, H3K9me3, H4K20me1), DNase I hypersensitive sites, transcriptional repressor CTCF binding sites, RNA polymerase II (PolII) binding sites, extended splice sites (defined as 5 base-pairs upstream and downstream of an intron start site, and 15 base-pairs upstream and downstream of an intron end site), as well as intron and coding annotations. In total, 16 variant annotations were tested for enrichment.

Figure A2 shows the enrichments in both primary and secondary CSs for the 12 out of 16 annotations that were significant at p-value < 10⁻⁴ in the primary signals (Fisher’s exact test, two-sided, no p-value adjustment for multiple comparisons). The strongest enrichment in both primary and secondary signals was for extended splice sites (odds ratio ≈ 5 in primary signals), which is reassuring given that these results are for splice QTLs. Other significantly enriched annotations in primary signals include PolII binding, several histone marks, and coding regions. The only annotation showing a significant depletion was introns. Results for secondary signals were qualitatively similar to those for primary, though all enrichments are less significant, which is most likely explained by the much smaller number of secondary CSs.

FIGURE A2. Splicing QTL enrichment analysis results.

Estimated odds ratios, and ± 2 standard errors, for each variant annotation, obtained by comparing the annotations of SNPs inside primary/secondary CSs against random “control” SNPs outside CSs. The p-values are from two-sided Fisher’s exact test, without multiple testing correction. The vertical line in each plot is posited at odds ratio = 1.

DATA AND RESOURCES

SuSiE is implemented in the R package susieR available at https://github.com/stephenslab/susieR. Source code and a website detailing the analysis steps for numerical comparisons and data applications are available at our manuscript resource repository (Wang et al., 2020b), also available at https://github.com/stephenslab/susie-paper.