Approximating posteriors with high-dimensional nuisance parameters via integrated rotated Gaussian approximation

Summary

Posterior computation for high-dimensional data with many parameters can be challenging. This article focuses on a new method for approximating posterior distributions of a low- to moderate-dimensional parameter in the presence of a high-dimensional or otherwise computationally challenging nuisance parameter. The focus is on regression models and the key idea is to separate the likelihood into two components through a rotation. One component involves only the nuisance parameters, which can then be integrated out using a novel type of Gaussian approximation. We provide theory on approximation accuracy that holds for a broad class of forms of the nuisance component and priors. Applying our method to simulated and real data sets shows that it can outperform state-of-the-art posterior approximation approaches.

1. Introduction

Consider the regression model

$$y~N\left(X\beta +\eta ,{\sigma }^2{I}_n\right),$$ (1)

where y is an n-dimensional vector of observations, X is an n × p design matrix, β is a p-dimensional parameter of interest, η is an n-dimensional nuisance parameter, and σ² is the error variance. The nuisance parameter can for instance capture the effect of a large set of covariates not included in X, or of non-Gaussian errors. Our goal is Bayesian inference on the model in (1) when p is of moderate size such that p ≪ n − p with the focus on the posterior

$$\pi (\beta |y)=\int \pi (\beta ,\eta |y)d\eta =\frac{1}{\pi (y)}\int \pi (y|\beta ,\eta )\pi (\beta ,\eta )d\eta .$$ (2)

The integrals in (2) and π(y) are intractable to approximate accurately for certain priors π(β, η), with direct approximations such as Laplace’s method producing inaccurate results and Monte Carlo sampling being daunting computationally. Our key idea is to transform the hard problem with nuisance parameter η in a principled way to a p-dimensional one which can be written as a linear model including only β. Then, a low-dimensional inference technique can be applied to this p-dimensional model. The transformation uses a novel type of Gaussian approximation using a data rotation to integrate out η from (1).

Section 3 discusses special cases of the model in (1). Applications include epidemiology studies in which y is a health outcome, X consists of exposures and key clinical or demographic factors of interest, and η is the effect of high-dimensional biomarkers. The goal is inference on the effect of the exposures and the clinical or demographic covariates, but adjusting for the high-dimensional biomarkers. For example, η may result from genetic factors, such as single-nucleotide polymorphisms (SNPs), and we want to control for these in identifying an environmental main effect. It is often impossible to isolate the impact of individual genetic factors so we consider these effects as nuisance parameters. Another use of (1) is computation of posterior inclusion probabilities in high-dimensional Bayesian variable selection as detailed in § 3·2.

Data with a complex component η that is not of primary interest and only a moderate number p of parameters of interest, are more and more common. Unfortunately, the complexity of η can make accurate approximation of π(β | y) in (2) challenging even when p = 1. One naive approach is to ignore the nuisance parameter η by setting it to zero. The result can be problematic as omitting η changes the interpretation of the parameter of interest β, which therefore might take on a different value. For example, η might capture the effect of covariates with it being important to adjust for them to avoid misleading conclusions on β.

Many posterior approximation methods exist, including Monte Carlo (George & McCulloch, 1993, 1997; O’Hara & Sillanpää, 2009), variational Bayes (Carbonetto & Stephens, 2012; Ormerod et al., 2017), integrated nested Laplace approximations (Rue et al., 2009), and expectation propagation (Hernández-Lobato et al., 2015). However, these methods can be computationally expensive, do not apply to our setting, or lack theoretical results regarding approximation accuracy. A notable exception to the latter is the fast posterior approximation algorithm of Huggins et al. (2017) which comes with bounds on the approximation error under conditions on the prior such as log-concavity, Gaussianity, and smoothness. The class of priors that we allow on β and η is much larger. Our method and its analysis for instance apply to dimensionality reduction and shrinkage priors such as spike-and-slab, horseshoe, and Laplace distributions.

The main computational bottleneck of our method is calculation of the mean and variance of a nuisance term, for which one can choose any suitable algorithm. As a result, the computational cost of our method is comparable to that of the fast algorithm chosen for this step.

2. Integrated rotated Gaussian approximation

2·1. Notation and assumptions

Denote the multivariate Gaussian distribution with mean μ and covariance Σ by N(μ, Σ), and its density function evaluated at a by $N\left(a|\mu ,\Sigma \right)$. Denote the distribution of a conditional on b by $\prod \left(a|b\right)$ and its density, with respect to some dominating measure, evaluated at a by $\pi \left(a|b\right)$. We assume that β and η are a priori independent so that $\prod \left(\beta ,\eta \right)=\prod \left(\beta \right)\prod \left(\eta \right)$. We treat X and σ² as known constants unless otherwise noted. Assume that p ≤ n. We assume that X is full rank to simplify the exposition, but our method also applies to rank deficient X.

2·2. Description of the method

We integrate out η from (1) by splitting the model into two parts, one of which does not involve β. A data rotation provides such a model split. Specifically, consider as rotation matrix the n × n orthogonal matrix Q from the QR decomposition of X. Define the n × p matrix M and the n × (n − p) matrix S by (M, S) = Q. Then, the columns of M form an orthonormal basis for the column space of X since X is full rank by assumption (Golub & Van Loan, 1996, § 5.2). Since Q is orthogonal, the columns of S form an orthonormal basis for the orthogonal complement of the column space of X. Therefore, S^TX = 0_(n−p)×p, an (n − p) × p matrix of zeros, which can also be derived from the fact that Q^TX is upper triangular.

By the rotational invariance of the Gaussian distribution and $Q^{\text{T}}Q=I_n$, $Q^{\text{T}}y~N\left(Q^{\text{T}}X\beta +Q^{\text{T}}\eta ,{\sigma }^2{I}_n\right)$ is distributionally equivalent to (1). This rotated model splits as

$$M^{\text{T}}y~N\left(M^{\text{T}}X\beta +M^{\text{T}}\eta ,{\sigma }^2{I}_p\right),$$ (3a)

$$S^{\text{T}}y~N\left(S^{\text{T}}\eta ,{\sigma }^2{I}_{n-p}\right);$$ (3b)

using $S^{\text{T}}X=0_{(n-p)\times p}$. This transformation motivates a two-stage approach in which one first computes $\Pi \left(\eta |S^{\text{T}}y\right)$ from submodel (3b) and then uses this distribution as an updated prior for the projected nuisance term M^Tη in submodel (3a). Following this approach, the posterior of β can be expressed as $\Pi (\beta |y)\propto \Pi (\beta )\int N\left(M^{\text{T}}y|M^{\text{T}}X\beta +M^{\text{T}}\eta ,{\sigma }^2{I}_p\right)d\Pi \left(M^{\text{T}}\eta |S^{\text{T}}y\right)$.

In practice, $\Pi \left(M^{\text{T}}\eta |S^{\text{T}}y\right)$ may be intractable to compute exactly because of the complexity of Π(η). To alleviate this challenge, we consider an approximation $\widehat{\Pi }\left(M^{\text{T}}\eta |S^{\text{T}}y\right)$, which then leads to an approximation for the posterior of β:

$$\widehat{\Pi }(\beta |y)\propto \Pi (\beta )\int N\left(M^{\text{T}}y|M^{\text{T}}X\beta +M^{\text{T}}\eta ,{\sigma }^2{I}_p\right)d\widehat{\Pi }\left(M^{\text{T}}\eta |S^{\text{T}}y\right).$$ (4)

All distributions, densities, and probabilities resulting from this approximation carry a hat to distinguish them from their exact counterparts.

A Gaussian approximation is analytically convenient:

$$\widehat{\Pi }\left(M^{\text{T}}\eta |S^{\text{T}}y\right)=N(\widehat{\mu },\widehat{\Sigma }),$$ (5)

where $\widehat{\mu }$ and $\widehat{\Sigma }$ are estimates of the mean and covariance of $\Pi \left(M^{\text{T}}\eta |S^{\text{T}}y\right)$, respectively. In this case, (4) simplifies as

$$\widehat{\Pi }(\beta |y)\propto \Pi (\beta )N\left(M^{\text{T}}y|M^{\text{T}}X\beta +\widehat{\mu },{\sigma }^2{I}_p+\widehat{\Sigma }\right).$$ (6)

Only β is unknown such that the computational problems with (1) resulting from the complexity of Π(η) have been resolved in (6). Furthermore, (6) is equivalent to a Gaussian linear model with observations $M^{\text{T}}y-\widehat{\mu }$, design matrix M^TX, and parameter β. We have reduced a model with a potentially challenging nuisance parameter to a low-dimensional one with the nuisance integrated out while controlling for the effect of the nuisance parameter in a principled manner. Algorithm 1 summarizes our method when the Gaussian approximation from (5) is used.

Algorithm 1.

Integrated rotated Gaussian approximation.

Input: Data (y, X)

1. Compute the QR decomposition of X to obtain the rotation matrix Q = (M, S).

2. Compute the estimates $\widehat{\mu }$ and $\widehat{\Sigma }$ for the mean and covariance of $\Pi \left(M^{\text{T}}\eta |S^{\text{T}}y\right)$ based on submodel (3b) using an algorithm of choice.

3. Approximate the posterior $\Pi (\beta |y)$ according to (6).

Output: The approximate posterior $\widehat{\Pi }(\beta |y)$

2·3. Relation to other methods

Algorithm 1 has resemblances with other approximation methods. Integrated nested Laplace approximations (Rue et al., 2009) also approximate a nested part of a Bayesian model by a Gaussian distribution but with important differences. A Laplace approximation is applied without a data rotation and is done at two, rather than one, nested levels of the model. Moreover, a Laplace approximation matches the mode and curvature of the approximating Gaussian while (5) matches the moments. Laplace’s method (Tierney & Kadane, 1986) requires a continuous target distribution and integrated nested Laplace approximations assume a conditionally Gaussian prior on some parameters. Our Gaussian approximation needs no such conditions on priors but assumes a Gaussian error distribution. For instance, § 3 considers examples of priors on η that are not continuous or are non-Gaussian.

The approximation in (5) aims to match the first two moments of the exact Π(M^Tη | S^Ty). Such matching is the principle behind expectation consistent inference (Opper & Winther, 2005). Our method matches moments for the nuisance parameter but not for the parameter of interest β. This differs from applications of the expectation consistent framework in which moment matching is pervasive such as in expectation propagation (Hernández-Lobato et al., 2015). Implementations of expectation propagation are usually not able to capture dependence among dimensions of the posterior while our method allows for dependence in the p-dimensional β.

Effectively, our method integrates out the nuisance parameter η approximately. Integrating out nuisance parameters from the likelihood is not new (Berger et al., 1999), including doing so approximately (Severini, 2011). Previous approximations, however, do not apply a data rotation and consider cases where the distribution on the nuisance parameter is regular enough so that a Laplace approximation can be applied. Our method does not need such regularity conditions.

The rotation Q is similar to the projection in the Frisch-Waugh-Lovell theorem (Stachurski, 2016, Theorem 11.2.1) for least-squares estimation of a parameter subset. Our method applies beyond least squares. Also, our estimation of the nuisance parameter through the rotation is merely an intermediate step for inference on β. Our method reduces to the algorithm from van den Boom et al. (2015) when considering the example in § 3·2 with p = 1.

2·4. Estimating σ² and hyperparameters

So far, we have treated σ² as fixed and known. In practice, σ² usually needs to be estimated, as well as any unknown parameters in the prior on η. This estimation fits naturally into Step 2 of Algorithm 1 as the methods that can be used there frequently come with such estimation procedures: See for instance § S5·3 and § S6 of the Supplementary Material. The resulting estimates can then be plugged into Step 3. By doing so, only the (n − p)-dimensional submodel (3b) informs the estimates of these parameters and not the p-dimensional submodel (3a). We expect (3b) to contain the vast majority of information on the unknown parameters if (n − p) ⨠ p, which is often the case in scenarios of interest.

3. Examples of nuisance parameters η

3·1. Adjusting for high-dimensional covariates

Section 3 provides examples of the general setting of model (1) that demonstrate the utility of the integrated rotated Gaussian approximation in Algorithm 1. As a first example, consider η = Zα with Z a known n × q feature matrix and α an unknown q-dimensional parameter with q ⨠ n. Then, the model in (1) becomes $y~N\left(X\beta +Z\alpha ,{\sigma }^2{I}_n\right)$, so that we are adjusting for high-dimensional covariates Z in performing inference on the coefficients β on the predictors X of interest. One way to deal with the fact that the number of covariates q exceeds the number of observations n is by inducing sparsity in α via its prior Π(α). We consider the spike-and-slab prior, ${\alpha }_j~\lambda N(0,\psi )+(1-\lambda )\delta (0)$ independently for j = 1, … , q, where λ = pr(α_j ≠ 0) is the prior inclusion probability, ψ the slab variance, and δ(0) a point mass at zero. By specifying Π(α), we have also defined Π(η) = Π(Zα). Since each Π(α_j) is a mixture of a point mass and a Gaussian, Π(α) and thus Π(η) are mixtures of 2^q Gaussians. As a result, computation of π(β | y) in (2) involves summing over these 2^q components. This is infeasible for large q.

Algorithm 1 provides an approximation $\widehat{\Pi }(\beta |y)$ while avoiding the exponential computational cost. Step 2 in Algorithm 1 requires choice of an estimation algorithm. Substituting η = Zα into (3b) yields $S^{\text{T}}y~N\left(S^{\text{T}}Z\alpha ,{\sigma }^2{I}_{n-p}\right)$, which is a linear model with (n − p) observations and design matrix S^TZ. As such, methods for linear regression with spike-and-slab priors can produce an approximation to $\Pi \left(\alpha |S^{\text{T}}y\right)$ and thus the estimates $\widehat{\mu }$ and $\widehat{\Sigma }$ in (5). We choose vector approximate message passing (Rangan et al., 2017), detailed in § S5 of the Supplementary Material, to approximate $\Pi \left(\alpha |S^{\text{T}}y\right)$ because of its computational scalability and accuracy. The computational scalability limits the size of q. For instance, § 5·3 considers a subset of q = 10, 000 SNPs as using all SNPs was computationally infeasible. As a more scalable alternative, we consider the debiased lasso (Javanmard & Montanari, 2013) in § 5·2 as it can also approximate $\Pi \left(\alpha |S^{\text{T}}y\right)$ as detailed in § S6 of the Supplementary Material. A q in the millions is feasible with embarrassingly parallel split-and-merge strategies (Song & Liang, 2014). The q-dimensional distribution $\Pi \left(\alpha |S^{\text{T}}y\right)$ is possibly highly non-Gaussian, being a mixture of Gaussians. At the same time, the p-dimensional distribution $\Pi \left(M^{\text{T}}Z\alpha |S^{\text{T}}y\right)=\Pi \left(M^{\text{T}}\eta |S^{\text{T}}y\right)$ can be nearly Gaussian such that the approximation in (5) is accurate as discussed in § 4·2.

3·2. Bayesian variable selection

For a second application of (1), consider the linear model $y~N\left(A\theta ,{\sigma }^2{I}_n\right)$ where A is a known n × r design matrix and θ an unknown r-dimensional parameter. Variable selection is the problem of determining which entries of θ are non-zero. Modeling the data in a Bayesian fashion provides a natural framework to evaluate statistical evidence via the posterior Π(θ | y). A standard variable selection prior Π(θ) is the spike-and-slab prior defined by ${\theta }_j~\lambda N(0,\psi )+(1-\lambda )\delta (0)$ independently for j = 1, … , p. As in § 3·1, the cost of computing the exact posterior with a spike-and-slab prior grows exponentially in r. Therefore, computation of Π(θ | y) is infeasible for r beyond moderate size. A variety of approximation methods exist for larger r including Monte Carlo (George & McCulloch, 1993, 1997; O’Hara & Sillanpää, 2009), variational Bayes (Carbonetto & Stephens, 2012; Ormerod et al., 2017), and expectation propagation (Hernández-Lobato et al., 2015).

Monte Carlo methods do not scale well with the number of predictors r. For r even moderately large, the 2^r possible non-zero subsets of θ is so huge that there is no hope of visiting more than a vanishingly small proportion of models. The result is high Monte Carlo error in estimating posterior probabilities, with almost all models assigned zero probability as they are never visited. As an alternative to Monte Carlo sampling, fast approximation approaches for Bayesian variable selection include variational Bayes (Carbonetto & Stephens, 2012; Ormerod et al., 2017) and expectation propagation (Hernández-Lobato et al., 2015). Their accuracy, however, does not come with theory guarantees. Our method, which applies to variable selection as detailed in the next paragraph, allows for theoretical analysis as § 4 shows.

In variable selection, often the main question asked is whether θ_j ≠ 0 (j = 1, … , r) as measured by the posterior inclusion probability pr(θ_j ≠ 0 | y). Algorithm 1 can estimate pr(θ_j ≠ 0 | y): Let p < r elements from θ constitute β and let the other q = r − p elements in θ constitute α. Then, Aθ = Xβ + Zα where X and Z consist of the respective columns in A, and $\Pi (\alpha ,\beta )=\Pi (\alpha )\Pi (\beta )$ since $\Pi (\theta )={\prod }_{j=1}^r\Pi \left({\theta }_j\right)$. This set-up is the same as in § 3·1 and Algorithm 1 approximates Π(β | y) as in § 3·1. Assuming θ_j is contained in β, an approximation of Π(θ_j | y) can be obtained as a marginal distribution of $\widehat{\Pi }(\beta |y)$. Repeating Algorithm 1 with different splits of θ into β and α provides estimates of all $\mathrm{pr}\left({\theta }_j\ne 0|y\right)(j=1,\dots ,r)$. Computations for these different splits can run in parallel.

The approximation accuracy is not very sensitive to how θ is split into α and β, and to p per § S9 of the Supplementary Material. We therefore use simple sequential splitting, where the first p elements of θ constitute β in the first split, and recommend choosing p based on computational complexity. Assume that the number of CPU cores is less than the number of variables r. Then, computation time to obtain all $\widehat{\mathrm{pr}}\left({\theta }_j\ne 0|y\right)$ is a trade-off between the length p of β, which affects the cost of each execution of Algorithm 1, and the number r/p of executions of Algorithm 1. The order of r is limited by the order of q, which is again limited by the algorithm chosen for Step 2 of Algorithm 1 as discussed in § 3·1. The complexity in terms of p and r of computing all $\widehat{\mathrm{pr}}\left({\theta }_j\ne 0|y\right)$ is O(r² log² r) if p = O(log r) and vector approximate message passing is used as detailed in the next paragraph.

Step 1 of Algorithm 1 is the QR decomposition of an n × p matrix which has complexity O(np²) (Golub & Van Loan, 1996, § 5.2). Step 2 involves vector approximate message passing on n − p observations and q parameters, which has a complexity of O{(n − p + K) q min(n − p, q)} where K is the number of message passing iterations as detailed in § S5·2 of the Supplementary Material. Additionally for Step 2, computation of S^Ty and S^TZ, which are the observations and design matrix in (3b), and computing $\widehat{\mu }$ and $\widehat{\Sigma }$ in (5) from the message passing output is O(n²q). Computing Step 3 with the spike-and-slab prior Π(β) is O(2^p p³), ignoring dependence on n. The complexity of obtaining all $\widehat{\mathrm{pr}}\left({\theta }_j\ne 0|y\right)$ by applying Algorithm 1 r/p times is thus $O\left\{(r/p)\left(q+2^p{p}^3\right)\right\}=O\left\{(r/p)\left(r-p+2^p{p}^3\right)\right\}$, ignoring dependence on n and K. For p = O(log r), this complexity reduces to $O\left(r^2{\log }^{2}r\right)$.

3·3. Non-parametric adjustment for covariates

As a last example, let η_i = (g ○ f)(z_i) (i = 1, … , n) where $g:ℝ\to ℝ$ is a known, differentiable, non-linear function, $f:{ℝ}^q\to ℝ$ is an unknown function, $g○f:{ℝ}^q\to ℝ$ is g composed with f, and z_i is a q-dimensional feature vector. Then, η_i provides a non-parametric adjustment for the covariate z_i in performing inferences on the effect of x_i on y_i. Take f’s prior as a Gaussian process that induces a prior Π(η). Algorithm 1 applies if a Gaussian approximation $\widehat{\Pi }\left(M^{\text{T}}\eta |S^{\text{T}}y\right)$ is available: Submodel (3b) reduces to $S^{\text{T}}y~N\left\{S^{\text{T}}G(F),{\sigma }^2{I}_{n-p}\right\}$ where $F={\left\{f\left(z_1\right),\dots ,f\left(z_n\right)\right\}}^{\text{T}}$ and $G(F)={\left\{g\left(F_1\right),\dots ,g\left(F_n\right)\right\}}^{\text{T}}$, which is a non-linear Gaussian model as studied in Steinberg & Bonilla (2014). Linearizing G using a first-order Taylor series yields a Gauss-Newton algorithm for a Laplace approximation of Π(F | S^Ty) as detailed in § S7 of the Supplementary Material. Based on that approximation, compute $\widehat{\mu }$ and $\widehat{\Sigma }$ in (5), for instance by sampling F from a Laplace approximation $\widehat{\Pi }\left(F|S^{\text{T}}y\right)$ and computing the sample mean and covariance of M^TG(F) since $M^{\text{T}}\eta =M^{\text{T}}G(F)$.

4. Analysis of integrated rotated Gaussian approximation

4·1. Approximation accuracy

This section provides theoretical guarantees on the accuracy of our posterior approximation framework. We begin with a general upper bound in terms of the accuracy of the approximation for the projected nuisance parameter. For this, denote the distribution of the p-dimensional a + b where $b~N\left(0,{\sigma }^2{I}_p\right)$ by $\Pi (a)\ast N_{{\sigma }^2}$. Define the Kullback-Leibler divergence from Π(b) to Π(a) as $D\{\Pi (a)\Vert \Pi (b)\}=\int \log \{\pi (a)/\pi (b)\}d\Pi (a)$.

At a high level, it is clear that the accuracy of the approximation $\widehat{\Pi }\left(\beta |y\right)$ defined in (4) depends on the accuracy of the approximation $\widehat{\Pi }\left(M^{\text{T}}\eta |S^{\text{T}}y\right)$. The following result quantifies the nature of this dependence in the setting where the data are generated from the prior predictive distribution. This result applies generally for any approximation $\widehat{\Pi }\left(M^{\text{T}}\eta |S^{\text{T}}y\right)$ and thus includes the Gaussian approximation (5) used in Algorithm 1 as a special case.

Theorem 1. Let y be distributed according to the model in (1) with β ~ Π(β) and η ~ Π(η) distributed according to their priors. Conditional on any realization S^Ty, the posterior approximation $\widehat{\Pi }\left(\beta |y\right)$ described in (4) satisfies

$$E\left[D\{\Pi (\beta |y)\Vert \widehat{\Pi }(\beta |y)\}|S^{\text{T}}y\right]\le D\left\{\Pi \left(M^{\text{T}}\eta |S^{\text{T}}y\right)\ast N_{{\sigma }^2}\Vert \widehat{\Pi }\left(M^{\text{T}}\eta |S^{\text{T}}y\right)\ast N_{{\sigma }^2}\right\},$$

where the expectation on the left is with respect to the conditional distribution of y given S^Ty.

A particularly useful property of Theorem 1 is that the upper bound does not depend in any way on the prior Π(β). This differs from some of the related work on posterior approximation, such as Huggins et al. (2017), which requires additional smoothness constraints, and thus excludes certain priors such as the spike-and-slab prior in § 3·1. Another useful property of Theorem 1 is that it does not require any assumptions about the extent to which the exact posterior Π(M^Tη | S^Ty) is concentrated about the ground truth. As a consequence, this result is relevant for non-asymptotic settings where there may be high uncertainty about η.

4·2. Accuracy of the Gaussian approximation

Next, we provide theoretical justification for a Gaussian approximation to Π(M^Tη | S^Ty) by showing that such an approximation can be accurate even when the prior on η is highly non-Gaussian. Without loss of generality, we focus on the set-up of § 3·1 where the nuisance term has the form η = Zα with a known n × q feature matrix Z and unknown parameter vector α. In this setting, the projected nuisance parameter M^Tη can be expressed as M^TZα where M^TZ is a p × q matrix with p ≪ q. There are no constraints on the dimension n other than n ≥ p.

As motivation for a Gaussian approximation to the projected nuisance term, consider the special case where the conditional distribution Π(α | S^Ty) is a product measure with uniformly bounded second moments. Under regularity assumptions on the columns of M^TZ, the multivariate central limit theorem combined with the assumption p ≪ q implies that the distribution of the projection M^TZα is close to the Gaussian distribution with the same mean and covariance. By contrast, the unprojected n-dimensional nuisance term η = Zα can be very far from Gaussian, particularly if n is of a similar order to q.

More realistically, one may envision settings where the entries of Π(α | S^Ty) are not independent but are weakly correlated on average. In this case, the usual central limit theorem does not hold because one can construct counterexamples in which the normalized sum of dependent but uncorrelated variables is far from Gaussian. Nevertheless, a classic result due to Diaconis & Freedman (1984) suggests that these counterexamples are atypical. Specifically, if one considers a weighted linear combination of the entries in α, then approximate Gaussianity holds for most choices of the weights, where most is quantified with respect to the uniform measure on the sphere. The implications of this phenomenon have been studied extensively in the context of statistical inference (Hall & Li, 1993; Leeb, 2013), and Meckes (2012) and Reeves (2017) provide approximation bounds for the setting of multidimensional linear projections.

In the context of our approximation framework, these results imply that a Gaussian approximation is accurate for most, but not necessarily all, instances of the p × q feature matrix M^TZ. To make this statement mathematically precise, we consider the expected behavior when the rows of Z are drawn independently from the q-dimensional Gaussian distribution N(0, Λ) where Λ is positive definite. As in the rest of the paper, we assume that X is fixed and arbitrary. Under these assumptions, the rows of the projected matrices M^TZ and S^TZ are independent with the same distribution as in Z.

Our results depend on certain properties of the conditional distribution $\Pi \left(\alpha |S^{\text{T}}y,S^{\text{T}}Z\right)$. Let ξ and Ψ denote the mean and covariance of $\Pi \left(\alpha |S^{\text{T}}y,S^{\text{T}}Z\right)$, respectively. Define

$$m_1=E\left\{\left|\frac{{\Vert {\Lambda }^{\frac{1}{2}}(\alpha -\xi )\Vert }^2}{\mathrm{tr}(\Lambda \Psi )}-1\right||S^{\text{T}}y,S^{\text{T}}Z\right\},m_2=\frac{\mathrm{tr}\left\{{(\Lambda \Psi )}^2\right\}}{\mathrm{tr}{(\Lambda \Psi )}^2}.$$

The term m₁ provides a measure of the concentration of ${\Vert {\Lambda }^{1/2}\left(\alpha -\xi \right)\Vert }^2$ about its mean and satisfies 0 ≤ m₁ ≤ 2. The term m₂ provides a measure of the average correlation between the entries of Λ^1/2α and satisfies 1/q ≤ m₂ ≤ 1 with equality on the left when ΛΨ is proportional to the identity matrix and equality on the right when ΛΨ has rank one.

Given estimates $\widehat{\xi }$ and $\widehat{\Psi }$ that are functions of S^Ty and S^TZ, we consider the Gaussian approximation

$$\widehat{\Pi }\left(M^{\text{T}}\eta |S^{\text{T}}y,S^{\text{T}}Z\right)=N\left\{M^{\text{T}}Z\widehat{\xi },\text{tr}(\Lambda \widehat{\Psi })I_p\right\}.$$ (7)

The covariance is chosen independently of M^TZ and depends only on a scalar summary of the estimated covariance. The following result bounds the accuracy of this approximation in terms of the terms m₁ and m₂ and the accuracy of the estimated mean and covariance.

Theorem 2. Conditional on any S^Ty and S^TZ, the Gaussian approximation in (7) satisfies

$$E_{M^{\text{T}}Z}\left[D\left\{\Pi \left(M^{\text{T}}\eta |S^{\text{T}}y,S^{\text{T}}Z\right)\ast N_{{\sigma }^2}\Vert \widehat{\Pi }\left(M^{\text{T}}\eta |S^{\text{T}}y,S^{\text{T}}Z\right)\ast N_{{\sigma }^2}\right\}\right]\le {\delta }_1+{\delta }_2,$$

where the expectation is with respect to M^TZ and

$${\delta }_1=3p\left[m_1\log \left\{1+\frac{\text{tr}(\Lambda \Psi )}{{\sigma }^2}\right\}+m_2^{\frac{1}{4}}+m_2^{\frac{1}{2}}{\left\{1+\frac{3\text{tr}(\Lambda \Psi )}{{\sigma }^2}\right\}}^{\frac{p}{4}}\right],$$

$${\delta }_2=\frac{p{\Vert {\Lambda }^{\frac{1}{2}}(\xi -\widehat{\xi })\Vert }^2}{2{\sigma }^2}+\frac{p}{2{\sigma }^2}{\left\{\text{tr}{(\Lambda \Psi )}^{\frac{1}{2}}-\text{tr}{(\Lambda \widehat{\Psi })}^{\frac{1}{2}}\right\}}^2.$$

This result is meaningful when p ≪ q and the noise variance is non-negligible compared to the covariance of the nuisance term such the ratio tr(ΛΨ)/σ² is bounded from above. Then, δ₁ converges to zero as m₁ and m₂ become small. The term δ₂ quantifies the effect of mismatch between the first and second moments of Π(α | S^Ty, S^TZ) and their approximations. The dependence on the second moments appears only in the terms tr(ΛΨ) and $\text{tr}(\Lambda \widehat{\Psi })$. Thus, this bound can be small even if the approximation $\widehat{\Psi }$ is very different from the true covariance Ψ.

To illustrate the significance of our results, consider two scaling regimes. First, if n ≪ q then the same arguments used in the proof of Theorem 2 can be used to show that the distribution of the n-dimensional nuisance term η is also approximately Gaussian. Then, our approximation framework is well motivated, but does not differ fundamentally from existing approaches that apply a Laplace approximation directly on the unrotated data. The second, and more interesting, regime occurs when n ≈ q or n ⨠ q. Then, the n-dimensional nuisance term is non-Gaussian in general, because there exists a near isometry between η and α. Our approximation framework can provide significant gains by taking this non-Gaussianity into account when estimating the mean and covariance. Moreover, combining Theorems 1 and 2 provides an upper bound on the error of the approximation to the posterior of β described in Algorithm 1. In particular, if the approximations of the mean and covariance are accurate enough, then this approximation error converges to zero as the terms m₁ and m₂ become small.

4·3. Variable selection consistency

Finally, we provide guarantees for variable selection consistency of (6), which only considers β in contrast to § 3·2. Let the set γ ⊂ {1, … , p} contain all indices j such that β_j ≠ 0. Define γ⁰ analogously for a non-random β⁰. Variable selection consistency as in Fernández et al. (2001) and Liang et al. (2008) means that, for y ~ N (Xβ⁰ + η⁰, σ²I_n), the posterior probability of the true model γ⁰ converges to one, pr(γ = γ⁰ | y) → 1 as n → ∞ where p does not change with n. It is desirable for a posterior approximation to inherit this property. Monte Carlo approximations do, but only if they are run for an infinite amount of time. Our approximation bypasses the need for such sampling, instead requiring mean and variance estimation for (5), while inheriting the consistency property if $\widehat{\Pi }\left(M^T\eta |S^{T}y\right)$ concentrates appropriately. Relatedly, Ormerod et al. (2017) established such consistency for their variational Bayes algorithm. More recently, K. Ray and B. Szabó (arXiv:1904.07150) showed optimal convergence rates of variable selection using variational Bayes with different priors than we consider here.

Let the |γ|-dimensional vector β_γ consist of the elements in β with indices in γ, and the n × |γ| matrix X_γ consist of the columns of X with indices in γ. Then, specifying Π(γ) and Π(β_γ | γ) defines Π(β). We consider g-priors (Zellner, 1986):

$${\beta }_{\gamma }|\gamma ~N\left\{0,{\sigma }^2{g}_n{\left(X_{\gamma }^{\text{T}}{X}_{\gamma }\right)}^{-1}\right\},g_n\in (0,\infty ).$$ (8)

Liang et al. (2008) showed variable selection consistency for priors of this form. Our approximation inherits this property under the additional assumption (9) on $\widehat{\Pi }\left(M^{\text{T}}\eta |S^{\text{T}}y\right)$ and g_n. This is an assumption on g_n and σ² jointly since $\widehat{\Pi }\left(M^{\text{T}}\eta |S^{\text{T}}y\right)$ depends on σ². Otherwise, the sensitivity on σ² is limited since the property considers the asymptotic regime n → ∞, when the signal-to-noise ratio goes to infinity regardless of σ².

Theorem 3. Let Π(β_γ) be the g-prior on β_γ from (8). Assume that g_n in (8), Π(γ), and X satisfy $\text{pr}\left(\gamma ={\gamma }^0\right)>0$, ${\lim }_{n\to \infty }\Vert \{I_n-X_{\gamma }{\left(X_{\gamma }^{\text{T}}{X}_{\gamma }\right)}^{-1}{X}_{\gamma }^{\text{T}}\}X{\beta }^0\Vert /n>0$ for any γ not containing γ⁰, g_n → ∞, and log(g_n)/n → 0, which are standard assumptions used in Fernández et al. (2001) and Liang et al. (2008) as detailed in § S4 of the Supplementary Material. Let y be distributed according to the data-generating model in (1) with β and η fixed to β⁰ and η⁰, respectively. Assume that $\widehat{\Pi }\left(M^{\text{T}}\eta |S^{\text{T}}y\right)$ concentrates appropriately in that

$$\frac{{\Vert M^{\text{T}}\eta -M^{\text{T}}{\eta }^0\Vert }^2}{\log g_n}\to 0,$$ (9)

in probability with respect to $M^{\text{T}}\eta ~\widehat{\Pi }\left(M^{\text{T}}\eta |S^{\text{T}}y\right)$ and y. Let $\widehat{\Pi }\left(\beta |y\right)$ be as in (4). Then, $\widehat{\text{pr}}\left(\gamma ={\gamma }^0|y\right)\to 1$ in probability with respect to y as n → ∞.

5. Simulation studies and applications

5·1. Non-parametric adjustment for covariates

Consider the set-up from § 3·3 with g(a) = a² and q = 1. We assign $f:ℝ\to ℝ$ a zero-mean Gaussian process prior with a squared exponential covariance function such that $\mathrm{cov}\left\{f\left(z_i\right),f\left(z_j\right)\right\}=\exp \left\{-{\left(z_i-z_j\right)}^2/10\right\}(i,j=1,\dots ,n)$, and β ~ N(0, 16I_p). Set n = 100, p = 3, and σ² = 1. We draw the rows of X independently from N(0_p×1, Φ) where Φ is a Toeplitz matrix defined so that its first row equals (0.9⁰, … , 0.9^p). Then, the columns of X are correlated. The features z_i (i = 1, … , n) equal the ith element of the first column of X. Generate y according to (1) with f equal to a draw from its prior distribution and β = (4, −4, 4)^T.

We approximate the posterior Π(β | y) using a random walk Metropolis-Hastings algorithm on f with 10,000 burnin and 90,000 recorded iterations. We marginalize out β since Π(β | f, y) is analytically available, allowing approximation of π(β | y) with samples from Π(f | y). Algorithm 1 also provides $\widehat{\Pi }\left(\beta |y\right)$ per § 3·3. Lastly, ignoring the non-parametric nuisance parameter by setting η_i = (g ○ f)(z_i) = 0 yields a simpler approximation. The Metropolis-Hastings algorithm took 6 minutes while our method finished in 2 seconds. The resulting posterior density estimates for β_j (j = 1, … , p) are in Fig. 1. Taking the Metropolis-Hastings estimate as the gold standard, our method yields an approximation that matches the location and spread of the posterior better than the result from ignoring the non-parametric nuisance term η.

Fig. 1.

Marginal posterior density estimates from the simulation in § 5·1 with the solid line representing the Gibbs estimate π(β_j | y), the thick dotted line the estimate $\widehat{\pi }\left({\beta }_j|y\right)$ from Algorithm 1, and the thin dotted line the estimate resulting from ignoring the nuisance parameter.

5·2. Bayesian variable selection

We consider the diabetes data from Efron et al. (2004) as it is a popular example of variable selection with collinear predictors (Park & Casella, 2008; Polson et al., 2013). The outcome y measures disease progression one year after baseline for n = 442 patients with diabetes. The r = 64 predictors come from 10 covariates with their squares and interactions. The outcome and predictors are standardized to have zero mean and unit norm. Consider the variable selection set-up from § 3·2 with prior inclusion probability λ = 1/2 and ψ = 1. Usually, one would not use scalable approximations for such a moderate-dimensional problem as a Gibbs sampler can provide accurate estimates. The latter is why we include it here as these accurate estimates enable assessment of the approximation accuracy of scalable methods.

We estimate the posterior inclusion probabilities pr(θ_j ≠ 0 | y) (j = 1, … , r) using 1) a Gibbs sampler with 10,000 burnin and 90,000 recorded iterations, Algorithm 1 as described in § 3·2 using 2) vector approximate message passing and 3) the debiased lasso in Step 2 with p = 4 as suggested by p = O(log r) and parallelization across 8 CPU cores, 4) expectation propagation as in Hernández-Lobato et al. (2015), and 5) variational Bayes as in Carbonetto & Stephens (2012). To implement expectation propagation and variational Bayes, we used the R code from https://jmhl.org/publications/ dated January 2010 and the R package ‘varbvs’ version 2.5-7, respectively. Results from the variational Bayes algorithm by Ormerod et al. (2017) are omitted as the method from Carbonetto & Stephens (2012) outperforms it in the scenarios that we consider. Since the error variance is unknown, we assign it the prior 1/σ² ~ Ga(1, 1), a gamma distribution with unit shape and rate parameter. The Gibbs sampler incorporates this prior. Algorithm 1 estimates σ² as described in § 2·4, and § S5·3 and § S6 of the Supplementary Material. Expectation propagation estimates σ² by maximizing approximate evidence (Hernández-Lobato etal., 2015). The R package ‘varbvs’ (Carbonetto & Stephens, 2012) uses approximate maximum likelihood for σ² within the variational Bayes method.

As discussed in § 3·2, determining whether posterior inclusion probabilities from a Gibbs sampler are accurate is non-trivial. Overlapping batch means (Flegal & Jones, 2010, § 3) estimates their average Monte Carlo standard error as 0.0015 in this application.

Table 1 focuses on the errors in the posterior inclusion probability estimates. An approximation error of 0.01 is worse when the inclusion probability is 0.01 versus 0.5. We therefore transform the probabilities to log odds. Our method with vector approximate message passing outperforms expectation propagation and variational Bayes as its error is lowest in Table 1, though at a higher computational cost. Our method is slowest but still considerably faster than the Gibbs sampler which took 11 minutes to run. Since the debiased lasso yielded the worst approximation, we do not consider it in the remainder of this article.

Table 1.

Summary statistics of the absolute difference between the Gibbs sampler estimates and the approximations of the posterior log odds of inclusion for the application in § 5·2 with computation times. IRGA and VAMP stand for integrated rotated Gaussian approximation and vector approximate message passing, respectively.

Method	Min	Q1	Median	Q3	Max	Mean	Computation time (seconds)
IRGA with VAMP	0.003	0.036	0.076	0.133	10.7	0.599	4.1
IRGA with the debiased lasso	0.003	0.100	0.142	0.199	7.85	0.470	3.8
Expectation propagation	0.003	0.061	0.109	0.168	11.9	0.666	0.8
Variational Bayes	0.002	0.093	0.124	0.166	11.6	0.667	1.0

5·3. Controlling for single-nucleotide polymorphisms

The Geuvadis dataset from Lappalainen et al. (2013), available at https://www.ebi.ac.uk/Tools/geuvadis-das, contains gene expression data from lymphoblastoid cell lines of n = 462 individuals from the 1000 Genomes Project along with roughly 38 million SNPs. We focus on the gene E2F2, ensemble ID ENSG00000007968, as it plays a key role in the cell cycle (Attwooll et al., 2004). Our focus is on assessing whether expression differs between populations, even after adjusting for genetic variation between individuals. Specifically, we compare people from British descent with the four other populations given in Table 2. If such differences occur, they can be presumed to be due to environmental factors that differ between these populations and that relate to E2F2 expression. We therefore consider the set-up from § 3·1 with y the E2F2 gene expressions, X demographic factors, and Z containing SNPs we would like to control for.

Table 2.

Posterior inclusion probabilities for the demographic factors from the application in § 5·3. IRGA stands for integrated rotated Gaussian approximation.

		Population
Method	Gender	Utahn of European ancestry	Finnish	Tuscan	Yoruba
IRGA	0.83	0.96	0.96	0.92	0.00
Ignoring the SNPs	0.73	0.07	0.04	0.20	0.49

The demographics in X are gender and the 4 populations with British as the reference group. The matrix X thus has p = 5 columns. The covariates Z consist of q = 10, 000 SNPs selected using sure independence screening (Fan & Lv, 2008) as vector approximate message passing on all 38 million SNPs was infeasible. We standardize y and the columns of X and Z to have zero mean and unit variance. To complete the set-up from § 3·1, set λ = n/(10 q) and ψ = 1/n for the spike-and-slab prior on α while Π(β) is a spike-and-slab with prior inclusion probability 1/2 and slab variance 1 such that, a priori, the SNPs do not capture more variation in the outcome than the demographic factors. This may provide a reasonable default for SNP data, but in other settings, hyperparameter values should be reconsidered. Vector approximate message passing estimates σ² using the prior 1/σ² ~ Ga(1, 1) and employs damping to achieve convergence in this application, as described in § S5·3 and § S5·4 of the Supplementary Material, respectively.

Table 2 contains the resulting posterior inclusion probabilities for the demographic factors, also when not controlling for the SNPs. The results vary hugely by whether SNPs are controlled for, with more evidence of a difference in the expression of gene E2F2 by population when controlling for SNPs using Algorithm 1. Section S10 of the Supplementary Material contains additional comparisons with other high-dimensional inference methods.

Section S8 of the Supplementary Material contains additional simulation studies. They further show that integrated rotated Gaussian approximation outperforms variational Bayes and either beats or is on par with expectation propagation in terms of approximation accuracy. This improved accuracy comes with increased computational cost for our method in certain scenarios.

6. Discussion

Although our focus was Bayesian inference, our method marginalizes out nuisance parameters from the likelihood for β as an intermediate step. This approximate likelihood from (6) can be useful in frequentist inference. It is well known that priors used in Bayesian inference correspond to penalties in frequentist inference. One can think of the log prior for the nuisance parameter η as a penalty on η. An L₂ penalty might not be ideal due to the complex or high-dimensional nature of η. Instead, one might want to use sparsity-inducing penalties, such as L₁ or the non-convex smoothly clipped absolute deviation, which come with attractive theoretical properties (Pötscher & Leeb, 2009) but can be computationally challenging. Our method obtains the marginal likelihood for β with such penalties on η, resolving the main computational bottleneck for frequentist inference on β in the model of interest in (1).