Priors in Whole-Genome Regression: The Bayesian Alphabet Returns

Abstract

Whole-genome enabled prediction of complex traits has received enormous attention in animal and plant breeding and is making inroads into human and even Drosophila genetics. The term “Bayesian alphabet” denotes a growing number of letters of the alphabet used to denote various Bayesian linear regressions that differ in the priors adopted, while sharing the same sampling model. We explore the role of the prior distribution in whole-genome regression models for dissecting complex traits in what is now a standard situation with genomic data where the number of unknown parameters (p) typically exceeds sample size (n). Members of the alphabet aim to confront this overparameterization in various manners, but it is shown here that the prior is always influential, unless n ≫ p. This happens because parameters are not likelihood identified, so Bayesian learning is imperfect. Since inferences are not devoid of the influence of the prior, claims about genetic architecture from these methods should be taken with caution. However, all such procedures may deliver reasonable predictions of complex traits, provided that some parameters (“tuning knobs”) are assessed via a properly conducted cross-validation. It is concluded that members of the alphabet have a room in whole-genome prediction of phenotypes, but have somewhat doubtful inferential value, at least when sample size is such that n ≪ p.

WHOLE-genome enabled prediction of complex traits has received much attention in animal and plant breeding (e.g., Meuwissen et al. 2001; Heffner et al. 2009; Lorenz et al. 2011; de los Campos et al. 2012a; Heslot et al. 2012) and is making inroads into human and even Drosophila genetics (e.g., de los Campos et al. 2010, 2012b; Makowsky et al. 2011; Ober et al. 2012; Vázquez et al. 2012). This approach is known as “genomic selection” in breeding of agricultural species. The term “Bayesian alphabet” was coined by Gianola et al. (2009) to refer to a (growing) number of letters of the alphabet used to denote various Bayesian linear regressions used in genomic selection that differ in the priors adopted while sharing the same sampling model: a Gaussian distribution with mean vector represented by a regression on p markers, typically SNPs, and a residual variance, ${\sigma }_{\text{e}}^2$. A recent review of some of these methods is in de los Campos et al. (2012a). In addition to prediction, this whole-genome approach lends itself to investigation of “genetic architecture,” often defined as the number of genes affecting a quantitative trait, the allelic effects on phenotypes, and the frequency distribution spectrum of alleles at these genes (e.g., Hill 2012). If epistasis and pleiotropy are brought into the picture, this definition of genetic architecture needs to be expanded significantly.

Most researchers in genomic selection are familiar with most letters of the alphabet, but we provide a brief review of its ontogeny. The alphabet started with Bayes A and B (Meuwissen et al. 2001), but there has been rapid expansion since, as illustrated by the Bayes Cπ and Dπ methods (Habier et al. 2011). Apart from between-letter variation, there is also variation within letters, such as fast EM-Bayes A (Sun et al. 2012), fast Bayes B (Meuwissen et al. 2009), and BRR (Bayesian ridge regression on markers), which is equivalent to G-BLUP (Van Raden 2008) but with variance parameters estimated Bayesianly; the equivalence between G-BLUP and ridge regression is given, for example, in de los Campos et al. (2009a,b). The letter D has several variants: Bayes D0, D1, D2, and D3 (Wellman and Bennewitz 2012).

Here, L is used to denote the Bayesian Lasso (Park and Casella 2008; de los Campos et al. 2009a,b) while L1 and L2 can be used to refer to variants due to Legarra et al. (2011). There is also the EL Bayesian Lasso of Mutshinda and Sillanpää (2010), with EL standing for “extended Lasso.” An almost empty hiatus spans from letters D to R (Erbe et al. 2012) with Bayes RS emerging even more recently (Brondum et al. 2012). Wang et al. (2013) presented Bayes TA, TB, and TCπ, extensions of the corresponding letters to threshold models. The upper bound of the alphabet seems to have been defined by Larry Schaeffer (personal communication, Interbull Meeting, Guelph, 2011) when he threatened attendants of this conference with Bayes Z-Δ, although full details have not been published yet. The preceding review may not be comprehensive, as there may be other members of the alphabet that are unknown to the author. It is tempting to conjecture that there may be issues with individual members of the alphabet, as this continued growth is suggestive of a state of lack of satisfaction with any given letter.

This article explores the role of the prior distribution in whole-genome regression models for predicting or dissecting complex traits. In particular, we address a standard situation encountered in genomic selection: with genomic data, the number of unknown parameters exceeds sample size. Section General Setting presents the regression model and reminds readers that, for the preceding situation, regression coefficients on marker genotypes are not identified in the likelihood function so that the data do not contain information for inference that is uncontaminated from the effects of the prior, except in a subspace. Bayesian methods for confronting the blatant overparameterization of genomic selection models are reviewed in this section, where it is shown that the prior is always influential in this setting. The section Bayesian Shrinkage discusses how ridge regression produces frequency-dependent shrinkage, while Bayes A, Bayes B, Bayes L, and Bayes R effect a type of shrinkage that is both allelic frequency and effect-size dependent. After establishing in the preceding sections, hopefully in a firm manner, that all members of the alphabet do not lead to inferences that are devoid of the influence of the prior, it is argued in the Discussion that all such methods may deliver reasonable predictions of complex traits, provided that some parameters (“tuning knobs”) are assessed properly. It is concluded that, while members of the alphabet cannot be construed as providing solid inferences about “genetic architecture,” they do have a room in whole-genome prediction of phenotypes.

General Setting

Let y be an n × 1 vector of target responses (e.g., phenotypes, preprocessed data). Using molecular markers, all members of the alphabet pose the same linear regression of phenotypes on marker codes, that is

$$y=X\beta +e,$$ (1)

where X is an n × p matrix of marker codes (e.g., −1, 0, 1 for aa, Aa, and AA genotypes, respectively); when additive action is assumed, β = {β_j} is a vector of allelic substitution effects for each of p markers, and e is a vector of residuals typically assigned the normal distribution $e|{\sigma }_{\text{e}}^2\sim N\left(e|0,I{\sigma }_{\text{e}}^2\right)$ where ${\sigma }_{\text{e}}^2$ is the residual variance, defined earlier.

In the standard additive model of quantitative genetics (e.g., Falconer and Mackay 1996), the β_j are fixed parameters, while the elements x_ij of X are random variables; e.g., members of the jth column of X may be realizations from a Hardy–Weinberg distribution with corealizations in columns j and j′ reflecting some linkage disequilibrium distribution. The maximum-likelihood estimator of β treats X as a fixed matrix and satisfies the system of equations

$$X^{\prime }X{\beta }^{\left(0\right)}=X^{\prime }y,$$

where β⁽⁰⁾ may not be a unique solution (Searle 1971). If n < p X′X is singular so the maximum-likelihood estimator is not unique, as there is an infinite number of solutions to the equations above. Letting (X′X)⁻ be a generalized inverse of X′X, one solution is β⁽⁰⁾ = (X′X)⁻X′y with expectation E(β⁽⁰⁾|β) = (X′X)⁻X′Xβ, producing a biased estimator of β, with at least p − n of its elements being equal to 0. On the other hand, E(y|β) = Xβ = g (the genetic signal captured by markers) is estimated uniquely because its estimator, Xβ⁽⁰⁾ is unique, although this reproduces y exactly in the n < p situation. Fisher’s information content about β in the sample is $X\prime X{\sigma }_{\text{e}}^{-2}$ and, because this matrix is singular, one cannot speak about information pertaining to individual marker effects in a strict sense. However, the information content about g = {g_i} is $I{\sigma }_{\text{e}}^{-2}$, meaning that information about each genotypic value g_i is proportional to that conveyed by a sample of size 1. Hence, in an n < p model, maximum-likelihood cannot be used either as an inferential or as a predictive machine. In the latter case, it does not generalize to new samples, because it copies both noise and signal contained in model training data.

In their proposals for employing whole-genome markers in a linear regression model, Meuwissen et al. (2001) were inspired by the fact that animal breeders had dealt with the n ≪ p problem successfully in the context of predicting random effects via best linear unbiased prediction (BLUP); see Henderson (1984) for a review, with a gentler treatment in Mrode (2005). BLUP assumes that marker effects are drawn from some distribution with known variance components; only knowledge of the covariance structure is needed and the form of the distribution is immaterial, although a linear model must hold. An alternative is provided by the Bayesian treatment but, here, the meaning of probability and the manner in which unknowns are inferred are different from their frequentist counterparts (Gianola and Fernando 1986; Robinson 1991). The distinctions between these two views are emphasized next, but the two approaches confront n ≪ p by bringing external information to the problem, as noted early in the game by Robertson (1955).

The BLUP approach to whole-genome prediction assumes that β has a null mean vector and some known covariance matrix V_β; then E(y) = 0 and the best linear unbiased predictor of β is

$$\text{BLUP}\left(\beta \right)=V_{\beta }{X}^{\prime }{\left(X{V}_{\beta }{X}^{\prime }+I{\sigma }_e^2\right)}^{-1}y.$$

Here, β is regarded as a random draw from the distribution indicated above so, on average E_y[BLUP(β)] = 0, meaning that BLUP(β) is unbiased with respect to the mean of the random effects distribution, E(β|V_β). BLUP envisages a sampling scheme where one draws a different realization of marker effects in every repetition of the sampling, such that, over all repetitions, 0 is obtained, on average. BLUP estimates zero without bias! However, when one is interested about individual marker effects (or about the genetic values of a given individual), the inference to be made pertains to the specific item of interest, and not to the average of their distribution. If so, BLUP is biased with respect to specific marker effects (the classical fixed model of quantitative genetics) because

$$E\left[\text{BLUP}\left(\beta \right)|\beta \right]=V_{\beta }{X}^{\prime }{\left(X{V}_{\beta }{X}^{\prime }+I{\sigma }_e^2\right)}^{-1}X\beta ,$$

so that the bias is $\left[I_p-V_{\beta }{X}^{\prime }{\left(X{V}_{\beta }{X}^{\prime }+I{\sigma }_{\text{e}}^2\right)}^{-1}X\right]\beta ,$ where I_p is an identity matrix of order p. The random effects treatment results in that BLUP(β) is unique whether n ≪ p or not, but it produces a biased estimator of marker effects; this bias never disappears when n ≪ p. On the other hand, if n → ∞ and p stays fixed, bias goes away, given that the model is true. A toy example of the bias of BLUP with respect to the true, fixed, substitution effects is shown in the Appendix (Bias of BLUP with respect to marker effects).

For the n ≪ p situation, Fan and Li (2001) discuss estimators that induce sparsity. However, to meet their so-called “oracle properties” (e.g., asymptotic unbiasedness), Fisher’s information matrix must be nonsingular for p₀ nonzero parameters (p₀ < n), these being the “true” effects of some markers on quantitative traits. BLUP and most members of the Bayesian alphabet do not produce a sparse model automatically; rather, they produce shrinkage of regression coefficients. Consider a sequence of P models of increasing dimensionality fitted to the same data, with p₀ < p₁ < p₂ …< p_P. The size of the “true” signal is dictated by the “true” effects p₀ and the size of the models could be viewed as corresponding to the number of SNPs in platforms of increasing density applied to the same data set. As marker density increases while n and p₀ remain fixed, estimates of marker effects must become necessarily smaller. How can “true” effects be learned properly if the model forces estimates to become smaller as p grows? Given the rhythm of technology, it is unlikely that we will reach the situation where n ≫ p_P. At this point there is not much hope for learning marker effects in a manner that is free from making additional untestable assumptions. A minor complication: for the oracle properties to hold the true model must be “hit.” This is probably an unrealistic proposal when dealing with complex traits where many difficulties arise; for example, linkage disequilibrium creates ambiguity because many markers can act as proxies for others and complex forms of epistasis are bound to produce havoc in a naive linear model on additive effects.

One way of tackling the n ≪ p problem is by introducing restrictions on the size of the regression coefficients, i.e., shrinkage or “regularization.” In the machine-learning literature this is attained via ad hoc penalty functions that produce regularization (e.g., Bishop 2006; Hastie et al. 2009). Bayesian methods with proper priors produce regularization automatically, to an extent that depends on the prior adopted. The various members of the Bayesian alphabet effect shrinkage in different manners, an issue explored subsequently in this article. Let all unknown parameters of a model be represented by θ = (θ₁, θ₂), where θ₁ and θ₂ denote distinct parameters, e.g., marker effects and their apparent (the reason for these terms is made clear later) variances, respectively, in Bayes A. The posterior distribution of θ (assume, for simplicity, that the residual variance is known) is

$$p\left({\mathit{\theta }}_1,{\mathit{\theta }}_2|y,{\sigma }_{\text{e}}^2,H\right)\propto p\left(y|{\mathit{\theta }}_1,{\mathit{\theta }}_2,{\sigma }_{\text{e}}^2,H\right)p\left({\mathit{\theta }}_1,{\mathit{\theta }}_2|H\right)$$ (2)

$$\propto p\left(y|{\mathit{\theta }}_1,{\sigma }_{\text{e}}^2\right)p\left({\mathit{\theta }}_1,{\mathit{\theta }}_2|H\right).$$ (3)

Above, H is a set of more or less arbitrarily specified hyperparameters. Expression (3) results from the assumption that, given location effects θ₁ (e.g., allelic substitution effects), the data are conditionally independent of θ₂. Further,

$$p\left({\mathit{\theta }}_1,{\mathit{\theta }}_2|H\right)=p\left({\mathit{\theta }}_1|{\mathit{\theta }}_2,H\right)p\left({\mathit{\theta }}_2|H\right)=p\left({\mathit{\theta }}_1|{\mathit{\theta }}_2\right)p\left({\mathit{\theta }}_2|H\right),$$

with the expression on the right side resulting because, given θ₂, location effects θ₁ do not depend on H (e.g., Sorensen and Gianola 2002). Note that p(θ₁|θ₂) is a conditional prior distribution, while the marginal prior distribution actually assigned to θ₁ is

$$p\left({\theta }_1|H\right)={\displaystyle \int p\left({\mathit{\theta }}_1|{\mathit{\theta }}_2,H\right)p\left({\mathit{\theta }}_2|H\right)d{\mathit{\theta }}_2.}$$ (4)

Likewise, p(θ₁|y, H) and p(g(θ₁)|y, H) denote the marginal posterior distributions of θ₁ and g(θ₁), the latter being any function of θ₁. For example, if θ₁ is the vector β, one may be interested in the posterior distribution of Xβ, the marked signal. The results of a Bayesian analysis should not be interpreted from a frequentist perspective, as the meaning of probability is different in the two camps (Bernardo and Smith 1994; O’Hagan 1994; Sorensen and Gianola 2002). For example, BLUP is an unbiased predictor in conceptual repeated sampling, but corresponds to the posterior mean of marker effects in a Bayesian Gaussian model with known covariance structure. In the latter, the data are fixed; in the BLUP setting, the data vary at random.

An important issue is the influence of priors on inference. Theory on Bayesian asymptotics dictates that, as sample size grows, the influence of the prior vanishes gradually. In the limit, the posterior distribution becomes normal, centered at the maximum-likelihood estimator and with covariance matrix given by the inverse of Fisher’s information measure, so the prior matters little in large samples (Bernardo and Smith 1994). However, this result holds for parameters that are likelihood identifiable, i.e., when their maximum-likelihood estimator exists, but it must be kept in mind that markers are not QTL, so the marker-based model is arguably wrong. In an n ≪ p setting, true Bayesian learning can take place for at most n parameters or functions thereof, since p − n parameters are unidentified. Gelfand and Sahu (1999) show that one can learn about at most n linearly independent functions of marker effects, such as $x_i^{\prime }\beta$. Carlin and Louis (1996) and Sorensen and Gianola (2002) give an example where the marginal posterior distributions of unidentified parameters exist if these are assigned proper priors; however, the priors will always matter and their influence will never vanish asymptotically. In the n ≪ p setting, inferences about marker effects (often referred to as learning genetic architecture, e.g., inferring effects of some QTL) are always influenced by the priors adopted, apart from the fact that the model is wrong, as argued above. This means that stories that can be made from posterior distributions will depend on stories that are made a priori. For example, Lehermeier et al. (2013) demonstrated the influence of priors on predictive ability from various Bayesian models (Bayes A, B, L) with simulated and empirical data. Also, Gianola et al. (2009) showed that the priors in Bayes A and B drive inferences on variances of marker-specific effects.

A formal verification that individual marker effects are not identified from a Bayesian perspective using a definition by Dawid (1979) is presented in the Appendix (Marker effects are not identified from a Bayesian perspective in the n < p setting); this holds for any model, linear or nonlinear. A proof that is specific to the linear regression model on p markers with sample size n is given in the Appendix as well (Inferences in a linear model with unidentified parameters); there, it is shown that what is learned about β is a function of what is learned about Xβ. In other words, Bayesian learning occurs for n items but then this knowledge is “distributed” into p pieces via the relationship between β and Xβ induced by the prior.

In summary, proper Bayesian learning from data in a linear regression model with n < p takes place only for linear combinations of marker effects that are identified in the likelihood, that is, estimable. Any other marker effects or linear combinations thereof are redundant in the sampling model, but their posterior distributions exist and the posterior mean will differ from the prior mean. It follows that mechanistic conjectures about genetic architecture in the n < p situation are, to a large extent, driven by prior assumptions and not by data. This observation has been corroborated empirically (e.g., Heslot et al. 2012; Ober et al. 2012; Lehermeier et al. 2013): Bayesian models differing in their prior produce different inferences about individual marker effects, but most often deliver similar predictive abilities if tuned properly. Not surprisingly, the posterior distributions of $x_i^{\prime }\beta$ (the signal to be predicted for datum i) from varying models are more similar to each other than the corresponding priors, as this function is likelihood identifiable. In short, extant theory says that given that a model is “true” (oracle principle 1, Fan and Li 2001), the posterior mean of an identifiable parameter or of a likelihood-identified combination of parameters will converge to its true value, including any “true zero,” as sample size goes to infinity (oracle principle 2). This works for n > p or for some estimators where sparsity is built in automatically, but the model must be true; Fan and Li (2001) describe several such estimators.

A situation in which proper Bayesian learning can take place is presented in the Appendix (An example of proper Bayesian learning).

Bayesian Shrinkage

Given that learning about genetic architecture without contamination from effects of the prior does not take place whenever n ≪ p, a question is what the various different members of the alphabet do. We examine ridge regression (BLUP), Bayes A, Bayes B, Bayes L, and Bayes R and also give a warning about a commonly used description of a specific prior; these procedures are prototypical so there is no need to consider other letters of the alphabet. All these methods have been reported in the genomic selection literature. Since the marginal posterior distribution of marker effects (with the exception of that of BLUP under normality) cannot be arrived at analytically, the methods are appraised from a heuristic perspective.

BLUP (ridge regression)

The vector of marker effects β is assigned the normal prior $N\left(\beta |0,I{\sigma }_{\beta }^2\right)$. The structure of the problem is well known and the mixed model equations leading to BLUP satisfy

$$\left(X^{\prime }X+I\lambda \right)\tilde{\beta }=X^{\prime }y=X^{\prime }X{\beta }^{\left(0\right)},$$ (5)

where $\tilde{\beta }=\text{BLUP}\left(\beta \right),$ $\lambda ={\sigma }_{\text{e}}^2/{\sigma }_{\beta }^2$ is the variance ratio, and β⁽⁰⁾ = (X′X)⁻ X′y is as before. One can write

$$\tilde{\beta }={\left(X^{\prime }X+I\lambda \right)}^{-1}\left(X^{\prime }X{\beta }^{\left(0\right)}+\lambda \times 0\right),$$ (6)

so $\tilde{\beta }$ (unique) is a matrix weighted average of solution β⁽⁰⁾ (not unique) and of the prior mean 0, where the weights are X′X and λ, respectively. For fixed p, as n increases, the rank of X′X will increase, eventually exceeding p and, in the limit, the posterior distribution will be centered at the unique maximum-likelihood estimator (by consistency, this will converge to the “true” value of β, given the model).

Representation (5) suggests that the same amount of shrinkage is effected to all p markers (because the same variance ratio λ is added to every diagonal element of X′X), but this is not the case. This is clear from Equation 6 where, for each marker effect, the contributions from the data vary over markers; this is more transparent from inspection of the solutions in scalar form. For marker 1, as an example, the estimator of substitution effect is

$${\tilde{\beta }}_1=\frac{{\displaystyle {\sum }_{i=1}^n{x}_{i1}\left(y_i-x_{i2}{\tilde{\beta }}_2-\dots -x_{ip}{\tilde{\beta }}_p\right)}}{{\displaystyle {\sum }_{i=1}^n{x}_{i1}^2+\lambda }}=\frac{{\displaystyle {\sum }_{i=1}^n{x}_{i1}^2\hspace{0.17em}{\overleftrightarrow{\beta }}_1+\lambda \times 0}}{{\displaystyle {\sum }_{i=1}^n{x}_{i1}^2+\lambda }},$$

where

$${\overleftrightarrow{\beta }}_1=\frac{{\displaystyle {\sum }_{i=1}^n{x}_{i1}\left(y_i-x_{i2}{\tilde{\beta }}_2-\dots -x_{ip}{\tilde{\beta }}_p\right)}}{{\displaystyle {\sum }_{i=1}^n{x}_{i1}^2}}.$$

Then, the BLUP ${\tilde{\beta }}_1$ of the allele substitution effect can be viewed, heuristically, as a weighted average of a “data driven” estimate $\left({\overleftrightarrow{\beta }}_1\right)$ and of the mean of the prior distribution (0) where the respective weights are ${\displaystyle {\sum }_{i=1}^n{x}_{i1}^2}/{\displaystyle ({\sum }_{i=1}^n{x}_{i1}^2+\lambda )}$ and $\lambda /{\displaystyle ({\sum }_{i=1}^n{x}_{i1}^2+\lambda )},$ respectively. This suggests less shrinkage toward zero for markers (j, say) having larger values of ${\displaystyle {\sum }_{i=1}^n{x}_{ij}^2}$. Now, if for any marker genotypes are coded as −1, 0, 1 for aa, Aa, and AA, respectively, it follows (assuming Hardy–Weinberg proportions and centered marker codes) that $E\left(x_{ij}^2\right)=\text{Var}\left(x_{ij}\right)=2p_j\left(1-p_j\right)$ so $E\left({\displaystyle {\sum }_{i=1}^n{x}_{ij}^2}\right)=2p_j\left(1-p_j\right)n,$ where p_j is the frequency of the A-type allele at that locus. Hence, at fixed sample size n, BLUP effects less shrinkage toward zero of markers that have intermediate allelic frequencies, simply because 2p_j(1 − p_j) is maximum at $p_j=\frac{1}{2}$. To illustrate, we use this Hardy–Weiberg approximation and plot (Figure 1) the weight “assigned to the data” for marker j

$$W_j=\frac{{\displaystyle {\sum }_{i=1}^n{x}_{ij}^2}}{{\displaystyle {\sum }_{i=1}^n{x}_{ij}^2+\lambda }}\approx \frac{2p_j\left(1-p_j\right)}{2p_j\left(1-p_j\right)+\lambda /n},$$

against allelic frequency at $\lambda /n=1,0.1,0.01$ and 0.001, respectively. As depicted in Figure 1, the extent of shrinkage is frequency and sample size dependent, with some differential shrinkage (bottom two curves) taking place at large values of $\lambda /n$, that is, at small sample sizes, but with little or no differential shrinkage otherwise, unless alleles are rare. Then, the often-made statement that BLUP or ridge regression perform an homogeneous shrinkage of marker effects is not correct. In short, shrinkage is frequency and sample size dependent but effect-size independent.

Figure 1.

Approximate weight (W) assigned to the data (the weight assigned to prior information is 1 − W) as a function of allelic frequency at a marker locus. From top to bottom the lines give the trajectory for λ/n values of 0.001 (solid line), 0.01 (short dashes), 0.1 (long dashes), and 1 (dots).

Bayes A

Bayes A (Meuwissen et al. 2001) consists of a three-stage hierarchical model. The first stage is the normal regression (1); the second stage assigns a normal conditional prior to each of the marker effects, all possessing a null mean but with a variance that is specific to each marker; the third stage assigns the same scaled inverted chi-square distribution with known scale $\left(S_{\beta }^2\right)$ and degrees of freedom (ν) parameters to each of the marker variances. The mechanistic argument for the Bayes A prior was that markers may contribute differentially to genetic variance (they do, to an extent depending on their effects, allelic frequencies, and linkage disequilibrium relationships with causal variants), so it seemed a good idea to “estimate” such variances. There are two difficulties: the first one is that the marginal prior for the markers effects, resulting from deconditioning the second stage over the third stage as done in (4), is the same for all markers. Second, there is scant Bayesian learning for marker-specific variances. This was pointed out by Gianola et al. (2009), who showed that all markers have the same prior distribution: a $t\left({\beta }_j|0,\nu ,S_{\beta }^2\right)$ process with null mean and variance $\nu S_{\beta }^2/\left(\nu -2\right)$. Given that this prior is homoscedastic over markers, why is it that it behaves differently from ridge regression BLUP, where all individual marker effects are assigned the prior $N\left({\beta }_j|0,{\sigma }_{\beta }^2\right)?$

In Bayes A, the marginal posterior distribution of marker effects cannot be arrived at in closed form, but insight can be obtained from inspection of the joint mode of the posterior distribution of β, assuming that the residual variance is known; recall that $S_{\beta }^2$ and ν are known hyperparameters in Bayes A. The hierarchical model is then

$$\begin{array}{c}{y}_i|\beta ,{\sigma }_{\text{e}}^2\sim N\left(y_i|x_i^{\prime }\beta ,{\sigma }_{\text{e}}^2\right),i=1,2,\dots ,n; {\beta }_j|S_{\beta }^2,\nu \\ \sim \text{IID} t\left({\beta }_j|0,S_{\beta }^2,\nu \right), j=1,2,\dots ,p,\end{array}$$

where $x_i^{\prime }$ is the ith row of X. Conditionally on ${\sigma }_{\text{e}}^2$, $S_{\beta }^2$ and ν, the joint posterior density is

$$p\left(\beta |S_{\beta }^2,\nu ,{\sigma }_{\text{e}}^2,y\right)\propto {\displaystyle \prod _{i=1}^n\text{exp}\left[-\frac{1}{2{\sigma }_{\text{e}}^2}{\left(y_i-x_i^{\prime }\beta \right)}^2\right]}{\displaystyle \prod _{j=1}^p{\left[1+\frac{{\beta }_j^2}{S_{\beta }^2\nu }\right]}^{-\left(1+\nu \right)/2}.}$$ (7)

Using results presented in the Appendix (Mode of the conditional posterior distribution in Bayes A), an iterative scheme for locating a mode of (7) is given by

$${\beta }^{\left[t+1\right]}={\left(X^{\prime }X+W_{\beta }^{\left[t\right]}\right)}^{-1}{X}^{\prime }y={\left(X^{\prime }X+W_{\beta }^{\left[t\right]}\right)}^{-1}{X}^{\prime }X{\beta }^{\left(0\right)}$$ (8)

with successive updating; here,

$$W_{\beta }^{\left[t\right]}=\text{Diag}\left\{\frac{{\sigma }_{\text{e}}^2}{S_{\beta }^2}\frac{\left(1+1/\nu \right)}{\left(1+{\beta }_j^{2\left[t\right]}/S_{\beta }^2\nu \right)}\right\}$$

is a diagonal matrix. If this converges, it will do so to one of perhaps many stationary points, as it is known that t-regression models may produce multimodal log-posterior surfaces, especially if ν is small (McLachlan and Krishnan 1997). Hence, iteration (8) may lead to a point receiving little posterior plausibility.

The role of $W_{\beta }=\left\{w_{jj,{\beta }_j}\right\}$ in (8) parallels that of the inverse of the genetic variance–covariance matrix (times ${\sigma }_{\text{e}}^2$) in standard BLUP (Henderson 1984), so that the larger $w_{jj,{\beta }_j}$ is, the stronger the shrinkage toward 0 (mean of the prior distribution). However, while the variance ratio $\lambda ={\sigma }_{\text{e}}^2/{\sigma }_{\beta }^2$ is constant in ridge regression BLUP, here it varies over markers, as it takes the form $w_{jj,{\beta }_j}$. As ν → ∞ (the t-distribution approaches a normal one), ${\lambda }_j\to {\sigma }_{\text{e}}^2/S_{\beta }^2$, resembling λ of BLUP. On the other hand, if the t prior has a finite number of degrees of freedom, markers whose effects are closer to 0 are shrunk more strongly than those with larger absolute values, simply because λ_j is larger for the former. To illustrate, let ${\sigma }_{\text{e}}^2=S_{\beta }^2=1$ so that the “variance ratio” is ${\lambda }_j=\left(1+1/\nu \right)/\left(1+{\beta }^2/\nu \right)$. Figure 2 illustrates the impact of the marker effect on the “variance ratio” for ν = 4, 6, 10, and 1000. It is seen that λ_j becomes smaller (less shrinkage toward zero) as the absolute value of the effect of the marker increases; also, shrinkage increases as the degrees of freedom of the distribution increase, at any given marker effect. Eventually, when ν → ∞ (so that the prior is normal) the variance ratio takes the same value for all markers (thick line in Figure 2, almost horizontal, corresponding to ν = 1000). For markers with effects near zero, the t-distribution shrinks effects more strongly than the normal process, but it does not severely penalize markers having strong effects on the phenotype. Hence, in Bayes A shrinkage is marker-effect specific, with this specificity becoming milder as the value of ν increases. Note that (7) also induces frequency-specific shrinkage, due to the Bayesian compromise between the prior and X′X, as in the case of BLUP. Hence, apart from the effects of sample size, there are two sources of shrinkage in Bayes A, contrary to a single one in BLUP. This seems to confer Bayes A more flexibility than BLUP, but this is not necessarily good because the extra parameters ν and $S_{\beta }^2$ (this one playing the role of ${\sigma }_{\beta }^2$) are influential, and may affect “inference” of marker effects adversely (Lehermeier et al. 2013). Naturally, these parameters can be assigned priors and inferred from the resulting Bayesian model, but this was not suggested by Meuwissen et al. (2001).

Figure 2.

Impact of marker effect and of the degrees of freedom (d.f.) parameter on the extent of shrinkage toward the prior distribution in Bayes A: the larger the value in the y-axis, the stronger the shrinkage toward 0. d.f. = 4, solid line; d.f. = 6, long-dashed line; d.f. = 10, short-dashed line; d.f. = 1000, gray circles, almost horizontal.

Bayes B

A formulation of Bayes B as a mixture at the level of effects, but not of their variances, as in Meuwissen et al. (2001), is in Gianola et al. (2009) and Habier et al. (2011). The hierarchical model is

$$\begin{array}{c}{y}_i|\beta ,{\sigma }_{\text{e}}^2\sim N\left(x_i^{\prime }\beta ,{\sigma }_{\text{e}}^2\right),\\ {\beta }_j|S_{\beta }^2,\nu ,\pi \sim IID \{\begin{array}{l}0\hfill \\ t\left(0,S_{\beta }^2,\nu \right)\hfill \end{array}\begin{array}{l}\text{with}\hspace{0.5em}\text{probability}\pi \hfill \\ \text{with}\hspace{0.5em}\text{probability} 1-\pi \hfill \end{array}, j=1,2,\dots ,p.\end{array}$$

The prior is a mixture of a “0-state” (a point mass at 0) with a t-distribution, the mixing probabilities being π and 1 − π, respectively, where π is assumed known and specified arbitrarily. Recall (in informal notation) that

$$t\left({\beta }_j|0,S_{\beta }^2,\nu \right)={\displaystyle \int N\left({\beta }_j|0,{\sigma }_{{\beta }_j}^2\right){\chi }^{-2}\left({\sigma }_{{\beta }_j}^2|S_{\beta }^2,\nu \right)d{\sigma }_{{\beta }_j}^2,} \hspace{1em}j=1,2,\dots ,p,$$

where ${\chi }^{-2}\left({\sigma }_{{\beta }_j}^2|S_{\beta }^2,\nu \right)$ is a scaled-inverted chi-square distribution assigned as prior to the variance of the jth marker effect, ${\sigma }_{{\beta }_j}^2$. Meuwissen et al. (2001) formulated the mixture at the level of these variances, arguing as follows: “the distribution of genetic variances across loci is that there are many loci with no genetic variance (not segregating) and a few with genetic variance.” Gianola et al. (2009) were critical of this formulation, both from statistical and genetic points of view.

The hierarchical prior is deceiving because, in fact, Bayes B ends up assigning the same marginal prior to every marker. This follows from consideration of the mean and variance of a mixture, e.g., Gianola et al. (2006). The mean of a mixture is the weighted average of the means of the components (the weights being the mixing probabilities π and 1 − π) and the variance is the weighted average of the component variances, plus a term that can be interpreted as “variance” among component means. One has

$$E\left({\beta }_j|\pi \right)=\left(1-\pi \right)E\left[t\left({\beta }_j|0,S_{\beta }^2,\nu \right)\right]=0,\hspace{0.17em}j=1,2,\dots ,p,$$

where $E\left[t\left({\beta }_j|0,S_{\beta }^2,\nu \right)\right]=0$ is the mean of the t-distribution, and

$$\begin{array}{cc}\text{Var}\left({\beta }_j|\pi \right)=\left(1-\pi \right)\frac{S_{\beta }^2\nu }{\nu -2},& j=1,2,\dots ,p.\end{array}$$

Above, $\text{Var}\left[t\left({\beta }_j|0,S_{\beta }^2,\nu \right)\right]=S_{\beta }^2\nu /\left(\nu -2\right)$ is the variance of the t-distribution. It follows that Bayes B assigns, a priori, the same mean and variance to all marker effects and that it uses a prior that is even more precise than the prior in Bayes A (the prior variance is reduced by a fraction π in Bayes B, relative to that of Bayes A). This makes effective Bayesian learning even more difficult in Bayes B than in Bayes A, as it takes more information from the data to “neutralize” the prior of Bayes B than that of Bayes A. At any rate, none of these two regression models allows for proper learning about marker effects or genetic architecture in the n ≪ p setting, as argued earlier in this article.

As for Bayes A, no closed forms for the marginal posterior distributions of marker effects exist for Bayes B. The posterior expectation of β is

$$E_{\text{Bayes}\hspace{0.5em}\text{B}}\left(\beta |\pi ,S_{\beta }^2,\nu ,{\sigma }_{\text{e}}^2,y\right)=\left(1-\pi \right)E_{\text{Bayes}\hspace{0.17em}\text{A}}\left(\beta |\pi ,S_{\beta }^2,\nu ,{\sigma }_{\text{e}}^2,y\right).$$ (9)

This indicates that shrinkage toward 0 is stronger than in Bayes A since posterior means are smaller in Bayes B by a fraction π. Coupled with the arbitrary assignment of a value to π, the implication is that the the prior is even more influential in Bayes B than in Bayes A. This could have been expected intuitively, but the point has not been made before, at least in this manner.

Wimmer et al. (2012) noted that methods such as Bayes B have yielded better predictive abilities than BLUP in many simulation studies reported in the literature, but that this has not been observed with real data (e.g., Ober et al. 2012). Wimmer et al. (2012) investigated predictive abilities of these two methods in maize and in Arabidopsis. The target populations differed in effective population size and in extent of linkage disequilibrium. Despite expected differences in genetic architecture among populations and traits, predictive abilities delivered by BLUP and Bayes B did not differ significantly for their target traits. Further, they found via simulation (personal communication) that Bayes B was effective for learning genetic architecture in the n ≪ p setting only when the number of true nonzero marker effects (s) is such that s ≪ n, given the true model. Otherwise, the error of estimation of marker effects was as poor as that of BLUP, the latter found to be more robust over a wide range of situations (this is ironic, because BLUP or G-BLUP were not tailored for learning genetic architecture). In short, they confirmed that, provided one “hits” the true model (thus fulfilling oracle property 1 of Fan and Li 2001), effective learning of “true genetic architecture” is possible only if the model is very sparse relative to sample size. The condition s ≪ n would lead to oracle property 2, as anticipated by standard asymptotic Bayesian theory under regularity conditions.

BayesSSVS was proposed by Verbyla et al. (2009), but it is not discussed here because it is similar to Bayes B. Bayes Cπ of Habier et al. (2011) provides a more sensible formulation of the mixture, but it is similar in spirit and shares the same limitations of Bayes B, since parameter identification is not attained for most of the unknowns. An interesting example of consequences of overparameterization in Bayes Cπ is provided by Duchemin et al. (2012); these authors noted that as values of π went up in the sampling process, realizations of marker effect variances went down. Hints about genetic architecture from Bayes B or Bayes Cπ or from other members of the alphabet should be taken very cautiously, at least when n ≪ p.

Bayes L

Lasso regression (Tibshirani 1996) inspired the Bayesian Lasso (Bayes L here) of Park and Casella (2008), a method with followers such as Vázquez et al. (2010) and Crossa et al. (2010) and with an implementation available in the software R described by Pérez et al. (2010). The linear regression model is given in (1), but the prior assigned to marker effects is a Laplace (double exponential, DE) distribution. All marker effects are assumed to be independently and identically distributed as DE with the prior density being

$$p\left(\beta |\lambda \right)=\frac{\lambda }{2}\text{exp}\left(-\lambda \left|\beta \right|\right).$$ (10)

Here, E(β|λ) = 0 and $\text{Var}\left(\beta |\lambda \right)=2/{\lambda }^2$ for all markers; as λ increases the variance of the DE distribution decreases and the density becomes sharper. This prior assigns the same variance or prior uncertainty to all marker effects, but it possesses thicker tails than the normal prior. A comparative discussion of the DE prior is in de los Campos et al. (2012a). Even though Bayes L bears a parallel with the Lasso, it does not “kill” or remove markers from the model, contrary to what happens in variable selection approaches. Bayes L poses a leptokurtic prior, so it is expected to shrink effects more strongly toward zero than the Gaussian prior, as opposed to inducing sparsity in the strict sense of the Lasso.

Bayes L shrinks strongly:

To appraise how Bayes L shrinks marker effects, we examine the mode(s) of the joint posterior distribution of β using the DE prior (10), assuming that λ and the residual variance are known. As in Tibshirani (1996), write $\left|{\beta }_j\right|={\beta }_j^2/\left|{\beta }_j\right|;$ with this representation ${\displaystyle {\sum }_{j=1}^p\left|{\beta }_j\right|}={\beta }^{\prime }{W}_{\beta }^{-1}\beta$, where $W_{\beta }^{-1}=\text{Diag}\left\{1/\left|{\beta }_j\right|\right\}.$. Using this, the log-posterior (apart from an additive constant) is

$$L\left(\beta |y,\lambda ,{\sigma }_{\text{e}}^2\right)=-\frac{{\left(y-X\beta \right)}^{\prime }\left(y-X\beta \right)+{\sigma }_{\text{e}}^2\lambda \beta \prime W_{\beta }^{-1}\beta }{2{\sigma }_{\text{e}}^2},$$ (11)

If, as in Tibshirani (1996), it is ignored that $W_{\beta }^{-1}$ is a random matrix (because it is a function of |β_j|) this takes the form of a standard BLUP representation, so the mode of the conditional posterior distribution of β satisfies

$$\tilde{\beta }={\left(X\prime X+{\sigma }_{\text{e}}^2\lambda W_{\beta }^{-1}\right)}^{-1}X\prime y.$$ (12)

Contrary to BLUP–ridge regression where shrinkage factors are marker effect independent, these factors take the form ${\sigma }_{\text{e}}^2\lambda /\left|{\beta }_j\right|$ in Bayes L, implying that markers with tiny effects are shrunk more strongly toward zero, as a larger number is added to the diagonal elements of the coefficient matrix leading to solution $\tilde{\beta }$. Note, however, that (12) is not an explicit system, so it would make sense to iterate; details on an iterative scheme are in the Appendix (Mode of the conditional posterior distribution in Bayes L).

The preceding implies that Bayes L produces a more “effectively sparse” model. This can be seen from inspection of an “effective number of parameters” measure (e.g., Tibshirani 1996; Ruppert et al. 2003) given by

$$\text{d.f}{\text{.}}_{\text{ridge}}=\text{tr}\left[X{\left(X^{\prime }X+I\frac{{\sigma }_{\text{e}}^2}{{\sigma }_{\beta }^2}\right)}^{-1}{X}^{\prime }\right[=\text{tr}\left[{\left(X^{\prime }X+I\frac{{\sigma }_{\text{e}}^2}{{\sigma }_{\beta }^2}\right)}^{-1}X\prime X\right],$$

and

$$\text{d.f}{\text{.}}_{\text{Bayes}\hspace{0.5em}\text{L}}=\text{tr}\left[{\left(X\prime X+{\sigma }_{\text{e}}^2\lambda W_{\beta }^{-1}\right)}^{-1}X\prime X\right].$$

If X is orthonormalized, so that X′X = I (with dispersion parameters scaled accordingly)

$$\text{d.f}{\text{.}}_{\text{ridge}}=\text{tr}\left[{\left(I+\frac{{\sigma }_{\text{e}}^2}{{\sigma }_{\beta }^2}\right)}^{-1}\right]=p\frac{{\sigma }_{\beta }^2}{{\sigma }_{\beta }^2+{\sigma }_{\text{e}}^2},$$ (13)

and

$$\text{d.f}{\text{.}}_{\text{Bayes}\hspace{0.5em}\text{L}}=\text{tr}\left[{\left(I+{\sigma }_{\text{e}}^2\lambda W_{\beta }^{-1}\right)}^{-1}\right]={\displaystyle \sum _{j=1}^p\frac{\left|{\beta }_j\right|}{\left|{\beta }_j\right|+{\sigma }_{\text{e}}^2\lambda }}.$$ (14)

This enables us to see that, in ridge regression, every degree of freedom (contributor to model complexity) represented by a column of the orthonormalized marker matrix is attenuated by the same factor, ${\sigma }_{\beta }^2/\left({\sigma }_{\beta }^2+{\sigma }_{\text{e}}^2\right)$. On the other hand, in Bayes L markers having tiny effects are effectively, but not physically, wiped out of the model. Also, markers with strong effects receive a heavier weight in this overall measure of complexity.

We simulated P = 100, 000 marker effects from DE distributions with mean 0 and variances 10⁻¹⁶, 10⁻⁸, or 10⁻⁴; setting ${\sigma }_{\text{e}}^2=1$, the preceding three values can be interpreted as the contribution of an individual marker to variance relative to residual variability. When a marker effect had a large variance (10⁻⁴), the entire battery of markers, assuming a priori independence of effects, represented $\frac{10}{11}$ of the total variance; on the other hand, when markers were assigned a variance of 10⁻¹⁶, markers accounted for about ${10}^{-11}/\left({10}^{-11}+1\right)$ of the total variability. Since the variance of the DE distribution is $2/{\lambda }^2$ the settings led to λ values of $\sqrt{2}\times {10}^8$, $\sqrt{2}\times {10}^4$, and $\sqrt{2}\times {10}^2$, respectively; larger values of λ produce stronger shrinkage toward 0. The shrinkage factor is ${\sigma }_{\text{e}}^2\lambda /\left|{\beta }_j\right|$ for marker j in Bayes L vs. ${\sigma }_{\text{e}}^2/{\sigma }_{\beta }^2$ in ridge regression–BLUP. The contribution of a marker to the model was assessed as follows: from (13), in ridge regression each marker contributes the same amount, ${\sigma }_{\beta }^2/\left({\sigma }_{\beta }^2+{\sigma }_{\text{e}}^2\right),$ to model complexity, whereas in Bayes L the corresponding metric is $\left|{\beta }_j\right|/\left(\left|{\beta }_j\right|+{\sigma }_{\text{e}}^2\lambda \right)$, as given in (14). For ridge regression, the effective number of parameters was approximately 10⁻¹¹, 10⁻³, and 10, for ${\sigma }_{\beta }^2={10}^{-16},{10}^{-8}$, and 10⁻⁴, respectively. For Bayes L, the corresponding effective number of parameters was 4.96 × 10⁻¹², 4.98 × 10⁻⁴, and 4.98, respectively. Clearly, Bayes L produced a model that was more sparse than ridge regression–BLUP. Each of the markers made a tiny contribution to model complexity; for instance, when the variance of the double exponential of marker effects was 10⁻¹⁶, the relative contributions to the model of individual markers ranged from 0 to 10⁻¹⁶; when the variance was 10⁻⁸, these ranged from 0 to 10⁻⁸, while the range was 0 − 10⁻⁴ for ${\sigma }_{\beta }^2={10}^{-4}$. A plot of the density of the effective contributions to the model of each of the 100, 000 markers is in Figure 3, for the case ${\sigma }_{\beta }^2={10}^{-4}$; >95% of the markers contributed <2 × 10⁻⁴ effective degrees of freedom to the model. Hence, when a marker contributes to variance in a tiny manner, shrinkage of their individual effects toward 0 is very strong. Then, if a marker effect conveys the meaning of a fraction equal to 10⁻⁸, say, of some physical parameter, what can this tell us about the state of nature (i.e., genetic architecture) in the absence of effective Bayesian learning, as argued earlier in the article? Probably not much unless n ≫ p and the model fitted is the “true” one, the latter requiring the extraordinarily strong assumption that a complex trait is well represented by a (multiple) linear regression.

Figure 3.

Density (over 100,000 markers) of the “effective degrees of freedom” contributed by a marker to the model for a double exponential prior distribution with variance 10⁻⁴.

Bayes L With Gamma prior for λ²:

The DE density (10) is indexed by a single positive parameter λ, and if this is treated as unknown, the marginal prior density of a marker effect is

$$p\left(\beta \right)={\displaystyle {\int }_0^{\mathrm{\infty }}p\left(\beta |\lambda \right)p\left(\lambda \right)d\lambda ,}$$

where p(λ) is the prior density of λ. Clearly E(β) = E_λE(β|λ) = 0, but the prior variance of β will depend on the distribution assigned to λ. Typically, a Γ(r, δ) prior is placed on λ² with the density being

$$p\left({\lambda }^2|r,\delta \right)=\frac{{\delta }^r}{\Gamma \left(r\right)}{\left({\lambda }^2\right)}^{r-1}\text{exp}\left(-\delta {\lambda }^2\right),$$ (15)

and $E\left({\lambda }^2|r,\delta \right)=r/\delta$ and $\text{Var}\left({\lambda }^2|r,\delta \right)=r/{\delta }^2$. Since λ is positive, p(β|λ) = p(β|λ²), so that

$$\begin{array}{c}p\left(\beta |r,\delta \right)={\displaystyle {\int }_0^{\mathrm{\infty }}p\left(\beta |{\lambda }^2\right)p\left({\lambda }^2|r,\delta \right)d{\lambda }^2}\\ \propto {\displaystyle {\int }_0^{\mathrm{\infty }}{\left({\lambda }^2\right)}^{r+\frac{1}{2}-1}\text{exp}\left[-\left(\left|\beta \right|\sqrt{{\lambda }^2}+\delta {\lambda }^2\right)\right]d{\lambda }^2.}\end{array}$$ (16)

Using in the above expression an approximation given in the Appendix (Approximation of an integral in Bayes L), Equation 16 gives

$$p\left(\beta |r,\delta \right){\propto }_{\text{approx}.}{e}^{-\left|\beta \right|\sqrt{r/\delta }}\left\{1-\frac{1}{2}\sqrt{\frac{\delta }{r}}\left|\beta \right|\frac{1}{2\delta }+\frac{\delta }{8r}\left({\left|\beta \right|}^2+\sqrt{\frac{\delta }{r}}\left|\beta \right|\right)\frac{4r+3}{4{\delta }^2}\right\},$$ (17)

where ∝_approx.means “approximately proportional to.” If only the first term of the approximation is used, after normalization one gets

$$p_1\left(\beta |r,\delta \right)\approx \frac{e^{-\left|\beta \right|\sqrt{r/\delta }}}{{\displaystyle {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{e}^{-\left|\beta \right|\sqrt{r/\delta }}d\beta }},$$ (18)

and this is a DE density with parameter $\lambda =\sqrt{r/\delta }.$ If both the first and second terms of the approximation are employed, one gets

$$p_2\left(\beta |r,\delta \right)\approx \frac{e^{-\left|\beta \right|\sqrt{r/\delta }}\left(1-\frac{1}{2}\sqrt{\delta /r}\left|\beta \right|\left(1/2\delta \right)\right)}{{\displaystyle {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{e}^{-\left|\beta \right|\sqrt{r/\delta }}\left(1-\frac{1}{2}\sqrt{\delta /r}\left|\beta \right|\left(1/2\delta \right)\right)d\beta }}.$$ (19)

Next, we examine the shape of the unnormalized density (19) for two different Γ(r, δ) prior distributions of λ². Setting r = δ gives Gamma distributions with expected value 1 and variance $\frac{1}{\delta }$; use of r = δ = 4 and r = δ = 16 produces prior distributions with variances $\frac{1}{4}$ and $\frac{1}{16}$, respectively, and the corresponding densities are shown in Figure 4, top left. Taking into account that the prior distributions of marker effects have null means, the variance of approximation (19) to the marginal prior of β was evaluated by numerical integration between −9 and 9 as

$${\text{Var}}_2\left(\beta |r=\delta \right)=\frac{{\displaystyle {\int }_{-9}^9{\beta }^2{e}^{-\left|\beta \right|}\left(1-\frac{1}{2}\left|\beta \right|\frac{1}{2\delta }\right)d\beta }}{{\displaystyle {\int }_{-9}^9{e}^{-\left|\beta \right|}\left(1-\frac{1}{2}\left|\beta \right|\frac{1}{2\delta }\right)d\beta }},$$

yielding 1.73 (δ = 4) and 1.93(δ = 16). This produces a seemingly paradoxical situation, where the more uncertain prior for λ² (δ = 4) gives a marginal prior for the marker effect that is more precise (as measured by the variance) than that for δ = 16. The densities, shown in Figure 4, top right, seem indistinguishable. However, if the plot is zoomed in the middle and right tails of the distribution (left and right bottom, respectively) the prior with δ = 16 turns out to be less sharp and with thicker tails, thus explaining its larger variance. Also, the prior probability that a marker has an effect ranging from −0.3 to 0.3 is 0.274 for δ = 4 (more variable prior for λ²) and 0.263 for δ = 16; the probabilities that a marker has an effect from 2 to 7 are 0.058 (δ = 4) and 0.065 (δ = 16), respectively.

Figure 4.

(A) Gamma prior density of lambda square for r = δ = 4 (dot–dash) and r = δ = 16 (solid). (B) Marginal prior densities of marker effects for r = δ = 4 (dot–dash) and r = δ = 16 (solid). (C) and (D) focus on the middle and right tails of the densities displayed in B, respectively.

Bayes L with uniform prior on λ:

In an attempt to make the prior in a Bayesian analysis less aggressive, one may naively think that Bayes’s “principle of insufficient reason” (the uniform prior) may render the analysis objective. Let the uniform prior on λ be λ|L, U ∼ Uniform(L, U), where L and U are the lower and upper bounds, respectively, of the prior distribution. Mixing the DE distribution with parameter λ over this prior gives as marginal density

$$p\left(\beta |L,U\right)=\frac{1}{U-L}{\displaystyle {\int }_L^U\frac{\lambda }{2}\text{exp}\left(-\lambda \left|\beta \right|\right)d\lambda .}$$

As before, we employ a Taylor series to approximate exp(−λ|β_j|), but now around the expectation $m=\left(U+L\right)/2$ of the uniform distribution, giving

$$\text{exp}\left(-\lambda \left|\beta \right|\right)\approx e^{-m\left|\beta \right|}\left[1-\left|\beta \right|\left(\lambda -m\right)+\frac{1}{2}{\left|\beta \right|}^2{\left(\lambda -m\right)}^2\right].$$

Then

$$p\left(\beta |L,U\right){\propto }_{\text{approx}.}\frac{e^{-m\left|\beta \right|}}{U-L}{\displaystyle \underset{L}{\overset{U}{\int }}\left[1-\left|\beta \right|\left(\lambda -m\right)+\frac{1}{2}{\left|\beta \right|}^2{\left(\lambda -m\right)}^2\right]\frac{\lambda }{2}d\lambda .}$$ (20)

If the constant and the linear terms of the expansion are retained this produces

$$p_{\text{unif},1}\left(\beta |L,U\right){\propto }_{\text{approx}.}\frac{1}{U-L}{e}^{-m\left|\beta \right|}{\displaystyle {\int }_L^U\left[1-\left|\beta \right|\left(\lambda -m\right)\right]\frac{\lambda }{2}d\lambda .}$$

Since λ is positive, one can set L = 0 and $m=U/2$ yielding

$$p_{\text{unif},1}\left({\beta }_j|L=0,U\right){\propto }_{\text{approx}.}\frac{e^{-m\left|\beta \right|}}{U}\left(\frac{1}{4}{U}^2+\frac{\left|\beta \right|U^2}{2}\left(\frac{m}{2}-\frac{U}{3}\right)\right)=\frac{U}{4}{e}^{-U\left|\beta \right|/2}\left(1-\left|\beta \right|\frac{U}{6}\right).$$

A plot of p_unif,1(β_j|L = 0, U) is shown in Figure 5. As U increases, the prior distribution of the marker effect gets increasingly concentrated near 0, reaching a point mass in the limit. This implies that the regression model becomes effectively very simple if U is assigned large values, as most regression coefficients take values close to 0. In theory, this should produce underfitting and out of sample predictions that do not generalize well. It is thus intriguing why Legarra et al. (2011) obtained reasonable predictive accuracies when placing a uniform prior on λ, with L = 0 and U = 10⁶. This theoretical excursion suggests that a big warning should be inserted in documentation of software implementing DE regression models with a flat prior on the regularization parameter λ.

Figure 5.

Prior density of a marker effect when a uniform (0, U) prior is adopted for λ at varying values of upper bound U of the uniform distribution: 1 (solid line), 5 (dashes), 10 (dots–dashes).

On parameterizations of Bayes L:

How any Bayesian or “classical” model is parameterized depends on mechanistic (e.g., interpretation with respect to some theory) or computing considerations, but alternative parameterizations must be equivalent in terms of the inference attained. For example, a parameterization of the classical infinitesimal model (e.g., Hill 2012) in terms of additive genetic and environmental variances (V_A, V_E) must be equivalent to parameterization (V − V_E, V_E), where V is the phenotypic variance, or to parameterization (Vh², (1 − h²)V), where h² is heritability. The second and third parameterizations do not imply causally that the genetic variance depends on the environmental variance or that the environmental variance depends on heritability. In likelihood-based inference there is invariance of parameters under transformation. However, care must be exercised in Bayesian analysis because parameters are random, so any rotation of coordinates (some transformations involve nonlinear rotations) require intervention of the Jacobian of the transformation. One can go back and forth between parameterizations, provided that probability volumes are preserved properly. For instance, if one assigns independent priors to h² and V in a (Vh², (1 − h²)V) parameterization, those used in a V_A, V_E parameterization should be probabilistically consistent with the preceding, such that samples from the joint posterior of h² and V produce the same distribution as that obtained by sampling from the joint posterior of V_A, V_E. Further, conditioning and deconditioning may be necessary due to computing issues, e.g., the Gibbs sampler works with conditional distributions, but the algorithm automates the deconditioning. It is precisely in this context that Legarra et al. (2011) misinterpreted the parameterization of Bayes L in Park and Casella (2008), de los Campos et al. (2009b), Weigel et al. (2009), and Vázquez et al. (2010) who, instead of working directly with prior (10) adopted a conditional prior discussed further below. All these authors have applied this parameterization successfully using data from animals and plants.

For reasons related to the behavior of Markov chain Monte Carlo algorithms for Bayes L, Park and Casella (2008) introduced a conditional DE distribution, with density This distribution has mean $E\left(\beta |\lambda ,{\sigma }_{\text{e}}^2\right)=0$ and variance $\text{Var}\left(\beta |\lambda ,{\sigma }_{\text{e}}^2\right)=2\left({\sigma }_{\text{e}}^2/{\lambda }^2\right);$ this, of course, is not the variance of β_j. Legarra et al. (2011) incorrectly wrote $\text{Var}\left(\beta \right)\text{=2}\left({\sigma }_{\text{e}}^2/{\lambda }^{\text{2}}\right)\text{,}$ and made the statement: “we do expect the distribution of SNP effects not to be related to unobservable, unaccounted (residual) effects that can, for example, vary from site to site for the same individuals.“ It is fairly obvious that $\text{Var}\left(\beta |\lambda ,{\sigma }_{\text{e}}^2\right)$ cannot be Var(β_j) since

$$\text{Var}\left(\beta |\lambda \right)=E_{{\sigma }_{\text{e}}^2}\left[\text{Var}\left(\beta |\lambda ,{\sigma }_{\text{e}}^2\right)\right]+{\text{Var}}_{{\sigma }_{\text{e}}^2}\left[E\left(\beta |\lambda ,{\sigma }_{\text{e}}^2\right)\right]=E_{{\sigma }_{\text{e}}^2}\left[2\frac{{\sigma }_{\text{e}}^2}{{\lambda }^2}\right],$$

with the term ${\text{Var}}_{{\sigma }_{\text{e}}^2}\left[E\left(\beta |\lambda ,{\sigma }_{\text{e}}^2\right)\right]$ dropping because it is null. Hence, Var(β|λ) depends on the prior adopted for ${\sigma }_{\text{e}}^2$. If ${\sigma }_{\text{e}}^2$ is assigned a scaled inverted chi-square distribution on ν_e degrees of freedom and with scale $S_{\text{e}}^2$, with density as in (29),

$$\begin{array}{c}\text{Var}\left(\beta |\lambda ,{\nu }_{\text{e}},S_{\text{e}}^2\right)=E_{{\sigma }_{\text{e}}^2}\left(2\frac{{\sigma }_{\text{e}}^2}{{\lambda }^2}|\lambda \right)=\frac{2}{{\lambda }^2}{\displaystyle \underset{0}{\overset{\mathrm{\infty }}{\int }}{\sigma }_{\text{e}}^{2}p\left({\sigma }_{\text{e}}^2|{\nu }_{\text{e}},S_{\text{e}}^2\right)d{\sigma }_{\text{e}}^2}\\ =\frac{2{\nu }_{\text{e}}{S}_{\text{e}}^2}{{\lambda }^2\left(v_{\text{e}}-2\right)},\hspace{1em}\nu >2.\end{array}$$ (21)

$$f\left(\beta /\lambda ,{\sigma }_{\text{e}}^2\right)=\frac{\lambda }{2\sqrt{{\sigma }_{\text{e}}^2}}\text{exp}\left(-\frac{\lambda }{2\sqrt{{\sigma }_{\text{e}}^2}}\left|\beta \right|\right).$$

Therefore, the variance of the prior distribution of marker effects does not depend on ${\sigma }_{\text{e}}^2$ but, rather, on λ² and on the parameters of the prior distribution of ${\sigma }_{\text{e}}^2$. There is the additional complication that (21) does not take into account uncertainty associated with λ, and this is examined next.

Since λ must be positive, conditioning on λ is equivalent to conditioning on λ², so that $E\left({\beta }_j\right)=E_{{\lambda }^2}E\left({\beta }_j|{\lambda }^2\right)=0$, and

$$\text{Var}\left(\beta \right)=E_{{\lambda }^2}\text{Var}\left(\beta |{\lambda }^2\right)+{\text{Var}}_{{\lambda }^2}E\left(\beta |{\lambda }^2\right)=E_{{\lambda }^2}\text{Var}\left(\beta |{\lambda }^2\right).$$

Hence, unconditionally, use of (21) in $E_{{\lambda }^2}\text{Var}\left(\beta |{\lambda }^2\right)$ produces

$$\text{Var}\left(\beta |{\nu }_{\text{e}},S_{\text{e}}^2\right)=\frac{2{\nu }_{\text{e}}{S}_{\text{e}}^2}{v_{\text{e}}-2}{\displaystyle \int \frac{1}{{\lambda }^2}p\left({\lambda }^2\right)d\lambda .}$$

If a Γ(r, δ) prior is placed on λ², with density (13)

$$\text{Var}\left(\beta |{\nu }_{\text{e}},S_{\text{e}}^2,r,\delta \right)=\frac{2{\nu }_{\text{e}}{S}_{\text{e}}^2}{v_{\text{e}}-2}{\displaystyle \int \frac{1}{{\lambda }^2}\frac{{\delta }^r}{\Gamma \left(r\right)}{\left({\lambda }^2\right)}^{r-1}\text{exp}\left(-\delta {\lambda }^2\right)d{\lambda }^2.}$$

Changing variables to $\theta =1/{\lambda }^2$ gives

$$\text{Var}\left(\beta |{\nu }_{\text{e}},S_{\text{e}}^2,r,\delta \right)=\frac{2{\nu }_{\text{e}}{S}_{\text{e}}^2}{v_{\text{e}}-2}{\displaystyle \int \theta \frac{{\delta }^r}{\Gamma \left(r\right)}{\left(\theta \right)}^{-r+1}\text{exp}\left(-\frac{\delta }{\theta }\right)\frac{1}{{\theta }^2}d\theta =\frac{2{\nu }_{\text{e}}{S}_{\text{e}}^2}{v_{\text{e}}-2}{\displaystyle \int \theta \frac{{\delta }^r}{\Gamma \left(r\right)}{\left(\theta \right)}^{-r-1}\text{exp}\left(-\frac{\delta }{\theta }\right)d\theta .}}$$

The integral is the expected value of a random variable (θ) following an inverted Gamma distribution with parameters r and δ, which is $\delta /\left(r-1\right)$ (r > 1), so

$$\text{Var}\left(\beta |{\nu }_{\text{e}},S_{\text{e}}^2,r,\delta \right)=\frac{2{\nu }_{\text{e}}{S}_{\text{e}}^2\delta }{\left(v_{\text{e}}-2\right)\left(r-1\right)}.$$ (22)

As argued in Gianola et al. (2009), the connection between the variance of the prior distribution of marker effects and additive genetic variance is subtle and elusive. If $\text{Var}\left(\beta |{\nu }_{\text{e}},S_{\text{e}}^2,r,\delta \right)$ were to be viewed as the variance of an additive effect in some infinitesimal model, how are its different components interpreted? If the standard infinitesimal model is parameterized in terms of (V_E, h²) one can write

$$V_{\text{A}}=V_{\text{E}}\frac{h^2}{1-h^2}.$$

In (22) ${\nu }_{\text{e}}{S}_{\text{e}}^2/\left(v_{\text{e}}-2\right)$is the counterpart of V_E, since this is the expected value of the prior distribution assigned to the residual variance, ${\sigma }_{\text{e}}^2$. Then, $2\delta /\left(r-1\right)$ plays the role of $h^2/\left(1-h^2\right)$; since $\delta /\left(r-1\right)$ is the prior expectation of $1/{\lambda }^2$, it would turn out that ${\lambda }^2/2$ would be the counterpart of $\left(1-h^2\right)/h^2$.

The statements made in Legarra et al. (2011) are misleading due to an incorrect interpretation of the parameterization of Bayes L proposed by Park and Casella (2008), used to address a multimodality problem that seems to arise in nonhierarchical implementations of Bayes L in the sense of Kärkkäinen and Sillanpäa (2012). These authors reported that hierarchical and nonhierarchical versions of the Bayesian Lasso led to different posterior inferences, but could not find clear reasons for this discrepancy. It might be related to lack of convergence of the Markov chain Monte Carlo scheme in the nonhierarchical parameterization or perhaps to some impropriety. Additional basic research is needed to explain this paradox, but Kärkkäinen and Sillanpäa (2012) recommended the hierarchical implementation, possibly because of easier computation.

Bayes R

Erbe et al. (2012) presented this method as follows. Bayes R starts the hierarchical model with (1) and poses a mixture of four zero-mean normal distributions as a conditional prior for a specific SNP effect:

$$\begin{array}{l}p\left(\beta |{\sigma }_{{\beta }_1}^2=0,{\sigma }_{{\beta }_2}^2={10}^{-4}{\sigma }_{\text{g}}^2,{\sigma }_{{\beta }_3}^2={10}^{-3}{\sigma }_{\text{g}}^2,{\sigma }_{{\beta }_4}^2={10}^{-2}{\sigma }_{\text{g}}^2,{\pi }_1,{\pi }_2,{\pi }_3,{\pi }_4\right)\\ ={\pi }_1\times 0+{\pi }_{2}N\left(\beta |0,{10}^{-4}{\sigma }_{\text{g}}^2\right)\\ +{\pi }_{3}N\left(\beta |0,{10}^{-3}{\sigma }_{\text{g}}^2\right)+{\pi }_{4}N\left(\beta |0,{10}^{-2}{\sigma }_{\text{g}}^2\right).\end{array}$$ (23)

Here, if the SNP effect is generated from the first component of the mixture (with probability π₁) it will be 0 with complete certainty; if drawn from the second component it will have a normal distribution with null mean and variance ${\sigma }_{{\beta }_2}^2={10}^{-4}{\sigma }_{\text{g}}^2,$ and so on. In Bayes R, ${\sigma }_{\text{g}}^2=r^2{\sigma }^2$ is the assumed genetic variance, r² is the assumed reliability, and σ² is the variance of the target trait. Presumably, the assumption about r² is either model derived or based on prior cross-validation information, which is good Bayesian behavior, normatively. Makowsky et al. (2011) gave evidence that what one assumes about genetic variance from inference in training data is not recovered in cross-validation.

The mean of the mixture is obviously 0. Since the four components of the mixture have null means, the variance, given π = (π₁, π₂, π₃, π₄,), is

$$\text{Var}\left(\beta |\mathit{\pi }\right)=\left({\pi }_2\times {10}^{-4}+{\pi }_3\times {10}^{-3}+{\pi }_4\times {10}^{-2}\right){\sigma }_{\text{g}}^2.$$

Further,

$$\text{Var}\left(\beta \right)=E_{\pi }\left[\text{Var}\left(\beta |\mathit{\pi }\right)\right]+{\text{Var}}_{\pi }\left[E\left(\beta |\mathit{\pi }\right)\right]=E_{\pi }\left[\text{Var}\left(\beta |\mathit{\pi }\right)\right].$$

Erbe et al. (2012) used a Dirichlet distribution with parameter vector α = (α₁,α₂,α₃,α₄)′ as prior for the elements of π, so that

$$\text{Var}\left(\beta |\alpha \right)=E_{\pi }\left[\text{Var}\left({\beta }_j|\mathit{\pi }\right)\right]=\frac{\left({10}^{-4}{\alpha }_2+{10}^{-3}{\alpha }_3+{10}^{-2}{\alpha }_4\right)}{{\alpha }_1+{\alpha }_2+{\alpha }_3+{\alpha }_4}{\sigma }_{\text{g}}^2.$$ (24)

In particular, Erbe et al. (2012) took α₁ = α₂ = α₃ = α₄ = 1, producing a uniform distribution on π. It follows that all SNPs have the same marginal prior distribution, with null mean, and variance

$$\text{Var}\left({\beta }_j|\alpha \right)=\frac{r^2{\sigma }^2}{400}\left(1+\frac{1}{10}+\frac{1}{100}\right)=\frac{111}{4\times {10}^4}{r}^2{\sigma }^2.$$

This suggests that a simple ridge regression–BLUP obtained by solving

$$\left[X^{\prime }X+\frac{{\sigma }_{\text{e}}^2\left({\alpha }_1+{\alpha }_2+{\alpha }_3+{\alpha }_4\right)}{r^2{\sigma }^2\left({10}^{-4}{\alpha }_2+{10}^{-3}{\alpha }_3+{10}^{-2}{\alpha }_4\right)}\right]\widehat{\beta }=X^{\prime }y$$

may deliver predictive abilities that are similar to those of Bayes R, except that it would differ with respect to Bayes R on how marker effects are shrunk.

Insight on how shrinkage takes place in Bayes R is gained by inspecting the joint posterior density of all marker effects, given r², σ², and π. Here

$$\begin{array}{c}p\left(\beta |y,\mathit{\pi },r^2,{\sigma }^2\right)\propto \text{exp}\left(-\frac{\left(y-X\beta \right)\prime \left(y-X\beta \right)}{2{\sigma }_{\text{e}}^2}\right)\\ \times {\displaystyle \prod _{j=1}^p\left[{\pi }_1\times 0+{\pi }_{2}N\left({\beta }_j|0,{\sigma }_2^2\right)+{\pi }_{3}N\left({\beta }_j|0,{\sigma }_3^2\right)+{\pi }_{4}N\left({\beta }_j|0,{\sigma }_4^2\right)\right]},\end{array}$$ (25)

where ${\sigma }_2^2=r^2{\sigma }^2{10}^{-4},{\sigma }_3^2=r^2{\sigma }^2\hspace{0.5em}{10}^{-3}$, and ${\sigma }_4^2=r^2{\sigma }^2\hspace{0.5em}{10}^{-2}$ (these values can be modified a piacere). Taking derivatives of the log-posterior with respect to β gives (apart from an additive constant)

$$\begin{array}{c}\frac{\partial }{\partial \beta }\text{log}\left[p\left(\beta |y,\mathit{\pi },r^2,{\sigma }^2\right)\right]=\frac{1}{{\sigma }_{\text{e}}^2}\left(X^{\prime }y-X^{\prime }X\beta \right)\\ +\left\{\frac{{\displaystyle {\sum }_{i=2}^4{\pi }_i\frac{d}{d{\beta }_j}{\phi }_i\left({\beta }_j|0,{\sigma }_i^2\right)}}{{\pi }_2{\phi }_2\left({\beta }_j|0,{\sigma }_2^2\right)+{\pi }_3{\phi }_3\left({\beta }_j|0,{\sigma }_3^2\right)+{\pi }_4{\phi }_4\left({\beta }_j|0,{\sigma }_4^2\right)}\right\},\end{array}$$ (26)

where {.} denotes a p × 1 vector. Above, ${\phi }_i\left({\beta }_j|0,{\sigma }_i^2\right)$(i = 2, 3, 4) is the density of β_j under the normal distribution corresponding to component i of the mixture, with

$$\frac{d}{d{\beta }_j}{\phi }_i\left({\beta }_j|0,{\sigma }_i^2\right)=-\frac{{\phi }_i\left({\beta }_j|0,{\sigma }_i^2\right)}{{\sigma }_i^2}{\beta }_j.$$

Employing the preceding expression in Equation 26 yields

$$\frac{\partial }{\partial \beta }\text{log}\left[p\left(\beta |\mathit{\pi },r^2,{\sigma }^2\right)\right]=\left(\frac{1}{{\sigma }_{\text{e}}^2}\right)\left(X^{\prime }y-X^{\prime }X\beta \right)-\left\{\frac{{\displaystyle {\sum }_{i=2}^4{\pi }_i\frac{{\phi }_i\left({\beta }_j|0,{\sigma }_i^2\right)}{{\sigma }_i^2}}}{{\displaystyle {\sum }_{i=2}^4{\pi }_i{\phi }_i\left({\beta }_j|0,{\sigma }_i^2\right)}}{\beta }_j\right\}.$$ (27)

Setting this to zero and rearranging leads to iteration

$${\beta }^{\left[t+1\right]}={\left(X^{\prime }X+{\Omega }_{{\beta }^{\left[t\right]}}\right)}^{-1}{X}^{\prime }y,$$

where ${\Omega }_{{\beta }^{\left[t\right]}}$ is a p × p diagonal matrix with typical element

$${\Omega }_{jj,\mathit{\beta }}^{\left[t\right]}={\sigma }_{\text{e}}^2\frac{{\displaystyle {\sum }_{i=2}^4{\pi }_i{\phi }_i\left({\beta }_j^{\left[t\right]}|0,{\sigma }_i^2\right)\left(1/{\sigma }_i^2\right)}}{{\displaystyle {\sum }_{i=2}^4{\pi }_i{\phi }_i\left({\beta }_j^{\left[t\right]}|0,{\sigma }_i^2\right)}}={\displaystyle \sum _{i=2}^4{\pi }_{ij}^{\prime \left[t\right]}\frac{{\sigma }_{\text{e}}^2}{{\sigma }_i^2},}$$ (28)

where

$$\begin{array}{cc}{\pi }_{ij}^{\prime \left[t\right]}\left({\beta }_j\right)=\frac{{\pi }_i{\phi }_i\left({\beta }_j^{\left[t\right]}|0,{\sigma }_i^2\right)}{{\displaystyle {\sum }_{i=2}^4{\pi }_i{\phi }_i\left({\beta }_j^{\left[t\right]}|0,{\sigma }_i^2\right)}},& i=1,2,\dots ,4\text{and}j=1,2,\dots ,p\end{array}.$$

This is interpretable as the probability that a value β_j in the course of iteration comes from the ith component of the mixture, as the value of β_j changes iteratively. Note that Ω_jj,β is a weighted average of the shrinkage factors ${\sigma }_{\text{e}}^2/{\sigma }_i^2$ corresponding to those that would be employed if the variance parameter of the ith component of the mixture were to be used in ridge regression–BLUP. If ${\sigma }_i^2$ is taken as constant over the three “slab” components, Bayes R reduces to BLUP. On the other hand, when ${\sigma }_i^2$ varies over components, the ratio ${\sigma }_{\text{e}}^2/{\sigma }_i^2$ will be larger for components having the smallest variance. Observe that π₁ does not play a role in this posterior mode interpretation of how Bayes R effects shrinkage.

In summary, Bayes R assigns the same prior distribution to all markers in the battery of SNPs, one with null mean and variance (for a mixture of K components)

$$\text{Var}\left(\beta |\alpha \right)={\displaystyle \sum _{k=1}^K\frac{{\alpha }_k}{{\displaystyle {\sum }_{k=1}^K{\alpha }_k}}{\sigma }_k^2},$$

where the α′s are the parameters of the prior distribution of the mixing probabilities π. Bayes R takes ${\sigma }_1^2=0$.

The superior performance of Bayes R over other methods found by Erbe et al. (2012) probably results from using prior empirical knowledge about r², the assumed reliability. Bayes R has been extended to Bayes RS (Brondum et al. 2012). This is a minor variant of Bayes R in which the mixture (23) is expanded by a factor S, so that there are now S mixtures of four normal distributions each. The letter S denotes a number of chromosome segments constructed in some manner that reflects prior knowledge that some such segments contribute more variance than others. Using the arguments outlined above, it is easy to see that Bayes RS leads to a shrinkage that, instead of being component specific, is now region-component specific.

An incorrect prior often used in the Bayesian alphabet

The following statement is found at high frequency in the genomic selection literature: “The prior distribution of the residual variance is ${\chi }^{-2}\left({\sigma }_{\text{e}}^2|{\nu }_{\text{e}}=-2,S_{\text{e}}^2=0\right)$, meaning that the degrees of freedom of the prior is −2 and that the scale parameter is null.” Examples are Meuwissen et al. (2001) and Jia and Jannink (2012), respectively. Note that Bayes’s theorem returns with null posterior density or probability any parameter value that is assigned 0 density or mass a priori. If the prior density (or probability) of parameter θ is such that p(θ|hyperparameters) = 0, it must be that

$$p\left(\theta |\text{hyperparameters},y\right)=\frac{p\left(y|\theta ,\text{hyperparameters}\right)\times 0}{p\left(y|\text{hyperparameters}\right)}=0,$$

as well. The prior ${\chi }^{-2}\left({\sigma }_{\text{e}}^2|{\nu }_{\text{e}}=-2,S_{\text{e}}^2=0\right)$ is absurd for two reasons. First, a scaled inverted chi-square distribution exists only if both ν_e and $S_{\text{e}}^2$ are >0. To see the second reason, we write the prior density explicitly, that is,

$$p\left({\sigma }_{\text{e}}^2|{\nu }_{\text{e}},S_{\text{e}}^2\right)=\frac{{\left({\nu }_{\text{e}}{S}_{\text{e}}^2/2\right)}^{\frac{{\nu }_{\text{e}}}{2}}}{\Gamma \left({\nu }_{\text{e}}/2\right)}\times {\left({\sigma }_{\text{e}}^2\right)}^{-\left(\frac{{\nu }_{\text{e}}}{2}+1\right)}\text{exp}\left(-\frac{{\nu }_{\text{e}}{S}_{\text{e}}^2}{2{\sigma }_{\text{e}}^2}\right),$$ (29)

so for $S_{\text{e}}^2=0$ and any “legal” value of ν_e, $p\left({\sigma }_{\text{e}}^2|{\nu }_{\text{e}},S_{\text{e}}^2=0\right)=0\vee {\sigma }_{\text{e}}^2$. Then, it must be that $p\left({\sigma }_{\text{e}}^2|{\nu }_{\text{e}},S_{\text{e}}^2,y\right)=0\vee {\sigma }_{\text{e}}^2$ as well. Hence, a scaled inverted chi-square with a null scale parameter is not a probability model at all, as it does not assign appreciable density to any value of the unknown residual variance. It does not convey uncertainty whatsoever: any value of the residual variance is assigned a density of zero, prior and posterior to observing data.

A possible reason for this mistake is as follows: Sorensen and Gianola (2002), as many other Bayesians often do, write the prior as being proportional to the kernel of the scaled inverted chi-square density, that is, as

$$p\left({\sigma }_{\text{e}}^2|{\nu }_{\text{e}},S_{\text{e}}^2\right)\propto {\left({\sigma }_{\text{e}}^2\right)}^{-\left(\frac{{\nu }_{\text{e}}}{2}+1\right)}\text{exp}\left(-\frac{{\nu }_{\text{e}}{S}_{\text{e}}^2}{2{\sigma }_{\text{e}}^2}\right),$$

and note that this kernel reduces to a uniform distribution by taking ν_e = −2 and $S_{\text{e}}^2=0$, yielding $p\left({\sigma }_{\text{e}}^2|{\nu }_{\text{e}},S_{\text{e}}^2\right)\propto 1$. However, it takes more than a kernel to make a density, as multiplication of 1 times the integration constant ${\left({\nu }_{\text{e}}{S}_{\text{e}}^2/2\right)}^{{\nu }_{\text{e}}/2}/\Gamma \left({\nu }_{\text{e}}/2\right)$ produces zero.

Discussion

The main message from this article is that it is not clear how one can learn about genetic architecture from data in n ≪ p situations. This is because individual marker effects are not estimable from the likelihood, apart from the fact that it is unlikely that a multiple linear regression provides a sensible description of biological complexity. On the other hand, it is feasible to learn about the signal Xβ because there is information about this unknown vector in the data, although equivalent to that conveyed by a sample of size 1. Unfortunately, the Bayesian alphabet (Gianola et al. 2009) continues to grow under the incorrect perception that different specifications stemming from various choices of prior inform about genetic architecture; Erbe et al. (2012) and Brondum et al. (2012) provide good examples of this. It is difficult to defend such claims unless n ≫ p and provided that the model is “true” and effectively sparse (Wimmer et al. 2012). Otherwise, the prior always matters whenever n ≪ p and different priors lead to different claims about the state of nature, merely because their shrinkage behavior in finite samples varies. All members of the alphabet produce unique point and interval Bayesian estimates of marker effects, but the driver is the prior and not the data.

Mixtures of Gaussian distributions are widely used in nonparametric density estimation (Wasserman 2010) because most distributions can be approximated well. Mixtures can capture vagaries from cryptic distributions but at the expense of parsimony, thus posing the risk of copying noise, as opposed to signal, especially if the mixture model has too many parameters. McLachlan and Peel (2000) give a warning: estimation of the parameters of a mixture (Ψ) on the basis of data are meaningful only if Ψ is likelihood identifiable. In Bayes RS (apart from nuisance effects and the residual variance) the number of unknown parameters is 2p + 4S. Here, 2p comes from the fact that each marker is assigned a distinct variance; the 4S comes from the fact that there are S segments each having four segment-specific mixing probabilities π_s(s = 1, 2, …, S). Unfortunately, n ≪≪< 2p + 4S, and this creates a huge identification deficit relative to the information content in a sample of size n. In a Bayesian context, there is the additional issue (occurring even when n > p) called label switching, leading Celeux et al. (2000) to write: “Although somewhat presumptuous, we consider that almost the entirety of Markov chain Monte Carlo samplers for mixture models has failed to converge!” In view of these pitfalls, one wonders what meaningful mechanistic sense can be extracted from these richly parameterized specifications intended to inform about genetic architecture.

Although their inferential outcomes may be misleading, one should not dismiss the potential value of Bayes B, C, R, RS, or of any of the mixture models proposed so far as prediction machines. Predictive distributions stemming from the various members of the alphabet may be analytically distinct from each other, but such differences are seldom revealed in cross-validation (e.g., Heslot et al. 2012); an exception is Lehermeier et al. (2013). Below we review how the alphabet can be interpreted from a predictive perspective.

A pioneer of Bayesian predictive inference (Geisser 1993) wrote:

Clearly hypothesis testing and estimation as stressed in almost all statistics books involve parameters…this presumes the truth of the model and imparts an inappropriate existential meaning to an index or parameter…inferring about observables is more pertinent since they can occur and be validated to a degree that is not possible for parameters.

Bayesian methods play an important role in machine learning (e.g., Bishop 2006; Barber 2012; Dehmer and Basak 2012; Rogers and Girolami 2012). A reason is that Bayes’s theorem provides a predictive distribution automatically, something that has not been appreciated in full yet in the whole-genome prediction literature.

The problem of prediction can be cast as one of making statements about future data y_f, given past data y. A model M (e.g., Bayes L) with parameter vector θ is fitted (trained) to y, leading to the posterior distribution p(θ|y, H, M), where H denotes hyperparameters. If y_f is treated as an unknown, the prior becomes p(θ, y_f|H, M) = p(y_f|θ, M)p(θ|H, M) so that

$$p\left(\mathit{\theta },y_{\text{f}}|y,\text{H, M}\right)\propto p\left(y|\mathit{\theta },y_{\text{f}},\text{M}\right)p\left(y_{\text{f}}|\mathit{\theta },\text{M}\right)p\left(\mathit{\theta }|\text{H},\text{M}\right).$$

Since past observations do not depend on future observations, given the parameters, p(y|θ, y_f, M) = p(y|θ, M), so that

$$p\left(y_{\text{f}}|y,\text{H},\text{M}\right)\propto {\displaystyle \int p\left(y_{\text{f}}|\mathit{\theta },\text{M}\right)p\left(\mathit{\theta }|y,\text{H},\text{M}\right)d\mathit{\theta }}.$$ (30)

This is the predictive distribution, where parameters θ do not necessarily play an “existential role” in the sense of Geisser (1993); rather, they are tools enabling one to go from past to future observations. Note that

$$p\left(y_{\text{f}}|y,\text{H},\text{M}\right)=E_{\mathit{\theta }|y,\text{H},\text{M}}\left[p\left(y_{\text{f}}|\mathit{\theta },\text{M}\right)\right],$$

meaning that the predictive distribution weights an infinite number of predictions made at a specific values of θ, with the averaging distributions being p(θ|y, H, M); this posterior conveys the plausibility assigned to a specific value of θ, posterior to the observed data y. For example, for ridge regression–BLUP, the posterior distribution of β is $\beta |y,\text{variances}\sim N\left(\tilde{\beta },{\left(X^{\prime }X+I\lambda \right)}^{-1}{\sigma }_{\text{e}}^2\right),$ where $\tilde{\beta }$ is the solution to Equation 5. It follows that the posterior distribution of the signal is $X\beta |y,\text{variances}\sim N\left(X\tilde{\beta },X{\left(X^{\prime }X+I\lambda \right)}^{-1}{X}^{\prime }{\sigma }_{\text{e}}^2\right).$ This implies that the predictive distribution of a future vector of data y_f = X_fβ + e_f would also be normal

$$y_{\text{f}}|y,\text{variances}\sim N\left(X_{\text{f}}\tilde{\beta },X_{\text{f}}{\left(X^{\prime }X+I\lambda \right)}^{-1}{X}_{\text{f}}^{\prime }{\sigma }_{\text{e}}^2+I_{\text{f}}{\sigma }_{{\text{e}}_{\text{f}}}^2\right).$$

Here, the strong assumption that the stochastic process generating current and future data are the same is made; typically, it is assumed that ${\sigma }_{{\text{e}}_{\text{f}}}^2={\sigma }_{\text{e}}^2,$ but this may not be realistic. While the different priors of the alphabet lead to different predictive distributions, it is to be expected that at least the point predictions will be fairly similar. This is because Xβ is identified in the likelihood, so some Bayesian learning about the signal will take place, especially when y is a vector of preprocessed means (e.g., means of daughter yield deviations for a battery of dairy cattle bulls with a large number of progeny records). In the latter case, the various members of the alphabet are expected to differ minimally in predicting ability.

The predictive distribution can be used to check whether observed data are consistent with what a model would lead one to expect. Sorensen and Waagepetersen (2003) used this idea to examine goodness of fit of model for litter size in pigs. However, the predictive approach outlined above does not take uncertainty about the model into account, and this may understate variability seriously. Bayesians address this via model averaging, where the predictive distribution is averaged over models, that is,

$$p\left(y_{\text{f}}|y\right)={\displaystyle \int p\left(y_{\text{f}}|y,{\text{H}}_{\text{M}},\text{M}\right)d\mu \left(\text{M}|y\right)}.$$

This integral represents both the situation where the number of models is finite and countable or infinite. In the first case the integral is a sum and the measure μ(M|y) is the posterior probability of model M. In the second case the number of possible models may be huge, e.g., in variable selection approaches for linear models aiming to include or exclude p markers, there are 2^p possible specifications. If p is very large, the number of models is practically infinite, so the measure μ(M|y) is the posterior density assigned to a specific model.

Although p(y_f|y) provides a more sensible assessment of predictive uncertainty, in practice one proceeds by constructing cross-validation distributions, with respect to one or several competing models. Each prediction generates an error, and this error will have a cross-validation distribution. The relevance of cross-validation is another important contribution of Meuwissen et al. (2001) to whole-genome prediction. Here, hyperparameters of genomic selection models (e.g., π in Bayes Cπ) can be viewed as “tuning knobs” and evaluated over a grid. Unfortunately, the reality is that Manhattan plots tend to overwhelm cross-validation graphs in genome-wide association studies.

Also, differences in predictive ability are often masked by the variation conveyed by a properly constructed cross-validation distribution (e.g., González-Camacho et al. 2012). On the other hand, the various Bayesian predictive machines resulting from different priors may possess differential robustness in finite samples. For instance, some priors may be less sensitive with respect to differences in true genetic architecture (Wimmer et al. 2012).

Given that the data do not contain information about individual marker effects, variation in inference is an artifact caused by the various priors. This leads to the question: How much does one prior differ from another one? Information on this can be obtained by use of some notion of statistical distance between distributions, such as the Kullback–Leibler (KL) metric. For example, Gianola et al. (2009) used KL for debunking the notion that marker-specific-effect variances in Bayes A tell us something about genetic variability of chromosomal regions. Recently, Lehermeier et al. (2013) used a metric that is easier to interpret than KL, the Hellinger distance or HD (e.g., Roos and Held 2011). They found that Bayesian learning in Bayes A and Bayes B was more limited than with Bayes L or Bayesian ridge regression. In our context, the HD between prior $N\left(\beta |0,{\sigma }_{\beta }^2\right)$ assigned to a marker effect in ridge regression and prior $t\left(\beta |0,S_{\beta }^2,\nu \right)$ of Bayes A is

$$\text{HD}\left(N,t\right)=\sqrt{1-{\displaystyle \int \sqrt{N\left(\beta |0,{\sigma }_{\beta }^2\right)t\left(\beta |0,S_{\beta }^2,\nu \right)}d\beta }.}$$

HD takes values between 0 and 1, with 1 corresponding to the situation where, say, any realization from $t\left(\beta |0,S_{\beta }^2,\nu \right)$ is assigned 0 density under $N\left(\beta |0,{\sigma }_{\beta }^2\right)$, and vice versa. Similar expressions hold for HD(N, DE), where DE(β|0, λ) is the zero-mean double-exponential distribution with parameter λ that is used in Bayes L and for HD(t, DE). To compare these three priors, we took ${\sigma }_{\beta }^2=1$, $S_{\beta }^2\nu /\left(\nu -2\right)=1$, and $2/{\lambda }^2=1$, so that the three priors had the same variance; for the t-distribution we assigned ν = 6, to produce sufficiently thick tails. With these assignments $S_{\beta }^2=\frac{2}{3}$ and $\lambda =\sqrt{2},$ so that, using numerical integration between −10 and 10, HD(N, t) = 0.0690. Further, HD(N, DE) = 0.122, and

$$\text{HD}\left(t,\text{DE}\right)=\sqrt{1-{\displaystyle \int \sqrt{\frac{\Gamma \left[3.5\right]{\left[1+{\beta }^2/4\right]}^{-\left(3.5\right)}}{\Gamma \left[3\right]\sqrt{4\pi }}\frac{\exp \left(-\sqrt{2}\left|\beta \right|\right)}{\sqrt{2}}}d\beta }}=\mathrm{0.06.}$$

This shows, at least when variances are matched, that these three priors are not too different from each other, so differences in inference would stem from difference in the type and extent of shrinkage effected. However, if priors are not matched, these distances would be expected to increase. Since ridge regression–BLUP, Bayes A, and Bayes L postulate the same sampling model, whenever n ≪ p differences in posterior inferences between these three members of the Bayesian alphabet must be due to the fact that the priors are very different and influential.

To conclude, whole-genome prediction can be useful for providing locally valid predictions of complex traits. However, the additive regression models employed therein should not be taken at face value from an inferential perspective unless an additive model with many 0 coefficients turns out to hold as approximately true (oracle property 1 met), and n ≫> p₀, where p₀ is the number of nonzero coefficients (oracle property 2 met). If these two conditions are (ever) fulfilled, it may be that the genetic architecture of the very elusive additive QTL (on whose existence the statistical abstraction of marker-assisted inference is based) will be unraveled by statistical means.

The question of the extent to which an additive genetic model is a good representer of complexity is another issue yet to be sorted out. The Bayesian alphabet may expand further on this matter, e.g., Bayes A may grow into Bayes AAA if additive × additive × additive epistasis is included in a model. Additional expansions of the Bayesian alphabet to accommodate epistatic interactions will further exacerbate the inferential problems, because of a vast increase in number of regression coefficients. It is far from obvious how genetic architecture of complex traits can be learned via highly dimensional statistical models.

Appendix

Bias of BLUP with Respect to Marker Effects

As a toy example, let n = 3 and p = 4. The model includes an intercept plus the effect of 3 markers, and the incidence matrix is

$$X=\left[\begin{array}{cccc}1& 0& -1& 1\\ 1& 1& 0& 0\\ 1& 1& 1& -1\end{array}\right].$$

The first column contains the dummy variable for the intercept, and the remaining columns are the genotype codes for the markers at each of three loci. The first observation (row 1 of X) pertains to an individual that is Aa (coded as 0), bb(coded as −1), CC (coded as1), and so on. This matrix has rank 3, and a generalized inverse of X′X is

$${\left(X^{\prime }X\right)}^{-}=\left[\begin{array}{cccc}3& -4& 2& 0\\ -4& 6& -3& 0\\ 2& -3& 2& 0\\ 0& 0& 0& 0\end{array}\right].$$

If the true values of the intercept and of the marker effects are denoted as a, b, c, and d, respectively, the expected value of the maximum-likelihood estimator of the four parameters is

$$E\left({\beta }^{\left(0\right)}|\beta \right)=\left[\begin{array}{c}a\\ b\\ c-d\\ 0\end{array}\right],$$

with the expected value of the effect of the third marker being 0 instead of d because of the rank deficiency. Now, we use BLUP with $V_{\beta }=I_3{\sigma }_{\beta }^2$ and variance ratio $\lambda ={\sigma }_{\text{e}}^2/{\sigma }_{\beta }^2$ (${\sigma }_{\beta }^2$ is the variance of marker effects) and calculate it (Henderson 1984) as

$$\text{BLUP}\left(\beta \right)={\left(X^{\prime }X+I\lambda \right)}^{-1}{X}^{\prime }y.$$

For this example,

$${\left(X^{\prime }X+I\lambda \right)}^{-1}=\left[\begin{array}{cccc}\frac{{\lambda }^2+6\lambda +6}{k}& -\frac{2\lambda +8}{k}& \frac{2}{k}& -\frac{2}{k}\\ -\frac{2\lambda +8}{k}& \frac{{\lambda }^2+7\lambda +12}{k}& -\frac{\lambda +3}{k}& \frac{\lambda +3}{k}\\ \frac{2}{k}& -\frac{\lambda +3}{k}& \frac{{\lambda }^3+7{\lambda }^2+11\lambda +1}{k\lambda }& \frac{2{\lambda }^2+9\lambda +1}{k\lambda }\\ -\frac{2}{k}& \frac{\lambda +3}{k}& \frac{2{\lambda }^2+9\lambda +1}{k\lambda }& \frac{{\lambda }^3+7{\lambda }^2+11\lambda +1}{k\lambda }\end{array}\right],$$

where k = λ³ + 9λ² + 20λ + 2. Then

$$E\left(\text{BLUP}\left(\beta \right)|\beta \right)={\left(X^{\prime }X+I\lambda \right)}^{-1}X\prime X\left[\begin{array}{c}a\\ b\\ c\\ d\end{array}\right].$$

After tedious algebra, one arrives at

$$E\left(\text{BLUP}\left(\beta \right)|\beta \right)=\left[\begin{array}{c}a\left(\frac{3}{k}{q}_4-\frac{2}{k}{q}_6\right)+b\left(\frac{4}{k\lambda }+\frac{2}{k}{q}_4-\frac{2}{k}{q}_6\right)-c\left(\frac{1}{k}{q}_6-\frac{8}{k\lambda }\right)+d\left(\frac{1}{k}{q}_6-\frac{8}{k\lambda }\right)\\ a\left(\frac{2}{k}{c}_2-\frac{3}{k}{q}_6\right)-b\left(\frac{2}{k}{q}_6-\frac{2}{k}{q}_2+\frac{2}{k\lambda }{q}_5\right)+c\left(\frac{1}{k}{q}_2-\frac{4}{k\lambda }{q}_5\right)-d\left(\frac{1}{k}{q}_2-\frac{4}{k\lambda }{q}_5\right)\\ a\left(\frac{6}{k}-\frac{2}{k}{q}_5\right)+b\left(\frac{4}{k}-\frac{2}{k}{q}_5-\frac{1}{k\lambda }{q}_3+\frac{1}{k\lambda }{q}_1\right)-c\left(\frac{1}{k}{q}_5+\frac{2}{k\lambda }{q}_3-\frac{2}{k\lambda }{q}_1\right)+d\left(\frac{1}{k}{q}_5+\frac{2}{k\lambda }{q}_3-\frac{2}{k\lambda }{q}_1\right)\\ -a\left(\frac{6}{k}-\frac{2}{k}{q}_5\right)-b\left(\frac{4}{k}-\frac{2}{k}{q}_5-\frac{1}{k\lambda }{q}_3+\frac{1}{k\lambda }{c}_1\right)+c\left(\frac{1}{k}{q}_5+\frac{2}{k\lambda }{q}_3-\frac{2}{k\lambda }{q}_1\right)-d\left(\frac{1}{k}{q}_5+\frac{2}{k\lambda }{q}_3-\frac{2}{k\lambda }{q}_1\right)\end{array}\right],$$

where

$$\begin{array}{l}{q}_1={\lambda }^3+7{\lambda }^2+11\lambda +1,\\ q_2={\lambda }^2+7\lambda +12,\\ q_3=2{\lambda }^2+9\lambda +1,\\ q_4={\lambda }^2+6\lambda +6,\\ q_5=\lambda +3,\\ q_6=2\lambda +8.\end{array}$$

Conditionally on β, all marker effects are estimated with a bias that involves all other markers (and the intercept as well). Since inferences on genetic architecture are primarily based on point estimates (it should be noted that the biased estimator is more precise), it is quite clear that such inferences are not “clean.”

Marker Effects Are Not Identified from a Bayesian Perspective in the n < p Setting

Let a Bayesian linear model consist of location parameters θ_A and θ_B (this partition has a different meaning from the one given above), with likelihood p(y|θ_A, θ_B). If the conditional posterior density of θ_B is such that

$$p\left({\mathit{\theta }}_{\text{B}}|{\mathit{\theta }}_{\text{A}},y\right)=p\left({\mathit{\theta }}_{\text{B}}|{\mathit{\theta }}_{\text{A}}\right),$$

then θ_B is not identifiable, meaning that observation of data does not increase knowledge about θ_B beyond what is conveyed by the conditional prior p(θ_B|θ_A) (Dawid 1979; Gelfand and Sahu 1999). For the model in (1), in the n < p situation matrix X_n×p has rank n, and one can reorganize its columns into

$$X\beta =\left[\begin{array}{cc}{X}_1& X_2\end{array}\right]\left[\begin{array}{c}{\beta }_1\\ {\beta }_2\end{array}\right],$$

where X₁ is n × n with rank n, and X₂ is n × (p − n), with the vector of marker effects β partitioned accordingly. Changing variables as

$$\left[\begin{array}{c}{\mathit{\theta }}_{\text{A}}\\ {\mathit{\theta }}_{\text{B}}\end{array}\right]=\left[\begin{array}{cc}{X}_1& X_2\\ 0& I_{\left(p-n\right)\times \left(p-n\right)}\end{array}\right]\left[\begin{array}{c}{\beta }_1\\ {\beta }_2\end{array}\right]$$

produces the inverse transformations β₂ = θ_B and ${\beta }_1=X_1^{-1}\left({\mathit{\theta }}_{\text{A}}-X_2{\mathit{\theta }}_{\text{B}}\right)$; because the transformation is linear, the Jacobian does not involve the parameters. Using the new parameterization model (1) can now be written as

$$y={\mathit{\theta }}_{\text{A}}+e,$$

implying that the data contain information about θ_A but not about θ_B (the latter can represent any marker effect, by construction). Then, irrespective of the joint prior distribution assigned to θ_A and θ_B, the posterior is

$$p\left({\mathit{\theta }}_{\text{A}},{\mathit{\theta }}_{\text{B}}|y\right)\propto p\left(y|{\mathit{\theta }}_{\text{A}},{\mathit{\theta }}_{\text{B}}\right)p\left({\mathit{\theta }}_{\text{B}}|{\mathit{\theta }}_{\text{A}}\right)p\left({\mathit{\theta }}_{\text{A}}\right)\propto p\left(y|{\mathit{\theta }}_{\text{A}}\right)p\left({\mathit{\theta }}_{\text{B}}|{\mathit{\theta }}_{\text{A}}\right)p\left({\mathit{\theta }}_{\text{A}}\right),$$

so

$$p\left({\mathit{\theta }}_{\text{B}}|{\mathit{\theta }}_{\text{A}},y\right)=p\left({\mathit{\theta }}_{\text{B}}|{\mathit{\theta }}_{\text{A}}\right),$$

verifying that the p − n marker effects are not likelihood identified. As pointed out by Gelfand and Sahu (1999), this does not mean that there is not Bayesian learning about θ_B. It means, however, that data “speak” about θ_A and that what can be said about θ_B depends on what has been spoken about θ_A, with the pipe lining of knowledge done through the prior distribution. This can be seen more clearly by writing the posterior of θ_B as

$$\begin{array}{c}p\left({\mathit{\theta }}_{\text{B}}|y\right)={\displaystyle \int p\left({\mathit{\theta }}_{\text{B}}|{\mathit{\theta }}_{\text{A}},y\right)p\left({\mathit{\theta }}_{\text{A}}|y\right)d{\mathit{\theta }}_{\text{A}}}={\displaystyle \int p\left({\mathit{\theta }}_{\text{B}}|{\mathit{\theta }}_{\text{A}}\right)p\left({\mathit{\theta }}_{\text{A}}|y\right)d{\mathit{\theta }}_{\text{A}}}\\ =E_{p\left({\mathit{\theta }}_{\text{A}}|y\right)}\left[p\left({\mathit{\theta }}_{\text{B}}|{\mathit{\theta }}_{\text{A}}\right)\right].\end{array}$$

This representation enables one to see that marginal inferences about individual marker effects are the weighted average of an infinite number of inferences made from the conditional prior p(θ_B|θ_A), where the averaging distribution is the posterior of the signal p(θ_A|y). If θ_B is any marker effect, say β_j, the preceding becomes

$$p\left({\beta }_j|y\right)={\displaystyle \int \left[p\left({\beta }_j|X_1{\beta }_1\right)\right]p\left(X_1{\beta }_1|y\right)d\left(X_1{\beta }_1\right)}.$$

In conclusion, for any letter of the alphabet and for any prior distribution adopted, any inference made about genetic architecture always depends on the form of p(β_j|X₁β₁) or, more generally, of p(θ_B|θ_A), and these densities depend on the prior adopted, but not on the data. Proper Bayesian learning takes place for X₁β₁ only.

Inferences in a Linear Model with Unidentified Parameters

In the context of model (1), the likelihood function (assuming known ${\sigma }_{\text{e}}^2$) is

$$l\left(\beta |y,{\sigma }_{\text{e}}^2\right)\propto \text{exp}\left[-\frac{\left(y-X\beta \right)\prime \left(y-X\beta \right)}{2{\sigma }_{\text{e}}^2}\right].$$

For the n < p situation, and with β⁽⁰⁾ being a solution to the normal equations corresponding to generalized inverse (X′X)⁻, the (singular) likelihood is expressible as

$$l\left(\beta |y,{\sigma }_{\text{e}}^2\right)\propto \text{exp}\left(-\frac{\left(\beta -{\beta }^{\left(0\right)}\right)\prime X\prime X\left(\beta -{\beta }^{\left(0\right)}\right)}{2{\sigma }_{\text{e}}^2}\right).$$

Letting r = rank(X) and using results from linear model theory (if n ≪ p, then r ≤ n), it follows that

$$X_{n\times p}{\beta }_{p\times 1}=\left(X_{n\times p}{Q}_{1,p\times r}\right)\left(L_{r\times p}{\beta }_{p\times 1}\right)+\left(X_{n\times p}{Q}_{2,p\times \left(p-r\right)}\right)\left(H_{\left(p-r\right)\times p}{\beta }_{p\times 1}\right)=K_1{\alpha }_1+K_2{\alpha }_2,$$

where Q₁ and Q₂ are partitions of a p × p matrix of rank-preserving elementary operators (Searle 1966); α₁ = Lβ is an r × 1 vector of likelihood-identified estimable functions and α₂ = Hβ is a (p − r) × 1 vector of pseudoparameters; K₁ = XQ₁ and K₂ = XQ₂ are incidence matrices, with K₂ = 0 (α₂ is a pseudoparameter, because it is effectively wiped out of the model). The genetic signal is given by K₁α₁ but we include α₂ as well, to see what Bayesian inference does for something on which the data lack information.

If β is assigned the normal prior N(β|0, V_β),

$$\left[\begin{array}{c}{\alpha }_1\\ {\alpha }_2\end{array}\right]\mathrm{|}{V}_{\beta }\sim N\left(\left[\begin{array}{c}0\\ 0\end{array}\right],\left[\begin{array}{cc}L{V}_{\beta }L\prime & L{V}_{\beta }H\prime \\ H{V}_{\beta }L\prime & H{V}_{\beta }H\prime \end{array}\right]\right).$$ (A.1)

The model is now y = K₁α₁ + K₂α₂ + e, and the likelihood under the new parameterization becomes

$$l\left({\alpha }_1,{\alpha }_2|y,{\sigma }_{\text{e}}^2\right)\propto \text{exp}\left(-\frac{\left[\begin{array}{cc}\left({\alpha }_1-{\alpha }_1^{\left(0\right)}\right)\prime & \left({\alpha }_2-{\alpha }_2^{\left(0\right)}\right)\prime \end{array}\right]\left[\begin{array}{cc}{K}_1^{\prime }{K}_1& K_1^{\prime }{K}_2\\ K_2^{\prime }{K}_1& K_2^{\prime }{K}_2\end{array}\right]\left[\begin{array}{c}\left({\alpha }_1-{\alpha }_1^{\left(0\right)}\right)\\ \left({\alpha }_2-{\alpha }_2^{\left(0\right)}\right)\end{array}\right]}{2{\sigma }_{\text{e}}^2}\right)$$ (A.2)

$$=\text{exp}\left(-\frac{\left({\alpha }_1-{\alpha }_1^{\left(0\right)}\right)\prime K_1^{\prime }{K}_1\left({\alpha }_1-{\alpha }_1^{\left(0\right)}\right)}{2{\sigma }_{\text{e}}^2}\right).$$ (A.3)

Expression (A.3) indicates that at most, r parameters are likelihood identified, but (A.2) is retained to illustrate what the prior does. It is well known (e.g., Gianola and Fernando 1986; Sorensen and Gianola 2002) that combining (A.1) with (A.2) leads to the posterior distribution

$$\left[\begin{array}{c}{\alpha }_1\\ {\alpha }_2\end{array}\right]\mathrm{|}y,V_{\beta },{\sigma }_{\text{e}}^2\sim N\left(\left[\begin{array}{c}{\tilde{\alpha }}_1\\ {\tilde{\alpha }}_2\end{array}\right],{\left[\begin{array}{cc}{K}_1^{\prime }{K}_1+V^{11}& K_1^{\prime }{K}_2+V^{12}\\ K_2^{\prime }{K}_1+V^{21}& K_2^{\prime }{K}_2+V^{22}\end{array}\right]}^{-1}{\sigma }_{\text{e}}^2\right),$$ (A.4)

where

$$\begin{array}{c}\left[\begin{array}{c}{\tilde{\alpha }}_1\\ {\tilde{\alpha }}_2\end{array}\right]=\\ {\left[\begin{array}{cc}{K}_1^{\prime }{K}_1+{\sigma }_{\text{e}}^2{V}^{11}& K_1^{\prime }{K}_2+{\sigma }_{\text{e}}^2{V}^{12}\\ K_2^{\prime }{K}_1+{\sigma }_{\text{e}}^2{V}^{21}& K_2^{\prime }{K}_2+{\sigma }_{\text{e}}^2{V}^{22}\end{array}\right]}^{-1}\left[\begin{array}{c}{K}_1^{\prime }y\\ K_2^{\prime }y\end{array}\right]={\left[\begin{array}{cc}{K}_1^{\prime }{K}_1+{\sigma }_{\text{e}}^2{V}^{11}& {\sigma }_{\text{e}}^2{V}^{12}\\ {\sigma }_{\text{e}}^2{V}^{21}& {\sigma }_{\text{e}}^2{V}^{22}\end{array}\right]}^{-1}\left[\begin{array}{c}{K}_1^{\prime }y\\ 0\end{array}\right],\end{array}$$

since K₂ = 0, and where

$$\left[\begin{array}{cc}{V}^{11}& V^{12}\\ V^{21}& V^{22}\end{array}\right]={\left[\begin{array}{cc}L{V}_{\beta }L\prime & L{V}_{\beta }H\prime \\ H{V}_{\beta }L\prime & H{V}_{\beta }H\prime \end{array}\right]}^{-1}.$$

The p-dimensional distribution (A.4) is nonsingular, but it is based on a likelihood that is defined in r dimensions only! Note that the posterior mean satisfies

$$\left[\begin{array}{cc}{K}_1^{\prime }{K}_1+{\sigma }_{\text{e}}^2{V}^{11}& {\sigma }_{\text{e}}^2{V}^{12}\\ {\sigma }_{\text{e}}^2{V}^{21}& {\sigma }_{\text{e}}^2{V}^{22}\end{array}\right]\left[\begin{array}{c}{\tilde{\alpha }}_1\\ {\tilde{\alpha }}_2\end{array}\right]=\left[\begin{array}{c}{K}_1^{\prime }y\\ 0\end{array}\right].$$ (A.5)

The coefficient matrix in (A.5) is the counterpart of X′X and it is proportional to the negative matrix of second derivatives) of the log-posterior with respect to α₁ and α₂. This shows that proper Bayesian learning takes place only for α₁, as the information about α₂ and the co-information about α₁ and α₂ come from the prior only. Further, note the relationship

$${\tilde{\alpha }}_2=-{\left(V^{22}\right)}^{-1}{V}^{21}{\tilde{\alpha }}_1,$$ (A.6)

indicating that what is learned about α₂ is solely a function of what is learned about α₁. This is verified by inserting relationship (A.6) in Equations A.5 above, giving

$$\left(K_1^{\prime }{K}_1+{\sigma }_{\text{e}}^2{V}^{11}-{\sigma }_{\text{e}}^2{V}^{12}{\left(V^{22}\right)}^{-1}{V}^{21}\right){\tilde{\alpha }}_1=K_1^{\prime }y.$$

Using properties of inverses of partitioned matrices, $V_{11}^{-1}=V^{11}-V^{12}{\left(V^{22}\right)}^{-1}{V}^{21},$ so that

$${\tilde{\alpha }}_1={\left(K_1^{\prime }{K}_1+{\sigma }_{\text{e}}^2{V}_{11}^{-1}\right)}^{-1}{K}_1^{\prime }y.$$ (A.7)

The preceding confirms that the data inform about α₁ but not about α₂; what is learned about the latter from phenotypes is done indirectly, through α₁. Such an “indirect” inference parallels the concept of “prediction of breeding values of individuals without phenotypes” (Henderson 1977). In the molecular markers setting, n linear combinations of markers are learned from the data, but p − n remain at the mercy of the prior. In other words, one does not clearly know what marker effects are being learned from the data, unless the model is parameterized deliberately. This is clearly shown in the first section of the Appendix.

An example of Proper Bayesian Learning

To illustrate a case of proper Bayesian learning where a connection with genomic BLUP arises, consider inferring the signal g = Xβ. This is likelihood identified (estimable) because E(y|Xβ) = g, and the likelihood is

$$l\left(g|y\right)\propto \text{exp}\left[-\frac{\left(g-y\right)\prime \left(g-y\right)}{2{\sigma }_{\text{e}}^2}\right],$$

with a maximum at $\widehat{\mathbf{g}}=y$. The information matrix is

$$E_{y|g,{\sigma }_{\text{e}}^2}\left[\frac{{\partial }^2}{\partial g\partial g\prime }\frac{\left(g-y\right)\prime \left(g-y\right)}{2{\sigma }_{\text{e}}^2}\right]=\frac{1}{{\sigma }_{\text{e}}^2}{I}_{n\times n},$$

meaning that, for each individual signal, the information content is proportional to what is conveyed by a sample of size n = 1 (if the response variates are means of preprocessed data, the information content will be higher). For the prior N(β|0, V_β), the resulting prior for the signal is g|V_β ∼ N(0, XV_βX′) and standard results for Bayesian inference give $g|y,V_{\beta },{\sigma }_{\text{e}}^2\sim N\left(\tilde{g},V_{\text{g}}\right)$ as posterior distribution, where

$$\tilde{\mathbf{g}}={\left[\frac{1}{{\sigma }_{\text{e}}^2}I+{\left(X{V}_{\beta }X\prime \right)}^{-1}\right]}^{-1}y.$$

and

$$V_{\text{g}}={\left[\frac{1}{{\sigma }_{\text{e}}^2}I+{\left(X{V}_{\beta }X\prime \right)}^{-1}\right]}^{-1}.$$

A special case is when $V_{\beta }=I_p{\sigma }_{\beta }^2,$ so that for $\lambda ={\sigma }_{\text{e}}^2/{\sigma }_{\beta }^2$ being the variance ratio, $\tilde{\mathbf{g}}={\left[I+{\left(XX\prime \right)}^{-1}\lambda \right]}^{-1}y,$ and $V_{\text{g}}={\left[I+{\left(XX\prime \right)}^{-1}\lambda \right]}^{-1}{\sigma }_{\text{e}}^2.$ Using well-established results known from prediction of random variables dating back to Henderson (1977) but rediscovered recently (e.g., Janss et al. 2012) one can easily find the posterior distribution of β from that of g, and vice versa . Here, take α₁ = Xβ so that ${\alpha }_1\sim N_n\left(0,XX\prime {\sigma }_{\beta }^2\right)$ and ${\tilde{\alpha }}_1=\widetilde{X\beta }=\tilde{\mathbf{g}},$ which is known as genomic BLUP. Any marker effect can be learned indirectly from $\tilde{\mathbf{g}}$ using standard BLUP theory as $\tilde{\beta }=X\prime {\left(XX\prime \right)}^{-1}\tilde{\mathbf{g}}.$

Mode of the Conditional Posterior Distribution in Bayes A

Taking logs of (7) yields

$$L=\text{log}\left[p\left(\beta |S_{\beta }^2,\nu ,{\sigma }_{\text{e}}^2,y\right)\right]=-\frac{1}{2{\sigma }_{\text{e}}^2}{\displaystyle \sum _{i=1}^n{\left(y_i-X_i^{\prime }\beta \right)}^2}-\left(\frac{1+\nu }{2}\right){\displaystyle \sum _{i=1}^p\text{log}\left[1+\frac{{\beta }_j^2}{S_{\beta }^2\nu }\right]}.$$ (A.8)

The gradient vector is

$$\begin{array}{c}\frac{\partial L}{\partial \beta }=-\frac{1}{2{\sigma }_{\text{e}}^2}{\displaystyle \sum _{i=1}^n\frac{\partial {\left(y_i-x_i^{\prime }\beta \right)}^2}{\partial \beta }-\left(\frac{1+\nu }{2}\right)}{\displaystyle \sum _{i=1}^p\frac{\partial }{\partial \beta }\text{log}\left[1+\frac{{\beta }_j^2}{S_{\beta }^2\nu }\right]}=\frac{1}{{\sigma }_e^2}{\displaystyle \sum _{i=1}^n{x}_i\left(y_i-x_i^{\prime }\beta \right)}-\frac{\left(1+\nu \right)}{S_{\beta }^2\nu }\left[\begin{array}{c}\frac{{\beta }_1}{1+\frac{{\beta }_1^2}{S_{\beta }^2\nu }}\\ \frac{{\beta }_2}{1+\frac{{\beta }_2^2}{S_{\beta }^2\nu }}\\ .\\ .\\ .\\ \frac{{\beta }_p}{1+\frac{{\beta }_p^2}{S_{\beta }^2\nu }}\end{array}\right]\\ =\frac{1}{{\sigma }_e^2}X\prime y-\frac{1}{{\sigma }_{\text{e}}^2}\left(X\prime X+W_{\beta }\right)\beta \mathit{,}\end{array}$$ (A.9)

where $X^{\prime }y=\left\{{\displaystyle {\sum }_{i=1}^n{x}_i{y}_i}\right\},$ $X^{\prime }X={\displaystyle {\sum }_{i=1}^n{x}_i{\mathbf{x}}_i^{\prime }},$ and

$$W_{\beta }=\text{Diag}\left\{\frac{{\sigma }_e^2}{S_{\beta }^2}\frac{\left(1+1/\nu \right)}{\left(1+{\beta }_j^2/S_{\beta }^2\nu \right)}\right\}.$$

Setting to zero, to satisfy the first-order condition, leads to

$$\left(X^{\prime }X+W_{\beta }\right)\widehat{\beta }=X^{\prime }y.$$

This system is not explicit in β (because marker effects appear nonlinearly in W_β) but a functional iteration can be developed to locate stationary points.

Mode of the Conditional Posterior Distribution in Bayes L

As a side note, consider what happens if it is not ignored that $W_{\beta }^{-1}=\text{Diag}\left\{1/\left|{\beta }_j\right|\right\}$ is a random matrix, contrary to what was done by Tibshirani (1996) in a modal representation of Bayes L. Recalling that $\left|{\beta }_j\right|={\beta }_j^2/\left|{\beta }_j\right|$ and that $d\left|x\right|/dx=\text{sign}\left(x\right)$,

$$\frac{\partial }{\partial {\beta }_j}\left|{\beta }_j\right|=\frac{\partial }{\partial {\beta }_j}\left(\frac{{\beta }_j^2}{\left|{\beta }_j\right|}\right)=\frac{2{\beta }_j}{\left|{\beta }_j\right|}-\frac{{\beta }_j^2}{{\left|{\beta }_j\right|}^2}\text{sign}\left({\beta }_j\right)=\frac{2{\beta }_j}{\left|{\beta }_j\right|}-\text{sign}\left({\beta }_j\right).$$

Differentiating (11) with respect to β

$$\begin{array}{c}\frac{\partial L\left(\beta |y,\lambda ,{\sigma }_{\text{e}}^2\right)}{\partial \beta }=-\frac{\frac{\partial }{\partial \beta }\left(y-X\beta \right)\prime \left(y-X\beta \right)+{\sigma }_e^2\lambda \frac{\partial }{\partial \beta }{\displaystyle \sum _{j=1}^p\left|{\beta }_j\right|}}{2{\sigma }_{\text{e}}^2}\\ =-\frac{1}{2{\sigma }_{\text{e}}^2}\left[-2X^{\prime }\left(y-X\beta \right)+2{\sigma }_{\text{e}}^2\lambda W_{\beta }^{-1}\beta -{\sigma }_{\text{e}}^2\lambda s_{\beta }\right],\end{array}$$

where s_β is a vector containing the signs of the elements of β. Here, the first-order condition would lead to the iteration

$$\left(X^{\prime }X+{\sigma }_{\text{e}}^2\lambda W_{{\beta }^{\left[t\right]}}^{-1}\right){\beta }^{\left[t+1\right]}=X^{\prime }y+\frac{{\sigma }_{\text{e}}^2\lambda }{2}{s}_{\beta }^{\left[t\right]}.$$

Approximation of an Integral in Bayes L

The integral in (16) can be approximated using a second-order expansion around ${\lambda }^2=r/\delta$ such that (ignoring the subscript in β_j)

$$\text{exp}\left[-\left(\left|\beta \right|\sqrt{{\lambda }^2}\right)\right]\approx e^{-\left|\beta \right|\sqrt{r/\delta }}\left[1-\frac{1}{2}\sqrt{\frac{\delta }{r}}\left|\beta \right|\left({\lambda }^2-\frac{r}{\delta }\right)+\frac{\delta }{8r}\left({\left|\beta \right|}^2+\sqrt{\frac{\delta }{r}}\left|\beta \right|\right){\left({\lambda }^2-\frac{r}{\delta }\right)}^2\right].$$

Use of this in (16) produces

$$\begin{array}{c}{\displaystyle {\int }_0^{\mathrm{\infty }}{\left({\lambda }^2\right)}^{r+\frac{1}{2}-1}\text{exp}\left[-\left(\left|\beta \right|\sqrt{{\lambda }^2}+\delta {\lambda }^2\right)\right]d{\lambda }^2}={\displaystyle {\int }_0^{\mathrm{\infty }}\text{exp}\left[-\left|\beta \right|\sqrt{{\lambda }^2}\right]{\left({\lambda }^2\right)}^{r+\frac{1}{2}-1}\text{exp}\left(-\delta {\lambda }^2\right)d{\lambda }^2}\\ \approx e^{-\left|\beta \right|\sqrt{r/\delta }}{\displaystyle \underset{0}{\overset{\mathrm{\infty }}{\int }}{\left({\lambda }^2\right)}^{r+\frac{1}{2}-1}\text{exp}\left(-\delta {\lambda }^2\right)d{\lambda }^2}\\ -e^{-\left|\beta \right|\sqrt{r/\delta }}\frac{1}{2}\sqrt{\frac{\delta }{r}}\left|\beta \right|{\displaystyle \underset{0}{\overset{\mathrm{\infty }}{\int }}\left({\lambda }^2-\frac{r}{\delta }\right){\left({\lambda }^2\right)}^{r+\frac{1}{2}-1}\text{exp}\left(-\delta {\lambda }^2\right)d{\lambda }^2}\\ +e^{-\left|\beta \right|\sqrt{r/\delta }}\frac{\delta }{8r}\left({\left|\beta \right|}^2+\sqrt{\frac{\delta }{r}}\left|\beta \right|\right){\displaystyle {\int }_{\mathrm{0}}^{\mathrm{\infty }}{\left({\lambda }^2-\frac{r}{\delta }\right)}^2{\left({\lambda }^2\right)}^{r+\frac{1}{2}-1}\text{exp}\left(-\delta {\lambda }^2\right)d{\lambda }^2}.\end{array}$$

Note that ${\left({\lambda }^2\right)}^{r+\frac{1}{2}-1}\text{exp}\left(-\delta {\lambda }^2\right)$ is the kernel of a $\Gamma \left(r+\frac{1}{2},\delta \right)$ distribution, so that

$$\begin{array}{c}{\displaystyle {\int }_0^{\mathrm{\infty }}{\left({\lambda }^2\right)}^{r+\frac{1}{2}-1}\text{exp}\left(-\delta {\lambda }^2\right)d{\lambda }^2}={\left[\frac{{\delta }^{r+\frac{1}{2}}}{\Gamma \left(r+\frac{1}{2}\right)}\right]}^{-1},\\ {\displaystyle {\int }_0^{\mathrm{\infty }}\left({\lambda }^2-\frac{r}{\delta }\right){\left({\lambda }^2\right)}^{r+\frac{1}{2}-1}\text{exp}\left(-\delta {\lambda }^2\right)d{\lambda }^2}={\left[\frac{{\delta }^{r+\frac{1}{2}}}{\Gamma \left(r+\frac{1}{2}\right)}\right]}^{-1}\frac{1}{2\delta },\end{array}$$

and

$${\displaystyle {\int }_0^{\mathrm{\infty }}{\left({\lambda }^2-\frac{r}{\delta }\right)}^2{\left({\lambda }^2\right)}^{r+\frac{1}{2}-1}\text{exp}\left(-\delta {\lambda }^2\right)d{\lambda }^2}={\left[\frac{{\delta }^{r+\frac{1}{2}}}{\Gamma \left(r+\frac{1}{2}\right)}\right]}^{-1}\frac{4r+3}{4{\delta }^2}.$$