Properties and Implementation of Jeffreys's Prior in Binomial Regression Models

Abstract

We study several theoretical properties of Jeffreys's prior for binomial regression models. We show that Jeffreys's prior is symmetric and unimodal for a class of binomial regression models. We characterize the tail behavior of Jeffreys's prior by comparing it with the multivariate t and normal distributions under the commonly used logistic, probit, and complementary log–log regression models. We also show that the prior and posterior normalizing constants under Jeffreys's prior are linear transformation-invariant in the covariates. We further establish an interesting theoretical connection between the Bayes information criterion and the induced dimension penalty term using Jeffreys's prior for binomial regression models with general links in variable selection problems. Moreover, we develop an importance sampling algorithm for carrying out prior and posterior computations under Jeffreys's prior. We analyze a real data set to illustrate the proposed methodology.

1. INTRODUCTION

Jeffreys's prior is perhaps the most widely used noninformative prior in Bayesian analysis. For the binomial regression model, Jeffreys's prior is attractive because it is proper under mild conditions and requires no elicitation of hyperparameters whatsoever. The only requirement is a likelihood function from which the prior is then derived using Jeffrey's rule, which is to take the prior distribution to be the determinant of the square root of the Fisher information matrix. There has been an enormous literature on Jeffrey's prior and its properties for a wide variety of applications and models, as well as its connections to various reference priors proposed in the literature. This literature is too vast to list in its entirety here, but some relevant key references include works of Jeffreys (1946, 1961), Kass (1989, 1990), Ibrahim and Laud (1991), Kass and Wasserman (1996), and Berger (2000, 2006). Two excellent books discussing Jeffreys's prior include those of Box and Tiao (1973) and Berger (1985).

Although the literature on Jeffreys's prior is vast, there is very little discussion of theoretical properties of Jeffreys's prior for models in which it is proper, particularly for logistic regression. Such properties include its potential connections to normal or t distributions, the tail behavior of Jeffreys's prior, unimodality and symmetry properties, techniques for sampling, and its properties in variable selection problems. In Section 2 we carry out a detailed investigation of these theoretical properties of Jeffreys's prior for general binomial regression models. In Section 3 we propose an efficient importance sampling algorithm for computing the prior and posterior normalizing constants for Jeffreys's prior and examine its performance. The development of this importance sampling algorithm completely alleviates the need for more complex, time-consuming, and computationally intensive algorithms, such as Gibbs sampling or other Markov chain Monte Carlo (MCMC) algorithms. In Section 4 we use Jeffreys's prior in analyzing a prostate cancer data set. We give a brief discussion of our results and future research in Section 5. We give all proofs and the computational development in Appendixes A and B of the supplementary document.

2. PROPERTIES OF JEFFREYS'S PRIOR IN BINOMIAL REGRESSION MODELS

Suppose that {(x_i, y_i, n_i), i = 1, 2, . . . , n} are independent observations, where y_i is a binomial response variable taking a value between 0 and n_i (≥1), and x_i = (x_i0, x_i1, . . . , x_ik)′ is a (k + 1) × 1 vector of (possibly random) covariates and x_i0 = 1 for the intercept term. The binomial regression model is assumed for [y_i|x_i], which has conditional density given by

$$\begin{array}{cc}\hfill p(y_i\mid {\mathbf{x}}_i,n_i,\beta )=\left(\begin{array}{c}\hfill n_i\hfill \\ \hfill y_i\hfill \end{array}\right){\left\{F\left({\mathbf{x}}_i^{\prime }\beta \right)\right\}}^{y_i}{\{1-F\left({\mathbf{x}}_i^{\prime }\beta \right)\}}^{n_i-y_i}& ,\hfill \\ \hfill i=1,2,\dots ,n& ,\hfill \end{array}$$ (1)

where β = (β₀, β₁, . . . , β_k)′ denotes a (k + 1) vector of regression coefficients, F(·) denotes a cumulative distribution function (cdf), and F⁻¹ is called the link function. The likelihood function of β is

$$L(\beta \mid X,\mathbf{y})=\prod _{i=1}^n\left(\begin{array}{c}\hfill n_i\hfill \\ \hfill y_i\hfill \end{array}\right){\left\{F\left({\mathbf{x}}_i^{\prime }\beta \right)\right\}}^{y_i}\{1-F\left({\mathbf{x}}_i^{\prime }\beta \right)\}{}^{n_i-y_i},$$ (2)

where y = (y₁, y₂, . . . , y_n)′ and X = (x₁, x₂, . . . , x_n)′ is the n × (k + 1) design matrix. Throughout the article, we assume that F(·) is twice differentiable and that f(z) = dF (z)/dz denotes the probability density function (pdf). Then Jeffreys's prior for β under the binomial regression model (1) is given by

$$\pi (\beta \mid X)\propto \mid X^{\prime }W\left(\beta \right)X{\mid }^{1∕2},$$ (3)

where |X′W(β)X| denotes the determinant of the matrix X′W(β)X, W(β) = diag(w₁(β), w₂(β), . . . , w_n(β)), and

$$w_i\left(\beta \right)=\frac{n_i{\left\{f\left({\mathbf{x}}_i^{\prime }\beta \right)\right\}}^2}{F\left({\mathbf{x}}_i^{\prime }\beta \right)\{1-F\left({\mathbf{x}}_i^{\prime }\beta \right)\}}$$ (4)

for i = 1, 2, . . . , n. The Jeffreys prior given by (3) does not have a closed form in general, in the sense that it is known only up to a prior normalizing constant and is generally improper for most generalized linear models except for binary regression models, such as logistic, probit, and complementary log–log regression models, as shown by Ibrahim and Laud (1991). In addition, it has several additional attractive properties, which we formally state as follows.

Proposition 1

For the binomial regression model (1), assume that X is of full rank. Then Jeffreys's prior (3) for β is proper, and the corresponding moment-generating function of β exists.

Proposition 1 is a special case of theorem 2.1 of Ibrahim and Laud (1991).

Proposition 2

Assume that F(z) is symmetric in the sense that F(−z) = 1 − F(z) and f(−z) = f(z). Then Jeffreys's prior π(β|X) in (3) is symmetric about 0, that is, π(−β|X) = π(β|X), ∀β ∈ R^k+1, where R^k+1 denotes (k + 1)-dimensional Euclidean space.

The proof of Proposition 2 follows directly from the fact that w_i(−β) = w_i(β) for i = 1, 2, . . . , n.

Theorem 1

Let $q\left(z\right)=\text{log}\left[\frac{{\left\{f\left(z\right)\right\}}^2}{F\left(z\right)\{1-F\left(z\right)\}}\right]=2\text{log}f\left(z\right)-\text{log}F\left(z\right)-\text{log}\{1-F\left(z\right)\}$. Assume that (a) X is of full rank; (b) q(z) has a unique mode, z_mod; and (c) q′(z) < 0 if z > z_mod, q′(z_mod) = 0, and q′(z) > 0 if z < z_mod. Then Jeffreys's prior π(β|X) in (3) is unimodal, and its unique mode is β_mod = (z_mod, 0, . . . , 0)′.

Assumptions (b) and (c) in Theorem 1 are satisfied for several binomial regression models, including logistic, probit, and complementary log–log regressions. We formally state this result in the next theorem.

Theorem 2

Assumptions (b) and (c) in Theorem 1 hold for F(z) = exp(z)/{1 + exp(z)}, F(z) = Φ(z) [the N(0, 1) cdf], and F(z) = 1 − exp{−exp(z)}, corresponding to logistic, probit, and complementary log–log regression models. Furthermore, Jeffreys's prior π(β|X) has a unique mode β_mod = 0 for the logistic and probit regression models, and β_mod = (.466, 0, . . . , 0)′ for the complementary log–log regression model.

Our next result establishes an interesting property of Jeffreys's prior under logistic, probit, and complementary log–log regressions—that the tails of Jeffreys's prior, regardless of sample size, are always lighter than that of a multivariate t distribution. Toward this end, let g(β|Σ, ν) denote the pdf of a (k + 1)-dimensional multivariate t-distribution with ν degrees of freedom, location 0, and a positive definite dispersion matrix Σ, that is,

$$\begin{array}{cc}\hfill g(\beta \mid \Sigma ,\nu )& =\frac{\Gamma \{(\nu +k+1)∕2\}}{\Gamma (\nu ∕2){\left(\nu \pi \right)}^{(k+1)∕2}}{\mid \Sigma \mid }^{-1∕2}\hfill \\ \hfill & \times {\left(1+\frac{1}{\nu }{\beta }^{\prime }{\Sigma }^{-1}\beta \right)}^{-(\nu +k+1)∕2}.\hfill \end{array}$$ (5)

Then we are led to the following theorem, which characterizes the tail behavior of Jeffrey's prior.

Theorem 3

Assume that X is of full rank. Then Jeffreys's prior π(β|X) in (3) under logistic regression, probit regression, and complementary log–log regression has lighter tails than g(β|Σ, ν) for any ν > 0, that is, ${\text{lim}}_{\Vert \beta \Vert \to \infty }\frac{\pi (\beta \mid X)}{g(\beta \mid \Sigma ,\nu )}=0$.

In the following theorem, we examine the tail behavior of Jeffreys's prior compared with a (k + 1)-dimensional multivariate normal distribution.

Theorem 4

Let ϕ_k+1 (β|Σ_N) denote the probability density function of the (k + 1)-dimensional normal distribution N_k+1(0, Σ_N), where Σ_N is a (k + 1) × (k + 1) positive definite matrix. Then the following results hold:

1. Under logistic regression, we have ${\text{lim}}_{\Vert \beta \Vert \to \infty }\pi (\beta \mid X)∕{\varphi }_{k+1}(\beta \mid {\Sigma }_N)=\infty$, which implies that Jeffreys's prior π(β|X) under logistic regression always has heavier tails than the normal distribution, regardless of n.

2. Let $X_{i_1{i}_2\cdots i_{k+1}}^{\ast }={({\mathbf{x}}_{i_1},{\mathbf{x}}_{i_2},\dots ,{\mathbf{x}}_{i_{k+1}})}^{\prime }$ be a (k + 1) × (k + 1) submatrix of X. If there exists (i₁, i₂, . . . , i_k+1) such that $X_{i_1{i}_2\cdots i_{k+1}}^{\ast }$ is of full rank and ${\Sigma }_N^{-1}-\frac{1}{2}{\left(X_{i_1{i}_2\cdots i_{k+1}}^{\ast }\right)}^{\prime }\times X_{i_1{i}_2\cdots i_{k+1}}^{\ast }>0$ (i.e., positive definite), then the normal distribution N_k+1(0, Σ_N) has lighter tails than Jeffreys's prior π(β|X) under probit regression. If ${\Sigma }_N^{-1}-\frac{1}{2}{\left(X_{i_1{i}_2\cdots i_{k+1}}^{\ast }\right)}^{\prime }{X}_{i_1{i}_2\cdots i_{k+1}}^{\ast }<0$ (i.e., negative definite) for all (k + 1) × (k + 1) full-rank submatrixes $X_{i_1{i}_2\cdots i_{k+1}}^{\ast }$ of X, then Jeffreys's prior π(β|X) under probit regression has lighter tails than the normal distribution N_k+1(0, Σ_N).

3. Let β = rd, where r ≥ 0 and d = (d₀, d₁, d₂, . . . , d_k)′ denotes a (k + 1)-dimensional vector of unit direction such that $\Vert \mathbf{d}\Vert =\sqrt{{\mathbf{d}}^{\prime }\mathbf{d}}=1$. Under complementary log–log regression, Jeffreys's prior π(β|X) has lighter tails than a N_k+1(0, Σ_N) distribution in certain directions d, including d = (1, 0, 0, . . . , 0)′, and has heavier tails than a N_k+1(0, Σ_N) distribution in some other directions d, including d = (−1, 0, 0, . . . , 0)′.

Next we characterize the conditional Jeffreys's prior distribution of β₀.

Proposition 3

For Jeffreys's prior π(β|X) given in (3) for general binomial regression, the conditional prior distribution of β₀ (the intercept), given β₁ = . . . = β_k = 0, is given by

$$\pi ({\beta }_0\mid {\beta }_1=\cdots ={\beta }_k=0,X)\propto {\left[\frac{f^2\left({\beta }_0\right)}{F\left({\beta }_0\right)\{1-F\left({\beta }_0\right)\}}\right]}^{(k+1)∕2},$$

and the conditional posterior distribution of β₀, given β₁ = . . . = β_k = 0, is given by

$$\begin{array}{cc}\hfill & \pi ({\beta }_0\mid {\beta }_1=\cdots ={\beta }_k=0,X,\mathbf{y})\hfill \\ \hfill & \propto {\left\{f\left({\beta }_0\right)\right\}}^{k+1}\left\{F\left({\beta }_0\right)\right\}{\Sigma }_{i=1}^n{y}_i-(k+1)∕2\hfill \\ \hfill & \times \{1-F\left({\beta }_0\right)\}{\Sigma }_{i=1}^n(n_i-y_i)-(k+1)∕2.\hfill \end{array}$$

The proof of Proposition 3 is straightforward. Note that the results given in Proposition 3 imply that the conditional Jeffreys's prior distribution of β₀ does not depend on the sample size n, but the conditional posterior does. This result sheds light on the asymptotic behavior of Jeffreys's prior—namely, that it does not converge to any well-known distribution as n → ∞.

Let $C_0\left(X\right)={\int }_{R^{k+1}}{\mid X^{\prime }W\left(\beta \right)X\mid }^{1∕2}d\beta$ and $C\left(X\right)={\int }_{R^{k+1}}L(\beta \mid X,\mathbf{y}){\mid X^{\prime }W\left(\beta \right)X\mid }^{1∕2}d\beta$, which correspond to the prior and posterior normalizing constants. We present an interesting result—that the prior and posterior normalizing constants based on Jeffreys's prior are invariant under a one-to-one linear transformation of the covariates. Let V denote a (k + 1) × (k + 1) matrix. We are led to the following theorem.

Theorem 5

Assume that V is of full rank (k + 1). We have

$$C_0\left(X\right)=C_0\left(XV\right)\text{and}C\left(X\right)=C\left(XV\right).$$ (6)

Note that when V is of full rank, XV denotes a one-to-one linear transformation of the covariates. By taking V = diag(1, v₁, v₂, . . . , v_k) with v₁ > 0, v₂ > 0, . . . , v_k > 0, (6) implies that C₀(X) and C(X) are scale-invariant with respect to the covariates. The scale-invariance of C₀(X) and C(X) in the covariates is a desirable property in Bayesian variable selection. Because the posterior model probability under a uniform prior on the model space is a function of prior and posterior normalizing constants, the result given in Theorem 5 also implies that the posterior model probability is scale-invariant in the covariates. In this sense Jeffreys's prior is as attractive as Zellner's g-prior (Zellner 1986), which also leads to posterior model probabilities that are scale invariant in the covariates.

Finally, we examine a theoretical connection between the Bayes information criterion (BIC) (Schwarz 1978) and the ratio of the posterior normalizing constant and the prior normalizing constant under Jeffreys's prior. Toward this end, we assume that there are only k + 1 distinct rows in X, denoted by ${\tilde{\mathbf{x}}}_j^{\prime }$, j = 1, 2, . . . , k + 1. Under this assumption, we combine the binomial counts into k + 1 aggregated counts corresponding to those ${\tilde{\mathbf{x}}}_j$'s, and the aggregated likelihood function of (2) is given by

$$\begin{array}{cc}\hfill & L(\beta \mid X_A,\mathbf{y})\hfill \\ \hfill & =\left\{\prod _{i=1}^n\left(\begin{array}{c}\hfill n_i\hfill \\ \hfill y_i\hfill \end{array}\right)\right\}\left[\prod _{j=1}^{k+1}{\left\{F\left({\tilde{\mathbf{x}}}_j^{\prime }\beta \right)\right\}}^{y_{Aj}}{\{1-F\left({\tilde{\mathbf{x}}}_j^{\prime }\beta \right)\}}^{n_{Aj}-y_{Aj}}\right],\hfill \end{array}$$ (7)

where $X_A={({\tilde{\mathbf{x}}}_1,{\tilde{\mathbf{x}}}_2,\dots ,{\tilde{\mathbf{x}}}_{k+1})}^{\prime }$, $y_{Aj}={\Sigma }_{i:{\mathbf{x}}_i={\tilde{\mathbf{x}}}_j}{y}_i$, and $n_{Aj}={\Sigma }_{i:{\mathbf{x}}_i={\tilde{\mathbf{x}}}_j}{n}_i$. Using (7), Jeffreys's prior in (3) reduces to $\pi (\beta \mid X_A)\propto {\mid X_A^{\prime }{W}_A\left(\beta \right)X_A\mid }^{1∕2}$, where W_A(β) = diag(w_A1(β), . . . , w_A,k+1(β)) and $w_{Aj}\left(\beta \right)=\frac{n_{Aj}{\left\{f\left({\tilde{\mathbf{x}}}_j^{\prime }\beta \right)\right\}}^2}{F\left({\tilde{\mathbf{x}}}_i^{\prime }\beta \right)\{1-F\left({\tilde{\mathbf{x}}}_i^{\prime }\beta \right)\}}$. It is easy to see that when X_A is of full rank and n_Aj ≥ 1, π(β|X_A) is proper. Let $p_j=p_j\left(\beta \right)=F\left({\tilde{\mathbf{x}}}_i^{\prime }\beta \right)$ for j = 1, 2, . . . , k + 1. When X_A is of full rank, p₁, p₂, . . . , p_k+1 are (k + 1) unconstrained binomial probabilities. The aggregated likelihood given in (7) thus can be written as a function of (p₁, p₂, . . . , p_k+1) given by $L(p_1,p_2,\dots ,p_{k+1}\mid X_A,\mathbf{y})=\left\{{\prod }_{i=1}^n\left(\begin{array}{c}\hfill n_i\hfill \\ \hfill y_i\hfill \end{array}\right)\right\}\left[{\prod }_{j=1}^{k+1}{p}_j^{y_{Aj}}{(1-p_j)}^{n_{Aj}-y_{Aj}}\right]$. In terms of the binomial probabilities, Jeffreys's prior and the corresponding posterior distribution have closed-form expressions, Specifically, Jeffreys's prior and the corresponding posterior distribution of p₁, p₂, . . . , p_k+1 are given by

$$\begin{array}{cc}\hfill & \pi (p_1,p_2,\dots ,p_{k+1}\mid X_A)\hfill \\ \hfill & ={\left\{B\left(\frac{1}{2},\frac{1}{2}\right)\right\}}^{-(k+1)}\prod _{j=1}^{k+1}{\left\{p_j(1-p_j)\right\}}^{-1∕2}\hfill \end{array}$$ (8)

and

$$\begin{array}{cc}\hfill & \pi (p_1,p_2,\dots ,p_{k+1}\mid X_A,\mathbf{y})\hfill \\ \hfill & =\prod _{j=1}^{k+1}\frac{p_j^{y_{Aj}-1∕2}{(1-p_j)}^{n_{Aj}-y_{Aj}-1∕2}}{B(\frac{1}{2}+y_{Aj},\frac{1}{2}+n_{Aj}-y_{Aj})}.\hfill \end{array}$$ (9)

It is easy to show that

$$\begin{array}{cc}\hfill & \pi (\beta \mid X_A)\hfill \\ \hfill & =\pi (p_1,p_2,\dots ,p_{k+1}\mid X_A)\mid \frac{(\partial p_1,p_2,\dots ,\partial p_{k+1})}{\partial \beta }\mid ,\hfill \end{array}$$ (10)

where $\frac{(\partial p_1,\partial p_2,\dots ,\partial p_{k+1})}{\partial \beta }=\text{diag}(f\left({\tilde{\mathbf{x}}}_1^{\prime }\beta \right),\dots ,f\left({\tilde{\mathbf{x}}}_{k+1}^{\prime }\beta \right))X_A$. Using (8) and (10), we obtain a closed-form expression of the normalizing constant for Jeffreys's prior, given by

$$\begin{array}{cc}\hfill C_0\left(X\right)& ={\int }_{R^{k+1}}{\mid X_A^{\prime }{W}_A\left(\beta \right)X_A\mid }^{1∕2}d\beta \hfill \\ \hfill & ={\left(\prod _{j=1}^{k+1}{n}_{Aj}\right)}^{1∕2}{\left\{B\left(\frac{1}{2},\frac{1}{2}\right)\right\}}^{k+1}\hfill \\ \hfill & ={\pi }^{k+1}{\left(\prod _{j=1}^{k+1}{n}_{Aj}\right)}^{1∕2},\hfill \end{array}$$ (11)

where $B(a,b)=\frac{\Gamma \left(a\right)\Gamma \left(b\right)}{\Gamma (a+b)}$ denotes the beta function. Similarly, using (9) and (10), we obtain the posterior normalizing constant based on Jeffreys's prior as

$$\begin{array}{cc}\hfill C\left(X\right)& ={\int }_{R^{k+1}}L(\beta \mid X_A,\mathbf{y})\mid X_A^{\prime }{W}_A\left(\beta \right)X_A{\mid }^{1∕2}d\beta \hfill \\ \hfill & =\left\{\prod _{i=1}^n\left(\begin{array}{c}\hfill n_i\hfill \\ \hfill y_i\hfill \end{array}\right)\right\}{\left(\prod _{j=1}^{k+1}{n}_{Aj}\right)}^{1∕2}\hfill \\ \hfill & \times \left\{\prod _{j=1}^{k+1}B\left(\frac{1}{2}+y_{Aj},\frac{1}{2}+n_{Aj}-y_{Aj}\right)\right\}.\hfill \end{array}$$ (12)

Let $N={\Sigma }_{i=1}^n{n}_i,{\widehat{\alpha }}_j=\frac{n_{Aj}}{N}$ and ${\widehat{\mu }}_j=\frac{y_{Aj}}{N}$ for j = 1, 2, . . . , k + 1. Also, let $\widehat{\beta }$ denote the maximum likelihood estimate of β. Then, using (7), the BIC is given by

$$\begin{array}{cc}\hfill \mathrm{BIC}& =-2\log L(\widehat{\beta }\mid X_{A,\mathbf{y}})+(k+1)\log \left(N\right)\hfill \\ \hfill & =-2\log \left\{\prod _{i=1}^n\left(\begin{array}{c}\hfill n_i\hfill \\ \hfill y_i\hfill \end{array}\right)\right\}-2\sum _{j=1}^{k+1}\{y_{Aj}\log \left(\frac{y_{Aj}}{n_{Aj}}\right)\phantom{\}}\hfill \\ \hfill & \phantom{\{}+(n_{Aj}-y_{Aj})\log \left(\frac{n_{Aj}-y_{Aj}}{n_{Aj}}\right)\}\hfill \\ \hfill & +(k+1)\log \left(N\right).\hfill \end{array}$$ (13)

The following theorem characterizes the connection between the BIC and the normalizing constants.

Theorem 6

Assume that (a) ${\text{lim}}_{N\to \infty }{\widehat{\alpha }}_j={\alpha }_j$ and ${\text{lim}}_{N\to \infty }{\widehat{\mu }}_j={\mu }_j$ exist and (b) 0 < α_j < 1 and 0 < μ_j < α_j for j = 1, 2, . . . , k + 1. Then for large N, we have

$$\begin{array}{cc}\hfill & -2\{\log C\left(X\right)-\log C_0\left(X\right)\}\hfill \\ \hfill & =\mathrm{BIC}+\sum _{j=1}^{k+1}\log \left(\frac{\pi }{2}{\widehat{\alpha }}_j\right)+o\left(\frac{1}{N}\right),\hfill \end{array}$$ (14)

where C₀(X), C(X), and BIC are defined by (11), (12), and (13).

The proof of Theorem 6 follows directly from Stirling's formula, and we omit the details for brevity. The theoretical connection given in (14) is quite interesting. First, −2{logC(X) − logC₀(X)} acts quite similarly to the BIC. Second, in addition to the dimensional penalty of (k + 1)log N in the BIC, the dimensional penalty term in −2{logC(X) − logC₀(X)} also depends on (α₁, α₂, . . . , α_k+1), which describes the “joint distribution” of the covariate x_i, which takes on the k + 1 distinct values ${\tilde{\mathbf{x}}}_j$, j = 1, . . . , k + 1. Thus Bayesian variable selection based on Jeffreys's prior can yield very different results than the BIC, especially when N is small.

3. COMPUTATIONAL IMPLEMENTATION OF JEFFREYS'S PRIOR

As shown in Theorem 3, under logistic regression, probit regression, and complementary log–log regression, Jeffreys's prior π(β|X) in (3) has lighter tails than g(β|Σ, ν) in (5) for any ν > 0. In addition, the likelihood L(β|X, y) is bounded above, so it follows that the posterior has lighter tails than the t-distribution. Because of Theorems 3 and 4, we can completely avoid Gibbs sampling or other computationally intensive MCMC schemes for carrying out posterior computations. Toward this goal, we propose an importance sampling approach (see, e.g., Geweke 1989) to compute the prior and posterior quantities of interest. To obtain appropriate importance sampling densities for Jeffreys's prior and the corresponding posterior distribution, we consider a more general form of the multivariate t-distribution with density

$$\begin{array}{cc}\hfill & g(\beta \mid \mu ,\Sigma ,\nu )\hfill \\ \hfill & =\frac{\Gamma \{(\nu +k+1)∕2\}}{\Gamma (\nu ∕2){\left(\nu \pi \right)}^{(k+1)∕2}}{\mid \Sigma \mid }^{-1∕2}\hfill \\ \hfill & \times {\left(1+\frac{1}{\nu }{(\beta -\mu )}^{\prime }{\Sigma }^{-1}(\beta -\mu )\right)}^{-(\nu +k+1)∕2}.\hfill \end{array}$$ (15)

For computing the prior normalizing constant, we specify μ = β_mod in (15), because Jeffreys's prior has a unique mode at β_mod as shown in Theorem 1. To determine Σ in (15), we match the Hessians (curvatures) of Jeffreys's prior and the t-distribution at β_mod as

$$\begin{array}{cc}\hfill & {\kappa }_0{\phantom{\mid }\frac{{\partial }^2\log \pi (\beta \mid X)}{\partial \beta \partial {\beta }^{\prime }}\mid }_{\beta ={\beta }_{mod}}\hfill \\ \hfill & ={\phantom{\mid }\frac{{\partial }^2\log g(\beta ∕\mu ={\beta }_{mod},\Sigma ,\nu )}{\partial \beta \partial {\beta }^{\prime }}\mid }_{\beta ={\beta }_{mod}},\hfill \end{array}$$ (16)

where π(β|X) and g(β|μ, Σ, ν) are given by (3) and (15), and κ₀ > 0 is a fixed dispersion adjustment parameter. We follow a similar strategy for computing the posterior normalizing constant. We match the mode and the Hessian (curvature) at the mode of the posterior distribution μ and Σ in the importance sampling density (15). In doing so, we are led to $\mu =\widehat{\mu }={\text{arg max}}_{\beta \in R^{k+1}}\left\{\text{log}\left[L(\beta \mid X,\mathbf{y})\pi (\beta \mid X)\right]\right\}$ and

$${\Sigma }^{-1}=-{\kappa }_1\frac{\nu }{\nu +k+1}{\phantom{\mid }\frac{{\partial }^2\log L(\beta \mid X,\mathbf{y})\pi (\beta \mid X)}{\partial \beta \partial {\beta }^{\prime }}\mid }_{\beta =\widehat{\mu }},$$ (17)

where κ₁ > 0 is a fixed dispersion adjustment parameter. A detailed discussion of the specification of ν and κ₀ (or κ₁) in determining the importance sampling density g(β|μ, Σ, ν) is given in Appendix B of the supplementary document.

The steps of the importance sampling method are quite simple. Let $\mathcal{G}(\frac{\nu }{2},\frac{\nu }{2})$ denote the gamma distribution with shape parameter ν/2 and scale parameter ν/2. Then the importance sampling algorithm for computing the prior normalizing constant C₀ = C₀(X) can be stated as follows:

• Step 1: Generate a random sample {β₁, β₂, . . . , β_Q of size Q from g(β|μ = β_mod, Σ, ν), where for each q, independently generate ${\lambda }_q~\mathcal{G}(\frac{\nu }{2},\frac{\nu }{2})$ and β_q ∼ N_k+1(β_mod, Σ/λ_q).

• Step 2: Compute the Monte Carlo estimate of the prior normalizing constant C₀ as ${\widehat{C}}_0=\frac{1}{Q}{\Sigma }_{q=1}^Q\frac{{\mid X^{\prime }W\left({\beta }_q\right)X\mid }^{1∕2}}{g({\beta }_q\mid \mu ={\beta }_{mod},\Sigma ,\nu )}$.

In step 2 we also may calculate $\text{log}\left({\widehat{C}}_0\right)$ instead of ${\widehat{C}}_0$ for greater stability and accuracy. In addition, we compute the relative Monte Carlo (MC) standard error (SE) as $\mathrm{RSE}\left({\widehat{C}}_0\right)=\frac{1}{{\widehat{C}}_0}\{\frac{1}{Q(Q-1)}{\Sigma }_{q=1}^Q{[\frac{{\mid X^{\prime }W\left({\beta }_q\right)X\mid }^{1∕2}}{g({\beta }_q\mid \mu ={\beta }_{mod},\Sigma ,\nu )}-{\widehat{C}}_0]}^2{\}}^{1∕2}$. Note that by the standard delta method, we can show that $\mathrm{RSE}\left({\widehat{C}}_0\right)$ is indeed the MC SE of $\text{log}\left({\widehat{C}}_0\right)$. In practice, we recommend choosing Q and κ₀ so that $\mathrm{RSE}\left({\widehat{C}}_0\right)$ is approximately .01 or less. The importance sampling algorithm for computing the posterior normalizing constant C = C(X) is analogous to that for the prior normalizing constant.

To examine the performance of the proposed importance sampling algorithm, we consider the logistic regression model with a single binary covariate. We generate a simulated data set of size n = 100 with x_i1 ∼ Bernoulli(.6) and y_i|x_i1 ∼ Bernoulli(p_i), where $p_i=\frac{\text{exp}\left({\mathbf{x}}_i^{\prime }\beta \right)}{1+\text{exp}\left({\mathbf{x}}_i^{\prime }\beta \right)}$, x_i = (1, x_i1)′, and β = (.5, 1.0)′. We implemented the proposed importance sampling algorithm with various values of κ₀ and κ₁. The results are given in Table 1. The prior and posterior normalizing constants in this example involve a two-dimensional integral. As discussed in Appendix B, β_mod = 0 and ν = 3.37 is a guide value for ν in g(β|μ = 0, Σ, ν) for computing the prior normalizing constant under logistic regression. From Table 1, we see that both ν = 3.37 and ν = 5 work well for computing both the prior and posterior normalizing constants. The MC SEs are all smaller than .01 with Q = 10,000. Moreover, we see that $\text{log}\left({\widehat{C}}_0\right)$, $\text{log}\left({\widehat{C}}_0\right)$, and the MC SE are extremely consistent and robust for several values of ν, κ₀, and κ₁. Moreover, the MC SE for ν = 3.37 is smallest among all values of ν, confirming our theoretical results concerning the choice of ν.

Table 1.

Monte Carlo estimates of log C₀ and log C with MC size Q = 10,000

	Jeffreys's prior			Posterior
v	k0	log Ĉ0	MC SE	k1	log Ĉ	MC SE
1	1	6.137	.008	2	−56.907	.007
3.37		6.131	.002		−56.900	.003
5		6.133	.002		−56.895	.003
20		6.139	.012		−56.881	.007
3.37	.5	6.144	.005	1	−56.884	.006
	2	6.127	.005	3	−56.906	.004
5	.5	6.129	.004	1	−56.898	.004
	2	6.139	.008	3	−56.890	.005
True value	log C0 = 6.132			log C = −56.890

Next we discuss how to compute the posterior estimates of β under Jeffreys's prior via the proposed importance sampling method. Let {β₁, β₂, . . . , β_Q}, where β_q = (β_q0, β_q1, . . . , β_qk)′, q = 1, 2, . . . , Q, be a random sample of size Q from g(β|μ, Σ, ν). Then an MC estimate of the hth posterior moment of β_j is given by $\widehat{E}({\beta }_j^h\mid X,y)=\frac{{\Sigma }_{q=1}^Q{\beta }_{qj}^h{\omega }_q}{{\Sigma }_{q=1}^Q{\omega }_q}$, where ${\omega }_q=\frac{{\pi }^{\ast }({\beta }_q\mid X,y)}{g({\beta }_q\mid \mu ,\Sigma ,\nu )}$ for q = 1, 2, . . . , Q. Using $\widehat{E}({\beta }_j^h\mid X,y)$, we can easily compute the posterior mean and standard deviation of β_j. To compute the highest posterior density (HPD) interval of β_j through the importance sampling method, we use the MC method proposed by Chen and Shao (1999). Specifically, for 0 ≤ γ < 1, define ${\widehat{\beta }}_j^{\left(\gamma \right)}={\beta }_{j\left(1\right)}$ if γ = 0 and β_j(q) if ${\Sigma }_l^{q-1}{\omega }_l<\gamma \le {\Sigma }_{l=1}^q{\omega }_l$, where β_j(q) is the qth smallest of {β_j(l), l = 1, 2, . . . , Q}. To obtain a 100(1 − α)% HPD interval for β_j, we let $R_q\left(Q\right)=({\widehat{\beta }}_j^{(q∕Q)},{\widehat{\beta }}_j^{((q+\left[(1-\alpha )Q\right])∕Q)})$ for q = 1, 2, . . . , Q − [(1 − α)Q], where [(1 − α)Q] denotes the integer part of (1 − α)Q. Then the 100(1 − α)% HPD interval is R_q*(Q), the interval with the smallest width among all R_q(Q)'s.

4. ANALYSIS OF PROSTATE CANCER DATA

To further motivate the proposed methodology, we consider data from a retrospective cohort study of men treated with radical prostatectomy (n = 968) between 1988 and 2000, which is a subset of the data published by D'Amico et al. (2002). The primary endpoint that D'Amico et al. (2002) considered was prostate-specific antigen (PSA) recurrence free survival. In our analysis, we consider Pathological Extracapsular Extension (PECE) a binary response variable (y) that takes values 0 and 1, where 1 denotes that the cancer has penetrated the prostate wall and 0 indicates otherwise. We consider five prognostic factors: age, natural logarithm of PSA (LogPSA), percent positive prostate biopsies (ppb), biopsy Gleason score, and the 1992 American Joint Commission on Cancer (AJCC) clinical tumor category. The covariates age, LogPSA, and ppb are continuous. We dichotomize biopsy Gleason score as GS7 and GS8H, where (GS7, GS8H) takes values (0, 0), (1, 0), and (0, 1), corresponding to biopsy Gleason scores of ≤6, 7, and 8−10. Similarly, we dichotomize the clinical tumor category as T2b and T2c, where (T2b, T2c) takes values (0, 0), (1, 0), and (0, 1), corresponding to the clinical tumor categories T1c or T2a, T2b, and T2c.

First, we carried out variable selection. For all of 32 possible models, including an intercept with five covariates, we computed the Akaike information criterion (AIC), the BIC, and the posterior model probabilities based on Jeffreys's prior. To compute the posterior model probabilities based on Jeffreys's prior, we implemented the importance sampling algorithm proposed in Section 3 with an MC sample size of Q = 20,000, κ₀ = 1, κ₁ = 2, and ν = 3.37 and ν = 5 for the prior and posterior normalizing constants. The best model under the BIC and the model with the highest posterior probability based on Jeffreys's prior is (LogPSA, ppb, GS7, GS8H). For this best model, the BIC value is 925.627, and the posterior probability is .871. The second-best model under these two criteria is (age, LogPSA, ppb, GS7, GS8H), with a BIC value of 928.574 and posterior probability of .119. The AIC selects the full model (age, LogPSA, ppb, GS7, GS8H, T2b, T2c) as the best model. The AIC values are 899.168 for the full model and 901.251 for the best BIC model. We note that the AIC gives different results than the BIC and the highest posterior probability model, apparently with no good scientific reason.

Next, we computed the posterior estimates under Jeffreys's prior for the highest posterior probability model, (LogPSA, ppb, GS7, GS8H), using the MC method with a MC sample of size Q = 20,000, as discussed in Section 3. Table 2 gives the posterior means (Estimates), the posterior standard deviations (SDs), and 95% HPD intervals for the β_j's, along with the corresponding maximum likelihood (ML) estimates, standard errors (SEs), and p values. We can see from Table 2 that the posterior estimates based on Jeffreys's prior are very close to the ML estimates, which is intuitively appealing, thus demonstrating that Jeffreys's prior is indeed noninformative. We also can see from this table that all selected variables are highly significant, which implies that these are the most important prognostic factors in predicting the binary response PECE. We note that under the full model, age, T2b, and T2c all have 95% HPD intervals that contain 0.

Table 2.

Posterior estimates of β for the prostate cancer data

	ML estimates			Posterior estimates
Variable	Estimate	SE	p value	Estimate	SD	95% HPD interval
Intercept	−3.895	.304	<.0001	−3.896	.307	(−4.586, −3.222)
LogPSA	.696	.135	<.0001	.696	.135	(.400, 1.004)
ppb	2.376	.355	<.0001	2.376	.356	(1.612, 3.201)
GS7	.705	.182	.0001	.706	.182	(.283, 1.098)
GS8H	1.420	.337	<.0001	1.420	.337	(.639, 2.156)

5. DISCUSSION

We have examined theoretical and computational properties of Jeffreys's prior for inference in binomial regression models. We have derived several theoretical properties of Jefferey's prior for binomial regression models, including unimodality, symmetry, mode, and tail behavior. For variable selection problems, we have established a theoretical connection between the BIC and the ratio of the posterior and prior normalizing constants with Jeffreys's prior under binomial regression with a general link. As shown in Theorem 6, the dimension penalty under Jeffreys's prior depends on the covariates as well as on the penalty term imposed by the BIC. The penalty term for the BIC depends only on k and N, and thus the BIC does not have a penalty as sophisticated as that of Jeffreys's prior for variable selection problems. This is an interesting theoretical difference that may have an impact in variable selection problems. Future work includes investigating the performance of Jeffreys's prior in variable selection and carrying out computations for variable selection with Jeffreys's prior for high-dimensional problems.