Objective Bayesian Inference in Probit Models with Intrinsic Priors Using Variational Approximations

Abstract

There is not much literature on objective Bayesian analysis for binary classification problems, especially for intrinsic prior related methods. On the other hand, variational inference methods have been employed to solve classification problems using probit regression and logistic regression with normal priors. In this article, we propose to apply the variational approximation on probit regression models with intrinsic prior. We review the mean-field variational method and the procedure of developing intrinsic prior for the probit regression model. We then present our work on implementing the variational Bayesian probit regression model using intrinsic prior. Publicly available data from the world’s largest peer-to-peer lending platform, LendingClub, will be used to illustrate how model output uncertainties are addressed through the framework we proposed. With LendingClub data, the target variable is the final status of a loan, either charged-off or fully paid. Investors may very well be interested in how predictive features like FICO, amount financed, income, etc. may affect the final loan status.

1. Introduction

There is not much literature on objective Bayesian analysis for binary classification problems, especially for intrinsic prior related methods. By far, only two articles have explored intrinsic prior related methods on classification problems. Reference [1] implements integral priors into the generalized linear models with various link functions. In addition, reference [2] considers intrinsic priors for probit models. On the other hand, variational inference methods have been employed to solve classification problem with logistic regression ([3]) and probit regression ([4,5]) with normal priors. Variational approximation methods have been reviewed in [6,7], and more recently [8].

In this article, we propose to apply variational approximations on probit regression models with intrinsic priors. In Section 4, we review the mean-field variational method that will be used in this article. In Section 3, procedures for developing intrinsic priors for probit models will be introduced following [2]. Our work is presented in Section 5. Our motivations for combining intrinsic prior methodology and variational inference is as following

• Avoiding manually set ad hoc plugin priors by automatically generating a family of non-informative priors that are less sensible.

• Reference [1,2] do not consider inference of posterior distributions of parameters. Their focus is on model comparison. Although the development of intrinsic priors itself comes from a model selection background, we thought it would be interesting to apply intrinsic priors on inference problems. In fact, some recently developed priors that proposed to solve inference or estimation problems turned out to be also intrinsic priors. For example, the Scaled Beta2 prior [9] and the Matrix-F prior [10].

• Intrinsic priors concentrate probability near the null hypothesis, a condition that is widely accepted and should be required of a prior for testing a hypothesis.

• Also, intrinsic priors have flat tails that prevents finite sample inconsistency [11].

• For inference problems with large data set, variational approximation methods are much faster than MCMC-based methods.

As for model comparison, due to the fact that the output of variational inference methods cannot be employed directly to compare models, we propose in Section 5.3 to simply make use of the variational approximation of the posterior distribution as an importance function and get the Monte Carlo estimated marginal likelihood by importance sampling for model comparison.

2. Background and Development of Intrinsic Prior Methodology

2.1. Bayes Factor

The Bayesian framework of model selection coherently involves the use of probability to express all uncertainty in the choice of model, including uncertainty about the unknown parameters of a model. Suppose that models $M_1,M_2,...,M_q$ are under consideration. We shall assume that the observed data $\mathbf{x}=(x_1,x_2,...,x_n)$ is generated from one of these models but we do not know which one it is. We express our uncertainty through prior probability $P(M_j),j=1,2,...,q$. Under model $M_i$, $\mathbf{x}$ has density $f_i(\mathbf{x}|{\mathit{\theta }}_i,M_i)$, where ${\mathit{\theta }}_i$ are unknown model parameters, and the prior distribution for ${\mathit{\theta }}_i$ is ${\pi }_i({\mathit{\theta }}_i|M_i)$. Given observed data and prior probabilities, we can then evaluate the posterior probability of $M_i$ using Bayes’ rule

$$\begin{array}{c}\hfill P(M_i|\mathbf{x})=\frac{p_i(\mathbf{x}|M_i)P(M_i)}{{\sum }_{j=1}^q{p}_j(\mathbf{x}|M_j)P(M_j)},\end{array}$$ (1)

where

$$\begin{array}{c}\hfill p_i(\mathbf{x}|M_i)=\int f_i(\mathbf{x}|{\mathit{\theta }}_i,M_i){\pi }_i({\mathit{\theta }}_i|M_i)d{\mathit{\theta }}_i\end{array}$$ (2)

is the marginal likelihood of $\mathbf{x}$ under $M_i$, also called the evidence for $M_i$ [12]. A common choice of prior model probabilities is $P(M_j)=\frac{1}{q}$, so that each model has the same initial probability. However, there are other alternatives of assigning probabilities to correct for multiple comparison (See [13]). From (1), the posterior odds are therefore the prior odds multiplied by the Bayes factor

$$\begin{array}{c}\hfill \frac{P(M_j|\mathbf{x})}{P(M_i|\mathbf{x})}=\frac{P(M_j)p_j(\mathbf{x})}{P(M_i)p_i(\mathbf{x})}=\frac{P(M_j)}{P(M_i)}\times B_{ji}.\end{array}$$ (3)

where the Bayes factor of $M_j$ to $M_i$ is defined by

$$\begin{array}{c}\hfill B_{ji}=\frac{p_j(\mathbf{x})}{p_i(\mathbf{x})}=\frac{\int f_j(\mathbf{x}|{\mathit{\theta }}_j){\pi }_j({\mathit{\theta }}_j)d{\mathit{\theta }}_j}{\int f_i(\mathbf{x}|{\mathit{\theta }}_i){\pi }_i({\mathit{\theta }}_i)d{\mathit{\theta }}_i}.\end{array}$$ (4)

Here we omit the dependence on models $M_j,M_i$ to keep the notation simple. The marginal likelihood, $p_i(\mathbf{x})$ expresses the preference shown by the observed data for different models. When $B_{ji}>1$, the data favor $M_j$ over $M_i$, and when $B_{ji}<1$ the data favor $M_i$ over $M_j$. A scale for interpretation of $B_{ji}$ is given by [14].

2.2. Motivation and Development of Intrinsic Prior

Computing $B_{ji}$ requires specification of ${\pi }_i({\mathit{\theta }}_i)$ and ${\pi }_j({\mathit{\theta }}_j)$. Often in Bayesian analysis, when prior information is weak, one can use non-informative (or default) priors ${\pi }_i^N({\mathit{\theta }}_i)$. Common choices for non-informative priors are the uniform prior, ${\pi }_i^U({\mathit{\theta }}_i)\propto 1$; the Jeffreys prior, ${\pi }_i^J({\mathit{\theta }}_i)\propto {\left[det({\mathbf{I}}_i({\mathit{\theta }}_i))\right]}^{1/2}$ where ${\mathbf{I}}_i({\mathit{\theta }}_i)$ is the expected Fisher information matrix corresponding to $M_i$.

Using any of the ${\pi }_i^N$ in (4) would yield

$$\begin{array}{c}\hfill B_{ji}^N=\frac{p_j^N(\mathbf{x})}{p_i^N(\mathbf{x})}=\frac{\int f_j(\mathbf{x}|{\mathit{\theta }}_j){\pi }_j^N({\mathit{\theta }}_j)d{\mathit{\theta }}_j}{\int f_i(\mathbf{x}|{\mathit{\theta }}_i){\pi }_i^N({\mathit{\theta }}_i)d{\mathit{\theta }}_i}.\end{array}$$ (5)

The difficulty with (5) is that ${\pi }_i^N$ are typically improper and hence are defined only up to an unspecified constant $c_i$. So $B_{ji}^N$ is defined only up to the ratio $c_j/c_i$ of two unspecified constants.

An attempt to circumvent the ill definition of the Bayes factors for improper non-informative priors is the intrinsic Bayes factor introduced by [15], which is a modification of a partial Bayes factor [16]. To define the intrinsic Bayes factor we consider the set of subsamples $\mathbf{x}(l)$ of the data $\mathbf{x}$ of minimal size l such that $0<p_i^N(\mathbf{x}(l))<\infty$. These subsamples are called training samples (not to be confused with training sample in machine learning). In addition, there is a total number of L such subsamples.

The main idea here is that training sample $\mathbf{x}(l)$ will be used to convert the improper ${\pi }_i^N({\mathit{\theta }}_i)$ to proper posterior

$$\begin{array}{c}\hfill {\pi }_i^N({\mathit{\theta }}_i|\mathbf{x}(l))=\frac{f_i(\mathbf{x}(l)|{\mathit{\theta }}_i){\pi }_i^N({\mathit{\theta }}_i)}{p_i^N(\mathbf{x}(l))}\end{array}$$ (6)

where $p_i^N(\mathbf{x}(l))=\int f_i(\mathbf{x}(l)|{\mathit{\theta }}_i){\pi }_i^N({\mathit{\theta }}_i)d{\mathit{\theta }}_i$. Then, the Bayes factor for the remaining of the data $\mathbf{x}(n-l)$, where $\mathbf{x}(l)\cup \mathbf{x}(n-l)=\mathbf{x}$, using ${\pi }_i^N({\mathit{\theta }}_i|\mathbf{x}(l))$ as prior is called a “partial” Bayes factor,

$$\begin{array}{c}\hfill B_{ji}^N(\mathbf{x}(n-l)|\mathbf{x}(l))=\frac{\int f_j(\mathbf{x}(n-l)|{\mathit{\theta }}_j){\pi }_j^N({\mathit{\theta }}_j|\mathbf{x}(l))d{\mathit{\theta }}_j}{\int f_i(\mathbf{x}(n-l)|{\mathit{\theta }}_i){\pi }_i^N({\mathit{\theta }}_i|\mathbf{x}(l))d{\mathit{\theta }}_i}\end{array}$$ (7)

This partial Bayes factor is a well-defined Bayes factor, and can be written as $B_{ji}^N(\mathbf{x}(n-l)|\mathbf{x}(l))=B_{ji}^N(\mathbf{x})B_{ij}(\mathbf{x}(l))$, where $B_{ji}^N(\mathbf{x})=\frac{p_j^N(\mathbf{x})}{p_i^N(\mathbf{x})}$ and $B_{ij}(\mathbf{x}(l))=\frac{p_i^N(\mathbf{x}(l))}{p_j^N(\mathbf{x}(l))}$. Clearly, $B_{ji}^N(\mathbf{x}(n-l)|\mathbf{x}(l))$ will depend on the choice of the training samples $\mathbf{x}(l)$. To eliminate this arbitrariness and increase stability, reference [15] suggests averaging over all training samples and obtained the arithmetic intrinsic Bayes factor (AIBF)

$$\begin{array}{c}\hfill B_{ji}^{AIBF}(\mathbf{x})=B_{ji}^N(\mathbf{x})\frac{1}{L}\sum _{l=1}^L{B}_{ij}^N(\mathbf{x}(l)).\end{array}$$ (8)

The strongest justification of the arithmetic IBF is its asymptotic equivalence with a proper Bayes factor arising from Intrinsic priors. These intrinsic priors were identified through an asymptotic analysis (see [15]). For the case where $M_i$ is nested in $M_j$, it can be shown that the intrinsic priors are given by

$$\begin{array}{c}\hfill {\pi }_i^I({\mathit{\theta }}_i)={\pi }_i^N({\mathit{\theta }}_i)\mathrm{and}{\pi }_j^I({\mathit{\theta }}_j)={\pi }_j^N({\mathit{\theta }}_j)E_{M_j}\left[\frac{m_i^N(\mathbf{x}(l))}{m_j^N(\mathbf{x}(l))}|{\mathit{\theta }}_j\right].\end{array}$$ (9)

3. Objective Bayesian Probit Regression Models

3.1. Bayesian Probit Model and the Use of Auxiliary Variables

Consider a sample $\mathbf{y}=(y_1,...,y_n)$, where $Y_i,i=1,...,n$, is a $0-1$ random variable such that under model $M_j$, it follows a probit regression model with a $j+1$-dimensional vector of covariates $x_i$, where $j\le p$. Here, p is the total number of covariate variables under our consideration. In addition, this probit model $M_j$ has the form

$$Y_i|{\beta }_0,...,{\beta }_j,M_j\sim \mathrm{Bernoulli}(\Phi ({\beta }_0{x}_{0i}+{\beta }_1{x}_{1i}+...+{\beta }_j{x}_{ji})),1\le i\le n,$$ (10)

where $\Phi$ denotes the standard normal cumulative distribution function and ${\mathit{\beta }}_j=({\beta }_0,...,{\beta }_j)$ is a vector of dimension $j+1$. The first component of the vector $x_i$ is set equal to 1 so that when considering models of the form (10), the intercept is in any submodel. The maximum length of the vector of covariates is $p+1$. Let $\pi (\mathit{\beta })$, proper or improper, summarize our prior information about $\mathit{\beta }$. Then the posterior density of $\mathit{\beta }$ is given by

$$\begin{array}{c}\hfill \pi (\mathit{\beta }|y)=\frac{\pi (\mathit{\beta }){\prod }_{i=1}^n\Phi {(x_i^{\prime }\mathit{\beta })}^{y_i}(1-\Phi {(x_i^{\prime }\mathit{\beta })}^{1-y_i})}{\int \pi (\mathit{\beta }){\prod }_{i=1}^n\Phi {(x_i^{\prime }\mathit{\beta })}^{y_i}(1-\Phi {(x_i^{\prime }\mathit{\beta })}^{1-y_i})d\mathit{\beta }},\end{array}$$

which is largely intractable.

As shown by [17], the Bayesian probit regression model becomes tractable when a particular set of auxiliary variables is introduced. Based on the data augmentation approach [18], introducing n latent variables $Z_1,...,Z_n$, where

$$\begin{array}{c}\hfill Z_i|\mathit{\beta }\sim N(x_i^{\prime }\mathit{\beta },1).\end{array}$$

The probit model (10) can be thought of as a regression model with incomplete sampling information by considering that only the sign of $z_i$ is observed. More specifically, define $Y_i=1$ if $Z_i>0$ and $Y_i=0$ otherwise. This allows us to write the probability density of $y_i$ given $z_i$

$$\begin{array}{c}\hfill p(y_i|z_i)=\mathbb{I}(z_i>0)\mathbb{I}(y_i=1)+\mathbb{I}(z_i\le 0)\mathbb{I}(y_i=0).\end{array}$$

Expansion of the parameter set from $\{\mathit{\beta }\}$ to $\{\mathit{\beta },\mathbf{Z}\}$ is the key to achieving a tractable solution for variational approximation.

3.2. Development of Intrinsic Prior for Probit Models

For the sample $\mathbf{z}={(z_1,...,z_n)}^{\prime }$, the null normal model is

$$M_{\mathbf{1}}:\{N_n(\mathbf{z}|\alpha {\mathbf{1}}_n,{\mathbf{I}}_n),\pi (\alpha )\}.$$

For a generic model $M_j$ with $j+1$ regressors, the alternative model is

$$M_j:\{N_n(\mathbf{z}|{\mathbf{X}}_j{\mathit{\beta }}_j,{\mathbf{I}}_n),\pi ({\mathit{\beta }}_j)\},$$

where the design matrix ${\mathbf{X}}_j$ has dimensions $n\times (j+1)$. Intrinsic prior methodology for the linear model was first developed by [19], and was further developed in [20] by using the methods of [21]. This intrinsic methodology gives us an automatic specification of the priors $\pi (\alpha )$ and $\pi (\mathit{\beta })$, starting with the non-informative priors ${\pi }^N(\alpha )$ and ${\pi }^N(\mathit{\beta })$ for $\alpha$ and $\mathit{\beta }$, which are both improper and proportional to 1.

The marginal distributions for the sample $\mathbf{z}$ under the null model, and under the alternative model with intrinsic prior, are formally written as

$$\begin{array}{cc}\hfill & p_1(\mathbf{z})=\int N_n(\mathbf{z}|\alpha {\mathbf{1}}_n,{\mathbf{I}}_n){\pi }^N(\alpha )d\alpha ,\hfill \\ \hfill & p_j(\mathbf{z})=\int \int N_n(\mathbf{z}|{\mathbf{X}}_j{\mathit{\beta }}_j,{\mathbf{I}}_n){\pi }^I(\mathit{\beta }|\alpha ){\pi }^N(\alpha )d\alpha d\mathit{\beta }.\hfill \end{array}$$ (11)

However, these are marginals of the sample $\mathbf{z}$, but our selection procedure requires us to compute the Bayes factor of model $M_j$ versus the reference model $M_{\mathbf{1}}$ for the sample $\mathbf{y}=(y_1,...,y_n)$. To solve this problem, reference [2] proposed to transform the marginal $p_j(\mathbf{z})$ into the marginal $p_j(\mathbf{y})$ by using the probit transformations $y_i=1(z_i>0),i=1,...,n$. These latter marginals are given by

$$\begin{array}{c}\hfill p_j(\mathbf{y})={\int }_{A_1\times ...\times A_n}{p}_j(\mathbf{z})d\mathbf{z}\end{array}$$ (12)

where

$$\begin{array}{c}\hfill A_i=\left\{\begin{array}{c}(0,\infty )\mathrm{if}{y}_i=1,\hfill \\ (-\infty ,0)\mathrm{if}{y}_i=0.\hfill \end{array}\right.\end{array}$$ (13)

4. Variational Inference

4.1. Overview of Variational Methods

Variational methods have their origins in the 18th century with the work of Euler, Lagrange, and others on the calculus of variations (The derivation in this section is standard in the literature on variational approximation and will at times follow the arguments in [22,23]). Variational inference is a body of deterministic techniques for making approximate inference for parameters in complex statistical models. Variational approximations are a much faster alternative to Markov Chain Monte Carlo (MCMC), especially for large models, and are a richer class of methods than the Laplace approximation [6].

Suppose we have a Bayesian model and a prior distribution for the parameters. The model may also have latent variables, here we shall denote the set of all latent variables and parameters by $\mathit{\theta }$. In addition, we denote the set of all observed variables by $\mathbf{X}$. Given a set of n independent, identically distributed data, for which $\mathbf{X}=\{{\mathbf{x}}_1,...,{\mathbf{x}}_n\}$ and $\mathit{\theta }=\{{\mathit{\theta }}_1,...,{\mathit{\theta }}_n\}$, our probabilistic model (e.g., probit regression model) specifies the joint distribution $p(\mathbf{X},\mathit{\theta })$, and our goal is to find an approximation for the posterior distribution $p(\mathit{\theta }|\mathbf{X})$ as well as for the marginal likelihood $p(\mathbf{X})$. For any probability distribution $q(\mathit{\theta })$, we have the following decomposition of the log marginal likelihood

$$\begin{array}{c}\hfill lnp(\mathbf{X})=\mathcal{L}(q)+\mathrm{KL}(q||p)\end{array}$$

where we have defined

$$\begin{array}{c}\hfill \mathcal{L}(q)=\int q(\mathit{\theta })ln\left\{\frac{p(\mathbf{X},\mathit{\theta })}{q(\mathit{\theta })}\right\}d\mathit{\theta }\end{array}$$ (14)

$$\begin{array}{c}\hfill \mathrm{KL}(q||p)=-\int q(\mathit{\theta })ln\left\{\frac{p(\mathit{\theta }|\mathbf{X})}{q(\mathit{\theta })}\right\}d\mathit{\theta }\end{array}$$ (15)

We refer to (14) as the lower bound of the log marginal likelihood with respect to the density q, and (15) is by definition the Kullback–Leibler divergence of the posterior $q(\mathit{\theta }|\mathbf{X})$ from the density q. Based on this decomposition, we can maximize the lower bound $\mathcal{L}(q)$ by optimization with respect to the distribution $q(\mathit{\theta })$, which is equivalent to minimizing the KL divergence. In addition, the lower bound is attained when the KL divergence is zero, which happens when $q(\mathit{\theta })$ equals the posterior distribution $p(\mathit{\theta }|\mathbf{X})$. It would be hard to find such a density since the true posterior distribution is intractable.

4.2. Factorized Distributions

The essence of the variational inference approach is approximation to the posterior distribution $p(\mathit{\theta }|\mathbf{X})$ by $q(\mathit{\theta })$ for which the q dependent lower bound $\mathcal{L}(q)$ is more tractable than the original model evidence. In addition, tractability is achieved by restricting q to a more manageable class of distributions, and then maximizing $\mathcal{L}(q)$ over that class.

Suppose we partition elements of $\mathit{\theta }$ into disjoint groups $\{{\mathit{\theta }}_i\}$ where $i=1,...,M$. We then assume that the q density factorizes with respect to this partition, i.e.,

$$\begin{array}{c}\hfill q(\mathit{\theta })=\prod _{i=1}^M{q}_i({\mathit{\theta }}_i).\end{array}$$ (16)

The product form is the only assumption we made about the distribution. Restriction (16) is also known as mean-field approximation and has its root in Physics [24].

For all distributions $q(\mathit{\theta })$ with the form (16), we need to find the distribution for which the lower bound $\mathcal{L}(q)$ is largest. Restriction of q to a subclass of product densities like (16) gives rise to explicit solutions for each product component in terms of the others. This fact, in turn, leads to an iterative scheme for obtaining the solutions. To achieve this, we first substitute (16) into (14) and then separate out the dependence on one of the factors $q_j({\mathit{\theta }}_j)$. Denoting $q_j({\mathit{\theta }}_j)$ by $q_j$ to keep the notation clear, we obtain

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \mathcal{L}(q)& =\int \prod _{i=1}^M{q}_i\left\{lnp(\mathbf{X},\mathit{\theta })-\sum _{i=1}^{M}ln{q}_i\right\}d\mathit{\theta }\hfill \\ \hfill & =\int q_j\left\{\int lnp(\mathbf{X},\mathit{\theta })\prod _{i\ne j}{q}_{i}d{\mathit{\theta }}_i\right\}d{\mathit{\theta }}_j-\int q_{j}ln{q}_{j}d{\mathit{\theta }}_j+\mathrm{constant}\hfill \\ \hfill & =\int q_{j}ln\tilde{p}(\mathbf{X},{\mathit{\theta }}_j)d{\mathit{\theta }}_j-\int q_{j}ln{q}_{j}d{\mathit{\theta }}_j+\mathrm{constant}\hfill \end{array}\end{array}$$ (17)

where $\tilde{p}(\mathbf{X},{\mathit{\theta }}_j)$ is given by

$$\begin{array}{c}\hfill ln\tilde{p}(\mathbf{X},{\mathit{\theta }}_j)={\mathbb{E}}_{i\ne j}[lnp(\mathbf{X},\mathit{\theta })]+\mathrm{constant}.\end{array}$$ (18)

The notation ${\mathbb{E}}_{i\ne j}[·]$ denotes an expectation with respect to the q distributions over all variables ${\mathbf{z}}_i$ for $i\ne j$, so that

$$\begin{array}{c}\hfill {\mathbb{E}}_{i\ne j}[lnp(\mathbf{X},\mathit{\theta })]=\int lnp(\mathbf{X},\mathit{\theta })\prod _{i\ne j}{q}_{i}d{\mathit{\theta }}_i.\end{array}$$

Now suppose we keep the $\{q_{i\ne j}\}$ fixed and maximize $\mathcal{L}(q)$ in (17) with respect to all possible forms for the density $q_j({\mathit{\theta }}_j)$. By recognizing that (17) is the negative KL divergence between $\tilde{p}(\mathbf{X},{\mathit{\theta }}_j)$ and $q_j({\mathit{\theta }}_j)$, we notice that maximizing (17) is equivalent to minimize the KL divergence, and the minimum occurs when $q_j({\mathit{\theta }}_j)=\tilde{p}(\mathbf{X},{\mathit{\theta }}_j)$. The optimal $q_j^{*}({\mathit{\theta }}_j)$ is then

$$\begin{array}{c}\hfill ln{q}_j^{*}({\mathit{\theta }}_j)={\mathbb{E}}_{i\ne j}[lnp(\mathbf{X},\mathit{\theta })]+\mathrm{constant}.\end{array}$$ (19)

The above solution says that the log of the optimal $q_j$ is obtained simply by considering the log of the joint distribution of all parameter, latent and observable variables and then taking the expectation with respect to all the other factors $q_i$ for $i\ne j$. Normalizing the exponential of (19), we have    

$$\begin{array}{c}\hfill q_j^{*}({\mathit{\theta }}_j)=\frac{exp({\mathbb{E}}_{i\ne j}[lnp(\mathbf{X},\mathit{\theta })])}{\int exp({\mathbb{E}}_{i\ne j}[lnp(\mathbf{X},\mathit{\theta })])d{\mathit{\theta }}_j}.\end{array}$$

The set of equations in (19) for $j=1,...,M$ are not an explicit solution because the expression on the right hand side of (19) for the optimal $q_j^{*}$ depends on expectations taken with respect to the other factors $q_i$ for $i\ne j$. We will need to first initialize all of the factors $q_i({\mathit{\theta }}_i)$ and then cycle through the factors one by one and replace each in turn with an updated estimate given by the right hand side of (19) evaluated using the current estimates for all of the other factors. Convexity properties can be used to show that convergence to at least local optima is guaranteed [25]. The iterative procedure is described in Algorithm 1.

Algorithm 1 Iterative procedure for obtaining the optimal densities under factorized density restriction (16). The updates are based on the solutions given by (19).

1:Initialize $q_2^{\ast }({\mathit{\theta }}_2),\dots ,q_M^{\ast }({\mathit{\theta }}_M).$ 2: Cycle through $$\begin{array}{c}{q}_1^{\ast }({\mathit{\theta }}_1)\leftarrow \frac{exp({\mathbb{E}}_{i\ne 1}[lnp(\mathbf{X},\mathit{\theta })])}{\int exp({\mathbb{E}}_{i\ne 1}[lnp(\mathbf{X},\mathit{\theta })])d{\mathit{\theta }}_1}\hfill \\ ⋮\hfill \\ q_M^{\ast }({\mathit{\theta }}_M)\leftarrow \frac{exp({\mathbb{E}}_{i\ne M}[lnp(\mathbf{X},\mathit{\theta })])}{\int exp({\mathbb{E}}_{i\ne M}[lnp(\mathbf{X},\mathit{\theta })])d{\mathit{\theta }}_M}\hfill \end{array}$$ until the increase in $\mathcal{L}(q)$ is negligible.

5. Incorporate Intrinsic Prior with Variational Approximation to Bayesian Probit Models

5.1. Derivation of Intrinsic Prior to Be Used in Variational Inference

Let ${\mathbf{X}}_l$ be the design matrix of a minimal training sample (mTS) of a normal regression model $M_j$ for the variable $\mathbf{Z}\sim N({\mathbf{X}}_j{\mathit{\beta }}_j,{\mathbf{I}}_{j+1})$. We have, for the $j+1$-dimensional parameter ${\mathit{\beta }}_j$,

$$\begin{array}{c}\hfill \int N_{j+1}({\mathbf{z}}_l|{\mathbf{X}}_l{\mathit{\beta }}_j,{\mathbf{I}}_{j+1})d{\mathit{\beta }}_j=\left\{\begin{array}{cc}|{\mathbf{X}}_l^{\prime }{\mathbf{X}}_l{|}^{-1/2}\hfill & \mathrm{if}\mathrm{rank}\mathrm{of}{\mathbf{X}}_l\ge j+1\hfill \\ \infty \hfill & \mathrm{otherwise}\hfill \end{array}\right..\end{array}$$

Therefore, it follows that the mTS size is $j+1$ [2]. Given that priors for $\alpha$ and $\mathit{\beta }$ are proportional to 1, the intrinsic prior for $\beta$ conditional on $\alpha$ could be derived. Let ${\mathit{\beta }}_0$ denote the vector with the first component equal to $\alpha$ and the others equal to zero. Based on Formula (9), we have

$$\begin{array}{cc}\hfill {\pi }^I(\mathit{\beta }|\alpha )& ={\pi }_j^N(\mathit{\beta }){\mathbb{E}}_{{\mathbf{z}}_l|\mathit{\beta }}^{M_j}\left[\frac{p_1({\mathbf{z}}_l|\alpha )}{\int p_j({\mathbf{z}}_l|\mathit{\beta }){\pi }_j^N(\mathit{\beta })d\mathit{\beta }}\right]\hfill \\ \hfill & ={\mathbb{E}}_{{\mathbf{z}}_l|\mathit{\beta }}^{M_j}\left[\frac{exp\{-\frac{1}{2}{({\mathbf{z}}_l-{\mathbf{X}}_l{\mathit{\beta }}_0)}^{\prime }({\mathbf{z}}_l-{\mathbf{X}}_l{\mathit{\beta }}_0)\}}{\int exp\{-\frac{1}{2}{({\mathbf{z}}_l-{\mathbf{X}}_l\mathit{\beta })}^{\prime }({\mathbf{z}}_l-{\mathbf{X}}_l\mathit{\beta })\}d\mathit{\beta }}\right]\hfill \\ \hfill & ={(2\pi )}^{-\frac{(j+1)}{2}}{|{({\mathbf{X}}_l^{\prime }{\mathbf{X}}_l)}^{-1}|}^{-\frac{1}{2}}\times {\mathbb{E}}_{{\mathbf{z}}_l|\mathit{\beta }}^{M_j}\left[exp\{-\frac{1}{2}{({\mathbf{z}}_l-{\mathbf{X}}_l{\mathit{\beta }}_0)}^{\prime }({\mathbf{z}}_l-{\mathbf{X}}_l{\mathit{\beta }}_0)\}\right]\hfill \\ \hfill & ={(2\pi )}^{-\frac{(j+1)}{2}}{|2{({\mathbf{X}}_l^{\prime }{\mathbf{X}}_l)}^{-1}|}^{-\frac{1}{2}}exp\{-\frac{1}{2}[{(\mathit{\beta }-{\mathit{\beta }}_0)}^{\prime }\frac{{\mathbf{X}}_l^{\prime }{\mathbf{X}}_l}{2}(\mathit{\beta }-{\mathit{\beta }}_0)]\}.\hfill \end{array}$$

Therefore,

$$\begin{array}{c}\hfill {\pi }^I(\mathit{\beta }|\alpha )=N_{j+1}(\mathit{\beta }|{\mathit{\beta }}_0,2{({\mathbf{X}}_l^{\prime }{\mathbf{X}}_l)}^{-1}),\mathrm{where}{\mathit{\beta }}_0={\left(\begin{array}{c}\alpha \\ 0\\ ⋮\\ 0\end{array}\right)}_{(j+1)\times 1}.\end{array}$$

Notice that ${\mathbf{X}}_l^{\prime }{\mathbf{X}}_l$ is unknown because it is a theoretical design matrix corresponding to the training sample ${\mathbf{z}}_l$. It can be estimated by averaging over all submatrices containing $j+1$ rows of the $n\times (j+1)$ design matrix ${\mathbf{X}}_j$. This average is $\frac{j+1}{n}{\mathbf{X}}_j^{\prime }{\mathbf{X}}_j$ (See [26] and Appendix A in [2]), and therefore

$$\begin{array}{c}\hfill {\pi }^I(\mathit{\beta }|\alpha )=N_{j+1}(\mathit{\beta }|{\mathit{\beta }}_0,\frac{2n}{j+1}{({\mathbf{X}}_j^{\prime }{\mathbf{X}}_j)}^{-1}).\end{array}$$

Next, based on ${\pi }^I(\mathit{\beta }|\alpha )$, the intrinsic prior for $\mathit{\beta }$ can be obtained by

$$\begin{array}{c}\hfill {\pi }^I(\mathit{\beta })=\int {\pi }^I(\mathit{\beta }|\alpha ){\pi }^I(\alpha )d\alpha .\end{array}$$ (20)

Since we assume that ${\pi }^I(\alpha )={\pi }^N(\alpha )$ is proportional to one, set ${\pi }^N(\alpha )=c$ where c is an arbitrary positive constant. Denote $\frac{2n}{j+1}{({\mathbf{X}}_j^{\prime }{\mathbf{X}}_j)}^{-1}$ by ${\Sigma }_{\mathit{\beta }|\alpha }$, we obtain

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\pi }^I(\mathit{\beta })& =\int {\pi }^I(\mathit{\beta }|\alpha ){\pi }^I(\alpha )d\alpha \hfill \\ \hfill & =c·{(2\pi )}^{-\frac{j+1}{2}}{|{\Sigma }_{\mathit{\beta }|\alpha }|}^{-\frac{1}{2}}\int exp\{-\frac{1}{2}{(\mathit{\beta }-{\mathit{\beta }}_0)}^{\prime }{\Sigma }_{\mathit{\beta }|\alpha }^{-1}(\mathit{\beta }-{\mathit{\beta }}_0)\}d\alpha \hfill \\ \hfill & \propto exp\{-\frac{1}{2}{\mathit{\beta }}^{\prime }{\Sigma }_{\mathit{\beta }|\alpha }^{-1}\mathit{\beta }\}\times \int exp\{-\frac{1}{2}[{\mathit{\beta }}_0^{\prime }{\Sigma }_{\mathit{\beta }|\alpha }^{-1}{\mathit{\beta }}_0-2{\mathit{\beta }}^{\prime }{\Sigma }_{\mathit{\beta }|\alpha }^{-1}{\mathit{\beta }}_0]\}d\alpha \hfill \\ \hfill & \propto exp\{-\frac{1}{2}{\mathit{\beta }}^{\prime }{\Sigma }_{\mathit{\beta }|\alpha }^{-1}\mathit{\beta }\}\times \int exp\{-\frac{1}{2}({\Sigma }_{\mathit{\beta }|{\alpha }_{(1,1)}}^{-1}{\alpha }^2-2{\mathit{\beta }}^{\prime }{\Sigma }_{\mathit{\beta }|{\alpha }_{(·1)}}^{-1}\alpha )\}d\alpha \hfill \end{array}\end{array}$$ (21)

where ${\Sigma }_{\mathit{\beta }|{\alpha }_{(1,1)}}^{-1}$ is component of ${\Sigma }_{\mathit{\beta }|\alpha }^{-1}$ at position row 1 column 1 and ${\Sigma }_{\mathit{\beta }|{\alpha }_{(·1)}}^{-1}$ is the first column of ${\Sigma }_{\mathit{\beta }|\alpha }^{-1}$. Denote ${\Sigma }_{\mathit{\beta }|{\alpha }_{(1,1)}}^{-1}$ by ${\sigma }_{11}$ and ${\Sigma }_{\mathit{\beta }|{\alpha }_{(·1)}}^{-1}$ by ${\gamma }_1$, we then obtain

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\pi }^I(\mathit{\beta })& \propto exp\{-\frac{1}{2}{\mathit{\beta }}^{\prime }{\Sigma }_{\mathit{\beta }|\alpha }^{-1}\mathit{\beta }\}\times \int exp\{-\frac{1}{2}{\sigma }_{11}{(\alpha -\frac{{\mathit{\beta }}^{\prime }{\gamma }_1}{{\sigma }_{11}})}^2+\frac{1}{2}\frac{{(\mathit{\beta }}^{\prime }{\gamma }_1{)}^2}{{\sigma }_{11}}\}d\alpha \hfill \\ \hfill & \propto exp\{-\frac{1}{2}({\mathit{\beta }}^{\prime }{\Sigma }_{\mathit{\beta }|\alpha }^{-1}\mathit{\beta }-{\mathit{\beta }}^{\prime }\frac{{\gamma }_1{\gamma }_1^{\prime }}{{\sigma }_{11}}\mathit{\beta })\}\times \sqrt{2\pi }{\sigma }_{11}^{-1/2}\hfill \\ \hfill & \propto exp\{-\frac{1}{2}{\mathit{\beta }}^{\prime }({\Sigma }_{\mathit{\beta }|\alpha }^{-1}-\frac{{\gamma }_1{\gamma }_1^{\prime }}{{\sigma }_{11}})\mathit{\beta }\}.\hfill \end{array}\end{array}$$ (22)

Therefore, we have derived that

$$\begin{array}{c}\hfill {\pi }^I(\mathit{\beta })\propto N_{j+1}(\mathbf{0},{({\Sigma }_{\mathit{\beta }|\alpha }^{-1}-\frac{{\gamma }_1{\gamma }_1^{\prime }}{{\sigma }_{11}})}^{-1}).\end{array}$$ (23)

For model comparison, the specific form of the intrinsic prior may be needed, including the constant factor. Therefore, by following (21) and (22) we have

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\pi }^I(\mathit{\beta })& =c·{(2\pi )}^{-\frac{j+1}{2}}|{\Sigma }_{\mathit{\beta }|\alpha }{|}^{-\frac{1}{2}}{(2\pi )}^{\frac{j+1}{2}}{|{({\Sigma }_{\mathit{\beta }|\alpha }^{-1}-\frac{{\gamma }_1{\gamma }_1^{\prime }}{{\sigma }_{11}})}^{-1}|}^{\frac{1}{2}}\sqrt{2\pi }{\sigma }_{11}^{-1/2}\times N_{j+1}(\mathbf{0},{({\Sigma }_{\mathit{\beta }|\alpha }^{-1}-\frac{{\gamma }_1{\gamma }_1^{\prime }}{{\sigma }_{11}})}^{-1})\hfill \\ \hfill & =c·|{\Sigma }_{\mathit{\beta }|\alpha }({\Sigma }_{\mathit{\beta }|\alpha }^{-1}-\frac{{\gamma }_1{\gamma }_1^{\prime }}{{\sigma }_{11}}){|}^{-\frac{1}{2}}\sqrt{2\pi }{\sigma }_{11}^{-1/2}\times N_{j+1}(\mathbf{0},{({\Sigma }_{\mathit{\beta }|\alpha }^{-1}-\frac{{\gamma }_1{\gamma }_1^{\prime }}{{\sigma }_{11}})}^{-1})\hfill \\ \hfill & =c·\sqrt{2\pi }{\sigma }_{11}^{-1/2}{|(\mathbb{I}-\frac{{\gamma }_1{\gamma }_1^{\prime }}{{\sigma }_{11}}{\Sigma }_{\mathit{\beta }|\alpha })|}^{-\frac{1}{2}}\times N_{j+1}(\mathbf{0},{({\Sigma }_{\mathit{\beta }|\alpha }^{-1}-\frac{{\gamma }_1{\gamma }_1^{\prime }}{{\sigma }_{11}})}^{-1}).\hfill \end{array}\end{array}$$ (24)

5.2. Variational Inference for Probit Model with Intrinsic Prior

5.2.1. Iterative Updates for Factorized Distributions

We have that

$$\begin{array}{cc}\hfill & Z_i|\mathit{\beta }\sim N(x_i^{\prime }\mathit{\beta },1)\mathrm{and}\hfill \\ \hfill & p(y_i|z_i)=\mathbb{I}(z_i>0)\mathbb{I}(y_i=1)+\mathbb{I}(z_i\le 0)\mathbb{I}(y_i=0)\hfill \end{array}$$

in Section 3.1. We have shown in Section 5.1 that

$$\begin{array}{c}\hfill {\pi }^I(\mathit{\beta })\propto N_{j+1}({\mu }_{\mathit{\beta }},{\Sigma }_{\mathit{\beta }}),\end{array}$$

where ${\mu }_{\mathit{\beta }}=\mathbf{0}$ and ${\Sigma }_{\mathit{\beta }}={({\Sigma }_{\mathit{\beta }|\alpha }^{-1}-\frac{{\gamma }_1{\gamma }_1^{\prime }}{{\sigma }_{11}})}^{-1}$. Since $\mathbf{y}$ is independent of $\mathit{\beta }$ given $\mathbf{z}$, we have

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill p(\mathbf{y},\mathbf{z},\mathit{\beta })& =p(\mathbf{y}|\mathbf{z},\mathit{\beta })p(\mathbf{z}|\mathit{\beta })p(\mathit{\beta })\hfill \\ \hfill & =p(\mathbf{y}|\mathbf{z})p(\mathbf{z}|\mathit{\beta })p(\mathit{\beta }).\hfill \end{array}\end{array}$$ (25)

To apply the variational approximation to probit regression model, unobservable variables are considered in two separate groups, coefficient parameter $\mathit{\beta }$ and auxiliary variable $\mathbf{Z}$. To approximate the posterior distribution of $\mathit{\beta }$, consider the product form

$$\begin{array}{c}\hfill q(\mathbf{Z},\mathit{\beta })=q_{\mathbf{Z}}(\mathbf{Z})q_{\mathit{\beta }}(\mathit{\beta }).\end{array}$$

We proceed by first describing the distribution for each factor of the approximation, $q_{\mathbf{Z}}(\mathbf{Z})$ and $q_{\mathit{\beta }}(\mathit{\beta })$. Then variational approximation is accomplished by iteratively updating the parameters of each factor distribution.

Start with $q_{\mathbf{Z}}(\mathbf{Z})$, when $y_i=1$, we have

$$\begin{array}{cc}\hfill logp(\mathbf{y},\mathbf{z},\mathit{\beta })& =log\left(\prod _i\frac{1}{\sqrt{2\pi }}exp\{-\frac{{(z_i-x_i^{\prime }\mathit{\beta })}^2}{2}\}\times {\pi }^I(\mathit{\beta })\right)\mathrm{where}{z}_i>0.\hfill \end{array}$$

Now, according to (19) and Algorithm 1, the optimal $q_{\mathbf{Z}}$ is proportional to

So, we have the optimal $q_{\mathbf{Z}}$,

$$\begin{array}{cc}\hfill q_{\mathbf{Z}}^{*}(\mathbf{Z})& \propto exp\{-\frac{1}{2}{\mathbf{z}}^{\prime }\mathbf{z}+{\mathbb{E}}_{\mathit{\beta }}{[\mathit{\beta }]}^{\prime }{\mathbf{X}}^{\prime }\mathbf{z}+\mathrm{constant}\}\hfill \\ \hfill & \propto exp\{-\frac{1}{2}{(\mathbf{z}-\mathbf{X}{\mathbb{E}}_{\mathit{\beta }}[\mathit{\beta }])}^{\prime }(\mathbf{z}-\mathbf{X}{\mathbb{E}}_{\mathit{\beta }}[\mathit{\beta }])\}.\hfill \end{array}$$

Similar procedure could be used to develop cases when $y_i=0$. Therefore, we have that the optimal approximation for $q_{\mathbf{Z}}$ is a truncated normal distribution, where

$$\begin{array}{c}\hfill q_{\mathbf{Z}}^{*}(\mathbf{Z})=\left\{\begin{array}{c}{N}_{[0,+\infty )}(\mathbf{X}{\mathbb{E}}_{\mathit{\beta }}{[\mathit{\beta }]}_i,1)\mathrm{if}{y}_i=1,\hfill \\ N_{(-\infty ,0]}(\mathbf{X}{\mathbb{E}}_{\mathit{\beta }}{[\mathit{\beta }]}_i,1)\mathrm{if}{y}_i=0.\hfill \end{array}\right.\end{array}$$ (26)

Denote $\mathbf{X}{\mathbb{E}}_{\mathit{\beta }}[\mathit{\beta }]$ by ${\mu }_{\mathbf{z}}$, the location of distribution $q_{\mathbf{Z}}^{*}(\mathbf{Z})$. The expectation ${\mathbb{E}}_{\mathit{\beta }}$ is taken with respect to the density form of $q(\mathit{\beta })$ for which we shall derive now.

For $q_{\mathit{\beta }}(\mathit{\beta })$, given the joint form in (25), we have

$$\begin{array}{c}\hfill logp(\mathbf{y},\mathbf{z},\mathit{\beta })=-\frac{1}{2}exp\{{(\mathbf{z}-\mathbf{X}\mathit{\beta })}^{\prime }(\mathbf{z}-\mathbf{X}\mathit{\beta })\}-\frac{1}{2}exp\{{(\mathit{\beta }-{\mu }_{\mathit{\beta }})}^{\prime }{\Sigma }_{\mathit{\beta }}^{-1}(\mathit{\beta }-{\mu }_{\mathit{\beta }})\}+\mathbf{constant}.\end{array}$$

Taking expectation with respect to $q_{\mathbf{Z}}(\mathbf{z})$, we have

Again, based on (19) and Algorithm 1, the optimal $q_{\mathit{\beta }}(\mathit{\beta })$ is proportional to ${\mathbb{E}}_{\mathbf{Z}}[logp(\mathbf{y},\mathbf{z},\mathit{\beta })]$,

$$\begin{array}{c}\hfill q_{\mathit{\beta }}^{*}(\mathit{\beta })\propto -\frac{1}{2}{\mathit{\beta }}^{\prime }({\mathbf{X}}^{\prime }\mathbf{X}+{\Sigma }_{\mathit{\beta }}^{-1})\mathit{\beta }+({\mathbb{E}}_{\mathbf{Z}}{[\mathbf{Z}]}^{\prime }\mathbf{X}+{\mu }_{\mathit{\beta }}^{\prime }{\Sigma }_{\mathit{\beta }}^{-1})\mathit{\beta }.\end{array}$$

First notice that any constant terms, including constant factor in the intrinsic prior, were canceled out due to the ratio form of (19). Then by noticing the quadratic form in the above formula we have

$$\begin{array}{c}\hfill q_{\mathit{\beta }}^{*}(\mathit{\beta })=N({\mu }_{q_{\mathit{\beta }}},{\Sigma }_{q_{\mathit{\beta }}}),\end{array}$$ (27)

where

$$\begin{array}{cc}\hfill {\Sigma }_{q_{\mathit{\beta }}}& ={({\mathbf{X}}^{\prime }\mathbf{X}+{\Sigma }_{\mathit{\beta }}^{-1})}^{-1},\hfill \\ \hfill {\mu }_{q_{\mathit{\beta }}}& ={({\mathbf{X}}^{\prime }\mathbf{X}+{\Sigma }_{\mathit{\beta }}^{-1})}^{-1}({\mathbb{E}}_{\mathbf{Z}}{[\mathbf{Z}]}^{\prime }\mathbf{X}+{\mu }_{\mathit{\beta }}^{\prime }{\Sigma }_{\mathit{\beta }}^{-1}).\hfill \end{array}$$

Notice that ${\mu }_{q_{\mathit{\beta }}}$, i.e., ${\mathbb{E}}_{\mathit{\beta }}[\mathit{\beta }]$, depends on ${\mathbb{E}}_{\mathbf{Z}}[\mathbf{Z}]$. In addition, from our previous derivation, we found that the update for ${\mathbb{E}}_{\mathbf{Z}}[\mathbf{Z}]$ depends on ${\mathbb{E}}_{\mathit{\beta }}[\mathit{\beta }]$. Given that the density form of $q_{\mathbf{Z}}$ is truncated normal, we have

$$\begin{array}{c}\hfill {\mathbb{E}}_{\mathbf{Z}}[{\mathbf{Z}}_i]=\left\{\begin{array}{cc}\mathbf{X}{\mathbb{E}}_{\mathit{\beta }}{[\mathit{\beta }]}_i+\frac{\varphi (-\mathbf{X}{\mathbb{E}}_{\mathit{\beta }}{[\mathit{\beta }]}_i)}{1-\Phi {(-\mathbf{X}{\mathbb{E}}_{\mathit{\beta }}[\mathit{\beta }])}_i}\hfill & \mathrm{if}{y}_i=1,\hfill \\ \mathbf{X}{\mathbb{E}}_{\mathit{\beta }}{[\mathit{\beta }]}_i-\frac{\varphi (-\mathbf{X}{\mathbb{E}}_{\mathit{\beta }}{[\mathit{\beta }]}_i)}{\Phi {(-\mathbf{X}{\mathbb{E}}_{\mathit{\beta }}[\mathit{\beta }])}_i}\hfill & \mathrm{if}{y}_i=0,\hfill \end{array}\right.\end{array}$$

where $\varphi$ is the standard normal density and $\Phi$ is the standard normal cumulative density. Denote ${\mathbb{E}}_{\mathbf{Z}}[\mathbf{Z}]$ by ${\mu }_{q_{\mathbf{Z}}}$. See properties of truncated normal distribution in Appendix A. Updating procedures for parameters ${\mu }_{q_{\mathit{\beta }}}$ and ${\mu }_{q_{\mathbf{Z}}}$ of each factor distribution are summarized in Algorithm 2.

Algorithm 2 Iterative procedure for updating parameters to reach optimal factor densities $q_{\mathit{\beta }}^{\ast }$ and $q_{\mathbf{Z}}^{\ast }$ in Bayesian probit regression model. The updates are based on the solutions given by (26) and (27).

1:Initialize ${\mu }_{q_{\mathbf{Z}}}.$ 2: Cycle through $$\begin{array}{ccc}\hfill & {\mu }_{q_{\mathbf{\beta }}}\leftarrow {({\mathbf{X}}^{\prime }\mathbf{X}+{\Sigma }_{\mathbf{\beta }}^{-1})}^{-1}({\mu }_{q_{\mathbf{z}}}^{\prime }\mathbf{X}+{\mu }_{\mathbf{\beta }}^{\prime }{\Sigma }_{\mathbf{\beta }}^{-1}),\hfill & \hfill {\mu }_{q_{\mathbf{Z}}}\leftarrow \mathbf{X}{\mu }_{q_{\mathbf{\beta }}}+\frac{\varphi \left(\mathbf{X}{\mu }_{q_{\mathbf{\beta }}}\right)}{\Phi {\left(\mathbf{X}{\mu }_{q_{\mathbf{\beta }}}\right)}^{\mathbf{y}}{[\Phi \left(\mathbf{X}{\mu }_{q_{\mathbf{\beta }}}\right)-\mathbf{1}]}^{\mathbf{1}-\mathbf{y}}},\end{array}$$ until the increase in $\mathcal{L}\left(q\right)$ is negligible.

5.2.2. Evaluation of the Lower Bound $\mathcal{L}(q)$

During the process of optimization of variational approximation densities, the lower bound for the log marginal likelihood need to be evaluated and monitored to determine when the iterative updating process converges. Based on derivations from previous section, we now have the exact form for the variational inference density,

$$\begin{array}{c}\hfill q(\mathit{\beta },\mathbf{Z})=q_{\mathit{\beta }}(\mathit{\beta })q_{\mathbf{Z}}(\mathbf{Z}).\end{array}$$

According to (14), we can write down the lower bound $\mathcal{L}(q)$ with respect to $q(\mathit{\beta },\mathbf{Z})$.

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \mathcal{L}(q)& =\int q(\mathit{\beta },\mathbf{Z})log\left\{\frac{p(\mathbf{Y},\mathit{\beta },\mathbf{Z})}{q(\mathit{\beta },\mathbf{Z})}\right\}d\mathit{\beta }d\mathbf{Z}\hfill \\ \hfill & =\int q_{\mathit{\beta }}(\mathit{\beta })q_{\mathbf{Z}}(\mathbf{Z})log\left\{\frac{p(\mathbf{Y},\mathit{\beta },\mathbf{Z})}{q_{\mathit{\beta }}(\mathit{\beta })q_{\mathbf{Z}}(\mathbf{Z})}\right\}d\mathit{\beta }d\mathbf{Z}\hfill \\ \hfill & =\int q_{\mathit{\beta }}(\mathit{\beta })q_{\mathbf{Z}}(\mathbf{Z})log\{p(\mathbf{Y},\mathit{\beta },\mathbf{Z})\}d\mathit{\beta }d\mathbf{Z}-\int q_{\mathit{\beta }}(\mathit{\beta })q_{\mathbf{Z}}(\mathbf{Z})log\{q_{\mathit{\beta }}(\mathit{\beta })q_{\mathbf{Z}}(\mathbf{Z})\}d\mathit{\beta }d\mathbf{Z}\hfill \\ \hfill & ={\mathbb{E}}_{\mathit{\beta },\mathbf{Z}}[log\{p(\mathbf{Y},\mathbf{Z}|\mathit{\beta })\}]+{\mathbb{E}}_{\mathit{\beta },\mathbf{Z}}[{\pi }^I(\mathit{\beta })]-{\mathbb{E}}_{\mathit{\beta },\mathbf{Z}}[log\{q_{\mathit{\beta }}(\mathit{\beta })\}]-{\mathbb{E}}_{\mathit{\beta },\mathbf{Z}}[log\{q_{\mathbf{Z}}(\mathbf{Z})\}].\hfill \end{array}\end{array}$$ (28)

As we can see in (28), $\mathcal{L}(q)$ has been divided into four different parts with expectation taken over the variational approximation density $q(\mathit{\beta },\mathbf{Z})=q_{\mathit{\beta }}(\mathit{\beta })q_{\mathbf{Z}}(\mathbf{Z})$. We now find the expression of these expectations one by one.

Part 1: ${\mathbb{E}}_{\mathit{\beta },\mathbf{Z}}[log\{p(\mathbf{Y},\mathbf{Z}|\mathit{\beta })\}]$

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & =log{(2\pi )}^{-\frac{n}{2}}+\int \int q_{\mathit{\beta }}(\mathit{\beta })q_{\mathbf{Z}}(\mathbf{Z})\{-\frac{1}{2}{(\mathbf{z}-\mathbf{X}\mathit{\beta })}^{\prime }(\mathbf{z}-\mathbf{X}\mathit{\beta })\}d\mathit{\beta }d\mathbf{z}\hfill \\ \hfill & =log{(2\pi )}^{-\frac{n}{2}}+\int q_{\mathbf{Z}}(\mathbf{Z})\int q_{\mathit{\beta }}(\mathit{\beta })\{-\frac{1}{2}({\mathit{\beta }}^{\prime }{\mathbf{X}}^{\prime }\mathbf{X}\mathit{\beta }-2{\mathbf{z}}^{\prime }\mathbf{X}\mathit{\beta }+{\mathbf{z}}^{\prime }\mathbf{z})\}d\mathit{\beta }d\mathbf{z}\hfill \end{array}\end{array}$$ (29)

Deal with the inner integral first, we have

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \int q_{\mathit{\beta }}(\mathit{\beta })\{-\frac{1}{2}({\mathit{\beta }}^{\prime }{\mathbf{X}}^{\prime }\mathbf{X}\mathit{\beta }-2{\mathbf{z}}^{\prime }\mathbf{X}\mathit{\beta }+{\mathbf{z}}^{\prime }\mathbf{z})\}d\mathit{\beta }& =-\frac{1}{2}\int q_{\mathit{\beta }}(\mathit{\beta })[{\mathit{\beta }}^{\prime }{\mathbf{X}}^{\prime }\mathbf{X}\mathit{\beta }]d\mathit{\beta }+{\mathbf{z}}^{\prime }\mathbf{X}{\mathbb{E}}_{\mathit{\beta }}[\mathit{\beta }]-\frac{1}{2}{\mathbf{z}}^{\prime }\mathbf{z}\hfill \\ \hfill & =-\frac{1}{2}\int q_{\mathit{\beta }}(\mathit{\beta })[{\mathit{\beta }}^{\prime }{\mathbf{X}}^{\prime }\mathbf{X}\mathit{\beta }]d\mathit{\beta }+{\mathbf{z}}^{\prime }\mathbf{X}{\mu }_{q_{\mathit{\beta }}}-\frac{1}{2}{\mathbf{z}}^{\prime }\mathbf{z}\hfill \end{array}\end{array}$$ (30)

where

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill -\frac{1}{2}\int q_{\mathit{\beta }}(\mathit{\beta })[{\mathit{\beta }}^{\prime }{\mathbf{X}}^{\prime }\mathbf{X}\mathit{\beta }]d\mathit{\beta }& =-\frac{1}{2}\int q_{\mathit{\beta }}(\mathit{\beta })[{(\mathit{\beta }-{\mu }_{q_{\mathit{\beta }}}+{\mu }_{q_{\mathit{\beta }}})}^{\prime }{\mathbf{X}}^{\prime }\mathbf{X}(\mathit{\beta }-{\mu }_{q_{\mathit{\beta }}}+{\mu }_{q_{\mathit{\beta }}})]d\mathit{\beta }\hfill \\ \hfill & =-\frac{1}{2}\mathrm{trace}({\mathbf{X}}^{\prime }\mathbf{X}{\mathbb{E}}_{\mathit{\beta }}[(\mathit{\beta }-{\mu }_{q_{\mathit{\beta }}}){(\mathit{\beta }-{\mu }_{q_{\mathit{\beta }}})}^{\prime }])-\frac{1}{2}{\mu }_{q_{\mathit{\beta }}}^{\prime }{\mathbf{X}}^{\prime }\mathbf{X}{\mu }_{q_{\mathit{\beta }}}\hfill \\ \hfill & =-\frac{1}{2}\mathrm{trace}({\mathbf{X}}^{\prime }\mathbf{X}[{\mu }_{q_{\mathit{\beta }}}{\mu }_{q_{\mathit{\beta }}}^{\prime }+{\Sigma }_{q_{\mathit{\beta }}}]).\hfill \end{array}\end{array}$$ (31)

Substitute (31) into (30), we got

$$\begin{array}{c}\hfill \int q_{\mathit{\beta }}(\mathit{\beta })\{-\frac{1}{2}({\mathit{\beta }}^{\prime }{\mathbf{X}}^{\prime }\mathbf{X}\mathit{\beta }-2{\mathbf{z}}^{\prime }\mathbf{X}\mathit{\beta }+{\mathbf{z}}^{\prime }\mathbf{z})\}d\mathit{\beta }=-\frac{1}{2}\mathrm{trace}({\mathbf{X}}^{\prime }\mathbf{X}[{\mu }_{q_{\mathit{\beta }}}{\mu }_{q_{\mathit{\beta }}}^{\prime }+{\Sigma }_{q_{\mathit{\beta }}}])+{\mathbf{z}}^{\prime }\mathbf{X}{\mu }_{q_{\mathit{\beta }}}-\frac{1}{2}{\mathbf{z}}^{\prime }\mathbf{z}.\end{array}$$ (32)

Substituting (32) back into (29) gives

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathbb{E}}_{\mathit{\beta },\mathbf{Z}}[log\{p(\mathbf{Y},\mathbf{Z}|\mathit{\beta })\}]& =log{(2\pi )}^{-\frac{n}{2}}+\int q_{\mathbf{Z}}(\mathbf{z})\{-\frac{1}{2}\mathrm{trace}({\mathbf{X}}^{\prime }\mathbf{X}[{\mu }_{q_{\mathit{\beta }}}{\mu }_{q_{\mathit{\beta }}}^{\prime }+{\Sigma }_{q_{\mathit{\beta }}}])+{\mathbf{z}}^{\prime }\mathbf{X}{\mu }_{q_{\mathit{\beta }}}-\frac{1}{2}{\mathbf{z}}^{\prime }\mathbf{z}\}d\mathbf{z}\hfill \\ \hfill & =log{(2\pi )}^{-\frac{n}{2}}-\frac{1}{2}\mathrm{trace}({\mathbf{X}}^{\prime }\mathbf{X}[{\mu }_{q_{\mathit{\beta }}}{\mu }_{q_{\mathit{\beta }}}^{\prime }+{\Sigma }_{q_{\mathit{\beta }}}])-\frac{1}{2}{\mathbb{E}}_{\mathbf{Z}}[{\mathbf{z}}^{\prime }\mathbf{z}]+{\mu }_{q_{\mathbf{z}}}^{\prime }{\mu }_{\mathbf{z}}\hfill \\ \hfill & =log{(2\pi )}^{-\frac{n}{2}}-\frac{1}{2}\mathrm{trace}({\mathbf{X}}^{\prime }\mathbf{X}[{\mu }_{q_{\mathit{\beta }}}{\mu }_{q_{\mathit{\beta }}}^{\prime }+{\Sigma }_{q_{\mathit{\beta }}}])+{\mu }_{q_{\mathbf{z}}}^{\prime }{\mu }_{\mathbf{z}}\hfill \\ \hfill & -\frac{1}{2}\sum _{i=1}^n{[1+{\mu }_{{\mathbf{z}}_i}^2-{\mu }_{{\mathbf{z}}_i}\frac{\varphi (-{\mu }_{{\mathbf{z}}_i})}{\Phi (-{\mu }_{{\mathbf{z}}_i})}]}^{\mathbb{I}(y_i=0)}{[1+{\mu }_{{\mathbf{z}}_i}^2+{\mu }_{{\mathbf{z}}_i}\frac{\varphi (-{\mu }_{{\mathbf{z}}_i})}{1-\Phi (-{\mu }_{{\mathbf{z}}_i})}]}^{\mathbb{I}(y_i=1)}\hfill \\ \hfill & =log{(2\pi )}^{-\frac{n}{2}}-\frac{1}{2}\mathrm{trace}({\mathbf{X}}^{\prime }\mathbf{X}[{\mu }_{q_{\mathit{\beta }}}{\mu }_{q_{\mathit{\beta }}}^{\prime }+{\Sigma }_{q_{\mathit{\beta }}}])+{\mu }_{q_{\mathbf{z}}}^{\prime }{\mu }_{\mathbf{z}}\hfill \\ \hfill & -\frac{1}{2}\sum _{i=1}^n{[1+{\mu }_{q_{{\mathbf{z}}_i}}{\mu }_{{\mathbf{z}}_i}]}^{\mathbb{I}(y_i=0)}{[1+{\mu }_{q_{{\mathbf{z}}_i}}{\mu }_{{\mathbf{z}}_i}]}^{\mathbb{I}(y_i=1)}\hfill \\ \hfill & =log{(2\pi )}^{-\frac{n}{2}}-\frac{1}{2}\mathrm{trace}({\mathbf{X}}^{\prime }\mathbf{X}[{\mu }_{q_{\mathit{\beta }}}{\mu }_{q_{\mathit{\beta }}}^{\prime }+{\Sigma }_{q_{\mathit{\beta }}}])+\frac{1}{2}{\mu }_{q_{\mathbf{z}}}^{\prime }{\mu }_{\mathbf{z}}-\frac{n}{2}.\hfill \end{array}\end{array}$$ (33)

We applied properties of truncated normal distribution in Appendix B to find the expression of the second moment ${\mathbb{E}}_{\mathbf{Z}}[{\mathbf{z}}^{\prime }\mathbf{z}]$.

Part 2: ${\mathbb{E}}_{\mathit{\beta },\mathbf{Z}}[log{q}_{\mathbf{Z}}(\mathbf{z})]$

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & =\int \int q_{\mathit{\beta }}(\mathit{\beta })q_{\mathbf{Z}}(\mathbf{z})log{q}_{\mathbf{Z}}(\mathbf{z})d\mathit{\beta }d\mathbf{Z}\hfill \\ \hfill & =\int q_{\mathbf{Z}}(\mathbf{z})log{q}_{\mathbf{Z}}(\mathbf{z})d\mathbf{Z}\hfill \\ \hfill & =-\frac{n}{2}(log(2\pi )+1)\hfill \\ \hfill & +\sum _{i=1}^n\{{[log(\Phi (-{\mu }_{{\mathbf{z}}_i}))+{\mu }_{{\mathbf{z}}_i}\frac{\varphi (-{\mu }_{{\mathbf{z}}_i})}{2\Phi (-{\mu }_{{\mathbf{z}}_i})}]}^{\mathbb{I}(y_i=0)}{[log(1-\Phi (-{\mu }_{{\mathbf{z}}_i}))-{\mu }_{{\mathbf{z}}_i}\frac{\varphi (-{\mu }_{{\mathbf{z}}_i})}{2(1-\Phi (-{\mu }_{{\mathbf{z}}_i}))}]}^{\mathbb{I}(y_i=1)}\}\hfill \\ \hfill & =-\frac{n}{2}(log(2\pi )+1)-\frac{1}{2}{\mu }_{\mathbf{z}}^{\prime }{\mu }_{\mathbf{z}}+\frac{1}{2}{\mu }_{q_{\mathbf{z}}}^{\prime }{\mu }_{\mathbf{z}}+\sum _{i=1}^n\{{[log(\Phi (-{\mu }_{{\mathbf{z}}_i}))]}^{\mathbb{I}(y_i=0)}{[log(1-\Phi (-{\mu }_{{\mathbf{z}}_i}))]}^{\mathbb{I}(y_i=1)}\}\hfill \end{array}\end{array}$$ (34)

Again, see Appendix B for well-known properties of truncated normal distribution. Now subtracting (34) from (33) we got

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathbb{E}}_{\mathit{\beta },\mathbf{Z}}[log\{p(\mathbf{Y},\mathbf{Z}|\mathit{\beta })\}]-{\mathbb{E}}_{\mathit{\beta },\mathbf{Z}}[log{q}_{\mathbf{Z}}(\mathbf{z})]& =-\frac{1}{2}\mathrm{trace}({\mathbf{X}}^{\prime }\mathbf{X}[{\mu }_{q_{\mathit{\beta }}}{\mu }_{q_{\mathit{\beta }}}^{\prime }+{\Sigma }_{q_{\mathit{\beta }}}])+\frac{1}{2}{\mu }_{\mathbf{z}}^{\prime }{\mu }_{\mathbf{z}}+\hfill \\ \hfill & \sum _{i=1}^n\{{[log(\Phi (-{\mu }_{{\mathbf{z}}_i}))]}^{\mathbb{I}(y_i=0)}{[log(1-\Phi (-{\mu }_{{\mathbf{z}}_i}))]}^{\mathbb{I}(y_i=1)}\}.\hfill \end{array}\end{array}$$ (35)

Based on the exact expression of the intrinsic prior ${\pi }^I(\mathit{\beta })$, denoting all constant terms by C, we have

Part 3: ${\mathbb{E}}_{\mathit{\beta },\mathbf{Z}}[log{p}_{\mathit{\beta }}(\mathit{\beta })]$

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & =\int \int q_{\mathbf{Z}}(\mathbf{z})q_{\mathit{\beta }}(\mathit{\beta })log{\pi }^I(\mathit{\beta })d\mathit{\beta }d\mathbf{z}\hfill \\ \hfill & =logC-\frac{(j+1)}{2}log(2\pi )-\frac{1}{2}log|{\Sigma }_{\mathit{\beta }}|-\frac{1}{2}\int q_{\mathit{\beta }}(\mathit{\beta })[{\mathit{\beta }}^{\prime }{\Sigma }_{\mathit{\beta }}^{-1}\mathit{\beta }]d\mathit{\beta }\hfill \end{array}\end{array}$$ (36)

To find the expression for the integral, we have

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \int q_{\mathit{\beta }}(\mathit{\beta })[{\mathit{\beta }}^{\prime }{\Sigma }_{\mathit{\beta }}^{-1}\mathit{\beta }]d\mathit{\beta }& =\int q_{\mathit{\beta }}(\mathit{\beta }){(\mathit{\beta }-{\mu }_{q_{\mathit{\beta }}}+{\mu }_{q_{\mathit{\beta }}})}^{\prime }{\Sigma }_{\mathit{\beta }}^{-1}(\mathit{\beta }-{\mu }_{q_{\mathit{\beta }}}+{\mu }_{q_{\mathit{\beta }}})d\mathit{\beta }\hfill \\ \hfill & =\mathbb{E}[\mathrm{trace}({\Sigma }_{\mathit{\beta }}^{-1}(\mathit{\beta }-{\mu }_{q_{\mathit{\beta }}}){(\mathit{\beta }-{\mu }_{q_{\mathit{\beta }}})}^{\prime })]+{\mu }_{q_{\mathit{\beta }}}^{\prime }{\Sigma }_{\mathit{\beta }}^{-1}{\mu }_{q_{\mathit{\beta }}}\hfill \\ \hfill & =\mathrm{trace}({\Sigma }_{\mathit{\beta }}^{-1}{\Sigma }_{q_{\mathit{\beta }}})+{\mu }_{q_{\mathit{\beta }}}^{\prime }{\Sigma }_{\mathit{\beta }}^{-1}{\mu }_{q_{\mathit{\beta }}}\hfill \end{array}\end{array}$$ (37)

Substituting (37) back into (36), we obtained

$$\begin{array}{c}\hfill {\mathbb{E}}_{\mathit{\beta },\mathbf{Z}}[log{p}_{\mathit{\beta }}(\mathit{\beta })]=logC-\frac{(j+1)}{2}log(2\pi )-\frac{1}{2}log|{\Sigma }_{\mathit{\beta }}|-\frac{1}{2}[\mathrm{trace}({\Sigma }_{\mathit{\beta }}^{-1}{\Sigma }_{q_{\mathit{\beta }}})+{\mu }_{q_{\mathit{\beta }}}^{\prime }{\Sigma }_{\mathit{\beta }}^{-1}{\mu }_{q_{\mathit{\beta }}}].\end{array}$$ (38)

Part 4: ${\mathbb{E}}_{\mathit{\beta },\mathbf{Z}}[log{q}_{\mathit{\beta }}(\mathit{\beta })]$

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & =\int \int q_{\mathbf{Z}}(\mathbf{z})q_{\mathit{\beta }}(\mathit{\beta })log{q}_{\mathit{\beta }}(\mathit{\beta })d\mathit{\beta }\hfill \\ \hfill & =-\frac{j+1}{2}log(2\pi )-\frac{1}{2}log|{\Sigma }_{q_{\mathit{\beta }}}|-\frac{1}{2}\int q_{\mathit{\beta }}(\mathit{\beta }){(\mathit{\beta }-{\mu }_{q_{\mathit{\beta }}})}^{\prime }{\Sigma }_{q_{\mathit{\beta }}}^{-1}(\mathit{\beta }-{\mu }_{q_{\mathit{\beta }}})d\mathit{\beta }\hfill \\ \hfill & =-\frac{j+1}{2}log(2\pi )-\frac{1}{2}log|{\Sigma }_{q_{\mathit{\beta }}}|-\frac{1}{2}\mathrm{trace}({\Sigma }_{\mathit{\beta }}^{-1}{\Sigma }_{\mathit{\beta }})\hfill \\ \hfill & =-\frac{j+1}{2}(log(2\pi )+1)-\frac{1}{2}log|{\Sigma }_{q_{\mathit{\beta }}}|\hfill \end{array}\end{array}$$ (39)

Combining all four parts together, we get

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \mathcal{L}(q)& ={\mathbb{E}}_{\mathit{\beta },\mathbf{Z}}[log\{p(\mathbf{Y},\mathbf{Z}|\mathit{\beta })\}]+{\mathbb{E}}_{\mathit{\beta },\mathbf{Z}}[{\pi }^I(\mathit{\beta })]-{\mathbb{E}}_{\mathit{\beta },\mathbf{Z}}[log\{q_{\mathit{\beta }}(\mathit{\beta })\}]-{\mathbb{E}}_{\mathit{\beta },\mathbf{Z}}[log\{q_{\mathbf{Z}}(\mathbf{Z})\}]\hfill \\ \hfill & =\underset{{\mathbb{E}}_{\mathit{\beta },\mathbf{Z}}[log\{p(\mathbf{Y},\mathbf{Z}|\mathit{\beta })\}]-{\mathbb{E}}_{\mathit{\beta },\mathbf{Z}}[log\{q_{\mathbf{Z}}(\mathbf{Z})\}]}{\underbrace{-\frac{1}{2}\mathrm{trace}({\mathbf{X}}^{\prime }\mathbf{X}[{\mu }_{q_{\mathit{\beta }}}{\mu }_{q_{\mathit{\beta }}}^{\prime }+{\Sigma }_{q_{\mathit{\beta }}}])+\frac{1}{2}{\mu }_{\mathbf{z}}^{\prime }{\mu }_{\mathbf{z}}+\sum _{i=1}^n\{{[log(\Phi (-{\mu }_{{\mathbf{z}}_i}))]}^{\mathbb{I}(y_i=0)}{[log(1-\Phi (-{\mu }_{{\mathbf{z}}_i}))]}^{\mathbb{I}(y_i=1)}\}}}\hfill \\ \hfill & \underset{{\mathbb{E}}_{\mathit{\beta },\mathbf{Z}}[log{p}_{\mathit{\beta }}(\mathit{\beta })]-{\mathbb{E}}_{\mathit{\beta },\mathbf{Z}}[log{q}_{\mathit{\beta }}(\mathit{\beta })]}{\underbrace{+logC-\frac{1}{2}log|{\Sigma }_{\mathit{\beta }}|-\frac{1}{2}[\mathrm{trace}({\Sigma }_{\mathit{\beta }}^{-1}{\Sigma }_{q_{\mathit{\beta }}})+{\mu }_{q_{\mathit{\beta }}}^{\prime }{\Sigma }_{\mathit{\beta }}^{-1}{\mu }_{q_{\mathit{\beta }}}]+\frac{j+1}{2}+\frac{1}{2}log|{\Sigma }_{q_{\mathit{\beta }}}|}}.\hfill \end{array}\end{array}$$ (40)

5.3. Model Comparison Based on Variational Approximation

Suppose we want to compare two models, $M_1$ and $M_0$, where $M_0$ is the simpler model. An intuitive thought on comparing two models by variational approximation methods is just to compare the lower bounds $\mathcal{L}(q_1)$ and $\mathcal{L}(q_0)$. However, we should note that by comparing the lower bounds, we are assuming that the KL divergences in the two approximations are the same, so that we can use just these lower bounds as guide. Unfortunately, it is not easy to measure how tight in theory any particular bound can be, if this can be accomplished we could then more accurately estimate the log marginal likelihood from the beginning. As clarified in [27], when comparing two exact log marginal likelihood, we have

$$\begin{array}{cc}\hfill log{p}_1(\mathbf{X})-log{p}_0(\mathbf{X})& =[\mathcal{L}(q_1)+KL(q_1\Vert p_1)]-[\mathcal{L}(q_0)-KL(q_0\Vert p_0)]\hfill \end{array}$$ (41)

$$\begin{array}{cc}\hfill & =\mathcal{L}(q_1)-\mathcal{L}(q_0)+[KL(q_1\Vert p_1)-KL(q_0\Vert p_0)]\hfill \end{array}$$ (42)

$$\begin{array}{cc}\hfill & \ne \mathcal{L}(q_1)-\mathcal{L}(q_0).\hfill \end{array}$$ (43)

The difference in log marginal likelihood, $log{p}_1(\mathbf{X})-log{p}_0(\mathbf{X})$, is the quantity we wish to estimate. However, if we base this on the lower bounds difference, we are basing our model comparison on () rather than (41). Therefore, there exists a systematic bias towards simpler model when comparing models if $KL(q_1\Vert p_1)-KL(q_0\Vert p_0)$ is not zero.

Realizing that we have a variational approximation for the posterior distribution of $\mathit{\beta }$, we propose the following method to estimate $p(\mathbf{X})$ based on our variational approximation $q_{\mathit{\beta }}(\mathit{\beta })$ (27). First, writing the marginal likelihood as

$$\begin{array}{c}\hfill p(\mathbf{x})=\int \left[\frac{p(\mathbf{x}|\mathit{\beta }){\pi }^I(\mathit{\beta })}{q_{\mathit{\beta }}(\mathit{\beta })}\right]q_{\mathit{\beta }}(\mathit{\beta })d\mathit{\beta },\end{array}$$

we can interpret it as the conditional expectation

$$\begin{array}{c}\hfill p(\mathbf{x})=\mathbb{E}\left[\frac{p(\mathbf{x}|\mathit{\beta }){\pi }^I(\mathit{\beta })}{q_{\mathit{\beta }}(\mathit{\beta })}\right]\end{array}$$

with respect to $q_{\mathit{\beta }}(\mathit{\beta })$. Next, draw samples ${\mathit{\beta }}^{(1)},...,{\mathit{\beta }}^{(n)}$ from $q_{\mathit{\beta }}(\mathit{\beta })$ and obtain the estimated marginal likelihood

$$\begin{array}{c}\hfill \widehat{p_{\mathbf{X}}(\mathbf{x})}=\frac{1}{n}\sum _{i=1}^n\frac{p(\mathbf{x}|{\mathit{\beta }}^{(i)}){\pi }^I({\mathit{\beta }}^{(i)})}{q_{\mathit{\beta }}({\mathit{\beta }}^{(i)})}.\end{array}$$

Please note that this method proposed is equivalent to importance sampling with importance function being $q_{\mathit{\beta }}(\mathit{\beta })$, for which we know the exact form and the generation of the random ${\mathit{\beta }}^{(i)}$ is easy and inexpensive.

6. Modeling Probability of Default Using Lending Club Data

6.1. Introduction

LendingClub (https://www.lendingclub.com/) is the world’s largest peer-to-peer lending platform. LendingClub enables borrowers to create unsecured personal loans between $1000 and $40,000. The standard loan period is three or five years. Investors can search and browse the loan listings on LendingClub website and select loans that they want to invest in based on the information supplied about the borrower, amount of loan, loan grade, and loan purpose. Investors make money from interest. LendingClub makes money by charging borrowers an origination fee and investors a service fee. To attract lenders, LendingClub publishes most of the information available in borrowers’ credit reports as well as information reported by borrowers for almost every loan issued through its website.

6.2. Modeling Probability of Default—Target Variable and Predictive Features

Publicly available LendingClub data, from 2007 June to 2018 Q4, has a total of 2,260,668 issued loans. Each loan has a status, either Paid-off, Charged-off, or Ongoing. We only adopted loans with an end status, i.e., either paid-off or charged-off. In addition, that loan status is the target variable. We then selected following loan features as our predictive covariates.

• Loan term in months (either 36 or 60)

• FICO

• Issued loan amount

• DTI (Debt to income ratio, i.e., customer’s total debt divided by income)

• Number of credit lines opened in past 24 months

• Employment length in years

• Annual income

• Home ownership type (own, mortgage, of rent)

We took a sample from the original data set that has customer yearly income between $15,000 and $60,000 and end up with a data set of 520,947 rows.

6.3. Addressing Uncertainty of Estimated Probit Model Using Variational Inference with Intrinsic Prior

Using the process developed in Section 5, we can update the intrinsic prior for parameters (see Figure 1) of the probit model using variational inference, and get the posterior distribution for the estimated parameters. Based on the derived parameter distributions, questions of interest may be explored with model uncertainty being considered.

Figure 1.

Intrinsic Prior.

Investors will be interested in understanding how each loan feature affect the probability of default, given a certain loan term, either 36 or 60. To answer this question, we samples 6000 cases from the original data set and draw from derived posterior distribution 100 times. We end up with $6000\times 100$ calculated probability of default, where each one of the 6000 samples yield 100 different probit estimates based on 100 different posterior draws. We summarize some of our findings in Figure 2, where color red representing 36 months loans and green representing 60 months loans.

Figure 2.

Effect of term months and other covariates on probability of default

• In general, 60 months loans have higher risk of default.

• Given loan term months, there is a clear trend showing that high FICO means lower risk.

• Given loan term months, there is a trend showing that high DTI indicating higher risk.

• Given loan term months, there is a trend showing that more credit lines opened in past 24 months indicating higher risk.

• There is no clear pattern regarding income. This is probably because we only included customers with income between $15,000 and $60,000 in our training data, which may not representing the true income level of the whole population.

Model uncertainty could also be measured through credible intervals. Again, with the derived posterior distribution, the credible interval is just the range containing a particular percentage of estimated effect/parameter values. For instance, the $95\%$ credible interval of the estimated parameter value of FICO is simply the central portion of the posterior distribution that contains $95\%$ of the estimated values. Contrary to the frequentist confidence intervals, Bayesian credible interval is much more straightforward to interpret. Using the Bayesian framework created in this article, from Figure 3, we can simply state that given the observed data, the estimated effect of DTI on default has $89\%$ probability of falling within $[8.300,8.875]$. Instead of the conventional $95\%$, we used $89\%$ following suggestions in [28,29], which is just as arbitrary as any of the conventions.

Figure 3.

Credible intervals for estimated coefficients

One of the main advantages of using variational inference over MCMC is that variational inference is much faster. Comparisons were made between the two approximation frameworks on a 64-bit Windows 10 laptop, with 32.0 GB RAM. Using the data set introduced in Section 6.2, we have that

• with a conjugate prior and following the Gibbs sampling scheme proposed by [17], it took 89.86 s to finish 100 simulations for the Gibbs sampler;

• following our method proposed in Section 5.2, it took 58.38 s to get the approximated posterior distribution and sampling 10,000 times from that posterior.

6.4. Model Comparison

Following the procedure proposed in Section 5.3, we compare the following series of nested models. From the data set introduced in Section 6.2, 2000 records were sampled to estimate the likelihood $p(\mathbf{x}|{\mathit{\beta }}^{(i)})$. Where ${\mathit{\beta }}^{(i)}$ is one of the 2500 draws sampled directly from the approximated posterior distribution $q_{\mathit{\beta }}(\mathit{\beta })$, which serves as the importance function used to estimate the marginal likelihood $p(\mathbf{x})$.

• $M_2$: FICO + Term 36 Indicator

• $M_3$: FICO + Term 36 Indicator + Loan Amount

• $M_4$: FICO + Term 36 Indicator + Loan Amount + Annual Income

• $M_5$: FICO + Term 36 Indicator + Loan Amount + Annual Income + Mortgage Indicator

Estimated log marginal likelihood for each model is plotted in Figure 4. We can see that the model evidence has increased by adding predictive features Loan Amount and Annual Income sequentially. However, if we further adding home ownership information, i.e., Mortgage Indicator as a predictive feature, the model evidence decreased. We have the Bayes factor

$$B{F}_{45}=\frac{p(\mathbf{x}|M_4)}{p(\mathbf{x}|M_5)}=e^{-1014.78-(-1016.42)}=5.16,$$

which suggests a substantial evidence for model $M_4$, indicating home ownership information may be irrelevant in predicting probability of default given that all the other predictive features are relevant.

Figure 4.

Log marginal likelihood comparison

7. Further Work

The authors thank the reviewers for pointing out that mean-field variational Bayes underestimates the posterior variance. This could be an interesting topic for our future research. We plan to study the $\mathit{linearresponsevariationalBayes}$ (LRVB) method proposed in [30] to see if it can be applied on the framework we proposed in this article. To see if we can get the approximated posterior variance close enough to the true variance using our proposed method, comparisons should be made between normal conjugate prior with the MCMC procedure, normal conjugate prior with LRVB, and intrinsic prior with LRVB.

Appendix A. Density Function

Suppose $X\sim N(\mu ,{\sigma }^2)$ has a normal distribution and lies within the interval $X\in (a,b),-\infty \le a<b\le \infty$. Then X conditional on $a<X<b$ has a truncated normal distribution. Its probability density function, f, for $a\le X<b$, is given by

$$\begin{array}{c}\hfill f(x|\mu ,\sigma ,a,b)=\frac{\frac{1}{\sigma }\varphi (\frac{x-\mu }{\sigma })}{\Phi (\frac{b-\mu }{\sigma })-\Phi (\frac{a-\mu }{\sigma })}\end{array}$$

and by $f=0$ otherwise. Here

$$\begin{array}{c}\hfill \varphi (\xi )=\frac{1}{\sqrt{2\pi }}exp(-\frac{1}{2}{\xi }^2)\end{array}$$

is the probability density function of the standard normal distribution and $\Phi (·)$ is its cumulative distribution function. If $b=\infty$, then $\Phi (\frac{b-\mu }{\sigma })=1,$ and similarly, if $a=-\infty$, then $\Phi (\frac{a-\mu }{\sigma })=0.$ And the cumulative density for the truncated normal distribution is

$$\begin{array}{c}\hfill F(x|\mu ,\sigma ,a,b)=\frac{\Phi (\xi )-\Phi (\alpha )}{Z},\end{array}$$

where $\xi =\frac{x-\mu }{\sigma }$ and $Z=\Phi (\beta )-\Phi (\alpha )$.

Appendix B. Moments and Entropy

Let $\alpha =\frac{a-\mu }{\sigma }$ and $\beta =\frac{b-\mu }{\sigma }$. For two-sided truncation:

$$\begin{array}{cc}\hfill & \mathbb{E}(X|a<X<b)=\mu +\sigma \frac{\varphi (\alpha )-\varphi (\beta )}{\Phi (\beta )-\Phi (\alpha )},\hfill \\ \hfill & Var(X|a<X<b)={\sigma }^2\left[1+\frac{\alpha \varphi (\alpha )-\beta \varphi (\beta )}{\Phi (\beta )-\Phi (\alpha )}-{\left(\frac{\varphi (\alpha )-\varphi (\beta )}{\Phi (\beta )-\Phi (\alpha )}\right)}^2\right].\hfill \end{array}$$

For one sided truncation (upper tail):

$$\begin{array}{cc}\hfill & \mathbb{E}(X|X>a)=\mu +\sigma \lambda (\alpha )\hfill \\ \hfill & Var(X|X>a)={\sigma }^2[1-\delta (\alpha )],\hfill \end{array}$$

where $\alpha =\frac{a-\mu }{\sigma },\lambda (\alpha )=\frac{\varphi (\alpha )}{1-\Phi (\alpha )}$ and $\delta (\alpha )=\lambda (\alpha )[\lambda (\alpha )-\alpha ]$.

For one sided truncation (lower tail):

$$\begin{array}{cc}\hfill & \mathbb{E}(X|X<b)=\mu -\sigma \frac{\varphi (\beta )}{\Phi (\beta )}\hfill \\ \hfill & Var(X|X<b)={\sigma }^2\left[1-\beta \frac{\varphi (\beta )}{\Phi (\beta )}-{\left(\frac{\varphi (\beta )}{\Phi (\beta )}\right)}^2\right].\hfill \end{array}$$

More generally, the moment generating function for truncated normal distribution is

$$\begin{array}{c}\hfill e^{\mu t+{\sigma }^2{t}^2/2}·\left[\frac{\Phi (\beta -\sigma t)-\Phi (\alpha -\sigma t)}{\Phi (\beta )-\Phi (\alpha )}\right].\end{array}$$

For a density $f(x)$ defined over a continuous variable, the entropy is given by

$$\begin{array}{c}\hfill H[x]=-\int f(x)logf(x)dx.\end{array}$$

And the entropy for a truncated normal density is

$$\begin{array}{c}\hfill log(\sqrt{2\pi e}\sigma Z)+\frac{\alpha \varphi (\alpha )-\beta \varphi (\beta )}{2Z}.\end{array}$$

Author Contributions

Methodology, A.L., L.P. and K.W.; software, A.L.; writing–original draft preparation, A.L., L.P. and K.W.; writing–review and editing, A.L. and L.P.; visualization, A.L. All authors have read and agreed to the published version of the manuscript.

Funding

The work of L.R.Pericchi was partially funded by NIH grants U54CA096300, P20GM103475 and R25MD010399.

Conflicts of Interest

The authors declare no conflict of interest.