Lognormal and Gamma Mixed Negative Binomial Regression

Abstract

In regression analysis of counts, a lack of simple and efficient algorithms for posterior computation has made Bayesian approaches appear unattractive and thus underdeveloped. We propose a lognormal and gamma mixed negative binomial (NB) regression model for counts, and present efficient closed-form Bayesian inference; unlike conventional Poisson models, the proposed approach has two free parameters to include two different kinds of random effects, and allows the incorporation of prior information, such as sparsity in the regression coefficients. By placing a gamma distribution prior on the NB dispersion parameter r, and connecting a lognormal distribution prior with the logit of the NB probability parameter p, efficient Gibbs sampling and variational Bayes inference are both developed. The closed-form updates are obtained by exploiting conditional conjugacy via both a compound Poisson representation and a Polya-Gamma distribution based data augmentation approach. The proposed Bayesian inference can be implemented routinely, while being easily generalizable to more complex settings involving multivariate dependence structures. The algorithms are illustrated using real examples.

1. Introduction

In numerous scientific studies, the response variable is a count y = 0, 1, 2, ⋯, which we wish to explain with a set of covariates x = [1, x₁, ⋯, x_P]^T as $\mathbb{E}[y\mid \mathit{x}]=g^{-1}\left({\mathit{x}}^T\beta \right)$, where β = [β₀, ⋯, β_P]^T are the regression coefficients and g is the canonical link function in generalized linear models (GLMs) (McCullagh & Nelder, 1989; Long, 1997; Cameron & Trivedi, 1998;Agresti, 2002; Winkelmann, 2008). Regression models for counts are usually nonlinear and have to take into consideration the specific properties of counts, including discreteness and nonnegativity, and often characterized by overdispersion (variance greater than the mean). In addition, we may wish to impose a sparse prior in the regression coefficients for counts, which is demonstrated to be beneficial for regression analysis of both Gaussian and binary data (Tipping, 2001).

Count data are commonly modeled with the Poisson distribution y ~ Pois(λ), whose mean and variance are both equal to λ. Due to heterogeneity (difference between individuals) and contagion (dependence between the occurrence of events), the varance is often much larger than the mean, making the Poisson assumption restrictive. By placing a gamma distribution prior with shape r and scale p/(1 – p) on λ, a negative binomial (NB) distribution y ~ NB(r, p) can be generated as $f_Y\left(y\right)={\int }_0^{\infty }$ Pois(y; λ)Gamma $\left(\lambda ;r,\frac{p}{1-p}\right)d\lambda =\frac{\Gamma (r+y)}{y!\Gamma \left(r\right)}{(1-p)}^r{p}^y$, where Γ(·) denotes the gamma function, r is the nonnegative dispersion parameter and p is a probability parameter. Therefore, the NB distribution is also known as the gamma-Poisson distribution. It has a variance rp/(1 – p)² larger than the mean rp/(1 – p), and thus it is usually favored over the Poisson distribution for modeling overdispersed counts.

The regression analysis of counts is commonly performed under the Poisson or NB likelihoods, whose parameters are usually estimated by finding the maximum of the nonlinear log likelihood (Long, 1997;Cameron & Trivedi, 1998; Agresti, 2002; Winkelmann,2008). The maximum likelihood estimator (MLE), however, only provides a point estimate and does not allow the incorporation of prior information, such as sparsity in the regression coefficients. In addition, the MLE of the NB dispersion parameter r often lacks robustness and may be severely biased or even fail to converge if the sample size is small, the mean is small or if r is large (Saha & Paul, 2005; Lloyd-Smith, 2007).

Compared to the MLE, Bayesian approaches are able to model the uncertainty of estimation and to incorporate prior information. In regression analysis of counts, however, the lack of simple and efficient algorithms for posterior computation has seriously limited routine applications of Bayesian approaches, making Bayesian analysis of counts appear unattractive and thus underdeveloped. For instance, for the NB dispersion parameter r, the only available closed-form Bayesian solution relies on approximating the ratio of two gamma functions using a polynomial expansion (Bradlow et al., 2002); and for the regression coefficients β, Bayesian solutions usually involve computationally intensive Metropolis-Hastings algorithms, since the conjugate prior for β is not known under the Poisson and NB likelihoods (Chib et al., 1998; Chib & Winkelmann, 2001; Winkelmann, 2008).

In this paper we propose a lognormal and gamma mixed NB regression model for counts, with default Bayesian analysis presented based on two novel data augmentation approaches. Specifically, we show that the gamma distribution is the conjugate prior to the NB dispersion parameter r, under the compound Poisson representation, with efficient Gibbs sampling and variational Bayes (VB) inference derived by exploiting conditional conjugacy. Further we show that a lognormal prior can be connected to the logit of the NB probability parameter p, with efficient Gibbs sampling and VB inference developed for the regression coefficients β and the lognormal variance parameter σ², by generalizing a Polya-Gamma distribution based data augmentation approach in Polson & Scott (2011). The proposed Bayesian inference can be implemented routinely, while being easily generalizable to more complex settings involving multivariate dependence structures. We illustrate the algorithm with real examples on univariate count analysis and count regression, and demonstrate the advantages of the proposed Bayesian approaches over conventional count models.

2. Regression Models for Counts

The most basic regression model for counts is the Poisson regression model (Long, 1997; Cameron & Trivedi,1998; Winkelmann, 2008), which can be expressed as

$$y_i\sim \text{Pois}\left({\lambda }_i\right),{\lambda }_i=\exp \left({\mathit{x}}_i^T\beta \right)$$ (1)

where x_i = [1, x_i1, ⋯, x_iP]T is the covariate vector for sample i. The Newton-Raphson method can be used to iteratively find the MLE of β (Long, 1997). A serious constraint of the Poisson regression model is that it assumes equal-dispersion, i.e., $\mathbb{E}[y_i\mid {\mathit{x}}_i]=\mathrm{Var}[y_i\mid {\mathit{x}}_i]=$ = exp(${\mathit{x}}_i^T\beta$) In practice, however, count data are often overdispersed, due to heterogeneity and contagion (Winkelmann, 2008). To model overdispersed counts, the Poisson regression model can be modified as

$$y_i\sim \text{Pois}\left({\lambda }_i\right),{\lambda }_i=\exp \left({\mathit{x}}_i^T\beta \right){∊}_i$$ (2)

where ∊_i is a nonnegative multiplicative random-effect term to model individual heterogeneity (Winkelmann, 2008). Using both the law of total expectation and the law of total variance, it can be shown that

$$\mathbb{E}[y_i\mid {\mathit{x}}_i]=\exp \left({\mathit{x}}_i^T\beta \right)\mathbb{E}\left[{∊}_i\right]$$ (3)

$$\mathrm{Var}[y_i\mid {\mathit{x}}_i]=\mathbb{E}[y_i\mid {\mathit{x}}_i]+\frac{\mathrm{Var}\left[{∊}_i\right]}{{\mathbb{E}}^2\left[{∊}_i\right]}{\mathbb{E}}^2[y_i\mid {\mathit{x}}_i].$$ (4)

Thus Var$[y_i\mid {\mathit{x}}_i]\ge \mathbb{E}[y_i\mid {\mathit{x}}_i]$ and we obtain a regression model for overdispersed counts. We show below that both the gamma and lognormal distributions can be used as the nonnegative prior on ∊_i.

2.1. The Negative Binomial Regression Model

The NB regression model (Long, 1997; Cameron & Trivedi, 1998; Winkelmann, 2008; Hilbe, 2007) is constructed by placing a gamma prior on ∊_i as

$${∊}_i\sim \text{Gamma}(r,1∕r)=\frac{r^r}{\Gamma \left(r\right)}{{∊}_i}^{r-1}{e}^{-r{∊}_i}$$ (5)

where $\mathbb{E}\left[{∊}_i\right]=1$ and Var[∊_i] = r⁻¹. Marginalizing out ∊_i in (2), we have a NB distribution parameterized by mean ${\mu }_i=\exp \left({\mathit{x}}_i^T\beta \right)$ and inverse dispersion parameter ϕ (the reciprocal of r) as $f_Y\left(y_i\right)=\frac{\Gamma ({\varphi }^{-1}+y_i)}{y_i!\Gamma (\varphi -1)}{\left(\frac{{\varphi }^{-1}}{{\varphi }^{-1}+{\mu }_i}\right)}^{{\varphi }^{-1}}{\left(\frac{{\mu }_i}{{\varphi }^{-1}+{\mu }_i}\right)}^{y_i}$, thus

$$\mathbb{E}[y_i\mid {\mathit{x}}_i]=\exp \left({\mathit{x}}_i^T\beta \right)$$ (6)

$$\mathrm{Var}[y_i\mid {\mathit{x}}_i]=\mathbb{E}[y_i\mid {\mathit{x}}_i]+\varphi {\mathbb{E}}^2[y_i\mid {\mathit{x}}_i].$$ (7)

The MLEs of β and ϕ can be found numerically with the Newton-Raphson method (Lawless, 1987).

2.2. The Lognormal-Poisson Regression Model

A lognormal-Poisson regression model (Breslow, 1984;Long, 1997; Agresti, 2002; Winkelmann, 2008) can be constructed by placing a lognormal prior on ∊_i as

$${∊}_i\sim \ln \mathcal{N}(0,{\sigma }^2)$$ (8)

where $\mathbb{E}\left[{∊}_i\right]=e^{{\sigma }^2∕2}$ and $\mathrm{Var}\left[{∊}_i\right]=e^{{\sigma }^2}\left(e^{{\sigma }^2}-1\right)$. Using (3) and (4), we have

$$\mathbb{E}[y_i\mid {\mathit{x}}_i]=\exp ({\mathit{x}}_i^T\beta +{\sigma }^2∕2)$$ (9)

$$\mathrm{Var}[y_i\mid {\mathit{x}}_i]=\mathbb{E}[y_i\mid {\mathit{x}}_i]+\left(e^{{\sigma }^2}-1\right){\mathbb{E}}^2[y_i\mid {\mathit{x}}_i].$$ (10)

Compared to the NB model, there is no analytical form for the distribution of y_i if ∊_i is marginalized out and the MLE is less straightforward to calculate, making it less commonly used. However, Winkelmann (2008) suggests to reevaluate the lognormal-Poisson model, since it is appealing in theory and may fit the data better. The inverse Gaussian distribution prior can also be placed on ∊_i to construct a heavier-tailed alternative to the NB model (Dean et al., 1989), whose density functions are shown to be virtually identical to the lognormal-Poisson model (Winkelmann, 2008).

3. The Lognormal and Gamma Mixed Negative Binomial Regression Model

To explicitly model the uncertainty of estimation and incorporate prior information, Bayesian approaches appear attractive. Bayesian analysis of counts, however, is seriously limited by the lack of efficient inference, as the conjugate prior for the regression coefficients β is unknown under the Poisson and NB likelihoods (Winkelmann, 2008), and the conjugate prior for the NB dispersion parameter r is also unknown.

To address these issues, we propose a lognormal and gamma mixed NB regression model for counts, termed here the LGNB model, where a lognormal prior ln $\mathcal{N}$(0, σ²) is placed on the multiplicative random effect term ∊_i and a gamma prior is placed on r. Denoting $p_i=\frac{e^{{\psi }_i}}{1+e^{{\psi }_i}}=\frac{\exp \left({\mathit{x}}_i^T\beta \right){∊}_i}{1+\exp \left({\mathit{x}}_i^T\beta \right){∊}_i}$, and logit(p_i) = ln $\frac{p_i}{1-p_i}$, the LGNB model is constructed as

$$y_i\sim \mathrm{NB}(r,p_i),{\psi }_i=\text{logit}\left(p_i\right)={\mathit{x}}_i^T\beta +\ln {∊}_i$$ (11)

$${∊}_i\sim \ln \mathcal{N}(0,{\phi }^{-1}),\phi \sim \text{Gamma}(e_0,1∕f_0)$$ (12)

$$\beta \sim \prod _{p=0}^P\mathcal{N}(0,{\alpha }_p^{-1}),{\alpha }_p\sim \text{Gamma}(c_0,1∕d_0)$$ (13)

$$r\sim \text{Gamma}(a_0,1∕h),h\sim \text{Gamma}(b_0,1∕g_0)$$ (14)

where φ = σ⁻² and a₀, b₀, c₀, d₀, e₀, f₀ and g₀ are gamma hyperparameters (they are set as 0.01 in experiments). Since y_i ~ NB (r, p_i) in (11) can be augmented into a gamma-Poisson structure as y_i ~ Pois(λ_i), λ_i ~ Gamma $(r,\exp ({\mathit{x}}_i^T\beta \left){∊}_i\right)$, the LGNB model can also be considered as a lognormal-gamma-gamma-Poisson regression model. Denoting $\mathbf{X}={[{\mathit{x}}_1^T,\cdots ,{\mathit{x}}_N^T]}^T$, we may equivalently express ψ = [ψ₁, ⋯, ψ_N]^T in the above model as

$$\psi \sim \mathcal{N}(\mathbf{X}\beta ,{\phi }^{-1}\mathbf{I}).$$ (15)

If we marginalize out h in (14), we obtain a beta prime distribution prior r ~ β’(a₀, b₀, 1, g₀). If we marginalize out α_p in (13), we obtain a Student-t prior for β_p, the sparsity-promoting prior used in Tipping (2001);Bishop & Tipping (2000) for regression analysis of both Gaussian and binary data. Note that β is connected to p_i with a logit link, which is key to deriving efficient Bayesian inference.

3.1. Model Properties and Model Comparison

Using the laws of total expectation and total variance and the moments of the NB distribution, we have

$$\begin{array}{cc}\hfill \mathbb{E}[y_i\mid {\mathit{x}}_i]=& {\mathbb{E}}_{{∊}_i}\left[\mathbb{E}\right[y_i\mid {\mathit{x}}_i,{∊}_i\left]\right]=\exp ({\mathit{x}}_i^T\beta +{\sigma }^2∕2+\ln r)\hfill \\ \hfill \mathrm{Var}[y_i\mid {\mathit{x}}_i]& ={\mathbb{E}}_{{∊}_i}\left[\mathrm{Var}\right[y_i\mid {\mathit{x}}_i,{∊}_i\left]\right]+{\mathrm{Var}}_{{∊}_i}\left[\mathbb{E}\right[y_i\mid {\mathit{x}}_i,{∊}_i\left]\right]\hfill \end{array}$$ (16)

$$=\mathbb{E}[y_i\mid {\mathit{x}}_i]+\left(e^{{\sigma }^2}(1+r^{-1})-1\right){\mathbb{E}}^2[y_i\mid {\mathit{x}}_i].$$ (17)

We define the quasi-dispersion κ as the coefficient associated with the mean quadratic term in the variance. As shown in (7) and (10), κ = ϕ in the NB model and $\kappa =\left(e^{{\sigma }^2}-1\right)$ in the lognormal-Poisson model. Apparently, they have different distribution assumptions on dispersion, yet there is no clear evidence to favor one over the other in terms of goodness of fit. In the proposed LGNB model, there are two free parameters r and σ² to adjust both the mean in (16) and dispersion $\kappa =\left(e^{{\sigma }^2}(1+r^{-1})-1\right)$, which become the same as those of the NB model when σ² = 0, and the same as those of the lognormal-Poisson model when ϕ = r⁻¹ = 0. Thus the LGNB model has one extra degree of freedom to incorporate both kinds of random effects, with their proportion automatically inferred.

4. Default Bayesian Analysis Using Data Augmentation

As discussed in Section 3, the LGNB model has an advantage of having two free parameters to incorporate both kinds of random effects. We show below that it has an additional advantage in that default Bayesian analysis can be performed with two novel data augmentation approaches, with closed-form solutions and analytical update equations available for both Gibbs sampling and VB inference. One augmentation approach concerns the inference of the NB dispersion parameter r using the compound Poisson representation, and the other concerns the inference of the regression coefficients β using the Polya-Gamma distribution.

4.1. Inferring the Dispersion Parameter Under the Compound Poisson Representation

We first focus on inference of the NB dispersion parameter r and assume we know {p_i}_i=1,N and h, neglecting the remaining part of the LGNB model at this moment. We comment here that the novel Bayesian inference developed here can be applied to any other scenarios where the conditional posterior of r is proportional to ${\prod }_{i=1}^N\mathrm{NB}(y_i;r,p_i)$Gamma(r; a₀, 1/h), for which a hybrid Monte Carlo and a Metropolis-Hastings algorithms had been developed in Williamson et al. (2010) and Zhou et al. (2012), but VB solutions were not yet developed.

As proved in Quenouille (1949), y ~ NB(r, p) can also be generated from a compound Poisson distribution as

$$y=\sum _{l=1}^L{u}_l,L\sim \text{Pois}(-r\ln (1-p\left)\right),u_l\stackrel{\mathit{iid}}{\sim }\text{Log}\left(p\right)$$ (18)

where Log(p) corresponds to the logarithmic distribution (Barndor-Nielsen et al., 2010) with f_U(k) = −p^k/[k ln(1 – p)], k ∈ {1, 2, …}, whose probability-generating function (PGF) is

$$G_U\left(z\right)=\ln (1-\mathit{pz})∕\ln (1-p),\mid z\mid <p^{-1}.$$ (19)

Using the conjugacy between the gamma and Poisson distributions, it is evident that the gamma distribution is the conjugate prior for r under this augmentation.

4.1.1. Gibbs Sampling for r

Recalling (18), y_i ~ NB(r, p_i) can also be generated from the random sum $y_i={\sum }_{l=1}^{L_i}{u}_{\mathit{il}}$ with

$$u_{\mathit{il}}\stackrel{\mathit{iid}}{\sim }\text{Log}\left(p_i\right),L_i\sim \text{Pois}(-r\ln (1-p_i\left)\right).$$ (20)

Exploiting conjugacy between (14) and (20), given L_i, we have the conditional posterior of r as

$$(r\mid -)\sim \text{Gamma}\left(a_0+\sum _{i=1}^N{L}_i,\frac{1}{h-{\Sigma }_{i=1}^N\ln (1-p_i)}\right)$$ (21)

where here and below expressions like (r|−) correspond to random variable (RV) r, conditioned on all other RVs. The remaining challenge is finding the conditional posterior of L_i. Denote $w_{\mathit{ij}}={\sum }_{l=1}^j{u}_{\mathit{il}}$, j = 1, ⋯ , y_i. Since w_ij is the summation of j iid Log(p_i) distributed RVs, using (19), the PGF of w_ij is

$$G_{W_{\mathit{ij}}}\left(z\right)={\left[\ln \right(1-p_{i}z)∕\ln (1-p_i\left)\right]}^j,\mid z\mid <p_i^{-1}.$$

Therefore, we have L_i ≡ 0 if y_i = 0 and for 1 ≤ j ≤ y_i

$$\begin{array}{cc}\hfill \mathrm{Pr}& (L_i=j\mid -)\propto \mathrm{Pr}(w_{\mathit{ij}}=y_i)\text{Pois}(j;-r\ln (1-p_i\left)\right)\hfill \\ \hfill =& \frac{G_{W_{\mathit{ij}}}^{\left(y_i\right)}\left(0\right)}{y_i!}\text{Pois}(j;-r\ln (1-p_i\left)\right)\hfill \\ \hfill =& {\phantom{\mid }\frac{d^{y_i}}{{\mathit{dz}}^{y_i}}{f}_i^j\left(z\right)\mid }_{z=0}\frac{r^j}{j!y_i!}\exp \left(r\ln \right(1-p_i\left)\right)\hfill \\ \hfill =& F(y_i,j)r^j{p}_i^{y_i}\exp \left(r\ln \right(1-p_i\left)\right)\hfill \end{array}$$ (22)

where f_i(z) = − ln(1–p_iz) and F is a lower triangular matrix with F (1, 1) = 1, F(m, j) = 0 if j > m, and

$$F(m,j)=\frac{m-1}{m}F(m-1,j)+\frac{1}{m}F(m-1,j-1)$$ (23)

if 1 ≤ j ≤ m. Using (22), we have

$$\mathrm{Pr}(L_i=j\mid -)=R_r(y_i,j),j=0,\cdots ,y_i.$$ (24)

where R_r(0, 0) = 1 and

$$R_r(m,j)=F(m,j)r^j/\sum _{j^{\prime }=1}^m(m,j^{\prime })r^{j^{\prime }}.$$ (25)

The values of F can be iteratively calculated and each row sums to one, e.g., the 4th and 5th rows of F are

$$\left(\begin{array}{ccccccc}\hfill 6∕4!\hfill & \hfill 11∕4!\hfill & \hfill 6∕4!\hfill & \hfill 1∕4!\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill \cdots \hfill \\ \hfill 24∕5!\hfill & \hfill 50∕5!\hfill & \hfill 35∕5!\hfill & \hfill 10∕5!\hfill & \hfill 1∕5!\hfill & \hfill 0\hfill & \hfill \cdots \hfill \end{array}\right).$$

Note that to obtain (22), we use the relationship proved in Lemma 1 of the supplementary material that

$$\frac{1}{m!}{\phantom{\mid }\frac{d^m}{{\mathit{dz}}^m}{f}_i^j\left(z\right)\mid }_{z=0}=F(m,j)j!p_i^m,1\le j\le m.$$ (26)

Gibbs sampling for r proceeds by alternately sampling (24) and (21). Note that to ensure numerical stability when r > 1, instead of using (25), we may iteratively calculate R_r in the way we calculate F in (23). We show in Figure 1 of the supplementary material the matrices R_r for r = .1, 1, 10 and 100.

Figure 1.

a) The histogram and (b) autocorrelation of collected Gibbs samples of the NB dispersion parameter r. (c) The inferred probability density function of r using VB. (d) The VB lower bound. Note that the calculated lower bound shows variations after convergence, which is expected since the Monte Carlo integration is used to calculate non-analytical expectation terms such as ⟨ln Γ(r+y_i)⟩.

4.1.2. Variational Bayes Inference for r

Using VB inference (Bishop & Tipping, 2000; Beal,2003), we approximate the posterior p(r, L|X) with Q(r, L) = $Q_r\left(r\right){\prod }_{i=1}^N{Q}_{L_i}\left(L_i\right)$, and we have

$$Q_{L_i}\left(L_i\right)=\sum _{j=0}^{y_i}{R}_{\tilde{r}}(y_i,j){\delta }_j$$ (27)

$$Q_r\left(r\right)=\text{Gamma}(\tilde{a},1∕\tilde{h}).$$ (28)

where ⟨x⟩ = $\mathbb{E}\left[x\right]$, $\tilde{r}$ = exp (⟨ln r⟩), ψ(x) is the digamma function, and

$$\langle \ln r\rangle =\psi \left(\tilde{a}\right)-\ln \tilde{h},\langle L_i\rangle =\sum _{j=1}^{y_i}{R}_{\tilde{r}}(y_i,j)j$$ (29)

$$\tilde{a}=a_0+\sum _{i=1}^N\langle L_i\rangle ,\tilde{h}=h-\sum _{i=1}^N\langle \ln (1-p_i)\rangle .$$ (30)

Equations (29)-(30) constitute the VB inference for the NB dispersion parameter r, with ⟨r⟩ = $\tilde{a}∕\tilde{h}$.

4.2. Inferring the Regression Coefficients Using the Polya-Gamma Distribution

Denote ω_i as a random variable drawn from the Polya-Gamma (PG) distribution (Polson & Scott, 2011) as

$${\omega }_i\sim \mathrm{PG}(y_i+r,0).$$ (31)

We have ${\mathbb{E}}_{{\omega }_i}\left[\exp (-{\omega }_i{\psi }_i^2∕2)\right]={\text{cosh}}^{-(y_i+r)}({\psi }_i∕2)$. Thus the likelihood of ψ_i in (11) can be expressed as

$$\begin{array}{cc}\hfill \mathcal{L}\left({\psi }_i\right)& \propto \frac{{\left(e^{{\psi }_i}\right)}^{y_i}}{{(1+e^{{\psi }_i})}^{y_i+r}}=\frac{2^{-(y_i+r)}\exp \left(\frac{y_i-r}{2}{\psi }_i\right)}{{\text{cosh}}^{y_i+r}({\psi }_i∕2)}\hfill \\ \hfill & \propto \exp \left(\frac{y_i-r}{2}{\psi }_i\right){\mathbb{E}}_{{\omega }_i}\left[\exp (-{\omega }_i{\psi }_i^2∕2)\right].\hfill \end{array}$$ (32)

Given the values of {ω_i}_i=1,N and the prior in (15), the conditional posterior of ψ can be expressed as

$$(\psi \mid -)\propto \mathcal{N}(\psi ;\mathbf{X}\beta ,{\phi }^{-1}\mathbf{I})\prod _{i=1}^N{e}^{-\frac{{\omega }_i}{2}}{\left({\psi }_i-\frac{y_i-r}{2{\omega }_i}\right)}^2$$ (33)

and given the values of ψ and the prior in (31), the conditional posterior of ω_i can be expressed as

$$({\omega }_i\mid -)\propto \exp (-{\omega }_i{\psi }_i^2∕2)\mathrm{PG}({\omega }_i;y_i+r,0).$$ (34)

4.3. Gibbs Sampling Inference

Exploiting conditional conjugacy and the exponential tilting of the PG distribution in Polson & Scott (2011), we can sample in closed-form all latent parameters of the LGNB model described from (11) to (14) as

$$\text{Sampling}{L}_i\text{with}\left(24\right),\text{Sampling}r\text{with}\left(21\right)$$ (35)

$$({\omega }_i\mid -)\sim \mathrm{PG}(y_i+r,{\psi }_i)$$ (36)

$$(\psi \mid -)\sim \mathcal{N}(\mu ,\Sigma ),(\beta \mid -)\sim \mathcal{N}({\mu }_{\beta },{\Sigma }_{\beta })$$ (37)

$$(h\mid -)\sim \text{Gamma}(a_0+b_0,1∕(g_0+r\left)\right)$$ (38)

$$(\phi \mid -)\sim \text{Gamma}\left(e_0+\frac{N}{2},\frac{1}{f_0+{\Vert \psi -\mathbf{X}\beta \Vert }_2^2∕2}\right)$$ (39)

$$({\alpha }_p\mid -)\sim \text{Gamma}(c_0+1∕2,1∕(d_0+{\beta }_p^2∕2\left)\right)$$ (40)

where Ω = diag(ω₁⋯, ω_N), A = diag(α₀, ⋯ , α_P), y = [y₁, ⋯ , y_N]^T, Σ= (φI + Ω)⁻¹, μ = Σ[(y – r)/2 + φXβ], Σ_β= (φX^TX + A)⁻¹ and μ_β = φΣ_βX^Tψ. Note that a PG distributed random variable can be generated from an infinite sum of weighted iid gamma random variables (Devroye, 2009; Polson & Scott, 2011). We provide in the supplementary material a method for accurately truncating the infinite sum.

4.4. Variational Bayes Inference

Using VB inference (Bishop & Tipping, 2000; Beal,2003), we approximate the posterior distribution with $Q=Q_{\psi }\left(\psi \right)Q_{\beta }\left(\beta \right)Q_r\left(r\right)Q_h\left(h\right)Q_{\phi }\left(\phi \right){\prod }_{p=0}^P{Q}_{{\alpha }_p}\left({\alpha }_p\right){\prod }_{i=1}^N\left[Q_{L_i}\right(L_i\left)Q_{{\omega }_i}\right({\omega }_i\left)\right]$. To exploit conjugacy, defining Q_Li(L_i) as in (27), Q_r(r) as in (28), Q_ωi(ω_i) = PG(γ_i1, γ_i2), $Q_{\psi }\left(\psi \right)=\mathcal{N}(\tilde{\mu },\tilde{\Sigma })$, $Q_{\beta }\left(\beta \right)=\mathcal{N}({\tilde{\mu }}_{\beta },{\tilde{\Sigma }}_{\beta })$, Q_h(h) = Gamma($\tilde{b},1∕\tilde{g}$), Q_φ(φ) = Gamma($\tilde{e},1∕\tilde{f}$) and Q_αp(α_p) = Gamma(${\tilde{c}}_p,1∕{\tilde{d}}_p$), we have

$$\tilde{a}=a_0+\sum _{i=1}^N\langle L_i\rangle ,\tilde{h}=\langle h\rangle +\sum _{i=1}^N\langle \ln (1+e^{{\psi }_i})\rangle$$ (41)

$$\tilde{\Sigma }={(\langle \phi \rangle \mathbf{I}+\tilde{\Omega })}^{-1},\tilde{\mu }=\tilde{\Sigma }\left[\right(\mathit{y}-\langle r\rangle )∕2+\langle \phi \rangle \mathbf{X}\beta ]$$ (42)

$${\tilde{\Sigma }}_{\beta }={(\langle \phi \rangle {\mathbf{X}}^T\mathbf{X}+\langle \tilde{\mathbf{A}}\rangle )}^{-1},{\tilde{\mu }}_{\beta }=\langle \phi \rangle {\tilde{\Sigma }}_{\beta }{\mathbf{X}}^T\langle \psi \rangle$$ (43)

$$\tilde{b}=a_0+b_0,\tilde{g}=\langle r\rangle +g_0,\tilde{e}=e_0+N∕2$$ (44)

$$\tilde{f}=f_0+\frac{\langle {\psi }^T\psi \rangle }{2}-\langle {\psi \rangle }^T\mathbf{X}\langle \beta \rangle +\frac{\mathrm{tr}[\mathbf{X}\langle \beta {\beta }^T\rangle {\mathbf{X}}^T]}{2}$$ (45)

$${\tilde{c}}_p=c_0+1∕2,{\tilde{d}}_p=d_0+\langle {\beta }_p^2\rangle ∕2$$ (46)

where $\tilde{\Omega }$ = diag(⟨ω₁⟩, ⋯, ⟨ω_N⟩), $\tilde{\mathbf{A}}$ = diag(${\tilde{\alpha }}_0,\cdots ,{\tilde{\alpha }}_P$), tr[Σ] is the trace of Σ, ⟨ln r⟩ and ⟨L_i⟩ are calculated as in (29), ⟨r⟩ = $\tilde{a}∕\tilde{h}$, ⟨ψ^Tψ⟩ = ${\tilde{\mu }}^T\tilde{\mu }+\mathrm{tr}\left[\tilde{\Sigma }\right]$, ⟨β⟩ = ${\tilde{\mu }}_{\beta }$, ⟨β^Tβ⟩ = ${\tilde{\mu }}_{\beta }^T{\tilde{\mu }}_{\beta }+\mathrm{tr}\left[{\tilde{\Sigma }}_{\beta }\right]$, ⟨ββ^T⟩ = ${\tilde{\mu }}_{\beta }{\tilde{\mu }}_{\beta }^T+{\tilde{\Sigma }}_{\beta }$, ⟨h⟩ = $\tilde{b}∕\tilde{g}$, ⟨α⟩ = $\tilde{e}∕\tilde{f}$ and ⟨α_p⟩ = ${\tilde{c}}_p∕{\tilde{d}}_p$. Although we do not have analytical forms for γ_i1 and γ_i2 in Q_ωi(ω_i), we can use (36) to calculate ⟨ω_i⟩ as

$$\langle {\omega }_i\rangle ={\mathbb{E}}_{r,{\psi }_i}\left[\mathbb{E}\right[{\omega }_i\mid r,{\psi }_i,y_i\left]\right]=(y_i+\langle r\rangle )\langle \frac{\tanh ({\psi }_i∕2)}{2{\psi }_i}\rangle$$ (47)

where the mean property of the PG distribution¹ (Polson & Scott, 2011 is applied. To calculate ⟨ln(1+e^ψ_i)⟩ in (41) and $\langle \frac{\tanh ({\psi }_i∕2)}{2{\psi }_i}\rangle$ in (47), we use the Monte Carlo integration algorithm (Andrieu et al., 2003).

5. Example Results

5.1. Univariate Count Data Analysis

The inference of the NB dispersion parameter r by itself plays an important role not only for the NB regression (Lawless, 1987; Winkelmann, 2008) but also for univariate count data analysis (Bliss & Fisher, 1953;Clark & Perry, 1989; Saha & Paul, 2005; Lloyd-Smith,2007), and it also arises in some recently proposed latent variable models for count matrix factorization (Williamson et al., 2010; Zhou et al., 2012). Thus it is of interest to evaluate the proposed closed-form Gibbs sampling and VB inference for this parameter alone, before introducing the regression analysis part.

We consider a real dataset describing counts of red mites on apple leaves, given in Table 1 of Bliss & Fisher (1953). There were in total 172 adult female mites found in 150 randomly selected leaves, with a 0 count on 70 leaves, 1 on 38, 2 on 17, 3 on 10, 4 on 9, 5 on 3, 6 on 2 and 7 on 1. This dataset has a mean of 1.1467 and a variance of 2.2736, clearly overdispersed. We assume the counts are NB distributed and we intend to infer r with a hierarchical model as

$$y_i\stackrel{\mathit{iid}}{\sim }\mathrm{NB}(r,p),r\sim \text{Gamma}(a,1∕b),p\sim \text{Beta}(\alpha ,\beta )$$

where i = 1; ⋯; N and we set a = b = α = β = 0:01. We consider 20,000 Gibbs sampling iterations, with the first 10,000 samples discarded and every fifth sample collected afterwards. As shown in Figure 1, the autocorrelation of Gibbs samples decreases quickly as the lag increases, and the VB lower bound converges quickly even starting from a bad initialization (r is initialized two times the converged value).

Table 1.

The MLEs or posterior means of the lognormal variance parameter σ², NB dispersion parameter r, quasi-dispersion κ and regression coefficients β for the Poisson, NB and LGNB regression models on the NASCAR dataset, using the MLE, VB or Gibbs sampling for parameter estimations.

Model	Poisson	NB	LGNB	LGNB
Parameters	(MLE)	(MLE)	(VB)	(Gibbs)
σ 2	N/A	N/A	0.1396	0.0289
r	N/A	5.2484	18.5825	6.0420
β 0	−0.4903	−0.5038	−3.5271	−2.1680
β1 (Laps)	0.0021	0.0017	0.0015	0.0013
β2 (Drivers)	0.0516	0.0597	0.0674	0.0643
β3 (TrkLen)	0.6104	0.5153	0.4192	0.4200

The estimated posterior mean of r is 1.0812 with Gibbs sampling and 0.9988 with VB. Compared to the method of moments estimator (MME), MLE, and maximum quasi-likelihood estimator (MQLE) (Clark & Perry, 1989), which provides point estimates of 1.1667, 1.0246 and 0.9947², respectively, our algorithm is able to provide a full posterior distribution of r and is convenient to incorporate prior information. The calculating details of the MME, MLE and MQLE, the closed-form Gibbs sampling and VB update equations, and the VB lower bound are all provided in the supplementary material, omitted here for brevity.

5.2. Regression Analysis of Counts

We test the full LGNB model on two real examples, with comparison to the Poisson, NB, lognormal-Poisson and inverse-Gaussian-Poisson (IG-Poisson) regression models. The NASCAR dataset³, analyzed in Winner, consists of 151 NASCAR races during the 1975-1979 Seasons. The response variable is the number of lead changes in a race, and the covariates of a race include the number of laps, number of drivers and length of the track (in miles). The MotorIns dataset⁴, analyzed in Dean et al. (1989), consists of Swedish third-party motor insurance claims in 1977. Included in the data are the total number of claims for automobiles insured in each of the 315 risk groups, defined by a combination of DISTANCE, BONUS, and MAKE factor levels. The number of insured automobile-years for each group is also given. As in Dean et al. (1989), a 19 dimensional covariate vector is constructed for each group to represent levels of the factors. To test goodness-of-fit, we use the Pearson residuals, a metric widely used in GLMs (McCullagh & Nelder, 1989), calculated as

$$E=\sum _{i=1}^N{e}_i^2,e_i=\frac{y_i-{\widehat{\mu }}_i}{\sqrt{{\widehat{\mu }}_i(1+\widehat{\kappa }{\widehat{\mu }}_i)}}$$ (48)

where $\widehat{\mu }$ and $\widehat{\kappa }$ are the estimated mean and quasi-dispersion, respectively, whose calculations are described in detail in the supplementary material.

The MLEs for the Poisson and NB models are well-known and the update equations can be found in Winner; Winkelmann (2008). The MLE results for the IG-Poisson model on the MotorIns data were reported inDean et al. (1989). For the lognormal-Poisson model, no standard MLE algorithms are available and we choose Metropolis-Hastings (M-H) algorithms for parameter estimation. We also consider a LGNB model under the special setting that r = 1000. As discussed in Section 3.1, this would lead to a model which is approximately the lognormal-Poisson model, yet with closed-form Gibbs sampling inference. We use both VB and Gibbs sampling for the LGNB model. We consider 20,000 Gibbs sampling iterations, with the first 10,000 samples discarded and every fifth sample collected afterwards. As described in the supplementary material, we sample from the PG distribution with a truncation level of 2000. We initialize r as 100 and other parameters at random. Examining the samples in Gibbs sampling, we find that the autocorrelations of model parameters generally reduce to below 0.2 at the lag of 20, indicating fast mixing.

Shown in Table 1 are the MLEs or posterior means of key model parameters. Note that β₀ of the LGNB model differs considerably from that of the Poisson and NB models, which is expected since β₀ + σ²/2 + ln r in the LGNB model plays about the same role as β₀ in the Poisson and NB models, as indicated in (16).

As shown in Tables 2, in terms of goodness of fit measured by Pearson residuals, the Poisson model performs the worst due to its unrealistic equal-dispersion assumption; the NB model, assuming a gamma distributed multiplicative random effect term, significantly improves the performances compared to the Poisson model; the proposed LGNB model, modeling extra-Poisson variations with both the gamma and lognormal distributions, clearly outperforms both the Poisson and NB models. Since for the lognormal-Poisson model with the M-H algorithm, we were not able to obtain comparable results even after carefully tuning the proposal distribution, we did not include it here for comparison. However, since the LGNB model reduces to the lognormal-Poisson model as r → ∞, the results of the LGNB model with r ≡ 1000 would be able to indicate whether the lognormal distribution alone is appropriate to model the extra-Poisson variations. Despite the popularity of the NB model, which models extra-Poisson variations only with the gamma distribution, the results in Tables 2 suggest the benefits of incorporating the lognormal random effects. These observations also support the claim in (Winkelmann, 2008) that the lognormal-Poisson model should be reevaluated since it is appealing in theory and may fit the data better. Compared to the lognormal-Poisson model, the LGNB model has an additional advantage that its parameters can be estimated with VB inference, which is usually much faster than sampling based methods.

Table 2.

Test of goodness of fit with Pearson residuals.

Models (Methods)	NASCAR	MotorIns
Poisson (MLE)	655.6	485.6
NB (MLE)	138.3	316.5
IG-Poisson (MLE)	N/A	319.7
LGNB (r ≡ 1000, Gibbs)	117.8	296.7
LGNB(VB)	126.1	275.5
LGNB(Gibbs)	129.0	284.4

A clear advantage of the Bayesian inference over the MLE is that a full posterior distribution can be obtained, by utilizing the estimated posteriors of σ², r and β. For example, shown in Figure 2 are the estimated posterior distributions of the quasi-dispersion κ, represented with histograms. These histograms should be compared to κ = 0 in the Poisson model, and the NB model’s MLEs of κ = 0.1905 and κ = 0.0118, for the NASCAR and MotorIns datasets, respectively. We can also find that VB generally tends to overemphasize the regions around the mode of its estimated posterior distribution and consequently places low densities on the tails, whereas Gibbs sampling is able to explore a wider region. This is intuitive since VB relies on the assumption that the posterior distribution can be approximated with the product of independent Q functions, whereas Gibbs sampling only exploits conditional independence.

Figure 2.

The histograms of the quasi-dispersion κ = e^σ2 (1+1/r)–1 based on (a) the 2000 collected Gibbs samples for NASCAR, (b) the 2000 simulated samples using the VB Q functions for NASCAR, (c) the 2000 collected Gibbs samples for MotorIns, and (d) the 2000 simulated samples using the VB Q functions for MotorIns.

The estimated posteriors can also assist model interpretation. For example, based on ${\widehat{\Sigma }}_{\beta }$ in VB for the NASCAR dataset, we can calculate the correlation matrix for (β₁, β₂, β₃)^T as

$$\left(\begin{array}{ccc}\hfill 1.0000\hfill & \hfill -0.4824\hfill & \hfill 0.8933\hfill \\ \hfill -0.4824\hfill & \hfill 1.0000\hfill & \hfill -0.7171\hfill \\ \hfill 0.8933\hfill & \hfill -0.7171\hfill & \hfill 1.0000\hfill \end{array}\right)$$

which is typically not provided in MLE. Since β₁ (Laps) and β₃ (TrkLen) are highly positively correlated, we expect the corresponding covariates to be highly negatively correlated. This is confirmed, as the correlation coefficient between the number of laps and the track length is found to be as small as −0.9006.

6. Conclusions

A lognormal and gamma mixed negative binomial (LGNB) regression model is proposed for regression analysis of overdispersed counts. Efficient closed-form Gibbs sampling and VB inference are both presented, by exploiting the compound Poisson representation and a Polya-Gamma distribution based data augmentation approach. Model properties are examined, with comparison to the Poisson, NB and lognormal-Poisson models. As the univariate lognormal-Poisson regression model can be easily generalized to regression analysis of correlated counts, in which the derivatives and Hessian matrixes of parameters are used to construct multivariate normal proposals in a Metropolis-Hastings algorithm (Chib et al., 1998; Chib & Winkelmann, 2001; Ma et al., 2008; Winkelmann, 2008), the proposed LGNB model can be conveniently modified for multivariate count regression, in which we may be able to derive closed-form Gibbs sampling and VB inference. As the log Gaussian process can be used to model the intensity of the Poisson process, whose inference remains a major challenge (Møller et al., 1998;Adams et al., 2009; Murray et al., 2010; Rao & Teh, 2011), we may link the log Gaussian process to the logit of the NB probability parameter, leading to a log Gaussian NB process with tractable closed-form Bayesian inference. Furthermore, the NB distribution is shown to be important for the factorization of a term-document count matrix (Williamson et al., 2010; Zhou et al., 2012), and the multinomial logit has been used to model correlated topics in topic modeling (Blei & Lafferty, 2005; Paisley et al., 2011). Applying the proposed lognormal-gamma-NB framework and the developed closed-form Bayesian inference to these diverse problems is currently under active investigation.