Flexible Random Intercept Models for Binary Outcomes Using Mixtures of Normals

Abstract

Random intercept models for binary data are useful tools for addressing between-subject heterogeneity. Unlike linear models, the non-linearity of link functions used for binary data force a distinction between marginal and conditional interpretations. This distinction is blurred in probit models with a normally distributed random intercept because the resulting model implies a probit marginal link as well. That is, this model is closed in the sense that the distribution associated with the marginal and conditional link functions and the random effect distribution are all of the same family. It is shown that the closure property is also attained when the distributions associated with the conditional and marginal link functions and the random effect distribution are mixtures of normals. The resulting flexible family of models is demonstrated to be related to several others present in the literature and can be used to synthesize several seemingly disparate modeling approaches. In addition, this family of models offers considerable computational benefits. A diverse series of examples is explored that illustrates the wide applicability of this approach.

1. Introduction

Random intercept models for binary data are useful tools for addressing between subject heterogeneity. Typically, random intercept models are implemented by adding a normally distributed random effect into the linear predictor of a generalized linear model (or GLM, see Nelder and Wedderburn, 1972; McCullagh and Nelder, 1989), giving rise to a generalized linear mixed model (or GLMM, see Breslow and Clayton, 1993). Because of the non-linearity of the link functions for binary GLMMs, such models force a distinction between parameter interpretations conditional on random effects and marginal interpretations averaged over random effects.

Random intercept models for binary outcomes with a probit link function and normally distributed random intercept (probit-normal models) have the interesting property that the marginal link function is the inverse of a normal cumulative distribution function (CDF). In this case, we say the model is “closed” in the sense that the distributions associated with the marginal and conditional link functions and the random effect distribution are all of the same family.

In this manuscript we explore a general family of closed random intercept models. In particular, we consider instances when the distribution associated with the conditional link function and the random effect distribution are mixtures of normals. Simple properties of mixture of normals then imply that the distribution function associated with the marginal link function is also a mixture of normals. We emphasize both the conceptual and practical benefits of this class of models. Notably, we explore models that yield conditional and marginal interpretations of parameters.

To summarize the results, the proposed model offers the conceptual benefit of containing a wide class of common models for binary data as either special or limiting cases. Furthermore, we highlight some interesting practical advantages of these models. In particular, the added flexibility of placing a mixture distribution on the random effects protects against misspecification of this distribution. Placing a mixture distribution on the conditional link distribution both allows for easy post-hoc approximations of marginal link functions and easier fitting for marginalized multilevel models. Finally we demonstrate that the latent variable representations of the mixture distributions for the conditional link and random effect distributions can result in simple and elegant Gibbs samplers for Bayesian analysis.

The manuscript is laid out as follows. In Section 2 we present the notation and the model. In Section 3 we connect the mixture of normals model with several variants of random effect models in the literature. In Section 4 we illustrate with a diverse collection of useful applications of the mixture of normals approximation. Finally, in Section 5, we provide a summary and discussion of future work.

2. Random intercept model for binary outcomes

2.1. Notation

Consider the data given in Table 1, which arose from a teratology experiment (Weil, 1970), and was subsequently analyzed in Liang and Hanfelt (1994) and Heagerty and Zeger (1996). The objective is to compare the survival of rat pups in 16 control litters with that of the pups in the 16 treated litters. The treatment was a chemical agent administered to the mothers of each treated litter. We use this data set and experiment to motivate the model.

Table 1.

Teratology data. Numbers are (number survived, number dead) in each litter by treatment arm. For example, in the first control litter, all thirteen pups survived. Source Weil (1970).

	(number survived, number dead)
Control	(13, 0) (12, 0) (9, 0) (9, 0) (8, 0) (8, 0) (12, 1) (11, 1)
	(9, 1) (9, 1) (8, 1) (11, 2) (4, 1) (5, 2) (7, 3) (7, 3)
Treatment	(12, 0) (11, 0) (10, 0) (9, 0) (10, 1) (9, 1) (9, 1) (8, 1)
	(8, 1) (4, 1) (7, 2) (4, 3) (5, 5) (3, 3) (3, 7) (0, 7)

Assume that {Y_ij} are repeated binary responses for subject/cluster i = 1, …, I and response j = 1, …, J_i. Therefore, in the Teratology data set, Y_ij represents survival or not (1 versus 0 respectively) for pup j from litter i. Let x_ij be a vector of covariates associated with Y_ij. For the Teratology data x_ij = (1, x_ij1)^t, containing an intercept term and a treatment indicator, respectively.

Let $F_w^{-1}$ be a link function (see McCullagh and Nelder, 1989) that relates the probability of a success to a function of the covariates. As is typical for binary data, we assume that F_w (the inverse link function) is a distribution function, referred to as the “link distribution”. We assume that

$$Pr(Y_{ij}=1\mid U_i=u_i)=F_w({\mathrm{\Delta }}_{ij}-u_i),$$ (1)

where the {U_i} are cluster-specific random effects, used to model correlation and heterogeneity arising from unmeasured covariates specific to a cluster. The {U_i} are assumed to be independent and identically distributed random variables, having distribution function F_u. Throughout we assume that the {Y_ij} are conditionally independent given the {U_i}.

Users familiar with GLMMs will note two departures from common notation. First, the “transfer function”, Δ_ij, is typically omitted and replaced with a linear combination of the covariates and slope parameters, such as

$${\mathrm{\Delta }}_{ij}={\mathbf{x}}_{ij}^t{\mathit{\beta }}^c.$$ (2)

This departure is adopted to consider a broader class of marginal and conditional models, which we describe in detail. Secondly, the random effect is subtracted in (1) rather than added, a convention that will be discussed below.

2.2. Conditional models

A conditional model specifies Δ_ij as in (2). The superscript c on the slope effects is used to denote that the effects are conditional, having an interpretation on the conditional link function’s scale.

Defining the Δ_ij as such implies a marginal model. Specifically

$$Pr(Y_{ij}=1)=F_q({\mathrm{\Delta }}_{ij}),$$ (3)

where F_q is the distribution of the sum of independent random variables having distribution functions F_u and F_w. To prove this fact, let {W_ij} be iid draws from F_w, then note that

$$\begin{array}{ll}Pr(Y_{ij}=1)& =E_{U_i}[Pr(Y_{ij}=1\mid U_i=u_i)]\\ & =\int F_w({\mathrm{\Delta }}_{ij}-u_i)d{F}_u(u_i)\\ & =\int Pr(W_{ij}\le {\mathrm{\Delta }}_{ij}-u_i\mid U_i=u_i)d{F}_u(u_i)\\ & =E_{U_i}[Pr(W_{ij}+u_i\le {\mathrm{\Delta }}_{ij}\mid U_i=u_i)]\\ & =Pr(W_{ij}+u_i\le {\mathrm{\Delta }}_{ij})\\ & =F_q({\mathrm{\Delta }}_{ij}).\end{array}$$

From this proof, we hope that the reason for the somewhat unusual convention of subtracting the random intercept is now clear.

We summarize the basic properties of the conditional model as

$$\begin{array}{ll}\text{Conditional\hspace{0.17em}model}\hfill & Pr(Y_{ij}=1\mid U_i=u_i)=F_w({\mathrm{\Delta }}_{ij}-u_i)\hfill \\ \text{Transfer\hspace{0.17em}function}\hfill & {\mathrm{\Delta }}_{ij}={\mathbf{x}}_{ij}^t{\mathit{\beta }}^c\hfill \\ \text{Random\hspace{0.17em}effect\hspace{0.17em}distribution}\hfill & Pr(U_i\le u_i)=F_u(u_i)\hfill \\ \text{Implied\hspace{0.17em}marginal\hspace{0.17em}model}\hfill & Pr(Y_{ij}=1)=F_q({\mathbf{x}}_{ij}^t{\mathit{\beta }}^c)\hfill \end{array}.$$

As an example, consider again the Teratology data set. Assume that F_w is the standard normal distribution, F_u is a normal distribution with 0 mean and variance ${\sigma }_{u,1}^2$, and Δ_ij is defined as in Equation 2. This model then corresponds to a probit-normal GLMM. By the standard properties of the normal distribution, the distribution of the sum of a standard normal (F_w) and a normal with mean 0 and variance ${\sigma }_{u,1}^2$ (F_u) results in F_q being a normal distribution with 0 mean and variance $1+{\sigma }_{u,1}^2$. Thus, using Equation 3, we have the well known result (see Zeger et al., 1988, for example) that the induced marginal model is

$$Pr(Y_{ij}=1)=F_q({\mathrm{\Delta }}_{ij})=F_q({\beta }_0^c+x_{ij1}{\beta }_1^c)=\mathrm{\Phi }\left\{\frac{{\beta }_0^c+x_{ij1}{\beta }_1^c}{{(1+{\sigma }_{u,1}^2)}^{1/2}}\right\},$$

where Φ denotes the standard normal distribution function. Hence, the marginal link is also a probit, with the marginal effects being scaled versions of the conditional effects, ${\mathit{\beta }}^c/{(1+{\sigma }_{u,1}^2)}^{1/2}$.

To illustrate, the fitted values for the Teratology data set are ${\widehat{\beta }}_0^c=1.474$, ${\widehat{\beta }}_1^c=-.585$, σ̂_u,1 = .749. Hence −.585 estimates the conditional probit-scale change in the probability of survival comparing the treated to the untreated pups. Correspondingly, ${\widehat{\beta }}_1^c/{(1+{\widehat{\sigma }}_{u,1}^2)}^{1/2}=-.468$ estimates the marginal probit-scale change in the probability of death.

2.3. Marginal Models

Consider again the Teratology probit-normal example from the previous section - i.e. F_w is a standard normal and F_u is a normal with mean 0 and variance ${\sigma }_{u,1}^2$. Had we defined

$${\mathrm{\Delta }}_{ij}=({\beta }_0^m+x_{ij1}{\beta }_1^m){(1+{\sigma }_{u,1}^2)}^{1/2},$$

then the marginal probability of success would satisfy

$$Pr(Y_{ij}=1)=F_q({\mathrm{\Delta }}_{ij})=F_q\left\{({\beta }_0^m+x_{ij1}{\beta }_1^m){(1+{\sigma }_{u,1}^2)}^{1/2}\right\}=\mathrm{\Phi }({\beta }_0^m+x_{ij1}{\beta }_1^m).$$

Therefore, the estimated slope parameters would have a marginal probit interpretation without rescaling; hence the superscript m. By the invariance of the MLE, the marginal slope estimate is identical to those calculated from the conditional model, ${\widehat{\beta }}_1^m=-.468$.

The probit-normal example has emphasized that appropriately defining Δ_ij results in parameters with marginal interpretations. In fact, Heagerty and Zeger (2000) showed that this technique can be applied more generally. Specifically, consider defining

$${\mathrm{\Delta }}_{ij}=F_q^{-1}\{F_w({\mathbf{x}}_{ij}^t{\mathit{\beta }}^m)\}.$$ (4)

Under this definition for Δ_ij and using (3), the marginal probability of success satisfies

$$Pr(Y_{ij}=1)=F_q({\mathrm{\Delta }}_{ij})=F_q\left[F_q^{-1}\{F_w({\mathbf{x}}_{ij}^t{\mathit{\beta }}^m)\}\right]=F_w({\mathbf{x}}_{ij}^t{\mathit{\beta }}^m)$$

That is, under an appropriate modification of Δ_ij, the slope parameters can be given a marginal interpretation with F_w as the link distribution. We summarize the marginal model with

$$\begin{array}{ll}\text{Conditional\hspace{0.17em}model}\hfill & Pr(Y_{ij}=1\mid U_i=u_i)=F_w({\mathrm{\Delta }}_{ij}-u_i)\hfill \\ \text{Transfer\hspace{0.17em}function}\hfill & {\mathrm{\Delta }}_{ij}=F_q^{-1}\{F_w({\mathbf{x}}_{ij}^t{\mathit{\beta }}^m)\}\hfill \\ \text{Random\hspace{0.17em}effect\hspace{0.17em}distribution}\hfill & Pr(U_i\le u_i)=F_u(u_i)\hfill \\ \text{Implied\hspace{0.17em}marginal\hspace{0.17em}model}\hfill & Pr(Y_{ij}=1)=F_w({\mathbf{x}}_{ij}^t{\mathit{\beta }}^m)\hfill \end{array}.$$

Consider again the Teratology data set. Allowing F_w to be the logistic distribution yields ${\widehat{\beta }}_1^m=-.86$, which estimates the marginal log-odds ratio of survival.

Marginalized multilevel models defined as such offer several advantages over competing methods. Unlike generalized estimating equations (GEE, see Liang and Zeger, 1986), they enjoy the benefits of a completely specified model, which includes the ability to plot profile likelihoods, the availability of likelihood ratio tests and Bayesian analysis and the relaxation on assumptions for missing data. Also, these models are more parsimonious and extensible than other marginal likelihood based models (see Lang and Agresti, 1994).

2.4. Mixtures of normals

The distinction between the conditional and marginal approaches is especially interesting for the probit-normal model, because of the fact that the probit-normal model is closed - the conditional, random effect and marginal link distributions all belong to the same family. In this manuscript we present another closed random intercept model for binary data that is considerably more flexible than the probit-normal model. In particular, when F_w and F_u are mixtures of normal distributions, then so is F_q.

To prove this, consider a model of the form

$$F_w(w)=\underset{l=1}{\overset{L_w}{\Sigma }}{\pi }_{w,l}\mathrm{\Phi }\left(\frac{w-{\mu }_{w,l}}{{\sigma }_{w,l}}\right)\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{and\hspace{0.17em}}\hspace{0.17em}\hspace{0.17em}{F}_u(u)=\underset{l=1}{\overset{L_u}{\Sigma }}{\pi }_{u,l}\mathrm{\Phi }\left(\frac{u-{\mu }_{u,l}}{{\sigma }_{u,l}}\right),$$

where, the {π_w,l} and {π_u,l} are each assumed to be greater than 0 and sum to one. Using simple properties of mixtures of normals and Equation 3, we have that

$$F_q(q)=\underset{l=1}{\overset{L_w}{\Sigma }}\underset{l^{\prime }=1}{\overset{L_u}{\Sigma }}{\pi }_{w,l}{\pi }_{u,l^{\prime }}\mathrm{\Phi }\left\{\frac{q-{\mu }_{w,l}-{\mu }_{u,l^{\prime }}}{{({\sigma }_{w,l}^2+{\sigma }_{u,l^{\prime }}^2)}^{1/2}}\right\}.$$ (5)

That is, under this model, the random effect, conditional and marginal link distributions are all mixtures of normals. We summarize the model as

$$\begin{array}{ll}\text{Conditional\hspace{0.17em}model}\hfill & Pr(Y_{ij}=1\mid U_i=u_i)=\underset{l=1}{\overset{L_w}{\Sigma }}{\pi }_{w,l}\mathrm{\Phi }\left(\frac{{\mathrm{\Delta }}_{ij}-u_i-{\mu }_{w,l}}{{\sigma }_{w,l}}\right)\hfill \\ \text{Transfer\hspace{0.17em}function}\hfill & {\mathrm{\Delta }}_{ij}\text{defined\hspace{0.17em}by\hspace{0.17em}either\hspace{0.17em}}(2)\hspace{0.17em}\text{or\hspace{0.17em}}(4)\hfill \\ \text{Random\hspace{0.17em}effect\hspace{0.17em}distribution}\hfill & Pr(U_i\le u_i)=\underset{l=1}{\overset{L_u}{\Sigma }}{\pi }_{u,l}\mathrm{\Phi }\left(\frac{u_i-{\mu }_{u,l}}{{\sigma }_{u,l}}\right)\hfill \\ \text{Implied\hspace{0.17em}marginal\hspace{0.17em}model}\hfill & Pr(Y_{ij}=1)=\underset{l=1}{\overset{L_w}{\Sigma }}\underset{l^{\prime }=1}{\overset{L_u}{\Sigma }}{\pi }_{w,l}{\pi }_{u,l^{\prime }}\mathrm{\Phi }\left\{\frac{{\mathrm{\Delta }}_{ij}-{\mu }_{w,l}-{\mu }_{u,l^{\prime }}}{{({\sigma }_{w,l}^2+{\sigma }_{u,l^{\prime }}^2)}^{1/2}}\right\}\hfill \end{array}.$$ (6)

To summarize, the model of interest in this manuscript combines the conditional and marginal approaches, while adding the constraint that the conditional link and random effect distributions are both mixtures of normals. For completeness, we add that the log-likelihood for (6) is

$$\underset{i=1}{\overset{I}{\Sigma }}log{\int }_{u{}_i}\prod _{j=1}^{J_i}{F}_w{({\mathrm{\Delta }}_{ij}-u_i)}^{y_{ij}}{\{1-F_w({\mathrm{\Delta }}_{ij}-u_i)\}}^{1-y_{ij}}d{F}_u(u_i)$$ (7)

(an equation that holds regardless of whether F_w and F_u are mixtures of normals).

Of course, Model 6 is excessively rich with all of the mixture probabilities, means and variances left unspecified; estimating both the conditional link distribution and the random effect distribution is a hopeless cause for most binary data sets. However, by specifying components of one or both of the free mixture distributions, one can achieve a variety of important models. We emphasize the following uses. First, specifying the mixture components for the conditional link distribution yields the ability to (approximately) fit models with other link functions, such as the logit, while retaining many of the computational benefits of probit links. Secondly, estimating a small number of mixture components on the random effects can be used to protect against canonical forms of misspecification. Finally, we find that using mixtures of normals for the conditional link and the random effect distribution leads to particularly convenient Gibbs samplers for conditional models. While many of these ideas have been explored, to our knowledge, they have not been placed in a unified modeling framework combining both conditional and marginal approaches. In the following literature review we highlight connections with existing research.

3. Literature review

In this section we demonstrate that Model 6 contains several important random intercept models for binary data as special or limiting cases. Clearly if F_u is degenerate at 0 and ${\mathrm{\Delta }}_{ij}={\mathbf{x}}_{ij}^t{\beta }^c$, then the model yields a GLM for binary data. Extending this setting so that F_u is not degenerate and L_u = 1 and μ_u,1 = 0 yields a GLMM for binary data with a normally distributed random intercept (see Breslow and Clayton, 1993; Agresti et al., 2000).

To be technical, only those GLM and GLMMs for binary data whose conditional link distribution, F_w, is a mixture of normals are special cases of the model we have suggested. However, all of the common link functions (logit, complementary log-log) can be obtained as limiting cases. In Appendix C we provide an algorithm to solve for π_w,l, σ_w,l and μ_w,l that yields very accurate approximations for a finite number of mixture components.

As an example, consider a mixture of normals as an approximation of the logistic distribution. The results using the algorithm in Appendix C with 150 quadrature points and {μ_w,j} = {0} yields the values given in Table 2. Figure 1 shows how accurate the approximation is, by depicting the exact logistic quantiles by a mixture of normals approximation. The mixture of normals approximation, with 5 mixture components, is nearly exact to logits of ± 10. By comparison, the plot also shows the standard normal and T quantiles, both of which are also used as approximations to the logit (see Caffo and Griswold, 2005). The linearity of the probit approximations breaks down at logits of around ± 3, while the T approximation around ± 5. Furthermore, we note that the mixture of normals approximation applies generally, to links other than the logistic, and can be made more accurate by simply adding more mixture components.

Table 2.

Mixing probabilities, standard deviations and means of the mixture components for a mixture-of-normals approximation to the logistic distribution.

π1	π2	π3	π4	π5
0.126840496	0.543170220	0.261711982	0.066181589	0.002066853
σ1	σ2	σ3	σ4	σ5
2.8420536	1.8257138	1.1943048	1.0757749	0.5631853
μ1	μ2	μ3	μ4	μ5
0	0	0	0	0

Fig. 1.

Quantile-quantile plot of the logistic distribution (vertical axis) by three approximations: the mixture of normals (solid), the probit (dotted), the T (dashed). A reference identity line is depicted in grey. The corresponding probability scale is given on the right and upper axes.

Approximating the logistic distribution with a single normal distribution or mixture of normals has a rich history (see Demidenko, 2004, and the references therein). Perhaps most relevant, Monahan and Stefanski (1992) used weighted Gaussian distributions to explore the logistic-normal integral.

Representing the link function by a latent variable was considered in the proof of Equation 3. In Section 4.3 we consider a much more ambitious latent variable representation of Model 6, using latent variables to represent the normal mixture distributions as well. The general latent variable approach to binary data was considered in Albert and Chib (1993), who also introduced a Gibbs sampler that motivates the one presented in Section 4.3. Relevant extensions to multivariate settings were considered in Chib and Greenberg (1998); however they focused on probit links and more general covariance structures than the random intercept models considered here.

Consider again the instance where L_u = 1, μ_u,1 = 0 (the random intercept is normally distributed). As described in Section 2.3, Heagerty and Zeger (2000) defined the Δ_ij to be non-linear (see Equation 4), so that the slope parameters have linear interpretations on the marginal link’s scale. These marginalized multilevel models for binary data are a special case of the models presented (by appropriately defining Δ_ij). Moreover, later we demonstrate that using mixtures of normals for the link distribution can greatly facilitate computing for these models.

A potentially negative aspect of this model is that, because Δ_ij is defined non-linearly, the conditional model is non-linear. The degree to which this is true depends on how close to linear $F_q^{-1}{F}_w$ is. However, this may be of no concern whatsoever if only marginal interpretations are required (though see Lee and Nelder, 2004).

A clear generalization of the marginalized model would replace F_w in Equation 4 with any other desired link distribution, thus, allowing the conditional and marginal link functions to be different. This idea was explored in Griswold (2005) and extended to ordered multinomial data in Caffo and Griswold (2005). Again, this approach easily fits into the current framework by appropriately redefining Δ_ij.

There has been a relatively small amount of research using mixtures of normals to estimate the link distribution. Geweke and Keane (1997) used a mixture of normals as a link function for dichotomous choice models. They presented an MCMC algorithm for fitting the model, including estimating the mixture components. In related work, Erkanli et al. (1993) used mixtures of normals to estimate the link function for ordinal data models and also presented an MCMC algorithm for estimating the mixture components. These approaches are conceptually related to the proposed model by forcing the random effect distribution to be degenerate at 0 and estimating {π_w,l}, {μ_w,l} and { σ_w,l}.

In contrast, using mixtures to estimate the random effect distribution has received much more attention. Perhaps most relevant, Magder and Zeger (1996) used mixtures of normals as the random effect distribution and estimated the mixture parameters with an MCMC algorithm. This corresponds to estimating the {μ_u,l}, {σ_u,l} and {π_u,l}. Aitkin (1999) and Follmann and Lambert (1989) used discrete mixtures to non-parametrically estimate the random effect distribution using maximum likelihood. Such models are obtained under the current framework as the {σ_u,j} tend to 0 and {μ_u,j} and {π_u,j} are estimated.

4. Examples

In this section we explore a subset of practical considerations illustrated through four data sets. We explore two marginal and one conditional modeling settings, where computations are significantly simplified by using mixtures of normals. Moreover, we consider a case where mixture modeling of the random effect offers additional protection against model misspecification.

We consider four well studied data sets for illustration:

1. The Teratology data, introduced in Section 2.1.

2. The Approval Rating data set given in Table A.1. This 2×2 contingency table cross-classifies approval ratings of the British Prime Minister collected at two occasions. Here, Y_ij represents approval (1) or not (0) for individual i on occasion j, where j = 1, 2 for the two sampling occasions. The covariate vector, x_ij = (1, x_ij)^t, contains an intercept term and an indicator function representing occasion, taking the value 1 when j = 2.

3. The Crossover data, given in Table A.2, concerns a well-studied crossover study from Jones and Kenward (1987). Here, Y_ij represents an abnormal (1) or a normal (0) response for subject i during period j for j = 1, 2. The objective is to study the response in relation to the treatment and period. Thus, x_ij = (1, x_ij1, x_ij2)^t contains an intercept term, a treatment indicator and a period indicator, taking the value 1 for the second period.

4. The Item Response data, given in Table A.3, concerns subjects’ response to three scenarios (given in the table) on abortion stratified by gender. We let Y_ij be the response of subject i on question j, where a response of 1 is supportive of legalized abortion (and 0 is not). The covariate vector, x_ij = (1, x_ij)₁, x_ij2, x_ij3)^t, contains an intercept term, an indicator for male gender, an indicator for Scenario 1, and an indicator for Scenario 2, respectively. We use the Item Response data to illustrate an instance where a mixture random effect distribution is warranted.

To focus this discussion, we assume that the principal parameter of interest for each data set is: the (marginal or conditional) log odds-ratio comparing treated to controls in the Teratology data, the log odds-ratio comparing time 2 to time 1 for the Approval Rating data, the log odds-ratio comparing treated to controls in the Crossover data and the log odds-ratio comparing males to females in the Item response data. Therefore, in each case the regressor corresponding to the effect of interest is x_ij1.

4.1. Post-hoc calculation of marginal effects

Given results from a conditional random effect model, an obvious question asks, “What is the corresponding marginal effects and link distribution?”. Such a question is especially relevant in situations such as in interpreting published results, where only effect estimates (and not the original data) are available. Model 6 allows one to approximate the necessary calculations easily.

Consider the conditional logit model

$$\text{logit}\{Pr(Y_{ij}=1\mid U_i=u_i)\}={\mathbf{x}}_{ij}^t{\mathit{\beta }}^c-u_i\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{and\hspace{0.17em}}\hspace{0.17em}\hspace{0.17em}{U}_i\sim \text{N}(0,{\sigma }_{u,1}^2).$$ (8)

If we are willing to accept the approximation that F_w is the 5 component mixtures of normals, then Model 8 is simply a special case of Model 6. Hence, we have that

$$Pr(Y_{ij}=1)=F_q({\mathbf{x}}_{ij}^t{\widehat{\mathit{\beta }}}^c)=\underset{l=1}{\overset{L_w}{\Sigma }}{\pi }_{w,l}\mathrm{\Phi }\left\{\frac{{\widehat{\mathbf{x}}}_{ij}^t{\widehat{\mathit{\beta }}}^c-{\mu }_{w,l}}{{({\sigma }_{w,l}^2+{\sigma }_{u,1}^2)}^{1/2}}\right\},$$ (9)

where {π_w,l}, {μ_w,l} and {σ_w,l} are from Table 2.

Below we use this approximation to obtain marginal logit interpretations from conditional logit models. However, before doing so, we emphasize the benefits of Equation 9 over Monte Carlo and numerical integration, which can also give very accurate approximations of marginal effects. For example, unlike numerical integration or Monte Carlo approximations, the approximation (9) can be performed quickly and easily. In addition, obtaining delta method estimates of standard errors is also easy. Furthermore, the method applies to any conditional link function, provided the relevant mixture components are known. Finally, and perhaps most importantly, we note that this method leads to an accurate and simple approximation to the marginal link distribution, F_q, whereas quadrature or Monte Carlo approximations only yield F_q for specific values of the covariates.

Table 4 gives estimated marginal logit effects for the four data sets calculated using (9). To illustrate the calculations, consider the Teratology dataset. The fitted values (SE) from Model 8 using the SAS procedure NLMIXED are ${\widehat{\beta }}_0^c=2.63\hspace{0.17em}(0.48)$, ${\widehat{\beta }}_1^c=-1.08\hspace{0.17em}(0.63)$ and σ̂_u,1 = 1.35 (0.33). Plugging the estimated parameters into (9) yields a marginal probability of survival of 0.76 for the treated and 0.88 for the untreated. Then, the marginal log odds ratio of survival (SE) comparing the treated to the control litters is logit(0.76) − logit(0.88) = −0.86 (0.51) (see Appendix D for details about obtaining standard errors).

Table 4.

Marginal logit estimates (standard errors) for the examples from Section 4.1.

Data Set	Marginal Estimate ( $\widehat{{\beta }_1^m}$)
Teratology	−0.86 (0.51)
Approval	−0.16 (0.04)
Crossover	Period 1 0.59 (0.31)	Period 2 10.58 (0.29)
Item Response	Question 1 −0.002 (0.03)	Question 2 −0.002 (0.03)	Question 3 −0.002 (0.03)

Table 4 applies these techniques to the three other data sets as well, each time taking the conditional estimates output by SAS (Table 3). Because of the additional covariates in the Crossover and Item Response data sets, the estimated marginal logit effects are reported within strata.

Table 3.

Conditional Estimates (standard errors).

	Parameter
Data Set	σ̂u,1	${\widehat{\beta }}_0^c$	${\widehat{\beta }}_1^c$	${\widehat{\beta }}_2^c$	${\widehat{\beta }}_3^c$
Teratology	1.35 (0.33)	2.63 (0.48)	−1.08 (0.63)
Approval	5.16 (0.35)	1.24 (0.19)	−0.56 (0.14)
Crossover	4.94 (1.91)	2.22 (1.17)	1.86 (0.93)	−1.04 (0.82)
Item Response	8.75 (0.54)	−0.61 (0.34)	−0.013 (0.49)	0.83 (0.16)	0.29 (0.16)

4.2. Easier marginalized multilevel models

The previous section addressed the issue of obtaining marginal effects from conditional results, which is useful when interpreting published results without access to the underlying data. However, when the data are available and marginal interpretations are desired, direct fitting is preferable. This section illustrates how the mixture of normals modeling framework can ease the calculations required to directly obtain marginal estimates.

We consider the marginal Model 6 where Δ_ij is given by Equation 4. Furthermore, assume that the $U_i\sim N(0,{\sigma }_{u,1}^2)$ and F_w is the 5 component mixtures of normals approximation to the logistic distribution function.

The benefit of using the F_w as a mixture of normals rather than the exact logistic distribution is that there is a closed form for F_q (see Equation 4); also its quantiles can easily be calculated using Newton’s method. Hence, representing the logistic distribution as such eliminates the difficult task of numerically approximating the convolution integral defining F_q and its inverse. It should be emphasized that while defining F_w as a mixture of normals eases the calculation of F_q and hence Δ_ij, calculation of the likelihood (7) still requires numerical integration, for which we employed Gauss/Hermite quadrature.

We implemented this model for the four data sets. We highlight the use of profile likelihoods - the functions obtained by maximizing the likelihood for each value of the parameter of interest. See Royall (1997) for more information regarding the benefits and interpretation of profile likelihoods.

The results of the model fits are given in Table 5. For example, for the Teratology data, −0.86 (the estimate for ${\beta }_1^m$) estimates the change in the marginal log-odds of survival comparing a treated pup to a control. For each of the data sets, Figure 2 shows the profile likelihood with 1/8 and 1/16 reference lines see (see Royall, 1997) for the parameter of interest ( ${\beta }_1^m$) and the variance component (σ_u,1) for each of the four models.

Table 5.

Estimates (standard errors) for marginalized multilevel models from Section 4.2.

	Parameter
Data Set	σ̂u,1	${\widehat{\beta }}_0^m$	${\widehat{\beta }}_1^m$	${\widehat{\beta }}_2^m$	${\widehat{\beta }}_3^m$
Teratology	1.35 (0.33)	2.03 (0.39)	−0.87 (0.51)
Approval	5.16 (0.35)	0.36 (0.05)	−0.16 (0.04)
Crossover	4.94 (1.91)	0.68 (0.28)	0.58 (0.23)	−0.32 (0.23)
Item Response	8.71 (0.54)	−0.048 (0.054)	0.004 (0.074)	0.150 (0.028)	0.053 (0.028)

Fig. 2.

Profile likelihood plots with 1/8 and 1/16 reference lines, see (Royall, 1997) for ${\beta }_1^m$ and σ_u,l for the marginalized multilevel model from 4.2. The rows from top to bottom correspond to the Teratology, Approval, Crossover and Item Response data sets respectively.

4.3. Bayesian analysis

In this section, we illustrate how specific instances of Model (6) are particularly well suited for Bayesian analysis via MCMC. We note that similar methods utilizing latent variables have been proposed to simulate from the posterior distributions of parameters for binary and multinomial responses (see Albert and Chib, 1993; McCulloch and Rossi, 1994; Chib et al., 1998; Imai and van Dyk, 2005). In addition, close variants of the sampling schemes can be used for the Monte Carlo EM algorithm (see Chib et al., 1998; Natarajan et al., 2000).

We apply these methods to binary responses with random effects, using the mixture of normals link approximation (similar to Geweke and Keane, 1997; Erkanli et al., 1993). Consider the latent variable representation of Model (6) given by

1. {D_u,i} are iid discrete random variables with support 1, …, L_u so that Pr(D_u,i = l) = π_u,l,

2. the {U_i} given that the {D_u,i = d_u,i} are independent N (μ_u,d_u,i, ${\sigma }_{u,d_{u,i}}^2$),

3. the {D_w,ij} are discrete iid random variables with support 1, …, L_w so that Pr(D_w,ij = l) = π_w,l,

4. the {M_ij} given that the {D_w,ij = d_w,ij} and {U_i = u_i} are independent Normals with mean μ_w,d_w,ij + u_i − Δ_ij and variance ${\sigma }_{w,d_{w,ij}}^2$,

5. the {Y_ij} are 1 iff M_ij ≤ 0 and 0 otherwise,

6. each ${\mathrm{\Delta }}_{ij}={\mathbf{x}}_{ij}^t{\mathit{\beta }}^c$,

To summarize the model, items 1 and 2 yield the mixture model for F_u, items 3–5 yield the conditional model for the y_ij and item 6 forces a conditional interpretation for the β^c. To prove that 3–5 induces the mixture of normals model for the y_ij, consider

$$\begin{array}{ll}Pr(Y_{ij}=1\mid U_i=u_i)& =Pr(M_{ij}\le 0\mid U_i=u_i)\\ & =\underset{l=1}{\overset{L_w}{\Sigma }}Pr(M_{ij}\le 0\mid U_i=u_i,D_{w,ij}=l)Pr(D_{w,ij}=l)\\ & =\underset{l=1}{\overset{L_w}{\Sigma }}\mathrm{\Phi }\left(\frac{{\mathrm{\Delta }}_{ij}-{\mu }_{w,l}-u_i}{{\sigma }_{w,l}}\right){\pi }_{w,l}.\end{array}$$

We complete the Bayesian model by specifying that β^c ~ Normal(μ_β^c, Σ), ${\sigma }_{u,l}^2\sim \text{IG}(\nu ,\tau )$. In the examples where the random effect mixture distribution had more than one component, the {μ_u,l} were independent normals with mean η and variance θ and {π_u,l} were Dirichlet with shape parameters α. We note a small complication is that the mean of the random effect distribution is aliased with an intercept parameter. Therefore, throughout this section we assume that the intercept term is excluded and instead the random effect mean is estimated. A second complication could potentially arise when the random effect mixture distribution has more than one component and the {π_u,l}, {μ_u,l} and the {σ_u,l} are of primary interest, because of the non-identifiability of the parameters due to permutation invariance of the likelihood. In the settings we present this is not an issue, because we focus on the label invariant parameters, such as β^c. In our investigations of the label-dependent parameters, we addressed this issue by imposing an ordering on the {μ_u,l}. See Jasra et al. (2005) and the references therein for further information on the “label-switching” problem.

The benefit of this model specification is that all of the full conditionals are common distributions and an elegant Gibbs sampler, which does not employ any Metropolis/Hastings steps, is available for exploring the posterior. The full conditionals are as follows:

1. the full conditional for D_u,i is discrete so that the probability D_u,i takes value l is

   $$\frac{{\sigma }_{u,l}^{-1}exp\left\{-{(u_i-{\mu }_{u,l})}^2/2{\sigma }_{u,l}^2\right\}{\pi }_{u,l}}{{\Sigma }_k{\sigma }_{u,k}^{-1}exp\left\{-{(u_i-{\mu }_{u,k})}^2/2{\sigma }_{u,k}^2\right\}{\pi }_{u,k}};$$
2. the full conditional for U_i is normal with mean

   $${\left(\underset{j}{\Sigma }{\sigma }_{w,d_{w,ij}}^{-2}+{\sigma }_{u,d_{u,i}}^{-2}\right)}^{-1}\left(\underset{j}{\Sigma }\frac{m_{ij}-{\mu }_{w,d_{w,ij}}+{\mathrm{\Delta }}_{ij}}{{\sigma }_{w,d_{w,ij}}^2}+\frac{{\mu }_{d_{u,i}}}{{\sigma }_{u,d_{u,i}}^2}\right)$$

   and variance

   $${\left(\underset{j}{\Sigma }{\sigma }_{w,d_{w,ij}}^{-2}+{\sigma }_{u,d_{u,i}}^{-2}\right)}^{-1};$$
3. the full conditional for D_w,ij is discrete so that the probability D_w,ij takes value l is

   $$\frac{{\sigma }_{w,l}^{-1}exp\{-{(m_{ij}-{\mu }_{w,l}-u_i+{\mathrm{\Delta }}_{ij})}^2/2{\sigma }_{w,l}^2\}{\pi }_{w,l}}{{\Sigma }_k{\sigma }_{w,k}^{-1}exp\{-{(m_{ij}-{\mu }_{w,k}-u_i+{\mathrm{\Delta }}_{ij})}^2/2{\sigma }_{w,k}^2\}{\pi }_{w,k}};$$
4. the full conditional for M_ij is truncated normal with mean μ_w,d_w,ij+ u_i − Δ_ij and variance ${\sigma }_{w,d_{w,ij}}^2$ with M_ij ≤ 0 when y_ij = 1 and M_ij > 0 when y_ij = 0; that is, the distribution function is

   $$\frac{\mathrm{\Phi }\left\{(m_{ij}-{\mu }_{w,d_{w,ij}}-u_i+{\mathrm{\Delta }}_{ij})/{\sigma }_{w,d_{w,ij}}\right\}}{\mathrm{\Phi }\left\{(-{\mu }_{w,d_{w,ij}}-u_i+{\mathrm{\Delta }}_{ij})/{\sigma }_{w,d_{w,ij}}\right\}}I(m_{ij}\le 0)$$

   when y_ij = 1 and

   $$\frac{\mathrm{\Phi }\left\{(m_{ij}-{\mu }_{w,d_{w,ij}}-u_i+{\mathrm{\Delta }}_{ij})/{\sigma }_{w,d_{w,ij}}\right\}-\mathrm{\Phi }\left\{(-{\mu }_{w,d_{w,ij}}-u_i+{\mathrm{\Delta }}_{ij})/{\sigma }_{w,d_{w,ij}}\right\}}{1-\mathrm{\Phi }\left\{(-{\mu }_{w,d_{w,ij}}-u_i+{\mathrm{\Delta }}_{ij})/{\sigma }_{w,d_{w,ij}}\right\}}I(m_{ij}\ge 0)$$

   when y_ij = 0;
5. the full conditional for β^c is multivariate normal with mean

   $${({\mathbf{\Sigma }}^{-1}+{\mathbf{X}}^t{\mathbf{W}}^{-1}\mathbf{X})}^{-1}({\mathbf{\Sigma }}^{-1}{\mathit{\mu }}_{{\beta }^c}+{\mathbf{X}}^t{\mathbf{W}}^{-1}\mathit{\xi })$$

   where X is the design matrix, W is a diagonal matrix of the ${\sigma }_{w,d_{w,ij}}^2$ and ξ is a vector with elements μ_w,d_w,ij+ u_i − m_ij and variance

   $${({\mathbf{\Sigma }}^{-1}+{\mathbf{X}}^t{\mathbf{W}}^{-1}\mathbf{X})}^{-1};$$
6. the full conditional for ${\sigma }_{u,l}^2$ is inverted gamma with shape parameter

   $$\nu +\underset{i}{\Sigma }I(d_{u,i}=l)/2$$

   and rate parameter

   $$\tau +\underset{i}{\Sigma }I(d_{u,i}=l){(u_i-{\mu }_{u,l})}^2/2,$$
7. the full conditional for μ_u,l is normal with mean

   $${\left\{\underset{i}{\Sigma }I(d_{u,i}=l)\frac{1}{{\sigma }_{u,d_{u,i}}^2}+\frac{1}{\theta }\right\}}^{-1}\left\{\underset{i}{\Sigma }I(d_{u,i}=l)\frac{u_i}{{\sigma }_{u,d_{u,i}}^2}+\frac{n}{\theta }\right\}$$

   and variance

   $${\left\{\underset{i}{\Sigma }I(d_{u,i}=l)\frac{1}{{\sigma }_{u,d_{u,i}}^2}+\frac{1}{\theta }\right\}}^{-1},$$
8. the full conditional for the {π_u,l} is Dirichlet with shape parameter

   $$\mathit{\alpha }+\underset{i}{\Sigma }{\{I(d_{u,i}=1),\dots ,I(d_{u,i}=L_u)\}}^t.$$

We apply the Gibbs sampler to the four datasets employing diffuse priors with a single normal random intercept. Throughout we assume that F_w is the five component mixture of normals approximation to the logistic distribution. Figure 3 shows the estimated posterior distributions for the parameter of interest after 20, 000 simulations and 1, 000 burn-in samples, for each of the data sets employing 1, 2 and 3 mixture components for the random effect distribution. For each of the examples we set ν = 10⁻⁴, τ = 10⁻⁶, η = 0, θ = 10⁴, α = (1, …, 1)^t, μ_β^c = (0, …, 0)^t, and Σ as a diagonal matrix with entries 10. Starting values were obtained in an ad-hoc manner, though using the maximum likelihood estimates when available. Diagnostic checks were made by examining trace plots and plots of some sequential quantiles for parameters of interest (see Figure B.1 for the Item Response data as an example). The hyperparameter specification had little impact for a range of reasonably diffuse priors (results not reported). To investigate the impact of occasional extremely large values of σ_u,l, the gamma priors were truncated, which again did not impact results.

Fig. 3.

Estimated posterior densities using for the examples from Section 4.3 using one (solid), two (dashed) and three (dotted) component mixtures for the random effect distributions.

For the random effect distribution, the use of large numbers of mixture components or estimating the number of mixture components, is generally not advisable in this setting, because there is typically insufficient information to identify this distribution in such detail. Instead we emphasize the use of a small number of mixture components, specifically 2 and 3, to protect against some canonical types of misspecification in the form of skewness or a small number of modes (see Agresti et al., 2004). This is particularly interesting for the Item Response data, since a random effect distribution with three modes makes practical sense in this situation. Specifically, it is likely that three populations, one opposed to abortion under any circumstance, one in favor of abortion rights regardless of the circumstance, and a more heterogeneous group, dominate the random effect distribution.

Regardless, for the parameter of interest for these four data sets, misspecification of the random effect distribution does not appear to be impacting results. The estimated posterior densities appear to be the same regardless of the number of mixture components implemented (Figure 3).

5. Discussion

In this manuscript, we discussed the conceptual and computational benefits of using mixtures of normals as the conditional link distribution and random effect distribution for random intercept models for binary outcomes. The use of mixtures of normals could be exploited for further generalizations of the random intercept model. In particular, the extension to multivariate random effects, using mixtures of multivariate normals, is plausible. Furthermore, this mixture approach is potentially very useful for jointly modeling discrete and continuous outcomes. Finally, further work may also explore how the mixture approach facilitates description of the “bridge” random effect distribution as introduced by Wang and Louis (2003) and Wang and Louis (2004).

In closing we note that we have put all of the relevant code to reproduce all of the results, and the derivations of the Bayesian full conditionals at the corresponding author’s web site.

Appendix A. Data sets

Table A.1.

Prime minister approval rating. Source Agresti (2002).

First Survey	Second Survey
	Approve	Disapprove
Approve	794	150
Disapprove	86	570

Table A.2.

Crossover data, frequency of responses by treatment regimen. Source Jones and Kenward (1987).

Response		Treatment sequence
Period 1	Period 2	Drug-Placebo	Placebo-Drug
Normal	Normal	22	18
Abnormal	Normal	0	4
Normal	Abnormal	6	2
Abnormal	Abnormal	6	9

Table A.3.

Response to questions on abortion stratified by gender from Agresti (2002). A response of “1” was in favor of legalized abortion in a specific scenario while a response of “0” was not. The scenarios are i if the family has a very low income ii the woman is not married and does not want to marry the man iii for any reason.

	Sequence of Responses
Gender	111	110	011	010	101	100	001	000
male	342	26	6	21	11	32	19	356
female	440	25	14	18	14	47	22	457

Appendix B. Example diagnostic plots

Fig. B.1.

Diagnostic trace plots for ${\beta }_1^c$ for the Item Response data. The upper right plot displays the (.1, .2, .4, .6, .8, .9) sequential quantiles.

Appendix C. Approximating link functions with mixtures of normals

In this section we give an estimation procedure for approximating a distribution with a mixture of normals. For a given number of mixture elements, we chose to minimize the Kullback-Leibler divergence (Kullback and Leibler, 1951) between the mixture approximation and the true density. That is, if g is the density associated with the link function of interest and f is the mixture approximation, we minimize E_g[log{f (X)/g(X)}]. The algorithm was obtained as the limit of the standard EM algorithm for estimating normal mixture components as the number of observed data points goes to infinity.

Let ${\pi }_j^{(t)},{\sigma }_j^{(t)}$ and ${\mu }_j^{(t)}$ be the current estimates,

$$\begin{array}{l}{P}_j^{(t)}(x)=\frac{{\pi }_j^{(t)}\phi \{(x-{\mu }_j^{(t)})/{\sigma }_j^{(t)}\}/{\sigma }_j^{(t)}}{{\Sigma }_l{\pi }_l^{(t)}\phi \{(x-{\mu }_l^{(t)})/{\sigma }_l^{(t)}\}/{\sigma }_l^{(t)}}\\ {\pi }_j^{(t+1)}=E_g\left[P_j^{(t)}(X)\right]\\ {\mu }_j^{(t+1)}=E_g\left[X{P}_j^{(t)}(X)\right]/{\pi }_j^{(t+1)}\\ {\sigma }_j^{(t+1)}={\left\{E_g\left[X^2{P}_j^{(t)}(X)\right]/{\pi }_j^{(t+1)}-{\left({\mu }_j^{(t+1)}\right)}^2\right\}}^{1/2}\end{array}.$$

The expected values generally need to be evaluated numerically. In this manuscript we use Gauss/Hermite quadrature (see Lange, 1999).

Appendix D. Obtaining standard error estimates of marginal parameters using the Multivariate Delta Method

In this section, we detail how to obtain the standard error estimate for ${\widehat{\beta }}_1^m$ when there is one binary covariate. Note that ${\widehat{\beta }}_1^m$ is a function of ${\widehat{\beta }}_0^c$ and ${\widehat{\beta }}_1^c$:

$${\widehat{\beta }}_1^m=g\left(\begin{array}{l}{\beta }_0^c\hfill \\ {\beta }_1^c\hfill \end{array}\right)=log\left\{\frac{F_q({\widehat{\beta }}_0^c+{\widehat{\beta }}_1^c)}{1-F_q({\widehat{\beta }}_0^c+{\widehat{\beta }}_1^c)}\right\}-log\left\{\frac{F_q({\widehat{\beta }}_0^c)}{1-F_q({\widehat{\beta }}_0^c)}\right\},$$

with gradient

$$\nabla g^t=\left[\begin{array}{c}\frac{f_q({\widehat{\beta }}_0^c+{\widehat{\beta }}_1^c)}{F_q({\widehat{\beta }}_0^c+{\widehat{\beta }}_1^c)\{1-F_q({\widehat{\beta }}_0^c+{\widehat{\beta }}_1^c)\}}-\frac{f_q({\widehat{\beta }}_0^c)}{F_q({\beta }_0^c)\{1-F_q({\beta }_0^c)\}}\\ \frac{f_q({\widehat{\beta }}_0^c+{\widehat{\beta }}_1^c)}{F_q({\widehat{\beta }}_0^c+{\widehat{\beta }}_1^c)\{1-F_q({\widehat{\beta }}_0^c+{\widehat{\beta }}_1^c)\}}\end{array}\right]$$

Since ${({\widehat{\beta }}_0^c,{\widehat{\beta }}_1^c)}^t$ is normally distributed with covariance matrix V_β, say, we can apply the multivariate Delta Method to obtain a standard error estimate of ${\beta }_1^m$ is $\text{SE}({\widehat{\beta }}_1^m)=\nabla g\hspace{0.17em}{\mathbf{V}}_{\beta }\hspace{0.17em}\nabla g^t$.