MM Algorithms for Some Discrete Multivariate Distributions

Abstract

The MM (minorization–maximization) principle is a versatile tool for constructing optimization algorithms. Every EM algorithm is an MM algorithm but not vice versa. This article derives MM algorithms for maximum likelihood estimation with discrete multivariate distributions such as the Dirichlet-multinomial and Connor–Mosimann distributions, the Neerchal–Morel distribution, the negative-multinomial distribution, certain distributions on partitions, and zero-truncated and zero-inflated distributions. These MM algorithms increase the likelihood at each iteration and reliably converge to the maximum from well-chosen initial values. Because they involve no matrix inversion, the algorithms are especially pertinent to high-dimensional problems. To illustrate the performance of the MM algorithms, we compare them to Newton’s method on data used to classify handwritten digits.

1. INTRODUCTION

The MM algorithm generalizes the celebrated EM algorithm (Dempster, Laird, and Rubin 1977). In this article we apply the MM (minorization–maximization) principle to devise new algorithms for maximum likelihood estimation with several discrete multivariate distributions. A series of research papers and review articles (Groenen 1993; de Leeuw 1994; Heiser 1995; Hunter and Lange 2004; Lange 2004; Wu and Lange 2010) have argued that the MM principle can lead to simpler derivations of known EM algorithms. More importantly, the MM principle also generates many new algorithms of considerable utility. Some statisticians encountering the MM principle for the first time react against its abstraction, unfamiliarity, and dependence on the mathematical theory of inequalities. This is unfortunate because real progress can be made applying a few basic ideas in a unified framework. The current article relies on just three well-known inequalities. For most of our examples, the derivation of a corresponding EM algorithm appears much harder, the main hindrance being the difficulty of choosing an appropriate missing data structure.

Discrete multivariate distributions are seeing wider use throughout statistics. Modern data mining employs such distributions in image reconstruction, pattern recognition, document clustering, movie rating, network analysis, and random graphs. High-dimension data demand high-dimensional models with ten to hundreds of thousands of parameters. Newton’s method and Fisher scoring are capable of finding the maximum likelihood estimates of these distributions via the parameter updates

$${\theta }^{(n+1)}={\theta }^{(n)}+M{({\theta }^{(n)})}^{-1}\nabla L({\theta }^{(n)}),$$

where ∇L(θ) is the score function and M(θ) is the observed or the expected information matrix, respectively. Several complications can compromise the performance of these traditional algorithms: (a) the information matrix M(θ) may be expensive to compute, (b) it may fail to be positive definite in Newton’s method, (c) in high dimensions it is expensive to solve the linear system M(θ)x = ∇L(θ⁽ⁿ⁾), and (d) if parameter constraints and parameter bounds intrude, then the update itself requires modification. Although mathematical scientists have devised numerous remedies and safeguards, these all come at a cost of greater implementation complexity. The MM principle offers a versatile weapon for attacking optimization problems of this sort. Although MM algorithms have at best a linear rate of convergence, their updates are often very simple. This can tip the computational balance in their favor. In addition, MM algorithms are typically easy to code, numerically stable, and amenable to acceleration. For the discrete distributions considered here, there is one further simplification often missed in the literature. These distributions involve gamma functions. To avoid the complications of evaluating the gamma function and its derivatives, we fall back on a device suggested by Haldane (1941) that replaces ratios of gamma functions by rising polynomials.

Rather than tire the skeptical reader with more preliminaries, it is perhaps best to move on to our examples without delay. The next section defines the MM principle, discusses our three driving inequalities, and reviews two simple acceleration methods. Section 3 derives MM algorithms for some standard multivariate discrete distributions, namely the Dirichlet-multinomial and Connor–Mosimann distributions, the Neerchal–Morel distribution, the negative-multinomial distribution, certain distributions on partitions, and zero-truncated and zero-inflated distributions. Section 4 describes a numerical experiment comparing the performance of the MM algorithms, accelerated MM algorithms, and Newton’s method on model fitting of handwritten digit data. Our discussion concludes by mentioning directions for further research and by frankly acknowledging the limitations of the MM principle.

2. OVERVIEW OF THE MM ALGORITHM

As we have already emphasized, the MM algorithm is a principle for creating algorithms rather than a single algorithm. There are two versions of the MM principle, one for iterative minimization and another for iterative maximization. Here we deal only with the maximization version. Let f (θ) be the objective function we seek to maximize. An MM algorithm involves minorizing f (θ) by a surrogate function g(θ|θⁿ) anchored at the current iterate θⁿ of a search. Minorization is defined by the two properties

$$f({\theta }^n)=g({\theta }^n|{\theta }^n),$$ (2.1)

$$f(\theta )\ge g(\theta |{\theta }^n),\theta \ne {\theta }^n.$$ (2.2)

In other words, the surface θ ↦ g(θ|θⁿ) lies below the surface θ ↦ f (θ) and is tangent to it at the point θ = θⁿ. Construction of the surrogate function g(θ|θⁿ) constitutes the first M of the MM algorithm.

In the second M of the algorithm, we maximize the surrogate g(θ|θⁿ) rather than f (θ). If θⁿ⁺¹ denotes the maximum point of g(θ|θⁿ), then this action forces the ascent property f (θⁿ⁺¹) ≥ f (θⁿ). The straightforward proof

$$f({\theta }^{n+1})\ge g({\theta }^{n+1}|{\theta }^n)\ge g({\theta }^n|{\theta }^n)=f({\theta }^n)$$

reflects definitions (2.1) and (2.2) and the choice of θⁿ⁺¹. The ascent property is the source of the MM algorithm’s numerical stability. Strictly speaking, it depends only on increasing g(θ|θⁿ), not on maximizing g(θ|θⁿ).

The art in devising an MM algorithm revolves around intelligent choices of minorizing functions. This brings us to the first of our three basic minorizations

$$\text{ln}\left({\displaystyle \sum _{i=1}^m{\alpha }_i}\right)\ge {\displaystyle \sum _{i=1}^m\frac{{\alpha }_i^n}{{\displaystyle {\sum }_{j=1}^m{\alpha }_j^n}}}\text{ln}\left(\frac{{\displaystyle {\sum }_{j=1}^m{\alpha }_j^n}}{{\alpha }_i^n}{\alpha }_i\right),$$ (2.3)

invoking the chord below the graph property of the concave function ln x. Note here that all parameter values are positive and that equality obtains whenever ${\alpha }_i={\alpha }_i^n$ for all i. Our second basic minorization

$$-\text{ln}(c+\alpha )\ge -\text{ln}(c+{\alpha }^n)-\frac{1}{c+{\alpha }^n}(\alpha -{\alpha }^n)$$ (2.4)

restates the supporting hyperplane property of the convex function −ln(c + x). Our final basic minorization

$$-\text{ln}(1-\alpha )\ge -\text{ln}(1-{\alpha }^n)+\frac{{\alpha }^n}{1-{\alpha }^n}\text{ln}\left(\frac{\alpha }{{\alpha }^n}\right)$$ (2.5)

is just a rearrangement of the two-point information inequality

$${\alpha }^n\text{ln}{\alpha }^n+(1-{\alpha }^n)\text{ln}(1-{\alpha }^n)\ge {\alpha }^n\text{ln}\alpha +(1-{\alpha }^n)\text{ln}(1-\alpha ).$$

Here α and αⁿ must lie in (0, 1). Any standard text on inequalities, for example, the book by Steele (2004), proves these three inequalities. Because piecemeal minorization works well, our derivations apply the basic minorizations only to strategic parts of the overall objective function, leaving other parts untouched.

The convergence theory of MM algorithms is well known (Lange 2004). Convergence to a stationary point is guaranteed provided five properties of the objective function f (θ) and the MM algorithm map M(θ) hold: (a) f (θ) is coercive on its open domain; (b) f (θ) has only isolated stationary points; (c) M(θ) is continuous; (d) θ* is a fixed point of M(θ) if and only if it is a stationary point of f (θ); (e) f [M(θ*)] ≤ f (θ*), with equality if and only if θ* is a fixed point of M(θ). Most of these conditions are easy to verify for our examples, so the details will be omitted.

A common criticism of EM and MM algorithms is their slow convergence. Fortunately, MM algorithms can be easily accelerated (Jamshidian and Jennrich 1995; Lange 1995a; Jamshidian and Jennrich 1997; Varadhan and Rolland 2008). We will employ two versions of the recent square iterative method (SQUAREM) developed by Varadhan and Roland (2008). These simple vector extrapolation techniques require computation of two MM updates at each iteration. Denote the two updates by M(θⁿ) and M ◦ M(θⁿ), where M(θ) is the MM algorithm map. These updates in turn define two vectors

$$u=M({\theta }^n)-{\theta }^n,\upsilon =M◦M({\theta }^n)-M({\theta }^n)-u.$$

The versions diverge in how they compute the steplength constant s. SqMPE1 (minimal polynomial extrapolation) takes $s=\frac{u^{t}u}{u^t\upsilon }$ , while SqRRE1 (reduced rank extrapolation) takes $s=\frac{u^t\upsilon }{u^t\upsilon }$ . Once s is specified, we define the next accelerated iterate by θⁿ⁺¹ = θⁿ − 2su + s²υ. Readers should consult the original article for motivation of SQUAREM. Whenever θⁿ⁺¹ decreases the log-likelihood L(θ), we revert to the MM update θⁿ⁺¹ =M ◦ M(θⁿ). Finally, we declare convergence when

$$\frac{|L({\theta }^n)-L({\theta }^{n-1})|}{|L({\theta }^{n-1})|+1}<\epsilon .$$ (2.6)

In the numerical examples that follow, we use the stringent criterion ε = 10⁻⁹. More sophisticated stopping criteria based on the gradient of the objective function and the norm of the parameter increment lead to similar results.

3. APPLICATIONS

3.1 Dirichlet-Multinomial and Connor–Mosimann Distributions

When count data exhibit overdispersion, the Dirichlet-multinomial distribution is often substituted for the multinomial distribution. The multinomial distribution is characterized by a vector p = (p₁,…, p_d) of cell probabilities and a total number of trials m. In the Dirichlet-multinomial sampling, p is first drawn from a Dirichlet distribution with parameter vector α = (α₁,…,α_d). Once the cell probabilities are determined, multinomial sampling commences. This leads to the admixture density

$$\begin{array}{cc}h(\mathbf{x}|\alpha )\hfill & =\frac{\mathrm{\Gamma }(|\alpha |)}{\mathrm{\Gamma }({\alpha }_1)\cdots \mathrm{\Gamma }({\alpha }_d)}{\displaystyle {\int }_{{\mathrm{\Delta }}_d}\left(\begin{array}{c}\hfill m\hfill \\ \hfill \mathbf{x}\hfill \end{array}\right){\displaystyle \prod _{i=1}^d{p}_i^{x_i+{\alpha }_i-1}{\mathit{\text{dp}}}_1\cdots {\mathit{\text{dp}}}_d}}\hfill \\ \hfill & =\left(\begin{array}{c}\hfill m\hfill \\ \hfill \mathbf{x}\hfill \end{array}\right)\frac{\mathrm{\Gamma }({\alpha }_1+x_1)\cdots \mathrm{\Gamma }({\alpha }_d+x_d)}{\mathrm{\Gamma }(|\alpha |+m)}\frac{\mathrm{\Gamma }(|\alpha |)}{\mathrm{\Gamma }({\alpha }_1)\cdots \mathrm{\Gamma }({\alpha }_d)},\hfill \end{array}$$ (3.1)

where $\left|\alpha \right|={\displaystyle {\sum }_{i=1}^d{\alpha }_i}$ , Δ_d is the unit simplex in ℝ^d, and x = (x₁,…, x_d) is the vector of cell counts. Note that the count total $\left|x\right|={\displaystyle {\sum }_{i=1}^d{x}_i}$ is fixed at m. Standard calculations show that a random vector X drawn from h(x|α) has the means, variances, and covariances

$$\begin{array}{cc}\hfill \mathbf{E}({\mathbf{X}}_i)& =m\frac{{\alpha }_i}{\left|\alpha \right|},\hfill \\ \hfill \mathbf{\text{Var}}({\mathbf{X}}_i)& =m\frac{{\alpha }_i}{\left|\alpha \right|}\left(1-\frac{{\alpha }_i}{\left|\alpha \right|}\right)\frac{\left|\alpha \right|+m}{\left|\alpha \right|+1},\hfill \\ \hfill \mathbf{\text{Cov}}({\mathbf{X}}_i,{\mathbf{X}}_j)& =-m\frac{{\alpha }_i}{\left|\alpha \right|}\frac{{\alpha }_j}{\left|\alpha \right|}\frac{\left|\alpha \right|+m}{\left|\alpha \right|+1},i\ne j.\hfill \end{array}$$

If the fractions $\frac{{\alpha }_i}{\left|\alpha \right|}$ tend to constants p_i as |α| tends to ∞, then these moments collapse to the corresponding moments of the multinomial distribution with proportions p₁,…, p_d.

One of the most unappealing features of the density function h(x|α) is the occurrence of the gamma function. Fortunately, very early on Haldane (1941) noted the alternative representation

$$h(\mathbf{x}|\alpha )=\left(\begin{array}{c}\hfill m\hfill \\ \hfill \mathbf{x}\hfill \end{array}\right)\frac{{\displaystyle {\prod }_{j=1}^d{\alpha }_j({\alpha }_j+1)\cdots ({\alpha }_j+x_j-1)}}{\left|\alpha \right|(|\alpha |+1)\cdots (|\alpha |+m-1)}.$$ (3.2)

The replacement of gamma functions by rising polynomials is a considerable gain in simplicity. Bailey (1957) later suggested the reparameterization

$${\pi }_j=\frac{{\alpha }_j}{\left|\alpha \right|},j=1,\dots ,d,\theta =\frac{1}{\left|\alpha \right|}$$

in terms of the proportion vector π = (π₁,…, π_d) and the overdispersion parameter θ. In this setting, the discrete density function becomes

$$h(\mathbf{x}|\pi ,\theta )=\left(\begin{array}{c}\hfill m\hfill \\ \hfill \mathbf{x}\hfill \end{array}\right)\frac{{\displaystyle {\prod }_{j=1}^d{\pi }_j({\pi }_j+\theta )\cdots [{\pi }_j+(x_j-1)}\theta ]}{(1+\theta )\cdots [1+(m-1)\theta ]}.$$ (3.3)

This version of the density function is used to good effect by Griffiths (1973) in implementing Newton’s method for maximum likelihood estimation with the beta-binomial distribution.

In maximum likelihood estimation, we pass to log-likelihoods. This introduces logarithms and turns factors into sums. To construct an MM algorithm under the parameterization (3.2), we need to minorize terms such as ln(α_j + k) and −ln(|α| + k). The basic inequalities (2.3) and (2.4) are directly relevant. Suppose we draw t independent samples x₁,…, x_t from the Dirichlet-multinomial distribution with m_i trials for sample i. The term −ln(|α| + k) occurs in the log-likelihood for x_i if and only if m_i ≥ k + 1. Likewise the term ln(α_j + k) occurs in the log-likelihood for x_i if and only if x_ij ≥ k + 1. It follows that the log-likelihood for the entire sample can be written as

$$\begin{array}{c}\hfill L(\alpha )=-{\displaystyle \sum _k{r}_k\text{ln}(\left|\alpha \right|+k)+{\displaystyle \sum _j{\displaystyle \sum _k{s}_{\mathit{\text{jk}}}\text{ln}({\alpha }_j+k),}}}\hfill \\ \hfill r_k={\displaystyle \sum _i{1}_{\{m_i\ge k+1\}},s_{\mathit{\text{jk}}}=}{\displaystyle \sum _i{1}_{\{x_{\mathit{\text{ij}}}\ge k+1\}}.}\hfill \end{array}$$ (3.4)

The index k in these formulas ranges from 0 to max_i m_i −1.

Applying our two basic minorizations to L(α) yields the surrogate function

$$g(\alpha |{\alpha }^n)=-{\displaystyle \sum _k{r}_k}\frac{1}{\left|{\alpha }^n\right|+k}\left|\alpha \right|+{\displaystyle \sum _j{\displaystyle \sum _k{s}_{\mathit{\text{jk}}}\frac{{\alpha }_j^n}{{\alpha }_j^n+k}\text{ln}{\alpha }_j}}$$

up to an irrelevant additive constant. Equating the partial derivative of the surrogate with respect to α_j to 0 produces the simple MM update

$${\alpha }_j^{n+1}=\left({\displaystyle \sum _k\frac{s_{\mathit{\text{jk}}}{\alpha }_j^n}{{\alpha }_j^n+k}}\right)/\left({\displaystyle \sum _k\frac{r_k}{\left|{\alpha }^n\right|+k}}\right).$$ (3.5)

Minka (2003) derived these updates from a different perspective.

Under the parameterization (3.3), matters are slightly more complicated. Now we minorize the terms −ln(1 + kθ) and ln(π_j + kθ) via

$$-\text{log}(1+k\theta )\ge -\text{log}(1+k{\theta }^n)-\frac{1}{1+k{\theta }^n}(k\theta -k{\theta }^n)$$

and

$$\text{log}({\pi }_j+k\theta )\ge \frac{{\pi }_j^n}{{\pi }_j^n+k{\theta }^n}\text{log}\left(\frac{{\pi }_j^n+k{\theta }^n}{{\pi }_j^n}{\pi }_j\right)+\frac{k{\theta }^n}{{\pi }_j^n+k{\theta }^n}\text{log}\left(\frac{{\pi }_j^n+k{\theta }^n}{k{\theta }^n}k\theta \right).$$

These minorizations lead to the surrogate function

$$-{\displaystyle \sum _k{r}_k}\frac{k}{1+k{\theta }^n}\theta +{\displaystyle \sum _j{\displaystyle \sum _k{s}_{\mathit{\text{jk}}}}}\left\{\frac{{\pi }_j^n}{{\pi }_j^n+k{\theta }^n}\text{log}{\pi }_j+\frac{k{\theta }^n}{{\pi }_j^n+k{\theta }^n}\text{log}\theta \right\}$$

up to an irrelevant constant. Setting the partial derivative with respect to θ equal to 0 yields the MM update

$${\theta }^{n+1}=\left({\displaystyle \sum _j{\displaystyle \sum _k\frac{s_{\mathit{\text{jk}}}k{\theta }^n}{{\pi }_j^n+k{\theta }^n}}}\right)/\left({\displaystyle \sum _k\frac{r_{k}k}{1+k{\theta }^n}}\right).$$ (3.6)

The update of the proportion vector π must be treated as a Lagrange multiplier problem owing to the constraint Σ_j π_j = 1. Familiar arguments produce the MM update

$${\pi }_j^{n+1}=\left({\displaystyle \sum _k\frac{s_{\mathit{\text{jk}}}{\pi }_j^n}{{\pi }_j^n+k{\theta }^n}}\right)/\left({\displaystyle \sum _l{\displaystyle \sum _k\frac{s_{\mathit{\text{lk}}}{\pi }_l^n}{{\pi }_l^n+k{\theta }^n}}}\right).$$ (3.7)

The two updates summarized by (3.5), (3.6), and (3.7) enjoy several desirable properties. First, parameter constraints are built in. Second, stationary points of the log-likelihood are fixed points of the updates. Virtually all MM algorithms share these properties. The update (3.7) also reduces to the maximum likelihood estimate

$${\widehat{\pi }}_j=\frac{{\displaystyle {\sum }_k{s}_{\mathit{\text{jk}}}}}{{\displaystyle {\sum }_l{\displaystyle {\sum }_k{s}_{\mathit{\text{lk}}}}}}=\frac{{\displaystyle {\sum }_i{x}_{\mathit{\text{ij}}}}}{{\displaystyle {\sum }_i{m}_i}}$$ (3.8)

of the corresponding multinomial proportion when θⁿ = 0.

The estimate (3.8) furnishes a natural initial value ${\pi }_j^0$ . To derive an initial value for the overdispersion parameter θ, consider the first two moments

$$\mathbf{E}(P_j)=\frac{{\alpha }_j}{\left|\alpha \right|},\mathbf{E}(P_j^2)=\frac{{\alpha }_j({\alpha }_j+1)}{\left|\alpha \right|(\left|\alpha \right|+1)}$$

of a Dirichlet distribution with parameter vector α. These identities imply that

$${\displaystyle \sum _{j=1}^d\frac{\mathbf{E}(P_j^2)}{\mathbf{E}(P_j)}}=\frac{\left|\alpha \right|+d}{\left|\alpha \right|+1}=\rho ,$$

which can be solved for θ = 1/|α| in terms of ρ as θ = (ρ − 1)/(d − ρ). Substituting the estimate

$$\widehat{\rho }={\displaystyle \sum _j\frac{{{\displaystyle {\sum }_i(x_{\mathit{\text{ij}}}/m_i)}}^2}{{\displaystyle {\sum }_i(x_{\mathit{\text{ij}}}/m_i)}}}$$

for ρ gives a sensible initial value θ⁰.

To test our two MM algorithms, we now turn to the beta-binomial data of Haseman and Soares (1976) on male mice exposed to various mutagens. The two outcome categories are (a) dead implants and (b) survived implants. In their first dataset, there are t = 524 observations with between m = 1 and m = 20 trials per observation. Table 1 presents the final log-likelihood, number of iterations, and running time (in seconds) of the two MM algorithms and their SQUAREM accelerations on these data. All MM algorithms converge to the maximum point previously found by the scoring method (Paul, Balasooriya, and Banerjee 2005). For the choice ε = 10⁻⁹ in stopping criterion (2.6), the MM algorithm (3.5) takes 700 iterations and 0.1580 sec to converge on a laptop computer. The alternative MM algorithm given in the updates (3.6) and (3.7) takes 339 iterations and 0.1626 sec. Figure 1 depicts the progress of the MM iterates on a contour plot of the log-likelihood. The conventional MM algorithm crawls slowly along the ridge in the contour plot; the accelerated versions SqMPE1 and SqRRE1 significantly reduce both the number of iterations and the running time until convergence.

Table 1.

MM algorithms for the Haseman and Soares beta-binomial data.

	(π, θ) parameterization			α parameterization
Algorithm	L	# iters	Time	L	# iters	Time
MM	−777.79	339	0.1626	−777.79	700	0.1580
SqMPE1 MM	−777.79	10	0.0093	−777.79	14	0.0105
SqRRE1 MM	−777.79	18	0.0159	−777.79	14	0.0100

Figure 1.

MM Ascent of the Dirichlet-multinomial log-likelihood surface. A color version of this figure is available in the electronic version of this article.

The Dirichlet-multinomial distribution suffers from two restrictions that limit its applicability, namely the negative correlation of coordinates and the determination of variances by means. It is possible to overcome these restrictions by choosing a more flexible mixing distribution as a prior for the multinomial. Connor and Mosimann (1969) suggested a generalization of the Dirichlet distribution that meets this challenge. The resulting admixed distribution, called the generalized Dirichlet-multinomial distribution, has proved its worth in machine learning problems such as the modeling and clustering of images, handwritten digits, and text documents (Bouguila 2008). It is therefore helpful to derive an MM algorithm for maximum likelihood estimation with this distribution that avoids the complications of gamma/digamma/trigamma functions arising with Newton’s method (Bouguila 2008). The Connor–Mosimann distribution is constructed inductively by the mechanism of stick breaking. Imagine breaking the interval [0, 1] into d subintervals of lengths P₁,…, P_d by choosing d − 1 independent beta variates Z_i with parameters α_i and β_i. The length of subinterval 1 is P₁ = Z₁. Given P₁ through P_i, the length of subinterval i + 1 is P_i+1 = Z_i+1(1 − P₁ − ⋯−P_i). The last length P_d = 1 − (P₁ + ⋯ + P_d−1) takes up the slack. Standard calculations show that the P_i have the joint density

$$g(\mathbf{p}|\alpha ,\beta )={\displaystyle \prod _{j=1}^{d-1}\frac{\mathrm{\Gamma }({\alpha }_j+{\beta }_j)}{\mathrm{\Gamma }({\alpha }_j)\mathrm{\Gamma }({\beta }_j)}{p}_j^{{\alpha }_j-1}{\left(1-{\displaystyle \sum _{i=1}^j{p}_i}\right)}^{{\gamma }_j},\mathbf{p}\in {\mathrm{\Delta }}_d,}$$

where γ_j = β_j − α_j+1 − β_j+1 for j = 1,…, d − 2 and γ_d−1 = β_d−1 − 1. The univariate case (d = 2) corresponds to the beta distribution. The Dirichlet distribution is recovered by taking β_j = α_j+1 +⋯+α_d. With d − 2 more parameters than the Dirichlet distribution, the Connor–Mosimann distribution is naturally more versatile.

The Connor–Mosimann distribution is again conjugate to the multinomial distribution, and the marginal density of a count vector X over m trials is easily shown to be

$$\begin{array}{cc}\text{Pr}(\mathbf{X}=\mathbf{x})\hfill & ={\displaystyle {\int }_{{\mathrm{\Delta }}_d}\left(\begin{array}{c}\hfill m\hfill \\ \hfill \mathbf{x}\hfill \end{array}\right)}{\displaystyle \prod _{j=1}^{d-1}{p}_j^{x_j}g(\mathbf{p}|\alpha ,\beta )d\mathbf{p}}\hfill \\ \hfill & =\left(\begin{array}{c}\hfill m\hfill \\ \hfill \mathbf{x}\hfill \end{array}\right){\displaystyle \prod _{j=1}^{d-1}\frac{\mathrm{\Gamma }({\alpha }_j+x_j)}{\mathrm{\Gamma }({\alpha }_j)}\frac{\mathrm{\Gamma }({\beta }_j+y_{j+1})}{\mathrm{\Gamma }({\beta }_j)}\frac{\mathrm{\Gamma }({\alpha }_j+{\beta }_j)}{\mathrm{\Gamma }({\alpha }_j+{\beta }_j+y_j)},}\hfill \end{array}$$

where $y_j={\displaystyle {\sum }_{k=j}^d{x}_k}$ . If we adopt the reparameterization

$${\theta }_j=\frac{1}{{\alpha }_j+{\beta }_j},{\pi }_j=\frac{{\alpha }_j}{{\alpha }_j+{\beta }_j},j=1,\dots ,d-1,$$

and use the fact that x_j + y_j+1 = y_j, then the density can be re-expressed as

$$\left(\begin{array}{c}\hfill m\hfill \\ \hfill \mathbf{x}\hfill \end{array}\right){\displaystyle \prod _{j=1}^{d-1}\frac{{\pi }_j\cdots [{\pi }_j+(x_j-1){\theta }_j]\times (1-{\pi }_j)\cdots [1-{\pi }_j+(y_{j+1}-1){\theta }_j]}{1\cdots [1+(y_j-1){\theta }_j]}}.$$ (3.9)

Thus, maximum likelihood estimation of the parameter vectors π = (π₁,…, π_d−1) and θ = (θ₁,…, θ_d−1) by the MM algorithm reduces to the case of d − 1 independent beta-binomial problems.

Let x₁,…, x_t be a random sample from the generalized Dirichlet-multinomial distribution (3.9) with m_i trials for observation x_i. Following our reasoning for estimation with the Dirichlet-multinomial, we define the associated counts

$$r_{\mathit{\text{jk}}}={\displaystyle \sum _{i=1}^t{1}_{\{x_{\mathit{\text{ij}}}\ge k+1\},}{s}_{\mathit{\text{jk}}}={\displaystyle \sum _{i=1}^t{1}_{\{y_{\mathit{\text{ij}}}\ge k+1\}}}}$$

for 1 ≤ j ≤ d − 1. In this notation, the reader can readily check that the MM updates become

$$\begin{array}{c}{\pi }_j^{n+1}=\left({\displaystyle \sum _k\frac{r_{\mathit{\text{jk}}}{\pi }_j^n}{{\pi }_j^n+k{\theta }_j^n}}\right)/\left({\displaystyle \sum _k\left[\frac{r_{\mathit{\text{jk}}}{\pi }_j^n}{{\pi }_j^n+k{\theta }_j^n}+\frac{s_{j+1,k}(1-{\pi }_j^n)}{1-{\pi }_j^n+k{\theta }_j^n}\right]}\right),\hfill \\ {\theta }_j^{n+1}=\left({\displaystyle \sum _k\left[\frac{r_{\text{jk}}k{\theta }_j^n}{{\pi }_j^n+k{\theta }_j^n}+\frac{s_{j+1,k}k{\theta }_j^n}{1-{\pi }_j^n+k{\theta }_j^n}\right]}\right)/\left({\displaystyle \sum _k\frac{s_{\mathit{\text{jk}}}}{1+k{\theta }_j^n}}\right).\hfill \end{array}$$

3.2 Neerchal–Morel Distribution

Neerchal and Morel (1998, 2005) proposed an alternative to the Dirichlet-multinomial distribution that accounts for overdispersion by finite admixture. If x represents count data over m trials and d categories, then their discrete density is

$$h(\mathbf{x}|\pi ,\rho )={\displaystyle \sum _{j=1}^d{\pi }_j\left(\begin{array}{c}\hfill m\hfill \\ \hfill \mathbf{x}\hfill \end{array}\right){[(1-\rho ){\pi }_1]}^{x_1}\cdots {[(1-\rho ){\pi }_j+\rho ]}^{x_j}\cdots {[(1-\rho ){\pi }_d]}^{x_d},}$$ (3.10)

where π = (π₁,…, π_d) is a vector of proportions and ρ ∈ [0, 1] is an overdispersion parameter. The Neerchal–Morel distribution collapses to the multinomial distribution when ρ = 0. Straightforward calculations show that the Neerchal–Morel distribution has means, variances, and covariances

$$\begin{array}{cc}\hfill \mathbf{E}({\mathbf{X}}_i)& =m{\pi }_i,\hfill \\ \hfill \mathbf{\text{Var}}({\mathbf{X}}_i)& =m{\pi }_i(1-{\pi }_i)[1-{\rho }^2+m{\rho }^2],\hfill \\ \hfill \mathbf{\text{Cov}}({\mathbf{X}}_i,{\mathbf{X}}_j)& =-m{\pi }_i{\pi }_j[1-{\rho }^2+m{\rho }^2],i\ne j.\hfill \end{array}$$

These are precisely the same as the first- and second-order moments of the Dirichlet-multinomial distribution provided we identify π_i = α_i/|α| and ρ₂ = 1/(|α| + 1).

If we draw t independent samples x₁,…, x_t from the Neerchal–Morel distribution with m_i trials for sample i, then the log-likelihood is

$${\displaystyle \sum _i\text{ln}\left\{{\displaystyle \sum _j{\pi }_j\left(\begin{array}{c}\hfill m_i\hfill \\ \hfill {\mathbf{x}}_i\hfill \end{array}\right){[(1-\rho ){\pi }_1]}^{x_{i1}}\cdots {[(1-\rho ){\pi }_j+\rho ]}^{x_{\mathit{\text{ij}}}}\cdots {[(1-\rho ){\pi }_d]}^{x_{\mathit{\text{id}}}}}\right\}}.$$ (3.11)

It is worth bearing in mind that every mixture model yields to the minorization (2.3). This is one of the secrets to the success of the EM algorithm. As a practical matter, explicit minorization via inequality (2.3) is more mechanical and often simpler to implement than performing the E step of the EM algorithm. This is particularly true when several minorizations intervene before we reach the ideal surrogate. Here two successive minorizations are needed.

To state the first minorization, let us abbreviate

$${\mathrm{\Pi }}_{\mathit{\text{ij}}}={\pi }_j{[(1-\rho ){\pi }_1]}^{x_{i1}}\cdots {[(1-\rho ){\pi }_j+\rho ]}^{x_{\mathit{\text{ij}}}}\cdots {[(1-\rho ){\pi }_d]}^{x_{\mathit{\text{id}}}}$$

and denote by ${\prod }_{\mathit{\text{ij}}}^n$ the same quantity evaluated at the nth iterate. In this notation it follows that

$$\text{ln}\left({\displaystyle \sum _j{\mathrm{\Pi }}_{\mathit{\text{ij}}}}\right)\ge {\displaystyle \sum _j{w}_{\mathit{\text{ij}}}^n\text{ln}\left(\frac{{\mathrm{\Pi }}_{\mathit{\text{ij}}}}{w_{\mathit{\text{ij}}}^n}\right)}={\displaystyle \sum _j{w}_{\mathit{\text{ij}}}^n\text{ln}{\mathrm{\Pi }}_{\mathit{\text{ij}}}-{\displaystyle \sum _j{w}_{\mathit{\text{ij}}}^n\text{ln}{w}_{\mathit{\text{ij}}}^n}}$$

with weights $w_{\mathit{\text{ij}}}^n=\frac{{\prod }_{\mathit{\text{ij}}}^n}{{\displaystyle {\sum }_l{\prod }_{\mathit{\text{ij}}}^n}}$ . The logarithm splits ln ∏_ij into the sum

$$\text{ln}{\mathrm{\Pi }}_{\mathit{\text{ij}}}=m_i\text{ln}(1-\rho )+\text{ln}{\pi }_j+x_{i1}\text{ln}{\pi }_1+\cdots +x_{\mathit{\text{ij}}}\text{ln}({\pi }_j+\theta )+\cdots +x_{\mathit{\text{id}}}\text{ln}{\pi }_d$$

for θ = ρ/(1 − ρ). To separate the parameters π_j and θ in the troublesome term ln(π_j + θ), we apply the minorization (2.3) again. This produces

$$\text{ln}({\pi }_j+\theta )\ge \frac{{\pi }_j^n}{{\pi }_j^n+{\theta }^n}\text{ln}\left(\frac{{\pi }_j^n+{\theta }^n}{{\pi }_j^n}{\pi }_j\right)+\frac{{\theta }^n}{{\pi }_j^n+{\theta }^n}\text{ln}\left(\frac{{\pi }_j^n+{\theta }^n}{{\theta }^n}\theta \right),$$

and up to a constant the surrogate function takes the form

$${\displaystyle \sum _i{\displaystyle \sum _j{w}_{\mathit{\text{ij}}}^n\left[{\displaystyle \sum _k{x}_{\mathit{\text{ik}}}\text{ln}{\pi }_k}+\left(1-\frac{x_{\mathit{\text{ij}}}{\theta }^n}{{\pi }_j^n+{\theta }^n}\right)\text{ln}{\pi }_j\right]}}+{\displaystyle \sum _i{\displaystyle \sum _j{w}_{\mathit{\text{ij}}}^n\left[\left(m_i-\frac{x_{\mathit{\text{ij}}}{\theta }^n}{{\pi }_j^n+{\theta }^n}\right)\text{ln}(1-\rho )+\frac{x_{\mathit{\text{ij}}}{\theta }^n}{{\pi }_j^n+{\theta }^n}\text{ln}\rho \right].}}$$

Standard arguments now yield the updates

$$\begin{array}{c}{\pi }_k^{n+1}=\left({\displaystyle \sum _i{\displaystyle \sum _j{w}_{\mathit{\text{ij}}}^n{x}_{\mathit{\text{ik}}}}+{\displaystyle \sum _j{w}_{\mathit{\text{ik}}}^n\left(1-\frac{x_{\mathit{\text{ik}}}{\theta }^n}{{\pi }_k^n+{\theta }^n}\right)}}\right)/\left({\displaystyle \sum _l{\displaystyle \sum _i{\displaystyle \sum _j{w}_{\mathit{\text{ij}}}^n{x}_{\mathit{\text{il}}}}}+{\displaystyle \sum _l{\displaystyle \sum _i{w}_{\mathit{\text{il}}}^n\left(1-\frac{x_{\mathit{\text{il}}}{\theta }^n}{{\pi }_l^n+{\theta }^n}\right)}}}\right),\hfill \\ \begin{array}{cc}{\rho }^{n+1}=\left({\displaystyle \sum _i{\displaystyle \sum _j\frac{w_{\mathit{\text{ij}}}^n{x}_{\mathit{\text{ij}}}{\theta }^n}{{\pi }_j^n+{\theta }^n}}}\right)/\left({\displaystyle \sum _i{m}_i}\right),\hfill & {\theta }^{n+1}=\frac{{\rho }^{n+1}}{1-{\rho }^{n+1}}.\hfill \end{array}\hfill \end{array}$$

Table 2 lists convergence results for this MM algorithm and its SQUAREM accelerations on the previously discussed Haseman and Soares data.

Table 2.

Performance of the Neerchal–Morel MM algorithms.

Algorithm	L	# iters	Time
MM	−783.29	128	0.2289
SqMPE1 MM	−783.29	10	0.0207
SqRRE1 MM	−783.29	11	0.0221

3.3 Negative-Multinomial

The motivation for the negative-multinomial distribution comes from multinomial sampling with d + 1 categories assigned probabilities π₁,…, π_d+1. Sampling continues until category d + 1 accumulates β outcomes. At that moment we count the number of outcomes x_i falling in category i for 1 ≤ i ≤ d. For a given vector x = (x₁,…, x_d ), elementary combinatorics gives the probability

$$\begin{array}{cc}h(\mathbf{x}|\beta ,\pi )\hfill & =\left(\begin{array}{c}\hfill \beta +\left|\mathbf{x}\right|-1\hfill \\ \hfill \left|\mathbf{x}\right|\hfill \end{array}\right)\left(\begin{array}{c}\hfill \left|\mathbf{x}\right|\hfill \\ \hfill \mathbf{x}\hfill \end{array}\right){\displaystyle \prod _{i=1}^d{\pi }_i^{x_i}{\pi }_{d+1}^{\beta }}\hfill \\ \hfill & =\frac{\beta (\beta +1)\cdots (\beta +|\mathbf{x}|-1)}{x_1!\cdots x_d!}{\displaystyle \prod _{i=1}^d{\pi }_i^{x_i}{\pi }_{d+1}^{\beta }}.\hfill \end{array}$$ (3.12)

This formula continues to make sense even if the positive parameter β is not an integer. For arbitrary β > 0, the most straightforward way to construct the negative-multinomial distribution is to run d independent Poisson processes with intensities π₁,…, π_d. Wait a gamma distributed length of time with shape parameter β and intensity parameter π_d+1. At the expiration of this waiting time, count the number of random events X_i of each type i among the first d categories. The random vector X has precisely the discrete density (3.12).

The Poisson process perspective readily yields the moments

$$\begin{array}{cc}\hfill \mathbf{E}(X_i)& =\beta \frac{{\pi }_i}{{\pi }_{d+1}},\hfill \\ \hfill \mathbf{\text{Var}}(X_i)& =\beta \frac{{\pi }_i}{{\pi }_{d+1}}\left(1+\frac{{\pi }_i}{{\pi }_{d+1}}\right),\hfill \\ \hfill \mathbf{\text{Cov}}(X_i,X_j)& =\beta \frac{{\pi }_i}{{\pi }_{d+1}}\frac{{\pi }_j}{{\pi }_{d+1}},i\ne j.\hfill \end{array}$$ (3.13)

Compared to a Poisson distributed random variable with the same mean, the component X_i is overdispersed. Also in contrast to the multinomial and Dirichlet-multinomial distributions, the counts from a negative-multinomial are positively correlated. Negative-multinomial sampling is therefore appealing in many applications.

Let x₁,…, x_t be a random sample from the negative-multinomial distribution with m_i = |x_i|. To maximize the log-likelihood

$$\begin{array}{cc}\hfill L(\beta ,\pi )& ={\displaystyle \sum _k{r}_k\text{ln}(\beta +k)+{\displaystyle \sum _{j=1}^d{x}_{·j}\text{ln}{\pi }_j+t\beta \text{ln}{\pi }_{d+1}-{\displaystyle \sum _i{\displaystyle \sum _j\text{ln}{x}_{\mathit{\text{ij}}}!,}}}}\hfill \\ \hfill r_k& ={\displaystyle \sum _i{1}_{\{m_i\ge k+1\},}}{x}_{·j}={\displaystyle \sum _i{x}_{\mathit{\text{ij}}},}\hfill \end{array}$$

we must deal with the terms ln(β + k). Fortunately, the minorization (2.4) implies

$$\text{ln}(\beta +k)\ge \frac{{\beta }^n}{{\beta }^n+k}\text{ln}\left(\frac{{\beta }^n+k}{{\beta }^n}\beta \right)+\frac{k}{{\beta }^n+k}\text{ln}\left(\frac{{\beta }^n+k}{k}k\right),$$

leading to the surrogate function

$$g(\beta ,\pi |{\beta }^n,{\pi }^n)={\displaystyle \sum _k{r}_k}\frac{{\beta }^n}{{\beta }^n+k}\text{ln}\beta +{\displaystyle \sum _{j=1}^d{x}_{·j}\text{ln}{\pi }_j+t\beta \text{ln}{\pi }_{d+1}}$$

up to an irrelevant constant. In view of the constraint ${\pi }_{d+1}=1-{\displaystyle {\sum }_{j=1}^d{\pi }_j}$ the stationarity conditions for a maximum of the surrogate reduce to

$$0=\frac{1}{\beta }{\displaystyle \sum _k{r}_k\frac{{\beta }^n}{{\beta }^n+k}+t\text{ln}{\pi }_{d+1},0=\frac{x_{·j}}{{\pi }_j}-\frac{t\beta }{{\pi }_{d+1}},1\le j\le d.}$$ (3.14)

Unfortunately, it is impossible to solve this system of equations analytically. There are two resolutions to the dilemma. One is block relaxation (de Leeuw 1994) alternating the updates

$${\beta }^{n+1}=-\left({\displaystyle \sum _k{r}_k}\frac{{\beta }^n}{{\beta }^n+k}\right)/(t\text{ln}{\pi }_{d+1}^n)$$

and

$${\pi }_{d+1}^{n+1}=\frac{t{\beta }^{n+1}}{{\displaystyle {\sum }_{k=1}^d{x}_{·k}+t{\beta }^{n+1}}},{\pi }_j^{n+1}=\frac{x_{·j}}{{\displaystyle {\sum }_{k=1}^d{x}_{·k}+t{\beta }^{n+1}}},1\le j\le d.$$

This strategy enjoys the ascent property of all MM algorithms.

The other possibility is to solve the stationarity equations numerically. It is clear that the system of equations (3.14) reduces to the single equation

$$0=\frac{1}{\beta }{\displaystyle \sum _k{r}_k}\frac{{\beta }^n}{{\beta }^n+k}+t\text{ln}\left(\frac{t\beta }{{\displaystyle {\sum }_{k=1}^d{x}_{·k}+t\beta }}\right)$$

for β. Equivalently, if we let

$$\alpha ={\beta }^{-1},\overline{m}=\frac{1}{t}{\displaystyle \sum _i{\displaystyle \sum _j{x}_{\mathit{\text{ij}}},}{c}^n={\displaystyle \sum _k{r}_k}\frac{{\beta }^n}{{\beta }^n+k},}$$

then we must find a root of the equation f (α) = αcⁿ − t ln(αm̄ + 1) = 0. It is clear that f (α) is a strictly convex function with f (0) = 0 and lim_α→∞ f (α) = ∞. Furthermore, a little reflection shows that f′ (0) = cⁿ −tm̄ < 0. Thus, there is a single root of f (α) on the interval (0,∞). Owing to the convexity of f (α), Newton’s method will reliably find the root if started to the right of the minimum of f (α) at α = t/cⁿ − 1/m̄.

To find initial values, we again resort to the method of moments. Based on the moments (3.13), the mean and variance of |X| = Σ_k X_j are

$$\mathbf{E}(\left|\mathbf{X}\right|)=\frac{\beta (1-{\pi }_{d+1})}{{\pi }_{d+1}},\mathbf{\text{ Var}}(\left|\mathbf{X}\right|)=\frac{\beta (1-{\pi }_{d+1})}{{\pi }_{d+1}^2}.$$

These suggest that we take

$${\beta }^0=\frac{{\overline{x}}^2}{s^2-\overline{x}},{\pi }_{d+1}^0=\frac{\overline{x}}{s^2},{\pi }_j^0=\frac{{\pi }_{d+1}^0}{{\beta }^0}\frac{x_{·j}}{t},1\le j\le d,$$

where

$$\overline{x}=\frac{1}{t}{\displaystyle \sum _{i=1}^t{m}_i,s^2=}\frac{1}{t-1}{\displaystyle \sum _{i=1}^t{(m_i-\overline{x})}^2}.$$

When the data are underdispersed (s² < x̄), our proposed initial values are not meaningful, but a negative-multinomial model is a poor choice anyway.

3.4 Distributions on Partitions

A partition of a positive integer m into k parts is a vector a = (a₁,…, a_m) of non-negative integers such that Σ_i a_i = k and |a| = Σ_i ia_i = m. In population genetics, the partition distributions of Ewens (2004) and Pitman (Pitman 1995; Johnson, Kotz, and Balakrishnan 1997) find wide application. We now develop an MM algorithm for Pitman’s distribution, which generalizes Ewens’s distribution. Pitman’s distribution

$$\text{Pr}(\mathbf{A}=\mathbf{a}|m,\alpha ,\theta )=\frac{m!{\displaystyle {\prod }_{i=1}^{\left|\mathbf{a}\right|-1}(\theta +i\alpha )}}{(\theta +1)\cdots (\theta +m-1)}\times {\displaystyle \prod _{j=1}^m{\left[\frac{(1-\alpha )\cdots (1-\alpha +j-2)]}{i!}\right]}^{a_j}\frac{1}{a_j!}}$$

involves two parameters 0 ≤ α < 1 and θ > −α. Ewens’s distribution corresponds to the choice α = 0. We will restrict θ to be positive.

To estimate parameters given u independent partitions a₁,…, a_u from Pitman’s distribution, we use the minorizations (2.3) and (2.4) to derive the minorizations

$$\begin{array}{cc}\hfill \text{ln}(\theta +i\alpha )& \ge \frac{{\theta }^n}{{\theta }^n+i{\alpha }^n}\text{ln}\theta +\frac{i{\alpha }^n}{{\theta }^n+i{\alpha }^n}\text{ln}\alpha +c,\hfill \\ \hfill \text{ln}(1-\alpha +i)& \ge \frac{1-{\alpha }^n}{1-{\alpha }^n+i}\text{ln}(1-\alpha )+c,\hfill \\ \hfill -\text{ln}(\theta +i)& \ge -\frac{1}{{\theta }^n+i}\theta +c,\hfill \end{array}$$

where c is a different irrelevant constant in each case. Assuming a_j is a partition of the integer m_j, it follows that the log-likelihood is minorized by

$${\displaystyle \sum _i\frac{r_i{\theta }^n}{{\theta }^n+i{\alpha }^n}\text{ln}\theta +{\displaystyle \sum _i\frac{r_{i}i{\alpha }^n}{{\theta }^n+i{\alpha }^n}}\text{ln}\alpha +{\displaystyle \sum _i\frac{s_i(1-{\alpha }^n)}{1-{\alpha }^n+i}}\text{ln}(1-\alpha )-{\displaystyle \sum _i\frac{t_i}{{\theta }^n+i}\theta +c,}}$$

where

$$r_i={\displaystyle \sum _j{1}_{\{|{\mathbf{a}}_j|\ge i+1\},s_i={\displaystyle \sum _j{\displaystyle \sum _{k\ge i+2}{a}_{\mathit{\text{jk}}},}}}}{t}_i={\displaystyle \sum _j{1}_{\{m_i\ge i+1\}}.}$$

Standard arguments now yield the simple updates

$$\begin{array}{c}{\alpha }^{n+1}=\left({\displaystyle \sum _i\frac{r_{i}i{\alpha }^n}{{\theta }^n+i{\alpha }^n}}\right)/\left({\displaystyle \sum _i\frac{r_{i}i{\alpha }^n}{{\theta }^n+i{\alpha }^n}}+{\displaystyle \sum _i\frac{s_i(1-{\alpha }^n)}{1-{\alpha }^n+i}}\right),\hfill \\ {\theta }^{n+1}=\left({\displaystyle \sum _i\frac{r_i{\theta }^n}{{\theta }^n+i{\alpha }^n}}\right)/\left({\displaystyle \sum _i\frac{t_i}{{\theta }^n+i}}\right).\hfill \end{array}$$

If we set α⁰ = 0, then in all subsequent iterates αⁿ = 0, and we get the MM updates for Ewens’s distribution. Despite the availability of the moments of the parts A_i (Charalambides 2007), it is not clear how to initialize α and θ. Unfortunately, the alternative suggestion of Nobuaki (2001) does not guarantee that the initial values satisfy the constraints α ∈ [0, 1) and θ > 0.

3.5 Zero-Truncated and Zero-Inflated Data

In this section we briefly indicate how the MM perspective sheds fresh light on EM algorithms for zero-truncated and zero-inflated data. Once again mastery of a handful of inequalities rather than computation of conditional expectations drives the derivations.

In many discrete probability models, only data with positive counts are observed. Counts that are 0 are missing. If f (x|θ) represents the density of the complete data, then the density of a random sample x₁,…, x_t of zero-truncated data amounts to

$$h(x|\theta )={\displaystyle \prod _{i=1}^t\frac{f(x_i|\theta )}{1-f(0|\theta )}.}$$

Inequality (2.5) immediately implies the minorization

$$\text{ln}h(x|\theta )\ge {\displaystyle \sum _{i=1}^t\left[\text{ln}f(x_i|\theta )+\frac{f(0|{\theta }^n)}{1-f(0|{\theta }^n)}\text{ln}f(0|\theta )\right]+c,}$$

where c is an irrelevant constant. In many models, maximization of this surrogate function is straightforward.

For instance, with zero-truncated data from the binomial, Poisson, and negative-binomial distributions, the MM updates reduce to

$$\begin{array}{c}{p}^{n+1}=\left({\displaystyle \sum _i{x}_i}\right)/\left({\displaystyle \sum _i\frac{m_i}{1-{(1-p^n)}^{m_i}}}\right),{\lambda }^{n+1}=\left({\displaystyle \sum _i{x}_i}\right)/\left({\displaystyle \sum _i\frac{1}{1-e^{-\lambda n}}}\right),\hfill \\ p^{n+1}=\left({\displaystyle \sum _i\frac{m_i}{1-{(p^n)}^{m_i}}}\right)/\left({\displaystyle \sum _i\left[x_i+\frac{m_i}{1-{(p^n)}^{m_i}}\right]}\right).\hfill \end{array}$$

For observation i of the binomial model, there are x_i successes out of m_i trials with success probability p per trial. λ is the mean in the Poisson model. For observation i of the negative-binomial model, there are x_i failures before m_i required successes.

More complicated models can be handled in similar fashion. The key insight in each case is to augment every ordinary observation x_i > 0 by a total of f (0|θⁿ)/[1 − f (0|θⁿ)] pseudo-observations of 0 at iteration n. With this amendment, the two MM algorithms for the beta-binomial distribution implemented in (3.5), (3.6), and (3.7) remain valid except that the count variables r_k and s_jk defining the updated parameters at iteration n become

$$\begin{array}{c}\hfill s_{1k}={\displaystyle \sum _i{1}_{\{x_{i1}\ge k+1\}},s_{2k}={\displaystyle \sum _i\left[1_{\{x_{i2}\ge k+1\}}+\frac{f(0|{\pi }^n,{\theta }^n)}{1-f(0|{\pi }^n,{\theta }^n)}\right]}},\\ \\ \hfill r_k={\displaystyle \sum _i\left[1+\frac{f(0|{\pi }^n,{\theta }^n)}{1-f(0|{\pi }^n,{\theta }^n)}\right]}{1}_{\{m_i\ge k+1\}},\end{array}$$

where

$$f(0|{\pi }^n,{\theta }^n)=\frac{{\pi }_2^n({\pi }_2^n+{\theta }^n)\cdots [{\pi }_2^n+(m_i-1){\theta }^n]}{(1+{\theta }^n)\cdots [1+(m_i-1){\theta }^n]}.$$

Here category 1 represents success and category 2 failure. If we start with θ⁰ = 0, then we recover the updates for the zero-truncated binomial distribution.

Zero-inflated data are equally easy to handle. The density function is now

$$h(x|\theta ,\pi ){\displaystyle \prod _{i=1}^t{[(1-\pi )+\pi f(0|\theta )]}^{1_{\{x_i=0\}}}}{[\pi f(x_i|\theta )]}^{1_{\{x_i>0\}}}.$$

Inequality (2.3) entails the minorization

$$\begin{array}{cc}\hfill \text{ln}h(x|\theta ,\pi )& \ge {\displaystyle \sum _{i=1}^t{1}_{\{x_i=0\}}\{z^n\text{ln}(1-\pi )+(1-z_n)[\text{ln}\pi +\text{ln}f(0|{\theta }^n)]\}+{\displaystyle \sum _{i=1}^t{1}_{\{x_i>0\}}[\text{ln}\pi +\text{ln}f(x_i|\theta )],}}\hfill \\ \hfill z^n& =\frac{1-{\pi }^n}{1-{\pi }^n+{\pi }^{n}f(0|{\theta }^n)}.\hfill \end{array}$$

The MM update of the inflation-admixture parameter clearly is

$${\pi }^{n+1}=\frac{1}{t}{\displaystyle \sum _{i=1}^t[1_{\{x_i>0\}}+1_{\{x_i=0\}}(1-z^n)].}$$

As a typical example, consider estimation with the zero-inflated Poisson (Patil 2007). The mean λ of the Poisson component is updated by

$${\lambda }^{n+1}=\frac{{\displaystyle {\sum }_i{x}_i}}{{\displaystyle {\sum }_i[(1-z^n)1_{\{x_i=0\}}+1_{\{x_i>0\}}]}}.$$

In other words, every 0 observation is discounted by the amount zⁿ at iteration n. This makes intuitive sense.

4. A NUMERICAL EXPERIMENT

As a numerical experiment, we fit the Dirichlet-multinomial (two parameterizations) and the Neerchal–Morel distributions to the 3823 training digits in the handwritten digit data from the UCI machine learning repository (Asuncion and Newman 2007). Each normalized 32 × 32 bitmap is divided into 64 blocks of size 4 × 4, and the black pixels are counted in each block. This generates a 64-dimensional count vector for each bitmap. Bouguila (2008) successfully fit mixtures of Connor–Mosimann to the training data and used the estimated models to cluster the test data. For illustrative purposes we now fit the Dirichlet-multinomial (two parameterizations) and Neerchal–Morel models. Based on the majorization (2.3), it is straightforward to extend our MM algorithms to fit finite mixture models using any of the previously encountered multivariate discrete distributions.

Table 3 lists the final log-likelihoods, number of iterations, and running times of the different algorithms tested. The MM and accelerated MM algorithms were coded in plain Matlab script language. Newton’s method was implemented using the fmincon function in the Marla Optimization Toolbox under the interior-point option with user-supplied analytical gradient and Hessian. All iterations started from the initial points θ⁰ = 1 and ${\pi }^0={\alpha }^0=(\frac{1}{64},\dots ,\frac{1}{64})$ . The stopping criterion for Newton’s method was tuned to achieve precision comparable to the stopping criterion (2.6) for the MM algorithms. Running times in seconds were recorded from a laptop computer.

Table 3.

Numerical experiment. Row 1: MM; Row 2: SqMPE1 MM; Row 3: SqRRE1 MM; Row 4: Newton’s method using the (fmincon) function available in the Matlab Optimization Toolbox.

	DM (π, θ)			DM (α)			Neerchal–Morel
Digit	L	# iters	Time	L	# iters	Time	L	iters	Time
0	−37,358	232	0.18	−37,358	361	0.16	−38,828	15	0.08
	−37,358	18	0.04	−37,358	18	0.04	−38,828	7	0.10
	−37,358	21	0.04	−37,358	18	0.04	−38,828	7	0.09
	−37,359	11	0.13	−37,358	18	0.16	−38,828	13	106.42
1	−42,179	237	0.16	−42,179	120	0.06	−52,424	17	0.09
	−42,179	17	0.03	−42,179	12	0.03	−52,424	7	0.10
	−42,179	26	0.05	−42,179	13	0.03	−52,424	7	0.10
	−42,179	15	0.19	−42,179	14	0.13	−52,424	12	98.91
2	−39,985	213	0.14	−39,985	136	0.07	−47,723	14	0.07
	−39,985	17	0.04	−39,985	15	0.03	−47,723	6	0.08
	−39,985	17	0.04	−39,985	11	0.03	−47,723	6	0.08
	−39,986	15	0.19	−39,985	15	0.13	−47,721	14	113.15
3	−40,519	214	0.14	−40,519	173	0.08	−45,816	14	0.07
	−40,519	23	0.04	−40,519	15	0.03	−45,816	6	0.08
	−40,519	20	0.04	−40,519	11	0.03	−45,816	6	0.08
	−40,519	14	0.17	−40,519	15	0.13	−45,816	12	102.30
4	−43,489	203	0.13	−43,489	102	0.06	−55,432	14	0.07
	−43,489	17	0.04	−43,489	12	0.03	−55,432	6	0.08
	−43,489	19	0.04	−43,489	9	0.03	−55,432	6	0.08
	−43,489	13	0.17	−43,489	14	0.12	−55,432	14	114.40
5	−41,191	205	0.13	−41,191	116	0.06	−50,063	13	0.07
	−41,191	18	0.04	−41,191	12	0.03	−50,063	6	0.08
	−41,191	19	0.04	−41,191	12	0.03	−50,063	6	0.09
	−41,192	12	0.16	−41,191	15	0.13	−50,063	15	118.22
6	−37,703	232	0.15	−37,703	203	0.10	−41,888	20	0.10
	−37,703	19	0.04	−37,703	16	0.03	−41,888	8	0.11
	−37,703	21	0.04	−37,703	11	0.03	−41,888	8	0.11
	−37,703	15	0.19	−37,703	19	0.16	−41,888	13	104.25
7	−40,304	218	0.14	−40,304	141	0.07	−47,653	12	0.06
	−40,304	16	0.04	−40,304	15	0.03	−47,653	6	0.08
	−40,304	18	0.04	−40,304	11	0.03	−47,653	6	0.08
	−40,305	13	0.15	−40,304	15	0.13	−47,653	15	120.95
8	−43,131	227	0.15	−43,131	171	0.08	−48,844	17	0.09
	−43,131	19	0.04	−43,131	16	0.03	−48,844	7	0.10
	−43,131	23	0.04	−43,131	14	0.03	−48,844	7	0.09
	−43,132	10	0.13	−43,131	15	0.14	−48,844	13	107.22
9	−43,710	207	0.14	−43,710	116	0.06	−53,030	13	0.07
	−43,710	19	0.04	−43,710	12	0.03	−53,030	6	0.08
	−43,710	18	0.04	−43,710	11	0.03	−53,030	6	0.08
	−43,710	12	0.16	−43,710	15	0.14	−53,030	14	116.49

Inspection of Table 3 demonstrates that the MM algorithms outperform Newton’s method and that acceleration is often very beneficial. The cost of evaluating and inverting the observed information matrices of the Neerchal–Morel model significantly slows Newton’s method even in these problems with only 64 parameters. The observed information matrix of the Dirichlet-multinomial distribution possesses a special structure (diagonal plus rank-1 perturbation) that makes matrix inversion far easier. Table 3 does not show the human effort in devising, programming, and debugging the various algorithms. For Newton’s method, derivation and programming took in excess of one day. Formulas for the score and observed information of the Dirichlet-multinomial and Neerchal–Morel distributions are omitted for the sake of brevity. Fisher’s scoring algorithm was not implemented because it is even more cumbersome than Newton’s method (Neerchal and Morel 2005).

This numerical comparison is merely for illustrative purpose. Numerical analysts have developed quasi-Newton algorithms to mend the defects of Newton’s method. The limited-memory BFGS (LBFGS) algorithm (Nocedal and Wright 2006) is especially pertinent to high-dimensional problems. A systematic comparison of the two methods is worth pursuing.

5. DISCUSSION

In designing algorithms for maximum likelihood estimation, Newton’s method and Fisher scoring come immediately to mind. In the last generation, statisticians have added the EM principle. These are good mental reflexes, but the broader MM principle also deserves serious consideration. In many problems, the EM and MM perspectives lead to the same algorithm. In other situations such as image reconstruction in transmission tomography, it is possible to construct different EM and MM algorithms for the same purpose (Lange 2004). One of the most appealing features of the EM perspective is that it provides a statistical interpretation of algorithm intermediates. Although it is a matter of taste and experience whether inequalities or missing data offer an easier path to algorithm development, the fact that there are two routes adds to the possibilities for new algorithms.

One can argue that applications of minorizations (2.3) and (2.5) are just disguised EM algorithms. This objection misses the point in three ways. First, it does not suggest missing data structures explaining the minorization (2.4) and other less well-know minorizations. Second, it fails to weigh the difficulties of invoking simple inequalities versus calculating conditional expectations. When the creation of an appropriate surrogate function requires several minorizations, the corresponding conditional expectations become harder to execute. For example, although the EM principle dictates adding pseudo-observations for zero-truncated data, it is easy to lose sight of this simple interpretation in complicated examples such as the beta-binomial distribution. The genetic segregation analysis example appearing in chapter 2 of the book by Lange (2002) falls into the same category. Third, it fails to acknowledge the conceptual clarity of the MM principle, which shifts focus away from the probability spaces connected with missing data to the simple act of minorization. For instance, when one undertakes maximum a posteriori estimation, should the E step of the EM algorithm take into account the prior?

Some EM and MM algorithms are notoriously slow to converge. As we noted earlier, slow convergence is partially offset by the simplicity of each iteration. There is a growing body of techniques for accelerating MM algorithms (Jamshidian and Jennrich 1995; Lange 1995a; Jamshidian and Jennrich 1997; Varadhan and Rolland 2008). These techniques often lead to a ten-fold or even a hundred-fold reduction in the number of iterations. The various examples appearing in this article are typical in this regard. On problems with boundaries or nondifferentiable objective functions, acceleration may be less helpful.

Our negative-multinomial example highlights two useful tactics for overcoming complications in solving the maximization step of the EM and MM algorithms. It is a mistake to think of the various optimization algorithms in isolation. Often block relaxation (de Leeuw 1994) and Newton’s method can be combined creatively with the MM principle. Systematic application of Newton’s method in solving the maximization step of the MM algorithm is formalized in the MM gradient algorithm (Lange 1995b).

Parameter asymptotic standard errors are a natural byproduct of Newton’s method and scoring. With a modicum of additional effort, the EM and MM algorithms also deliver asymptotic standard errors (Meng and Rubin 1991; Hunter and Lange 2004). Virtually all optimization algorithms are prone to converge to inferior modes. For this reason, we have emphasized finding reasonable initial values. The overlooked article of Ueda and Nakano (1998) suggested an annealing approach to maximization with mixture models. Here the idea is to flatten the likelihood surface and eliminate all but the dominant mode. As the iterations proceed, the flat surface gradually warps into the true bumpy surface. Our recent work (Zhou and Lange 2010) extends this idea to many other EM and MM algorithms. A similar idea, called graduated non-convexity (GNC), appears in computer vision and signal processing literature (Blake and Zisserman 1987). In the absence of a good annealing procedure, one can fall back on starting an optimization algorithm from multiple random points, but this inevitably increases the computational load. The reassurance that a log-likelihood is concave is always welcome.

Readers may want to try their hands at devising their own MM algorithms. For instance, the Dirichlet-negative-multinomial distribution, the bivariate Poisson (Johnson, Kotz, and Balakrishnan 1997), and truncated multivariate discrete distributions yield readily to the techniques described. The performance of the MM algorithm on these problems is similar to that in our fully developed examples. Of course, many objective functions are very complicated, and devising a good MM algorithm is a challenge. The greatest payoffs are apt to be on high-dimensional problems. For simplicity of exposition, we have not tackled any extremely high-dimensional problems, but these certainly exist (Sabatti and Lange 2002; Ayers and Lange 2008; Lange and Wu 2008). In any event, most mathematicians and statisticians keep a few tricks up their sleeves. The MM principle belongs there, waiting for the right problems to come along.