A Legacy of EM Algorithms

Summary

Nan Laird has an enormous and growing impact on computational statistics. Her paper with Dempster and Rubin on the expectation-maximisation (EM) algorithm is the second most cited paper in statistics. Her papers and book on longitudinal modelling are nearly as impressive. In this brief survey, we revisit the derivation of some of her most useful algorithms from the perspective of the minorisation-maximisation (MM) principle. The MM principle generalises the EM principle and frees it from the shackles of missing data and conditional expectations. Instead, the focus shifts to the construction of surrogate functions via standard mathematical inequalities. The MM principle can deliver a classical EM algorithm with less fuss or an entirely new algorithm with a faster rate of convergence. In any case, the MM principle enriches our understanding of the EM principle and suggests new algorithms of considerable potential in high-dimensional settings where standard algorithms such as Newton’s method and Fisher scoring falter.

1. INTRODUCTION

As of November 2021, the landmark paper on expectation-maximisation (EM) algorithms of Dempster et al. (1977) is the second most cited paper across all of statistics, boasting a cumulative count of 64,769 citations according to Google Scholar. This exposition explores variations of the algorithms derived by Dempster et al. (1977). These algorithms exemplify some of the most fundamental ideas Nan Laird has contributed to or inspired in statistical science: EM algorithms, the closely related minorisation-maximisation (MM) algorithms, and longitudinal data analysis by mixed models (Garrett et al., 2004).

The MM and EM algorithms replace the objective function by a simpler surrogate function. By design, optimising the surrogate function sends the objective function downhill in minimisation and uphill in maximisation. In constructing the surrogate function for an EM algorithm, statisticians rely on notions of missing data. The more general MM algorithm calls on skills in inequalities and convex analysis. More often than not, concrete problems also involve parameter constraints. Modern penalty methods incorporate the constraints by imposing penalties on the objective function. A tuning parameter scales the strength of the penalties. In the classical penalty method, the constrained solution is recovered as the tuning parameter tends to infinity. In the augmented Lagrangian method, the constrained solution emerges for a finite value of the tuning parameter.

In the remaining sections, we adopt several notational conventions. Vectors and matrices appear in boldface type; for the most part parameters appear as Greek letters. The differential df (θ) of a scalar-valued function f (θ) equals its row vector of partial derivatives; the transpose ∇f (θ) of the differential is the gradient. The second differential d²f (θ) is the Hessian matrix of second partial derivatives. The Euclidean norm of a vector b and the spectral norm of a matrix A are denoted by ∥b∥ and ∥A∥, respectively. All other norms will be appropriately subscripted. The nth entry b_n of a vector b must be distinguished from the nth vector b_n in a sequence of vectors. To maintain consistency, b_ni denotes the ith entry of b_n. A similar convention holds for sequences of matrices. For symmetric matrices, the relation A⪯B means that B − A is positive semidefinite.

2. THE EM AND MM ALGORITHMS

The numerical analysts Ortega & Rheinboldt (1970) first articulated the MM principle; de Leeuw (1976) saw its potential and created the first MM algorithm of value in statistics. Building on earlier work of Weiszfeld (1937), Voß & Eckhardt (1980) illuminated some convergence properties of MM algorithms. The MM principle currently enjoys its greatest vogue in computational statistics (Hunter & Lange, 2004; Lange et al., 2000; Lange, 2016). The basic idea is to convert a hard optimisation problem into a sequence of simpler ones. In minimisation, the MM principle majorises the objective function f (θ) by a surrogate function g(θ|θ_n) anchored at the current point θ_n. Majorisation combines the tangency condition g(θ_n|θ_n) = f (θ_n) and the domination condition g(θ|θ_n) ≥ f (θ) for all θ. The next iterate of the MM algorithm is defined to minimise g(θ|θ_n). Because

$$f\left({\theta }_{n+1}\right)\le g\left({\theta }_{n+1}\mid {\theta }_n\right)\le g\left({\theta }_n\mid {\theta }_n\right)=f\left({\theta }_n\right),$$

the MM iterates generate a deiscent algorithm drivng the objective function downhill. Strictly speaking, the descent property depends only on decreasing g(θ|θ_n), not on minimising it. Constraint satisfaction is automatically enforced in finding θ_{n + 1}. Under appropriate regularity conditions, an MM algorithm is guaranteed to converge to a local minimum of the objective function (Lange, 2010; Lange et al., 2021). In maximisation, we first minorise and then maximise. Thus, the acronym MM does double duty in the forms majorise-minimise and minorise-maximise.

When it is successful, the MM algorithm simplifies optimisation by (a) separating the variables of a problem, (b) avoiding large matrix inversions, (c) linearising a problem, (d) restoring symmetry, (e) dealing with equality and inequality constraints gracefully, and (f) turning a nondifferentiable problem into a smooth problem. The art in devising an MM algorithm lies in choosing a tractable surrogate function g(θ|θ_n)that hugs the objective function f (θ)as tightly possible.

The majorisation relation between functions is closed under the formation of sums, nonnegative products, limits, and composition with an increasing function. These rules allow one to work piecemeal in simplifying complicated objective functions. Skill in dealing with inequalities is crucial in constructing majorisations. Classical inequalities such as Jensen’s inequality, the information inequality, the arithmetic-geometric mean inequality, and the Cauchy–Schwartz inequality prove useful in many problems. The supporting hyperplane property of a convex function and the quadratic upper bound principle (Böhning & Lindsay, 1988; de Leeuw & Lange, 2009) also find wide application.

The derivation of the EM principle hinges upon a missing data structure. Let f (θ) be the log-likelihood of the observed data with parameter vector θ. In the E step, a surrogate function Q(θ|θ_n) is calculated as the conditional expectation of the complete data log-likelihood given the observed data and the current parameter iterate θ_n. Well-known calculations (Dempster et al., 1977) based on the information inequality demonstrate that the Q function satisfies the domination inequality

$$f(\theta )\ge Q\left(\theta \mid {\theta }_n\right)-Q\left({\theta }_n\mid {\theta }_n\right)+f\left({\theta }_n\right)$$

for all θ. The tangency condition obviously holds at θ = θn. This effectively validates Q(θ|θ_n)as a minorisation of f (θ) up to an additive constant. Figure 1 displays the Q function of the EM algorithm and the minorisation function of an MM algorithm for the variance component model studied in Section 4. In this example, MM differs from EM, and the MM minorisation function hugs the log-likelihood function tighter than the Q function of the EM algorithm, resulting in faster convergence of MM than EM as depicted later in Figure 2.

Figure 1.

The Q function of EM and the corresponding MM surrogate minorise the log-likelihood surface of a univariate response, two variance component model at the point (18.5, 0.7). In this example, the MM surrogate hugs the log-likelihood surface tighter than the EM surrogate

Figure 2.

MM algorithm converges faster than EM for a multi-response variance component model with d = 4 responses for m = 500 subjects, k = 3 variance components, and p = 3 fixed effect covariates. L^⋆ – L_n indicates the difference in log-likelihood between the found MLE and the nth iterate. EM and MM algorithms start from the same point in each of the 100 simulation replicates

3. MM ALGORITHMS FOR TRADITIONAL PROBLEMS

Convexity and concavity figure prominently in the construction of many MM algorithms. The supporting hyperplane minorisation

$$f(x)\ge f\left(x_n\right)+df\left(x_n\right)\left(x-x_n\right)$$

of a convex function f (x) is natural in many applications. For the choice f (x) = −ln(x), this reads

$$-\text{ln\hspace{0.17em}}x\ge -\text{\hspace{0.17em}ln}{x}_n-\frac{x-x_n}{x_n}.$$

Jensen’s inequality is instrumental in majorising composite functions of the form f [u(x)+v(x)], where f (y) is convex and u and v are positive functions of some underlying parameter vector x. In practice, it is often convenient to split the contributions of u and v. The majorisation (De Pierro, 1993)

$$f(u+v)\le \frac{u_n}{u_n+v_n}f\left(\frac{u_n+v_n}{u_n}u\right)+\frac{v_n}{u_n+v_n}f\left(\frac{u_n+v_n}{v_n}v\right)$$ (1)

achieves this goal. Equality clearly holds whenever (u, v) = (u_n, v_n). For the special case f (x) = −ln x, the minorisation

$$\text{ln}(u+v)\ge \frac{u_n}{u_n+v_n}\text{ln\hspace{0.17em}}u+\frac{v_n}{u_n+v_n}\text{ln\hspace{0.17em}}v+c_n$$

relies on a constant c_n depending only on (u_n, v_n). This minorisation is handy in splitting log-likelihoods in maximum likelihood estimation with mixture models. It extends to from two to multiple summands. Armed with these ideas, we now explore four examples.

Example Power Series Distributions

A random variable X concentrated on an interval [r, ∞) of nonnegative integers is said to have a power series distribution if $\text{Pr}(X=k)=\frac{c_k{\theta }^k}{q(\theta )}$ for all k ∈ [r, ∞). In this definition, θ is a positive parameter, the coefficients c_k are nonnegative, and $q(\theta )={\sum }_{k=r}^{\infty }{c}_k{\theta }^k$ is the appropriate normalising constant (Rao, 1973). Examples include the binomial, negative binomial, Poisson, and logarithmic families and versions of these families truncated at any nonnegative number r, especially r = 1. For example, the negative binomial distribution has $c_k=\left(\begin{array}{c}j+k-1\\ k\end{array}\right)$ and q(θ) = (1 – θ)^–j for r = 0 under no truncation. If x₁, …, x_m is a random sample from the power series density and q(θ) is log-concave, then the log-likelihood of the data is minorised, via the supporting hyperplane inequality, by

$$\begin{gathered} L(\theta )\ge \sum _{i=1}^m{x}_i\text{ln\hspace{0.17em}}\theta -m\text{\hspace{0.17em}ln\hspace{0.17em}}q\left({\theta }_n\right)-m{\left[\text{ln\hspace{0.17em}}q\left({\theta }_n\right)\right]}^{\prime }\left(\theta -{\theta }_n\right)+c_n \\ =\sum _{i=1}^m{x}_i\text{ln\hspace{0.17em}}\theta -m\text{\hspace{0.17em}ln\hspace{0.17em}}q\left({\theta }_n\right)-m\frac{q^{\prime }\left({\theta }_n\right)}{q\left({\theta }_n\right)}\left(\theta -{\theta }_n\right)+c_n, \end{gathered}$$

where c_n is a constant independent of θ. Setting the derivative of this surrogate function equal to 0 leads to the MM update

$${\theta }_{n+1}=\frac{\overline{x}q\left({\theta }_n\right)}{q^{\prime }\left({\theta }_n\right)},$$

where $\overline{x}$ is the sample average of the observations x_i. Anderson et al. (2007) derive a straightforward test for log-concavity of q(θ). Namely, if the coefficients c_k are positive and the ratio (k + 1)c_{k + 1/}c_k is decreasing in k, then q(θ) is log-concave. The negative binomial fails this test, but the Poisson and binomial distributions qualify. These ideas are pursued in more depth by Wu & Lange (2010).□

Example Cauchy Location and Scale

The Cauchy density with location μ and scale σ can be written as

$$f(x)=\frac{1}{\pi \sigma \left[1+{\left(\frac{x-\mu }{\sigma }\right)}^2\right]}.$$

The usual approach to maximum likelihood estimation of μ and σ involves finding the roots of polynomials of degree 2m − 1 and 2m, respectively, for m sample points x₁, …, x_m. However, this process tends to be complicated by the existence of multiple local maxima. From the MM perspective, one can exploit the convexity of the function h(y) = −log(1 + y) via the supporting hyperplane minorisation. If we substitute ${\left(\frac{x-\mu }{\sigma }\right)}^2$ for y, then the log-likelihood is minorised by

$$L(\mu ,\sigma )\ge -m\text{\hspace{0.17em}log\hspace{0.17em}}\sigma -\sum _{i=1}^m{w}_{ni}{\left(\frac{x_i-\mu }{\sigma }\right)}^2+c_n$$

$$w_{ni}=\frac{1}{1+{\left(\frac{x_i-{\mu }_n}{{\sigma }_n}\right)}^2}$$

at iteration n, where c_n is an irrelevant constant. The MM algorithm for estimating μ and σ now reduces to weighted least squares with updates

$${\mu }_{n+1}=\frac{\sum _{i=1}^m{w}_{ni}{x}_i}{\sum _{i=1}^m{w}_{ni}}\text{ and }{\sigma }_{n+1}=\sqrt{\frac{2\sum _{i=1}^m{w}_{ni}{\left(x_i-{\mu }_{n+1}\right)}^2}{m}}\text{. }$$

These updates stably increase the likelihood at each iteration. The median of the x_i serves as a starting value for μ. As recommended by Wikipedia, half the sample inter-quartile range is a reasonable starting value for σ. This example is a special case of the broader algorithm discussed in the next example. □

Example Elliptically Symmetric Distributions

An elliptically symmetric probability density takes the form

$$f(y)=\frac{e^{-\frac{1}{2}}\kappa \left({\delta }^2\right)}{{(2\pi )}^{\frac{p}{2}}{(\text{det}\Omega )}^{\frac{1}{2}}},$$

where $y\in {ℝ}^p$ and δ² = (y – μ)* Ω^–1 (y – μ) denotes the Mahalanobis distance between y and μ. Here, we assume that the function κ(s) is strictly increasing and strictly concave and that the matrix Ω is positive definite. Such densities serve as substitutes for the multivariate normal distribution in robust estimation (Huber, 2004; Lange et al., 1989; Lange & Sinsheimer, 1993).

Dutter and Huber (Huber, 2004) introduced an MM algorithm driven by the affine majorisation

$$\kappa (t)\le \kappa \left(t_n\right)+{\kappa }^{\prime }\left(t_n\right)\left(t-t_n\right).$$

If y₁, …, y_m is a random sample from the density (3), then the multivariate normal log-likelihood

$$g\left(\theta \mid {\theta }_n\right)=-\frac{1}{2}\sum _{i=1}^m\left[w_{ni}{\delta }_i^2(\theta )+\text{ln\hspace{0.17em}det}\Omega \right]+c_n$$

with weights $w_{ni}={\kappa }^{\prime }\left[{\delta }_i^2\left({\theta }_n\right)\right]$ and irrelevant constant c_n minorises the log-likelihood of the data under the elliptically symmetric density. The array of techniques from linear algebra for estimating the parameters of a multivariate normal distribution can be brought to bear on maximising g(θ|θ_n). For normal/independent distributional families such as the multivariate t, the Dutter–Huber algorithm reduces to an EM algorithm (Dempster, 1980; Lange & Sinsheimer, 1993). Given an unstructured mean vector and covariance matrix, the MM updates (Lange et al., 1989; Lange & Sinsheimer, 1993; Little & Rubin, 2019) are

$${\mu }_{n+1}=\frac{1}{s_n}\sum _{i=1}^m{w}_{ni}{y}_i$$

$${\Omega }_{n+1}=\frac{1}{m}\sum _{i=1}^m{w}_{ni}\left(y_i-{\mu }_{n+1}\right){\left(y_i-{\mu }_{n+1}\right)}^{\top },$$

where $S_n={\sum }_{i=1}^m{w}_{ni}$ is the sum of the case weights. For the multivariate t, Kent et al. (1994) suggest a faster algorithm that replaces the update of Ω by

$${\Omega }_{n+1}=\frac{1}{s_n}\sum _{i=1}^m{w}_{ni}\left(y_i-{\mu }_{n+1}\right){\left(y_i-{\mu }_{n+1}\right)}^{\top }.$$

Meng & Van Dyk (1997) justify this amendment within the EM framework.

Example EM for Mixture Models

Many naive data scientists conflate the EM principle with the EM algorithm for normal mixtures. This level of ignorance is testimony to the importance of this special case. Dempster et al. (1977) review the history of the EM clustering algorithm and demonstrate that it possesses the critical ascent property. The EM algorithm makes soft cluster assignments in contrast to the hard assignments of k-means clustering. The alternative of soft choices is possible with admixture models (McLachlan & Krishnan, 2007; Mengersen et al., 2011). An admixture probability density h(y) can be written as a convex combination

$$h(y)=\sum _{j=1}^k{\pi }_j{h}_j(y),$$

where the π_j are nonnegative probabilities that sum to 1 and h_j(y) is the probability density of group j.

Suppose the observations y₁, …, y_m represent a random sample from the admixture density (3). In practice, we want to estimate the admixture proportions π_j and whatever further parameters θ characterise the densities h_j(y|θ). An EM algorithm is natural in this context with group membership as the missing data (Dempster et al., 1977). The EM updates can be derived by invoking the Jensen minorisation (3) for each observation y_i in the form

$$\text{ln}\left[\sum _{j=1}^k{\pi }_j{h}_j\left(y_i\mid \theta \right)\right]\ge \sum _{j=1}^k{w}_{nij}\left[\text{ln}{\pi }_j+\text{ln}{h}_j\left(y_i\mid \theta \right)\right]+c_n,$$

where c_n is an irrelevant constant and w_nij is the posterior probability that y_i belongs to cluster j given the current admixture vector π_n and density vector θ_n. Fortunately, this minorisation separates the π parameters from the θ parameters. The problem of maximising the objective ${\sum }_{j=1}^k{d}_j\text{\hspace{0.17em}ln\hspace{0.17em}}{\pi }_j$ for $d_j={\sum }_{i=1}^m{w}_{nij}$ is standard with intuitive solution π_{n + 1, j} = d_j/m. Updating the remaining parameters is possible for elliptically symmetric distributions as discussed in the previous example. This involves a second minorisation, which is often ill advised because it generates slowing converging algorithms. It is viable here if we employ a common scale matrix across the groups and the Kent et al. (1994) acceleration for the multivariate t. □

4. MULTI-RESPONSE VARIANCE COMPONENT MODELS

In this example, we contrast the derivations of an EM algorithm and an MM algorithm for the multi-response variance component model. This model involves an m × d response matrix Y with mean E(Y) = XB and covariance

$$\Omega =\text{Cov}(\text{vec\hspace{0.17em}}Y)=\sum _{j=1}^k{\Gamma }_j\otimes V_j.$$

The p × d coefficient matrix B collects the fixed effects, the d × d covariance matrices Γ_j collect the unknown variance components, and the m × m covariance matrices V_j collect the known variance components. When the vector vecY is normally distributed, Y equals a sum of independent matrix normal distributions (Gupta & Nagar, 1999). We now make this assumption and pursue estimation of B and the Γ_j, which we collectively denote as Γ. Under the normality assumption, Roth’s Kronecker product identity vec(CDE) = (E^⊤ ⊗C)vec(D) yields the log-likelihood

$$L(B,\Gamma )=-\frac{1}{2}\text{ln\hspace{0.17em}det}\Omega -\frac{1}{2}{\left(\text{vec\hspace{0.17em}}Y-Z_{\text{vec}}B\right)}^{\top }{\Omega }^{-1}\left(\text{vec\hspace{0.17em}}Y-Z_{\text{vec}}B\right),$$ (2)

where Z = I_d⊗X.

4.1. MM Derivation

Updating B given Γ_n is accomplished by solving the general least squares problem met earlier in the univariate case. Updating Γ_j given B_n is difficult due to the positive semidefiniteness constraint. Typical solutions involve reparameterization of the covariance matrix (Pinheiro & Bates, 1996). The MM algorithm derived in this section gracefully accommodates this constraint.

Updating Γ given B requires two minorisations. The convexity of the function −ln detΩ implies the supporting hyperplane minorisation

$$-\frac{1}{2}\text{ln\hspace{0.17em}det}\Omega \ge -\frac{1}{2}\text{ln\hspace{0.17em}det}{\Omega }_n-\frac{1}{2}\text{tr}\left[{\Omega }_n^{-1}\left(\Omega -{\Omega }_n\right)\right]\text{. }$$ (3)

We must also generalise Jensen’s majorisation (1). This is accomplished by noting that the function

$$f(X,M)=\{\begin{array}{ll}\frac{1}{2}{v}^{*}{X}^{*}{M}^{-}Xv\hfill & Xv\in \text{Range}(M)\hfill \\ \infty \hfill & Xv\notin \text{Range}(M)\hfill \end{array}$$

is convex for v fixed, where M is positive semidefinite, X is conformable to M, and M^– is the pseudo-inverse of M (Lange, 2016). Given this fact and the identities (A⊗B)(C⊗D) = (AC)⊗(BD), (A⊗B)^–1 = A⁻¹⊗B⁻¹, and AA⁻¹A = A, we have

$$\begin{gathered} {\Omega }_n{\Omega }^{-1}{\Omega }_n=k\left[\frac{1}{k_j}\sum _{j=1}^k{\Gamma }_{nj}\otimes V_j\right]{\left[\frac{1}{k}\sum _{i=1}^k{\Gamma }_j\otimes V_j\right]}^{-1}\left[\frac{1}{k}\sum _{i=1}^k{\Gamma }_{nj}\otimes V_j\right] \\ ⩽k\sum _{j=1}^k\frac{1}{k}\left({\Gamma }_{nj}\otimes V_j\right){\left({\Gamma }_j\otimes V_j\right)}^{-1}\left({\Gamma }_{nj}\otimes V_j\right) \\ =\sum _{j=1}^k\left({\Gamma }_{nj}{\Gamma }_j^{-1}{\Gamma }_{nj}\right)\otimes V_j, \end{gathered}$$

or equivalently

$${\Omega }^{-1}⩽{\Omega }_n^{-1}\left[\sum _{j=1}^k\left({\Gamma }_{nj}{\Gamma }_j^{-1}{\Gamma }_{nj}\right)\otimes V_j\right]{\Omega }_n^{-1}.$$ (4)

This derivation relies on the invertibility of the matrices V_j. One can relax this assumption by substituting V_{ϵ, j} = V_j + ϵI_m for V_j and sending ϵ to 0.

Up to an irrelevant constant, the majorisations (3) and (4) jointly yield the surrogate

$$\begin{gathered} g\left(\Gamma \mid {\Gamma }_{n1},\dots ,{\Gamma }_{nk}\right) \\ =-\frac{1}{2}\sum _{j=1}^k\left\{\text{tr}\left[{\Omega }_n^{-1}\left({\Gamma }_j\otimes V_j\right)\right]+\text{vec}{\left(R_n\right)}^{\top }\left[\left({\Gamma }_{nj}{\Gamma }_j^{-1}{\Gamma }_{nj}\right)\otimes V_j\right]\text{vec}\left(R_n\right)\right\}, \end{gathered}$$

where R_n is the m × d matrix satisfying

$$\text{vec}\left(R_n\right)={\Omega }_n\text{vec}\left(Y-X{B}_n\right)\text{.}$$ (5)

The first trace here is linear in Γ_j and can be expressed as

$$\begin{gathered} \text{tr}\left[{\Omega }_n^{-1}\left({\Gamma }_j\otimes V_j\right)\right]=\text{tr}\left(M_{nj}{\Gamma }_j\right) \\ {M}_{nj}={\left(I_d\otimes 1_m\right)}^{\top }\left[\left(1_d{1}_d^{\top }\otimes V_j\right)\odot {\Omega }_n^{-1}\right]\left(I_d\otimes 1_m\right), \end{gathered}$$ (6)

where ⊙ takes the Hadamard (pointwise) product of two matrices. To prove this fact, note that if ${\left({\Omega }_n^{-1}\right)}_{rS}$ is the (r, s)th m × m block of ${\Omega }_n^{-1}$, then the coefficient of the entry (Γ_j)_rs is equal to

$$\text{tr}\left[{\left({\Omega }_n^{-1}\right)}_{rs}{V}_j\right]=1_m^{\top }\left[V_j\odot {\left({\Omega }_n^{-1}\right)}_{rs}\right]1_m.$$

Furthermore, (I_d⊗1_m) is a diagonal block matrix with each diagonal block equal to 1_m and (1_d1_d^⊤ ⊗V_j) is a block matrix with all blocks equal to V_j.

The second trace of g(Γ|Γ_n) simplifies owing to the Kronecker identities vec(CDE) = (E^⊤ ⊗C)vec(D) and vec(A)^⊤ vec(B) = tr(A^⊤ B). It follows that the surrogate can be rewritten as

$$\begin{gathered} g\left(\Gamma \mid {\Gamma }_{n1},\dots ,{\Gamma }_{nk}\right) \\ =-\frac{1}{2}\sum _{j=1}^k\left\{\text{tr}\left[{\Omega }_n^{-1}\left({\Gamma }_j\otimes V_j\right)\right]+\text{tr}\left(R_n^{\top }{V}_j{R}_n{\Gamma }_{nj}{\Gamma }_j^{-1}{\Gamma }_{nj}^{\top }\right)\right\} \\ =-\frac{1}{2}\sum _{j=1}^k\left\{\text{tr}\left(M_{nj}{\Gamma }_j\right)+\text{tr}\left({\Gamma }_{nj}^{\top }{R}_n^{\top }{V}_j{R}_n{\Gamma }_{nj}{\Gamma }_j^{-1}\right)\right\}. \end{gathered}$$ (7)

The directional derivative of g(Γ|Γ_n1, …, Γ_nk) with respect to Γ_j in the direction Δ_j is

$$\begin{gathered} -\frac{1}{2}\text{tr}\left(M_{nj}{\Delta }_j\right)+\frac{1}{2}\text{tr}\left({\Gamma }_{nj}^{\top }{R}_n^{\top }{V}_j{R}_n{\Gamma }_{nj}{\Gamma }_j^{-1}{\Delta }_j{\Gamma }_j^{-1}\right) \\ =-\frac{1}{2}\text{tr}\left(M_{nj}{\Delta }_j\right)+\frac{1}{2}\text{tr}\left({\Gamma }_j^{-1}{\Gamma }_{nj}{R}_n^{\top }{V}_j{R}_n{\Gamma }_{nj}{\Gamma }_j^{-1}{\Delta }_j\right). \end{gathered}$$

Because all directional derivatives of the surrogate vanish at a stationary point, the matrix equation

$$M_{nj}={\Gamma }_j^{-1}{\Gamma }_{nj}{R}_n^{\top }{V}_j{R}_n{\Gamma }_{nj}{\Gamma }_j^{-1}$$ (8)

holds. Fortunately, this equation admits an explicit solution. For positive scalers a and b, the solution to the equation b = 1x^–1ax^–1 is $x=\pm \sqrt{a/b}$ The matrix analogue of this equation is the Riccati equation B = X^–1 AX^–1, whose solution is summarised in the next lemma.

Lemma 1. Assume A and B are positive definite and L is the Cholesky factor of B. Then Y = (L^–1)^⊤ (L^⊤AL)^1/2L^–1 is the unique positive definite solution to the matrix equation B = X^–1AX^–1.

The Cholesky factor L in Lemma 4.1 can be replaced by the symmetric square root of B. The solution, which is unique, remains the same. The Cholesky decomposition is preferred for its cheaper computational cost and better numerical stability.

Algorithm 1 summarises the MM algorithm for fitting the multi-response model (1). Each iteration invokes k Cholesky decompositions and symmetric square roots of d × d positive definite matrices. Fortunately in most applications, d is a small number.

4.2. EM Derivation

The landmark paper (Dempster et al., 1977) by Nan Laird and co-authors features the EM derivation for variance component models with univariate response. Later extensions to multivariate responses include Reinsel (1984) and Glanz & Carvalho (2018). We give a self-contained derivation here for ease of comparison with the MM algorithm. Derivation of the EM algorithm hinges upon the missing data and conditional expectation. If the response matrix Y can be written as the sum Y = XB + Z₁ + … + Z_k of independent random matrices with vecZ_j ~ N(0, Ω_j), then vecY ~ N(vec(Xβ); Ω), where $\Omega =\sum _{j=1}^k{\Omega }_j$. Under the matrix normal assumption, Ω_j = Γ_j⊗V_j. The complete data log-likelihood for the unobserved Z_j is

$$-\frac{1}{2}\sum _{j=1}^k{\text{lndet}}^{+}{\Omega }_j-\frac{1}{2}\sum _{j=1}^k{\left(\text{vec}{Z}_j\right)}^T{\Omega }_j^{+}\left(\text{vec}{Z}_j\right),$$

where det⁺ Ω_j denotes the pseudo-determinant of Ω_j and ${\Omega }_j^{+}$ the pseudo-inverse of Ω_j. To compute the surrogate function for the EM algorithm, one needs the conditional expectations

$$E_{nj}=\text{E}\left(\text{vec}{Z}_j\mid Y,{\theta }_n\right)={\Omega }_{nj}{\Omega }_n^{-1}\text{vec}\left(Y-X{B}_n\right)$$

and the conditional covariances

$$F_{nj}=\text{Cov}\left(\text{vec}{Z}_j\mid Y,{\theta }_n\right)={\Omega }_{nj}-{\Omega }_{nj}{\Omega }_n^{-1}{\Omega }_{nj},$$

where θ is the parameter vector. These are employed to compute the conditional second moments

$$G_{nj}=\text{E}\left[\left(\text{vec}{Z}_j\right){\left(\text{vec}{Z}_j\right)}^{\top }\mid Y,{\theta }_n\right]=F_{nj}+E_{nj}{\left(E_{nj}\right)}^{\top }.$$

Here, the random vector Z_j should be replaced by Z_k – XB_n when j = k.

One can readily check that ${\Omega }_j^{+}={\Gamma }_j^{+}\otimes V_j^{+}={\Gamma }_j^{-1}\otimes V_j^{+}$ for Γ_j invertible. Because the pseudo-determinant of a positive semidefinite matrix equals the product of its positive eigenvalues, the formulas

$${\text{det}}^{+}{\Omega }_j={\left(\text{det}{\Gamma }_j\right)}^{r_i}{\left({\text{det}}^{+}{V}_j\right)}^{s_i}$$

$${\text{lndet}}^{+}{\Omega }_j=r_j\text{\hspace{0.17em}ln\hspace{0.17em}det}{\Gamma }_j+s_i{\text{lndet}}^{+}{V}_j$$

apply, where $r_j=\text{rank}\left(V_j^{+}\right)$ and $s_j=\text{rank}\left({\Gamma }_j^{+}\right)$. In the M step of the EM algorithm, one maximises the surrogate

$$-\frac{1}{2}\sum _{j=1}^k{r}_j\text{ln\hspace{0.17em}det}{\Gamma }_j-\frac{1}{2}\sum _{j=1}^k\text{tr}\left[\left({\Gamma }_j^{-1}\otimes V_j^{+}\right)G_{nj}\right].$$ (9)

For Γ_j unstructured, we substitute ${\Lambda }_j={\Gamma }_j^{-1}$ and maximise with respect to Λ_j. Fortunately, the next lemma can be invoked.

Lemma 2. If the matrices A, B, and C are d × d, m × m, and dm × dm respectively, then

$$\text{tr}\left[(A\otimes B)C^{\top }\right]=\text{tr}\left\{{\left(I_d\otimes 1_m\right)}^{\top }\left[\left(1_d{1}_d^{\top }\otimes B\right)\odot C\right]\left(I_d\otimes 1_m\right)A^{\top }\right\}.$$

Proof This trace identity is essentially proved in our derivation of the corresponding MM algorithm in Section 4.1. □

Lemma 4.2 yields

$$\text{tr}\left[\left({\Lambda }_j\otimes V_j^{+}\right)G_j^{(t)}\right]=\text{tr}\left\{{\left(I_d\otimes 1_m\right)}^{\top }\left[\left(1_d{1}_d^{\top }\otimes V_j^{+}\right)\odot G_j^{(t)}\right]\left(I_d\otimes 1_m\right){\Lambda }_j\right\}.$$

The stationarity condition

$$0=\frac{1}{2}{r}_j{\Lambda }_j^{-1}-\frac{1}{2}{\left(I_d\otimes 1_m\right)}^{\top }\left[\left(1_d{1}_d^{\top }\otimes V_j^{+}\right)\odot G_j^{(t)}\right]\left(I_d\otimes 1_m\right)$$

now entails the update

$$\begin{gathered} {\Gamma }_{n+1,j}=r_j^{-1}{\left(I_d\otimes 1_m\right)}^{\top }\left[\left(1_d{1}_d^{\top }\otimes V_j^{+}\right)\odot G_{nj}\right]\left(I_d\otimes 1_m\right) \\ ={\Gamma }_{nj}-r_j^{-1}{\Gamma }_{nj}{M}_{nj}{\Gamma }_{nj}+r_j^{-1}{\Gamma }_{nj}{R}_n^{\top }{V}_j{R}_n{\Gamma }_{nj}, \end{gathered}$$ (10)

where M_nj is the d × d matrix defined by (6) and R_n is the m × d matrix defined by (5). The second equation invokes the identities $V_j{V}_j^{+}{V}_j=V_j,\text{tr}\left(V_j{V}_j^{+}\right)=\text{rank}\left(V_j\right)$, and the cyclic permutation property of the trace.

In the case k = 1, the single update reduces to

$${\Gamma }_{n+1}=\frac{1}{r}{\left(Y-X{B}_n\right)}^{\top }{V}^{+}\left(Y-X{B}_n\right),$$

which matches the earlier result of Glanz & Carvalho (2018). When Γ_j is the scalar ${\sigma }_j^2$, the update (10) reduces to the classical update (Dempster et al., 1977)

$${\sigma }_{n+1,j}^2={\sigma }_{n,j}^2-\frac{{\sigma }_{n,j}^4}{r_j}\left[\text{tr}\left({\Omega }_n^{-1}{V}_j\right)-{\left(y-X{\beta }_n\right)}^{\top }{\Omega }_n^{-1}{V}_j{\Omega }_n^{-1}\left(y-X{\beta }_n\right)\right].$$

Algorithm 2 summarises the EM algorithm for fitting the multi-response model (2). The additive update of Γ_j in the EM algorithm differs markedly from the multiplicative update in the MM algorithm. The computational cost of each EM iteration is similar to that of MM. Both are dominated by the inversion of the md × md covariance matrix Ω_n.

For the univariate response case d = 1, Zhou et al. (2019) show that the MM algorithm enjoys a faster convergence rate than EM. Here, we verify the same behaviour empirically for a variance component model with d = 4 responses for m = 500 subjects, k = 3 variance components, and p = 3 fixed effect covariates. We start both algorithms from the same initial point in each of 100 simulation replicates. Figure 2 shows that MM algorithm converges faster than EM in all replicates.

4.3. A Problem Involving a Moore–Penrose Inverse

The derivations so far assume that the variance components Γ_j are unstructured covariance matrices with kd(d + 1)/2 parameters. Under a limited sample size, Γ_j cannot be estimated reliably, especially when the numbers of responses d and variance components k are large. A more parsimonious model imposes a low rank structure on all Γ_j except that associated with V_k = I_m. Then maximising the MM surrogate function (7) boils down to the problem of minimizing

$$\text{tr}(MX)+\text{tr}\left(N{X}^{+}\right)$$

where M and N are known d × d positive definite matrices and X is a positive semidefinite d × d matrix of rank r ≤ d. Denote the thin eigendecomposition X = UΣU^⊤, where U^⊤ U = I_r and Σ = diag(σ₁, …, σ_r) with σ_i > 0. We first determine the optimal eigenvalues σ_i and then eigenvectors U. The objective is

$$\begin{gathered} \text{tr}\left(U^{\top }MU\Sigma \right)+\text{tr}\left(U^{\top }NU{\Sigma }^{+}\right) \\ =\text{tr}(A\Sigma )+\text{tr}\left(B{\Sigma }^{+}\right) \\ =\sum _{i=1}^r{\sigma }_i{a}_{ii}+\sum _{i=1}^r{\sigma }_i^{-1}{b}_{ii}, \end{gathered}$$

where A = U^⊤ MU, B = U^⊤ NU. Setting the derivatives to zero yields the optimal eigenvalues ${\sigma }_i=\sqrt{b_{ii}/a_{ii}}$.

Now the task is to minimise

$$f(U)=2\sum _{i=1}^r\sqrt{a_{ii}{b}_{ii}}=2\sum _{i=1}^r\sqrt{u_i^{\top }M{u}_i{u}_i^{\top }N{u}_i}$$

under the orthogonality constraint U^⊤ U = I_r, which is subject to a suite of algorithms for manifold optimisation such as Manopt (Boumal et al., 2014) or a simple projected gradient descent algorithm. We record the gradient as

$${\nabla }_{u_i}f(U)=2\sqrt{\frac{u_i^{\top }N{u}_i}{u_i^{\top }M{u}_i}M}{u}_i+2\sqrt{\frac{u_i^{\top }M{u}_i}{u_i^{\top }N{u}_i}}N{u}_i.$$

Alternatively, split variables by replacing U by A and U^⊤ by B^⊤. Then impose the constraints A = B and B^⊤ A = I_r. The penalised objective

$$\text{tr}\left(B^{\top }MA\Sigma \right)+\text{tr}\left(B^{\top }NA{\Sigma }^{+}\right)+\frac{\rho }{2}{\Vert B^{\top }A-I_r\Vert }_F^2+\frac{\rho }{2}\Vert A-B{\Vert }_F^2$$

for ρ > 0 large has differential with respect to A of

$$\Sigma B^{\top }M+{\Sigma }^{+}{B}^{\top }N+\rho {\left(B^{\top }A-I_r\right)}^{\top }{B}^{\top }+\rho {(A-B)}^{\top }.$$

Set this equal to 0 and solve for A in the form

$$A^{\top }=-\left({\rho }^{-1}\Sigma B^{\top }M+{\rho }^{-1}{\Sigma }^{+}{B}^{\top }N-2B^{\top }\right){\left(I_d+B{B}^{\top }\right)}^{-1}.$$

A similar update holds for B.

5. DISCUSSION

The senior (citizen) author of this paper remembers being mesmerised by Nan Laird’s EM seminar at UCLA in the late 1970s. Nan opened an entirely new toolbox of optimisation. The beautiful abstraction and generality of the EM principle has served the statistics community well for decades. The principle is capable of generating maximum likelihood algorithms motivated by intermediate quantities of natural statistical interest. It is worth stressing that EM and Fisher scoring are unique contributions by statisticians to optimisation practice. However, there is no panacea in optimisation. Each problem class presents unique challenges and deserves to be attacked from a variety of perspectives. Often hybrid algorithms work best.

The MM principle distils the essence of EM and frees it from the sometimes elusive notion of missing data. As we have witnessed, EM and MM algorithms for the same problem do not necessarily coincide. When they differ, their rates of convergence and computational complexity can also differ. Our exposition of EM and MM algorithms for variance component models illustrates these points. In this case, the MM algorithm appears faster.

The current paper offers, at best, a snapshot of the current state of the MM art. New applications are in the pipeline. Our recent research on constrained optimisation shows how the MM principle, set projection, and the Courant penalty method can cooperate to solve constrained problems involving nonconvexity and sparsity (Chi et al., 2014; Keys et al., 2019; Landeros et al., 2022; Xu et al., 2017). Many challenges remain in theory, numerical practice, and software development. Fortunately, current researchers stand on the shoulders of giants such as Nan Laird and Jan de Leeuw in attacking these issues. We are profoundly grateful to Nan for her many advances in computational statistics. Only a handful of statisticians can claim a legacy of such distinction.