A Brief Survey of Modern Optimization for Statisticians

Abstract

Modern computational statistics is turning more and more to high-dimensional optimization to handle the deluge of big data. Once a model is formulated, its parameters can be estimated by optimization. Because model parsimony is important, models routinely include nondifferentiable penalty terms such as the lasso. This sober reality complicates minimization and maximization. Our broad survey stresses a few important principles in algorithm design. Rather than view these principles in isolation, it is more productive to mix and match them. A few well chosen examples illustrate this point. Algorithm derivation is also emphasized, and theory is downplayed, particularly the abstractions of the convex calculus. Thus, our survey should be useful and accessible to a broad audience.

Introduction

Modern statistics represents a confluence of data, algorithms, practical inference, and subject area knowledge. As data mining expands, computational statistics is assuming greater prominence. Surprisingly, the confident prediction of the previous generation that Bayesian methods would ultimately supplant frequentist methods has given way to a realization that Markov chain Monte Carlo (MCMC) may be too slow to handle modern data sets. Size matters because large data sets stress computer storage and processing power to the breaking point. The most successful compromises between Bayesian and frequentist methods now rely on penalization and optimization. Penalties serve as priors and steer parameter estimates in realistic directions. In classical statistics estimation usually meant least squares and maximum likelihood with smooth objective functions. In a search for sparse representations, mathematical scientists have introduced nondifferentiable penalties such as the lasso and the nuclear norm. To survive in this alien terrain, statisticians are being forced to master exotic branches of mathematics such as convex calculus [39, 40]. Thus, the uneasy but productive relationship between statistics and mathematics continues, but in a different guise and mediated by new concerns.

The purpose of this survey article is to provide a few glimpses of the new optimization algorithms being crafted by computational statisticians and applied mathematicians. Although a survey of convex calculus for statisticians would certainly be helpful, our emphasis is more concrete. The truth of the matter is that a few broad categories of algorithms dominate. Furthermore, difficult problems require that several algorithmic pieces be assembled into a well coordinated whole. Put another way, from a handful of basic ideas, computational statisticians often weave a complex tapestry of algorithms that meets the needs of a specific problem. No algorithm category should be dismissed a priori in tackling a new problem. There is plenty of room for creativity and experimentation. Algorithms are made for tinkering. When one part fails or falters, it can be replaced by a faster or more robust part.

This survey will treat the following methods: (a) block descent, (b) steepest descent, (c) Newton’s method, quasi-Newton methods, and scoring, (d) the MM and EM algorithms, (e) penalized estimation, (f) the augmented Lagrangian method for constrained optimization, and (g) acceleration of fixed point algorithms. As we have mentioned, often the best algorithms combine several themes. We will illustrate the various themes by a sequence of examples. Although we avoid difficult theory and convergence proofs, we will try to point out along the way a few motivating ideas that stand behind most algorithms. For example, as its name indicates, steepest descent algorithms search along the direction of fastest decrease of the objective function. Newton’s method and its variants all rely on the notion of local quadratic approximation, thus correcting the often poor linear approximation of steepest descent. In high dimensions, Newton’s method stalls because it involves calculating and inverting large matrices of second derivatives.

The MM and EM algorithms replace the objective function by a simpler surrogate function. By design, optimizing the surrogate function sends the objective function downhill in minimization and uphill in maximization. In constructing the surrogate function for an EM algorithm, statisticians rely on notions of missing data. The more general MM algorithm calls on skills in inequalities and convex analysis. More often than not, concrete problems also involve parameter constraints. Modern penalty methods incorporate the constraints by imposing penalties on the objective function. A tuning parameter scales the strength of the penalties. In the classical penalty method, the constrained solution is recovered as the tuning parameter tends to infinity. In the augmented Lagrangian method, the constrained solution emerges for a finite value of the tuning parameter.

In the remaining sections, we adopt several notational conventions. Vectors and matrices appear in boldface type; for the most part parameters appear as Greek letters. The differential df(θ) of a scalar-valued function f(θ) equals its row vector of partial derivatives; the transpose ▿f(θ) of the differential is the gradient. The second differential d²f(θ) is the Hessian matrix of second partial derivatives. The Euclidean norm of a vector b and the spectral norm of a matrix A are denoted by ∥b∥ and ∥A∥, respectively. All other norms will be appropriately subscripted. The nth entry b_n of a vector b must be distinguished from the nth vector b_n in a sequence of vectors. To maintain consistency, b_ni denotes the ith entry of b_n. A similar convention holds for sequences of matrices.

Block Descent

Block relaxation (either block descent or block ascent) divides the parameters into disjoint blocks and cycles through the blocks, updating only those parameters within the pertinent block at each stage of a cycle [21]. For the sake of brevity, we consider only block descent. In updating a block, we minimize the objective function over the block. Hence, block descent possesses the desirable descent property of always forcing the objective function downhill. When each block consists of a single parameter, block descent is called cyclic coordinate descent. The coordinate updates need not be explicit. In high-dimensional problems, implementation of one-dimensional Newton searches is often compatible with fast overall convergence. Block descent is best suited to unconstrained problems where the domain of the objective function reduces to a Cartesian product of the subdomains associated with the different blocks. Obviously, exact block updates are a huge advantage. Constraints can present insuperable barriers to coordinate descent because parameters get locked into place. In some problems it is advantageous to consider overlapping blocks.

Example 0.1. Nonnegative Least Squares

For a positive definite matrix A = (a_ij) and vector b = (b_i), consider minimizing the quadratic function

$$f\left(\theta \right)=\frac{1}{2}{\theta }^t\mathit{A}\theta +b^t\theta +c$$

subject to the constraints θ_i ≥ 0 for all i. In the case of least squares, A = X^tX and b = −X^ty for some design matrix X and response vector y. Equating the partial derivative of f(θ) with respect to θ_i to 0 gives

$$0=\sum _j{a}_{\mathit{ij}}{\theta }_j+b_i.$$

Rearrangement now yields the unrestricted minimum

$${\theta }_{n+1,i}={\theta }_{\mathit{ni}}-\frac{1}{a_{ii}}\left(b_i+\sum _j{a}_{\mathit{ij}}{\theta }_{\mathit{nj}}\right).$$

Taking into account the nonnegativity constraint, this must be amended to

$${\theta }_{n+1,i}=\max \left\{0,{\theta }_{\mathit{ni}}-\frac{1}{a_{ii}}\left(b_i+\sum _j{a}_{\mathit{ij}}{\theta }_{\mathit{nj}}\right)\right\}$$

at stage n + 1 to construct the coordinate descent update of θ_i.

Example 0.2. Matrix Factorization by Alternating Least Squares

In the 1960s Kruskal [47] applied the method of alternating least squares to factorial ANOVA. Later the subject was taken up by de Leeuw and colleagues [32]. Suppose U is a m × q matrix whose columns u₁, … , u_q represent data vectors. In many applications it is reasonable to postulate a reduced number of prototypes v₁, … , v_p and write

$$u_j\approx \sum _{k=1}^p{\upsilon }_k{w}_{\mathit{kj}}$$

for certain nonnegative weights w_kj. The matrix W = (w_kj) is p × q. If p is small compared to q, then the representation U ≈ VW compresses the data for easier storage and retrieval. Depending on the circumstances, one may want to add further constraints [24]. For instance, if the entries of U are nonnegative, then it is often reasonable to demand that the entries of V be nonnegative as well [55, 68]. If we want each u_j to equal a convex combination of the prototypes, then constraining the column sums of W to equal 1 is indicated.

One way of estimating V and W is to minimize the squared Frobenius norm

$${\mid \mid U-\mathit{VW}\mid \mid }_F^2=\sum _{i=1}^m\sum _{j=1}^q{\left(u_{\mathit{ij}}-\sum _{k=1}^p{\upsilon }_{\mathit{ik}}{w}_{kj}\right)}^2.$$

No explicit solution is known, but alternating least squares offers an iterative attack. If W is fixed, then we can update the ith row of V by minimizing the sum of squares

$$\sum _{j=1}^q{\left(u_{\mathit{ij}}-\sum _{k=1}^p{\upsilon }_{\mathit{ik}}{w}_{kj}\right)}^2.$$

Similarly, if V is fixed, then we can update the jth column of W by minimizing the sum of squares

$$\sum _{i=1}^n{\left(u_{\mathit{ij}}-\sum _{k=1}^p{\upsilon }_{\mathit{ik}}{w}_{kj}\right)}^2.$$

Thus, block descent solves a sequence of least squares problems, some of which are constrained.

Steepest Descent

The first-order Taylor expansion

$$f(\theta +\gamma )=f\left(\theta \right)+\mathit{df}\left(\theta \right)\gamma +o(\mid \mid \gamma \mid \mid )$$

of a differentiable function f(θ) around θ motivates the method of steepest descent. In view of the Cauchy-Schwarz inequality, the choice

$$\gamma =-\nabla f\left(\theta \right)∕\mid \mid \nabla f\left(\theta \right)\mid \mid$$

minimizes the linear term df(θ)γ of the expansion over the sphere of unit vectors. Of course, if ▿f(θ) = 0, then θ is a stationary point. The steepest descent algorithm iterates according to

$${\theta }_{n+1}={\theta }_n-s\nabla f\left({\theta }_n\right)$$ (1)

for some scalar s > 0. If s is sufficiently small, then the descent property f(θ_n+1) < f(θ_n) holds. The most sophisticated version of the algorithm determines s by searching for the minimum of the objective function along the direction of steepest descent. Among the many methods of line search, the methods of false position, cubic interpolation, and golden section stand out [53]. These are all local search methods, and unless some guarantee of convexity exists, confusion of local and global minima can occur.

The method of steepest descent often exhibits zigzagging and a painfully slow rate of convergence. For these reasons it was largely replaced in practice by Newton’s method and its variants. However, the sheer scale of modern optimization problems has led to a re-evaluation. The avoidance of second derivatives and Hessian approximations is now viewed as an virtue. Furthermore, the method has been generalized to nondifferentiable problems by substituting the forward directional derivative

$$d_{\nu }f\left(\theta \right)=\underset{s\downarrow 0}{\lim }\frac{f(\theta +s\nu )-f\left(\theta \right)}{s}$$

for the gradient [84]. Here the idea is to choose a unit search vector ν to minimize d_νf(θ). In some instances this secondary problem can be attacked by linear programming. For a convex problem, the condition d_νf(θ) ≥ 0 for all ν is both necessary and sufficient for θ to be a minimum point. If the domain of f(θ) equals a convex set C, then only tangent directions ν = μ−θ with μ ∈ C come into play.

Steepest descent also has a role to play in constrained optimization. Suppose we want to minimize f(θ) subject to the constraint θ ∈ C for some closed convex set. The projected gradient method capitalizes on the steepest descent update (1) by projecting it onto the set C [35, 56, 79]. It is well known that for a point x external to C, there is a closest point P_C(x) to x in C. Explicit formulas for the projection operator P_C(x) exist when C is a box, Euclidean ball, hyperplane, or halfspace. Fast algorithms for computing P_C(x) exist for the unit simplex, the l₁ ball, and the cone of positive semidefinite matrices [27, 62].

Choice of the scalar s in the update (1) is crucial. Current theory suggests taking s to equal r/L, where L is a Lipschitz constant for the gradient ▿f(θ) and r belongs to the interval (0, 2). In particular, the Lipschitz inequality

$$\mid \mid \nabla f\left(\theta \right)-\nabla f\left(\gamma \right)\mid \mid \le L\mid \mid \theta -\gamma \mid \mid$$

is valid for L = sup_θ ∥d²f(θ)∥, whenever this quantity is finite. In practice, the Lipschitz constant L must be estimated. Any induced matrix norm ∥ · ∥_† can be substituted for the spectral norm ∥ · ∥ in the defining supremum and will give an upper bound on L.

Example 0.3. Coordinate Descent versus the Projected Gradient Method

As a test problem, we generated a random 100 × 50 design matrix X with i.i.d. standard normal entries, a random 50 × 1 parameter vector θ with i.i.d. uniform [0,1] entries, and a random 100 × 1 error vector e with i.i.d. standard normal entries. In this setting the response y = Xθ + e. We then compared coordinate descent, the projected gradient method (for L equal to the spectral radius of X^tX and r equal to 1.0, 1.75, and 2.0), and the MM algorithm explained later in Example 0.6. All computer runs start from the common point θ₀ whose entries are filled with i.i.d. uniform [0,1] random deviates. Figure 1 plots the progress of each algorithm as measured by the relative difference

$$\frac{f\left(b_n\right)-f\left(b_{\infty }\right)}{f\left(b_{\infty }\right)},$$ (2)

between the loss at the current iteration and the ultimate loss at convergence. It is interesting how well coordinate descent performs compared to projected gradient descent. The slower convergence of the MM algorithm is probably a consequence of the fact that its multiplicative updates slow down as they approach the 0 boundary. Note also the importance of choosing a good step size in the projected gradient algorithm. Inflated steps accelerate convergence, but excessively inflated steps hamper it.

Figure 1.

Comparing the rate of convergence of three algorithms on a nonnegative least squares problem. CD = coordinate descent, PG = projected gradient, and MM = majorize-minimize.

Variations on Newton’s Method

The primary advantage of Newton’s method is its speed of convergence in low-dimensional problems. Its many variants seek to retain its fast convergence while taming its defects. The variants all revolve around the core idea of locally approximating the objective function by a strictly convex quadratic. At each iteration the quadratic approximation is optimized subject to safeguards that keep the iterates from overshooting and veering toward irrelevant stationary points.

Consider minimizing the real-valued function f(θ) defined on an open set S ⊂ R^p. Assuming that f(θ) is twice differentiable, we have the second order Taylor expansion

$$f\left(\gamma \right)=f\left(\theta \right)+\mathit{df}\left(\theta \right)(\gamma -\theta )+\frac{1}{2}{(\gamma -\theta )}^2{d}^{2}f\left(\alpha \right)(\gamma -\theta )$$

for some α on the line segment [θ, γ]. This expansion suggests that we substitute d²f(θ) for d²f(α) and approximate f(γ) by the resulting quadratic. If we take this approximation seriously, then we can solve for its minimum point γ as

$$\gamma =\theta -d^{2}f{\left(\theta \right)}^{-1}\nabla f\left(\theta \right).$$

In Newton’s method we iterate according to

$${\theta }_{n+1}={\theta }_n-{\mathit{sd}}^{2}f{\left({\theta }_n\right)}^{-1}\nabla f\left({\theta }_n\right)$$ (3)

for step length constant s with default value 1. Any stationary point of f(θ) is a fixed point of Newton’s method.

There is nothing to prevent Newton’s method from heading uphill rather than downhill. The first order expansion

$$f\left({\theta }_{n+1}\right)=f\left({\theta }_n\right)-s\mathit{df}\left({\theta }_n\right)d^{2}f{\left({\theta }_n\right)}^{-1}\nabla f\left({\theta }_n\right)+o\left(s\right)$$

makes it clear that the descent property holds provided s > 0 is small enough and the Hessian matrix d²f(θ_n) is positive definite. When d²f(θ_n) is not positive definite, it is usually replaced by a positive definite approximation H_n in the update (3).

Backtracking is crucial to avoid overshooting. In the step-halving version of backtracking, one starts with s = 1. If the descent property holds, then one takes the Newton step. Otherwise, $\frac{s}{2}$ is substituted for s, θ_n+1 is recalculated, and the descent property is rechecked. Eventually, a small enough s is generated to guarantee f(θ_n+1) < f(θ_n).

In the next two examples we adopt standard statistical language. The outcome of a statistical experiment is summarized by a loglikelihood L(θ). Its gradient ▿L(θ) is called the score, and its second differential d²L(θ), after a change in sign, is call the observed information. In maximum likelihood estimation, one maximizes L(θ) with respect to the parameter vector θ.

Example 0.4. Newton’s Method for Binomial Regression

Consider binomial regression with m independent responses y₁, … , y_m. Each y_i represents a count between 0 and k_i with success probability π_i(θ) per trial. The loglikelihood, score, and observed information amount to

$$\begin{array}{cc}\hfill L\left(\theta \right)=& \sum _{i=1}^m\left[y_i\ln {\pi }_i\right(\theta )+(k_i-y_i\left)\ln \right[1-{\pi }_i\left(\theta \right)]\hfill \\ \hfill \nabla L\left(\theta \right)=& \sum _{i=1}^m\frac{y_i-k_i{\pi }_i\left(\theta \right)}{{\pi }_i\left(\theta \right)]1-{\pi }_i(\theta \left)\right]}\nabla {\pi }_i\left(\theta \right)\hfill \\ \hfill -d^{2}L\left(\theta \right)=& -\sum _{i=1}^m\frac{y_i-k_i{\pi }_i\left(\theta \right)}{{\pi }_i\left(\theta \right)[1-{\pi }_i(\theta \left)\right]}{d}^2{\pi }_i\left(\theta \right)\hfill \\ \hfill & +\sum _{i=1}^m\left\{\frac{y_i}{{\pi }_i{\left(\theta \right)}^2}+\frac{k_i-y_i}{{[1-{\pi }_i(\theta \left)\right]}^2}\right\}\nabla {\pi }_i\left(\theta \right)d{\pi }_i\left(\theta \right).\hfill \end{array}$$

Because E(y_i) = k_iπ_i(θ), the observed information can be approximated by

$$\begin{array}{cc}\hfill -d^{2}L\left(\theta \right)\approx & \sum _{i=1}^m\left\{\frac{y_i}{{\pi }_1{\left(\theta \right)}^2}+\frac{k_i-y_i}{{[1-{\pi }_i(\theta \left)\right]}^2}\right\}\nabla {\pi }_i\left(\theta \right)d{\pi }_i\left(\theta \right)\hfill \\ \hfill \approx & \sum _{i=1}^m\left\{\frac{k_i}{{\pi }_i\left(\theta \right)}+\frac{k_i}{[1-{\pi }_i(\theta \left)\right]}\right\}\nabla {\pi }_i\left(\theta \right)d{\pi }_i\left(\theta \right).\hfill \end{array}$$

Because we seek to maximize rather than minimize L(θ), we want −d²L(θ) to be positive definite. Fortunately, both approximations fulfill this requirement. The second approximation leads to the scoring algorithm discussed later.

Example 0.5. Poisson Multigraph Model

In a graph the number of edges between any two nodes is 0 or 1. A multigraph allows an arbitrary number of edges between any two nodes. Multigraphs are natural structures for modeling the internet and gene and protein networks. Here we consider a multigraph with a random number of edges X_ij connecting every pair of nodes {i, j}. In particular, we assume that the X_ij are independent Poisson random variables with means μ_ij. As a plausible model for ranking nodes, we take μ_ij = θ_iθ_j, where θ_i and θ_j are nonnegative propensities [72]. The loglikelihood of the observed edge counts x_ij = x_ji amounts to

$$\begin{array}{cc}\hfill L\left(\theta \right)& =\sum _{\{i,j\}}(x_{\mathit{ij}}\ln {\mu }_{\mathit{ij}}-{\mu }_{\mathit{ij}}-\ln x_{\mathit{ij}}!)\hfill \\ \hfill & =\sum _{\{i,j\}}\left[x_{\mathit{ij}}\right(\ln {\theta }_i+\ln {\theta }_j)-{\theta }_i{\theta }_j-\ln x_{\mathit{ij}}!].\hfill \end{array}$$

The score vector has entries

$$\frac{\partial }{\partial {\theta }_i}L\left(\theta \right)=\sum _{j\ne i}\left(\frac{x_{\mathit{ij}}}{{\theta }_i}-{\theta }_j\right),$$

and the observed information matrix has entries

$$-\frac{{\partial }^2}{\partial {\theta }_i\partial {\theta }_j}L\left(\theta \right)\{\begin{array}{cc}1\hfill & i\ne j\hfill \\ \frac{1}{{\theta }_i^2}\sum _{k\ne i}{x}_{\mathit{ik}}\hfill & i=j.\hfill \end{array}\phantom{\}}$$

For p nodes the matrix −d²L(p) is p × p, and inverting it seems out of the question when p is large. Fortunately, the Sherman-Morrison formula comes to the rescue. If we write −d²L(θ) as D + 11^t with D diagonal, then the explicit inverse

$${(D+{11}^t)}^{-1}=D^{-1}-\frac{1}{1+1^t{D}^{-1}1}{D}^{-1}{11}^t{D}^{-1}$$

is available. This makes Newton’s method trivial to implement as long as one respects the bounds θ_i ≥ 0. More generally, it is always cheap to invert a low-rank perturbation of an explicitly invertible matrix.

In maximum likelihood estimation, the method of steepest ascent replaces the observed information matrix −d²L(θ) by the identity matrix I. Fisher’s scoring algorithm makes the far more effective choice [67] of replacing the observed information matrix by the expected information matrix J(θ) = E[−d²L(θ)]. The alternative representation J(θ) = Var[▿rL(θ)] of J(θ) as a variance matrix demonstrates that it is positive semidefinite. Usually it is positive definite as well and serves as an excellent substitute for −d²L(θ) in Newton’s method. The inverse matrices $-d^{2}L{\left(\widehat{\theta }\right)}^{-1}$ and $J{\left(\widehat{\theta }\right)}^{-1}$ immediately supply the asymptotic variances and covariances of the maximum likelihood estimate $\widehat{\theta }$ [73].

The score and expected information simplify considerably for exponential families of densities [8, 11, 36, 44, 63]. Recall that the density of a vector random variable Y from an exponential family can be written as

$$f(y\mid \theta )=g\left(y\right)e^{\beta \left(\theta \right)+h{\left(y\right)}^t\gamma \left(\theta \right)}$$ (4)

relative to some measure ν [25, 73]. The function h(y) in equation (4) is the sufficient statistic. The maximum likelihood estimate of the parameter vector θ depends on an observation y only through h(y). Predictors of y are incorporated into the functions β(θ) and γ(θ). If γ(θ) is linear in θ, then J(θ) = −d²L(θ) = −d²β(θ), and scoring coincides with Newton’s method. If in addition J(θ) is positive definite, then L(θ) is strictly concave and possesses at most a single local maximum, which is necessarily the global maximum.

Both the score vector and expected information matrix can be expressed succinctly in terms of the mean vector μ(θ) = E[h(y)] and the variance matrix Σ(θ) = Var[h(y)] of the sufficient statistic. Standard arguments show that

$$\begin{array}{cc}\hfill \nabla L\left(\theta \right)& =d\mu {\left(\theta \right)}^t\sum {\left(\theta \right)}^{-1}\left[h\right(y)-\mu (\theta \left)\right]\hfill \\ \hfill J\left(\theta \right)& =d\mu {\left(\theta \right)}^t\sum {\left(\theta \right)}^{-1}d\mu \left(\theta \right).\hfill \end{array}$$

These formulas have had an enormous impact on nonlinear regression and fitting generalized linear models. Applied statistics as we know it would be nearly impossible without them. Implementation of scoring is almost always safeguarded by step halving and upgraded to handle linear constraints and parameter bounds. The notion of quadratic approximation is still the key, but each step of constrained scoring must solve a quadratic program.

In parallel with developments in statistics, numerical analysts sought substitutes for Newton’s method. Their e orts a generation ago focused on quasi-Newton methods for generic smooth functions [23, 65]. Once again the core idea was successive quadratic approximation. A good quasi-Newton method: (a) minimizes a quadratic function f(θ) from R^p to R in p steps, (b) avoids evaluation of d²f(θ), (c) adapts readily to simple parameter constraints, and (d) exploits inexact line searches.

Quasi-Newton methods update the current approximation H_n to the second differential d²f(θ) of an objective function f(θ) by a rank-one or rank-two perturbation satisfying a secant condition. The secant condition captures the first-order Taylor approximation

$$\nabla f\left({\theta }_{n+1}\right)-\nabla f\left({\theta }_n\right)\approx d^{2}f\left({\theta }_n\right)({\theta }_{n+1}-{\theta }_n).$$

If we define the gradient and argument differences

$$\begin{array}{cc}\hfill g_n& =\nabla f\left({\theta }_{n+1}\right)-\nabla f\left({\theta }_n\right)\hfill \\ \hfill d_n& ={\theta }_{n+1}-{\theta }_n,\hfill \end{array}$$

then the secant condition reads H_n+1d_n = g_n. Davidon [19] discovered that the unique symmetric rank-one update to H_n satisfying the secant condition is

$$H_{n+1}=H_n+c_n{\upsilon }_n{\upsilon }_n^t,$$

where the constant c_n and the vector v_n are determined by

$$\begin{array}{cc}\hfill c_n& =-\frac{1}{{(H_n{d}_n-g_n)}^2{d}_n}\hfill \\ \hfill {\upsilon }_n& =H_n{d}_n-g_n.\hfill \end{array}$$

When the inner product (H_nd_n − g_n)^td_n is too close to 0, there are two possibilities. Either the secant adjustment is ignored, and the value H_n is retained for H_n+1, or one resorts to a trust region strategy [65].

In the trust region method, one minimizes the quadratic approximation to f(θ) subject to the spherical constraint ∥θ − θ_n∥² ≤ r² for a fixed radius r. This constrained optimization problem has a solution regardless of whether H_n is positive definite. Working within a trust region prevents absurdly large steps in the early stages of minimization. With appropriate safeguards, some numerical analysts [18, 45] consider Davidon’s rank-one update superior to the widely used BFGS update, named after Broyden, Fletcher, Goldfarb, and Shanno. This rank-two perturbation is guaranteed to maintain positive definiteness and is better understood theoretically than the symmetric rank-one update. Also of interest is the DFP (Davidon, Fletcher, and Powell) rank-two update, which applies to the inverse $H_n^{-1}$ of H_n. Although the DFP update ostensibly avoids matrix inversion, the consensus is that the BFGS update is superior to it in numerical practice [23].

The MM and EM Algorithms

The numerical analysts Ortega and Rheinboldt [66] first articulated the MM principle; de Leeuw [20] saw its potential and created the first MM algorithm. The MM algorithm currently enjoys its greatest vogue in computational statistics [41, 54, 90]. The basic idea is to convert a hard optimization problem into a sequence of simpler ones. In minimization the MM principle majorizes the objective function f(θ) by a surrogate function g(θ ∣ θ_n) anchored at the current point θ_n. Majorization combines the tangency condition g(θ_n ∣ θ_n) = f(θ_n) and the domination condition g(θ ∣ θ_n) ≥ f(θ) for all θ. The next iterate of the MM algorithm is defined to minimize g(θ ∣ θ_n). Because

$$f\left({\theta }_{n+1}\right)\le g({\theta }_{n+1}\mid {\theta }_n)\le g({\theta }_n\mid {\theta }_n)=f\left({\theta }_n\right),$$

the MM iterates generate a descent algorithm driving the objective function downhill. Strictly speaking, the descent property depends only on decreasing g(θ ∣ θ_n), not on minimizing it. Constraint satisfaction is automatically enforced in finding θ_n+1. Under appropriate regularity conditions, an MM algorithm is guaranteed to converge to a local minimum of the objective function [52]. In maximization, we first minorize and then maximize. Thus, the acronym MM does double duty in the forms majorize-minimize and minorize-maximize.

When it is successful, the MM algorithm simplifies optimization by: (a) separating the variables of a problem, (b) avoiding large matrix inversions, (c) linearizing a problem, (d) restoring symmetry, (e) dealing with equality and inequality constraints gracefully, and (f) turning a nondifferentiable problem into a smooth problem. The art in devising an MM algorithm lies in choosing a tractable surrogate function g(θ ∣ θ_n) that hugs the objective function f(θ) as tightly possible.

The majorization relation between functions is closed under the formation of sums, nonnegative products, limits, and composition with an increasing function. These rules allow one to work piecemeal in simplifying complicated objective functions. Skill in dealing with inequalities is crucial in constructing majorizations. Classical inequalities such as Jensen’s inequality, the information inequality, the arithmetic-geometric mean inequality, and the Cauchy-Schwartz prove useful in many problems. The supporting hyperplane property of a convex function and the quadratic upper bound principle of Böhning and Lindsay [5] also find wide application.

Example 0.6. An MM Algorithm for Nonnegative Least Squares

Sha et al [81] devised an MM algorithm for Example 0.1. The diagonal terms $a_{\mathit{ii}}{\theta }_i^2$ they retain as presented. The off-diagonal terms a_ijθ_iθ_j they majorize according to the sign of the coefficient a_ij. When the sign of a_ij is positive, they apply the majorization

$$\mathit{xy}\le \frac{y_n}{2x_n}{x}^2+\frac{x_n}{2y_n}{y}^2,$$

which just a rearrangement of the inequality

$$0\le {\left(\sqrt{\frac{y_n}{x_n}}x-\sqrt{\frac{x_n}{y_n}}y\right)}^2,$$

with equality when x = x_n and y = y_n. When the sign of a_ij is negative, they apply the majorization

$$-\mathit{xy}\le -x_n{y}_n\left[1+\ln \left(\frac{x}{x_n}\right)+\ln \left(\frac{y}{y_n}\right)\right],$$

which is just a rearrangement of the simple inequality z ≥ 1 + ln z with z = xy/(x_ny_n). The value z = 1 gives equality in the inequality. Both majorizations separate parameters and allow one to minimize the surrogate function parameter by parameter. Indeed, if we define matrices A⁺ and A⁻ with entries max{a_ij, 0} and −min{a_ij, 0}, respectively, then the resulting MM algorithm iterates according to

$${\theta }_{n+1,i}={\theta }_{n,i}\left[\frac{-b_i+\sqrt{b_i^2+4{\left(A^{+}{\theta }_n\right)}_i{\left(A^{-}{\theta }_n\right)}_i}}{2{\left(A^{+}{\theta }_n\right)}_i}\right].$$

All entries of the initial point θ₀ should be positive; otherwise, the MM algorithm stalls. The updates occur in parallel. In contrast, the cyclic coordinate descent updates are sequential. Figure 1 depicts the progress of the MM algorithm on our nonnegative least squares problem.

Example 0.7. Locating a Gunshot

Locating the time and place of a gunshot is a typical global positioning problem [82]. In a certain city m sensors located at the points x₁, … , x_m are installed. A signal, say a gunshot sound, is sent from an unknown location θ at unknown time α and known speed s and arrives at location j at time y_j observed with random measurement error. The problem is to estimate the vector θ and the scalar α from the observed data y₁, … , y_m. Other problems of this nature include pinpointing the epicenter of an earthquake and the detonation point of a nuclear explosion. This estimation problem can be attacked by a combination of block descent and the MM principle.

If we assume Gaussian random errors, then maximum likelihood estimation reduces to minimizing the criterion

$$\begin{array}{cc}\hfill f(\theta ,\alpha )& =\frac{1}{2}\sum _{j=1}^m{(y_j-s^{-1}\mid \mid \theta -x_j\mid \mid -\alpha )}^2\hfill \\ \hfill & =\frac{1}{2s^2}\sum _{j=1}^m{({\mathit{sy}}_j-\mid \mid \theta -x_j\mid \mid -\alpha s)}^2.\hfill \end{array}$$

The equivalence of the two representations of f(θ, α) shows that it suffices to solve the problem with speed s = 1. In the remaining discussion we make this assumption. For fixed θ estimation of α reduces to a least squares problem with the obvious solution

$$\alpha =\frac{1}{m}\sum _{j=1}^m\left(y_j-\mid \mid \theta -x_j\mid \mid \right).$$

.

To update θ with α fixed, we rewrite f(θ, α) as

$$f(\theta ,\alpha )=\frac{1}{2}\sum _{j=1}^m[{(y_j-\alpha )}^2-2(y_j-\alpha )\mid \mid \theta -x_j{\mid \mid +\mid \mid \theta -x_j\mid \mid }^2].$$

The middle terms −2(y_j − α)∥θ − x_j∥ are awkward to deal with in minimization. Depending on the sign of the coefficient −2(y_j −α), we majorized them in two different ways. If the sign is negative, then we employ the Cauchy-Schwarz majorization

$$-\mid \mid \theta -x_j\mid \mid \le -{\mid \mid {\theta }_n-x_j\mid \mid }^{-1}{({\theta }_n-x_j)}^t(\theta -x_j).$$

If the sign is positive, then we employ the more subtle majorization

$$\mid \mid \theta -x_j\mid \mid \le \mid \mid {\theta }_n-x_j\mid \mid +\frac{1}{2\mid \mid {\theta }_n-x_j\mid \mid }({\mid \mid \theta -x_j\mid \mid }^2-\mid \mid {\theta }_n-x_j\mid \mid ).$$

To derive this second majorization, note that $\sqrt{u}$ is a concave function on (0, ∞). It therefore satisfies the dominating hyperplane inequality

$$\sqrt{u}\le \sqrt{u_n}+\frac{1}{2\sqrt{u_n}}(u-u_n).$$

Now substitute ∥θ − x_j∥² for u. These maneuvers separate parameters and reduce the surrogate to a sum of linear terms and squared Euclidean norms. The minimization of the surrogate yields the MM update

$${\theta }_{n+1}=\frac{\sum _{j=1}^m\left[1+\frac{(\alpha -y_j)}{\mid \mid {\theta }_n-x_j\mid \mid }\right]x_j-\left[\sum _{\alpha \le y_j}\frac{(\alpha -y_j)}{\mid \mid {\theta }_n-x_j\mid \mid }\right]{\theta }_n}{m+\sum _{\alpha >y_j}\frac{\alpha -y_j}{\mid \mid {\theta }_n-x_j\mid \mid }}$$

of θ for α fixed. The condition α > y_j in this update is usually vacuous. By design f(θ, α) decreases after each cycle of updating α and θ.

The celebrated expectation-maximization (EM) algorithm is one the most potent optimization tools in the statistician’s toolkit [22, 59]. The E step in the EM algorithm creates a surrogate function, the Q function in the literature, that minorizes the loglikelihood. Thus, every EM algorithm is an MM algorithm. If y is the observed data and x is the complete data, then the Q function is defined as the conditional expectation

$$Q(\theta \mid {\theta }_n)=\mathrm{E}\left[\ln f\right(X\mid \theta )\mid Y=y,{\theta }_n],$$

where f(x ∣ θ) denotes the complete data loglikelihood, upper case letters indicate random vectors, and lower case letters indicate corresponding realizations of these random vectors. In the M step of the EM algorithm, one calculates the next iterate θ_n+1 by maximizing Q(θ ∣ θ_n) with respect to θ.

Example 0.8. MM versus EM for the Dirichlet-Multinomial Distribution

When multivariate count data exhibit over-dispersion, the Dirichlet-multinomial distribution is preferred to the multinomial distribution. In the Dirichlet-multinomial model, the multinomial probabilities p = (p₁, … , p_d) follow a Dirichlet distribution with parameter vector α = (α₁, … , _d) having positive components. For a multivariate count vector x = (x₁, … , x_d) with batch size $\mid x\mid ={\sum }_{j=1}^d{x}_j$, the probability mass function is accordingly

$$\begin{array}{cc}\hfill h(x\mid \alpha )& ={\int }_{{\Delta }_d}\left(\begin{array}{c}\hfill \mid x\mid \hfill \\ \hfill x\hfill \end{array}\right)\prod _{j=1}^d{p}_j^{x_j}\frac{\Gamma (\mid \alpha \mid )}{\prod _{j=1}^d\Gamma \left({\alpha }_j\right)}\prod _{j=1}^d{p}_j^{{\alpha }_j-1}\mathit{dp}\hfill \\ \hfill & =\left(\begin{array}{c}\hfill \mid x\mid \hfill \\ \hfill x\hfill \end{array}\right)\frac{\prod _{j=1}^d\Gamma ({\alpha }_j+x_j)}{\Gamma (\mid \alpha \mid +\mid x\mid )}\frac{\Gamma (\mid \alpha \mid )}{\prod _{j=1}^d\Gamma \left({\alpha }_j\right)}\hfill \\ \hfill & =\left(\begin{array}{c}\hfill \mid x\mid \hfill \\ \hfill x\hfill \end{array}\right)\frac{\prod _{j=1}^d\left({\alpha }_j\right)x_j}{{(\mid \alpha \mid )}_{\mid x\mid }},\hfill \end{array}$$ (5)

where Δ_d is the unit simplex in d dimensions, ∣α∣ equals ${\sum }_{j=1}^d{\alpha }_j$, and ${\left(a\right)}_k={\prod }_{i=0}^{k-1}(a+i)$ denotes a rising factorial. The last equality in (6) follows from the factorial property Γ(a+1)/Γ(a) = a of the gamma function. Given independent data points x₁, … , x_m, the loglikelihood is

$$L\left(\alpha \right)=\sum _{i=1}^m\ln \left(\begin{array}{c}\hfill \mid x_i\mid \hfill \\ \hfill x\hfill \end{array}\right)+\sum _{i=1}^m\sum _{j=1}^d\sum _{k=0}^{x_{\mathit{ij}}-1}\ln ({\alpha }_j+k)-\sum _{i=1}^m\sum _{k=0}^{\mid x_i\mid -1}\ln (\mid \alpha \mid +k).$$

The lack of concavity of L(α) may cause instability in Newton’s method when it is started far from the optimal point. Fisher’s scoring algorithm is computationally prohibitive because calculation of the expected information matrix involves numerous evaluations of beta-binomial tail probabilities. The ascent property makes EM and MM algorithms attractive.

In deriving an EM algorithm, we treat the unobserved multinomial probabilities p_j in each case as missing data. The complete data likelihood is then the integrand in the integral (5). A straightforward calculation shows that p possesses a posterior Dirichlet distribution with parameters α₁ + x_i1 through α_d + x_id for case i. If we now differentiate the identity

$$1=\frac{\Gamma (\mid \alpha \mid +\mid x_i\mid )}{\prod _{j=1}^d\Gamma ({\alpha }_j+x_{ij})}{\int }_{{\Delta }_d}\prod _{j=1}^d{p}_j^{x_{\mathit{ij}}+{\alpha }_j-1}\mathit{dp}$$

with respect to α_j, then the identity

$$\mathrm{E}(\ln p_j\mid {\alpha }_{n,}{x}_{\mathit{ij}})=\Psi (x_{\mathit{ij}}+{\alpha }_{\mathit{nj}})-\Psi (\mid x_i\mid +\mid {\alpha }_n\mid )$$

emerges, where Ψ(z) = Γ’(z)/Γ(z) is the digamma function. It follows that up to an irrelevant additive constant the surrogate function is

$$\begin{array}{cc}\hfill Q(\alpha \mid {\alpha }_n)=& \sum _{i=1}^m\sum _{j=1}^d{\alpha }_j\left[\Psi \right(x_{ij}+{\alpha }_{nj})-\Psi (\mid x_i\mid +\mid {\alpha }_n\mid \left)\right]\hfill \\ \hfill & +m\ln \Gamma (\mid \alpha \mid )-m\sum _{j=1}^d\ln \Gamma \left({\alpha }_j\right).\hfill \end{array}$$

Maximizing Q(α ∣ α_n) is non-trivial because involves it special functions and intertwining of the α_j parameters.

Directly invoking the MM principle produces a more malleable surrogate function. Consider the logarithm of the third form of the likelihood function (5). Applying Jensen’s inequality to ln(α_j + k) gives

$$\begin{array}{cc}\hfill \ln ({\alpha }_j+k)& \ge \frac{{\alpha }_{nj}}{{\alpha }_{nj}+k}\ln \left(\frac{{\alpha }_{\mathit{nj}}+k}{{\alpha }_{\mathit{nj}}}\cdot {\alpha }_j\right)+\frac{k}{{\alpha }_{\mathit{nj}}+k}\ln \left(\frac{{\alpha }_{\mathit{nj}}+k}{k}\cdot k\right)\hfill \\ \hfill & =\frac{{\alpha }_{\mathit{nj}}}{{\alpha }_{\mathit{nj}}+k}\ln {\alpha }_j+c_n.\hfill \end{array}$$

Likewise, applying the supporting hyperplane inequality to −ln(∣α∣ + k) gives

$$-\ln (\mid \alpha \mid +k)\ge -\ln (\mid {\alpha }_n\mid +k)-\frac{\mid \alpha \mid -\mid {\alpha }_n\mid }{\mid {\alpha }_n+k\mid }=\frac{\mid \alpha \mid }{\mid {\alpha }_n\mid +k}+c_n.$$

Overall, these minorizations yield the surrogate function

$$\begin{array}{cc}\hfill g(\alpha \mid {\alpha }_n)& =-\sum _k\frac{r_k}{\mid {\alpha }_n\mid +k}\mid \alpha \mid +\sum _j\sum _k\frac{s_{\mathit{jk}}{\alpha }_{\mathit{nj}}}{{\alpha }_{\mathit{nj}}+k}\ln {\alpha }_j+c_n,\hfill \\ \hfill s_{jk}& =\sum _{i=1}^m{1}_{\{x_{\mathit{ij}}\ge k+1\}},r_k=\sum _{i=1}^m{1}_{\{m_i\ge k+1\}},\hfill \end{array}$$

which completely separates the parameter α_j. This suggests the simple MM updates

$${\alpha }_{n+1,j}={\alpha }_{\mathit{nj}}\frac{\sum _k\frac{s_{\mathit{jk}}}{{\alpha }_{\mathit{nj}}+k}}{\sum _k\frac{r_k}{\mid {\alpha }_n\mid +k}},j=1,\dots ,d.$$

The positivity constraints are always satisfied when all initial values α_0j > 0. Parameter separation can be achieved in the EM algorithm by a further minorization of the lnΓ(∣α∣) term in Q(α ∣ α_n). This action yields a viable EM-MM hybrid algorithm. The reference [92] contains more details and a comparison of the convergence rates of the three algorithms.

Finally, let us mention various strategies for handling exceptional cases. In the MM algorithm it may be impossible to optimize the surrogate function g(θ ∣ θ_n) explicitly. There are two obvious remedies. One is to institute some form of block relaxation in updating g(θ ∣ θ_n) [61]. There is no need to iterate to convergence since the purpose is merely to improve g(θ ∣ θ_n) and hence the objective function f(θ). Another obvious remedy is to optimize the surrogate function by Newton’s method. It turns out that a single step of Newton’s method suffices to preserve the local rate of convergence of the MM algorithm [50]. The ascent property is sacrificed initially, but it kicks in as one approaches the optimal point. In an unconstrained problem this variant MM algorithm can be phrased as

$$\begin{array}{cc}\hfill {\theta }_{n+1}& ={\theta }_n+d^{2}g{({\theta }_n\mid {\theta }_n)}^{-1}\nabla g({\theta }_n\mid {\theta }_n)\hfill \\ \hfill & ={\theta }_n+d^{2}g{({\theta }_n\mid {\theta }_n)}^{-1}\nabla f\left({\theta }_n\right),\hfill \end{array}$$

where the substitution of ▿f(θ_n) for ▿g(θ_n ∣ θ_n) is justified by the tangency and domination conditions satisfied by g(θ ∣ θ_n) and f(θ).

A more pressing concern in the EM algorithm is intractability of the E step. If f(X ∣ θ) denotes the complete data likelihood, then in the stochastic EM algorithm [43, 75, 87] one estimates the surrogate function by a Monte Carlo average

$$Q(\theta \mid {\theta }_n)\approx \frac{1}{m}\sum _{i=1}^m\ln f(x_i\mid \theta )$$ (6)

over realizations x_i of the complete data X conditional on the observed data Y = y and the current parameter iterate θ_n. Sampling can be done by rejection sampling, importance sampling, Markov chain Monte Carlo, or quasi-Monte Carlo. The next iterate θ_n+1should maximize the average (6). The sample size mshould increase as the iteration count nincreases. Determining the rate of increase of m and setting a reasonable convergence criterion are both subtle issues. The ascent property of the EM algorithm fails because of the inherent sampling noise. The combination of slow convergence and Monte Carlo sampling makes the stochastic EM algorithm unattractive in large-scale problems. In smaller problems it fills a useful niche.

The stochastic EM algorithm generalizes the Robbins-Monro algorithm [76] for root finding and the Kiefer-Wolfowitz algorithm [46] for function maximization. In unconstrained maximum likelihood estimation, one seeks a root of the likelihood equation, so both methods are relevant. Under suitable assumptions, the Kiefer-Wolfowitz algorithm converges to a local maximum almost surely. Since this cluster of topics is tangential to our overall emphasis on deterministic methods of optimization, we refer readers to the books [13, 49, 75] for a fuller discussion.

Penalization

Penalization is a device for imposing parsimony. For purposes of illustration, we discuss two penalized estimation problems of considerable utility in applied statistics. Both of these examples generate convex programs with nondifferentiable objective functions. In the interests of accessibility, we will derive estimation algorithms for both problems without invoking the machinery of convex analysis.

Example 0.9. Lasso Penalized Regression

Lasso penalized regression has been pursued for a long time in many application areas [14, 16, 26, 80, 83, 85]. Modern versions consider a generalized linear model where y_i is the response for case i, x_ij is the value of predictor j for case i, and θ_j is the regression coefficient corresponding to predictor j. When the number of predictors p exceeds the number of cases m, θ cannot be uniquely estimated. In an era of big data, this quandary is fairly common. One remedy is to perform model selection by imposing a lasso penalty on the loss function l(θ). In least squares estimation

$$\ell \left(\theta \right)=\frac{1}{2}\sum _{i=1}^m{\left(y_i-\sum _j{x}_{\mathit{ij}}{\theta }_j\right)}^2.$$

For a generalized linear model [69], l(θ) is the negative loglikelihood of the data. Lasso penalized estimation minimizes the criterion

$$f\left(\theta \right)=\ell \left(\theta \right)+\rho \sum _j{w}_j\mid {\theta }_j\mid ,$$

where the nonnegative weights w_j and the tuning constant ρ > 0 are given. If θ_j is the intercept for the model, then its weight w_j is usually set to 0. For the remaining predictors the choice w_j = 1 is reasonable provided the predictors are standardized to have mean 0 and variance 1. To improve the asymptotic properties of the lasso estimates, the adaptive lasso [95] defines the weights $w_j={\mid {\widehat{\theta }}_j\mid }^{-1}$ for any consistent estimate ${\widehat{\theta }}_j$ of θ_j In a Bayesian context, imposing a lasso penalty is equivalent to placing a Laplace prior with mean 0 on each θ_j. The elastic net [96] adds a ridge penalty $\lambda {\sum }_j{\theta }_j^2$ to the lasso penalty.

The primary difference between lasso and ridge regression is that the lasso penalty forces most parameters to 0 while the ridge penalty merely reduces them. Thus, the ridge penalty relaxes its grip too quickly for model selection. Unfortunately, the lasso penalty tends to select one predictor from a group of correlated predictors and ignore the others. The elastic net ameliorates this defect. To overcome severe shrinkage, many statisticians discard penalties after the conclusion of model selection and re-estimate the selected parameters. Cross-validation [37] and stability selection [60] are effctive in choosing the penalty tuning constant and the selected predictors, respectively.

Coordinate descent works particularly well when only a few predictors enter a model [29, 89]. Consider what happens when we visit parameter θ_j and the loss function is the least squares criterion. If we define the amended response ${\mathcal{z}}_{\mathit{ni}}=y_i-{\sum }_{k\ne j}{x}_{\mathit{ik}}{\theta }_{\mathit{nk}}$, then the problem reduces to minimizing

$$\frac{1}{2}\sum _{i=1}^m{({\mathcal{z}}_{\mathit{ni}}-x_{\mathit{ij}}{\theta }_j)}^2+\rho w_j\mid {\theta }_j\mid .$$

Now divide the domain of θ_j into the two intervals (−∞, 0] and [0, ∞). On the right interval, elementary calculus suggests the update

$${\theta }_{n+1,j}=\frac{\sum _{i=1}^m{\mathcal{z}}_{\mathit{ni}}{x}_{\mathit{ij}}-\rho w_j}{\sum _{i=1}^m{x}_{\mathit{ij}}^2}.$$

This is invalid when it is negative and must be replaced by 0. Likewise, on the left interval, we have the update

$${\theta }_{n+1,j}=\frac{\sum _{i=1}^m{\mathcal{z}}_{\mathit{ni}}{x}_{\mathit{ij}}-\rho w_j}{\sum _{i=1}^m{x}_{\mathit{ij}}^2}$$

unless it is positive. On both intervals, shrinkage pulls the usual least squares estimate toward 0. In underdetermined problems with just a few relevant predictors, most parameters never budge from their starting values of 0. This circumstance plus the complete absence of matrix operations explains the speed of coordinate descent. It inherits its numerical stability from the descent property enjoyed by any coordinate descent algorithm.

With a generalized linear model, say logistic regression, the same story plays out. Now, however, we must institute a line search for the minimum on each of the two half intervals. Newton’s method, scoring, and even golden section search work well. When f(θ) is convex, and θ_j = 0, it is prudent to check the forward directional derivatives d_ejf(θ) and d_−ejf(θ) along the current coordinate direction e_j and its negative. If both forward directional derivatives are nonnegative, then no progress can be made by moving off 0. Thus, a parameter parked at 0 is left there. Other computational savings are possible that make coordinate descent even faster. For example, computations can be organized around the the linear predictor ∑_jx_ijθ_j for each case i. When θ_j changes, it is trivial to update this inner product. The references [88, 89] illustrate the potential of coordinate descent on some concrete genetic examples.

Example 0.10. Matrix Completion

The matrix completion problem became famous when the movie distribution company Netflix offered a million dollar prize for improvements to its movie rating system [1]. The idea was that customers would submit ratings on a small subset of movie titles, and from these ratings Netflix would infer their preferences and recommend additional movies for their consideration. Imagine therefore a very sparse matrix Y = (y_ij) whose rows are individuals and whose columns are movies. Completed cells contain a rating from 1 to 5. Most cells are empty and need to be filled in. If the matrix is sufficiently structured and possesses low rank, then it is possible to complete the matrix in a parsimonious way. Although this problem sounds specialized, it has applications far beyond this narrow setting. For example, filling in missing genotypes in genome scans for disease genes benefits from matrix completion [15].

Following the references [9, 10, 58, 12], let Δ denote the set of index pairs (i, j) such that y_ij is observed. The Lagrangian formulation of matrix completion minimizes the criterion

$$f\left(X\right)=\frac{1}{2}\sum _{(i,j)\in \Delta }{(y_{\mathit{ij}}-x_{\mathit{ij}})}^2+\rho \sum _k{\sigma }_k$$ (7)

with respect to a compatible matrix X = (x_ij) with singular values σ_k. Recall that the singular value decomposition

$$X=\sum _i{\sigma }_i{u}_i{\upsilon }_i^t$$

represents X as a sum of outer products involving a collection of orthogonal left singular vectors u_i, a corresponding collection of orthogonal right singular vectors v_i, and a descending sequence of nonnegative singular values σ_i. Alternatively, we can factor X in the form UΣV^t for orthogonal matrices U and V and a rectangular diagonal matrix Σ.

The nuclear norm ∥X∥_nuc = ∑_kσ_k plays the same role in low-rank matrix approximation that the l₁ norm ∥b∥₁ = ∑_k ∣b_k plays in sparse regression. For a more succinct representation of the criterion (7), we introduce the Frobenius norm

$${\mid \mid U\mid \mid }_F=\sqrt{\sum _i\sum _j{u}_{\mathit{ij}}^2}$$

induced by the trace inner product tr(UV^t) and the projection operator P_Δ(Y) with entries

$$P_{\Delta }\left(Y\right)=\{\begin{array}{cc}{y}_{\mathit{ij}}\hfill & (i,j)\in \Delta \hfill \\ 0\hfill & (i,j)\notin \Delta .\hfill \end{array}\phantom{\}}$$

In this notation, the criterion (7) becomes

$$\frac{1}{2}{\mid \mid P_{\Delta }\left(Y\right)-P_{\Delta }\left(X\right)\mid \mid }_F^2+\rho {\mid \mid X\mid \mid }_{\mathrm{nuc}}.$$

To derive an algorithm for estimating X, we again exploit the MM principle. The general idea is to restore the symmetry of the problem by imputing the missing data [58]. Suppose X_n is our current approximation to X. We simply replace a missing entry y_ij of Y by the corresponding entry x_nij of X_n and add the term $\frac{1}{2}{(x_{\mathit{nij}}-x_{\mathit{ij}})}^2$ to the criterion (7). Since the added terms majorize 0, they create a legitimate surrogate function and lead to an MM algorithm. One can rephrase the problem in matrix terms by defining the orthogonal complement $P_{\Delta }^{\perp }\left(Y\right)$ of P_Δ(Y) according to the rule $P_{\Delta }^{\perp }\left(Y\right)+P_{\Delta }\left(Y\right)=Y$. The matrix $Z_n=P_{\Delta }\left(Y\right)+P_{\Delta }^{\perp }\left(X_n\right)$ temporarily completes Y and yields the surrogate function

$$\begin{array}{cc}\hfill g(X\mid X_n)& =\frac{1}{2}{\mid \mid Z_n-X\mid \mid }_F^2+\rho {\mid \mid X\mid \mid }_{\mathrm{nuc}}\hfill \\ \hfill & =\frac{1}{2}{\mid \mid Z_n\mid \mid }_F^2-\mathrm{tr}\left(Z_n{X}^t\right)+\frac{1}{2}{\mid \mid X\mid \mid }_F^2+\rho {\mid \mid X\mid \mid }_{\mathrm{nuc}}.\hfill \end{array}$$

At this juncture it is helpful to recall some mathematical facts. First, the Frobenius norm is invariant under left and right multiplication of its argument by an orthogonal matrix. Thus, ${\mid \mid X\mid \mid }_F^2={\sum }_k{\sigma }_k^2$ depends only on the singular values of X. The inner product −tr(Z_nX^t) presents a greater barrier to progress, but it ultimately succumbs to a matrix analogue of the Cauchy-Schwarz inequality. Fan’s inequality [6] says that

$$\mathrm{tr}\left(Z_n{X}^t\right)\le \sum _k{\omega }_k{\sigma }_k$$

for the ordered singular values ω_k of Z_n. Equality is attained in Fan’s inequality if and only if the right and left singular vectors for the two matrices coincide. Thus, in minimizing g(X ∣ X_n) we can assume that the singular vectors of X coincide with those of Z_n and rewrite the surrogate function as

$$\begin{array}{cc}\hfill g(X\mid X_n)& =\frac{1}{2}\sum _k{\omega }_k^2-\sum _k{\omega }_k{\sigma }_k+\frac{1}{2}\sum _k{\sigma }_k^2+\rho \sum _k{\sigma }_k\hfill \\ \hfill & =\frac{1}{2}\sum _k{({\omega }_k-{\sigma }_k)}^2+\rho \sum _k{\sigma }_k.\hfill \end{array}$$

Application of the forward directional derivative test

$$d_{\nu }\left[\frac{1}{2}\sum _k{({\omega }_k-{\sigma }_k)}^2+\rho \sum _k{\sigma }_k\right]=\sum _k({\sigma }_k-{\omega }_k){\nu }_k+\rho \sum _k{\nu }_k\ge 0$$

for all tangent directions ν identifies the shrunken singular values

$${\sigma }_k=\max \{{\omega }_k-\rho ,0\}$$

as optimal. In practice, one does not have to extract the full singular value decomposition of Z_n. Only the singular values ω_k > ρ are actually relevant in constructing X_n+1

In many applications the underlying structure of the observation matrix Y is corrupted by a few noisy entries. This tempts one to approximate Y by the sum of a low rank matrix X plus a sparse matrix W. To estimate X and W, we introduce a positive tuning constant λ and minimize the criterion

$$f(X,W)=\frac{1}{2}\sum _{(i,j)\in \Delta }{(y_{\mathit{ij}}-x_{\mathit{ij}}-w_{\mathit{ij}})}^2+\rho \sum _k{\sigma }_k+\lambda \sum _i\sum _j\mid w_{\mathit{ij}}\mid$$

by block descent. We have already indicated how to update X for W fixed. To minimize f(X, W) for X fixed, we set w_ij = 0 for any pair (i, j) ∉ Δ. Because the remaining W parameters separate in f(X, W), the shrinkage updates

$$w_{n+1,i,j}=\{\begin{array}{cc}{y}_{\mathit{ij}}-x_{\mathit{nij}}-\lambda \hfill & y_{\mathit{ij}}-x_{\mathit{nij}}-\lambda >0\hfill \\ y_{\mathit{ij}}-x_{\mathit{nij}}+\lambda \hfill & y_{\mathit{ij}}-x_{nij}+\lambda <0\hfill \\ 0\hfill & \text{otherwise}\hfill \end{array}\phantom{\}}$$

are trivial to derive.

Augmented Lagrangians

The augmented Lagrangian method is one of the best ways of handling parameter constraints [38, 65, 70, 77]. For the sake of simplicity, we focus on the problem of minimizing f(θ) subject to the equality constraints g_i(θ) = 0 for i = 1, … , q. We will ignore inequality constraints and assume that f(θ) and the g_i(θ) are smooth. At a constrained minimum the classical Lagrange multiplier rule

$$0=\Delta f\left(\theta \right)+\sum _{i=1}^q{\lambda }_i\nabla g_i\left(\theta \right)$$ (8)

holds provided the gradients ▿g_i(θ) are linearly independent. The augmented Lagrangian method optimizes the perturbed function

$${\mathcal{L}}_{\rho }(\theta ,\lambda )=f\left(\theta \right)+\sum _{i=1}^q{\lambda }_i{g}_i\left(\theta \right)+\frac{\rho }{2}\sum _{i=1}^q{g}_i{\left(\theta \right)}^2$$

with respect to θ. It then adjusts the current multiplier vector λ in the hope of matching the true Lagrange multiplier vector. The penalty term $\frac{\rho }{2}{g}_i{\left(\theta \right)}^2$ punishes violations of the equality constraint g_i(θ) = 0. At convergence the gradient ρg_i(θ)▿g_i(θ) of $\frac{\rho }{2}{g}_i{\left(\theta \right)}^2$ vanishes, and we recover the standard multiplier rule (8). This process can only succeed if the degree of penalization ρ is sufficiently large.

Thus, we must either take ρ initially large or gradually increase it until it hits the finite transition point where the constrained and unconstrained solutions merge. Updating λ is more subtle. If θ_n furnishes the unconstrained minimum of ${\mathcal{L}}_{\rho }(\theta ,{\lambda }_n)$, then the stationarity condition reads

$$\begin{array}{cc}\hfill 0& =\nabla f\left({\theta }_n\right)+\sum _{i=1}^q{\lambda }_{\mathit{ni}}\nabla g_i\left({\theta }_n\right)+\rho \sum _{i=1}^q{g}_i\left({\theta }_n\right)\nabla g_i\left({\theta }_n\right).\hfill \\ \hfill & =\nabla f\left({\theta }_n\right)+\sum _{i=1}^q[{\lambda }_{\mathit{ni}}+\rho g_i({\theta }_n\left)\right]\nabla g_i\left({\theta }_n\right).\hfill \end{array}$$

The last equation motivates the standard update

$${\lambda }_{n+1,i}={\lambda }_{\mathit{ni}}+\rho g_i\left({\theta }_n\right).$$

The alternating direction method of multipliers (ADMM) [30, 33] minimizes the sum f(θ) + h(γ) subject to the affine constraints Aθ + Bγ = c. Although the objective function is separable in the block variables θ and γ, the affine constraints frustrate a direct attack. However, the problem is ripe for a combination of the augmented Lagrangian method and a single round of block descent per iteration. The augmented Lagrangian is

$${\mathcal{L}}_{\rho }(\theta ,\gamma ,\lambda )=f\left(\theta \right)+h\left(\gamma \right)+{\lambda }^t(A\theta +B\gamma -c)+\frac{\rho }{2}{\mid \mid A\theta +B\gamma -c\mid \mid }^2.$$

Minimization is performed over θ and γ by block descent before updating the multiplier vector λ via

$${\lambda }_{n+1}={\lambda }_n+\rho (A{\theta }_{n+1}+B{\gamma }_{n+1}-c).$$

Introduction of block descent simplifies the usual augmented Lagrangian method, which minimizes ${\mathcal{L}}_{\rho }(\theta ,\gamma ,\lambda )$ jointly over θ and γ. This modest change keeps the convergence theory intact [7, 28] and has led to a resurgence in the popularity of ADMM in machine learning [4, 7, 12, 71, 74, 91].

Example 0.11. Fused Lasso

ADMM is helpful in reducing difficult optimization problems to simpler ones. The easiest fused lasso problem [86] minimizes the criterion

$$\frac{1}{2}{\mid \mid y-\theta \mid \mid }_2^2+\mu \sum _i\mid {\theta }_{i+1}-{\theta }_i\mid .$$

The l₁ penalty on the increments θ_i+1 −θ_i favors piecewise constant solutions. Unfortunately, this twist on the standard lasso penalty renders coordinate descent inefficient. We can reformulate the problem as minimizing the criterion $\frac{1}{2}{\mid \mid y-\theta \mid \mid }^2+\mu {\mid \mid \gamma \mid \mid }_1$ subject to the constraint γ = Dθ, where

$$d_{\mathit{ij}}=\{\begin{array}{cc}1\hfill & j=i+1\hfill \\ -1\hfill & j=i\hfill \\ 0\hfill & \text{otherwise}.\hfill \end{array}\phantom{\}}$$

In the augmented Lagrangian framework, updating θ amounts to minimizing $\frac{1}{2}{\mid \mid y-\theta \mid \mid }^2+\frac{\rho }{2}{\mid \mid \gamma -\frac{1}{\rho }\lambda -D\theta \mid \mid }^2$. It is straightforward to solve this least squares problem. Updating γ involves minimizing $\frac{\rho }{2}{\mid \mid D\theta -\gamma \mid \mid }^2+\mu {\mid \mid \gamma \mid \mid }_1$, which is a standard lasso problem. Thus, ADMM decouples the problematic linear transformation Dθ from the lasso penalty.

Algorithm Acceleration

Many MM and block descent algorithms converge very slowly. In partial compensation, the computational work per iteration may be light. Even so, diminishing the number of iterations until convergence by one or two orders of magnitude is an attractive proposition [3, 42, 48, 51, 78, 93]. In this section we discuss a generic method for accelerating a wide variety of algorithms [93]. Consider a differentiable algorithm map θ_n+1 = A(θ_n) for optimizing an objective function f(θ), and suppose stationary points of f(θ) correspond to fixed points of A(θ). Equivalently, stationary points correspond to roots of the equation B(θ) = θ − A(θ) = 0. Within this framework it is natural to apply Newton’s method

$$\begin{array}{cc}\hfill {\theta }_{n+1}& ={\theta }_n-\mathit{dB}{\left({\theta }_n\right)}^{-1}B\left({\theta }_n\right)\hfill \\ \hfill & ={\theta }_n-{[I-dA({\theta }_n\left)\right]}^{-1}B\left({\theta }_n\right)\hfill \end{array}$$ (9)

to find the root and accelerate the overall process. This is a realistic expectation because Newton’s method converges at a quadratic rate in contrast to the linear rates of MM and block descent algorithms.

There are two principal impediments to implementing algorithm (9) in high dimensions. First, it appears to require evaluation and storage of the Jacobi matrix dA(θ), whose rows are the differentials of the components of A(θ). Second, it also appears to require inversion of the matrix I − dA(θ). Both problems can be attacked by secant approximations. Close to the optimal point θ_∞, the linear approximation

$$AoA\left({\theta }_n\right)-A\left({\theta }_n\right)\approx \mathit{dA}\left({\theta }_{\infty }\right)\left[A\right({\theta }_n)-{\theta }_n]$$

is valid. This suggests that we take two ordinary steps and gather information in the process on the matrix M = A(θ_∞). If we let v be the vector A ○ A(θ_n) − A(θ_n) and u be the vector A(θ_n) − θ_n, then the secant condition reads Mu = v. In practice it is advisable to exploit multiple secant conditions Mu_i = v_i as long as their number does not exceed the number of parameters p. The secant conditions can be generated one per iteration over the current and previous q − 1 iterations. Let us represent the conditions collectively in the matrix form MU = V for U = (u₁, … , u_q), and V = (v₁, … , v_q).

The principle of parsimony suggests that we replace M by the smallest matrix satisfying the secant conditions. If we pose this problem concretely as minimizing the criterion $\frac{1}{2}{\mid \mid M\mid \mid }_F^2$ subject to the constraints MU = V, then a straightforward exercise in Lagrange multipliers [52] gives the solution M = V(U^tU)⁻¹U^t. The matrix M has rank at most q, and the Sherman Morrison formula yields that explicit inverse

$${[I-V{\left(U^{t}U\right)}^{-1}{U}^t]}^{-1}=I+V{[U^{t}U-U^{t}V]}^{-1}{U}^t.$$

Fortunately, it involves inverting just the q × q matrix U^tU − U^tV. Furthermore, the Newton update (9) boils down to

$$\begin{array}{cc}\hfill {\theta }_{n+1}& ={\theta }_n-{[I-V{\left(U^{t}U\right)}^{-1}{U}^t]}^{-1}[{\theta }_n-A({\theta }_n\left)\right]\hfill \\ \hfill & =A\left({\theta }_n\right)-V{(U^{t}U-U^{t}V)}^{-1}{U}^t[{\theta }_n-A({\theta }_n\left)\right].\hfill \end{array}$$

The advantages of this procedure include: (a) it avoids large matrix inverses, (b) it relies on matrix times vector multiplication rather than matrix times matrix multiplication, (c) it requires only storage of the small matrices U and V, and (d) it respects linear parameter constraints. Nonnegativity constraints may be violated. The number of secants q should be fixed in advance, say between 1 and 15, and the matrices U and V should be updated by substituting the latest secant pair generated for the earliest secant pair retained. If an accelerated step fails the descent test, than one can revert to the ordinary MM or block descent step.

Acceleration of non-smooth algorithms is more problematic [40]

For gradient descent and its generalizations [17] to non-smooth problems, Nesterov [64] has suggested a potent acceleration. As noted by Beck and Teboulle [2], the accelerated iterates in ordinary gradient descent depend on an intermediate scalar t_n and an intermediate vector φ according to the formulas

$$\begin{array}{cc}\hfill t_{n+1}& =\frac{1+\sqrt{1+4t_n^2}}{2}\hfill \\ \hfill \phi & ={\theta }_n+\frac{t_n-1}{t_{n+1}}({\theta }_n-{\theta }_{n-1})\hfill \\ \hfill {\theta }_{n+1}& =\phi -\frac{1}{L}\nabla f\left(\phi \right)\hfill \end{array}$$

with initial values t₁ = 1 and φ = θ₀. In other words, instead of taking a steepest descent step from the current iterate, one takes a steepest descent step from the extrapolated point φ, which depends on both the current iterate θ_n and the previous iterate θ_n−1. This mysterious extrapolation algorithm can yield impressive speed ups for essentially the same computational cost as gradient descent.

Discussion

The fault lines in optimization separate smooth from non-smooth problems, unconstrained from constrained problems, and small-scale problems from large-scale problems. Smooth, unconstrained, small-scale problems are easy to solve. Mathematical scientists are beginning to tackle non-smooth, constrained, large-scale problems at the opposite end of the difficulty spectrum. The most spectacular successes usually rely on convexity. We can expect further progress because some of the best minds in applied mathematics, computer science, and statistics have taken up the challenge. What is unlikely to occur is the discovery of a universally valid algorithm. Optimization is apt to remain as much art as science for a long time to come.

We have emphasized a few key ideas in this survey. Our examples demonstrate some of the possibilities for mixing and matching the different algorithm themes. Although we cannot predict the future of computational statistics with any certainty, the key ideas mentioned here will not disappear. For instance, penalization is here to stay, the descent property of an algorithm is always desirable, and quadratic approximation will always be superior to linear approximation for smooth functions. As computing devices hit physical constraints, the importance of parallel algorithms will also likely increase. This argues that block descent and parameter separated MM algorithms will played a larger role in the future [94]. Although we have de-emphasized convex calculus, readers who want to devise their own algorithms are well advised to learn this inherently subtle subject. There is a difference, after all, between principled algorithms and ad hoc procedures.