Path Following in the Exact Penalty Method of Convex Programming

Abstract

Classical penalty methods solve a sequence of unconstrained problems that put greater and greater stress on meeting the constraints. In the limit as the penalty constant tends to ∞, one recovers the constrained solution. In the exact penalty method, squared penalties are replaced by absolute value penalties, and the solution is recovered for a finite value of the penalty constant. In practice, the kinks in the penalty and the unknown magnitude of the penalty constant prevent wide application of the exact penalty method in nonlinear programming. In this article, we examine a strategy of path following consistent with the exact penalty method. Instead of performing optimization at a single penalty constant, we trace the solution as a continuous function of the penalty constant. Thus, path following starts at the unconstrained solution and follows the solution path as the penalty constant increases. In the process, the solution path hits, slides along, and exits from the various constraints. For quadratic programming, the solution path is piecewise linear and takes large jumps from constraint to constraint. For a general convex program, the solution path is piecewise smooth, and path following operates by numerically solving an ordinary differential equation segment by segment. Our diverse applications to a) projection onto a convex set, b) nonnegative least squares, c) quadratically constrained quadratic programming, d) geometric programming, and e) semidefinite programming illustrate the mechanics and potential of path following. The final detour to image denoising demonstrates the relevance of path following to regularized estimation in inverse problems. In regularized estimation, one follows the solution path as the penalty constant decreases from a large value.

1 Introduction

Penalties and barriers are both potent devices for solving constrained optimization problems [1, 2, 3, 4, 5, 6]. The general idea is to replace hard constraints by penalties or barriers and then exploit the well-oiled machinery for solving unconstrained problems. Penalty methods operate on the exterior of the feasible region and barrier methods on the interior. The strength of a penalty or barrier is determined by a tuning constant. In classical penalty methods, a single global tuning constant is gradually sent to ∞; in barrier methods, it is gradually sent to 0. Either strategy generates a sequence of solutions that converges in practice to the solution of the original constrained optimization problem.

Barrier methods are now generally conceded to offer a better approach to solving convex programs than penalty methods. Application of log barriers and carefully controlled versions of Newton’s method make it possible to follow the central path reliably and quickly to the constrained minimum [1]. Nonetheless, penalty methods should not be ruled out. Augmented Lagrangian methods [7] and exact penalty methods [4] are potentially competitive with interior point methods for smooth convex programming problems. Both methods have the advantage that the solution of the constrained problem kicks in for a finite value of the penalty constant. This avoids problems of ill conditioning as the penalty constant tends to ∞.

The disadvantage of exact penalties over traditional quadratic penalties is lack of differentiability of the penalized objective function. In the current paper, we argue that this impediment can be finessed by path following. Our path following method starts at the unconstrained solution and follows the solution path as the penalty constant increases. In the process, the solution path hits, exits, and slides along the various constraint boundaries. The path itself is piecewise smooth with kinks at the boundary hitting and escape times. One advances along the path by numerically solving a differential equation for the Lagrange multipliers of the penalized problem. In the special case of quadratic programming with affine constraints, the solution path is piecewise linear, and one can easily anticipate entire path segments [8]. This special case is intimately related to the linear complementarity problem [9] in optimization theory.

Homotopy (continuation) methods for the solution of nonlinear equations and optimization problems have been pursued for many years and enjoyed a variety of successes [4, 10, 11, 12]. To our knowledge, however, there has been no exploration of path following as an implementation of the exact penalty method. Our modest goal here is to assess the feasibility and versatility of exact path following for constrained optimization. Comparing its performance to existing methods, particularly the interior point method, is probably best left for later, more practically oriented papers. In our experience, coding the algorithm is straightforward in Matlab. The rich numerical resources of Matlab include differential equation solvers that alert the user when certain events such as constraint hitting and escape occur.

The rest of the paper is organized as follows. Section 2 briefly reviews the exact penalty method for optimization and investigates sufficient conditions for uniqueness and continuity of the solution path. Section 3 derives the path following strategy for general convex programs, with particular attention to the special cases of quadratic programming and convex optimization with affine constraints. Section 4 presents various applications of the path algorithm. Our most elaborate example demonstrates the relevance of path following to regularized estimation. The particular problem treated, image denoising, is typical of many inverse problems in applied mathematics and statistics [13]. In such problems one follows the solution path as the penalty constant decreases. Finally, Section 5 discusses the limitations of the path algorithm and hints at future generalizations.

2 Exact Penalty Methods

In this paper we consider the convex programming problem of minimizing the convex objective function f (x) subject to r affine equality constraints g_i (x) = 0 and s convex inequality constraints h_j (x) ≤ 0. We will further assume that f (x) and the h_j (x) are twice differentiable. The differential df (x) is the row vector of partial derivatives of f (x); the gradient ∇ f (x) is the transpose of df (x). The second differential d² f (x) is the Hessian matrix of second partial derivatives of f (x). Similar conventions hold for the differentials of the constraint functions.

Exact penalty methods [4, 5] minimize the surrogate function

$${ℰ}_{\rho }(\mathit{x})=f(\mathit{x})+\rho {\displaystyle \sum _{i=1}^r|g_i(\mathit{x})|+}\rho {\displaystyle \sum _{j=1}^s\max \{0,h_j(\mathit{x})\}}.$$ (1)

This definition of ℰ_ρ (x) is meaningful regardless of whether the contributing functions are convex. If the program is convex, then ℰ_ρ (x) is itself convex. It is interesting to compare ℰ_ρ (x) to the Lagrangian function

$$ℒ(\mathit{x})=f(\mathit{x})+{\displaystyle \sum _{i=1}^r{\lambda }_i{g}_i(\mathit{x})+}{\displaystyle \sum _{j=1}^s{\mu }_j{h}_j(\mathit{x})},$$

which captures the behavior of f (x) near the optimum. At a constrained local minimum y, the Lagrangian satisfies the stationarity condition ∇ℒ (y) = 0; its inequality multipliers μ_j are nonnegative and satisfy the complementary slackness conditions μ_j h_j (y) = 0. In an exact penalty method one takes

$$\rho >\max \{|{\lambda }_1|,\dots ,|{\lambda }_r|,{\mu }_1,\dots ,{\mu }_s\}.$$ (2)

This choice creates the favorable circumstances

$$\begin{array}{l}ℒ(\mathit{x})\le {ℰ}_{\rho }(\mathit{x})\text{for all}\mathit{x}\\ ℒ(\mathit{z})\le f(\mathit{z})={ℰ}_{\rho }(\mathit{z})\text{for all feasible}\mathit{z}\\ ℒ(\mathit{y})=f(\mathit{y})={ℰ}_{\rho }(\mathit{y})\text{for}\mathit{y}\text{optimal}\end{array}$$

with profound consequences. As the next proposition proves, minimizing ℰ_ρ (x) is effective in minimizing f (x) subject to the constraints.

Proposition 1

Suppose the objective function f (x) and the constraint functions are twice differentiable and satisfy the Lagrange multiplier rule at the local minimum y. If inequality (2) holds and υ *d² ℒ (y) v > 0 for every vector v ≠ 0 satisfying dg_i (y) v = 0 and dh_j (y) v ≤ 0 for all active inequality constraints, then y furnishes an unconstrained local minimum of ℰ_ρ(x). For a convex program satisfying Slater’s constraint qualification and inequality (2), y is a minimum of ℰ_ρ(x) if and only if y is a minimum of f (x) subject to the constraints. No differentiability assumptions are required for convex programs.

Proof The conditions imposed on the quadratic form v*d² ℒ (y) v are well-known sufficient conditions for a local minimum. Theorems 6.9 and 7.21 of the reference [5] prove all of the foregoing assertions.

As previously stressed, the exact penalty method turns a constrained optimization problem into an unconstrained minimization problem. Furthermore, in contrast to the quadratic penalty method [4, Section 17.1], the constrained solution in the exact method is achieved for a finite value of ρ. Despite these advantages, minimizing the surrogate function ℰ_ρ (x) is complicated. For one thing, it is no longer globally differentiable. For another, one must minimize ℰ_ρ (x) along an increasing sequence ρ_n because the Lagrange multipliers (2) are usually unknown in advance. These hurdles have prevented wide application of exact penalty methods in convex programming.

As a prelude to our derivation of the path following algorithm for convex programs, we record several properties of ℰ_ρ (x) that mitigate the failure of differentiability.

Proposition 2

The surrogate function ℰ_ρ (x) is increasing in ρ. Furthermore, ℰ_ρ (x) is strictly convex for one ρ > 0 if and only if it is strictly convex for all ρ > 0. Likewise, it is coercive for one ρ > 0 if and only if is coercive for all ρ > 0. Finally, if f (x) is strictly convex (or coercive), then all ℰ_ρ (x) are strictly convex (or coercive).

Proof The first assertion is obvious. For the second assertion, consider more generally a finite family u₁ (x), …, u_q (x) of convex functions, and suppose a linear combination ${\displaystyle {\sum }_{k=1}^q{c}_k{u}_k(\mathit{x})}$ with positive coefficients is strictly convex. It suffices to prove that any other linear combination ${\displaystyle {\sum }_{k=1}^q{b}_k{u}_k(\mathit{x})}$ with positive coefficients is strictly convex. For any two points x ≠ y and any (0 scalar α ∈ (0, 1), we have

$$u_k[\alpha \mathit{x}+(1-\alpha )\mathit{y}]\le \alpha u_k(\mathit{x})+(1-\alpha )u_k(\mathit{y}).$$ (3)

Since ${\displaystyle {\sum }_{k-1}^q{c}_k{u}_k(\mathit{x})}$ is strictly convex, strict inequality must hold for at least one k. Hence, multiplying inequality (3) by b_k and adding gives

$${\displaystyle \sum _{k=1}^q{b}_k{u}_k\left[\alpha \mathit{x}+(1-\alpha )\mathit{y}\right]}<\alpha {\displaystyle \sum _{k=1}^q{b}_k{u}_k(\mathit{x})+(1-\alpha ){\displaystyle \sum _{k=1}^q{b}_k{u}_k(\mathit{y})}}.$$

The third assertion follows from the fact that a convex function is coercive if and only if its restriction to each half-line is coercive [14, Proposition 3.2.2]. Given this result, suppose ℰ_ρ (x) is coercive, but ${ℰ}_{{\rho }^{\ast }}(\mathit{x})$ is not coercive. Then there exists a point x, a direction v, and a sequence of scalars t_n tending to ∞ such that ${ℰ}_{{\rho }^{\ast }}(\mathit{x}+t_n\mathit{v})$ is bounded above. This requires the sequence f (x + t_n v) and each of the sequences |g_i (x + t_n v)| and max {0, h_j (x + t_n v) to remain bounded above. But in this circumstance the sequence ℰ_ρ (x + t_n v) also remains bounded above. The final two assertions are obvious.

3 The Path Following Algorithm

In this section, we take a different point of view. Instead of minimizing ℰ_ρ (x) for an increasing sequence ρ_n, we study how the solution x (ρ) changes continuously with ρ and devise a path following strategy starting from ρ = 0. For some finite value of ρ, the path locks in on the solution of the original convex program. In regularized statistical estimation and inverse problems, the primary goal is to select relevant predictors rather than to find a constrained solution. Thus, the entire solution path commands more interest than any single point along it [15, 16, 17, 8, 13]. Although our theory will focus on constrained estimation, readers should bear in mind this second application area of path following.

The path algorithm relies critically on the first order optimality condition that characterizes the optimum point of the convex function ℰ_ρ (y).

Proposition 3

For a convex program, a point x = x (ρ) minimizes the function ℰ_ρ (y) if and only if x satisfies the stationarity condition

$$0=\nabla f(\mathit{x})+\rho {\displaystyle \sum _{i=1}^r{s}_i\nabla g_i(\mathit{x})}+\rho {\displaystyle \sum _{j=1}^s{t}_j\nabla h_j(\mathit{x})}$$ (4)

for coefficient sets ${\{s_i\}}_{i=1}^r$ and ${\{t_j\}}_{j=1}^s$. These sets can be characterized as

$$s_i\in \{\begin{array}{ll}\{-1\}& g_i(\mathit{x})<0\\ [-1,1]& g_i(\mathit{x})=0\\ \{1\}& g_i(\mathit{x})>0\end{array}\mathit{and}{t}_i\in \{\begin{array}{ll}\{0\}& h_j(\mathit{x})<0\\ [0,1]& h_j(\mathit{x})=0\\ \{1\}& h_j(\mathit{x})>0\end{array}.$$ (5)

At most one point achieves the minimum of ℰ_ρ (y) for a given ρ when ρ (y) is strictly convex.

Proof According to Fermat’s rule, x minimizes ℰ_ρ (y) if and only if 0 belongs to the subdifferential ∂ℰ_ρ (x) of ℰ_ρ (y). To derive the subdifferential displayed in equations (4) and (5), one applies the addition and chain rules of the convex calculus. The sets defining the possible values of s_i and t_j are the subdifferentials of the functions |s| and t₊ = max {t, 0}, respectively. For more details see Theorem 3.5 and ancillary material in the book [5]. Finally, it is well known that strict convexity guarantees a unique minimum.

To speak coherently of solution paths, one must validate the existence, uniqueness, and continuity of the solution x (ρ) to the system of equations (1). Uniqueness follows from strict convexity as already noted. Existence and continuity are more subtle.

Proposition 4

If ℰ_ρ (y) is strictly convex and coercive, then the solution path x (ρ) of equation (1) exists and is continuous in ρ. If the gradient vectors {∇ g_i (x): g_i (x) = 0} ∪ {∇ h_j (x): h_j (x) = 0} of the active constraints are linearly independent at x (ρ) for ρ > 0, then the coefficients s_i (ρ) and t_j (ρ) are unique and continuous near ρ as well.

Proof The special case of a least squares objective with affine constraints appears in our previous paper [8]. The following proof applies to general convex programming. In accord with Proposition 2, we assume that either f (x) is strictly convex and coercive or restrict our attention to the open interval (0, ∞). Consider a subinterval [a, b] and fix a point x in the common domain of the functions ℰ_ρ (y). The coercivity of ℰ_a (y) and the inequalities

$${ℰ}_a\left[\mathit{x}(\rho )\right]\le {ℰ}_{\rho }\left[\mathit{x}(\rho )\right]\le {ℰ}_{\rho }(\mathit{x})\le {ℰ}_b(\mathit{x})$$

demonstrate that the solution vector x (ρ) is bounded over [a, b]. To prove continuity, suppose that it fails for a given ρ ∈ [a, b]. Then there exists an ε > 0 and a sequence ρ_n tending to ρ such ‖x (ρ_n) − x (ρ_n)‖₂ ≥ ε for all n. Since x (ρ_n) is bounded, we can pass to a subsequence if necessary and assume that x (ρ_n) converges to some point y. Taking limits in the inequality ℰ_ρn [x (ρ_n)] ≤ ℰ_ρn (x) demonstrates that ℰ_ρ (y) ≤ ℰ_ρ (x) for all x. Because x (ρ) is unique, we reach the contradictory conclusions ‖y − x (ρ)‖₂ ≥ ε and y = x (ρ)

Verification of the second claim is deferred to permit further discussion of path following. The claim says that an active constraint (g_i (x) = 0 or h_j (x) = 0) remains active until its coefficient hits an endpoint of its subdifferential. Because the solution path is, in fact, piecewise smooth, one can follow the coefficient path by numerically solving an ordinary differential equation (ODE).

Our path following algorithm works segment-by-segment. Along the path we keep track of the following index sets

$$\begin{array}{c}{\mathcal{N}}_{\mathrm{E}}=\{i:g_i(\mathit{x})<0\}\\ {\mathcal{Z}}_{\mathrm{E}}=\{i:g_i(\mathit{x})=0\}\\ {\mathcal{P}}_{\mathrm{E}}=\{i:g_i(\mathit{x})>0\}\end{array}\begin{array}{c}{\mathcal{N}}_{\mathrm{I}}=\{j:h_j(\mathit{x})<0\}\\ {\mathcal{Z}}_{\mathrm{I}}=\{j:h_j(\mathit{x})=0\}\\ {\mathcal{P}}_{\mathrm{I}}=\{j:h_j(\mathit{x})>0\}\end{array}$$ (6)

determined by the signs of the constraint functions. For the sake of simplicity, assume that at the beginning of the current segment s_i does not equal −1 or 1 when i ∈ $Z_{\mathrm{E}}$. and t_j does not equal 0 or 1 when $Z_{\mathrm{I}}$. In other words, the coefficients of the active constraints occur on the interior of their subdifferentials. Let us show in this circumstance that the solution path can be extended in a smooth fashion. Our plan of attack is to reparameterize by the Lagrange multipliers for the active constraints. Thus, set λ_i = ρs_i for i ∈ $Z_{\mathrm{E}}$ and ω_j = ρt_j for j ∈ $Z_{\mathrm{I}}$. The multipliers satisfy −ρ < λ_i < ρ and 0 < ω_j < ρ.

The stationarity condition now reads

$$\begin{array}{c}0=\nabla f(\mathit{x})-\rho {\displaystyle \sum _{i\in {\mathcal{N}}_{\mathrm{E}}}\nabla g_i(\mathit{x})}+\rho {\displaystyle \sum _{i\in {\mathcal{P}}_{\mathrm{E}}}\nabla g_i(\mathit{x})}+\rho {\displaystyle \sum _{j\in {\mathcal{P}}_{\mathrm{I}}}\nabla h_j(\mathit{x})}\\ +{\displaystyle \sum _{i\in {\mathcal{Z}}_{\mathrm{E}}}{\mathit{\lambda }}_i\nabla g_i(\mathit{x})}+{\displaystyle \sum _{j\in {\mathcal{Z}}_{\mathrm{I}}}{\mathit{\omega }}_j\nabla h_j(\mathit{x})}.\end{array}$$

To this we concatenate the constraint equations 0 = g_i (x) for i ∈ $Z_{\mathrm{E}}$ and 0 = h_j (x) for j ∈ $Z_{\mathrm{I}}$

For convenience now define

$${\mathit{U}}_{\mathcal{Z}}(\mathit{x})=\left[\begin{array}{c}d{h}_{{\mathcal{Z}}_{\mathrm{I}}}(\mathit{x})\\ d{g}_{{\mathcal{Z}}_{\mathrm{E}}}(\mathit{x})\end{array}\right],u_{\overline{\mathcal{Z}}}(\mathit{x})=-{\displaystyle \sum _{i\in {\mathcal{N}}_{\mathrm{E}}}\nabla g_i(\mathit{x})}+{\displaystyle \sum _{i\in {\mathcal{P}}_{\mathrm{E}}}\nabla g_i(\mathit{x})}+{\displaystyle \sum _{j\in {\mathcal{P}}_{\mathrm{I}}}\nabla h_j(\mathit{x})}.$$

In this notation the stationarity equation can be recast as

$$0=\nabla f(\mathit{x})+\rho u_{\overline{\mathcal{Z}}}(\mathit{x})+{\mathit{U}}_{\mathcal{Z}}^t(\mathit{x})\left[\begin{array}{c}\mathit{\lambda }\\ \mathit{\omega }\end{array}\right].$$

Under the assumption that the matrix ${\mathit{U}}_Z(\mathit{x})$ has full row rank, one can solve for the Lagrange multipliers in the form

$$\left[\begin{array}{c}{\mathit{\lambda }}_{{\mathcal{Z}}_E}\\ {\mathit{\omega }}_{{\mathcal{Z}}_{\mathit{I}}}\end{array}\right]=-{[{\mathit{U}}_{\mathcal{Z}}(\mathit{x}){\mathit{U}}_{\mathcal{Z}}^t(\mathit{x})]}^{-1}{\mathit{U}}_{\mathcal{Z}}(\mathit{x})[\nabla f(\mathit{x})+\rho u_{\overline{\mathcal{Z}}}(\mathit{x})].$$ (7)

Hence, the multipliers are unique. Continuity of the multipliers is a consequence of the continuity of the solution vector x (ρ) and all functions in sight on the right-hand side of equation (7). This observation completes the proof of Proposition 4.

Collectively the stationarity and active constraint equations can be written as the vector equation 0 = k (x, λ, ω, ρ). To solve for x, λ and ω in terms of ρ, we apply the implicit function theorem [18, 19]. This requires calculating the differential of k (x, λ, ω, ρ) with respect to the underlying dependent variables x, λ, and ω and the independent variable ρ. Because the equality constraints are affine, a brief calculation gives

$$\begin{array}{l}{\partial }_{\mathit{x},\mathit{\lambda },\mathit{\omega }}k(\mathit{x},\mathit{\lambda },\mathit{\omega },\rho )=\left[\underset{{\mathit{U}}_{\mathcal{Z}}(\mathit{x})}{d^{2}f(\mathit{x})+\rho {\displaystyle {\sum }_{j\in {\mathcal{P}}_{\mathrm{I}}}{d}^2{h}_j(\mathit{x})+{\displaystyle {\sum }_{j\in {\mathcal{Z}}_{\mathrm{I}}}{\mathit{\omega }}_j{d}^2{h}_j(\mathit{x})}}}\underset{0}{{\mathit{U}}_{\mathcal{Z}}^t(\mathit{x})}\right]\\ {\partial }_{\rho }k(\mathit{x},\mathit{\lambda },\mathit{\omega },\rho )=\left(\underset{0}{u_{\overline{\mathcal{Z}}}(\mathit{x})}\right).\end{array}$$

The matrix ∂_x,λ,ω k (x, λ, ω, ρ) is nonsingular when its upper-left block is positive definite and its lower-left block has full row rank [20, Proposition 11.3.2]. Given that it is nonsingular, the implicit function theorem applies, and we can in principle solve for x, λ and ω in terms of ρ. More importantly, the implicit function theorem supplies the derivative

$$\frac{d}{d\rho }\left[\begin{array}{c}\mathit{x}\\ {\mathit{\lambda }}_{{\mathcal{Z}}_{\mathrm{E}}}\\ {\mathit{\omega }}_{{\mathcal{Z}}_{\mathrm{I}}}\end{array}\right]=-{\partial }_{\mathit{x},\mathit{\lambda },\mathit{\omega }}k{(\mathit{x},\mathit{\lambda },\mathit{\omega },\rho )}^{-1}{\partial }_{\rho }k(\mathit{x},\mathit{\lambda },\mathit{\omega },\rho ),$$ (8)

which is the key to path following. We summarize our findings in the next proposition. A similar proof appears in our application paper [21], where we implement path following in a convex clustering context.

Proposition 5

Suppose the surrogate function ℰ_ρ (y) is strictly convex and coercive. If at the point x (ρ₀) the matrix ∂_x,λ,ω k (x, λ, ω, ρ) is nonsingular and the coefficient of each active constraints occurs on the interior of its subdifferential, then the solution path x (ρ) and Lagrange multipliers λ (ρ) and ω (ρ) satisfy the differential equation (8) in the vicinity of x (ρ₀).

In practice one traces the solution path along the current time segment until either an inactive constraint becomes active or the coefficient of an active constraint hits the boundary of its subdifferential. The earliest hitting time or escape time over all constraints determines the duration of the current segment. When the hitting time for an inactive constraint occurs first, we move the constraint to the appropriate active set ${\mathcal{Z}}_{\mathrm{E}}$ or ${\mathcal{Z}}_{\mathrm{I}}$ and keep the other constraints in place. Similarly, when the escape time for an active constraint occurs first, we move the constraint to the appropriate inactive set and keep the other constraints in place. In the second scenario, if s_i hits the value −1, then we move i to ${\mathcal{N}}_{\mathrm{E}}$; If s_i hits the value 1, then we move i to ${\mathcal{P}}_{\mathrm{E}}$. Similar comments apply when a coefficient t_j hits 0 or 1. Once this move is executed, we commence path following along the new segment. Path following continues until for sufficiently large ρ, the sets ${\mathcal{N}}_{\mathrm{E}}$, ${\mathcal{P}}_{\mathrm{E}}$, and the ${\mathcal{P}}_{\mathrm{I}}$ are exhausted, $u_{\overline{\mathcal{Z}}}=0$, and solution vector x (ρ) stabilizes.

Algorithm 1 summarizes the above path following strategy. It makes the convenient assumptions that there are no active constraints ${\mathcal{Z}}_{\mathrm{E}}={\mathcal{Z}}_{\mathrm{I}}=\varnothing$ at the unconstrained solution x (0) and that the hitting and/or escape times do not occur simultaneously for any ρ > 0. Our previous paper [8] suggests remedies in the very rare situations where these assumptions fail.

Algorithm 1: Exact path algorithm for convex programming.

Path following simplifies considerably in two special cases. Consider convex quadratic programming with objective function $f(\mathit{x})=\frac{1}{2}{\mathit{x}}^t\mathit{A}\mathit{x}+b^t\mathit{x}$ and equality constraints V x = d and inequality constraints W x ≤ e, where A is positive semi-definite. The exact penalized objective function becomes

$${ℰ}_{\rho }(\mathit{x})=\frac{1}{2}{\mathit{x}}^t\mathit{A}\mathit{x}+b^t\mathit{x}+\rho {\displaystyle \sum _{i=1}^s|{\mathit{v}}_i^t\mathit{x}-d_i|+\rho {\displaystyle \sum _{j=1}^t({\mathit{w}}_j^t\mathit{x}-e_j)}+.}$$

Since both the equality and inequality constraints are affine, their second derivatives vanish. Both ${\mathit{U}}_{\mathcal{Z}}$ and $u_{\overline{\mathcal{Z}}}$ are constant on the current path segment, and the path x (ρ) satisfies

$$\frac{d}{d\rho }\left[\begin{array}{c}\mathit{x}\\ {\mathit{\lambda }}_{{\mathcal{Z}}_{\mathrm{E}}}\\ {\mathit{\omega }}_{{\mathcal{Z}}_{\mathrm{I}}}\end{array}\right]=-{\left(\begin{array}{c}\mathit{A}\\ {\mathit{U}}_{\mathcal{Z}}\end{array}\begin{array}{c}{\mathit{U}}_{\mathcal{Z}}^t\\ 0\end{array}\right)}^{-1}\left(\begin{array}{c}{u}_{\overline{\mathcal{Z}}}\\ 0\end{array}\right).$$ (9)

This implies that the solution path x (ρ) is piecewise linear [8]. When V is the identity matrix, d = 0, and there are no inequality constraints, Algorithm 1 reduces to the well-known path algorithm for lasso regression [15, 16]. Tibshirani and Taylor [17] construct a piecewise-linear path algorithm for the generalized lasso problem with a general V and d = 0. Their approach seems to circumvent the need for linear independence of the active constraints.

On the next rung on the ladder of generality are convex programs with affine constraints. For the exact surrogate

$${ℰ}_{\rho }(\mathit{x})=f(\mathit{x})+\rho {\displaystyle \sum _{i=1}^s|{\mathit{v}}_i^t\mathit{x}-d_i|+\rho {\displaystyle \sum _{j=i}^t({\mathit{w}}_j^t\mathit{x}-e_j)}}+,$$

the matrix ${\mathit{U}}_{\mathcal{Z}}$ and vector $u_{\overline{\mathcal{Z}}}$ are still constant along a path segment. The relevant differential equation becomes

$$\frac{d}{d\rho }\left[\begin{array}{c}\mathit{x}\\ {\mathit{\lambda }}_{{\mathcal{Z}}_{\mathrm{E}}}\\ {\mathit{\omega }}_{{\mathcal{Z}}_{\mathrm{I}}}\end{array}\right]=-{\left(\begin{array}{c}{d}^{2}f(\mathit{x})\\ {\mathit{U}}_{\mathcal{Z}}\end{array}\begin{array}{c}{\mathit{U}}_{\mathcal{Z}}^t\\ 0\end{array}\right)}^{-1}\left(\begin{array}{c}{u}_{\overline{\mathcal{Z}}}\\ 0\end{array}\right).$$ (10)

There are two approaches for computing the right-hand side of equation (10). When A = d² f (x) is positive definite and $\mathit{B}={\mathit{U}}_{\mathcal{Z}}$ has full row rank, the relevant inverse amounts to

$${\left(\begin{array}{cc}\mathit{A}& {\mathit{B}}^t\\ \mathit{B}& 0\end{array}\right)}^{-1}=\left(\begin{array}{cc}{\mathit{A}}^{-1}-{\mathit{A}}^{-1}{\mathit{B}}^t{\left[\mathit{B}{\mathit{A}}^{-1}{\mathit{B}}^t\right]}^{-1}\mathit{B}{\mathit{A}}^{-1}& {\mathit{A}}^{-1}{\mathit{B}}^t{\left[\mathit{B}{\mathit{A}}^{-1}{\mathit{B}}^t\right]}^{-1}\\ {\left[\mathit{B}{\mathit{A}}^{-1}{\mathit{B}}^t\right]}^{-1}\mathit{B}{\mathit{A}}^{-1}& -{\left[\mathit{B}{\mathit{A}}^{-1}{\mathit{B}}^t\right]}^{-1}\end{array}\right).$$

The numerical cost of computing the inverse scales as $O(n^3)+O({|\mathcal{Z}|}^3)$. When d² f (x) is a constant, the inverse is computed once. Sequentially updating it for different active sets $\mathcal{Z}$ is then conveniently organized around the sweep operator of computational statistics [8]. For a general convex function f (x), every time x changes, the inverse must be recomputed. This burden plus the cost of computing the entries of d² f (x) slow the path algorithm for general convex problems. At the same time, Section 4 demonstrates how a greater level of generality opens up many new applications. See our recent paper [21] on convex clustering for yet another application of path following.

In many applications f (x) is convex but not necessarily strictly convex. By Proposition 2, ℰ_ρ is also convex for all ρ > 0. Furthermore, ℰ_ρ may admit a unique global minimum for large ρ. In this case the proof of Proposition 4 still applies, and the solution path is continuous. One can circumvent problems in inverting d² f (x) by reparameterizing [4]. For the sake of simplicity, suppose that all of the constraints are affine and that ${\mathit{U}}_{\mathcal{Z}}$ has full row rank. The set of points x satisfying the active constraints can be written as x = w+Y y, where w is a particular solution, y is free to vary, and the columns of $\mathit{Y}\in {\mathrm{R}}^{n\times (n-|Z|)}$ span the null space of ${\mathit{U}}_{\mathcal{Z}}$ and hence are orthogonal to the rows of ${\mathit{U}}_{\mathcal{Z}}$. Under the null space reparameterization, $\frac{d\mathit{x}}{d\rho }=\mathit{Y}\frac{d\mathit{y}}{d\rho }$. Furthermore,

$$\begin{array}{l}{\mathit{Y}}^t{d}^{2}f(\mathit{x})\mathit{Y}=d_{\mathit{y}}^{2}f(\mathit{w}+{\mathit{Y}}_{\mathit{y}})\\ {\mathit{Y}}^t{u}_{\overline{\mathcal{Z}}}(\mathit{x})={\nabla }_{\mathit{y}}[-\rho {\displaystyle \sum _{i\in {\mathcal{N}}_{\mathrm{E}}}{g}_i(\mathit{w}+{\mathit{Y}}_{\mathit{y}})}\\ +\rho {\displaystyle \sum _{i\in {\mathcal{P}}_{\mathrm{E}}}{g}_i(\mathit{w}+{\mathit{Y}}_{\mathit{y}})+\rho {\displaystyle \sum _{j\in {\mathcal{P}}_{\mathrm{I}}}{g}_i(\mathit{w}+{\mathit{Y}}_{\mathit{y}})}}].\end{array}$$

It follows that equation (10) becomes

$$\begin{array}{l}\frac{d}{d\rho }\mathit{y}=-{\left[{\mathit{Y}}^t{d}^{2}f(\mathit{x})\mathit{Y}\right]}^{-1}{\mathit{Y}}^t{u}_{\overline{\mathcal{Z}}}\\ \frac{d}{d\rho }\mathit{x}=-\mathit{Y}{\left[{\mathit{Y}}^t{d}^{2}f(\mathit{x})\mathit{Y}\right]}^{-1}{\mathit{Y}}^t{u}_{\overline{\mathcal{Z}}}.\end{array}$$ (11)

Differentiating equation (7) gives the multiplier derivatives

$$\frac{d}{d\rho }\left[\begin{array}{c}{\mathit{\lambda }}_{{\mathcal{Z}}_{\mathrm{E}}}\\ {\mathit{w}}_{{\mathcal{Z}}_{\mathrm{I}}}\end{array}\right]=-{({\mathit{U}}_{\mathcal{Z}}{\mathit{U}}_{\mathcal{Z}}^t)}^{-1}{\mathit{U}}_{\mathcal{Z}}\left(d^{2}f(\mathit{x})\frac{d\mathit{x}}{d\rho }+u_{\overline{\mathcal{Z}}}\right).$$ (12)

The obvious advantage of using equation (11) is that the matrix Y^t d² f (x) Y can be nonsingular when d² f (x) is singular. The computational cost of evaluating the right-hand sides of equations (11) and (12) is $O([n-{|\mathcal{Z}|}^3])+O({|\mathcal{Z}|}^3)$ When $n-|\mathcal{Z}|$ and $|\mathcal{Z}|$ are small to compared n, this is an improvement over the cost $O(n^3)+O({|\mathcal{Z}|}^3)$ of computing the right-hand side of equation (8). Balanced against this gain is the requirement of finding a basis of the null space of ${\mathit{U}}_{\mathcal{Z}}$. Fortunately, the matrix Y is constant over each path segment and in practice can be computed by taking the QR decomposition of the active constraint matrix ${\mathit{U}}_{\mathcal{Z}}$. At each kink of the solution path, either one constraint enters $\mathcal{Z}$ or one leaves. Therefore, Y can be sequentially computed by standard updating and downdating formulas [22, 4]. Which ODE (8) or (11) is preferable depends on the specific application. When the loss function f (x) is not strictly convex, for example when the number of parameters exceeds the number of cases in regression, path following requires the ODE (11). Interested readers are referred to the book [4] for a more extended discussion of range-space versus null-space optimization methods.

For a general convex program, one can employ Euler’s update

$$\left[\begin{array}{c}\mathit{x}(\rho +\Delta \rho )\\ \mathit{\lambda }(\rho +\Delta \rho )\\ \mathbf{\omega }(\rho +\Delta \rho )\end{array}\right]=\left[\begin{array}{c}\mathit{x}(\rho )\\ \mathit{\lambda }(\rho )\\ \mathbf{\omega }(\rho )\end{array}\right]+\Delta \rho \frac{d}{d\rho }\left[\begin{array}{c}\mathit{x}(\rho )\\ \mathit{\lambda }(\rho )\\ \mathbf{\omega }(\rho )\end{array}\right]$$

to advance the solution of the ODE (8). Euler’s formula may be inaccurate for ∆ρ large. One can correct it by fixing ρ and performing one step of Newton’s method to re-connect with the solution path. This amounts to replacing the position-multiplier vector by

$$\left[\begin{array}{c}\mathit{x}\\ \mathit{\lambda }\\ \mathbf{\omega }\end{array}\right]-{\partial }_{\mathit{x},\mathit{\lambda },\mathbf{\omega }}k{(\mathit{x},\mathit{\lambda },\mathbf{\omega },\rho )}^{-1}k(\mathit{x},\mathit{\lambda },\mathbf{\omega },\rho ).$$

In practice, it is certainly easier and probably safer to rely on ODE packages such as the ODE45 function in Matlab to advance the solution of the ODE.

4 Examples of Path Following

Our examples are intended to illuminate the mechanics of path following and showcase its versatility. As we emphasized in the introduction, we forgo comparisons with other methods. Comparisons depend heavily on programming details and problem choices, so a premature study might well be misleading.

Example 1 Projection onto the Feasible Region

Finding a feasible point is the initial stage in many convex programs. Dykstra’s algorithm [23, 24] was designed precisely to solve the problem of projecting an exterior point onto the intersection of a finite number of closed convex sets. The projection problem also yields to our generic path following algorithm. Consider the toy example of projecting a point b ∈ ℝ² onto the intersection of the closed unit ball and the closed half space x₁ ≥ 0 [18]. This is equivalent to solving

$$\begin{array}{l}\text{minimize}f(\mathit{x})=\frac{1}{2}{\Vert \mathit{x}-\mathit{b}\Vert }^2\\ \text{subject to}{h}_1(\mathit{x})=\frac{1}{2}{\Vert \mathit{x}\Vert }^2-\frac{1}{2}\le 0,h_2(\mathit{x})=-{\mathit{x}}_1\le 0.\end{array}$$

The relevant gradients and second differentials are

$$\begin{array}{l}\nabla f(\mathit{x})=\mathit{x}-\mathit{b},\nabla h_1(\mathit{x})=\mathit{x},\nabla h_2(\mathit{x})=-\left(\begin{array}{c}1\\ 0\end{array}\right)\\ d^{2}f(\mathit{x})=d^2{h}_1(\mathit{x})={\mathit{I}}_2,d^2{h}_2(\mathit{x})=0.\end{array}$$

Path following starts from the unconstrained solution x (0) = b; the direction of movement is determined by formula (8). For x ∈ {x :‖x‖² > 1, x₁ > 0}, the path

$$\frac{d}{d\rho }\mathit{x}=-{[(1+\rho ){\mathit{I}}_2]}^{-1}\mathit{x}=-\frac{1}{1+\rho }\mathit{x}$$

heads toward the origin. For x ∈ {x : |x₂| > 1, x₁ = 0}, the path

$$\frac{d}{d\rho }\left(\begin{array}{c}\mathit{x}\\ {\mathit{\omega }}_2\end{array}\right)=-{\left(\begin{array}{ccc}1+\rho & 0& -1\\ 0& 1+\rho & 0\\ -1& 0& 0\end{array}\right)}^{-1}\left(\begin{array}{c}{\mathit{x}}_1\\ {\mathit{x}}_2\\ 0\end{array}\right)=-\frac{1}{1+\rho }\left(\begin{array}{c}0\\ {\mathit{x}}_2\\ 0\end{array}\right)$$

also heads toward the origin. For x ∈ {x : ‖x‖² > 1, x₁ < 0}, the path

$$\frac{d}{d\rho }\mathit{x}=-{[(1+\rho ){\mathit{I}}_2]}^{-1}\left(\begin{array}{c}{\mathit{x}}_1-1\\ {\mathit{x}}_2\end{array}\right)=-\frac{1}{1+\rho }\left(\begin{array}{c}{\mathit{x}}_1-1\\ {\mathit{x}}_2\end{array}\right).$$

heads toward the point (1, 0)^t. For x ∈ {x : ‖x‖² = 1, x₁ < 0}, the path

$$\frac{d}{d\rho }\left(\begin{array}{c}\mathit{x}\\ {\mathit{\omega }}_1\end{array}\right)=-{\left(\begin{array}{ccc}1+{\mathit{\omega }}_1& 0& {\mathit{x}}_1\\ 0& 1+{\mathit{\omega }}_1& {\mathit{x}}_2\\ {\mathit{x}}_1& {\mathit{x}}_2& 0\end{array}\right)}^{-1}\left(\begin{array}{c}-1\\ 0\\ 0\end{array}\right)=\left(\begin{array}{c}-\frac{{\mathit{x}}_2^2}{1+{\mathit{\omega }}_1}\\ \frac{{\mathit{x}}_1{\mathit{x}}_2}{1+{\mathit{\omega }}_1}\\ -{\mathit{x}}_1\end{array}\right)$$

is tangent to the circle. Finally, for x ∈ {x : ‖x‖² < 1, x₁ < 0}, the path

$$\frac{d}{d\rho }\mathit{x}=-{\mathit{I}}_2^{-1}\left(\begin{array}{c}-1\\ 0\end{array}\right)=\left(\begin{array}{c}1\\ 0\end{array}\right)$$

heads toward the x₂-axis. The left panel of Figure 1 plots the vector field $\frac{d}{d\rho }\mathit{x}$ at the time ρ = 0. The right panel shows the solution path for projection from the points (−2, 0.5)^t, (−2, 1.5)^t, (−1, 2)^t, (2, 1.5)^t, (2, 0)^t, (1, 2)^t, and (−0.5, −2)^t onto the feasible region. In projecting the point b = (−1, 2)^t onto (0, 1)^t, the ODE45 solver of Matlab evaluates derivatives at 19 different time points. Dykstra’s algorithm by comparison takes about 30 iterations to converge [18].

Fig. 1.

Projection to the positive half disk. Left: Derivatives at ρ = 0 for projection onto the half disc. Right: Projection trajectories from various initial points.

Example 2 Nonnegative Least Squares (NNLS) and Nonnegative Matrix Factorization (NNMF)

Non-negative matrix factorization (NNMF) is an alternative to principle component analysis and is useful in modeling, compressing, and interpreting nonnegative data such as observational counts and images. The articles [25, 26, 27] discuss in detail estimation algorithms and statistical applications of NNMF. The basic idea is to approximate an m × n data matrix X = (x_ij) with nonnegative entries by a product V W of two low rank matrices V = (v_ik) and W = (w_kj) with nonnegative entries. Here V and W are m × r and r × n respectively, with r ≪ min {m, n}. One version of NNMF minimizes the criterion

$$f(\mathit{V},\mathit{W})={\Vert \mathit{X}-\mathit{V}\mathit{W}\Vert }_{\mathrm{F}}^2={{\displaystyle \sum _i{\displaystyle \sum _j\left(x_{ij}-{\displaystyle \sum _k{v}_{ik}{w}_{kj}}\right)}}}^2,$$ (13)

where ‖⋅‖_F denotes the Frobenius norm. In a typical imaging problem, m (number of images) might range from 10³ to 10⁴, n (number of pixels per image) might surpass 10⁴, and a rank r = 50 approximation might adequately capture X.

Minimization of the objective function (13) is nontrivial because it is not jointly convex in V and W. Multiple local minima are possible. The well-known multiplicative algorithm [26, 27] enjoys the descent property, but it is not guaranteed to converge to even a local minimum [25]. An alternative algorithm that exhibits better convergence is alternating least squares (ALS). In updating W with V fixed, ALS solves the n separated nonnegative least square (NLS) problems

$$\begin{array}{c}\min \\ {\mathit{w}}_j\end{array}{\Vert {\mathit{x}}_j-\mathit{V}{\mathit{w}}_j\Vert }_2^2\text{subject to}{\mathit{w}}_j\ge 0,$$ (14)

where x_j and w_j denote the j-th columns of the corresponding matrices. Similarly, in updating V with W fixed, ALS solves m separated NNLS problems. The unconstrained solution W (0) = (V^t V)⁻¹ V^t X of W for fixed V requires just one QR decomposition of V or one Cholesky decomposition of V^t V. The exact path algorithm for solving the subproblem problem (14) commences with W (0). If W (ρ) stabilizes with just a few zeros, then the path algorithm ends quickly and is extremely efficient. For a NNLS problem, the path is piecewise linear, and one can straightforwardly project the path to the next hitting or escape time using the sweep operator [8]. Figure 2 shows a typical piecewise linear path for a problem with r = 50 predictors. Each projection to the next event requires 2r² flops. The number of path segments (events) roughly scales as the number of negative components in the unconstrained solution.

Fig. 2.

Piecewise linear paths of the regression coefficients for a NNLS problem with 50 predictors.

Example 3 Quadratically Constrained Quadratic Programming (QCQP)

Example 1 is a special case of quadratically constrained quadratic programming (QCQP). In convex QCQP [1, Section 4.4], one minimizes a convex quadratic function over an intersection of ellipsoids and affine subspaces.

Mathematically, this amounts to the problem

$$\begin{array}{l}\text{minimize}f(\mathit{x})=\frac{1}{2}{\mathit{x}}^t{P}_0\mathit{x}+{\mathit{b}}_0^t\mathit{x}+c_0\\ \text{subject to}{g}_i(\mathit{x})=a_i^t\mathit{x}-d_i=0,i=1,\dots ,r\\ h_j(\mathit{x})=\frac{1}{2}{\mathit{x}}^t{P}_j\mathit{x}+{\mathit{b}}_j^t\mathit{x}+c_j\le 0,j=1,\dots ,s,\end{array}$$

where P₀ is a positive definite matrix and the P_j are positive semidefinite matrices. Our algorithm starts with the unconstrained minimum $\mathit{x}(0)=-P_0^{-1}{\mathit{b}}_0$ and proceeds along the path determined by the derivative

$$\frac{d}{d\rho }\left[\begin{array}{c}\mathit{x}\\ {\mathit{\lambda }}_{{\mathcal{Z}}_{\mathrm{E}}}\\ {\mathbf{\omega }}_{{\mathcal{Z}}_{\mathrm{I}}}\end{array}\right]=-{\left(\begin{array}{c}{P}_0+\rho {\displaystyle {\sum }_{j\in {\mathcal{P}}_{\mathrm{I}}}{P}_j+}{\displaystyle {\sum }_{j\in {\mathcal{Z}}_{\mathrm{I}}}{\mathit{\omega }}_j{P}_j}\\ {\mathit{U}}_{\mathcal{Z}}(\mathit{x})\end{array}\begin{array}{c}{\mathit{U}}_{\mathcal{Z}}^t(\mathit{x})\\ 0\end{array}\right)}^{-1}\left(\begin{array}{c}{u}_{\overline{\mathcal{Z}}}(\mathit{x})\\ 0\end{array}\right),$$

where ${\mathit{U}}_{\mathcal{Z}}$ (x) has rows $a_i^t$ for i ∈ ${\mathcal{Z}}_{\mathrm{E}}$ and (P_j x + b_j)^t for j ∈ ${\mathcal{Z}}_{\mathrm{I}}$, and

$$u_{\overline{\mathcal{Z}}}(\mathit{x})=-{\displaystyle \sum _{i\in {\mathcal{N}}_{\mathrm{E}}}{a}_i}+{\displaystyle \sum _{i\in {\mathcal{P}}_{\mathrm{E}}}{a}_i}+{\displaystyle \sum _{i\in {\mathcal{P}}_{\mathrm{I}}}(P_j\mathit{x}+{\mathit{b}}_j)}.$$

Affine inequality constraints can be accommodated by setting one or more of the P_j equal to 0.

As a numerical illustration, consider the bivariate problem

$$\begin{array}{l}\text{minimize}f(\mathit{x})=\frac{1}{2}{\mathit{x}}_1^2+{\mathit{x}}_2^2-{\mathit{x}}_1{\mathit{x}}_2+\frac{1}{2}{\mathit{x}}_1-2{\mathit{x}}_2\\ \text{subject to}{h}_1(\mathit{x})={\left({\mathit{x}}_1-\frac{1}{2}\right)}^2+{\mathit{x}}_2^2-1\le 0\\ h_2(\mathit{x})={\left({\mathit{x}}_1+\frac{1}{2}\right)}^2+{\mathit{x}}_2^2-1\le 0\\ h_3(\mathit{x})={\mathit{x}}_1^2+{\left({\mathit{x}}_2-\frac{1}{2}\right)}^2-1\le 0.\end{array}$$ (15)

Here the feasible region is given by the intersection of three disks with centers (0.5, 0)^t, (−0.5, 0)^t, and (0, 0.5)^t, respectively, and a common radius of 1. Figure 3 displays the solution trajectory. Starting from the unconstrained minimum x (0) = (1, 1.5)^t, it hits, slides along, and exits two circles before its journey ends at the constrained minimum (0.059, 0.829)^t. The ODE45 solver of Matlab evaluates derivatives at 72 time points along the path.

Fig. 3.

Trajectory of the exact penalty path algorithm for a QCQP problem (15). The solid lines are the contours of the objective function f (x). The dashed lines are the contours of the constraint functions h_j (x).

Example 4 Geometric Programming

As a branch of convex optimization theory, geometric programming stands just behind linear and quadratic programming in importance [28, 29, 30, 31]. It has applications in chemical equilibrium problems [32], structural mechanics [29], digit circuit design [33], maximum likelihood estimation [34], stochastic processes [35], and a host of other subjects [28, 29]. Geometric programming deals with posynomials, which are functions of the form

$$f(\mathit{x})={\displaystyle \sum _{\alpha \in S}{c}_{\alpha }}{\displaystyle \prod _{i=1}^n{\mathit{x}}_i^{{\alpha }_i}=}{\displaystyle \sum _{\alpha \in S}{c}_{\alpha }{e}^{{\alpha }^t\mathit{y}}}=f(\mathit{y}).$$ (16)

In the left-hand definition of this equivalent pair of definitions, the index set S ⊂ ℝⁿ is finite, and all coefficients c_α and all components x₁, …, x_n of the argument x of f (x) are positive. The possibly fractional powers α_i corresponding to a particular α may be positive, negative, or zero. For instance, ${\mathit{x}}_1^{-1}+2{\mathit{x}}_1^3{\mathit{x}}_2^{-2}$ is a posynomial on ℝ². In geometric programming, one minimizes a posynomial f (x) subject to posynomial inequality constraints of the form h_j (x) ≤ 1 for 1 ≤ j ≤ s. In some versions of geometric geometric programming, equality constraints of monomial type are permitted [28]. The right-hand definition in equation (16) invokes the exponential reparameterization ${\mathit{x}}_i=e^{{\mathit{y}}_i}$. This simple transformation has the advantage of rendering a geometric program convex. In fact, any posynomial f (y) in the exponential parameterization is log-convex and therefore convex. The concise representations

$$\nabla f(\mathit{y})={\displaystyle \sum _{\alpha \in S}{c}_{\alpha }{e}^{{\alpha }^t\mathit{y}}\alpha },d^{2}f(\mathit{y})={\displaystyle \sum _{\alpha \in S}{c}_{\alpha }{e}^{{\alpha }^t\mathit{y}}\alpha {\alpha }^t}$$

of the gradient and the second differential are helpful in both theory and computation.

Without loss of generality, one can repose geometric programming as

$$\begin{array}{l}\text{minimize}\ln f(\mathit{y})\\ \text{subject to}\ln g_i(\mathit{y})=0,1\le i\le r\\ \ln h_j(\mathit{y})\le 0,1\le j\le s,\end{array}$$ (17)

where f (y) and the h_j (y) are posynomials and the equality constraints ln g_i (y) are affine. In this exponential parameterization setting, it is easy to state necessary and sufficient conditions for strict convexity and coerciveness.

Proposition 6

The objective function f (y) in the geometric program (17) is strictly convex if and only if the subspace spanned by the vectors {α}_α∈S is all of ℝⁿ; f (y) is coercive if and only if the polar cone {z : z^t α ≤ 0 for all α ∈ S} reduces to the origin 0. Equivalently, f (y) is coercive if the origin 0 belongs to the interior of the convex hull of the set S.

Proof These claims are proved in detail in our paper [36].

According to Propositions 1 and 4, the strict convexity and coerciveness of f (y) guarantee the uniqueness and continuity of the solution path in y. This in turn implies the uniqueness and continuity of the solution path in the original parameter vector x. The path directions are related by the chain rule

$$\frac{d}{d\rho }{x}_i(\rho )=\frac{d{x}_i}{d{y}_i}\frac{d}{d\rho }{y}_i(\rho )=x_i\frac{d}{d\rho }{y}_i(\rho ).$$

As a concrete example, consider the problem

$$\begin{array}{l}\text{minimize}{x}_1^{-3}+3x_1^{-1}{x}_2^{-2}+x_1{x}_2\\ \text{subject to}\frac{1}{6}{x}_1^{1/2}+\frac{2}{3}{x}_2\le 1,x_1>0,x_2>0.\end{array}$$ (18)

It is easy to check that the vectors {(−3, 0)^t, (−1_, 2)^t, (1, 1)^t} span ℝ² and generate a convex hull strictly containing the origin 0. Therefore, f (y) is strictly convex and coercive. It achieves its unconstrained minimum at the point $x(0)={(\sqrt[5]{6},\sqrt[5]{6})}^t$, or equivalently y (0) = (ln 6/5, ln 6/5)^t. To solve the constrained minimization problem, we follow the path dictated by the revised geometric program (17). Figure 4 plots the trajectory from the unconstrained solution to the constrained solution in the original x variables. The solid lines in the figure represent the contours of the objective function f (x), and the dashed lines represent the contours of the constraint function h (x). The ODE45 solver of Matlab evaluates derivatives at seven time points along the path.

Fig. 4.

Trajectory of the exact penalty path algorithm for the geometric programming problem (18). The solid lines are the contours of the objective function f (x). The dashed lines are the contours of the constraint function h (x) at levels 1, 1.25, and 1.5.

Example 5 Semidefinite Programming (SDP)

The linear semidefinite programming problem [37] consists in minimizing the trace function X → tr (C X) over the cone of positive semidefinite matrices $S_{+}^n$ subject to the linear constraints tr (A_i X) = b_i for 1 ≤ i ≤ p. Here C and the A_i are assumed symmetric. According to Sylvester’s criterion, the constraint $\mathit{X}\in S_{+}^n$ involves a complicated system of inequalities involving nonconvex functions. One way of cutting through this morass is to focus on the minimum eigenvalue v₁ (X) of X. Because the function v₁ −(X) is convex, one can enforce positive semidefiniteness by requiring v₁ (X) ≤ 0. Thus, the linear semidefinite programming problem is a convex program in the standard functional form.

It simplifies matters enormously to assume that v₁ (X) has multiplicity 1. Let u be the unique, up to sign, unit eigenvector corresponding to v₁ (X). The matrix X is parameterized by the entries of its lower triangle. With these conventions, the following formulas

$$-\frac{\partial }{\partial x_{ij}}{\mathit{v}}_1(\mathit{X})=-u^t\frac{\partial }{\partial x_{ij}}\mathit{X}u$$ (19)

$$\begin{array}{l}-\frac{{\partial }^2}{\partial x_{ij}\partial x_{kl}}{\mathit{v}}_1(\mathit{X})=-u^t\frac{\partial }{\partial x_{ij}}\mathit{X}{({\mathit{v}}_1\mathit{I}-\mathit{X})}^{-}\frac{\partial }{\partial x_{kl}}\mathit{X}u\\ -u^t\frac{\partial }{\partial x_{kl}}\mathit{X}{({\mathit{v}}_1\mathit{I}-\mathit{X})}^{-}\frac{\partial }{\partial x_{ij}}\mathit{X}u\\ =-2u^t\frac{\partial }{\partial x_{ij}}\mathit{X}{({\mathit{v}}_1\mathit{I}-\mathit{X})}^{-}\frac{\partial }{\partial x_{kl}}\mathit{X}u\end{array}$$ (20)

for the first and second partial derivatives of −v₁(X) are well known [19]. Here the matrix (v₁I − X)⁻ is the Moore-Penrose inverse of v₁ I − X. The partial derivative of X with respect to its lower triangular entry x_ij equals E_ij + 1_{i≠j} E_ji, where E_ij is the matrix consisting of all 0’s excepts for a 1 in position (i, j). Note that $u^t{E}_{ij}=u_i{e}_j^t$ and E_kl u = u_l e_k for the standard unit vectors e_j and e_k. The second partial derivatives of X vanish. The Moore-Penrose inverse is most easily expressed in terms of the spectral decomposition of X. If we denote the ith eigenvalue of X by v_i and the corresponding ith unit eigenvector by u_i, then we have

$${(\mathit{X}-{\mathit{v}}_1\mathit{I})}^{-}={\displaystyle \sum _{i>1}\frac{1}{{\mathit{v}}_i-{\mathit{v}}_1}}{u}_i{u}_i^t$$

Finally, the formulas

$$\begin{array}{l}\mathrm{tr}({\mathit{A}}_i\mathit{X})-{\mathit{b}}_i={\displaystyle \sum _k{({\mathit{A}}_i)}_{kk}{x}_{kk}+2}{\displaystyle \sum _k{\displaystyle \sum _{l<k}{({\mathit{A}}_i)}_{kl}{x}_{kl}-{\mathit{b}}_i}}\\ \frac{\partial }{\partial x_{kl}}[\mathrm{tr}({\mathit{A}}_i\mathit{X})-{\mathit{b}}_i]={({\mathit{A}}_i)}_{kl}+1_{\{k\ne l\}}{({\mathit{A}}_i)}_{lk}\end{array}$$

express the linear constraints and their partial derivatives in terms of the lower triangular entries of X.

Initiating path following is problematic because tr (CX) has minimum −∞. A good strategy is to amend the surrogate function ℰ_ρ (x) by adding the term $\frac{\epsilon (\rho )}{2}{\Vert \mathit{X}\Vert }_{\mathrm{F}}^2$, where ℰ (ρ) is a smooth positive function that decreases to 0. Taking ℰ (ρ) = e^−cρ for c positive works well in practice. The new surrogate function $(C\mathit{X})+\frac{\epsilon (\rho )}{2}{\Vert \mathit{X}\Vert }_{\mathrm{F}}^2$ is strictly convex and possesses a unique minimum for all ρ ≥ 0. In view of the identities ${\Vert \mathit{X}\Vert }_{\mathrm{F}}^2={\displaystyle {\sum }_i{\displaystyle {\sum }_j{x}_{ij}^2}}$ and tr (CX) = Σ_i Σ_j c_ij x_ij for X = (x_ij) and C = (c_ij), the initial condition X (0) = − ε (0)⁻¹ C is straightforward to deduce.

Path following must be modified to accommodate the new surrogate function. In the notation of [19], let x = v (X) be the $\frac{1}{2}n(n+1)$ vector obtained from vec (X) by eliminating all supradiagonal entries, and let D be the $n^2\times \frac{1}{2}n(n+1)$ duplication matrix satisfying vec (X) = Dx. Applying the chain rule to the obvious identities ${\Vert \mathit{X}\Vert }_{\mathrm{F}}^2=\mathit{x}{D}^{t}D\mathit{x}$ and tr (CX) = vec (C)^t Dx, one can extend the derivation of Proposition 5 and prove that

$$\begin{array}{l}\frac{d}{d\rho }\left[\begin{array}{c}\mathit{x}\\ {\mathit{\lambda }}_{{\mathcal{Z}}_{\mathrm{E}}}\\ {\mathbf{\omega }}_{{\mathcal{Z}}_{\mathrm{I}}}\end{array}\right]\\ =-{\left[\begin{array}{c}\epsilon (\rho )D^{t}D-{\mathit{\omega }}_{{\mathcal{Z}}_{\mathrm{I}}}{d}^2{\mathit{v}}_1(\mathit{x})1_{\{{\mathit{v}}_1(\mathit{X})=0\}}\\ {\mathit{U}}_{\mathcal{Z}}\end{array}\begin{array}{c}{\mathit{U}}_{\mathcal{Z}}^t\\ 0\end{array}\right]}^{-1}\\ \times \left(\begin{array}{c}\frac{d\epsilon (\rho )}{d\rho }{D}^{t}D\mathit{x}-{\displaystyle {\sum }_{i\in {\mathcal{N}}_{\mathrm{E}}}{D}^t\text{vec}({\mathit{A}}_i)+{\displaystyle {\sum }_{i\in {\mathcal{P}}_{\mathrm{E}}}{D}^t\text{vec}({\mathit{A}}_i)}-\nabla }{\mathit{v}}_1(\mathit{x})1_{\{{\mathit{v}}_1(\mathit{X})<0\}}\\ 0\end{array}\right).\end{array}$$

Path following proceeds until all constraints are satisfied and ℰ (ρ) is negligible.

For didactic purposes, considering the problem of minimizing tr (CX) subject to

$$\mathrm{tr}({\mathit{A}}_1\mathit{X})=1,\mathrm{tr}({\mathit{A}}_2\mathit{X})=2,\text{and}\mathit{X}\in S_{+}^2,$$

where

$$C=\left(\begin{array}{cc}0& \frac{1}{2}\\ \frac{1}{2}& 0\end{array}\right),{\mathit{A}}_1=\left(\begin{array}{cc}1& 0\\ 0& 0\end{array}\right),\text{and}{\mathit{A}}_2=\left(\begin{array}{cc}0& 0\\ 0& 1\end{array}\right).$$

Figure 5 displays the solution paths of the entries x_ij of X and the minimum eigenvalue v₁ Here we use ℰ (ρ) = e^−ρ. The path starts with X (0) = −C, hits, slides along, and exits various constraints, and ends at the constrained solution $\left(\begin{array}{cc}1& -\sqrt{2}\\ -\sqrt{2}& 2\end{array}\right)$.

Fig. 5.

Solution path of a semidefinite programming example.

Example 6 Image Denoising

Image analysis is another fertile field for path following. Here we explore how to restore or enhance images by removing noise. This example differs from previous examples in that the fully constrained solution is trivial. The solution path itself is the object of interest. Suppose that w = (w_ij) ∈ ℝ^m×n represents the recorded gray levels across a 2D array of pixels from a noisy image with true gray levels u = (u_ij). The well-known denoising model of Rudin-Osher-Fatemi (ROF) [38] minimizes the total variation regularized least squares criterion

$$\begin{array}{l}\frac{1}{2}{\Vert \mathit{w}-\mathit{u}\Vert }_2^2+\rho \mathrm{TV}(\mathit{u})\\ =\frac{1}{2}{\displaystyle \sum _{ij}{({\mathit{w}}_{ij}-u_{ij})}^2}+\rho {\displaystyle \sum _{i,j}\sqrt{{(u_{i+1,j}-u_{ij})}^2+{(u_{i,j+1}-u_{ij})}^2}}.\end{array}$$ (21)

The total variation penalty serves to smooth the reconstructed image and preserve its edges. A similar effect can be achieved by replacing the isotropic penalty TV (u) by the anisotropic penalty

$${\mathrm{TV}}_1(\mathit{u})={\displaystyle \sum _{i,j}(|u_{i+1,j}-u_{ij}|+|u_{i,j+1}-u_{ij}|)}.$$ (22)

In this example we focus on path following for the anisotropic penalty and a more general convex loss function f(u). The objective function is now

$$f(\mathit{u})+\rho {\Vert \mathit{D}\mathit{u}\Vert }_1.$$ (23)

For instance, the amended loss function $f(\mathit{u})=\frac{1}{2}{\Vert \mathit{w}-\mathit{K}\mathit{u}\Vert }_2^2$ with a Gaussian or motion blurring matrix K is appropriate in many imaging problems. Poisson count data are relevant to image reconstruction in X-ray and positron tomography [20] and to image denoising in certain circumstances [39]. With Poisson noise, the least squares criterion is replaced by a negative loglikeli-hood. The difference matrix D captures the ℓ₁ penalty (22). Note that the matrices w and u are now viewed as vectors. For an m × n 2D image, the difference matrix D has 2mn − m − n rows (penalties) and mn columns (pixels). This matrix is very sparse, with just 2(2mn − m − n) nonzero entries equal to ±1. When m and n are both at least 2, D has more rows than columns and a reduced column rank of mn −1.

For sufficiently large ρ, the minimum of the objective functions (21) reduces to a constant vector (blank image) equal to the average value $\overline{w}$ of the w_ij. The goal of image denoising is to find a ρ such that the recovered image is judged satisfactory by visual inspection or other more quantitative criteria. Notable computational advances in solving this problem include Chambolle’s algorithm [40] and split Bregman iteration [41]. These methods minimize the objective functions (21) and (23) for a fixed value of ρ. The web site of UCLA’s Computational and Applied Math Group summarizes the most recent progress in this area. In reality, outer iterations are almost always required to tune the parameter ρ. Path following is an attractive option because it provides the whole solution path at about the same computational cost as recovering the solution for an individual ρ.

Although it is tempting to minimize the criterion (23) by path following, the regularization matrix D has linearly dependent rows and deficient rank. Because the assumptions of Proposition 4 are violated, the multipliers λ_E of the active constraints in equations (7) and (9) are not uniquely determined. One can intuitively understand the difficulty by considering a square with four pixels. Whenever any three constraints are active, the fourth is automatically active as well. This constraint redundancy can be remedied by reparameterizing the model in terms of neighboring pixel differences x = Du. Unfortunately, the rank deficiency of D is also an issue. Adding the same constant to all of the components of u yields exactly the same x. To circumvent this problem, we simply append a bottom row to D with all entries 0 except for a 1 in the last position. If V is the amended version of D, then V has full column rank, and the vector x = Vu uniquely determines the image. Indeed, one can solve for x in the form u = (V^t V)⁻¹ V^t x. The bottom entry of x is obviously the gray level of the last pixel of the image.

Despite the presence of the inverse of the huge mn × mn matrix V^t V, the transformation u = (V^t V)⁻¹ V^t x is not as daunting as it appears. First of all, multiplication by the sparse matrix V^t is trivial. More importantly, the matrix V^t V is symmetric, banded, and extremely sparse. To count its nonzero entries, note that except for diagonal entries, these entries occur in the same positions as the nonzero entries of the adjacency matrix of a corresponding graph with 2mn − m − n edges and mn nodes. Because an adjacency matrix has twice as many nonzero entries as edges, the matrix V^t V has at most 2(2mn − m − n) + mn = 5mn −2m −2n nonzero entries. These occur within a band of width min{m, n} along the main diagonal, depending on whether we stack columns or concatenate rows. The most convenient way to solve equations of the kind V^t V a = b is to extract the Cholesky decomposition L of V^t V and execute forward and backward substitution. Although extraction of L is cheap for banded matrices, it is even cheaper for banded matrices with just a handful of nonzero entries per row. In our experience, the computational complexity of extracting L scales linearly in the product mn. Since L itself is sparse, forward and backward substitution are also very cheap. For instance with a 256 × 256 image, Matlab computes L (a 65536 × 65536 matrix) in 0.26 seconds on a laptop; L contains just 1,971,395 nonzero entries. The sparsity of L suggests that it be computed once and stored in compressed format for all images of a given size. Many of its nonzero entries are close to zero. Thus, a fairly light truncation of the non-diagonal entries of L gives an even sparser matrix realizing nearly the same transformation. Figure 6 displays the sparsity pattern of the matrix V^t V and its permuted Cholesky factor L for 64 × 64 images. Images of other sizes show similar sparsity patterns.

Fig. 6.

Sparsity patterns of V^t V and its Cholesky decomposition L for 64-by-64 images.

The problem of minimizing the objective function $\frac{1}{2}\Vert \mathit{w}-\mathit{K}\mathit{u}{\Vert }_2^2+\rho \Vert \mathit{D}\mathit{u}{\Vert }_1$ in the transformed variable x turns out to coincide with lasso penalized regression, for which an efficient path algorithm is known [15, 16]. Let us sketch how path following works in the more general case. The objective function is f (Bx) + ρ‖x₋‖₁, where B = (V^t V)⁻¹V^t and x₋ denotes the vector x with its last entry deleted. The penalty contributions correspond to affine equality constraints in constrained minimization. In path following, the penalty constant ρ starts large and moves downward. The initial image is flat with gray level determined by taking x₋ = 0 and adjusting the last entry of x to minimize f (Bx). Call this point x_∞. The first escape time occurs at ρ_max = max_j |(B^t ∇ f (Bx_∞)_j|. At this juncture path following begins in earnest. Under the x parameterization, the loss function has gradient B^t ∇ f (Bx) and second differential B^td² f (Bx) B. Because f (Bx) is not strictly convex, our previous reparameterization from x to y variables is needed. Based on equation (11), the path ODEs reduce to

$$\begin{array}{l}\frac{d}{d\rho }{\mathit{x}}_{\overline{\mathcal{Z}}}=-{({\mathit{B}}_{\overline{\mathcal{Z}}}^t{d}^{2}f(\mathit{B}\mathit{x}){\mathit{B}}_{\overline{\mathcal{Z}}})}^{-1}\mathrm{sgn}({\mathit{x}}_{\overline{\mathcal{Z}}}),\frac{d}{d\rho }{\mathit{x}}_{\mathcal{Z}}=0,\\ \frac{d}{d\rho }{\mathit{\lambda }}_{\mathcal{Z}}=-{\mathit{B}}_{\mathcal{Z}}^t{d}^{2}f(\mathit{B}\mathit{x}){\mathit{B}}_{\overline{\mathcal{Z}}}\frac{d}{d\rho }{\mathit{x}}_{\overline{\mathcal{Z}}}.\end{array}$$ (24)

Observe that the updates of equation (11) drastically simplify because the rows of the active constraint matrix ${\mathit{U}}_{\mathcal{Z}}$ and the columns of its null space matrix Y are populated by standard Euclidean unit vectors. Furthermore, for the ROF model of image denoising, d² f (Bx) is a diagonal matrix. Alternatively, one can derive the ODEs (24) from first principles by implicitly differentiating the stationary conditions. Path following solves the coupled ODEs (24) segment by segment.

For a quadratic loss function, the second differential is constant, and the solution path is piecewise linear. Thus no ODE solving is involved. With a blurring matrix K, the second differential is B^t d² f (Bx) B = B^t K^t K B. After each path extension, the path directions (24) yield the next event time ρ_j at which a nonzero component x_j hits zero, or a multiplier λ_j of a zero component x_j hits ρ or −ρ. The path is then extended to the closest of these event times. In deblurring or denoising, the inverse of ${\mathit{B}}_{\overline{\mathcal{Z}}}^t{K}^{t}K{\mathit{B}}_{\overline{\mathcal{Z}}}$ is best computed via a QR decomposition of ${\mathit{B}}_{\overline{\mathcal{Z}}}K$. At each kink in the path, ${\mathit{B}}_{\overline{\mathcal{Z}}}K$ changes by adding or deleting a column of BK. As we mentioned earlier, it is straightforward to update or downdate the QR decomposition [22]. In the original ROF model, traversing one time segment requires about O (p) operations for p = mn total pixels. The whole process ends when T differences x_j becomes nonzero. In practice, a large value of T recovers too grainy an image, so T is typically much smaller than p. The total cost of computing the solution path is approximately O (K p), where K is the number of traversed kinks. Empirically K is on the order of O (T) for our examples.

Figure 7 illustrates denoising of a 112 × 91 image of a lighthouse. The corrupted image appears in the top-left corner of the figure. The p = 10, 192 pixels generate 20, 182 transformed variables. It takes our Matlab script about one minute of desktop computing time to traverse K = 2, 500 segments along the regularization path from ρ = 87.9881 (blank image) to ρ = 0.5206 (a nearly optimal image), where T = 2148 differences become nonzero. In the process, the lighthouse clearly emerges from the fog of oversmoothing. Figure 7 displays selected snapshots along the regularization path. We emphasize that path following based on equation (23) reveals the entire path for the interval [0.5206,87.9881] of ρ values. In practice, one can accelerate path following by starting from a ρ nearer to the ultimate destination.

Fig. 7.

A noisy image and snapshots along the regularization path.

5 Discussion

Our path following algorithm for constrained convex optimization builds on but differs from the tradition of path following in homotopy methods [12] and interior point programming [1]. The paths encountered in the exact penalty method introduce the novelty of piecewise differentiability, which can be effectively handled by tracking the Langrange multipliers. Computational statisticians deserve credit for exploring this difficult terrain [15, 16, 8, 13, 42, 43]. To our knowledge we are the first to make the connection to exact penalty methods.

Our algorithm enjoys the dual advantages of simplicity and generality. Given the rich numerical resources of Matlab, it is straightforward to solve the required ODEs segment by segment. Regardless of whether path following is faster or slower than existing optimization methods, it supplies the whole solution path. In regularized estimation, this level of detail offers unprecedented insight into how penalties and predictors interact. Our example on image denoising is a case in point.

In quadratic programming with affine equality and inequality constraints, the solution path is piecewise linear [8]. This permits path following to take large steps. Furthermore, each step can be implemented very efficiently by the sweep operator of computational statistics. Despite the loss of these advantages in more complicated examples, the real culprit in path-following deceleration in many applications is an excessive number of constraints to be navigated. Our image denoising example suffers from this defect. On the positive side of the ledger, in nonconvex problems path following may well prove to be more reliable than competing methods in separating global from local minima [44].

Various extensions of path following are in order. First, the current algorithm commences from the unconstrained solution. Our development relies on the strict convexity and coerciveness of the objective function to ensure a unique starting point. In principle, path initiation should work for any problem with a unique unconstrained minimum. Similarly, path continuation should be possible whenever the interior solution is well defined and piecewise smooth. As the image denoising example suggests, reparametrization can play an important role in correcting defects in strict convexity. Another possibility is to amend the surrogate function ℰ_ρ (x). In our semidefinite programming example, we add the term $e^{-c\rho }{\Vert \mathit{X}\Vert }_{\mathrm{F}}^2$ to enforce strict convexity and coerciveness. A similar tactic obviously works in other examples.

A second generalization is to expand the list of penalty functions. For instance, Euclidean penalties of the form ‖M x + a‖₂ are useful in grouping parameters in statistical problems. It should be straightforward to extend path following to include such penalties. A third generalization is to remove convexity restrictions altogether. As we have noted, the exact penalty method applies equally to nonconvex programming. Path following in this setting is nontrivial since the solution path is no longer necessarily continuous. This poses a real challenge, and it is unclear to us whether one can construct a theory as satisfying as that standing behind modern interior point methods. We invite the optimization community to tackle this broader issue. In the meantime, we are happy to share our Matlab code with interested researchers.