Extensions to the Proximal Distance Method of Constrained Optimization

Abstract

The current paper studies the problem of minimizing a loss f(x) subject to constraints of the form Dx ∈ S, where S is a closed set, convex or not, and D is a matrix that fuses parameters. Fusion constraints can capture smoothness, sparsity, or more general constraint patterns. To tackle this generic class of problems, we combine the Beltrami-Courant penalty method of optimization with the proximal distance principle. The latter is driven by minimization of penalized objectives $f(x)+\frac{\rho }{2}\text{dist}{(Dx,S)}^2$ involving large tuning constants ρ and the squared Euclidean distance of Dx from S. The next iterate x_n+1 of the corresponding proximal distance algorithm is constructed from the current iterate x_n by minimizing the majorizing surrogate function $f(x)+\frac{\rho }{2}{\Vert Dx-{𝒫}_S\left(D{x}_n\right)\Vert }^2$. For fixed ρ and a subanalytic loss f(x) and a subanalytic constraint set S, we prove convergence to a stationary point. Under stronger assumptions, we provide convergence rates and demonstrate linear local convergence. We also construct a steepest descent (SD) variant to avoid costly linear system solves. To benchmark our algorithms, we compare their results to those delivered by the alternating direction method of multipliers (ADMM). Our extensive numerical tests include problems on metric projection, convex regression, convex clustering, total variation image denoising, and projection of a matrix to a good condition number. These experiments demonstrate the superior speed and acceptable accuracy of our steepest variant on high-dimensional problems. Julia code to replicate all of our experiments can be found at https://github.com/alanderos91/ProximalDistanceAlgorithms.jl

1. Introduction

The generic problem of minimizing a continuous function f(x) over a closed set S of ${ℝ}^p$ can be attacked by a combination of the penalty method and distance majorization. The classical penalty method seeks the solution of a penalized version h_ρ(x) = f(x) + ρq(x) of f(x), where the penalty q(x) is nonnegative and 0 precisely when x ∈ S. If one follows the solution vector x_ρ as ρ tends to ∞, then in the limit one recovers the constrained solution (Beltrami, 1970; Courant, 1943). The function

$$q(x)=\frac{1}{2}\text{dist}{(x,S)}^2=\frac{1}{2}\underset{y\in S}{\text{min}}\Vert x-y{\Vert }^2,$$

is one of the most fruitful penalties in this setting. Our previous research for solving this penalized minimization problem has focused on an MM (majorization-minimization) algorithm based on distance majorization (Chi et al., 2014; Keys et al., 2019). In distance majorization one constructs the surrogate function

$$g_{\rho }\left(x\mid x_n\right)=f(x)+\frac{\rho }{2}{\Vert x-𝒫\left(x_n\right)\Vert }^2,$$

using the Euclidean projection 𝒫(x_n) of the current iterate x_n onto S. The minimum of the surrogate occurs at the proximal point

$$x_{n+1}={\text{prox}}_{{\rho }^{-1}f}\left[𝒫\left(x_n\right)\right].$$ (1)

According to the MM principle, this choice of x_n+1 decreases g_ρ(x | x_n) and hence the objective h_ρ(x) as well. As we note in our previous JMLR paper (Keys et al., 2019), the update (1) reduces to the classical proximal gradient method when S is convex (Parikh, 2014).

We have named this iterative scheme the proximal distance algorithm (Keys et al., 2019; Lange, 2016). It enjoys several virtues. First, it allows one to exploit the extensive body of results on proximal maps and projections. Second, it does not demand that the constraint set S be convex. If S is merely closed, then the map 𝒫(x) may be multivalued, and one must choose a representative element from the projection 𝒫(x_n). Third, the algorithm does not require the objective function f(x) to be differentiable. Fourth, the algorithm dispenses with the chore of choosing a step length. Fifth, if sparsity is desirable, then the sparsity level can be directly specified rather than implicitly determined by the tuning parameter of the lasso or other penalty.

Traditional penalty methods have been criticized for their numerical instability. This hazard is mitigated in the proximal distance algorithm by its reliance on proximal maps, which are often highly accurate. The major defect of the proximal distance algorithm is slow convergence. This can be ameliorated by Nesterov acceleration (Nesterov, 2013). There is also the question of how fast one should send ρ to ∞. Although optimal schedules are rarely known, simple numerical experiments yield a good choice. Finally, soft constraints can be achieved by stopping the steady increase of ρ at a finite value.

1.1. Proposed Framework

Distance majorization can be generalized in various ways. For instance, it can be expanded to multiple constraint sets. In practice, at most two constraint sets usually suffice. Another generalization is to replace the constraint x ∈ S by the constraint Dx ∈ S, where D is a compatible matrix. Again, the original case D = I is allowed. By analogy with the fused lasso of Tibshirani et al. (2005), we will call the matrix D a fusion matrix. With these ideas in mind, we examine the general problem of minimizing a differentiable function f(x) subject to r fused constraints D_ix ∈ S_i. We approach this problem by extending the proximal distance method. For a fixed penalty constant ρ, the objective function and its MM surrogate now become

$$h_{\rho }(x)=f(x)+\frac{\rho }{2}\sum _{i=1}^r\text{dist}{\left(D_{i}x,S_i\right)}^2,$$

$$g_{\rho }\left(x\mid x_n\right)=f(x)+\frac{\rho }{2}\sum _{i=1}^r{\Vert D_{i}x-{𝒫}_i\left(D_i{x}_n\right)\Vert }^2,$$

where 𝒫_i(y) denotes the projection of y onto S_i. Any or all of the fusion matrices D_i can be the identity I. Our motivating premise posits that projection onto each set S_i is more straightforward than projection onto its preimage $D_i^{-1}\left(S_i\right)$. When the contrary is true, one should clearly favor projection onto the preimage.

Fortunately, we can simplify the problem by defining S to be the Cartesian product ${\prod }_{i=1}^r{S}_i$ and D to be the stacked matrix

$$D=\left(\begin{array}{c}{D}_1\\ ⋮\\ D_r\end{array}\right).$$

Our objective and surrogate then revert to the less complicated forms

$$h_{\rho }(x)=f(x)+\frac{\rho }{2}\text{dist}{(Dx,S)}^2,$$ (2)

$$g_{\rho }\left(x\mid x_n\right)=f(x)+\frac{\rho }{2}{\Vert Dx-𝒫\left(D{x}_n\right)\Vert }^2,$$ (3)

respectively, where 𝒫(x) is the Cartesian product of the projections 𝒫_i(x). Note that all closed sets S_i with simple projections, including sparsity sets, are fair game.

1.2. Our Contributions

In the framework described above, we summarize the contributions of the current paper.

1. Section 2 describes solution algorithms for minimizing the penalized loss h_ρ(x). Our first algorithm is based on Newton’s method applied to the surrogate g_ρ(x | x_n). For some important problems, Newton’s method reduces to least squares. Our second method is a steepest descent algorithm on g_ρ(x | x_n) tailored to high dimensional problems.

2. For a sufficiently large ρ, we show that when f(x) and S are convex and f(x) possesses a unique minimum point in the preimage of S under D, y ∈ D⁻¹(S), the penalized loss h_ρ(x) attains its minimum value. This is the content of Proposition 3.1. Similarly, Proposition 3.2 shows that the surrogate g_ρ(x | x_n) also attains its minimum.

3. If in addition f(x) is differentiable, then Proposition 3.3 demonstrates that the MM iterates x_n for minimizing h_ρ(x) satisfy

   $$h_{\rho }\left(x_n\right)-h_{\rho }\left(z_{\rho }\right)=O\left(\frac{\rho }{n}\right),$$

   where z_ρ minimizes h_ρ(x). If f(x) is also L-smooth and μ-strongly convex, then Proposition 3.4 shows that z_ρ is unique and the iterates x_n converge to z_ρ at a linear rate.

4. More generally, Proposition 4.1 proves that the iterates x_n of a generic MM algorithm converge to a stationary point of a coercive subanalytic function h(x) with a good surrogate. Our objectives and their surrogates fall into this category. Proposition 4.2 specializes the results of Proposition 4.1 to proximal distance algorithms. Propositions 4.3 and 4.4 further specialize to proximal distance algorithms with sparsity constraints and demonstrate a linear rate of convergence. Sparsity sets appear in model selection.

5. Finally, we discuss a competing alternating direction method of multipliers (ADMM) algorithm and note its constituent updates. Our extensive numerical experiments compare the two proximal distance algorithms to ADMM. We find that proximal distance algorithms are competitive with and often superior to ADMM in terms of accuracy and running time.

1.3. Notation

The symbols x and y and their decorated variants are typically reserved for optimization variables. Fusion operators D are problem specific but understood as real matrices. Generally, D is neither injective nor surjective; the inverse image of S is denoted D⁻¹(S). The function f(x) represents the loss in the constrained minimization problem

$$\underset{x}{\text{min}}f(x)\text{such}\text{that}Dx\in S,$$

whereas $h_{\rho }(x)=f(x)+\frac{\rho }{2}\text{dist}{(Dx,S)}^2$ is the the penalized loss. Our point to set distance functions are based on the Euclidean norm via

$$\text{dist}(y,S)=\underset{z\in S}{\text{inf}}\Vert y-z{\Vert }_2\hspace{1em}\text{and}\hspace{1em}\text{dist}(Dx,S)=\underset{z\in S}{\text{inf}}\Vert Dx-z{\Vert }_2.$$

Finally, the notation 𝒫_S(y) indicates the unique Euclidean projection of y onto the closed convex set S. If S is merely closed, then 𝒫_S(y) may be set-valued. Depending on the context, 𝒫_S(y) also denotes a particular representative of this set.

2. Different Solution Algorithms

Unless f(x) is a convex quadratic, exact minimization of the surrogate g_ρ(x | x_n) is likely infeasible. As we have already mentioned, to reduce the objective h_ρ(x) in (2), it suffices to reduce the surrogate (3). For the latter task, we recommend Newton’s method on small and intermediate-sized problems and steepest descent on large problems. The exact nature of these generic methods are problem dependent. The following section provides a high-level overview of each strategy and we defer details on our later numerical experiments to the appendices.

2.1. Newton’s Method and Least Squares

Unfortunately, the proximal operator ${\text{prox}}_{{\rho }^{-1}f}(y)$ is no longer relevant in calculating the MM update x_n+1. When f(x) is smooth, Newton’s method for the surrogate g_ρ(x | x_n) defined in Equation (3) employs the update

$$x_{n+1}=x_n-{\left[H_n+\rho D^{t}D\right]}^{-1}\left\{\nabla f\left(x_n\right)+\rho D^t\left[D{x}_n-𝒫\left(D{x}_n\right)\right]\right\},$$

where H_n = d²f(x_n) is the Hessian. To enforce the descent property, it is often prudent to substitute a positive definite approximation H_n for d²f(x_n). In statistical applications, the expected information matrix is a natural substitute. It is also crucial to retain as much curvature information on f(x) as possible. Newton’s method has two drawbacks. First, it is necessary to compute and store d²f(x_n). This is mitigated in statistical applications by the substitution just mentioned. Second, there is the necessity of solving a large linear system. Fortunately, the matrix H_n + ρD^tD is often well-conditioned relative to H_n, for example, when D has full column rank and D^tD is positive definite. The method of conjugate gradients can be called on to solve the linear system in this ideal circumstance.

To reduce the condition number of the matrix H_n + ρD^tD even further, one can sometimes rephrase the Newton step as iteratively reweighted least squares. For instance, in a generalized linear model, the gradient ∇f(x) and the expected information H can be written as

$$\nabla f(x)=-Z^t{W}^{1/2}r\hspace{1em}\text{and}\hspace{1em}H=Z^{t}WZ,$$

where r is a vector of standardized residuals, Z is a design matrix, and W is a diagonal matrix of case weights (Lange, 2010; Nelder and Wedderburn, 1972). The Newton step is now equivalent to minimizing the least squares criterion

$$\begin{array}{l}\frac{1}{2}{x}^{*}{H}_{n}x-\nabla f{\left(x_n\right)}^{*}x={\Vert W_n^{1/2}Zx-W_n^{-1/2}\nabla f\left(x_n\right)\Vert }^2\\ ={\Vert \left(\begin{array}{c}{W}_n^{1/2}Z\\ \sqrt{\rho }D\end{array}\right)x-\left(\begin{array}{c}{W}_n^{1/2}Z{x}_n+r_n\\ \sqrt{\rho }𝒫\left(D{x}_n\right)\end{array}\right)\Vert }^2.\end{array}$$

In this context a version of the conjugate gradient algorithm adapted to least squares is attractive. The algorithms LSQR (Paige and Saunders, 1982) and LSMR (Fong and Saunders, 2011) perform well when the design is sparse or ill conditioning is an issue. Given the numerical stability of the iteratively reweighted least squares updates, we favor them over the explicit Newton steps in implementing proximal distance algorithms.

2.2. Proximal Distance by Steepest Descent

In high-dimensional optimization problems, gradient descent is typically employed to avoid matrix inversion. Determination of an appropriate step length is now a primary concern. In the presence of fusion constraints Dx ∈ S and a convex quadratic loss $f(x)=\frac{1}{2}{x}^{t}Ax+b^{t}x$, the gradient of the proximal distance objective at x_n amounts to

$$v_n=A_{n}x+b+\rho D^t\left[D{x}_n-𝒫\left(D{x}_n\right)\right].$$

For the steepest descent update x_n+1 = x_n − t_nv_n, one can show that the optimal step length is

$$t_n=\frac{{\Vert v_n\Vert }^2}{v_n^{t}A{v}_n+\rho {\Vert D{v}_n\Vert }^2}.$$

This update obeys the descent property and avoids matrix inversion. One can also substitute a local convex quadratic approximation around x_n for f(x). If the approximation majorizes f(x), then the descent property is preserved. In the failure of majorization, the safeguard of step halving is trivial to implement.

In addition to Nesterov acceleration, gradient descent can be accelerated by the subspace MM technique (Chouzenoux et al., 2010). Let G_n be the matrix with k columns determined by the k most current gradients of the objective h_ρ(x), including ∇h_ρ(x_n). Generalizing our previous assumption, suppose f(x) has a quadratic surrogate with Hessian H_n at x_n. Overall we get the quadratic surrogate

$$q_{\rho }\left(x\mid x_n\right)=g_{\rho }\left(x_n\mid x_n\right)+\nabla g_{\rho }{\left(x_n\mid x_n\right)}^t\left(x-x_n\right)+\frac{1}{2}{\left(x-x_n\right)}^t\left(H_n+\rho D^{t}D\right)\left(x-x_n\right).$$

of g_ρ(x | x_n). We now seek the best linear perturbation x_n + G_nβ of x_n by minimizing q_ρ(x_n + G_nβ | x_n) with respect to the coefficient vector β. To achieve this end, we solve the stationary equation

$$0=G_n^t\nabla g_{\rho }\left(x_n\mid x_n\right)+G_n^t\left(H_n+\rho D^{t}D\right)G_n\beta ,$$

and find $\beta =-{\left[G_n^t\left(H_n+\rho D^{t}D\right)G_n\right]}^{-1}{G}_n^t\nabla g_{\rho }\left(x_n\mid x_n\right)$, where the gradient is

$$\nabla g_{\rho }\left(x_n\mid x_n\right)=\nabla h_{\rho }\left(x_n\right)=\nabla f\left(x_n\right)+\rho D^t\left[D{x}_n-𝒫\left(D{x}_n\right)\right].$$

The indicated matrix inverse is just k × k.

2.3. ADMM

ADMM (alternating direction method of multipliers) is a natural competitor to the proximal distance algorithms just described (Hong et al., 2016). ADMM is designed to minimize functions of the form f(x) + g(Dx) subject to x ∈ C, where C is closed and convex. Splitting variables leads to the revised objective f(x) + g(y) subject to x ∈ C and y = Dx. ADMM invokes the augmented Lagrangian

$${𝒜}_{\mu }(x,y,\lambda )=f(x)+g(y)+{\lambda }^t(Dx-y)+\frac{\mu }{2}\Vert Dx-y{\Vert }^2,$$

with Lagrange multiplier λ and step length μ > 0. At iteration n + 1 of ADMM one calculates successively

$$x_{n+1}={\text{argmin}}_{x\in C}\left[f(x)+\frac{\mu }{2}{\Vert Dx-y_n+{\lambda }_n\Vert }^2\right],$$ (4)

$$y_{n+1}={\text{argmin}}_y\left[g(y)+\frac{\mu }{2}{\Vert D{x}_{n+1}-y+{\lambda }_n\Vert }^2\right],\text{and}$$ (5)

$${\lambda }_{n+1}={\lambda }_n+\mu \left(D{x}_{n+1}-y_{n+1}\right).$$ (6)

Update (4) succumbs to Newton’s method when f(x) is smooth and $C={ℝ}^p$, and update (5) reduces to a proximal map of g(y). Update (6) of the Lagrange multiplier λ amounts to steepest ascent on the dual function. A standard extension to the scheme in (4) through (6) is to vary the step length μ by considering the magnitude of residuals (Boyd et al., 2011). For example, letting r_n = Dx − y and s_n = μD^t(y_n−1 − y_n) denote primal and dual residuals at iteration n, we make use of the heuristic

$${\mu }_{n+1}=\{\begin{array}{ll}2{\mu }_n,\hfill & \text{if}\Vert r_n\Vert /\Vert s_n\Vert >10\hfill \\ {\mu }_n/2,\hfill & \text{if}\Vert r_n\Vert /\Vert s_n\Vert <10\hfill \\ {\mu }_n,\hfill & \text{otherwise}\hfill \end{array},$$

which (a) keeps the primal and dual residuals within an order of magnitude of each other, (b) makes ADMM less sensitive to the choice of step length, and (c) improves convergence.

Our problem conforms to the ADMM paradigm when S is equal to the Cartesian product ${\prod }_{i=1}^r{S}_i$ and $g(y)=\frac{\rho }{2}\text{dist}{(y,S)}^2$. Fortunately, the y update (5) reduces to a simple formula (Bauschke and Combettes, 2017). To derive this formula, note that the proximal map y = prox_αg(z) satisfies the stationary condition

$$0=y-z+\alpha [y-𝒫(y)],$$

for any z, including z = Dx_n+1 + λ_n, and any α, including α = ρ/μ. Since the projection map 𝒫(y) has the constant value 𝒫(z) on the line segment [z, 𝒫(z)], the value

$${\text{prox}}_{\alpha g}(z)=\frac{\alpha }{1+\alpha }𝒫(z)+\frac{1}{1+\alpha }z,$$

satisfies the stationary condition. Because the explicit update (5) for y decreases the Lagrangian even when S is nonconvex, we will employ it generally.

The x update (4) is given by the proximal map ${\text{prox}}_{{\mu }^{-1}f}\left({\lambda }_n-y_n\right)$ when $S={ℝ}^p$ and D = I. Otherwise, the update of x is more problematic. Assuming f(x) is smooth and $S={ℝ}^p$, Newton’s method gives the approximate update

$$x_{n+1}=x_n-{\left[d^{2}f\left(x_n\right)+\mu D^{t}D\right]}^{-1}\left[\nabla f\left(x_n\right)+\mu D^t\left(D{x}_n-y_n+{\lambda }_n\right)\right].$$

Our earlier suggestion of replacing d²f(x_n) by a positive definite approximation also applies here. Let us emphasize that ADMM eliminates the need for distance majorization. Although distance majorization is convenient, it is not necessarily a tight majorization. Thus, one can hope to see gains in rates of convergence. Balanced against this positive is the fact that ADMM is often slow to converge to high accuracy.

2.4. Proximal Distance Iteration

We conclude this section by describing proximal distance algorithms in pseudocode. As our theoretical results will soon illustrate, the choice of penalty parameter ρ is tied to the convergence rate of any proximal distance algorithm. Unfortunately, a large value for ρ is necessary for the iterates to converge to the constraint set S. We ameliorate this issue by slowly sending ρ → ∞ according to annealing schedules from the family of geometric progressions ρ(t) = r^t−1 with t ≥ 1. Here we parameterize the family by an initial value ρ = 1 and a multiplier r > 1. Thus, our methods approximate solutions to min f(x) subject to Dx ∈ S by solving a sequence of increasingly penalized subproblems, min _xh_ρ(t)(x). Algorithm 1 gives a high-level sketch of proximal distance iteration.

In practice we can only solve a finite number of subproblems so we prescribe the following convergence criteria

$$\Vert \nabla h_{\rho (t)}\left(x_n\right)\Vert \le {\delta }_h,$$ (7)

$$\text{dist}\left(D{x}_t,S\right)\le {\delta }_d,\text{or}$$ (8)

$$\left|\text{dist}\left(D{x}_t,S\right)-\text{dist}\left(D{x}_{t-1},S\right)\right|\le {\delta }_q\left[\text{dist}\left(D{x}_{t-1},S\right)+1\right].$$ (9)

Condition (7) is a guarantee that a solution estimate x_n is close to a stationary point after n inner iterations for the fixed value of ρ = ρ(t). In conditions (8) and (9), the vector x_t denotes the δ_h-optimal solution estimate once condition (7) is satisfied for a particular subproblem along the annealing path. Condition (8) is a guarantee that solutions along the annealing path adhere to the fusion constraints at level δ_d. In general, condition (8) can only be satisfied for large values of ρ(t). Finally, condition (9) is used to terminate the annealing process if the relative progress made in decreasing the distance penalty becomes too small as measured by δ_q. Algorithm 2 describes the flow of proximal distance iteration in practice. Nesterov acceleration in inner iterations is a key ingredient. Warm starts in solving subsequent subproblems are implicit in our formulation.

3. Convergence Analysis: Convex Case

In this section we summarize convergence results for proximal distance algorithms on convex problems. Proofs of all propositions appear in Section 7. We begin our discussion by recalling the definition of the next iterate

$$x_{n+1}\in \text{argmin}{g}_{\rho }\left(x\mid x_n\right)\equiv \underset{z_n\in 𝒫\left(D{x}_n\right)}{\cup }\underset{x}{\text{argmin}}\left[f(x)+\frac{\rho }{2}{\Vert Dx-z_n\Vert }^2\right],$$ (10)

and the descent property

$$h_{\rho }\left(x_{n+1}\right)\stackrel{\text{domination}}{\le }{g}_{\rho }\left(x_{n+1}\mid x_n\right)\stackrel{\text{definition}}{\le }{g}_{\rho }\left(x_n\mid x_n\right)\stackrel{\text{tangency}}{=}{h}_{\rho }\left(x_n\right),$$

enjoyed by all MM algorithms. As noted earlier, we can assume a single fusion matrix D and a single closed convex constraint set S. With these ideas firmly in mind, we state a sufficient condition for the existence of a minimum point of the penalized loss h_ρ(x). Further constraints on x beyond those imposed in the distance penalties are rolled into the essential domain of the convex loss f(x). This is particularly beneficial for a quadratic loss with an affine constraint (Keys et al., 2019; Lange, 2016).

Proposition 3.1 Suppose the convex function f(x) on ${ℝ}^p$ possesses a unique minimum point y on the closed convex set T = D⁻¹(S). Then for all sufficiently large ρ, the objective $h_{\rho }(x)=f(x)+\frac{\rho }{2}\text{dist}{(Dx,S)}^2$ is coercive and therefore attains its minimum value.

Next we show that the surrogate function g_ρ(x | x_n+1) defined in equation (3) attains its minimum value for large enough ρ. This ensures that the algorithm map (10) is well-defined.

Proposition 3.2 Under the conditions of Proposition 3.1, for sufficiently large ρ, every surrogate $g_{\rho }\left(x\mid x_n\right)=f(x)+\frac{\rho }{2}{\Vert Dx-𝒫\left(D{x}_n\right)\Vert }^2$ is coercive and therefore attains its minimum value. If

$$f(x)\ge f\left(x_n\right)+v_n^t\left(x-x_n\right)+\frac{1}{2}{\left(x-x_n\right)}^{t}A\left(x-x_n\right),$$

for all x and some positive semidefinite matrix A and subgradient v_n at x_n, and if the inequality u^tAu > 0 holds whenever ∥Du∥ = 0 and u ≠ 0, then for ρ sufficiently large, g_ρ(x | x_n) is strongly convex and hence coercive.

As an illustration of Proposition 3.2, suppose that D is the forward difference operator with unit spacing. By virtue of our convexity assumption, it suffices to prove that the surrogate g_ρ(x | x_n+1) is coercive along all rays emanating from the origin. The only vectors with Dv = 0 are multiples of 1. Thus, we only need the map t ↦ f(t1) to tend to ∞ as |t| does. This is much weaker condition than the strong convexity of f(x). In other words f(x) must compensate for the penalty where the penalty is not coercive. Uniqueness of x_n+1 holds whenever g_ρ(x | x_n) is strictly convex regardless of whether the conditions imposed by the proposition are true.

In our ideal convex setting we have a first convergence result for fixed ρ.

Proposition 3.3 Supposes (a) that S is closed and convex, (b) that the loss f(x) is convex and differentiable, and (c) that the constrained problem possesses a unique minimum point. For ρ sufficiently large, let z_ρ denote a minimal point of the objective h_ρ(x) defined by equation (2). Then the MM iterates (10) satisfy

$$0\le h_{\rho }\left(x_n\right)-h_{\rho }\left(z_{\rho }\right)\le \frac{\rho }{2(n+1)}{\Vert D\left(z_{\rho }-x_0\right)\Vert }^2.$$

Furthermore, the iterate values h_ρ(x_n) systematically decrease.

In even more restricted circumstances, one can prove linear convergence of function values in the framework of Karimi et al. (2016). Specifically, our result hinges on deriving a Polyak-Łojasiewicz inequality for (sub)gradients of h_ρ(x), from which linear convergence follows almost immediately.

Proposition 3.4 Suppose that S is a closed and convex set and that the loss f(x) is L-smooth and μ-strongly convex. Then the objective $h_{\rho }(x)=f(x)+\frac{\rho }{2}\text{dist}{(Dx,S)}^2$ possesses a unique minimum point z_ρ, and the proximal distance iterates x_n satisfy

$$0\le h_{\rho }\left(x_n\right)-h_{\rho }\left(z_{\rho }\right)\le {\left[1-\frac{{\mu }^2}{2{\left(L+\rho \Vert D{\Vert }^2\right)}^2}\right]}^n\left[h_{\rho }\left(x_0\right)-h_{\rho }\left(z_{\rho }\right)\right].$$

Convergence of ADMM is well studied in the optimization literature (Beck, 2017). Appendix B summarizes the main findings as they apply to the ADMM algorithms of Section 2.3.

4. Convergence Analysis: General Case

We now depart the comfortable confines of convexity. Our analysis relies on the Fréchet subdifferential (Kruger, 2003) and the theory of semialgebraic sets and functions. Readers unfamiliar with these topics are encouraged to read Appendix A for a brief review of the relevant theory.

The presentation of our results relies on the prior chain of reasoning established by Keys et al. (2019). Specifically, our arguments invoke Zangwill’s global convergence theorem for algorithm maps (Luenberger et al., 1984, Chapter 7, Section 7). The key ingredients of the theory are (i) a solution set (for instance a set of stationary points), (ii) an algorithm map that is closed outside the solution set, (iii) a compact set containing the iterates x_n generated by the map, and (iv) a Lyapunov function that decreases along the iterates. Keys et al. (2019) set the stage in their Propositions 5 through 8 by showing that proximal distance algorithms with D = I satisfy the restrictions (i)-(iv) imposed by Zangwill’s global convergence theorem. Note that our algorithm maps inherit their multivalent nature from multivalent Euclidean projections onto nonconvex sets. In any event, the main hurdles to overcome in proving convergence with fixed ρ > 0 are

1. establishing coercivity of h_ρ(x),

2. demonstrating strong convexity of g_ρ(x | x_n), and

3. showing h_ρ(x) satisfies a Polyak-Łojasiewicz inequality.

In the present case with D ≠ I, the coercivity assumption (a) is tied to requirement (iii) in Zangwill’s theorem as stated above. The strong convexity assumption (b) is fortunately imposed on the surrogate rather than the objective. It is also crucial that the Euclidean distance dist(x, S) to a semialgebraic set S is a semialgebraic function. In view of the closure properties of such functions, the function $\frac{\rho }{2}\text{dist}{(Dx,S)}^2$ is also semialgebraic. Semialgebraic theory is quite general, and many common set constraints fall within its purview. For example, the nonnegative orthant ${ℝ}_{+}^p$ and the unit sphere 𝒮^p−1 are semialgebraic.

The next proposition considers the convergence of the iterates of a generic MM algorithm to a stationary point. Readers seeking a proof may consult Section A.10 in the appendix of Keys et al. (2019).

Proposition 4.1 In an MM algorithm suppose the objective h(x) is coercive, continuous, and subanalytic and all surrogates g(x | x_n) are continuous, μ-strongly convex, and satisfy the Lipschitz condition

$$\Vert \nabla g\left(a\mid x_n\right)-\nabla g\left(b\mid x_n\right)\Vert ,\le L\Vert a-b\Vert$$

on the compact set {x : h(x) ≤ h(x₀)}. Then the MM iterates x_n+1 = argmin_x g(x | x_n) converge to a stationary point.

This result builds on theoretical contributions extending Kurdyka’s, Łojasiewicz’s, and Polyak’s inequalities to nonsmooth analysis, the generic setting of semialgebraic sets and functions, and proximal algorithms (Bolte et al., 2007; Attouch et al., 2010; Kang et al., 2015; Cui et al., 2018; Le Thi et al., 2018). Note that the stationary point may represent a local minimum or even a saddle point rather than a global minimum.

Before stating a precise result for our proposed proximal distance algorithms, let us clarify the nature of the Fréchet subdifferential in the current setting. This entity is determined by the identity

$$\frac{1}{2}\text{dist}{(Dx,S)}^2=\frac{1}{2}\underset{z\in S}{\text{min}}\Vert Dx-z{\Vert }^2,$$

for which Danskin’s theorem yields the directional derivative

$$d_v\frac{1}{2}\text{dist}{(Dx,S)}^2=\underset{z\in S(x)}{\text{min}}{(Dx-z)}^{t}Dv.$$

Here S(x) is the solution set where the minimum is attained. The Fréchet differential

$$\begin{array}{l}{\partial }^F{h}_{\rho }(x)=\nabla f(x)+\rho \left\{u:{(Dx-z)}^{t}Dv\ge u^{t}v,z\in S(x)\text{and}\text{all}v\right\}\\ =\nabla f(x)+\rho \left\{u:u=D^t(Dx-z),z\in S(x)\right\},\end{array}$$

holds owing to Corollary 1.12.2 and Proposition 1.17 of Kruger (2003) since dist(Dx, S)² is locally Lipschitz. The latter fact follows from the identity a² − b² = (a + b)(a − b) with a = dist(Dy, S)² and b = dist(Dx, S)², given that dist(w, S) is Lipschitz and bounded on bounded sets.

In any event, a stationary point x satisfies 0 = ∇f(x) + ρD^t(Dx − z) for all z ∈ S(x). As we expect, the stationary condition is necessary for x to furnish a global minimum. Indeed, if it fails, take z ∈ S(x) with surrogate satisfying ∇g_ρ(x | x) ≠ 0. Then the negative gradient −∇g_ρ(x | x) is a descent direction for g_ρ(x | x), which majorizes h_ρ(x). Hence, −∇g_ρ(x | x) is also a descent direction for h_ρ(x). This conclusion is inconsistent with x being a local minimum of the objective.

Having clarified Proposition 4.1 in our context, we state a convergence result on h_ρ(x).

Proposition 4.2 Suppose in our proximal distance setting that ρ is sufficiently large, the closed constraint sets S_i and the loss f(x) are semialgebraic, and f(x) is differentiable with a locally Lipschitz gradient. Under the coercive assumption made in Proposition 3.2, the proximal distance iterates x_n converge to a stationary point of the objective h_ρ(x).

The coercivity assumption requires that h_ρ(x) be coercive for sufficiently large ρ. This is not as restrictive as it sounds. If S is compact and f(x) is convex or bounded below, then the primary hindrances are the directions v where Dv = 0. Unless D is the trivial matrix 0, this null space has Lebesgue measure 0.

We conclude this section by communicating two results involving the set S_k ⊂ R^p whose members have at most k nonzero components. The sparsity constraint defining S_k is usually expressed as ∥x∥₀ ≤ k for all feasible vectors x. Projection of x onto S_k preserves the top k components of x in absolute value but sends all other components to 0. Sparsity sets can be extended to matrix-valued variables X with the sparsity constraint applying to the entries, rows, or columns. Projection in the second case ranks rows by their norms and replaces the lowest ranked rows by the 0 vector. The next proposition establishes that S_k is semialgebraic in a general setting.

Proposition 4.3 The order statistics of a finite set ${\left\{f_i(x)\right\}}_{i=1}^n$ of semialgebraic functions are semialgebraic. Hence, sparsity sets are semialgebraic.

For sparsity constrained problems, one can establish a linear rate of convergence under the right hypotheses. The next proposition proves convergence for a wide class of fused models.

Proposition 4.4 Suppose in our proximal distance setting that ρ is sufficiently large, the constraint set is a sparsity set S_k, and the loss f(x) is semialgebraic, strongly convex, and has a Lipschitz gradient. Then the proximal distance iterates x_n converge to a stationary point x_∞. Convergence occurs at a linear rate provided Dx_∞ has k unambiguous largest components in magnitude. When the rows of D are unique, the complementary set of points x where Dx has k ambiguous largest components in magnitude has Lebesgue measure 0.

5. Numerical Examples

This section considers five concrete examples of constrained optimization amenable to distance majorization with fusion constraints, with D denoting the fusion matrix in each problem. In each case, the loss function is both strongly convex and differentiable. The specific examples that we consider are the metric projection problem, convex regression, convex clustering, image denoising with a total variation penalty, and projection of a matrix to one with a better condition number. Each example is notable for the large number of fusion constraints and projections to convex constraint sets, except in convex clustering. In convex clustering we encounter a sparsity constraint set. Quadratic loss models feature prominently in our examples. Interested readers may consult our previous work for nonconvex examples with D = I (Keys et al., 2019; Xu et al., 2017).

5.1. Mathematical Descriptions

Here we provide the mathematical details for each example.

5.1.1. Metric Projection

Solutions to the metric projection problem restore the triangle inequality to noisy distance data represented as m nodes of a graph (Brickell et al., 2008; Sra et al., 2005). Specifically, data are encoded in an m × m dissimilarity matrix Y = (y_ij) with nonnegative weights in the matrix W = (w_ij). Metric projection requires finding the symmetric semi-metric X = (x_ij) minimizing

$$f(X)=\sum _{i>j}{w}_{ij}{\left(x_{ij}-y_{ij}\right)}^2,$$

subject to all $\left(\begin{array}{c}m\\ 2\end{array}\right)$ nonnegativity constraints x_ij ≥ 0 and all $3\left(\begin{array}{c}m\\ 3\end{array}\right)$ triangle inequality constraints x_ij − x_ik − x_kj ≤ 0. The diagonal entries of Y, W, and X are zero by definition. The fusion matrix D has $\left(\begin{array}{c}m\\ 2\end{array}\right)+3\left(\begin{array}{c}m\\ 3\end{array}\right)$ rows, and the projected value of DX must fall in the set S of symmetric matrices satisfying all pertinent constraints.

One can simplify the required projection by stacking the nonredundant entries along each successive column of X to create a vector x with $\left(\begin{array}{c}m\\ 2\end{array}\right)$ entries. This captures the lower triangle of X. The sparse matrix D is correspondingly redefined to be $\left[\left(\begin{array}{c}m\\ 2\end{array}\right)+3\left(\begin{array}{c}m\\ 3\end{array}\right)\right]\times \left(\begin{array}{c}m\\ 2\end{array}\right)$. These maneuvers simplify constraints to Dx ≥ 0, and projection involves sending each entry u of Dx to max{0, u}. Putting everything together, the objective to minimize is

$$h_{\rho }(x)=\frac{1}{2}{\Vert W^{1/2}(x-y)\Vert }_2^2+\frac{\rho }{2}\text{dist}{\left(Tx,{ℝ}_{+}^{p_1}\right)}^2+\frac{\rho }{2}\text{dist}{\left(x,{ℝ}_{+}^{p_2}\right)}^2,$$

where D consists of blocks T and I_p2 and p₁ and p₂ count the number of triangle inequality and nonnegativity constraints, respectively. The linear system (I + ρD^tD)x = b appears in both the MM and ADMM updates for x_n. Application of the Woodbury and Sherman-Morrison formulas yield an exact solution to the linear system and allow one to forgo iterative methods. The interested reader may consult Appendix C for further details.

5.1.2. Convex Regression

Convex regression is a nonparametric method for estimating a regression function under shape constraints. Given m responses y_i and corresponding predictors $x_i\in {ℝ}^d$, the goal is to find the convex function ψ(x) minimizing the sum of squares $\frac{1}{2}{\sum }_{i=1}^m{\left[y_i-\psi \left(x_i\right)\right]}^2$. Asymptotic and finite sample properties of this convex estimator have been described in detail by Seijo and Sen (2011). In practice, a convex regression program can be restated as a finite dimensional problem of finding the value θ_i and subgradient ${\xi }_i\in {ℝ}^d$ of ψ(x) at each sample point (y_i, x_i). Convexity imposes the supporting hyperplane constraint ${\theta }_j+{\xi }_j^t\left(x_i-x_j\right)\le {\theta }_i$ for each pair i ≠ j. Thus, the problem becomes one of minimizing $\frac{1}{2}\Vert y-\theta {\Vert }^2$ subject to these m(m − 1) inequality constraints. In the proximal distance framework, we must minimize

$$h_{\rho }(\theta ,\Xi )=\frac{1}{2}\Vert y-\theta {\Vert }^2+\frac{\rho }{2}\text{dist}{\left(A\theta +B\mathbf{\Xi },{ℝ}_{-}^{m(m-1)}\right)}^2,$$

where D = [A B] encodes the required fusion matrix. The reader may consult Appendix D for a description of each algorithm map.

5.1.3. Convex Clustering

Convex clustering of m samples based on d features can be formulated in terms of the regularized objective

$$F_{\gamma }(U)=\frac{1}{2}\sum _{i=1}^m{\Vert u_i-x_i\Vert }^2+\gamma \sum _{i>j}{w}_{ij}\Vert u_i-u_j\Vert ,$$

based on columns x_i and u_i of $X\in {ℝ}^{d\times m}$ and $U\in {ℝ}^{d\times m}$, respectively. Here each $x_i\in {ℝ}^d$ is a sample feature vector and the corresponding u_i represents its centroid assignment. The predetermined weights w_ij have a graphical interpretation under which similar samples have positive edge weights w_ij and distant samples have 0 edge weights. The edge weights are chosen by the user to guide the clustering process. In general, minimization of F_γ(U) separates over the connected components of the graph. To allow all sample points to coalesce into a single cluster, we assume that the underlying graph is connected. The regularization parameter γ > 0 tunes the number of clusters in a nonlinear fashion and potentially captures hierarchical information. Previous work establishes that the solution path U(γ) varies continuously with respect to γ (Chi and Lange, 2015). Unfortunately, there is no explicit way to determine the number of clusters entailed by a particular value of γ prior to fitting U(γ).

Alternatively, we can attack the problem using sparsity and distance majorization. Consider the penalized objective

$$h_{\rho ,k}(U)=\frac{1}{2}\Vert U-X{\Vert }_F^2+\frac{\rho }{2}\text{dist}{\left(UD,S_k\right)}^2.$$

The fusion matrix D has $\left(\begin{array}{c}m\\ 2\end{array}\right)$ columns w_ij(e_i − e_j) and serves to map the centroid matrix U to a $d\times \left(\begin{array}{c}m\\ 2\end{array}\right)$ matrix V encoding the weighted differences w_ij(u_i − u_j). The members of the sparsity set S_k are $d\times \left(\begin{array}{c}m\\ 2\end{array}\right)$ matrices with at most k non-zero columns. Projection of UD onto the closed set S_k forces some centroid assignments to coalesce, and is straightforward to implement by sorting the Euclidean lengths of the columns of UD and sending to 0 all but the k most dominant columns. Ties are broken arbitrarily.

Our sparsity-based method trades the continuous penalty parameter γ > 0 in the previous formulation for an integer sparsity index $k\in \left\{0,1,2,\dots ,\left(\begin{array}{c}m\\ 2\end{array}\right)\right\}$. For example with k = 0, all differences u_i − u_j are coerced to 0, and all sample points cluster together. The other extreme $k=\left(\begin{array}{c}m\\ 2\end{array}\right)$ assigns each point to its own cluster. The size of the matrices D and UD can be reduced by discarding column pairs corresponding to 0 weights. Appendix E describes the projection onto sparsity sets and provides further details.

5.1.4. Total Variation Image Denoising

To approximate an image U from a noisy input W matrix, Rudin et al. (1992) regularize a loss function f(U) by a total variation (TV) penalty. After discretizing the problem, the least squares loss leads to the objective

$$F_{\gamma }(U)=\sum _{i,j}{\left(U_{i,j}-W_{i,j}\right)}^2+\gamma \sum _{i,j}\sqrt{{\left(U_{i+1,j}-U_{i,j}\right)}^2+{\left(U_{i,j+1}-U_{i,j}\right)}^2},$$

where U, $W\in {ℝ}^{m\times p}$ are rectangular monochromatic images and γ controls the strength of regularization. The anisotropic norm

$${\text{TV}}_1(U)=\sum _{i,j}\left|U_{i+1,j}-U_{i,j}\right|+\left|U_{i,j+1}-U_{i,j}\right|={\Vert D_{m}U\Vert }_1+{\Vert U{D}_p^t\Vert }_1,$$

is often preferred because it induces sparsity in the differences. Here D_p is a forward difference operator on p data points. Stacking the columns of U into a vector u = vec(U) allows one to identify a fusion matrix D and write TV₁(U) compactly as TV₁(u) = ∥Du∥₁. We append a row with a single 1 in the last component to make D full rank. In this context we reformulate the denoising problem as minimizing f(u) subject to the set constraint ∥Du∥₁ ≤ γ. This revised formulation directly quantifies the quality of a solution in terms of its total variation and brings into play fast pivot-based algorithms for projecting onto multiples of the ℓ₁ unit ball (Condat, 2016). Appendix F provides descriptions of each algorithm.

5.1.5. Projection of a Matrix to a Good Condition Number

Consider an m × p matrix M with m ≥ p and full singular value decomposition M = UΣV^t. The condition number of M is the ratio σ_max/σ_min of the largest to the smallest singular value of M. We denote the diagonal of Σ as σ. Owing to the von Neumann-Fan inequality, the closest matrix N to M in the Frobenius norm has the singular value decomposition N = UXV ^t, where the diagonal x of X satisfies inequalities pertinent to a decent condition number (Borwein and Lewis, 2010). Suppose c ≥ 1 is the maximum condition number. Then every pair (x_i, x_j) satisfies x_i − cx_j ≤ 0. Note that x_i − cx_i > 0 if and only if x_i < 0. Thus, nonnegativity of the entries of x is enforced. The proximal distance approach to the condition number projection problem invokes the objective and majorization

$$\begin{array}{l}{h}_{\rho }(x)=\frac{1}{2}\Vert \sigma -x{\Vert }^2+\frac{\rho }{2}\sum _{(i,j)}\text{dist}{\left(x_i-c{x}_j,ℝ\_\right)}^2\\ =\frac{1}{2}\Vert \sigma -x{\Vert }^2+\frac{\rho }{2}\sum _{(i,j)}{\left(x_i-c{x}_j\right)}_{+}^2\\ \le \frac{1}{2}\Vert \sigma -x{\Vert }^2+\frac{\rho }{2}\sum _{(i,j)}{\left(x_i-c{x}_j-q_{nij}\right)}^2,\end{array}$$

at iteration n, where q_nij = min{x_ni − cx_nj,0}. We can write the majorization more concisely as

$$h_{\rho }(x)\le \frac{1}{2}{\Vert A_{\rho }x-r_n\Vert }^2,\hspace{1em}{A}_{\rho }=\left(\begin{array}{c}{I}_p\\ \sqrt{\rho }D\end{array}\right),\hspace{1em}{r}_n=\left(\begin{array}{c}\sigma \\ \sqrt{\rho }\text{vec}{Q}_n\end{array}\right),$$

where vecQ_n stacks the columns of Q_n = (q_nij) and the p² × p fusion matrix D satisfies (Dx)_k = x_i − cx_j for each component k. The minimum of the surrogate occurs at the point $x_{n+1}={\left(A_{\rho }^t{A}_{\rho }\right)}^{-1}{A}_{\rho }^t{r}_n$. This linear system can be solved exactly. Appendix G provides additional details.

5.2. Numerical Results

Our numerical experiments compare various strategies for implementing Algorithm 2. We consider two variants of proximal distance algorithms. The first directly minimizes the majorizing surrogate (MM), while the second performs steepest descent (SD) on it. In addition to the aforementioned methods, we tried the subspace MM algorithm described in Section 2.2. Unfortunately, this method was outperformed in both time and accuracy comparisons by Nesterov accelerated MM; the MM subspace results are therefore omitted. We also compare our proximal distance approach to ADMM as described in Section 2.3. In many cases updates require solving a large linear system; we found that the method of conjugate gradients sacrificed little accuracy and largely outperformed LSQR and therefore omit comparisons. The clustering and denoising examples are exceptional in that the associated matrices D^tD are sufficiently ill-conditioned to cause failures in conjugate gradients. Table 1 summarizes choices in control parameters across each example.

Table 1:

Summary of control parameters used in each example.

	δ h	δ d	δ q	ρ(t) = rt
Metric Projection	10−3	10−2	10−6	min{108, 1.2t−1}
Convex Regression	10−3	10−2	10−6	min{108, 1.2t−1}
Convex Clustering	10−2	10−5	10−6	min{108, 1.2t−1}
Image Denoising	10−1	10−1	10−6	min{108, 1.5t−1}
Condition Numbers	10−3	10−2	10−6	min{108, 1.2t−1}

We now explain example by example the implementation details behind our efforts to benchmark the three strategies (MM, SD, and ADMM) in implementing Algorithm 2. In each case we initialize the algorithm with the solution of the corresponding unconstrained problem. Performance is assessed in terms of speed in seconds or milliseconds, number of iterations until convergence, the converged value of the loss f(x), and the converged distance to the constraint set dist(Dx, S), as described in Algorithm 2. Additional metrics are highlighted where applicable. The term inner iterations refers to the number of iterations to solve a penalized subproblem argmin h_ρ(x) for a given ρ whereas outer iterations count the total number of subproblems solved. Lastly, we remind readers that the approximate solution to argmin h_ρ(t)(x) is used as a warm start in solving argmin h_ρ(t+1)(x).

5.2.1. Metric Projection.

In our comparisons, we use input matrices $Y\in {ℝ}^{m\times m}$ whose iid entries y_ij are drawn uniformly from the interval [0, 10] and set weights w_ij = 1. Each algorithm is allotted a maximum of 200 outer and 10⁵ inner iterations, respectively, to achieve a gradient norm of δ_h = 10⁻³ and distance to feasibility of δ_d = 10⁻². The relative change parameter is set to δ_q = 0 and the annealing schedule is set to ρ(t) = min{10⁸, 1.2^t−1} for the proximal distance methods. Table 2 summarizes the performance of the three algorithms. Best values appear in boldface. All three algorithms converge to a similar solution as indicated by the final loss ∥y − x∥² and distance values. It is clear that SD matches or outperforms MM and ADMM approaches on this example. Notably, the linear system appearing in the MM update admits an exact solution (see Appendix C.5), yet SD has a faster time to solution with fewer iterations taken. The selected convergence metrics in Figure 1 vividly illustrate stability of solutions x_ρ = argmin h_ρ(x) along an annealing path from ρ = 1 to ρ = 1.2⁴⁰ ≈ 1470. Specifically, solving each penalized subproblem along the sequence results in marginal increase in the loss term with appreciable decrease in the distance penalty. Except for the first outer iteration, there is minimal decrease of the loss, distance penalty, or penalized objective within a given outer iteration even as the gradient norm vanishes. The observed tradeoff between minimizing a loss model and minimizing a nonnegative penalty is well-known in penalized optimization literature (Beltrami, 1970; Lange, 2016, see Proposition 7.6.1 on p. 183).

Table 2:

Metric projection. Times are averaged over 3 replicates with standard deviations in parentheses. Reported iteration counts reflect the total inner iterations taken with outer iterations in parentheses.

	Time (s)			Loss ×10−3			Distance ×103			Iterations
m	MM	SD	ADMM	MM	SD	ADMM	MM	SD	ADMM	MM	SD	ADMM
16	0.0429 (0:000709)	0.0338 (0:000137)	0.107 (0.000858)	0.237	0.237	0.237	9.24	9.24	9.24	4980 (37)	3920 (37)	7030 (37)
32	1.24 (0.00309)	1:28 (0.00617)	2.6 (0.0149)	1.14	1.14	1.14	9.13	9.13	9.13	16000 (41)	15400 (41)	17300 (41)
64	19.5 (0.00835)	16.7 (0.0148)	43.9 (0.0616)	4.69	4.69	4.69	8.7	8.7	8.7	30100 (44)	24200 (44)	33700 (44)
128	171 (0.613)	150 (0.28)	725 (0.772)	18.4	18.4	18.4	9.44	9.44	9.44	29900 (44)	23700 (44)	51900 (44)
256	1670 (0.702)	1570 (5.63)	9110 (75.4)	75.3	75.3	75.3	8.68	8.68	8.68	32500 (46)	26700 (46)	76100 (46)

Figure 1:

Loss, distance, penalized objective, and gradient norm for SD on metric projection problem with 32 nodes, labeled by outer iteration.

5.2.2. Convex Regression.

In our numerical examples the observed functional values y_i are independent Gaussian deviates with means ψ(x_i) and common variance σ² = 0.1. The predictors are iid deviates sampled from the uniform distribution on [−1, 1]^d. We choose the simple convex function ψ(x_i) = ∥x_i∥² for our benchmarks for ease in interpretation; the interested reader may consult the work of Mazumder et al. (2019) for a detailed account of the applicability of the technique in general. Each algorithm is allotted a maximum of 200 outer and 10⁴ inner iterations, respectively, to converge with δ_h = 10⁻³, δ_d = 10⁻², and δ_q = 10⁻⁶. The annealing schedule is set to ρ(t) = min{10⁸, 1.2^t−1}.

Table 3 demonstrates that although the SD approach is appreciably faster than both MM and ADMM, the latter appear to converge on solutions with marginal improvements in minimizing the loss ∥y − θ∥², distance, and mean squared error (MSE) measured using ground truth functional values ψ(x_i) and estimates θ_i. Interestingly, increasing both the number of features and samples does not necessarily increase the amount of required computational time in using a proximal distance approach; for example, see results with d = 2 and d = 20 features. This may be explained by sensitivity to the annealing schedule.

Table 3:

Convex regression. Times are reported as averages over 3 replicates.

		Time (s)			Loss ×103			Distance ×104			MSE ×102
d	m	MM	SD	ADMM	MM	SD	ADMM	MM	SD	ADMM	MM	SD	ADMM
1	50	0.015 (0.00477)	0.0109 (0.00162)	0.0171 (0.0033)	454	454	454	87.3	87.4	87.5	70.1	70.1	70.1
	100	0.0382 (0.002)	0.0311 (0.000301)	0.0578 (0.00179)	510	510	510	94.3	94.4	94.5	75.3	75.3	75.3
	200	0.138 (0.00662)	0.0991 (0.00138)	0.205 (0.00213)	471	471	471	92.2	92.2	92.2	71	71	71
	400	0.565 (0.012)	0.464 (0.000493)	0.862 (0.0184)	501	501	501	96.9	97	97	79.1	79.1	79.1
2	50	1.71 (0.00334)	0.693 (0.00341)	16 (0.00511)	122	126	118	85.9	85.3	85.5	72.2	73.1	70.8
	100	11.8 (0.0189)	3.33 (0.00704)	66.6 (0.0844)	162	163	162	99.4	98	98.4	95.4	95.8	95.1
	200	51.4 (0.0701)	14.1 (0.00805)	230 (0.273)	233	234	233	98.1	94.2	96.8	123	123	122
	400	200 (1.06)	50.3 (0.299)	917 (1.48)	239	239	238	94.3	90.8	91.9	140	140	140
10	50	0.19 (0.00281)	0.00722 (9.99 × 10−5)	0.196 (0.00743)	0.000891	0.00109	0.000838	0.821	2.23	0.488	8.59	8.63	8.61
	100	0.854 (0.0002)	0.0644 (0.000503)	0.873 (0.00191)	0.000937	0.00097	0.000943	0.154	0	0.11	10.3	10.2	10.3
	200	3.77 (0.00486)	0.398 (0.00888)	3.89 (0.0132)	0.000883	0.00099	0.00101	0.281	0	0.292	9.64	9.63	9.65
	400	26.8 (0.405)	3.17 (0.0183)	27.6 (0.0488)	0.000992	0.000997	0.000999	0.185	0.288	0.176	9.41	9.42	9.39
20	50	0.39 (0.00132)	0.00791 (2.6 × 10−5)	0.399 (0.00193)	0.000991	0.00542	0.00091	0.027	7.06	0.0696	9.77	9.34	9.57
	100	1.46 (0.000164)	0.0684 (0.000174)	1.58 (0.0329)	0.000995	0.000965	0.000996	0.0308	0	0	9.13	9.22	9
	200	7.7 (0.166)	0.414 (0.00109)	7.78 (0.0105)	0.000984	0.00113	0.000961	0.251	1.74	0.438	9.6	9.61	9.6
	400	30.2 (0.0275)	3.03 (0.00605)	30 (0.0157)	0.000997	0.00105	0.00142	0.0921	1.68	0.646	10	10.2	10.2

5.2.3. Convex Clustering.

To evaluate the performance of the different methods on convex clustering, we consider a mixture of simulated data and discriminant analysis data from the UCI Machine Learning Repository (Dua and Graff, 2019). The simulated data in gaussian300 consists of 3 Gaussian clusters generated from bivariate normal distributions with means μ = (0.0, 0.0)^t, (2.0, 2.0)^t, and (1.8, 0.5)^t, standard deviation σ = 0.1, and class sizes n₁ = 150, n₂ = 50, n₃ = 100. This easy dataset is included to validate Algorithm 3 described later as a reasonable solution path heuristic. The data in iris and zoo are representative of clustering with purely continuous or purely discrete data, respectively. In these two datasets, samples with same class label form a cluster. Finally, the simulated data spiral500 is a classic example that thwarts k-means clustering. Each algorithm is allotted a maximum of 10⁴ inner iterations to solve a ρ-penalized subproblem at level δ_h = 10⁻². The annealing schedule is set to ρ(t) = min{10⁸, 1.2^t−1} over 100 outer iterations with δ_d = 10⁻⁵ and δq = 10−6.

Because the number of clusters is usually unknown, we implement the search heuristic outlined in Algorithm 3. The idea behind the heuristic is to gradually coerce clustering without exploring the full range of the hyperparameter k. As one decreases the number of admissible nonzero centroid differences k from k_max to 0, sparsity (1 − k/k_max) in the columns of UD increases to reflect coalescing centroid assignments. Thus, Algorithm 1 generates a list of candidate clusters that can be evaluated by various measures of similarity (Vinh et al., 2010). For example, the adjusted Rand index (ARI) provides a reasonable measure of the distance to the ground truth in our examples as it accounts for both the number of identified clusters and cluster assignments. We also report the related normed Mutual Information (NMI). The ARI takes values on [−1, 1] whereas NMI appears on a [0, 1] scale.

ADMM, as implemented here, is not remotely competitive on these examples given its extremely long compute times and failure to converge in some instances. These times are only exacerbated by the search heuristic and therefore omit ADMM from this example. The findings reported in Table 4 indicate the same accuracy for MM (using LSQR) and SD as measured by loss and distance to feasibility. Here we see that the combination of the proximal distance algorithms and the overall search heuristic (Algorithm 3) yields perfect clusters in the gaussian300 example on the basis of ARI and NMI. To its disadvantage, the search heuristic is greedy and generally requires tuning. Both MM and SD achieve similar clusters as indicated by ARI and NMI. Notably, SD generates candidate clusters faster than MM.

Table 4:

Convex clustering. Times reflect the total time spent generating candidate clusterings using Algorithm 3. Additional metrics correspond to the optimal clustering on the basis of maximal ARI. Time and clustering indices are averaged over 3 replicates with standard deviations reported in parentheses.

				Time (s)		Loss		Distance ×105		ARI		NMI
dataset	features	samples	classes	MM	SD	MM	SD	MM	SD	MM	SD	MM	SD
zoo	16	101	7	95.6 (0.268)	77.1 (2.02)	1600	1600	8.62	8.62	0.841 (0)	0.848 (0.0118)	0.853 (0)	0.856 (0.00256)
iris	4	150	3	76.4 (0.129)	62.8 (2)	596	596	8.8	8.8	0.575 (0)	0.575 (0)	0.734 (0)	0.734 (0)
gaussian300	2	300	3	190 (0.173)	155 (0.177)	598	598	8.98	8.98	1 (0)	1 (0)	1 (0)	1 (0)
spiral500	2	500	2	715 (18.5)	561 (0.501)	998	998	8.98	8.98	0.133 (0)	0.133 (0)	0.366 (0)	0.366 (0)

5.2.4. Total Variation Image Denoising.

To evaluate our denoising algorithm, we consider two standard test images, cameraman and peppers_gray. White noise with σ = 0.2 is applied to an image and then reconstructed using our proximal distance algorithms. Only MM and SD are tested with a maximum of 100 outer and 10⁴ inner iterations, respectively, and convergence thresholds δ_h = 10⁻¹, δ_d = 10⁻¹, and δ_q = 10⁻⁶. A moderate schedule ρ(t) = {10⁸, 1.5^t−1} performs well even with such lax convergence criteria. Table 5 reports convergence metrics and image quality indices, MSE and Peak Signal-to-Noise Ratio (PSNR). Timings reflect the total time spent generating solutions, starting from a 0% reduction in the total variation of the input image U up to 90% reduction in increments of 10%. Explicitly, we take γ₀ = TV₁(U) and vary the control parameter γ = (1 − s)γ₀ with s ∈ [0, 1] to control the strength of denoising. Figure 5.2.4 depicts the original and reconstructed images along the solution path.

Table 5:

Image denoising. Times reflect the total time spent generating candidate images, averaged over 3 replicates, ultimately achieving 90% reduction in total variation

			Time (s)		Loss		Distance ×103		MSE ×105		PSNR
image	width	height	MM	SD	MM	SD	MM	SD	MM	SD	MM	SD
cameraman	512	512	557 (5.01)	181 (0.906)	8090	8090	93.1	93.1	284	284	25.5	25.5
peppers_gray	512	512	553 (6.28)	183 (2.11)	8020	8020	92.6	92.5	290	290	25.4	25.4

Figure 2:

Sample images along the solution path of the search heuristic. Images are arranged from left to right as follows: reference, noisy input, and 90% reduction.

5.2.5. Projection of a Matrix to a Good Condition Number.

We generate base matrices $M\in {ℝ}^{p\times p}$ as random correlation matrices using MatrixDepot.jl (Zhang and Higham, 2016), which relies on Davies’ and Higham’s refinement (Davies and Higham, 2000) of the Bendel-Mickey algorithm (Bendel and Mickey, 1978). Simulations generate matrices with condition numbers c(M) in the set {119, 1920, 690}. Our subsequent analyses target condition number decreases by a factor a such a ∈ {2, 4, 16, 32}. Each algorithm is allotted a maximum of 200 outer and 10⁴ inner iterations, respectively with choices δ_h = 10⁻³, δ_d = 10⁻², δ_q = 10⁻⁶, and ρ(t) = 1.2^t−1. Table 6 summarizes the performance of the three algorithms. The quality of approximate solutions is similar across MM, SD, and ADMM in terms of loss, distance, and final condition number metrics. Interestingly, the MM approach requires less time to deliver solutions of comparable quality to SD solutions as the size p of the input matrix $M\in {ℝ}^{p\times p}$ increases.

Table 6:

Condition number experiments. Here c(M) is the condition number of the input matrix, a is the decrease factor, and c(X) is the condition number of the solution.

			Time (ms)			Loss ×103			Distance ×105			c(X)
p	c(M)	a	MM	SD	ADMM	MM	SD	ADMM	MM	SD	ADMM	MM	SD	ADMM
10	119	2	0.576 (0.104)	0.391 (0.00629)	0.582 (0.172)	0.549	0.548	0.549	41.1	41.1	40.6	59.4	59.4	59.4
		4	0.413 (0.00321)	0.359 (0.00129)	0.543 (0.101)	4.92	4.92	4.92	236	234	236	29.7	29.7	29.7
		16	0. 439 (0.00319)	0.424 (0.00427)	0.535 (0.00395)	130	130	130	926	929	927	7.44	7.44	7.44
		32	0.662 (0.00336)	0.496 (0.00357)	0.824 (0.00459)	821	821	821	984	983	983	3.72	3.72	3.72
100	1920	2	37.7 (2.89)	51.1 (3.77)	149 (0.972)	0.00119	0.00119	0.00119	0.208	0.208	0.204	960	960	960
		4	17.1 (0.164)	26.5 (2.8)	66.4 (0.485)	0.0107	0.0107	0.0107	0.845	0.845	0.877	480	480	480
		16	10.1 (0.301)	11. (0.0472)	35.3 (0.437)	0.436	0.436	0.436	17.8	17.9	17.9	120	120	120
		32	26 (2.23)	33.4 (0.49)	74 (0.893)	3.26	3.26	3.26	95.2	95.2	95.2	60	60	60
1000	59400	2	60.3 (0.83)	75.3 (1.35)	157 (3.98)	1.15 × 10−6	1.15 × 10−6	1.15 × 10−6	0	0	0	29700	29700	29600
		4	57.4 (3.37)	71.8 (0.882)	131 (8.44)	1.04 × 10−5	1.04 × 10−5	1.04 × 10−5	0	0	0	14800	14800	14800
		16	55.6 (3.3)	71.9 (2.05)	152 (38.8)	0.000258	0.000258	0.000261	0	0	0	3710	3710	3690
		32	23200 (855)	30000 (337)	87400 (1340)	0.0011	0.0011	0.00111	0.11	0.11	0	1860	1860	1850

6. Discussion

We now recapitulate the main findings of our numerical experiments. Tables 2 through 6 show a consistent pattern of superior speed by the steepest descent (SD) version of the proximal distance algorithm. This is hardly surprising since unlike ADMM and MM, SD avoids solving a linear system at each iteration. SD’s speed advantage tends to persist even when the linear system can be solved exactly. The condition number example summarized in Table 6 is an exception to this rule. Here the MM updates leverage a very simple matrix inverse. MM is usually faster than ADMM. We attribute MM’s performance edge to the extra matrix-vector multiplications involving the fusion matrix D required by ADMM. In fairness, ADMM closes the speed gap and matches MM on convex regression.

The choice of annealing schedule can strongly impact the quality of solutions. Intuitively, driving the gradient norm ∥∇h_ρ(x)∥ to nearly 0 for a given ρ keeps the proximal distance methods on the correct annealing path and yields better approximations. Provided the penalized objective is sufficiently smooth, one expects the solution x_ρ ≈ argmin h_ρ(x) to be close to the solution x_ρ′ ≈ argmin h_ρ′(x) when the ratio ρ′/ρ > 1 is not too large. Thus, choosing a conservative δ_h for the convergence criterion ∥∇h_ρ(x)∥ ≤ δ_h may guard against a poorly specified annealing schedule. Quantifying sensitivity of intermediate solutions x_ρ with respect to ρ is key in avoiding an increase in inner iterations per subproblem; for example, as observed in Figure 1. Given the success of our practical annealing recommendation to overcome the unfortunate coefficients in Propositions 3.3 and 3.4, this topic merits further consideration in future work.

In practice, it is sometimes unnecessary to impose strict convergence criteria on the proximal distance iterates. It is apparent that the convergence criteria on convex clustering and image denoising are quite lax compared to choices in other examples, specifically in terms of δ_h. Figure 1 suggests that most of the work in metric projection involves driving the distance penalty downhill rather than in fitting the loss. Surprisingly, Table 4 shows that our strict distance criterion δ_d = 10⁻⁵ in clustering is achieved. This implies dist(Dx, S)² ≤ 10⁻¹⁰ on the selected solutions with ρ ≤ 10⁸, yet we only required ∥∇h_ρ(x)∥ ≤ 10⁻² on each subproblem. Indeed, not every problem may benefit from precise solution estimates. The image processing example underscores this point as we are able to recover denoised images with the choices δ_h = δ_d = 10⁻¹. Problems where patterns or structure in solutions are of primary interest may stand to benefit from relaxed convergence criteria.

Our proximal distance method, as described in Algorithm 2, enjoys several advantages. First, fusion constraints fit naturally in the proximal distance framework. Second, proximal distances enjoy the descent property. Third, there is a nearly optimal step size for gradient descent when second-order information is available on the loss. Fourth, proximal distance algorithms are competitive if not superior to ADMM on many problems. Fifth, proximal distance algorithms like iterative hard thresholding rely on set projection and are therefore helpful in dealing with hard sparsity constraints. The main disadvantages of the proximal distance methods are (a) the overall slow convergence due to the loss of curvature information on the distance penalty and (b) the need for a reasonable annealing schedule. In practice, a little experimentation can yield a reasonable schedule for an entire class of problems. Many competing methods are only capable of dealing with soft constraints imposed by the lasso and other convex penalties. To their detriment, soft constraints often entail severe parameter shrinkage and lead to an excess of false positives in model selection.

Throughout this manuscript we have stressed the flexibility of the proximal distance framework in dealing with a wide range of constraints as a major strength. From our point of view, proximal distance iteration adequately approximates feasible, locally optimal solutions to constrained optimization problems for well behaved constraint sets, for instance convex sets or semialgebraic sets. Combinatorially complex constraints or erratic loss functions can cause difficulties for the proximal distance methods. The quadratic distance penalty dist(Dx, S)² is usually not an issue, and projection onto the constraint should be fast. Poor loss functions may either lack second derivatives or may possess a prohibitively expensive and potentially ill-conditioned Hessian d²f(x). In this setting techniques such as coordinate descent and regularized and quasi-Newton methods are viable alternatives for minimizing the surrogate g_ρ(x | x_n) generated by distance majorization. In any event, it is crucial to design a surrogate that renders each subproblem along the annealing path easy to solve. This may entail applying additional majorizations in f(x). Balanced against this possibility is the sacrifice of curvature information with each additional majorization.

We readily acknowledge that other algorithms may perform better than MM and proximal distance algorithms on specific problems. The triangle fixing algorithm for metric projection is a case in point (Brickell et al., 2008), as are the numerous denoising algorithms based on the ℓ₁ norm. This objection obscures the generic utility of the proximal distance principle. ADMM can certainly be beat on many specific problems, but nobody seriously suggests that it be rejected across the board. Optimization, particularly constrained optimization, is a fragmented subject, with no clear winner across problem domains. Generic methods serve as workhorses, benchmarks, and backstops.

As an aside, let us briefly note that ADMM can be motivated by the MM principle, which is the same idea driving proximal distance algorithms. The optimal pair (x, y) and λ furnishes a stationary point of the Lagrangian. Because the Lagrangian is linear in λ, its maximum for fixed (x, y) is ∞. To correct this defect, one can add a viscosity minorization to the Lagrangian. This produces the modified Lagrangian

$${ℒ}_{\mu }(x,y,\lambda )=f(x)+g(y)+\mu {\lambda }^t(Dx-y)+\frac{\mu }{2}\Vert Dx-y{\Vert }^2-\frac{\alpha }{2}{\Vert \lambda -{\lambda }_n\Vert }^2.$$

The penalty term has no impact on the x and y updates. However, the MM update for λ is determined by the stationary condition

$$0=\mu \left(D{x}_{n+1}-y_{n+1}\right)-\alpha \left(\lambda -{\lambda }_n\right),$$

so that

$${\lambda }_{n+1}={\lambda }_n+\frac{\mu }{\alpha }\left(D{x}_{n+1}-y_{n+1}\right).$$

The choice α = 1 gives the standard ADMM update. Thus, the ADMM algorithm alternates decreasing and increasing the Lagrangian in a search for the saddlepoint represented by the optimal trio (x, y, λ).

In closing we would like to draw the reader’s attention to some generalizations of the MM principle and connections to other well-studied algorithm classes. For instance, a linear fusion constraint Dx ∈ S can in principle by replaced by a nonlinear fusion constraint M(x) ∈ S. The objective and majorizer are then

$$h_{\rho }(x)=f(x)+\frac{\rho }{2}\text{dist}{[M(x),S]}^2$$

$$g\left(x\mid x_n\right)=f(x)+\frac{\rho }{2}{\Vert M(x)-{𝒫}_S\left[M\left(x_n\right)\right]\Vert }^2.$$

The objective has gradient g = ∇f(x) + ρdM(x)^t{M(x)− 𝒫_S[M(x)]}. The second differential of the majorizer is approximately d²f(x) + ρdM(x)^tdM(x) for M(x) close to 𝒫_S[M(x)]. Thus, gradient descent can be implemented with step size

$$\gamma =\frac{{\Vert g_n\Vert }^2}{g_n^t{d}^{2}f\left(x_n\right)g_n+\rho {\Vert dM\left(x_n\right)g_n\Vert }^2},$$

assuming the denominator is positive.

Algebraic penalties such as ∥g(x)∥² reduce to distance penalties with constraint set {0}. The corresponding projection operator sends any vector y to 0 so that the algebraic penalty ∥g(x)∥² = dist[g(x), {0}]². This observation is pertinent to constrained least squares with g(x) = d − Cx (Golub and Van Loan, 1996). The proximal distance surrogate can be expressed as

$$\frac{1}{2}\Vert y-Ax{\Vert }^2+\frac{\rho }{2}\Vert d-Cx{\Vert }^2=\frac{1}{2}{\Vert \left(\begin{array}{c}y\\ \sqrt{\rho }d\end{array}\right)-\left[\begin{array}{c}A\\ \sqrt{\rho }C\end{array}\right]x\Vert }^2,$$

and minimized by standard least squares algorithms. No annealing is necessary. Inequality constraints g(x) ≤ 0 behave somewhat differently. The proximal distance majorization $\text{dist}{\left[g(x),{ℝ}_{-}^m\right]}^2\le {\Vert g(x)-{𝒫}_{{ℝ}_{-}^m}\left[g\left(x_n\right)\right]\Vert }^2$ is not the same as the Beltrami quadratic penalty $g{(x)}_{+}^2$ (Beltrami, 1970). However, the standard majorization (Lange, 2016)

$$g{(x)}_{+}^2\le {\Vert g(x)-{𝒫}_{{ℝ}_{-}^m}\left[g\left(x_n\right)\right]\Vert }^2,$$

brings them back into alignment.

7. Proofs

In this section we provide proofs for the convergence results discussed in Section 3 and Section 4 for the convex and nonconvex cases, respectively.

7.1. Proposition 3.1

Proof Without loss of generality we can translate the coordinates so that y = 0. Let B be the unit sphere {x: ∥x∥ = 1}. Our first aim is to show that h_ρ(x) > f(0) throughout B. Consider the set B ∩ T, which is possibly empty. On this set the infimum b of f(x) is attained, so b > f(0) by assumption. The set B\T will be divided into two regions, a narrow zone adjacent to T and the remainder. Now let us show that there exists a δ > 0 such that h_ρ(x) ≥ f(x) ≥ f(0) + δ for all x ∈ B with dist(Dx, S) ≤ δ. If this is not so, then there exists a sequence x_n ∈ B with $f\left(x_n\right)<f(0)+\frac{1}{k}$ and $\text{dist}\left(D{x}_n,S\right)\le \frac{1}{k}$. By compactness, some subsequence of x_n converges to z ∈ B ∩ T with f(z) ≤ f(0), contradicting the uniqueness of y. Finally, let a = min_x∈B f(x). To deal with the remaining region take ρ large enough so that $a+\frac{\rho }{2}{\delta }^2>f(0)$. For such ρ, h_ρ(x) > f(0) everywhere on B. It follows that on the unit ball {x : ∥x∥ ≤ 1}, h_ρ(x) is minimized at an interior point. Because h_ρ(x) is convex, a local minimum is necessarily a global minimum.

To show that the objective h_ρ(x) is coercive, it suffices to show that it is coercive along every ray {tv: t ≥ 0, ∥v∥ = 1} (Lange, 2016). The convex function r(t) = h_ρ(tv) satisfies $r(t)\ge r(1)+r_{+}^{\prime }(1)(t-1)$. Because r(0) < r(1), the point 1 is on the upward slope of r(t), and the one-sided derivative $r_{+}^{\prime }(1)>0$. Coerciveness follows from this observation. ■

7.2. Proposition 3.2

Proof The first assertion follows from the bound g_ρ(x | x_n) ≥ h_ρ(x). To prove the second assertion, we note that it suffices to prove the existence of some constant ρ > 0 such that the matrix A + ρD^tD is positive definite (Debreu, 1952). If no choice of ρ renders A + ρD^tD positive definite, then there is a sequence of unit vectors u_m and a sequence of scalars ρ_m tending to ∞ such that

$$u_m^{t}A{u}_m+{\rho }_m{u}_m^t{D}^{t}D{u}_m\le 0.$$ (11)

By passing to a subsequence if needed, we may assume that the sequence u_m converges to a unit vector u. On the one hand, because D^tD is positive semidefinite, inequality (11) compels the conclusions $u_m^{t}A{u}_m\le 0$, which must carry over to the limit. On the other hand, dividing inequality (11) by ρ_m and taking limits imply u^tD^tDu ≤ 0 and therefore ∥Du∥ = 0. Because the limit vector u violates the condition u^tAu > 0, the required ρ > 0 exists. ■

7.3. Proposition 3.3

Proof Systematic decrease of the iterate values h_ρ(x_n) is a consequence of the MM principle. The existence of z_ρ follows from Proposition 3.1. To prove the stated bound, first observe that the function $g_{\rho }\left(x\mid x_n\right)-\frac{\rho }{2}\Vert Dx{\Vert }^2$ is convex, being the sum of the convex function f(x) and a linear function. Because ∇g_ρ(x_n+1 | x_n)^t(x − x_n+1) ≥ 0 for any x in S, the supporting hyperplane inequality implies that

$$g_{\rho }\left(x\mid x_n\right)-\frac{\rho }{2}\Vert Dx{\Vert }^2\ge g_{\rho }\left(x_{n+1}\mid x_n\right)-\frac{\rho }{2}{\Vert D{x}_{n+1}\Vert }^2-\rho x_{n+1}^t{D}^{t}D\left(x-x_{n+1}\right),$$

or equivalently

$$g_{\rho }\left(x\mid x_n\right)\ge g_{\rho }\left(x_{n+1}\mid x_n\right)+\frac{\rho }{2}{\Vert D\left(x-x_{n+1}\right)\Vert }^2.$$ (12)

Now note that the difference

$$d(x\mid y)=\frac{1}{2}\Vert x-𝒫(y){\Vert }^2-\frac{1}{2}\Vert x-𝒫(x){\Vert }^2,$$

has gradient

$$\nabla d(x\mid y)=𝒫(x)-𝒫(y).$$

Because 𝒫(x) is non-expansive, the gradient ∇d(x | y) is Lipschitz with constant 1. The tangency conditions d(y | y) = 0 and ∇d(y | y) = 0 therefore yield

$$\begin{array}{l}d(x\mid y)\le d(y\mid y)+\nabla d{(y\mid y)}^t(x-y)+\frac{1}{2}\Vert x-y{\Vert }^2\\ =\frac{1}{2}\Vert \mathit{x}-\mathit{y}{\Vert }^2,\end{array}$$ (13)

for all x. At a minimum z_ρ of h_ρ(x), combining inequalities (12) and (13) gives

$$\begin{array}{l}{h}_{\rho }\left({\mathit{x}}_{n+1}\right)+\frac{\rho }{2}{\Vert D\left({\mathit{z}}_{\rho }-{\mathit{x}}_{n+1}\right)\Vert }^2\\ \le g_{\rho }\left(x_{n+1}\mid x_n\right)+\frac{\rho }{2}{\Vert D\left(z_{\rho }-x_{n+1}\right)\Vert }^2\\ \le g_{\rho }\left({\mathit{z}}_{\rho }\mid x_n\right)\\ =h_{\rho }\left(z_{\rho }\right)-\frac{\rho }{2}{\Vert D{z}_{\rho }-𝒫\left(D{z}_{\rho }\right)\Vert }^2+\frac{\rho }{2}{\Vert D{z}_{\rho }-𝒫\left(D{x}_n\right)\Vert }^2\\ =h_{\rho }\left(z_{\rho }\right)+\rho d\left(D{z}_{\rho }\mid D{x}_n\right)\\ \le h_{\rho }\left({\mathit{z}}_{\rho }\right)+\frac{\rho }{2}{\Vert D{\mathit{z}}_{\rho }-D{x}_n\Vert }^2.\end{array}$$

Adding the result

$$h_{\rho }\left({\mathit{x}}_{n+1}\right)-h_{\rho }\left({\mathit{z}}_{\rho }\right)\le \frac{\rho }{2}\left[{\Vert D\left({\mathit{z}}_{\rho }-{\mathit{x}}_n\right)\Vert }^2-{\Vert D\left({\mathit{z}}_{\rho }-{\mathit{x}}_{n+1}\right)\Vert }^2\right]\text{, }$$

over n and invoking the descent property h_ρ(x_n+1) ≤ h_ρ(x_n), telescoping produces the desired error bound

$$\begin{array}{l}{h}_{\rho }\left({\mathit{x}}_{n+1}\right)-h_{\rho }\left({\mathit{z}}_{\rho }\right)\le \frac{\rho }{2(n+1)}\left[{\Vert D\left({\mathit{z}}_{\rho }-{\mathit{x}}_0\right)\Vert }^2-{\Vert D\left({\mathit{z}}_{\rho }-{\mathit{x}}_{n+1}\right)\Vert }^2\right]\\ \le \frac{\rho }{2(n+1)}{\Vert D\left(z_{\rho }-x_0\right)\Vert }^2.\end{array}$$

This is precisely the asserted bound. ■

7.4. Proposition 3.4

Proof The existence and uniqueness of z_ρ are obvious. The remainder of the proof hinges on the facts that h_ρ(x) is μ-strongly convex and the surrogate g_ρ(x | w) is L + ρ∥D∥²-smooth for all w. The latter assertion follows from

$$\nabla g_{\rho }(x\mid w)-\nabla g_{\rho }(y\mid w)=\nabla f(x)-\nabla f(y)+\rho D^{t}D(x-y).$$

These facts together with ∇g_ρ(z_ρ | z_ρ) = ∇h_ρ(z_ρ) = 0 imply

$$\begin{array}{l}{h}_{\rho }(x)-h_{\rho }\left(z_{\rho }\right)\le g_{\rho }\left(x\mid z_{\rho }\right)-g_{\rho }\left(z_{\rho }\mid z_{\rho }\right)\\ \le \nabla g_{\rho }{\left(z_{\rho }\mid z_{\rho }\right)}^t\left(x-z_{\rho }\right)+\frac{L+\rho \Vert D{\Vert }^2}{2}{\Vert x-z_{\rho }\Vert }^2\\ =\frac{L+\rho \Vert D{\Vert }^2}{2}{\Vert \mathit{x}-{\mathit{z}}_{\rho }\Vert }^2.\end{array}$$ (14)

The strong convexity condition

$$0\ge h_{\rho }\left(z_{\rho }\right)-h_{\rho }(x)\ge \nabla h_{\rho }{(x)}^t\left(z_{\rho }-x\right)+\frac{\mu }{2}{\Vert z_{\rho }-x\Vert }^2$$

entails

$$\Vert \nabla h_{\rho }(\mathit{x})\Vert \cdot \Vert {\mathit{z}}_{\rho }-\mathit{x}\Vert \ge -\nabla h_{\rho }{(\mathit{x})}^t\left({\mathit{z}}_{\rho }-\mathit{x}\right)\ge \frac{\mu }{2}{\Vert {\mathit{z}}_{\rho }-\mathit{x}\Vert }^2.$$

It follows that $\Vert \nabla h_{\rho }(\mathit{x})\Vert \ge \frac{\mu }{2}\Vert \mathit{x}-{\mathit{z}}_{\rho }\Vert$. This last inequality and inequality (14) produce the Polyak-Łojasiewicz bound

$$\frac{1}{2}{\Vert \nabla h_{\rho }(x)\Vert }^2\ge \frac{{\mu }^2}{2\left(L+\rho \Vert D{\Vert }^2\right)}\left[h_{\rho }(x)-h_{\rho }\left(z_{\rho }\right)\right].$$

Taking c = L + ρ∥D∥² and

$$x=x_n-c^{-1}\nabla g_{\rho }\left(x_n\mid x_n\right)=x_n-c^{-1}\nabla h_{\rho }\left(x_n\right),$$

the Polyak-Łojasiewicz bound gives

$$\begin{array}{l}{h}_{\rho }\left(x_{n+1}\right)-h_{\rho }\left(x_n\right)\le g_{\rho }\left(x_{n+1}\mid x_n\right)-g_{\rho }\left(x_n\mid x_n\right)\\ \le g_{\rho }\left(x\mid x_n\right)-g_{\rho }\left(x_n\mid x_n\right)\\ \le -c^{-1}\nabla g_{\rho }{\left(x_n\mid x_n\right)}^t\nabla h_{\rho }\left(x_n\right)+\frac{c}{2}{\Vert c^{-1}\nabla h_{\rho }\left(x_n\right)\Vert }^2\\ =-\frac{1}{2c}{\Vert \nabla h_{\rho }\left(x_n\right)\Vert }^2\\ \le -\frac{{\mu }^2}{2c^2}\left[h_{\rho }\left(x_n\right)-h_{\rho }\left(z_{\rho }\right)\right].\end{array}$$

Rearranging this inequality yields

$$h_{\rho }\left(x_{n+1}\right)-h_{\rho }\left(z_{\rho }\right)\le \left[1-\frac{{\mu }^2}{2c^2}\right]\left[h_{\rho }\left(x_n\right)-h_{\rho }\left(z_{\rho }\right)\right)],$$

which can be iterated to give the stated bound. ■

7.5. Proposition 4.2

Proof To validate the subanalytic premise of Proposition 4.1, first note that semialgebraic functions and sets are automatically subanalytic. The penalized loss

$$h_{\rho }(x)=f(x)+\frac{\rho }{2}\text{dist}{(x,S)}^2,$$

is semialgebraic by the sum rule. Under the assumption stated in Proposition 3.2, g_ρ(x | x_n) is strongly convex and coercive for ρ sufficiently large. Continuity of g_ρ(x | x_n) is a consequence of the continuity of f(x). The Lipschitz condition follows from the fact that the sum of two Lipschitz functions is Lipschitz. Under these conditions and regardless of which projected point P_S(x) is chosen, the MM iterates are guaranteed to converge to a stationary point. ■

7.6. Proposition 4.3

Proof The first claim is true owing to the inclusion-exclusion formula

$$f_{(k)}(x)=\sum _{j=k}^n\sum _{\left|S\right|=j}{(-1)}^{j-k}\left(\begin{array}{l}j-1\hfill \\ k-1\hfill \end{array}\right)\underset{i\in S}{\max }{f}_i(x),$$

and the previously stated closure properties. For n = 3 and k = 2 the inclusion-exclusion formula reads f₍₂₎ = max{f₁, f₂} + max{f₁, f₃} + max{f₂, f₃}−2max{f₁, f₂, f₃}. To prove the second claim, note that a sparsity set in ${ℝ}^p$ with at most k nontrivial coordinates can be expressed as the zero set {x: y_(p−k) = 0}, where y_i = |x_i|. Thus, the sparsity set is semialgebraic. ■

7.7. Proposition 4.4

Proof Proposition 4.2 proves that the proximal distance iterates x_n converge to x_∞. Suppose that Dx_∞ has k unambiguous largest components in magnitude. Then Dx_n shares this property for large n. It follows that all $p_n={𝒫}_{S_k}\left(D{x}_n\right)$ occur in the same k-dimensional subspace S for large n. Thus, we can replace the sparsity set S_k by the subspace S in minimization from some n onward. Convergence at a linear rate now follows from Proposition 3.4.

To prove that the set A of points x such that Dx has k ambiguous largest components in magnitude has measure 0, observe that it is contained in the set T where two or more coordinates tie. Suppose x satisfies the tie condition $d_i^{t}x=d_j^{t}x$ for two rows $d_i^t$ and $d_j^t$ of D. If the rows of D are unique, then the equality (d_i − d_j)^tx = 0 defines a hyperplane in x space and consequently has measure 0. Because there are a finite number of row pairs, T as a union has measure 0. ■

Appendix A. Relevant Theory for Nonconvex Analysis

Let us first review the notion of a Fréchet subdifferential (Kruger, 2003). If h(x) is a function mapping ${ℝ}^p$ into $ℝ\cup \{+\infty \}$, then its Fréchet subdifferential at x ∈ dom f is defined as

$${\partial }^{F}h(x)=\left\{v:\underset{y\to x}{\lim \inf }\frac{h(y)-h(x)-v^t(y-x)}{\Vert y-x\Vert }\ge 0\right\}.$$

The set ∂^Fh(x) is closed, convex, and possibly empty. If h(x) is convex, then ∂^Fh(x) reduces to its convex subdifferential. If h(x) is differentiable, then ∂^Fh(x) reduces to its ordinary differential. At a local minimum x, Fermat’s rule 0 ∈ ∂^Fh(x) holds. For a locally Lipschitz and directionally differentiable function, the Fréchet subdifferential becomes

$${\partial }^{F}h(x)=\left\{v:d_{u}h(x)\ge v^{t}u\text{ for all directions }u\right\}.$$

Here d_uh(x) is the directional derivative of h(x) at x in the direction u. This result makes it clear that at a critical point, all directional derivatives are flat or point uphill.

We will also need some notions from algebraic geometry (Bochnak et al., 2013). For simplicity we focus on the class of semialgebraic functions and the corresponding class of semialgebraic subsets of ${ℝ}^p$. The latter is the smallest class that:

1. contains all sets of the form {x: q(x) > 0} for a polynomial q(x) in p variables, and

2. is closed under the formation of finite unions, finite intersections, set complements, and Cartesian products.

A function $a:{ℝ}^p\mapsto {ℝ}^r$ is said to be semialgebraic if its graph is a semialgebraic set of ${ℝ}^{p+r}$. The class of real-valued semialgebraic functions contains all polynomials p(x) and all 0/1 indicators of algebraic sets. It is closed under the formation of sums and products and therefore constitutes a commutative ring with identity. The class is also closed under the formation of absolute values, reciprocals when a(x) ≠ 0, nth roots when a(x) ≥ 0, and maxima max{a(x), b(x)} and minima min{a(x), b(x)}. Finally, the composition of two semialgebraic functions is semialgebraic.

Appendix B. Convergence Properties of ADMM

To avail ourselves of the known results, we define three functions

$$H_{\rho }(x,y)=f(x)+\frac{\rho }{2}\text{dist}{(y,S)}^2,$$

$$\begin{array}{l}{ℒ}_{\rho }(x,y,\lambda )=H_{\rho }(x,y)+{\lambda }^t(Dx-y),\text{ and}\\ q(\lambda )=\underset{x,y}{\min }{\mathit{ℒ}}_{\rho }(x,y,\lambda ),\end{array}$$

the second and third being the Lagrangian and dual function. This notation leads to following result; see Beck (2017) for an accessible proof.

Proposition B.1 Suppose that S is closed and convex and that the loss f(x) is proper, closed, and convex with domain whose relative interior is nonempty. Also assume the dual function q(λ) achieves its maximum value. If the objective $f(x)+\frac{\rho }{2}\Vert Dx{\Vert }^2+a^{t}x$ achieves its minimum value for all a ≠ 0, then the ADMM running averages

$${\overline{x}}_n=\frac{1}{n}\sum _{k=1}^n{x}_k\text{ and }{\overline{y}}_n=\frac{1}{n}\sum _{k=1}^n{y}_k,$$

satisfy

$$\begin{array}{l}\left|H_{\rho }\left({\overline{x}}_n,{\overline{y}}_n\right)-h_{\rho }\left(x_{\rho }\right)\right|=O\left(\frac{\rho }{n}\right),and\\ \Vert D{\overline{x}}_n-y_n\Vert =O\left(\frac{1}{k}\right).\end{array}$$

Note that Proposition 3.2 furnishes a sufficient condition under which the functions $f\text{(x)+}\frac{\rho }{2}\Vert Dx{\Vert }^2+a^{t}x$ achieve their minima. Linear convergence holds under stronger assumptions.

Proposition B.2 Suppose that S is closed and convex, that the loss f(x) is L-smooth and μ-strongly convex, and that the map determined by D is onto. Then the ADMM iterates converge at a linear rate.

Giselsson and Boyd (2016) proved Proposition B.2 by operator methods. A range of convergence rates is specified there.

Appendix C. Additional Details for Metric Projection Example

Given a n × n dissimilarity matrix C = (c_ij) with non-negative weights w_ij, our goal is to find a semi-metric X = (x_ij). We start by denoting trivec an operation that maps a symmetric matrix X to a vector x, x = trivec(X) (Figure 3). Then we write the metric projection objective as

$$h_{\rho }(x)=\frac{1}{2}{\Vert W^{1/2}(x-c)\Vert }_2^2+\frac{\rho }{2}\text{dist}{\left(Tx,{ℝ}_{+}^{m_1}\right)}^2+\frac{\rho }{2}\text{dist}{\left(x,{ℝ}_{+}^{m_2}\right)}^2,$$

where c = trivec(C). Here T encodes triangle inequalities and the m_i count the number of constraints of each type. The usual distance majorization furnishes a surrogate

$$\begin{gathered} g_{\rho }\left(x\mid x_n\right)=\frac{1}{2}{\Vert W^{1/2}(x-c)\Vert }_2^2+\frac{\rho }{2}{\Vert Tx-𝒫\left(T{x}_n,{ℝ}_{+}^{m_1}\right)\Vert }_2^2+\frac{\rho }{2}{\Vert x-𝒫\left(x_n,{ℝ}_{+}^{m_2}\right)\Vert }_2^2 \\ =\frac{1}{2}{\Vert W^{1/2}(x-c)\Vert }_2^2+\frac{\rho }{2}{\Vert Dx-𝒫\left(D{x}_n\right)\Vert }_2^2. \end{gathered}$$

The notation 𝒫(·, S) denotes projection onto a set S. The fusion matrix D = [T ; I] stacks the two operators; the joint projection operates in a block-wise fashion.

Figure 3:

Example of a symmetric matrix X and its minimal representation x = trivec(X).

C.1. MM

We rewrite the surrogate explicitly as a least squares problem minimizing ${\Vert Ax-b_n\Vert }_2^2$:

$$x_{n+1}=\underset{x}{\text{argmin}}\frac{1}{2}{\Vert \left[\begin{array}{l}{W}^{1/2}\hfill \\ \sqrt{\rho }D\hfill \end{array}\right]x-\left[\begin{array}{c}c\\ \sqrt{\rho }𝒫\left(D{x}_n\right)\end{array}\right]\Vert }_2^2,$$

where c ≡ y from the main text. Updating the RHS b_n in the linear system reduces to evaluating the projection and copy operations. It is worth noting that triangle fixing algorithms that solve the metric nearness problem operate in the same fashion, except they work one triangle a time. That is, each iteration solves $\left(\begin{array}{l}n\hfill \\ 3\hfill \end{array}\right)$ least squares problems compared to 1 in this formulation. A conjugate gradient type of algorithm solves the normal equations directly using A^tA, whereas LSQR type methods use only A and A^t.

C.2. Steepest Descent

The updates x_n+1 = x_n − t_n∇h_ρ(x_n) admit an exact solution for the line search parameter t_n. Recall the generic formula from the main text:

$$t_n=\frac{{\Vert v_n\Vert }^2}{v_n^{t}A{v}_n+\rho {\Vert D{v}_n\Vert }^2}.$$

Identifying v_n with ∇h_ρ(x_n) we have

$$\begin{array}{l}\nabla h_{\rho }\left(x_n\right)=W\left(x_n-c\right)+\rho D^t\left[D{x}_n-𝒫\left(D{x}_n\right)\right]\\ =\mathit{W}\left({\mathit{x}}_n-c\right)+\rho \left(I+T^{t}T\right){\mathit{x}}_n-\rho \left[T^t𝒫\left(T{x}_n,{ℝ}_{+}^{m_1}\right)+𝒫\left(x_n,{ℝ}_{+}^{m_1}\right)\right],\\ t_n=\frac{{\Vert v_n\Vert }^2}{{\Vert W^{1/2}{v}_n\Vert }^2+\rho {\Vert D{v}_n\Vert }^2}.\end{array}$$

C.3. ADMM

Taking y as the dual variable and λ as scaled multipliers, the updates for each ADMM block are

$$\begin{array}{l}{x}_{n+1}=\underset{x}{\text{argmin}}{\Vert \left[\begin{array}{l}{W}^{1/2}\hfill \\ \sqrt{\mu }D\hfill \end{array}\right]x-\left[\begin{array}{c}c\\ \sqrt{\mu }\left(y_n-{\lambda }_n\right))\end{array}\right]\Vert }_2^2,\\ y_{n+1}=\frac{\alpha }{1+\alpha }𝒫\left(z_n\right)+\frac{1}{1+\alpha }{z}_n;\hspace{1em}{z}_n=D{x}_{n+1}+{\lambda }_n,\alpha =\rho /\mu .\end{array}$$

Finally, the Multipliers follow the standard update.

C.4. Properties of the Triangle Inequality Matrix

These results have been documented before and are useful in designing fast subroutines for Dx and D^tDx. Recall that m counts the number of nodes in the problem and $p=\left(\begin{array}{c}m\\ 2\end{array}\right)$ is the number of parameters. In this notation $\mathit{D}=\left(\begin{array}{l}\mathit{T}\hfill \\ {\mathit{I}}_p\hfill \end{array}\right)$ and D^tD = T^tT + I_p.

Proposition C.1 The matrix T has $3\left(\begin{array}{c}m\\ 3\end{array}\right)$ rows and $\left(\begin{array}{c}m\\ 2\end{array}\right)$ columns.

Proof Interpret X as the adjacency matrix for a complete directed graph on m nodes without self-edges. When X is symmetric the number of free parameters is therefore $\left(\begin{array}{c}m\\ 2\end{array}\right)$. An oriented 3-cycle is formed by fixing 3 nodes so there are $\left(\begin{array}{c}m\\ 3\end{array}\right)$ such cycles. Now fix the orientation of the 3-cycles and note that each triangle encodes 3 metric constraints. The number of constraints is therefore $3\left(\begin{array}{c}m\\ 3\end{array}\right)$. ■

Proposition C.2 Each column of T has 3(m − 2) nonzero entries.

Proof In view of the previous result, the entries T_ij encode whether edge j participates in constraint i. We proceed by induction on the number of nodes m. The base case m = 3 involves one triangle and is trivial. Note that a triangle encodes 3 inequalities.

Now consider a complete graph on m nodes and suppose the claim holds. Without loss of generality, consider the collection of 3-cycles oriented clockwise and fix an edge j. Adding a node to the graph yields 2m new edges, two for each of the existing m nodes. This action also creates one new triangle for each existing edge. Thus, edge j appears in 3(m − 2)+3 = 3[(m + 1)−2] triangle inequality constraints based on the induction hypothesis. ■

Proposition C.3 Each column of T has m − 2 +1s and 2(m − 2) −1s.

Proof Interpret the inequality x_ij ≤ x_ik + x_kj with i > k > j as the ordered triple x_ij, x_ik, x_kj. The statement is equivalent to counting

$$a(N)=\text{number}\text{of}\text{times}{x}_{ij}\text{appears}\text{in}\text{position}1,\text{and},$$

$$b(N)=\text{number}\text{of}\text{times}{x}_{ij}\text{appears}\text{in}\text{position}2,\text{or}3,$$

where N denotes the number of constraints. In view of the previous proposition, it is enough to prove a(N) = m − 2. Note that a(3) = 1, meaning that x_ij appears in position 1 exactly once within a given triangle. Given that an edge (i, j) appears in 3(m−2) constraints, divide this quantity by the number of constraints per triangle to arrive at the stated result. ■

Proposition C.4 The matrix T has full column rank.

Proof It is enough to show that A = T^tT is full rank. The first two propositions imply

$$a_{ii}=\langle T_i,T_i\rangle =\sum {(\pm 1)}^2=3(m-2).$$

To compute the off-diagonal entries, fix a triangle and note that two edges i and j appear in all three of its constraints of the form x_i ≤ x_j + x_k. There are three possibilities for a given constraint c:

$$T_{c,i}{T}_{c,j}=\{\begin{array}{cc}-1,& \text{ if }i\text{ LHS, }j\text{ RHS or vice-versa }\hfill \\ 1,& \text{ if }i\text{ and }j\text{ both appear on RHS }\hfill \\ 0,& \text{if one of }i\text{ or }j\text{ is missing}.\hfill \end{array}$$

It follows that

$$a_{ij}=\langle T_i,T_j\rangle =\{\begin{array}{cc}-1,& \text{ if edges }i\text{ and }j\text{ overlap in constraints }\hfill \\ 0,& \text{ otherwise}.\hfill \end{array}$$

By Proposition C.2, an edge i appears in 3(m − 2) constraints. Imposing the condition that edge j also appears reduces this number by m − 2, the number of remaining nodes that can contribute edges in our accounting. The calculation

$$\sum _{j\ne i}\left|a_{ij}\right|=2(m-2)<3(m-2)=\left|a_{ii}\right|,$$

establishes that A is strictly diagonally dominant and hence full rank. ■

Proposition C.5 The matrix T^tT has at most 3 distinct eigenvalues of the form m − 2, 2m − 2, and 3m − 4 with multiplicities 1, m − 1, and $\frac{1}{2}m(m-3)$, respectively.

Proof Let $M\in {\{0,1\}}^{\left(\begin{array}{c}m\\ 2\end{array}\right)\times m}$ be the incidence matrix of a complete graph with m vertices. That is M has entry m_e,v = 1 if vertex v occurs in edge e and 0 otherwise. Each row of M has two entries equal to 1; each column of M has m−1 entries equal to 1. It is easy to see

$$T^{t}T=(3m-4)I_{\left(\begin{array}{c}m\\ 2\end{array}\right)}-M{M}^t.$$

The Gram matrices M^tM and MM^t share the same positive eigenvalues. Since $M^{t}M=(m-2)I_m+m\left(1_m/\sqrt{m}\right){\left(1_m/\sqrt{m}\right)}^t$ has eigenvalue 2m−2 with multiplicity 1 and eigenvalue m−2 with multiplicity m − 1, MM^t has eigenvalue 2m − 2 with multiplicity 1, eigenvalue m − 2 with multiplicity m − 1, and eigenvalue 0 with multiplicity m(m − 3)/2. Therefore the eigenvalues of T^tT are m − 2, 2m − 2, and 3m − 4 with multiplicities 1, m − 1, and m(m − 3)/2 respectively. ■

In general, it is easy to check that the matrix m × m matrix aI + b11^t has the eigenvector 1 with eigenvalue a + mb and m − 1 orthogonal eigenvectors

$$u_i=\frac{1}{i-1}\sum _{j=1}^{i-1}{e}_j-e_i,\hspace{1em}i=2,\dots ,m$$

with eigenvalue a. Note that each u_i is perpendicular to 1. None of these eigenvectors is normalized to have length 1. Although the eigenvectors u_i are certainly convenient, they are not unique.

To recover the eigenvectors of T^tT, and hence those D^tD, we can leverage the eigenvectors of M^tM, which we know. The following generic observations are pertinent. If a matrix A has full SVD USV ^t, then its transpose has full SVD A^t = VSU^t. As mentioned AA^t and A^tA share the same nontrivial eigenvalues. These can be recovered as the nontrivial diagonal entries of S². Suppose we know the eigenvectors U of AA^t = US²U^t. Since A^tU = VS, then presumably we can recover some of the eigenvectors V as A^tUS⁺, where S⁺ is the diagonal pseudo-inverse of S.

C.5. Fast Subroutines for Solving Linear Systems

Using the Woodbury formula, the inverse of T^tT can be expressed as

$$\begin{array}{l}{\left({\mathit{T}}^t\mathit{T}\right)}^{-1}\\ ={\left[(3m-4)I_{\left(\begin{array}{c}m\\ 2\end{array}\right)}-\mathit{M}{\mathit{M}}^t\right]}^{-1}\\ ={(3m-4)}^{-1}{I}_{\left(\begin{array}{c}m\\ 2\end{array}\right)}-{(3m-4)}^{-2}M{\left[-I_m+{(3m-4)}^{-1}{M}^{t}M\right]}^{-1}{M}^t\\ ={(3m-4)}^{-1}{I}_{\left(\begin{array}{c}m\\ 2\end{array}\right)}-{(3m-4)}^{-1}M{\left[-(2m-2)I_m+1_m{1}_m^t\right]}^{-1}{M}^t\\ ={(3m-4)}^{-1}{I}_{\left(\begin{array}{c}m\\ 2\end{array}\right)}-{(3m-4)}^{-1}M\left[-{(2m-2)}^{-1}{I}_m-{(2m-2)}^{-1}{(m-2)}^{-1}{1}_m{1}_m^t\right]M^t\\ =\frac{1}{3m-4}{I}_{\left(\begin{array}{c}m\\ 2\end{array}\right)}+\frac{2}{(3m-4)(m-1)(m-2)}{1}_{\left(\begin{array}{c}m\\ 2\end{array}\right)}{1}_{\left(\begin{array}{c}m\\ 2\end{array}\right)}^t+\frac{1}{2(3m-4)(m-1)}\mathit{M}{\mathit{M}}^t.\end{array}$$

Solving linear system T^tT invokes two matrix vector multiplications involving the incidence matrix M. Mv corresponds to taking pairwise sums of the components of a vector v of length m. M^tw corresponds to taking a combination of column and row sums of a lower triangular matrix with the lower triangular part populated by the components of a vector w with length $\left(\begin{array}{c}m\\ 2\end{array}\right)$. Both operations cost O(m²) flops. This result can be extended to the full fusion matrix D^tD that incorporates non-negativity constraints and, more importantly, to the linear system I + ρD^tD:

$$\begin{array}{cccc}{\left[I_{\left(\begin{array}{c}m\\ 2\end{array}\right)}+\rho D^{t}D\right]}^{-1}& ={\left[I_{\left(\begin{array}{c}m\\ 2\end{array}\right)}+\rho \left(T^{t}T+I_{\left(\begin{array}{c}m\\ 2\end{array}\right)}\right)\right]}^{-1}& & a={[3(m-1)\rho +1]}^{-1}\\ & =a{I}_{\left(\begin{array}{c}m\\ 2\end{array}\right)}+ab\rho M{M}^t+4abc{\rho }^2{1}_{\left(\begin{array}{c}m\\ 2\end{array}\right)}{1}_{\left(\begin{array}{c}m\\ 2\end{array}\right)}^t;& & b={[(2m-1)\rho +1]}^{-1}\\ & & & c={[(m-1)\rho +1]}^{-1}.\end{array}$$

Appendix D. Additional Details for Convex Regression Example

We start by formulating the proximal distance version of the problem:

$$h_{\rho }(v)=\frac{1}{2}\Vert Mv-y{\Vert }_2^2+\frac{\rho }{2}\text{dist}{\left(Dv,{ℝ}_{-}^m\right)}^2,$$

where v = [θ; vec(Ξ)] stacks each optimization variable into a vector of length n(1 + d). This maneuver introduces matrices

$$M=\left[\begin{array}{ll}{I}_{n\times n}\hfill & 0_{n\times nd}\hfill \end{array}\right],\hspace{1em}\mathit{D}=\left[\begin{array}{ll}A\hfill & B\hfill \end{array}\right],$$

where [Aθ]_k = θ_j − θ_i and [B vec(Ξ)]_k = 〈x_i − x_j, ξ_j〉 according to the ordering i > j.

D.1. MM

We rewrite the surrogate explicitly a least squares problem minimizing ${\Vert \tilde{M}v-{\tilde{b}}_n\Vert }_2^2$:

$$v_{n+1}=\underset{v}{\text{argmin}}\frac{1}{2}{\Vert \left[\begin{array}{c}M\\ \sqrt{\rho }D\end{array}\right]v-\left[\begin{array}{c}b\\ \sqrt{\rho }𝒫\left(D{v}_n\right)\end{array}\right]\Vert }_2^2,$$

where b ≡ y to avoid clashing with notation in ADMM below. In this case it seems better to store D explicitly in order to avoid computing x_i − x_j each time one applies D, D^t, or D^tD.

D.2. Steepest Descent

The updates v_n+1 = v_n − t_n∇h_ρ(v_n) admit an exact solution for the line search parameter t_n. Taking q_n = ∇h_ρ(v_n) as the gradient we have

$$q_n=A^{t}A\left(v_n-b\right)+\rho D^t\left[D{v}_n-𝒫\left(D{v}_n\right)\right],$$

$$t_n=\frac{{\Vert q_n\Vert }^2}{{\Vert A{q}_n\Vert }^2+\rho {\Vert D{q}_n\Vert }^2}.$$

Note that Aq_n = ∇_θh_ρ(v_n), the gradient with respect to function values θ.

D.3. ADMM

Take y as the dual variable and λ as scaled multipliers. Then the ADMM updates are

$$\begin{array}{l}{v}_{n+1}=\underset{v}{\text{argmin}}\frac{1}{2}{\Vert \left[\begin{array}{c}A\\ \sqrt{\mu }D\end{array}\right]v-\left[\begin{array}{c}b\\ \sqrt{\mu }\left(y_n-{\lambda }_n\right)\end{array}\right]\Vert }_2^2,\\ y_{n+1}=\frac{\alpha }{1+\alpha }𝒫\left(z_n\right)+\frac{1}{1+\alpha }{z}_n;\hspace{1em}{z}_n=D{v}_{n+1}+{\lambda }_n,\alpha =\rho /\mu ,\end{array}$$

and with the update for the multipliers being standard.

Appendix E. Additional Details for Convex Clustering Example

We write u = vec(U) and x = vec(X), so the surrogate becomes

$$g_{\rho }\left(u\mid u_n\right)=\frac{1}{2}\Vert u-x{\Vert }_2^2+\frac{\rho }{2}{\Vert Du-{𝒫}_{S_k}\left(D{u}_n\right)\Vert }^2.$$

E.1. MM

Rewrite the surrogate explicitly a least squares problem minimizing ${\Vert Au-b_n\Vert }_2^2$:

$$u_{n+1}=\underset{u}{\text{argmin}}\frac{1}{2}{\Vert \left[\begin{array}{c}I\\ \sqrt{\rho }D\end{array}\right]u-\left[\begin{array}{c}x\\ \sqrt{\rho }{𝒫}_{S_k}\left(D{u}_n\right)\end{array}\right]\Vert }_2^2.$$

E.2. Steepest Descent

The updates u_n+1 = u_n − t_n∇h_ρ(u_n) admit an exact solution for the line search parameter t_n. Taking q_n = ∇h_ρ(u_n) as the gradient we have

$$\begin{array}{l}{q}_n=\left(u_n-x\right)+\rho D^t\left[D{u}_n-{𝒫}_{S_k}\left(D{u}_n\right)\right],\\ t_n=\frac{{\Vert q_n\Vert }^2}{{\Vert q_n\Vert }^2+\rho {\Vert D{q}_n\Vert }^2}.\end{array}$$

Note that blocks in ${\left[D{u}_n-{𝒫}_{S_k}\left(D{u}_n\right)\right]}_{\ell }$ are equal to 0 whenever the projection of block [Du_n]_ℓ is non-zero.

E.3. ADMM

Take y as the dual variable and λ as scaled multipliers. Minimizing the u block involves solving a single linear system:

$$\begin{array}{l}{u}_{n+1}=\underset{u}{\text{argmin}}\frac{1}{2}{\Vert \left[\begin{array}{c}I\\ \sqrt{\mu }D\end{array}\right]u-\left[\begin{array}{c}x\\ \sqrt{\mu }\left(y_n-{\lambda }_n\right)\end{array}\right]\Vert }_2^2,\\ y_{n+1}=\frac{\alpha }{1+\alpha }{𝒫}_{S_k}\left(z_n\right)+\frac{1}{1+\alpha }{z}_n;\hspace{1em}{z}_n=D{u}_{n+1}+{\lambda }_n,\alpha =\rho /\mu .\end{array}$$

Multipliers follow the standard update.

E.4. Blockwise Sparse Projection

The projection ${𝒫}_{S_k}$ maps a matrix to a sparse representation with k non-zero columns (or blocks in the case of the vectorized version). In the context of clustering, imposing sparsity permits a maximum of k violations in consensus of centroid assignments, u_i = u_j. Letting Δ_ℓ ≡ Δ_ij = ∥u_i − u_j∥ denote pairwise distances and $M=\left(\begin{array}{c}m\\ 2\end{array}\right)$ denote the number of unique pairwise distances, we define the projection along blocks v_ℓ = u_i − u_j for each pair as

$${𝒫}_{S_k}\left(v_{\ell }\right)=\{\begin{array}{ll}{v}_{\ell },\hfill & \text{ if }{\Delta }_{\ell }\in \left\{{\Delta }_{(M)},{\Delta }_{(M-1)},\dots {\Delta }_{(M-k+1)}\right\}\hfill \\ \mathbf{0},\hfill & \text{ otherwise}.\hfill \end{array}$$

Here the notation x_(i) represents the i-th element in an ascending list. Concretely, the magnitude of a difference v_ℓ must be within the top k distances. An alternative, helpful definition is based on the smallest distances

$${𝒫}_{S_k}\left(v_{\ell }\right)=\{\begin{array}{ll}\mathbf{0},\hfill & \text{ if }{\Delta }_k\in \left\{{\Delta }_{(1)},{\Delta }_{(2)},\dots {\Delta }_{(k)}\right\}\hfill \\ v_{\ell },\hfill & \text{ otherwise }\hfill \end{array}$$

Thus, it is enough to find a pivot Δ_(M−k+1) or Δ_(k) that splits the list into the top k elements. Because the hyperparameter k has a finite range in $\left\{0,1,2,\dots ,\left(\begin{array}{c}m\\ 2\end{array}\right)\right\}$ one can exploit symmetry to reduce the best/average computational complexity in a search procedure. We implement this projection using a partial sorting algorithm based on quicksort, and note that it is set-valued in general.

Appendix F. Additional Details for Image Denoising Example

Here we restate the total variation denoising problem to take advantage of proximal operators in the proximal distance framework. We minimize the penalized objective

$$h_{\rho }(U)=\frac{1}{2}\Vert u-w{\Vert }_F^2+\frac{\rho }{2}\text{dist}{\left(Du,S_{\gamma }\right)}^2,$$

where w = vec(W) is a noisy input image and S_γ is the ℓ₁ ball with radius γ. Thus, γ may be interpreted as the target total variation of the reconstructed image. Distance majorization yields the surrogate

$$g_{\rho }\left(u\mid u_n\right)=\frac{1}{2}\Vert u-w{\Vert }_2^2+\frac{\rho }{2}{\Vert Du-{𝒫}_{\gamma }\left(D{u}_n\right)\Vert }^2.$$

Here 𝒫γ(Du) enforces sparsity in all derivatives through projection onto the ℓ₁ ball with radius γ. Because D is ill-conditioned, we append an additional row with zeros everywhere except the last entry; that is, D = [D_n, D_p, e_p] with $u\in {ℝ}^p$. In this case, the sparse projection applies to all but the last component of Du.

F.1. MM

Rewrite the surrogate explicitly as a least squares problem:

$$u_{n+1}=\underset{x}{\text{argmin}}\frac{1}{2}{\Vert \left[\begin{array}{c}I\\ \sqrt{\rho }D\end{array}\right]u-\left[\begin{array}{c}w\\ \sqrt{\rho }{𝒫}_{\gamma }\left(D{u}_n\right)\end{array}\right]\Vert }_2^2.$$

F.2. Steepest Descent

The updates u_n+1 = u_n − t_n∇h_ρ(u_n) admit an exact solution for the line search parameter t_n. Taking q_n = ∇h_ρ(u_n) as the gradient we have

$$\begin{array}{l}{q}_n=\left(u_n-w\right)+\rho D^t\left[D{u}_n-{𝒫}_{\gamma }\left(D{u}_n\right)\right],\\ t_n=\frac{{\Vert q_n\Vert }^2}{{\Vert q_n\Vert }^2+\rho {\Vert D{q}_n\Vert }^2}.\end{array}$$

F.3. ADMM

We denote by y the dual variable and λ the scaled multipliers. Minimizing the u block involves solving a single linear system:

$$\begin{array}{l}{u}_{n+1}=\underset{x}{\text{argmin}}\frac{1}{2}{\Vert \left[\begin{array}{c}I\\ \sqrt{\mu }D\end{array}\right]u-\left[\begin{array}{c}w\\ \sqrt{\mu }\left(y_n-{\lambda }_n\right)\end{array}\right]\Vert }_2^2,\\ y_{n+1}=\frac{\alpha }{1+\alpha }{𝒫}_{\gamma }\left(z_n\right)+\frac{1}{1+\alpha }{z}_n;\hspace{1em}{z}_n=D{u}_{n+1}+{\lambda }_n,\alpha =\rho /\mu .\end{array}$$

Multipliers follow the standard update.

Appendix G. Additional Details for Condition Number Example

Given a matrix M = UΣV⁻¹ with singular values σ₁ ≥ σ₂ ≥ … ≥ σ_p, we seek a new matrix N = UXV⁻¹ such that cond(B) = x₁/x_p ≤ c. We minimize the penalized objective

$$h_{\rho }(x)=\frac{1}{2}\Vert x-\sigma {\Vert }^2+\frac{\rho }{2}\text{dist}{\left(Dx,{ℝ}_{-}^{p^2}\right)}^2,$$

as suggested by the Von Neumann-Fan inequality. The fusion matrix D = C + S encodes the constraints x_i − cx_j ≤ 0. Distance majorization yields the surrogate

$$g_{\rho }\left(x\mid x_n\right)=\frac{1}{2}\Vert x-w{\Vert }_2^2+\frac{\rho }{2}{\Vert Dx-{𝒫}_{-}\left(D{x}_n\right)\Vert }^2.$$

To be specific, the matrix C = − c1_p ⊗ I_p scales the p × p identity matrix by −c and stacks it p times. Similarly, the matrix S = I_p ⊗ 1_p stacks p matrices of dimension p × p. Each of these stacked matrices has (p−1) 0_p columns and one shifted 1_p column. For example, for p = 2

$$S=\left[\begin{array}{l}\left(\begin{array}{ll}1\hfill & 0\hfill \\ 1\hfill & 0\hfill \end{array}\right)\hfill \\ \left(\begin{array}{ll}0\hfill & 1\hfill \\ 0\hfill & 1\hfill \end{array}\right)\hfill \end{array}\right].$$

G.1. MM

Rewrite the surrogate explicitly a least squares problem minimizing ${\Vert Ax-b_n\Vert }_2^2$:

$$x_{n+1}=\underset{x}{\text{argmin}}\frac{1}{2}{\Vert \left[\begin{array}{c}I\\ \sqrt{\rho }D\end{array}\right]x-\left[\begin{array}{c}\sigma \\ \sqrt{\rho }𝒫\left(D{x}_n\right)\end{array}\right]\Vert }_2^2.$$

Applying the matrix inverse from before yields an explicit formula (with a and b defined as before):

$$x_{n+1}=\frac{1}{a}\left[z_n-\frac{1^t{z}_n}{p-(a/b)}1\right];\hspace{1em}{z}_n=\sigma +\rho D^t𝒫\left(D{x}_n\right),a=1+\rho p\left(c^2+1\right),b=2\rho c.$$

G.2. Steepest Descent

The updates x_n+1 = x_n − t_n∇h_ρ(x_n) admit an exact solution for the line search parameter t_n. Taking q_n = ∇h_ρ(x_n) as the gradient we have

$$\begin{array}{l}{q}_n=\left(x_n-u\right)+\rho D^t\left[D{x}_n-{𝒫}_{\nu }\left(D{x}_n\right)\right],\\ t_n=\frac{{\Vert q_n\Vert }^2}{{\Vert q_n\Vert }^2+\rho {\Vert D{q}_n\Vert }^2}.\end{array}$$

G.3. ADMM

Take y as the dual variable and λ as scaled multipliers. The formula for the MM algorithm applies in updating x_n, except we replace ρ with μ and (Dx_n) with y_n − λ_n:

$$\begin{array}{ll}{x}_{n+1}=\frac{1}{a}\left[z_n^1-\frac{1^t{z}_n^1}{p-(a/b)}1\right];\hfill & z_n^1=\sigma +\mu D^t\left(y_n-{\lambda }_n\right),a=1+\mu p\left(c^2+1\right),b=2\mu c\hfill \\ y_{n+1}=\frac{\alpha }{1+\alpha }𝒫\left(z_n^2\right)+\frac{1}{1+\alpha }{z}_n^2;\hfill & z_n^2=D{x}_{n+1}+{\lambda }_n,\alpha =\rho /\mu .\hfill \end{array}$$

Multipliers follow the standard update.

G.4. Explicit Matrix Inverse

Both ADMM and MM reduce to solving a linear system. Fortunately, the Hessian for h_ρ(x) reduces to a Householder-like matrix. First we note that it is trivial to multiply either C^t or S^t by a p²-vector. The more interesting problem is calculating ${\left(A_{\rho }^t{A}_{\rho }\right)}^{-1}$, where $\nabla h_{\rho }^2=A_{\rho }^t{A}_{\rho }=I_p+\rho D^{t}D$. The reader can check the identities

$$\begin{array}{l}{C}^{t}C=c^{2}p{I}_p\text{and}{C}^{t}S=-c{1}_p{1}_p^t,\\ S^{t}C=-c{1}_p{1}_p^t\text{and}{S}^{t}S=p{I}_p.\end{array}$$

It follows that $A_{\rho }^t{A}_{\rho }=\left(1+\rho p\left(c^2+1\right)\right)I_p-2c\rho 1_p{1}_p^t$. Applying the Sherman-Morrison formula to results in

$$\begin{array}{l}{\left[I_p+\rho D^{t}D\right]}^{-1}={\left[a{I}_p-b{1}_p{1}_p^t\right]}^{-1}\\ =-b^{-1}\left[-(b/a)I_p-\frac{{(b/a)}^2{1}_p{1}_p^t}{1-(a/b)1_p^t{1}_p}\right]\\ =\frac{1}{a}\left[I_p-\frac{1_p{1}_p^t}{p-a/b}\right],\end{array}$$

where a = 1 + ρp(c² + 1) and b = 2ρc. These simplifications make the exact proximal distance updates easy to compute.

Table 7:

Performance of MM on convex regression using CG and LSQR. Both inner and outer iterations are reported with the latter in parentheses.

		Time (s)			Loss ×106			Distance ×106		Iterations
features	samples	CG	LSQR		CG		LSQR	CG	LSQR	CG	LSQR
20	50	0.39 (0.00132)		0.68 (0.00607)	0.991	0.991		2.7	2.65	104 (2)	104 (2)
20	100	1.46 (0.000164)		2.67 (0.00555)	0.995	0.994		3.08	3.16	190 (2)	190 (2)
20	200	7.7 (0.166)		14.7 (0.00913)	0.984	0.984		25.1	25.1	298 (2)	298 (2)
20	400	30.2 (0.0275)		63 (0.268)	0.997	0.997		9.21	8.89	412 (2)	412 (2)

Table 8:

Performance of ADMM on convex regression using CG and LSQR. Both inner and outer iterations are reported with the latter in parentheses.

		Time (s)		Loss ×106		Distance ×106		Iterations
features	samples	CG	LSQR	CG	LSQR	CG	LSQR	CG	LSQR
20	50	0.399 (0.00193)	0.677 (0.00507)	0.91	0.91	6.96	6.96	98 (2)	98 (2)
20	100	1.58 (0.0329)	2.83 (0.00398)	0.996	0.996	0	0	194 (2)	194 (2)
20	200	7.78 (0.0105)	15 (0.0386)	0.961	0.961	43.8	44	296 (2)	296 (2)
20	400	30 (0.0157)	60.4 (0.0841)	1.42	1.45	64.6	67.8	377 (2)	376 (2)

Appendix H. Choice of Linear Solver

Updating parameters using MM or ADMM requires solving large-scale linear systems of the form (I + c_tD^tD)x = b. Here c_t is a scalar that depends on the outer iteration number t, in general, and the matrix on the LHS is square, symmetric, and often reasonably well-conditioned. Standard factorization methods like Cholesky and spectral decompositions cannot be applied without efficient update rules based on c_t. Instead, we turn to iterative methods, specifically conjugate gradients (CG) and LSQR, and use a linear map approach to adequately address sparsity, structure, and computational efficiency in matrix-vector multiplication. Tables 7 and 8 summarize performance metrics for MM and ADMM using both iterative linear solvers on instances of the convex regression problem. Times are averages taken over 3 replicates with standard deviations in parentheses, and iteration counts reflect the total number of inner iterations with outer counts in parentheses. We find no appreciable difference between CG and LSQR except on timing, and therefore favor CG in all our benchmarks.

Appendix I. Software & Computing Environment

Code for our implementations and numerical experiments is available at https://github.com/alanderos91/ProximalDistanceAlgorithms.jl and is based on the Julia language (Bezanson et al., 2017). Additional packages used include Plots.jl (Breloff, 2021), GR.jl (Heinen et al., 2021), and (Udell et al., 2014). Numerical experiments were carried out on a Manjaro Linux 5.10.89–1 desktop environment using 8 cores on an Intel 10900KF at 4.9 GHz and 32 GB RAM.